Architecting Fast Data Applications - TechArch Day · Kafka Storage Akka Streams Kafka Streams......

transcript

TechArch Day 2018

Architecting Fast Data Applications

Gerard Maas

Architecting Fast Data Applications

TechArch Day 2018 - Helsinki, October 3, 2018

Gerard MaasSenior SW Engineer, Lightbend, Inc.

Gerard MaasSeñor SW Engineer

gerard.maas@lightbend.com

@maasg

https://github.com/maasg

https://www.linkedin.com/in/gerardmaas/

https://stackoverflow.com/users/764040/maasg

Why Streaming?

Fast Data Architectures

((( Quick Demo )))

The Future of Fast Data

Why Streaming?

Fast Data Use Cases

Real-time marketing based on behavior, location, inventory levels, product promotions, etc.

Real-time Personalization

Drive better business outcomes through real-time risk, fraud detection, compliance, audit, governance, etc.

Real-time Financial ProcessesPredictive Analytics

Apply ML models to large volumes of device data to pre-empt failures / outages

Real-time consumer and industrial Device and Supply Chain management at scale

https://www.lightbend.com/customers

Fast Data Use Cases

Real-time marketing based on behavior, location, inventory levels, product promotions, etc.

Real-time Personalization

Drive better business outcomes through real-time risk, fraud detection, compliance, audit, governance, etc.

Real-time Financial ProcessesPredictive Analytics

Apply ML models to large volumes of device data to pre-empt failures / outages

Real-time consumer and industrial Device and Supply Chain management at scale

https://www.lightbend.com/customers

•ML models applied to device telemetry to detect anomalies

•Preemptive maintenance prevents potential failures that would impact users

Predictive Analytics

Predictive Analytics - Core Idea

AnomalyHandler

TelemetryRecords

Probable Anomalies

CorrectiveActions

Anomaly Detection:

Predictive Analytics - Core Idea

AnomalyHandler

TelemetryRecords

Probable Anomalies

CorrectiveActions

Ingest telemetry from edge devices.

Train models to look for anomalies… and score

incoming telemetry.

Handle anomaly: move activity off component, schedule maintenance window to replace it.

Anomaly Detection

microservicesmicroservicesMicroservice

Storage

Akka Streams

Kafka Streams

Low LatencyMicroservices

Data Center

Streaming, Batch Processing

Device SessionMicroservices

Example Architecture

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Session Mngmt,Restful msvcs

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Model Scoring

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Model Training

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Ingest Device Telemetry

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Periodically Train Anomaly Detection Model

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Updated model parameters are written back to Kafka

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Updated model parameters are also written to secondary storage for resilience

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Ingest model parameters in the low-latency microservice for serving

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Ingest telemetry data to score it, looking for anomalies.

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Write detected anomalies back to Kafka in a new topic

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Read anomaly information into microservices that manage the devices remotely

Storage

Akka Streams

Kafka Streams

Data Center

ModelTraining

ModelStorage

ModelServing

ScoresIngestion

Telemetry

CorrectiveAction

Take corrective action, e.g., download patches, disable, reset, ...

Fast Data Architectures

Requirements?● Latency. How Low?● Throughput. How High?● Which kind of data processing?● How do you want to build, deploy,

and manage those services?

Kubernetes, Mesos, YARN

Cloud | On-prem

On-Prem Cloud

Substrate

Cloud | On-prem

Substrate Messaging Backbone

Cloud | On-prem

Streams Streams

Substrate Messaging Backbone ProcessingEngine

Stream ProcMicroservices

Akka Streams

Kafka Streams

Cloud | On-prem

Substrate Messaging Backbone ProcessingEngine

Microservices

Storage

Stream Proc

Reactive Platform

Go node.js

Microservices

Akka Streams

Kafka Streams

SQL NoSQL

Cloud | On-prem

Substrate Messaging Backbone ProcessingEngineMicroservices Storage

Storage

Stream Proc

Reactive Platform

Go node.js

Microservices

Akka Streams

Kafka Streams

SQL NoSQL

Cloud | On-prem

Substrate Messaging Backbone ProcessingEngineMicroservices Storage Monitoring &Management

Storage

Stream Proc

Reactive Platform

Go node.js

Microservices

Akka Streams

Kafka Streams

SQL NoSQL

Sockets

Cloud | On-prem

((( Quick Demo )))

Aggregator records

Call-Record-Aggregatordatamodelakka-cdr-ingestor

akka-java-aggr-outspark-aggregation

Call-Record-Aggregatordatamodelakka-cdr-ingestor

akka-java-aggr-outspark-aggregation

HTTPIngress

Aggregator

ConsoleOut

The Future of Fast Data

ReactiveSystems

FastData

Applications

BigData

Microservices

The Evolution of Current Trends

Growing number of workloads are moving from “data at rest” to “data in motion”

Streaming data pipelines are being served by Microservices

Microservices Stream Processing

ContainerizationOrchestration

"This is what makes time travel possible." —Doc Brown

Img src: http://backtothefuture.wikia.com/wiki/Flux_capacitor

The Flux Capacitor

Microservices are a part of a streaming pipeline

A pipeline can now be exposed as Microservices

Packaged, deployed and managed as cloud-native applications with the right tools

lightbend.com/fast-data-platform

Learn more

lightbend.com/fast-data-platform

Thank you

Architecting Fast Data Applications - TechArch Day · Kafka Storage Akka Streams Kafka Streams......

Documents