Streaming Patterns Revolutionary Architectures with the Kafka API

transcript

Streaming Patterns, Revolutionary Architectures Carol McDonald

Agenda Streams Core Components

•  Topics, Partitions •  Fault Tolerance •  High Availability

Patterns •  Event Sourcing •  Duality of Streams and Databases •  Command Query Responsibility Separation •  Polyglot Persistence, Multiple Materialized Views •  Turning the Database Upside Down

Real World Examples •  Fraud Detection •  Healthcare Exchange

Which products are we discussing?

Streams Core Components

What’s a Stream ?

Producers Consumers Events_Stream

A stream is an unbounded sequence of events carried from a set of producers to a set of consumers.

Events

What is Streaming Data? Got Some Examples?

Data Collection Devices

Smart Machinery Phones and Tablets Home Automation

RFID Systems Digital Signage Security Systems Medical Devices

Why Streams?

Trigger Events: •  Stock Prices •  User Activity •  Sensor Data

Many Big Data sources are Event Oriented

Stream Stream Stream

Event Data

Topic Topic

Real-Time Analytics

Analyze Data What if you need to analyze data as it arrives?

It was hot at 6:05

yesterday!

Batch Processing with HDFS

Analyze

6:01 P.M.: 72° 6:02 P.M.: 75° 6:03 P.M.: 77° 6:04 P.M.: 85° 6:05 P.M.: 90° 6:06 P.M.: 85° 6:07 P.M.: 77° 6:08 P.M.: 75°

90° 90° 6:01 P.M.: 72° 6:02 P.M.: 75° 6:03 P.M.: 77° 6:04 P.M.: 85° 6:05 P.M.: 90° 6:06 P.M.: 85° 6:07 P.M.: 77° 6:08 P.M.: 75°

Event Processing with Streams

6:05 P.M.: 90° Topic

Stream

Temperature

Turn on the air conditioning!

Organize Data What if you need to organize data as it arrives?

Integrating Many Data Sources and Applications

Sources (Producers)

Applications (Consumers)

Unorganized, Complicated, and Tightly Coupled.

Organize Data into Topics with MapR Streams Topics Organize Events into Categories and Decouple Producers from Consumers

Consumers

MapR Cluster

Topic: Pressure

Topic: Temperature

Topic: Warnings

Consumers

Kafka API Kafka API

Process High Volume of Data What if you need to process a high volume of data as it arrives?

What if BP had detected problems before the oil hit the water ?

•  1M samples/sec •  High performance at

scale is necessary!

Legacy Messaging

Millions of Sources

Hundreds of Destinations insert

Legacy Message Queue:

Message rate <100K/s

Publish Acks

delete

Consume Acks

Mechanisms for Decoupling Traditional message queues? •  Huge performance hit for persistence:

•  message acknowledgement per message per consumer •  Lots of Non sequential disk I/O when messages added/removed

Scalable Messaging with MapR Streams

Server 1

Partition1: Topic - Pressure Partition1: Topic - Temperature Partition1: Topic - Warning

Server 2

Server 3

Topics are partitioned for throughput and scalability

Producers are load balanced between partitions

Kafka API

Consumers

Consumer groups can read in parallel

Kafka API

Core Components: Partitions

Consumers

MapR Cluster

Topic: Admission / Server 1

Consumers

Partition

Partitions: –  Messages are

appended in order

Offset: –  Sequential id of a

message in a partition Partition

Partition

6 5 4 3 2 1

5 4 3 2 1

Producers

New Message

6 5 4 3 2 1 Old

Message

Read Cursors •  Read cursor: offset ID of most recent read message •  Producers Append New messages to tail •  Consumers Read from head

MapR Cluster

6 5 4 3 2 1 Consumer

group Producers

Read cursors

Consumer group

Consumers

MapR Cluster

Consumers

Partition

6 5 4 3 2 1

5 4 3 2 1

Producers

Events are delivered in the order they are received, like a queue. Partitioned, Sequential Access = High Performance

New Message

6 5 4 3 2 1 Old

Message

Unlike a queue, events are persisted even after they’re delivered

Messages remain on the partition, available to other consumers Minimizes Non-Sequential disk read-writes

MapR Cluster (1 Server)

Topic: Warning

Partition 1

3 2 1 Unread Events

Get Unread

Client Library Consumer Poll

Considering a Messaging Platform Kafka-esque Logs?

•  Sequential writing/reading disk: •  Messages are persisted sequentially as produced, and read sequentially when consumed •  Performance plus persistence •  performance of up to a billion messages per second at millisecond-level delivery times.

Kafka model is BLAZING fast

•  Kafka 0.9 API with message sizes at 200 bytes •  MapR Streams on a 5 node cluster sustained 18 million events / sec •  Throughput of 3.5GB/s and over 1.5 trillion events / day

When Are Messages Deleted? •  Messages can be persisted forever Or •  Older messages can be deleted automatically based on time to live

MapR Cluster (1 Server)

6 5 4 3 2 1 Partition 1

Older message

Parallelism When Reading To read messages from the same Topic in parallel: •  create consumer groups •  consumers with same group.id •  partitions assigned dynamically round-robin

Consumer group: Oil Wells

Consumer A

Consumer B

Consumer C

MapR Cluster

Partition 4: Warning

Fault Tolerance Consumption: Partitions Re-Assigned Dynamically If consumer goes offline, partitions re-assigned

Consumer group.id: Oil Wells

Consumer A

Consumer C

MapR Cluster

Partition4: Warning

Partition3: Warning

Partition2: Warning

Partition1: Warning

Partition5: Warning

Processing Same Message for Different Views

Consumers

Producers

MapR-FS

Kafka API Kafka API

Pub Sub: Multiple Consumers, Multiple Destinations

Partition Fault Tolerance

Message Recovery What if you need to recover messages in case of server failure?

Partitions are Replicated for Fault Tolerance

Producer

Server 2 Partition2: Topic - Warning

Producer

Server 2

Server 3

Server 1

Server 3

Server 1

Server 2

Partition1: Warning Partition2: Warning Replica Partition3: Warning Replica

Partition1: Warning Replica

Partition1: Warning Replica Partition2: Warning Replica Partition3: Warning

Producer

Server 1

Server 2

Server 3

Security Investigation & Event Management

Operational Intelligence

Real-time Analytics

Partition2: Warning

Producer

Real-time Analytics

Server 1

Server 2

Server 3

Partition2: Warning

Partitions are Replicated for Fault tolerance

Producer

Real-time Analytics

Server 1

Server 2

Server 3

Partition2: Warning

Streams and High Availability

•  Stream: –  collection of topics managed together

•  Manage stream: –  replication –  security –  time-to-live –  number of partitions

Core Components: Streams

Stream

Pressure Temperature Warning

Stream

Consumers

Producers

Replication

Real-time Access What if you need real-time access to live data distributed across multiple clusters and multiple data centers?

Lack of Global Replication

Topic: C

Streams and Replication Streams:

•  are a collection of topics •  can be replicated worldwide

Topic: A

Topic: B

Topic: C

Topic: A

Topic: B

Topic: C

Replicating to another cluster

Streams and Replication

Topic: A

Topic: B

Topic: C

Fail Over

Streams: •  high availability •  disaster recovery

Replicating Streams: Master-Slave Replication

Venezuela_HA Cluster

Metrics Stream

Metrics Producers

Venezuela Cluster

Metrics Stream

Metrics

Consumers

High Availabiltiy Backup for Venezula

Master Slave

Replicating Streams: Many-to-One Replication

Houston

Metrics Stream

Metrics

Producers Venezuela

Metrics Stream

Metrics Consumers

Consumers

Producers Mexico

Metrics Stream

Metrics Consumers Analyze all data from Houston

Replicating Streams: Multi-Master Replication

Producers Seoul

Metrics Stream

Metrics Consumers

Producers San Francisco

Metrics Stream

Metrics Consumers

Both send and receive updates

Stream Replication

Stream

Ship picks up containers…

Singapore

Arrives at destination…

While enroute to next destination…

Washington

Where does the data live…

Singapore Washington

What is important about this? Data is generated on the ship

•  Must have an easy way (i.e. foolproof) to move the data off the ship

Each port stores the data from the ship

•  Moving data between locations •  Analytics could happen at any location

This is a multi-data center time series data use case

•  Events from sensors = metrics •  Same concepts as data center monitoring

Patterns

Event Sourcing

Updates

Imagine each event as a change to an entry in a database.

Account Id Balance WillO 80.00 BradA 20.00

1: WillO : Deposit : 100.00 2: BradA : Deposit : 50.00 3: BradA : Withdraw : 30.00 4: WillO : Withdraw: 20.00

https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Change log

4 3 2 1

credit, debit events

current account balances

Replication

Change Log

https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

3 2 1 3 2 1 3 2 1

Duality of Streams and Tables: Database: captures data at rest Stream: captures data change

Master: Append writes

Slave: Apply writes in order

Which Makes a Better System of Record?

Which of these can be used to reconstruct the other?

Account Id Balance WillO 80.00 BradA 20.00

Change Log 3 2 1

Rewind: Reprocessing Events

MapR Cluster

6 5 4 3 2 1 Producers

Reprocess from oldest message

Consumer

Create new view, Index, cache

Rewind Reprocessing Events

MapR Cluster

6 5 4 3 2 1 Producers

To Newest message

Consumer

new view

Read from new view

Event Sourcing, Command Query Responsibility Separation: Turning the Database Upside Down

Key-Val Document Graph

Wide Column

Time Series Relational

??? Events Updates

What Else Do I Use My Stream For?

Lineage - “how did BradA’s balance get so low?” Auditing - “who deposited/withdrew from BradA’s account?” History – to see the status of the accounts last year Integrity - “can I trust this data hasn’t been tampered with?”

•  Yup - Streams are immutable

What Do I Need For This to Work?

Infinitely persisted events A way to query your persisted stream data An integrated security model across the stream and databases

Fraud Detection

Point of Sale -> Data Center is Transaction Fraud ? •  Lots of requests •  Need answer within ~ 50 100 milliseconds

Data Center

Point of Sale

Location, time, card#

Fraud yes/no ?

Traditional Solution

POS 1..n

Fraud detector

Last card use

1.  Look up last card use 2.  Compute the card velocity:

•  Subtract last location, time from current location, time

3.  Update last card use

What Happens Next?

POS 1..n

Fraud detector

Last card use

POS 1..n

Fraud detector

POS 1..n

Fraud detector

1.  Look up last card use 2.  Compute the card velocity 3.  Update last card use

Bottleneck !

Service Isolation: Separate Read from Write

POS 1..n

Fraud detector

Last card use

Updater

card activity

Read last card use

Separate Read Model from the Write Model: Command Query Responsibility Separation

POS 1..n

Fraud detector

Last card use

Updater

card activity

Event last card use

Write last card use

Event Sourcing: New Uses of Data Processing Same Message for Multiple Views

POS 1..n

Fraud detector

Last card use

Updater

Card location history

card activity

Scaling Through Isolation allows Multiple Consumers

POS 1..n

Last card use

Updater

POS 1..n

Last card use

Updater

card activity

Fraud detector

Multiple fraud detectors can use the same message queue

•  De-coupling and isolation are key

•  Propagate events, not table updates

Decoupled Architecture

Producer

Activity Handler

Producer

Producer Historical

Interesting Data Real-time

Analysis

Results Dashboard

Anomaly Detection

more than one component can make use of the same stream of messages for a variety of uses

Lessons De-coupling and isolation are key Propagate events, not table updates

Building Enterprise Software vs Internet Companies

Enterprise Software: Complexity of domain => Business logic, Business rules Banking, Healthcare, Telecom Compliance=> Security

Internet Companies: Volume of data => Complex data infrastructure Large Scale Availability, Recovery

Reference Martin Kleppmann

Building Enterprise Software vs Internet Companies

Enterprise Software: Event Sourcing

Internet Companies: Stream Processing

Reference Martin Kleppmann

Real World Solution

Credit Card Fraud Model Building

Serve NoSQL Storage Data Ingest

Fraud Stream Processing Architecture

Stream Processing Source

MapR-FS

MapR-DB

Topic: A

Topic: B

Topic: C

Topic: A

Topic: B

Topic: C

Streams Messaging

Fraud Processing

Stream Processing

Derive features

enriched

alerts

process

Batch Processing

MapR-FS

MapR-DB

enriched

alerts

build model update model

Streams Messaging

Fraud Event Processing

Stream Processing

NoSQL Storage

MapR-FS

MapR-DB

Enriched

1.  Parse raw event 2.  read card holder

profile from MapR-DB 3.  Derive features 4.  Get prediction from

model with features 5.  Publish not fraud to

enriched topic 6.  Publish fraud to

fraud topic

Fraud Processing Same Message for Different Views

Partition1: Topic – Raw Trans Partition1: Topic – Enriched Partition1: Topic – Fraud Alert

Partition2: Topic – Raw Trans Partition2: Topic - Enriched Partition2: Topic – Fraud Alert

Partition3: Topic – Raw Trans Partition3: Topic - Enriched Partition3: Topic – Fraud Alert

Consumers MapR-FS

MapR-DB

Consumers

Consumers MapR-FS

MapR-DB

Consumers

Consumers MapR-FS

MapR-DB

Consumers

Real World Solution

JSON DB (MapR-DB)

Graph DB (Titan on

MapR-DB)

Search Engine (Elastic-Search)

Transforming the Health Care Ecosystem

Electronic Medical Records

“The Stream is the System of Record” –Brad Anderson VP Big Data Informatics

Liaison ALLOY™ Platform

Data Integration

ingest syndicatetransform

Data Management

masterdeduplicateharmonize

relatemerge

tokenize

store / persist

analyzesummarize

reportdistill

recommend

explorequery

sandboxbatch transform

learntraverse

Use Case: Streaming System of Record for Healthcare

Objective: •  Build a flexible, secure

healthcare exchange

Records Analysis Applications

Challenges: •  Many different data models •  Security and privacy issues •  HIPAA compliance

Records

ALLOY Health: Exchange State HIE

Clinical Data Viewer

Analytics queries like: What are the outcomes in the entire state on diabetes? Are there doctors that are doing this better than others?

Clinical Data

Financial Data Provider

Organizations

OrdersAnywhere PORTAL (no EHR)

EHR with HL7 ONLY

EHR with WORKFLOW INTEGRATION

RADIOLOGY

This is a PAIN !

COMPLIANCE

SECURITY CONTROLS

COMPLIANCE FEATURES

PRIVACY

PCI DSS 3.0

21 CFR Part 11

SSAE16 / SOC2

HIPAA/HITECH

WHY NOW?

84 http://bit.ly/29aBatK

WHY NOW?

2014 FQ4 profit

$ -440 M Total Cost Estimate

$ -12 B

Why Now? The Relational database is not the only tool

Attribute Value

patient_id 1234

Name Jon Smith

Age 50

Attribute Value

patient_id 999

Name Jonathan Smith

DOB Jun 1965

Attribute Value

provider_id 86

Name Dr. Nora Paige

Specialty Diabetes

Attribute Value

rx_id 9876

Name Sitagliptin

Dosage 325mg

Visited

Prescribed

WasPrescribed

Patient

Prescription

Provider

Context and Relationships

WHY NOW? Mind the Gap

Streaming System of Record for Healthcare

Stream

Records

Applications

6 5 4 3 2 1

Search

Graph DB

Micro Service

Streaming System of Record Materialized Views

89 Immutable Log

Raw Data

workflow

Key/Value (MapR-DB)

materialized

workflow

Search Engine

materialized

k v v v v v

k v v v

k v v v v

k v v v

k v v v v v

Document Log (MapR-FS)

pre-processor

workflow

Graph (ArangoDB)

materialized

workflow

Time Series

(OpenTSDB)

materialized view

micro service

App App App

The Promised Land Compliance Auditor

The Promised Land

Auditor smiley faces •  Data Lineage •  Audit Logging •  Wire-level encryption •  At Rest encryption

Replication

•  Disaster Recovery •  EU – data can’t leave

Non-Stream / Non-”Big Data” •  Software Development Lifecycle •  System Hardening •  Separation of Concerns

-  Dev vs Ops •  Patch Management

Compliance Auditor

Solution Design/architecture solved some

•  Streams •  Data Lineage/System of Record •  Kappa Architecture (Kreps/Kleppman)

MapR solved others •  Unified Security •  Replication DC to DC •  Converge Kafka/HBase/Hadoop to one cluster •  Multi-tenancy (lots of topics, for lots of tenants)

Sample Producer: All Together public class SampleProducer { String topic=“/streams/pump:warning”; public static KafkaProducer producer; public static void main(String[] args) { producer=setUpProducer(); for(int i = 0; i < 3; i++) { String txt = “msg ” + i; ProducerRecord<String, String> rec = new ProducerRecord<String, String>(topic, txt); producer.send(rec); System.out.println("Sent msg number " + i); } producer.close(); }

public class MyConsumer { public static String topic = "/stream/pump:warning”; public static KafkaConsumer consumer; public static void main(String[] args) { configureConsumer(args); consumer.subscribe(topic); while (true) { ConsumerRecords<String, String> msg= consumer.poll(pollTimeOut); Iterator<ConsumerRecord<String, String>> iter = msg.iterator(); while (iter.hasNext()) { ConsumerRecord<String, String> record = iter.next(); System.out.println(”read " + record.toString()); } } consumer.close(); } }

Sample Consumer: All Together

Summary

Can we get “Extreme” ?

1+ Trillion Events

•  per day Millions of Producers

•  Billions of events per second Multiple Consumers

•  Potentially for every event Multiple Data Centers

•  Plan for success •  Plan for drastic failure

Think that is crazy? Consider having 100 servers and performing: Monitoring and Application logs…

•  100 metrics per server •  60 samples per minute •  50 metrics per request •  1,000 log entries per request (abnormally

small, depends on level) •  1million requests per day

~ 2 billion events per day, for one small (ish) use case

Extreme Average Reality

Stream Processing

Building a Complete Data Architecture

MapR File System (MapR-FS)

MapR Converged Data Platform

MapR Database (MapR-DB) MapR Streams

Sources/Apps Bulk Processing

bit.ly/jjug-aug2016 Find my slides & other related materials to this talk here:

or search:

MapR Blog

• https://www.mapr.com/blog/

…helping you put data technology to work

●  Find answers

●  Ask technical questions

●  Join on-demand training course discussions

●  Follow release announcements

●  Share and vote on product ideas

●  Find Meetup and event listings

Connect with fellow Apache Hadoop and Spark professionals

community.mapr.com