Streaming Analytics with Apache Flink - Meetupfiles.meetup.com/18824486/Flink @ DC Flink...

transcript

Stephan Ewen

@stephanewen

Streaming Analyticswith Apache Flink

Apache Flink Stack

DataStream APIStream Processing

DataSet APIBatch Processing

RuntimeDistributed Streaming Data Flow

Libraries

Streaming and batch as first class citizens.

DataStream APIStream Processing

DataSet APIBatch Processing

RuntimeDistributed Streaming Data Flow

Libraries

Streaming is the next programming paradigmfor data applications, and you need to start

thinking in terms of streams.

Streaming technology is enabling the obvious: continuous processing on data that is

continuously produced

A brief History of Flink

January ‘10 December ‘14

v0.5 v0.6 v0.7

March ‘16

Flink Project

Incubation

Top Level

Project

v0.8 v0.10

Release

Project

Stratosphere

(Flink precursor)

April ‘14

A brief History of Flink

January ‘10 December ‘14

v0.5 v0.6 v0.7

March ‘16

Flink Project

Incubation

Top Level

Project

v0.8 v0.10

Release

Project

Stratosphere

(Flink precursor)

April ‘14

The academia gap:Reading/writing papers,

teaching, worrying about thesis

Realizing this might be interesting to people

beyond academia(even more so, actually)

A Stream Processing Pipeline

collect log analyze serve

Programs and Dataflows

Source

Transformation

val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…))

val events: DataStream[Event] = lines.map((line) => parse(line))

val stats: DataStream[Statistic] = stream.keyBy("sensor").timeWindow(Time.seconds(5)).sum(new MyAggregationFunction())

stats.addSink(new RollingSink(path))

Source[1]

map()[1]

keyBy()/window()/

apply()[1]

Sink[1]

Source[2]

map()[2]

keyBy()/window()/

apply()[2]

StreamingDataflow

Why does Flink stream flink?

Low latency

High Throughput

Well-behavedflow control

(back pressure)

Make more sense of data

Works on real-timeand historic data

TrueStreaming

Event Time

APIsLibraries

StatefulStreaming

Globally consistentsavepoints

Exactly-once semanticsfor fault tolerance

Windows &user-defined state

Flexible windows(time, count, session, roll-your own)

Complex Event Processing

Counting

Continuous counting

A seemingly simple application, but generally an unsolved problem

E.g., count visitors, impressions, interactions, clicks, etc

Aggregations and OLAP cube operations are generalizations of counting

Counting in batch architecture

Continuous ingestion

Periodic (e.g., hourly) files

Periodic batch jobs

Problems with batch architecture

High latency

Too many moving parts

Implicit treatment of time

Out of order event handling

Implicit batch boundaries

Counting in λ architecture

"Batch layer": what we had before

"Stream layer": approximate early results

Problems with batch and λ

Way too many moving parts (and code dup)

Implicit treatment of time

Out of order event handling

Implicit batch boundaries

Counting in streaming architecture

Message queue ensures stream durability and replay

Stream processor ensures consistent counting

Counting in Flink DataStream API

Number of visitors in last hour by country

DataStream<LogEvent> stream = env.addSource(new FlinkKafkaConsumer(...)) // create stream from Kafka .keyBy("country") // group by country .timeWindow(Time.minutes(60)) // window of size 1 hour .apply(new CountPerWindowFunction()); // do operations per window

Counting hierarchy of needs

Continuous counting

... with low latency,

... efficiently on high volume streams,

... fault tolerant (exactly once),

... accurate and

repeatable,

... queryable

Based on Maslow's

hierarchy of needs

Continuous counting

... accurate and

repeatable,

Continuous counting

... accurate and

repeatable,

... queryable1.1+

Rest of this talk

Continuous counting

... accurate and

repeatable,

... queryable

Streaming Analytics by Example

Time-Windowed Aggregations

case class Event(sensor: String, measure: Double)

val env = StreamExecutionEnvironment.getExecutionEnvironment

val stream: DataStream[Event] = env.addSource(…)

stream.keyBy("sensor").timeWindow(Time.seconds(5)).sum("measure")

Time-Windowed Aggregations

stream.keyBy("sensor").timeWindow(Time.seconds(60), Time.seconds(5)).sum("measure")

Session-Windowed Aggregations

stream.keyBy("sensor").window(EventTimeSessionWindows.withGap(Time.seconds(60))).max("measure")

Session-Windowed Aggregations

stream.keyBy("sensor").window(EventTimeSessionWindows.withGap(Time.seconds(60))).max("measure")

Flink 1.1 syntax

Pattern Detection

case class Event(producer: String, evtType: Int, msg: String)case class Alert(msg: String)

val stream: DataStream[Event] = env.addSource(…)stream

.keyBy("producer")

.flatMap(new RichFlatMapFuncion[Event, Alert]() {

lazy val state: ValueState[Int] = getRuntimeContext.getState(…)

def flatMap(event: Event, out: Collector[Alert]) = {val newState = state.value() match {

case 0 if (event.evtType == 0) => 1case 1 if (event.evtType == 1) => 0case x => out.collect(Alert(event.msg, x)); 0

}state.update(newState)

Pattern Detection

case class Event(producer: String, evtType: Int, msg: String)case class Alert(msg: String)

val stream: DataStream[Event] = env.addSource(…)stream

.keyBy("producer")

.flatMap(new RichFlatMapFuncion[Event, Alert]() {

lazy val state: ValueState[Int] = getRuntimeContext.getState(…)

def flatMap(event: Event, out: Collector[Alert]) = {val newState = state.value() match {

case 0 if (event.evtType == 0) => 1case 1 if (event.evtType == 1) => 0case x => out.collect(Alert(event.msg, x)); 0

}state.update(newState)

Embedded key/valuestate store

Many more

Joining streams (e.g. combine readings from sensor)

Detecting Patterns (CEP)

Applying (changing) rules or models to events

Training and applying online machine learning models

(It's) About Time

The biggest change in moving frombatch to streaming is

handling time explicitly

Example: Windowing by Time

case class Event(id: String, measure: Double, timestamp: Long)

stream.keyBy("id").timeWindow(Time.seconds(15), Time.seconds(5)).sum("measure")

Example: Windowing by Time

Different Notions of Time

Event Producer Message QueueFlink

Data SourceFlink

Window Operator

partition 1

partition 2

EventTime

IngestionTime

WindowProcessing

1977 1980 1983 1999 2002 2005 2015

Processing Time

Episode

Event Time

Event Time vs. Processing Time

Processing Time

val env = StreamExecutionEnvironment.getExecutionEnvironmentenv.setStreamTimeCharacteristic(ProcessingTime)

Window by operator's processing time

Ingestion Time

val env = StreamExecutionEnvironment.getExecutionEnvironmentenv.setStreamTimeCharacteristic(IngestionTime)

Event Time

val env = StreamExecutionEnvironment.getExecutionEnvironmentenv.setStreamTimeCharacteristic(EventTime)

Event Time

val stream: DataStream[Event] = env.addSource(…)val tsStream = stream.assignAscendingTimestamps(_.timestamp)

tsStream.keyBy("id").timeWindow(Time.seconds(15), Time.seconds(5)).sum("measure")

Event Time

val stream: DataStream[Event] = env.addSource(…)val tsStream = stream.assignTimestampsAndWatermarks(

new MyTimestampsAndWatermarkGenerator())

tsStream.keyBy("id").timeWindow(Time.seconds(15), Time.seconds(5)).sum("measure")

Watermarks

W(11)W(17)

11159121417122220 171921

WatermarkEvent

Event timestamp

Stream (in order)

W(11)W(20)

Watermark

991011141517

Event timestamp

1820 192123

Stream (out of order)

Watermarks in Parallel

Source(1)

Source(2)

map(1)

map(2)

window(1)

window(2)

A|30B|31

F|15G|18H|20

Watermark

Event Timeat the operator

Event[id|timestamp]

Event Timeat input streams

WatermarkGeneration

M|39N|39Q|44

L|22O|23R|37

Per Kafka Partition Watermarks

Source(1)

Source(2)

map(1)

map(2)

window(1)

window(2)

A|30B|73

F|15G|91H|94

WatermarkGeneration

L|35N|39

O|97 M|89

I|21Q|23

T|99 S|97

Per Kafka Partition Watermarks

val kafka = new FlinkKafkaConsumer09(topic, schema, props)kafka.assignTimestampsAndWatermarks(

new MyTimestampsAndWatermarkGenerator())

val stream: DataStream[Event] = env.addSource(kafka)stream

.keyBy("id")

.timeWindow(Time.seconds(15), Time.seconds(5))

.sum("measure")

Matters of State(Fault Tolerance, Reinstatements, etc)

Back to the Aggregation Example

val stream: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(topic, schema, properties))

Stateful

Fault Tolerance

Prevent data loss (reprocess lost in-flight events)

Recover state consistency (exactly-once semantics)• Pending windows & user-defined (key/value) state

Checkpoint based fault tolerance• Periodicaly create checkpoints

• Recovery: resume from last completed checkpoint

• Async. Barrier Snapshots (ABS) Algorithm

Checkpoints

data stream

newer records older records

State of the dataflowat point Y

State of the dataflowat point X

Checkpoint Barriers

Markers, injected into the streams

Checkpoint Procedure

Savepoints

A "Checkpoint" is a globally consistent point-in-time snapshot of the streaming program (point in stream, state)

A "Savepoint" is a user-triggered retained checkpoint

Streaming programs can start from a savepoint

Savepoint B Savepoint A

(Re)processing data (in batch)

Re-processing data (what-if exploration, to correct bugs, etc.)

Usually by running a batch job with a set of old files

Tools that map files to times

2016-3-112:00 am

2016-3-11:00 am

2016-3-12:00 am

2016-3-1111:00pm

2016-3-1212:00am

2016-3-121:00am…

Collection of files, by ingestion time

2016-3-1110:00pm

To the batchprocessor

Unclear Batch Boundaries

2016-3-112:00 am

2016-3-11:00 am

2016-3-12:00 am

2016-3-1111:00pm

2016-3-1212:00am

2016-3-121:00am…

2016-3-1110:00pm

To the batchprocessor

What about sessions across batches?

(Re)processing data (streaming)

Draw savepoints at times that you will want to start new jobs from (daily, hourly, …)

Reprocess by starting a new job from a savepoint• Defines start position in stream (for example Kafka offsets)

• Initializes pending state (like partial sessions)

Savepoint

Run new streamingprogram from savepoint

Continuous Data Sources

2016-3-112:00 am

2016-3-11:00 am

2016-3-12:00 am

2016-3-1111:00pm

2016-3-1212:00am

2016-3-121:00am

2016-3-1110:00pm …

partition

Savepoint

Stream of Kafka Partitions

Stream view over sequence of files

Kafka offsets +Operator state

File mod timestamp +File position +Operator state

WIP (Flink 1.1?)

Upgrading Programs

A program starting from a savepoint can differ from the program that created the savepoint• Unique operator names match state and operator

Mechanism be used to fix bugs in programs, to evolve programs, parameters, libraries, …

State Backends

Large state is a collection of key/value pairs

State backend defines what data structure holds the state, plus how it is snapshotted

Most common choices• Main memory – snapshots to master

• Main memory – snapshots to dist. filesystem

• RocksDB – snapshots to dist. filesystem

Complex Event Processing Primer

Event Types

Defining Patterns

Generating Alerts

Latency and Throughput

Low Latency and High Throughput

Frequently though to be mutually exclusive

• Event-at-a-time low latency, low throughput

• Mini batch high latency, high throughput

The above is not true!

Very little latency has to be sacrificed for very high throughput

Latency and Throughput

The Effect of Buffering

Network stack does not alwaysoperate in event-at-a-timemode

Optional buffering addssome milliseconds latencybut increases throughput

No effect on application logic

An Outlook on Things to Come

Roadmap

Dynamic Scaling, Resource Elasticity Stream SQL CEP enhancements Incremental & asynchronous state snapshotting Mesos support More connectors, end-to-end exactly once API enhancements (e.g., joins, slowly changing inputs)

Security (data encryption, Kerberos with Kafka)

I stream*, do you?

* beyond Netflix movies

Why does Flink stream flink?

Low latency

High Throughput

Well-behavedflow control

(back pressure)

Make more sense of data

Works on real-timeand historic data

TrueStreaming

Event Time

APIsLibraries

StatefulStreaming

Globally consistentsavepoints

Exactly-once semanticsfor fault tolerance

Windows &user-defined state

Flexible windows(time, count, session, roll-your own)

Complex Event Processing

Addendum

On a technical level

Decouple all things

Clocks• Wall clock time (processing time)• Event time (watermarks & punctuations)• Consistency clock (logical checkpoint timestamps)

Buffering• Windows (application logic)• Network (throughput tuning)

Decoupling clocks

Stream Alignment

High Availability Checkpoints

JobManager

Client

TaskManagers

Apache Zookeeper™

1. Take snapshots

JobManager

Client

TaskManagers

Apache Zookeeper™

1. Take snapshots2. Persist snapshots3. Send handles to JM

JobManager

Client

TaskManagers

Apache Zookeeper™

1. Take snapshots2. Persist snapshots3. Send handles to JM4. Create global checkpoint

JobManager

Client

TaskManagers

Apache Zookeeper™

1. Take snapshots2. Persist snapshots3. Send handles to JM4. Create global checkpoint5. Persist global checkpoint

JobManager

Client

TaskManagers

Apache Zookeeper™

1. Take snapshots2. Persist snapshots3. Send handles to JM4. Create global checkpoint5. Persist global checkpoint6. Write handle to ZooKeeper

Streaming Analytics with Apache Flink - Meetupfiles.meetup.com/18824486/Flink @ DC Flink...

Documents