Single-Pass Graph Stream Analytics with Apache Flink

transcript

@GraphDevroom

Single-pass Graph Stream Analytics with Apache Flink

Rethinking graph processing for dynamic data

Vasiliki Kalavri <vasia@apache.org> Paris Carbone <senorcarbone@apache.org>

@GraphDevroom

Real Graphs are dynamic

Graphs created by events happening in real-time • liking a post • buying a book • listening to a song • rating a movie • packet switching in computer networks • bitcoin transactions

Each event adds an edge to the graph

@GraphDevroom

In a batch world

We create and analyze a snapshot of the real graph • all events / interactions / relationships that

happened between t0 and tn • the Facebook social network on January 30 2016 • user web logs gathered between March 1st 12:00 and 16:00 • retweets and replies for 24h after the announcement of the

death of David Bowie

@GraphDevroom

Batch Graph Processing

@GraphDevroom

In a streaming world

• We receive and consume the events as they are happening, in real-time

• We analyze the evolving graph and receive results continuously

@GraphDevroom

Streaming Graph Processing

@GraphDevroom

Sounds expensive?

Challenges • maintain the graph structure

• how to apply state updates efficiently?

• update the result • re-run the analysis for each event? • design an incremental algorithm? • run separate instances on multiple snapshots?

• compute only on most recent events

@GraphDevroom

The Apache Flink Stack

Execution

DataStreamDataSet

Distributed Dataflow

Deployment

• Bounded Data Sources • Structured Iterations • Blocking Operations

• Unbounded Data Sources • Asynchronous Iterations • Incremental Operations

@GraphDevroom

Unifying Data Processing

Job Manager • scheduling tasks • monitoring/recovery

Client

• task pipelining • blocking

• execution plan building • optimisation

DataStreamDataSet

Deployment

DataSet<String> text = env.readTextFile(“hdfs://…”); text.map(…).groupReduce(…)…

DataStream<String> events = env.addSource(new KafkaConsumer(…)); events.map(…).filter(…).window(…).fold(…)…

@GraphDevroom Graph Processing on

Apache Flink

DataStreamDataSet

Deployment

• Static Graphs • Multi-Pass Algorithms • Full Computations

DataStream

@GraphDevroom

Data Streams as ADTs

• Direct access to the execution graph / topology

• Suitable for engineers

• Abstract Data Type Transformations hide operator details

• Suitable data analysts and engineers

similar to: PCollection, DStream

DataStream

@GraphDevroom

Nature of a DataStream Job

• Tasks are long running in a pipelined execution.

• State is kept within tasks.

• Transformations are applied per-record or per-window.

Execution Graph

unbounded data sinks

unbounded data sources

• operator parallelism• stream partitioning

Execution Properties

@GraphDevroom

Working with DataStreams

Creation TransformationsDataStream<String> myStream =

-for supported data sources: env.addSource(new FlinkKafkaConsumer<String>(…)); env.addSource(new RMQSource<String>(…)); env.addSource(new TwitterSource(propsFile)); env.socketTextStream(…);

-for testing: env.fromCollection(…); env.fromElements(…);

-for adding any custom source: env.addSource(MyCustomSource(…));

PropertiesmyStream.setParallelism(3)

myStream.broadcast(); .rebalance(); .forward();

.keyBy(key);

partitioning

partition stream and operator state by key

myStream.map(…); myStream.flatMap(…); myStream.filter(…); myStream.union(myOtherStream);

-for aggregations on partitioned-by-key streams:

myKeyStream.reduce(…); myKeyStream.fold(…); myKeyStream.sum(…);

@GraphDevroom

Example

env.setParallelism(2); //default parallelism DataStream<Tuple2<String, Integer>> counts = env

.socketTextStream("localhost", 9999) .flatMap(new Splitter()) //transformation .keyBy(0) //partitioning .sum(1) //rolling aggregation

.setParallelism(4); counts.print();

“cool, gelly is cool”

<“gelly", 1><“is”, 1> <“cool”,1><“cool”,1>

<“is”, 1> <“gelly”, 1>

<“cool”,2> <“cool”,1>

printsum

flatMap

@GraphDevroom

Working with Windows

Why windows? We are often interested in fresh data!

Highlight: Flink can form and trigger windows consistently under different notions of time and deal with late events!

#sec40 80

SUM #2

SUM #1

20 60 100

#sec40 80

SUM #3

SUM #2

SUM #1

20 60 100

15 38 65 88

110 120

myKeyStream.timeWindow( Time.of(60, TimeUnit.SECONDS), Time.of(20, TimeUnit.SECONDS));

1) Sliding windows

2) Tumbling windowsmyKeyStream.timeWindow( Time.of(60, TimeUnit.SECONDS));

window buckets/panes

@GraphDevroom

Example

env.setParallelism(2); //default parallelism DataStream<Tuple2<String, Integer>> counts = env

.socketTextStream("localhost", 9999) .flatMap(new Splitter()) //transformation .keyBy(0) //partitioning

.window(Time.of(5, TimeUnit.MINUTES)) .sum(1) //rolling aggregation

.setParallelism(4); counts.print();

10:48 - “cool, gelly is cool”

printwindow sumflatMap

11:01 - “dataflow is cool too”

<“gelly”,1>… <“cool”,2>

<“dataflow”,1>… <“cool”,1>

@GraphDevroom Single-Pass Graph Streaming

with Windows• Each event represents an edge addition

• Each edge is processed once and thrown away, i.e. the graph structure is not explicitly maintained

• The state maintained corresponds to a graph summary, a continuously improving property, an aggregation

• Recent events can be grouped in a graph window and processed independently

@GraphDevroom

What’s the benefit?

• Get results faster • No need to wait for the job to finish • Sometimes, early approximations are better than late exact

answers • Get results continuously

• Process unbounded number of events • Use less memory

• single-pass algorithms don’t store the graph structure • run computations on a graph summary

@GraphDevroom

What can you do in this model?

• transformations, e.g. mapping, filtering vertex / edge values, reverse edge direction

• continuous aggregations, e.g. degree distribution

@GraphDevroom

1 2 3 4

Streaming Degrees Distribution#v

degree

@GraphDevroom

1 2 3 4

degree

Streaming Degrees Distribution

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

1 2 3 4

degree

@GraphDevroom

• spanners for distance estimation • sparsifiers for cut estimation • sketches for homomorphic properties

graph summary

algorithm algorithm~R1 R2

@GraphDevroom

• neighborhood aggregations on windows, e.g. triangle counting, clustering coefficient (no iterations… yet!)

@GraphDevroom

Examples

@GraphDevroom

Batch Connected Components

• State: the graph and a component ID per vertex (initially equal to vertex ID)

• Iterative Computation: For each vertex:

• choose the min of neighbors’ component IDs and own component ID as new ID

• if component ID changed since last iteration, notify neighbors

@GraphDevroom

i=13 4

1 2 4 5

@GraphDevroom

Streaming Connected Components

• State: a disjoint set data structure for the components

• Computation: For each edge

• if seen for the 1st time, create a component with ID the min of the vertex IDs

• if in different components, merge them and update the component ID to the min of the component IDs

• if only one of the endpoints belongs to a component, add the other one to the same component

@GraphDevroom

ComponentID Vertices

@GraphDevroom

1 1, 3

@GraphDevroom

2 2, 5

1 1, 3

@GraphDevroom

2 2, 4, 5

1 1, 3

@GraphDevroom

2 2, 4, 5

1 1, 3

6 6, 7

@GraphDevroom

2 2, 4, 5

1 1, 3

6 6, 7, 8

@GraphDevroom

41 ComponentID Vertices

2 2, 4, 5

1 1, 3

6 6, 7, 8

@GraphDevroom

2 2, 4, 5

1 1, 3

6 6, 7, 8

@GraphDevroom

6 6, 7, 8

1 1, 2, 3, 4, 5

@GraphDevroom

6 6, 7, 8

1 1, 2, 3, 4, 5

@GraphDevroom

6 6, 7, 8

1 1, 2, 3, 4, 5

@GraphDevroom Distributed Streaming Connected

Components

@GraphDevroom

Streaming Bipartite Detection

Similar to connected components, but

• each vertex is also assigned a sign, (+) or (-)

• edge endpoints must have different signs

• when merging components, if flipping all signs doesn’t work => the graph is not bipartite

@GraphDevroom

(+) (-)

(+)(-)

(+) (-)

@GraphDevroom

(+) (-)

(+)(-)

(+) (-)

@GraphDevroom

(+) (-)

(+)(-)

(+) (-)

@GraphDevroom

(+) (-)

(-)(+)

(+) (-)

@GraphDevroom

(+) (-)

(-)(+)

(+) (-)

(-)Can’t flip signs and stay consistent

=> not bipartite!

@GraphDevroom

The GraphStream

DataStreamDataSet

Deployment

Gelly Gelly-Stream

• Static Graphs • Multi-Pass Algorithms • Full Computations

• Dynamic Graphs • Single-Pass Algorithms • Incremental Computations

DataStream

@GraphDevroom

Introducing Gelly-Stream

• Gelly-Stream enriches the DataStream API with two new additional ADTs:

• GraphStream:

• A representation of a data stream of edges.

• Edges can have state (e.g. weights).

• Supports property streams, transformations and aggregations.

• GraphWindow:

• A “time-slice” of a graph stream.

• It enables neighborhood aggregations (and iterations in the future)

@GraphDevroom

Graph Property Streams

A B C D A CGraph Stream:

.getEdges()

.getVertices()

.numberOfVertices()

.numberOfEdges()

.getDegrees()

.inDegrees()

.outDegrees()

GraphStream -> DataStream

@GraphDevroom

.mapEdges();

.distinct();

.filterVertices();

.filterEdges();

.reverse();

.undirected();

.union();

Transform Graph Streams

A B C D A CGraph Stream:

GraphStream -> GraphStream

@GraphDevroom

Graph Stream Aggregations

result aggregate

property streamgraph stream

(window) fold

combine

reduce

partitioned aggregates

global aggregates

global aggregates can be persistent or transient

graphStream.aggregate(new MyGraphAggregation(window, update, fold, combine, merge))

@GraphDevroom

Graph Stream Aggregations

result aggregate

property streamgraph stream

(window) fold

combine merge

graphStream.aggregate(new MyGraphAggregation(window, fold, combine, merge))

reduce map

partitioned aggregates

global aggregates

@GraphDevroom

Connected Components

graph stream

combine merge

graphStream.aggregate(new ConnectedComponents(window, update, fold, combine, merge))

reduce map31

@GraphDevroom

graph stream

combine mergereduce map

graphStream.aggregate(new ConnectedComponents(window, update, fold, combine, merge)) 1

@GraphDevroom

graph stream

@GraphDevroom

graph stream

{4,5}76

@GraphDevroom

graph stream

@GraphDevroom

graph stream

TODO:: show blocking reduce instead?

{2,5}{6,8}

{1,3}{4,5}

@GraphDevroom

graph stream

{1,3}{2,4,5}

{6,7,8}

@GraphDevroom

graph stream

{1,3}{2,4,5}

{6,7,8}

@GraphDevroom

graph stream

{1,3}{2,4,5}

{6,7,8}

@GraphDevroom

graph stream

{1,3}{2,4,5}

{6,7,8}

{1,2,4}

{3,4}{7,8}

@GraphDevroom

graph stream

{1,2,4,5}{6,7,8}

{3,4}{7,8}

@GraphDevroom

graph stream

{1,2,3,4,5}{6,7,8}

@GraphDevroom

Slicing Graph Streams

graphStream.slice(Time.of(1, MINUTE));

11:40 11:41 11:42 11:43

@GraphDevroom

Aggregating Slices

graphStream.slice(Time.of(1, MINUTE), direction)

.reduceOnEdges();

.foldNeighbors();

.applyOnNeighbors();

• Slicing collocates edges by vertex information

• Neighbourhood aggregations are now enabled on sliced graphs

source

target

Aggregations

@GraphDevroom

Finding matches nearby

graphStream.slice(Time.of(1, MINUTE)).applyOnNeighbors(FindPairs())

slice applyOnNeighbors

TODO: make it more interactive with transitions

@GraphDevroom

Summary

• Many graph analysis problems can be covered in single-pass

• Processing dynamic graphs requires an incremental graph processing model

• We introduce Gelly-Stream, a simple yet powerful library for graph streams

Single-Pass Graph Stream Analytics with Apache Flink

Data & Analytics