Extending the Yahoo Streaming Benchmark

transcript

Extending the Yahoo! Streaming Benchmark

Jamie Grier@jamiegrierjamie@data-artisans.com

Who am I?• Director of Applications Engineering at data

Artisans• Previously working on streaming

computation at Twitter, Gnip and Boulder Imaging

• Involved in various kinds of stream processing for about a decade

• High-speed video, social media streaming, general frameworks for stream processing

Overview• Yahoo! performed a benchmark comparing

Apache Flink, Storm and Spark• The benchmark never actually pushed Flink

to it’s throughput limits but stopped at Storms limits

• I knew Flink was capable of much more so I repeated the benchmarks myself

• I did a follow up blog post explaining my findings and will summarize them here

Yahoo! Benchmark• Count ad impressions grouped by

campaign• Compute aggregates over a 10 second

window• Emit current value of window aggregates

to Redis every second for query• Map ads to campaigns using Redis as well

Any questions so far?

Storm Code

Flink Code

Hardware Specs• 10 Kafka brokers with 2 partitions each• 10 compute nodes (Flink / Storm)• Each machine has 1 Xeon E3-1230-V2@3.30GHz CPU

• 4 cores w/ hyperthreading• 32 GB RAM (only 8GB allocated to JVMs)

• 10 GigE Ethernet between compute nodes• 1 GigE Ethernet between Kafka cluster and compute

Logical Deployment

Data Generat

orKafka Source Filter Project Join

Window Sink Redis

Stream Processor

Apache StormDeployment

Source Filter Project Join Window Sink

FlinkData Generator

Shuffle

Apache Storm10 Gige Link1 Gige Link

FlinkData Generator

Shuffle

10 Gige Link1 Gige Link

Source / Filter Project Join Window Sink

FlinkData Generator

Shuffle

Source / Filter / Project Join Window Sink

FlinkData Generator

Shuffle

Source / Filter / Project / Join Window Sink

FlinkData Generator

Shuffle

Window / Sink

FlinkData Generator

Shuffle

Source / Filter / Project / Join

FlinkData Generator

Shuffle

Window / SinkSource / Filter / Project / Join

FlinkData Generator

Shuffle

Apache FlinkDeployment

Apache Flink

Processing Guarantees

Apples and OrangesApache Storm Apache Flink

At least once semantics

Exactly once semantics

Double counting after failures No double counting

Lost state after failures No state loss

Benchmark

0 750,000 1,500,000 2,250,000 3,000,000 3,750,000

Baseline

Throughput: msgs/sec

Bottleneck AnalysisApache Storm

FlinkData Generator

Shuffle

Bottleneck AnalysisApache Storm

FlinkData Generator

Shuffle

FlinkData Generator

Shuffle

Bottleneck AnalysisApache Flink

Apache Flink

FlinkData Generator

Shuffle

Bottleneck AnalysisApache Flink

Apache Flink

Network

FlinkData Generator

Shuffle

Eliminate theBottleneck

Apache Flink

FlinkData Generator

Shuffle

Apache Flink

Shuffle

Apache Flink

DataGenerator

Shuffle

Apache Flink

DataGenerator

Apache FlinkDeployment

Round 2

Benchmark

0 750,000 1,500,000 2,250,000 3,000,000 3,750,000

Baseline

BenchmarkRound 2

Flink (10 GigE)

0 4,000,000 8,000,000 12,000,000 16,000,000

10 GigE end-to-end

Results• Apache Flink achieved 15 million messages

/ sec on Yahoo! benchmark• Much stronger processing guarantees:

Exactly once• 80x higher than what was reported in the

original Yahoo! benchmark on similar hardware

Questions?

Storm Compatibility• Lot’s of companies already have applications

written using the Storm API• Flink provides a Storm compatibility layer• Run your Storm jobs on Flink with a one line

code change• Flink also allows you to reuse your existing

Storm spout and bolt code from a Flink job• Give it a try!

Thanks!

Extending the Yahoo Streaming Benchmark

Data & Analytics