+ All Categories
Transcript
Page 1: Spark meetup  - Zoomdata Streaming

Interactive Visualization of Data powered by Spark

Page 2: Spark meetup  - Zoomdata Streaming

Streaming Data @ Zoomdata

Visualizations react to new data delivered

Users start, stop, pause the stream

Users select a rolling window or pin a start time to capture cumulative metrics

Page 3: Spark meetup  - Zoomdata Streaming

Drivers for Streaming Data

Data Freshness Time to Analytic Business Context

Page 4: Spark meetup  - Zoomdata Streaming

Challenges

● Time

● Frequency

● Retention

● Synchronization

● Order

● Updates

Page 5: Spark meetup  - Zoomdata Streaming

Addressing the Problem @ Zoomdata

Historical Revised

Receive Data JMS Kafka

Manipulate Stream Single JVM in Memory Spark Streaming

Hold Data in Buffer MongoDB Pluggable

Interact with Data Custom Code Pluggable

Page 6: Spark meetup  - Zoomdata Streaming

Technology Cast

● The Stream - Kafka, Kinesis, JMS

● Processing Fabric - Spark Streaming

● Landing Area - MemSQL, Solr, Kudu, Others

Page 7: Spark meetup  - Zoomdata Streaming

How it looks

Page 8: Spark meetup  - Zoomdata Streaming

With the rest of the app

Page 9: Spark meetup  - Zoomdata Streaming

Scale Out

Page 10: Spark meetup  - Zoomdata Streaming

Benefits

● Contextual Expressiveness with Streaming Data● Independent scalability (scale-up, scale-around)● Expressiveness powered by Spark -- using

Windowing (dataframe API with stream)

Page 11: Spark meetup  - Zoomdata Streaming

Side Benefits

● Separation of concerns● Disaster Recovery, COOP, other Data management

concerns● Restatements● Options!

Page 12: Spark meetup  - Zoomdata Streaming

Demo

● Twitter Producer● Spark Streaming● MemSQL & Solr Sinks

Page 13: Spark meetup  - Zoomdata Streaming

Future Work

● Cross Stream Synchronization & Fusion

● On-demand scale out and resource management via Mesos

● Schema Evolution

● Storage Tiering

Page 14: Spark meetup  - Zoomdata Streaming

Thanks

For more information contact:

[email protected]@zoomdata.com


Top Related