Anton Gorshkov

Real-Time Analytics Visualized Kafka Streamliner MemSQL ZoomData

Please note that during the course of this presentation ZoomData products will be used and shown on the screen. Goldman Sachs has an ownership interest in ZoomData, Inc. and may have other business relationships with ZoomData, Inc. Nothing herein shall constitute an offer to sell or a solicitation of an offer to buy an interest in any entity or product.

Learn more at GS.com/Engineering

>docker run kafka>docker run memsql>docker run zoomdata

Initial Set-Up

2 4-CPU / 16GB / 80GB SSD / Intel Xeon E5-2670 @ 2.5GHzX

Context

Start with one producer & a queue

Producer 1 Kafka

Add-a-Sink

Producer 1 Kafka In-Memory SQL RDB

Consumer

Add a Real-Time Visualization

Producer 1 Kafka In-Memory SQL RDB

RT VisConsumer

Enterprise Grade?

Resilience- Don’t lose data- Be Up- Deliver (at least once and in-order)

Bad Data

SchemaEvolution

How “I” is the BI?- Is it a “view” or a “do work” layer?- Data-at-Rest vs Data-at-Motion- Pull vs Push- Consistency to other vis layers

Will It Scale?Throughput, Concurrency, Size(and at what cost…)

Will it Scale?

Producer 1

Kafka In-Memory SQL RDB

RT VisProducer 2

Producer n

Consumer

Audience Participation Time…

Kafka In-Memory SQL RDB

RT Vis

(620) 487-2222

Consumer

Audience Participation Time…

send a text [Fruit] [Quantity]

(620) 487-2222example: mango 540

Adaptability

Producer 1

Kafka

In-Memory SQL RDB

RT Vis

Producer 2

Producer nElastic Kibana

Consumer

KafkaConnect

Representative Deployment

Direct Sink

Spark Streaming

IMRDBMS

Data Lake

Kafka

SourceDBs

Order Mgmt

Batch ETL

Cache

Vert

X

Learn more at GS.com/Engineering