A Gentle Introduction To Storm And Kafka

The Leader in Big Data Consulting

www.mammothdata.com | @mammothdataco

A Gentle Introduction of Kafka and Storm

{Percona University | Raleigh}

http://www.mammothdata.com/

https://twitter.com/mammothdataco

www.mammothdata.com | @mammothdatacowww.mammothdata.com | @mammothdataco

Open Software Integrators

Open Software Integrators is a Big Data consulting and services company specializing in Hadoop, Cassandra, MongoDB and other NoSQL technologies. OSI focuses on executive strategy, initial install, design and implementation.

Founded January 2008 by Andrew C. Oliver

Based in downtown Durham, NC

Partnered with Hortonworks, MongoDB, DataStax, Cloudera, Couchbase, Cloudbees & Neo Technology






A Gentle Introduction

What Kafka and Storm are?What they can be used for?What they excel at?






Kafka

Kafka and Storm




What is Apache Kafka?

Kafka is a distributed, partitioned, replicated commit log service.






The Commit Log

An append-only, immutable sequence of records ordered by time.

firstrecord

next writtenrecord






Kafka is:

● fast● durable● distributed● scalable






Kafka Abstractions

● Topic: feeds of messages in categories● Broker: a host running Kafka● Producer: a process that publishes messages● Consumer: a process that pulls messages● Partition: portion of a topic’s stream of messages






What Kafka is used for:

Enterprise-grade event streaming






What Kafka is not good at:

Doing anything other than being a commit log.






Storm

Kafka and Storm




What is Apache Storm?

Storm is a distributed, real time computation system






Stream Processing

● AKA Event Sourcing ● Command and Query Responsibility Segregation● Complex Event Processing● etc.

Several process fail into the domain of stream processing.






● Simple API● Guaranteed data processing● Fault tolerant● Scalable● Usable with any language

What Storm Does




Three abstractions:● Spouts● Bolts● Topology

Storm Abstractions

SpoutSpout

BoltBoltBolt

Bolt




Processes:● UI● Nimbus● Supervisor● Worker

Storm Processes

SupervisorWorker

Worker

SupervisorWorker

Worker

Zookeeper

Web UI Nimbus




● Worker process● Executors● Tasks

Storm Parallelism Model




Use Case: Security

Kafka and Storm




Security customer analytics platform ● Pulling data from customer sites, ● Placed data in a SQL database ● Performing analysis to spot anomalous traffic ● Pushing results back to client to blocking traffic sources

Use Case: Security




Original system mean turn around time: 4.5 hoursStorm / Kafka solution, maximum processing time:

2.6 seconds

Use Case: Security




Thank You

Kafka and Storm




Kafka: http://kafka.apache.org/Storm: http://storm.apache.org/

Links



http://kafka.apache.org/

http://storm.apache.org/

Date post:	12-Apr-2017
Category:	Technology
Upload:	mammoth-data
View:	115 times
Download:	0 times