A Gentle Introduction To Storm And Kafka

transcript

The Leader in Big Data Consulting

www.mammothdata.com | @mammothdataco

A Gentle Introduction of Kafka and Storm

{Percona University | Raleigh}

http://www.mammothdata.com/

https://twitter.com/mammothdataco

www.mammothdata.com | @mammothdatacowww.mammothdata.com | @mammothdataco

Open Software Integrators

Open Software Integrators is a Big Data consulting and services company specializing in Hadoop, Cassandra, MongoDB and other NoSQL technologies. OSI focuses on executive strategy, initial install, design and implementation.

Founded January 2008 by Andrew C. Oliver

Based in downtown Durham, NC

Partnered with Hortonworks, MongoDB, DataStax, Cloudera, Couchbase, Cloudbees & Neo Technology

A Gentle Introduction

What Kafka and Storm are?What they can be used for?What they excel at?

Kafka

Kafka and Storm

What is Apache Kafka?

Kafka is a distributed, partitioned, replicated commit log service.

The Commit Log

An append-only, immutable sequence of records ordered by time.

firstrecord

next writtenrecord

Kafka is:

● fast● durable● distributed● scalable

Kafka Abstractions

● Topic: feeds of messages in categories● Broker: a host running Kafka● Producer: a process that publishes messages● Consumer: a process that pulls messages● Partition: portion of a topic’s stream of messages

What Kafka is used for:

Enterprise-grade event streaming

What Kafka is not good at:

Doing anything other than being a commit log.

Storm

Kafka and Storm

What is Apache Storm?

Storm is a distributed, real time computation system

Stream Processing

● AKA Event Sourcing ● Command and Query Responsibility Segregation● Complex Event Processing● etc.

Several process fail into the domain of stream processing.

● Simple API● Guaranteed data processing● Fault tolerant● Scalable● Usable with any language

What Storm Does

Three abstractions:● Spouts● Bolts● Topology

Storm Abstractions

SpoutSpout

BoltBoltBolt

Bolt

Processes:● UI● Nimbus● Supervisor● Worker

Storm Processes

SupervisorWorker

Worker

SupervisorWorker

Worker

Zookeeper

Web UI Nimbus

● Worker process● Executors● Tasks

Storm Parallelism Model

Use Case: Security

Kafka and Storm

Security customer analytics platform ● Pulling data from customer sites, ● Placed data in a SQL database ● Performing analysis to spot anomalous traffic ● Pushing results back to client to blocking traffic sources

Use Case: Security

Original system mean turn around time: 4.5 hoursStorm / Kafka solution, maximum processing time:

2.6 seconds

Use Case: Security

Thank You

Kafka and Storm

Kafka: http://kafka.apache.org/Storm: http://storm.apache.org/

Links

http://kafka.apache.org/

http://storm.apache.org/