A real time architecture using Hadoop and Storm @ FOSDEM 2013

A real-time architecture using Hadoop and Storm.

A real-time architecture using Hadoop & Storm. 2

Speakers

Nathan Bijnens@nathan_gs

Geert Van Landeghem@gvanlandeghem

http://twitter.com/nathan_gs

http://twitter.com/gvanlandeghem


Our Vision

Big Data

test

Volume


Big Data

test

Velocity


Our Vision

Volume

test

Variety


Credits

Nathan MarzEngineer at Backtype(now Twitter).

Storm

Cascalog

ElephantDB

manning.com/marz

http://manning.com/marz


A Data System


Not all information is equal. Some information is derived from other pieces of

information.

Data is more than Information


Eventually you will reach the most

This is the information you hold true, simple because it exists.

Data is more than Information


EventsEverything we do generates events:- Pay with Credit Card

- Commit to Git

- Click on a webpage

- Tweet


Events used to manipulate the master data.

Events - Before


Today, events are the master data.

Events - After


everything.

Data System


Data is Immutable

Events


Data is Time Based

Events


Capturing change traditionally

Person Location

Nathan Antwerp

Geert Dendermonde

John Ghent

Person Location

Nathan Ghent

Geert Dendermonde

John Ghent


Capturing change

Person Location Time

Nathan Antwerp 2005-01-01

Geert Dendermonde 2011-10-08

John Ghent 2010-05-02

Nathan Ghent 2013-02-03






The data you query is often transformed, aggregated, ...

Query


Query

Query = function ( data )


Number of people living in each city.





Nathan Ghent 2013-02-03

Location Count

Ghent 2

Dendermonde 1


Query

All Data Query


Query: Precompute

All Data QueryPrecomputed

View


Layered Architecture

Speed Layer

Batch Layer

Serving Layer


Layered Architecture

HadoopElephant

DB

Qu

ery

Incoming Data

Cassandra


Batch Layer


Batch Layer

HadoopElephant

DB

Incoming Data


Unrestrained computation.

Batch Layer


Horizontal scalable.

Batch Layer


High Latency.

matter.

Batch Layer


Stores master copy of data set...

Batch Layer

append only.


Batch Layer


Batch: View generation

Master Dataset

View #1

View #3

View #2MapReduce


1. Take a large problem and divide it into sub-problems

2. Perform the same function on all sub-problems

3. Combine the output from all sub-problems

…

…

Output

MA

PRED

UC

E

MapReduce

DoWork() DoWork() DoWork()…


Read only database.No random writes required.

Batch View Database


Batch View DatabaseElephantDB

Splout


Batch Layer

Not yet absorbed.

Data absorbed into Batch Views

Time No

w

Just a few hours of data.


Speed Layer


Overview

HadoopElephant

DB

Incoming Data

Cassandra


Stream processing.

Speed Layer


Continuous computation.

Speed Layer


Transactional.

Speed Layer


Storing a limited window of data.Compensating for the last few hours of data.

Speed Layer


All the complexity is isolated in the Speed layer auto-

corrected.

Speed Layer


CAPYou have a choice between:

Availability- Queries are eventual consistent.

Consistency- Queries are consistent.


Some algorithms are hard to implement in real time. For those cases we could

estimate the results.

Eventual accuracy


Speed Layer

Incoming Data

Real Time

View 1

Real Time

View 2


StormMessage passing.

Distributed processing.

Horizontally scalable.

Incremental algorithms.

Fast.

Data in motion.


StormMessage passing.

Distributed processing.

Horizontally scalable.

Incremental algorithms.

Fast.

Data in motion.


Storm

Nimbus Zookeeper

Worker Node

Supervisor

Wo

rker

Wo

rker

Wo

rker

Worker Node

Supervisor

Wo

rker

Wo

rker

Wo

rkerWorker Node

SupervisorW

orker

Wo

rker

Wo

rker


StormTuple

Stream


StormSpout

Bolt


StormGrouping


Speed Layer ViewsThe views are stored in Read & Write database.- Cassandra

- Hbase

- MongoDB

- MySQL

- ElasticSearch

-

Much more complex than a read only view.


Serving Layer


Overview

HadoopElephant

DB

Qu

ery

Incoming Data

Cassandra


This layer queries the Batch & Real Time views and merges it.

Serving Layer


Serving Layer

Real Time Views

Merge

Batch Views


Overview


Overview

HadoopElephant

DB

Qu

ery

Incoming Data

Cassandra


Lambda ArchitectureCan discard any view, batch and real time, and just recreate everything from the master data.

Mistakes are corrected via recomputation.- Write bad data? Remove the data & recompute.

- Bug in view generation? Just recompute the view.

Data storage is highly optimized.


Recommendations


Serialization & Schema

Catch errors as quickly as they happen. Validation on write vs on read.


Serialization & Schema

CSV is actually a serialization language that is just poorly defined.


Serialization & SchemaUse a format with a schema.- Thrift

- Avro

- Protobuffers


Questions?

What are your needs?@nathan_gs & @gvanlandeghem


DataCrunchers

We enable companies in envisioning, defining and implementing a data strategy.

A one-stop-shop for all your Big Data needs.

The first Big Data Consultancy agency in Belgium.


Jobs

We are [email protected]

Date post:	26-Jan-2015
Category:	Technology
Upload:	nathan-bijnens
View:	105 times
Download:	3 times

A real time architecture using Hadoop and Storm @ FOSDEM 2013

Technology