%w(map reduce).first A Tale About Rabbits,

Latency and Slim Crontabs

Paolo Negri

thanks to: www.autoscout24.de

Summary:

Map

http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html

http://www.flickr.com/photos/myxi/448253580

rabbitMQ

crontab diet

http://www.flickr.com/photos/tim_norris/2600843131/

Map Reduce

“Programming model for processing and generating large

data sets”

(Google)

Map Reduce "Map" step

the master node takes the input, chops it up into smaller sub-problems, and distributes

those to worker nodes.(Wikipedia)

The problem

Invoicing our clients

Is it as simple as...

clients.map do |client| client.invoiceend

No!

•distributed

•concurrent

Because the process is:

Problems:

•How many nodes?

•How many workers?

•Distribution mechanism to feed the workers?

What about queuing?

• the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue

• workers independently consume the content of the queue

Here comes

• RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging

• It’s opensource

• Can be used to manage queues

• Written in Erlang

Erlang?

• Erlang is a general-purpose concurrent programming language designed by Ericsson

• distributed

• fault tolerant

• soft real time

• high availability

Install it

• sudo apt-get install rabbitmq

• sudo gem install tmm1-amqp

Do it - master node

Use it - worker node

What and where

RabbitMQ(Erlang) TCP/IP

Master(ruby)

Worker(ruby)

Worker(ruby)

Get for free

• Decoupling master/worker

• Workers take care of feeding themselves

• Flexible number of workers

Get for free

• RabbitMQ can be clustered

• Support of message acknowledgement

• Queues can be persisted on disk(at a price)

• low latency

Queue

• Is an actual entity

• has a name

• can be inspected and managed

EventMachine

EventMachine

• Non blocking IO and lightweight concurrency

• eliminate the complexities of high-performance threaded network programming

Is an implementation of Reactor Pattern

EventMachine

EventMachine

amqp gem is built on EventMachine => you’re in a

context where you can leverage concurrent

programming

EM - Deferrables

EM - Deferrables

“The Deferrable pattern allows you to specify any number of Ruby code blocks that will be

executed at some future time when the status of the Deferrable object changes “

EM - Deferrables

EM - Deferrables

Deferrables

ClientStat

Arrears

without deferrables with deferrables

Time

ClientStat Arrears

Achieved so far

• Easy distribution of tasks

• Architecture that supports arbitrary number of workers (and masters)

• Concurrency within the single worker

More rabbitsAnalogy with email system

Multicasting - producer

Multicasting - consumer

Multicasting

Exchangemsg A

Queue1

Queue3

Queue2

Cons1

Cons2

Cons3

Publisher

Multicasting

Exchange

Queue1

Queue3

Queue2

msg A

msg A

msg A

Cons1

Cons2

Cons3

Publisher

Not only queues then

• communication across hosts, heterogeneous systems

• low latency

• clustering

Use messages distribution to build the nervous system of your app

Where to start?

crontab -l

5 * * * * bin/do_the_quick_thing.rb0 2 * * * bin/do_the_scary_thing.rb

Cron

• Simple

• Reliable

• No maintenance

• Status is not explicit

• Locking?

• Shot and forget

Queue

• Distributed easily

• Reliable

• Can be inspected

• Add/decrease workers

•Makes you think!• Adds more complexity

On github - Projects

• eventmachine/eventmachine

• tmm1/amqp

• macournoyer/thin

• famoseagle/carrot

• celldee/bunny

• ezmobius/nanite

Q&A

?

Thanks!

Paolo Negri

/ hungryblank