Post on 30-Oct-2014
description
transcript
one system to rule them all
the challenge
• write a backend that scales well
• handle various frontends (web, iPhone, ...)
• runs “somewhere in the cloud”
• easy to maintain and extend
the solution
RabbitMQ+CouchDB=Awesome
Part I
• AMQP based Message Queue
• lots of additional transports (XMPP, HTTP, STOMP, ...)
• erlang based
RabbitMQ
RabbitMQ
daemon
daemon
daemon daemon
frontend frontend frontend
• every functionality gets a backed daemon
• every daemon has a queue associated
• communication via a JSON message that gets passed around
the JSON message
{ "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" }}
• RPC style communication
• if the flow through the system is static
• fast transactions
works great for ...
gets fiddly when ...
• there are asynchronous callbacks from outside
• message flow can change based on various factors
• tasks can take days to complete
Part II
workflows
... again ...
• various frontends
• backend daemons
• JSON messages
what we have ...
what we need
• a way to describe message flow
• a easy flexible way to write and update it
CouchDB
... to the rescue
workflow
• really only a path definition through backends
• some sort of condition management
• a backend definition
{ "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ]}
... and the backend definition
{ "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc"}
now we need a workflow daemon
RabbitMQ
daemon
daemon
daemon daemon
frontend frontend frontend
workflow
where are we now?
• we can call backends as before
• we can call workflows via the workflow queue
• we can manage workflows in CouchDB
Part III
asynchronous callbacks
problem
• we start a workflow
• we call some external API
• the external API somehow notifies us (Mail, Callback URL, ...) about progress
• we need to continue in the workflow
we need a persistent message store
{ "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill"}
one more problemhow do we correlate messages
... with some CouchDB awesome ...
• the backend daemon for the callback polls a view in CouchDB
• we can search for any value in the request we sent
• all we need is a view that searches for the very value this external API uses as correlation ID
RabbitMQ
daemon
daemon
daemon daemon
frontend frontend frontend
workflow
externalcallback
Lenz Gschwendtner
CTO iWantMyName
@norbu09
me
credits
• RabbitMQ
• CouchDB
• Erlang and Perl
more credits
• http://www.flickr.com/photos/djpd
• http://www.flickr.com/photos/lilyapp
• http://www.flickr.com/photos/b-tal
• http://www.flickr.com/photos/mikebaird
• http://www.flickr.com/photos/eyetwist