RestMQ

Redis based Message Queue

WHAT !?

RestMQ - Message Queue based on Redis

Data manipulation: HTTP GET/POST/DELETE, COMET and WebSockets

No signaling or specific protocol (optional JSON protocol)

http://www.restmq.com/

Gleicon Moraeshttp://github.com/gleiconhttp://zenmachine.wordpress.comhttp://www.7co.cc

Key/Value storage

Think memcached, GET and SET operations

[key] = value

>> SET key meh>> GET keymeh

>> SET fucounter 0>> INCR fucounter1>> GET fucounter1

Redis

Key-Value databaseAtomic operationsSemi-persistent or persistent storagePublish/Subscription channelsDifferent Data types: Sets, Sorted Sets, Lists, Hashes

http://code.google.com/p/redis/http://github.com/antirez/redishttp://rediscookbook.org/

Redis

Data types

StringsListsSets and Ordered SetsHashesPublish/Subscribe channels

Redis

Things you can do:

CacheInverted indexesFast countersThrottle controlCooler stuff at Redis CookbookLoad Balancing

Things you can't do:

Search inside all values for a given string

Redis

Things you can't do:

- Search inside all values for a given key- Search inside all values for a given key- Search inside all values for a given key

Message Queues

Does everything really needs to be tightly coupled ?Order: first come, first servedTransactions: there is no transaction, just trackingJob SchedulingPublish/SubscribeMap/Reduce

Message Queues

SMTP: Oldest Message Queue aroundAvoid hitting your DB real time with MQReal time: Ratings, Voting and CommentsTwitterDo I need NoSQL or I really need to clean my mess ?

RestMQ - Endpoints Diagram

RestMQ - Messages

/q/<queue> : Queue handling (GET/POST/DELETE)

$ curl -X POST -d "value=foobar" http://localhost:8888/q/testqtestq:1

$ curl -X POST -d "value=foobar" http://localhost:8888/q/testqtestq:2

$ curl http://localhost:8888/q/testq { "count": 0, "value": "foobar", "key": "testq:1" }

RestMQ - Messages

/c/<queue> : Comet Endpoint

$ curl http://localhost:8888/c/testq (hangs until there is a message to be received)

/ws/<queue> : WebSocket Endpoint

Needs the proper javascript code

RestMQ - MessagesJSON Protocol inspired by Amazon SQSFirst prototype at http://jsonqueue.appspot.com

add

{ "cmd" : "add", "queue" : "testq", "value" : "abacab" }

del

{ "cmd" : "del", "queue" : "testq", "key" : "testq:31" }

get

{ "cmd" : "get", "queue" : "testq", }

take

{ "cmd" : "take", "queue" : "testq", }

RestMQ - Status

/stats/<queue> : Queue status $ curl http://localhost:8888/stats/test

{ "queue" : "test", "redis": "127.0.0.1:6379 - 10 connection(s)", "len": {"len": 1} }

/stats : Server status $ curl http://localhost:8888/stats/

{ "count": 4, "queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"], "redis": "127.0.0.1:6379 - 10 connection(s)" }

RestMQ - Controls

/p/<queue> : Distribution Policy for COMET and WS

Round Robin a message for each consumer Broadcast a message to all consumers

/control/<queue> : Queue start/stop

Pause all COMET consumers (useful for job schedulers)

RestMQ - Data structure Diagram

RestMQ - Algorithm

Algorithm for pushing messages into a queue named 'testq'

SADD testq into a SET called queuesetINCR a per-queue UUID generator called testq:uuidSET testq:<<id>>, messageLPUSH testq:<<id>> in to a LIST named testq:queue

Works for new and existing queues. On-the-fly queue creation.

RestMQ - Algorithm

Algorithm for reading messages from a queue named 'testq'

RPOP a key from a LIST named testq:queueGET the value associated to this key

For non-destructive GET operations:

LINDEX -1 to get a key in testq:queue LISTINCR key:refcount +1GET the value stored for key

RestMQ - Implementation(the core functionality can be implemented on any language)

Main branch 'Enterprise Edition'

PythonTwistedCycloneRedis async client

Embedded version

Sinatra + Ruby Reduced endpoints (GET/POST)

RestMQ - Sinatra Implementationhttp://gist.github.com/524240

GET /q - List all queues

RestMQ - Sinatra Implementationhttp://gist.github.com/524240

GET /q/<queue> - Get a message from <queue>

RestMQ - Sinatra Implementationhttp://gist.github.com/524240

POST /q/<queue> - Insert a message in <queue>

RestMQ - Map/Reduce

Questions ?

Thanks