Home >Technology >Redis as a message queue

Redis as a message queue

Date post:24-Jan-2015
View:4,157 times
Download:0 times
Share this document with a friend
Using Redis as a Message Queue
  • 1. Redis As a Message Queue 1Wednesday, October 9, 13

2. The Plug Essentially, a screenshot site MMORPG (Massively-Multiplayer, Online Role-Playing Game) niche market Site visitors browse games, games have albums Players can upload screenshots 2Wednesday, October 9, 13 3. The Architecture (2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10 (1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1 (2) layer-4 ipvsadm load balancers in master/backup failover via keepalived (3) PostgreSQL master/slave database (3) nginx+php-fpm (5.5) app servers Replicated le system via GlusterFS for images Memcached instance for php sessions Redis for caching and message queue Amazon Cloudfront for CDN (content delivery network) PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API 3Wednesday, October 9, 13 4. Message Queue? Push messages into a stack Pop (pull) messages off a stack You put (push) data (messages) into the queue (stack) Messages are things you want to process later by pulling them from the queue 4Wednesday, October 9, 13 5. Messages? A message is just some data Could be an array Could be a string 5Wednesday, October 9, 13 6. The Use-Case When a user uploads images, it is bad to block the user while your server processes the photos Image resizing Creating multiple versions Creating database entries 6Wednesday, October 9, 13 7. The User Flow User selects images Images are uploaded to app server App server generates a UUID lename and copies the images to the replicated le system App server increments a Redis counter App server creates an array with uuid, user id, game id and pushes this message into the Redis list (message queue) 7Wednesday, October 9, 13 8. Batch Processing A separate PHP/other process running on any of the app servers (crontab, forking daemon) pops the Redis list to get a message By using GlusterFS as a replicated le system all app servers have all the uploads/images When a message is obtained from Redis queue, it processes the le upload, creates various image sizes Creates a database entry in PostgreSQL 8Wednesday, October 9, 13 9. Lets push! // Create a new redis connection $r = new Redis(); $r->connect($host, $port); // Create the message array $message = [uuid => $uuid, userId => $userId, gameId => $gameId]; // Get a new message id by implementing the builtin increment feature of Redis $id = $r->incr('mmoscreens:screens-queue'); // Push the id into the queue (Redis list) $r->lpush('mmoscreens:screens-queue', $id); // Set the $message into Redis $r->hmset("mmoscreens:screens-queue:$id", $message); 9Wednesday, October 9, 13 10. Lets pull! // Create a new redis connection $r = new Redis()->connect($host, $port); // pop the message ID from the queue, or block until one is available $id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME); // brPop returns [listName, element] isset($id[1]) && $id = $id[1]; // Get the message from Redis $message = $r->hgetall("mmoscreens:screens-queue:$id"); $message: [ uuid => fef4430c-e4a8-4874-b380-63edd2fa8472, userId => 10611, gameId => 8 ] 10Wednesday, October 9, 13

Click here to load reader

Embed Size (px)