+ All Categories
Home > Business > RabbitMQ Data Ingestion

RabbitMQ Data Ingestion

Date post: 11-May-2015
Category:
Upload: alvaro-videla
View: 8,063 times
Download: 2 times
Share this document with a friend
Description:
This talk explains the use of RabbitMQ Federation to build a distributed system with RabbitMQ
Popular Tags:
90
Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
Transcript
Page 1: RabbitMQ Data Ingestion

Building a Distributed Data Ingestion System

with RabbitMQAlvaro Videla - RabbitMQ

Page 2: RabbitMQ Data Ingestion

Alvaro Videla

• Developer Advocate at Pivotal / RabbitMQ!

• Co-Author of RabbitMQ in Action!

• Creator of the RabbitMQ Simulator!

• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!

• @old_sound — [email protected] — github.com/videlalvaro

Page 3: RabbitMQ Data Ingestion

About Me

Co-authored!!

RabbitMQ in Action!

http://bit.ly/rabbitmq

Page 4: RabbitMQ Data Ingestion

About this Talk

• Exploratory Talk

• A ‘what could be done’ talk instead of ‘this is how you do it’

Page 5: RabbitMQ Data Ingestion

Agenda

• Intro to RabbitMQ

• The Problem

• Solution Proposal

• Improvements

Page 6: RabbitMQ Data Ingestion

What is RabbitMQ

Page 7: RabbitMQ Data Ingestion

RabbitMQ

Page 8: RabbitMQ Data Ingestion

RabbitMQ

Page 9: RabbitMQ Data Ingestion

RabbitMQ• Multi Protocol Messaging Server

Page 10: RabbitMQ Data Ingestion

• Multi Protocol Messaging Server!• Open Source (MPL)

RabbitMQ

Page 11: RabbitMQ Data Ingestion

• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot

RabbitMQ

Page 12: RabbitMQ Data Ingestion

• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot!• Written in Erlang/OTP

RabbitMQ

Page 14: RabbitMQ Data Ingestion

Polyglot

Page 15: RabbitMQ Data Ingestion

Polyglot

Page 16: RabbitMQ Data Ingestion

Polyglot

• Java

Page 17: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js

Page 18: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js!• Erlang

Page 19: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js!• Erlang!• PHP

Page 20: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby

Page 21: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net

Page 22: RabbitMQ Data Ingestion

Polyglot

• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net!• Haskell

Page 23: RabbitMQ Data Ingestion

Polyglot

Even COBOL!!!11

Page 24: RabbitMQ Data Ingestion

Some users of RabbitMQ

Page 25: RabbitMQ Data Ingestion

• Instagram

Some users of RabbitMQ

Page 26: RabbitMQ Data Ingestion

• Instagram!• Indeed.com

Some users of RabbitMQ

Page 27: RabbitMQ Data Ingestion

• Instagram!• Indeed.com!• Telefonica

Some users of RabbitMQ

Page 28: RabbitMQ Data Ingestion

• Instagram!• Indeed.com!• Telefonica!• Mercado Libre

Some users of RabbitMQ

Page 29: RabbitMQ Data Ingestion

• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS

Some users of RabbitMQ

Page 30: RabbitMQ Data Ingestion

• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS!• Mozilla

Some users of RabbitMQ

Page 31: RabbitMQ Data Ingestion

http://www.rabbitmq.com/download.html

Unix - Mac - Windows

Page 32: RabbitMQ Data Ingestion

Messaging with RabbitMQ

A demo with the RabbitMQ Simulator

https://github.com/RabbitMQSimulator/RabbitMQSimulator

Page 33: RabbitMQ Data Ingestion

http://tryrabbitmq.com

Page 34: RabbitMQ Data Ingestion

RabbitMQ Simulator

Page 35: RabbitMQ Data Ingestion

The Problem

Page 36: RabbitMQ Data Ingestion

Distributed Application

App

App

App

App

Page 37: RabbitMQ Data Ingestion

Distributed Application

App

App

App

App

Page 38: RabbitMQ Data Ingestion

Ad-hoc solution

Page 39: RabbitMQ Data Ingestion

A process that replicates data to the remote server

Page 40: RabbitMQ Data Ingestion

Possible issues• Remote server is offline

• Prevent unbounded local buffers

• Prevent message loss

• Prevent unnecessary message replication

• No need for those messages on remote server

• Messages that became stale

Page 41: RabbitMQ Data Ingestion

Can we do better?

Page 42: RabbitMQ Data Ingestion

RabbitMQ Federation

Page 43: RabbitMQ Data Ingestion

RabbitMQ Federation• Supports replication across different administrative domains

• Supports mix of Erlang and RabbitMQ versions

• Supports Network Partitions

• Specificity - not everything has to be federated

Page 44: RabbitMQ Data Ingestion

RabbitMQ Federation

Page 45: RabbitMQ Data Ingestion

RabbitMQ Federation

Page 46: RabbitMQ Data Ingestion

RabbitMQ Federation

Page 47: RabbitMQ Data Ingestion

RabbitMQ Federation

• It’s a RabbitMQ Plugin

Page 48: RabbitMQ Data Ingestion

RabbitMQ Federation

• It’s a RabbitMQ Plugin

• Internally uses Queues and Exchanges Decorators

Page 49: RabbitMQ Data Ingestion

RabbitMQ Federation

• It’s a RabbitMQ Plugin

• Internally uses Queues and Exchanges Decorators

• Managed using Parameters and Policies

Page 50: RabbitMQ Data Ingestion

Enabling the Plugin

rabbitmq-plugins enable rabbitmq_federation

Page 51: RabbitMQ Data Ingestion

Enabling the Plugin

rabbitmq-plugins enable rabbitmq_federation

rabbitmq-plugins enable rabbitmq_federation_management

Page 52: RabbitMQ Data Ingestion

Federating an Exchange

rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}'

Page 53: RabbitMQ Data Ingestion

Federating an Exchange

rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}' !

rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\." \ '{"federation-upstream-set":"all"}'

Page 54: RabbitMQ Data Ingestion

Federating an Exchange

Page 55: RabbitMQ Data Ingestion

Configuring Federation

Page 56: RabbitMQ Data Ingestion

Config Optionsrabbitmqctl set_parameter federation-upstream \ name ‘json-object’

Page 57: RabbitMQ Data Ingestion

Config Optionsrabbitmqctl set_parameter federation-upstream \ name ‘json-object’ !

json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm }

http://www.rabbitmq.com/federation-reference.html

Page 58: RabbitMQ Data Ingestion

Prevent unbound buffers

expires: N // ms. message-ttl: N // ms.

Page 59: RabbitMQ Data Ingestion

Prevent message forwarding

max-hops: N

Page 60: RabbitMQ Data Ingestion

Speed vs No Message Loss

ack-mode: on-confirm ack-mode: on-publish

ack-mode: no-ack

Page 61: RabbitMQ Data Ingestion

AMQP URI:

amqp://user:pass@host:10000/vhost

http://www.rabbitmq.com/uri-spec.html

Page 62: RabbitMQ Data Ingestion

Config can be applied via

• CLI using rabbitmqctl

• HTTP API

• RabbitMQ Management Interface

Page 63: RabbitMQ Data Ingestion

RabbitMQ Federation

Page 64: RabbitMQ Data Ingestion

Scaling the Setup

Page 65: RabbitMQ Data Ingestion

The Problem

Page 66: RabbitMQ Data Ingestion

The Problem

• Queues contents live in the node where the Queue was declared

Page 67: RabbitMQ Data Ingestion

The Problem

• Queues contents live in the node where the Queue was declared

• A cluster can access the queue from every connected node

Page 68: RabbitMQ Data Ingestion

The Problem

• Queues contents live in the node where the Queue was declared

• A cluster can access the queue from every connected node

• Queues are an Erlang process (tied to one core)

Page 69: RabbitMQ Data Ingestion

The Problem

• Queues contents live in the node where the Queue was declared

• A cluster can access the queue from every connected node

• Queues are an Erlang process (tied to one core)

• Adding more nodes doesn’t really help

Page 70: RabbitMQ Data Ingestion

Sharded Queues

Page 71: RabbitMQ Data Ingestion

Pieces of the Puzzle

• consistent hash exchange

• good ol’ queues

Page 72: RabbitMQ Data Ingestion

Sharded Queues

Page 73: RabbitMQ Data Ingestion

Sharded Queues

Page 74: RabbitMQ Data Ingestion

Sharded Queues

Page 75: RabbitMQ Data Ingestion

Sharded Queues

• Declare Queues with name: nodename.queuename.index

Page 76: RabbitMQ Data Ingestion

Sharded Queues

• Declare Queues with name: nodename.queuename.index

• Bind the queues to a consistent hash exchange

Page 77: RabbitMQ Data Ingestion

Sharded Queues

• Declare Queues with name: nodename.queuename.index

• Bind the queues to a consistent hash exchange

• Get the consumer to randomly build the queue name

Page 78: RabbitMQ Data Ingestion

We need more scale!

Page 79: RabbitMQ Data Ingestion

Federated Queues

Page 80: RabbitMQ Data Ingestion

Federated Queues

• Load-balance messages across federated queues

• Only moves messages when needed

Page 81: RabbitMQ Data Ingestion

Federating a Queue

rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}'

Page 82: RabbitMQ Data Ingestion

Federating a Queue

rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}' !

rabbitmqctl set_policy --apply-to queues federate-me "^images\." \ '{"federation-upstream-set":"all"}'

Page 83: RabbitMQ Data Ingestion

With RabbitMQ we can

Page 84: RabbitMQ Data Ingestion

With RabbitMQ we can

• Ingest data using various protocols: AMQP, MQTT and STOMP

Page 85: RabbitMQ Data Ingestion

With RabbitMQ we can

• Ingest data using various protocols: AMQP, MQTT and STOMP

• Distribute that data globally using Federation

Page 86: RabbitMQ Data Ingestion

With RabbitMQ we can

• Ingest data using various protocols: AMQP, MQTT and STOMP

• Distribute that data globally using Federation

• Scale up using Sharding

Page 87: RabbitMQ Data Ingestion

With RabbitMQ we can

• Ingest data using various protocols: AMQP, MQTT and STOMP

• Distribute that data globally using Federation

• Scale up using Sharding

• Load balance consumers with Federated Queues

Page 88: RabbitMQ Data Ingestion

Credits

world map: wikipedia.org

federation diagrams: rabbitmq.com

Page 89: RabbitMQ Data Ingestion

Questions?

Page 90: RabbitMQ Data Ingestion

ThanksAlvaro Videla - @old_sound


Recommended