intro and messaging patternsjavier arias losada @javier_arilos

amqp and rabbitmq

introduction to messaging

messaging

● provides asynchronous communications● loosely coupled modules / processes● MoM => Message Oriented Middleware

amqp concepts (i)

amqp is a message oriented wire-level protocol

● message broker: receives and dispatches msgs using AMQP

● connection physical connection (eg: tcp/ip)● channel: allows n clients over one connection

consumerproducer

producer consumer

amqp concepts (ii)

consumerproducer

producer consumer

Clients produce and consume messages.

Exchanges Route and filter messages to queues: binding rules, direct, one-to-one, fanout, topic, headers

Queues buffer messages between producers and consumers

Messages are always: >> sent to exchanges>> consumed from queues

escenario 1

● producing and consuming, simplest routing

examples using Python, pika and rabbitmq, everything should apply to other

languages, libraries and amqp brokers

‘important’ messages must be sent to queue ‘important-jobs’

consumerproducer

source code for the example: https://gist.github.com/javierarilos/9348168

step 1- connect & channel setup

from pika import BlockingConnection, ConnectionParameters conn = BlockingConnection(ConnectionParameters('localhost'))ch = conn.channel()

consumerproducer

Same code for consumer and producer

step 2- declare exchange

Declare an exchange, important parameters:

● name: if exists, nothing is done● type: direct, fanout, pub-sub, headers● durable: must exchange be kept between server restarts?● autodelete: must exchange be deleted when not in use?● internal: is it for internal routing use, or public to clients?

ch.exchange_declare(exchange='important', type='direct')

consumerproducerexchange: importanttype: direct

step 3- declare queue

Declare a queue, important parameters:

● name: if exists, nothing is done● durable: must queue declaration be kept between server restarts?● exclusive: Is this the only connection that can consume from the queue?● autodelete: must queue be deleted when not in use?

ch.queue_declare(queue='important-jobs')

consumerproducerexchange: importanttype: direct

queue: important-jobs

step 4- bind queue and exchange

ch.queue_bind(exchange='important', queue='important-jobs', routing_key='important')

consumerproducerexchange: importanttype: direct

queue: important-jobs

routing_key: important

Binding a queue and exchange:

● establishes a route (exchange => queue)● based on a criteria (exchange type + routing key)

Here: when producer sends a message to ‘important’ exchange with routing key ‘important’, message will be forwarded to queue ‘important-jobs’.

step 5- send the message

ch.basic_publish(exchange='important', routing_key='important', body='new important task')

consumerproducerexchange: importanttype: direct

queue: important-jobs

routing_key: important

Bonus track: default exchange, forwards to a queue, without any exchange declaration:default exchange’s name is empty string = ‘’routing_key is the queue name ‘important-jobs’

ch.basic_publish(exchange=’’, routing_key='important-jobs', body=’def exch important task')

step 6- consume the message

method_frame, header_frame, body = ch.basic_get('important-jobs')print body ch.basic_ack(method_frame.delivery_tag)

consumerproducerexchange: importanttype: direct

queue: important-jobs

routing_key: important

Messages must be acknowledged

escenario 2

● default exchange in more detail● message to two queues, depending on routing key

‘important’ messages must be sent to queues ‘important-jobs’ and ‘traces’

consumerproducer

consumer

step 1- create & bind new queue

ch.queue_declare(queue='traces')ch.queue_bind(exchange='important', queue='traces', routing_key='important')ch.basic_publish(exchange='important', routing_key='important', body='[another task to be handled]')

consumerproducerexchange: importanttype: direct

queue: important-jobs

routing_key: important

consumerrouting_key: important

queue: traces

step 2- consuming from both

consumerproducerexchange: importanttype: direct

queue: important-jobs

consumerrouting_key: important

queue: traces

routing_key: important

● Binding the new ‘traces’ queue to existing ‘important’ exchange does not affect publishing code.

● Consumers and queues may be added dynamically without affecting the producer.

escenario 3● 2 x (binding + routing-key) => to 1 queue● exchange to exchange binding● headers exchange

‘customer’ messages to ‘important’ exchange must be sent to different queues depending on the operation to perform (‘signup’, ‘update’) and to ‘traces’

traces consumerproducer

update consumer

signup consumer

step 1- bind traces, declare q’s

routing_key: customer traces

consumerproducer

update consumer

signup consumer

exchange: importanttype: direct

queue: traces

queue: signup

queue: update

‘traces’ queue is bound to ‘important’ exch with two routing keys: ‘customer’ and ‘important’

ch.queue_bind(exchange='important', queue='traces', routing_key='customer')ch.queue_declare(queue='signup')ch.queue_declare(queue='update')

exchange: importanttype: direct

traces consumerproducer

update consumer

signup consumer

queue: traces

queue: signup

queue: update

exchange: customertype: headers

routing_key: customer

step 2- customer exchange & bind

ch.exchange_declare(exchange='customer', type='headers')ch.exchange_bind(source='important', destination='customer', routing_key='customer')ch.queue_bind(exchange='customer', queue='signup', arguments={'operation': 'signup', 'x-match':'any'})ch.queue_bind(exchange='customer', queue='update', arguments={'operation': 'update', 'x-match':'any'})

step 3- customer signup message

ch.basic_publish(exchange='important', routing_key='customer', body='cust num=25', properties=BasicProperties(headers=s{'operation': 'signup'})) method_frame, header_frame, msg = ch.basic_get('signup')print "msg received from queue 'signup' : ", msgch.basic_ack(method_frame.delivery_tag)

routing_key: customer traces

consumerproducer

update consumer

signup consumer

exchange: importanttype: direct

queue: traces

queue: signup

queue: update

exchange: customertype: headers

msg also routed to traces queue

bonus: topic exchanges

traces consumer

producer

customer consumer

signup consumerexchange: operations

type: topic

queue: traces

queue: signup

queue: update

r_k: #

r_k: *.signup

r_k: customer.*

topic exchanges allow to route messages based on topics. Examples from escenario 3:

○ Producer sends with routing keys: ‘customer.signup’, ‘customer.update’○ ‘traces’ consumer subscribes to ‘#’ that means all routing keys○ ‘signup’ customer consumer subscribes to ‘customer.signup’○ Consumer wanting all customer operations: ‘customer.*’○ Consumer wanting all signup operations: *.signup’

escenario 4● DeadLetter exchanges in RabbitMQ

What will we do with a message RabbitMQ cannot deliver?

(on client rejection, timeout, queue length exceeded)

By default those messages are dropped, we want not to lose them

consumerproducer

step 1- create rejected-jobs

ch.exchange_declare(exchange='rejected-jobs', type='direct')

ch.queue_declare(queue='rejected-jobs')

ch.queue_bind(exchange='rejected-jobs', queue='rejected-jobs', routing_key='important')

consumerproducer

queue: rejected-jobsexchange: rejected-jobstype: direct

step 2- deadlettr important-jobs

ch.queue_delete('important-jobs')ch.queue_declare(queue='important-jobs', arguments={'x-dead-letter-exchange': 'rejected-jobs'})ch.queue_bind(exchange='important', queue='important-jobs', routing_key='important')

consumerproducer

queue: rejected-jobsexchange: rejected-jobstype: direct

exchange: importanttype: direct

routing_key: important

x-dead-letter-exchange: rejected-jobs

step 3- consumer rejects job

ch.basic_publish(exchange='important', routing_key='important', body='[unparseable message]')

method_frame, header_frame, important_job = ch.basic_get('important-jobs')ch.basic_reject(method_frame.delivery_tag, requeue=False)

method_frame, header_frame, rejected_job = ch.basic_get('rejected-jobs')ch.basic_ack(method_frame.delivery_tag)

consumerproducer

queue: rejected-jobsexchange: rejected-jobstype: direct

exchange: importanttype: direct

routing_key: important

x-dead-letter-exchange: rejected-jobs

What are you waiting to start working with RabbitMQ?

It’s fun!

Questions?

Thank you for attending.