Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product...

© 2018 Rock Solid Knowledge.

Messaging and RabbitMQ

© 2018 Rock Solid Knowledge 2

Agenda

• The problem space

• Why queues?

• Reliability

• Failure Semantics

• Scalability and Availability


Terminology

• Broker

• Overall controller of messaging infrastructure

• Queue

• FIFO structure that stores messages

• Publisher

• A part of a system that creates messages and sends to a queue (also known as a producer)

• Consumer

• A part of a system that reads messages from a queue and performs any resultant processing


Why Queues?

• Decouple publisher and consumer

• Fully async

• Don’t have to be running concurrently

Publisher Consumer

Publisher’s World Consumer’s World


RabbitMQ

• Open Source Queuing Product

• Written in Erlang

• Designed for high performance

• Google have installation processing 1,000,000 message/second

• Flexible for most situations

• May require a little work

• .NET library on Nuget

• RabbitMQ.Client


Connections and Models

• Access RabbitMQ via a connection

• Authentication details

• Server (broker) address

• Perform actions via a model

• Recommended 1 model per thread

ConnectionFactory factory = new ConnectionFactory{

HostName = host,UserName = user,Password = password

};

IConnection connection = factory.CreateConnection();

IModel model = connection.CreateModel();


Publishing via Exchanges

• Publish to Exchange rather than queue

• Queues bound to exchanges via routing criteria

• Level of indirection increases flexibility

• Publication specifies a routing key for the message

• Determines routing matches for queues

• Routing matching rules depend on type of exchange

model.ExchangeDeclare(exchangeName, "direct", false, false);

string bodyMessage = "Hello, World!";byte[] body = Encoding.UTF8.GetBytes(bodyMessage);

model.BasicPublish(exchangeName, routingKey, false, null, body);


Messages

• Messages very flexible in RabbitMQ

• Can have body

• Can have headers

• Can modify behaviour with properties


Queues and Bindings

• Queues define where messages are stored

• Binding determines which queues receive a message when published

• All matches receive copy

• No match may or may not be an issue


model.QueueDeclare(queueName, false, false, false, null);

model.QueueBind(queueName, exchangeName, routingKey, null);


Consuming a queue

• Pass IBasicConsumer to model

• RabbitMQ library provides EventingBasicConsumer

• Each message arrives as an event

• Two consumers on same queue will not get same message

var consumer = new EventingBasicConsumer(model);

consumer.Received += (s, e) =>{

string bodyMessage = Encoding.UTF8.GetString(e.Body);

Console.WriteLine(bodyMessage); };

model.BasicConsume(queueName, true, consumer);


Load Leveling

• Queues provide load leveling automatically

• Consumer does not have to be able to process peak load

• Handling average load is enough unless processing latency is an issue


Messaging Patterns

• Can implement different messaging patterns in RabbitMQ

• Pub/Sub

• Broadcast

• Async Request/Response

• Different types of exchanges assist

• Direct

• Topic

• Header

• Fanout


Direct Exchange

• Published message routingKey must exact match routingKey of queue binding

• Most performant


model.QueueDeclare(queueName, true, false, false, null);

model.QueueBind(queueName, exchangeName, routingKey, null);

model.BasicPublish(exchangeName, routingKey, true, props, null);


Topic Exchange

• Published message routingKey pattern matched against routingKey of queue binding

• Pub/Sub common use case

• Routing key typically set of dot separated segments

• JobType.SalesReport.AcmeCorp

model.ExchangeDeclare(exchangeName, "topic", false, false);


model.QueueBind("reportQueue", exchangeName, "JobType.SalesReport.#", null);



Headers Exchange

• Published message headers must match headers specification of queue binding

• Can match on one or more headers

• Match type all or any

model.ExchangeDeclare(exchangeName, "headers", false, false);


var spec = new Dictionary<string, object>{

["priority"] = 10,["retryAttempts"] = 2,["x-match"] = "all"

};

model.QueueBind(”reportQueue", exchangeName, "", spec);


Fanout Exchange

• Published message delivered to all bound queues irrespective of routingKey

• Used for broadcast

FanoutExchange


Queues and Reliability

• Messages can be persisted

• Systems can recover easily after crashes

• Must be managed by message broker

Publisher Consumer

Publisher’s World Consumer’s World


RabbitMQ, Persistence and Durability

• Durability and Persistence are often confused in RabbitMQ

• Durability is about exchanges and queues – if the broker restarts are they automatically recreated?

• Persistence is about messages – if a machine restarts are the messages still available?


Durability

• Durable flag when declaring exchanges and queues

• Exchange/Queue will be recreated if broker restarts

• Can also mark an exchange/queue as auto-delete

• Will be deleted when last connection disconnects

• Also flag on declaration

model.ExchangeDeclare(exchangeName, "direct", true, false);


model.ExchangeDeclare(exchangeName, "direct", true, false);



Persistence

• Message can be persisted

• Queue must be durable

• Performance implications (1 – 2 orders of magnitude slower)

• Specify DeliveryMode = 2 in message properties

var props = model.CreateBasicProperties();

props.DeliveryMode = 2; // persistent message

model.BasicPublish(exchangeName, routingKey + "." + suffix, true, props, null);


Reliability (cont’d)

• Need to ensure message published and message processed

• Traditionally done through transactions

Publisher Consumer

Tx1 Tx2


RabbitMQ and Reliability

• RabbitMQ supports transactions but discouraged

• Heavy performance impact

• Instead RabbitMQ typically uses confirmations and acknowledgements

• Receiver of message must state success (Ack) or failure (NAck)

• Broker reports to publisher

• Consumer reports to broker


Tracking acknowledgements

• Every message published or received on a model is given a model specific numeric id

• DeliveryTag

• Incremented by one for each message

• Success or failure reported for each id

• Can report multiple successes or failures in one round trip

• Important for high volume systems


autoAck

• By default consumer uses automatic acknowledgements (autoAck)

• Message Ack’d once received but before processed so message can be lost

• Can disable autoAck in BasicConsume

• Consumer must now manually ack

model.BasicConsume(queueName, false, consumer);

consumer.Received += (s, e) => {try {

ProcessMessage(e.Body);model.BasicAck(e.DeliveryTag, false);

}catch (Exception exception) {

model.BasicNack(e.DeliveryTag, false, true);}

};


Publishing with Confirms

• Did the broker receive the message OK?

• Can turn on confirmations

• Broker fires model events: BasicAcks or BasicNacks

• DeliveryTag in event args identifies message

• May get multiple confirms in one event

model.ConfirmSelect();

model.BasicAcks += (s, e) =>{

// ...};

model.BasicNacks += (s, e) =>{

// ...};


Routing Failures

• What if message cannot be routed to a queue?

• Default to drop on floor

• Often valid for pub/sub scenarios

• Can specify the successful routing is mandatory



Failed mandatory routing

• Broker returns message to publisher using BasicReturn

• Need someway to match message publication to message return

• Message not identified with DeliveryTag

• Can create messageId header to track


Failure Semantics

• What happens if I fail to publish to queue?

• Depends on queue implementation

• What happens if I fail to process a message?

• Drop on floor?

• Requeue?

Consumer

Poison Message?

Poison Message Queue


RabbitMQ and Poisoned Messages

• Can configure dead letter exchange on queue

• Message sent to dead letter exchange if message NAck’d with requeue false

var args = new Dictionary<string, object>{

["x-dead-letter-exchange"] = "DeadLetters"};

model.QueueDeclare(queueName, true, false, false, args);


Is message poisoned?

• No built in poison message identification

• Cannot tell how many times a message has attempted processing

• Can keep track out of band if using unique message id

• Can add retry count into header – means manual requeuing rather than NAck


Security and Virtual Hosts

• RabbitMQ uses Virtual Hosts for isolation

• Exchanges and queues live in a virtual host

• Default virtual host is “/”

• User account authorization based on virtual host

• Configure

• Write

• Read

• Pattern matched to artifacts using regex


Summary

• Messaging/Queues powerful way to connect (micro)services that are not temporally bound

• Load balancing / leveling simple to achieve

• RabbitMQ popular messaging platform

• Very flexible

• Designed for high performance

• May need to build architecture for some features

Date post:	11-Aug-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product...

Documents