An Introduction to AMQP with Code Samples

transcript

What’s Messaging?

An Introduction

History of Message Queuing

Manual Telegraphy Machine Assisted Telegraphy

1920s1911 - 192019th Century 1900s

Telegrams sent using“Store and Forward”

Electronic Telegraphy

1950s1940s1950s1940s

Electronic Telegram Machines, eg Plan 55-A

IBM System/360 with BTAM & QTAM

Message Switching

Machine Assisted Telegraphy

1920s1911 - 1920 1930s

Telcos UseElectronic Telegraphy

1950s1940s 1960s

IBM System/360 with BTAM & QTAM

Message Switching

First Electronic Mail Solutions

Banking Users

IBM TCAM which is the first true solution

Retired 1990!

First Electronic Mail

Financial Trading UsersBanking Users FMCG & Utilities

1980s 1990s

Growth of SMTPOrigins of Tibco in

Stock Price Messaging

IBM Launch MQSeries(now WebsphereMQ)

IBM TCAM which is the first true solution

Retired 1990!

First Electronic Mail

FMCG & Utilities

IBM Launch MQSeries(now WebsphereMQ)

Machine Assisted Telegraphy Corporates Large Websites YOU

Noughties Today

Sun Release Java JMS, Reinvigorating

Enterprise Messaging

AMQP Working Group Formed by Investment Banks

Cloud Enables and Drives StormMQ

Adoption

The Integration Tag Soup

WS – File Transfer

!"#$%&'(#)#*#+%+

Message QueuingSOAP

!).$%&'/$.0%&1$%+

234%5)6!%$7%$

SOAMessage

OrientatedMiddleware

MQSeries

Amazon SQS

StormMQ

RabbitMQ

TCP/IPDCE / RPC

RMIXML-RPC

.NET Remoting

Remote Procedure Call

XML-RPC

The Integration Tag Soup

WS – File Transfer

!"#$%&'(#)#*#+%+

Message QueuingSOAP

!).$%&'/$.0%&1$%+

234%5)6!%$7%$

SOAMessage

OrientatedMiddleware

MQSeries

Amazon SQS

StormMQ

RabbitMQ

TCP/IPDCE / RPC

RMIXML-RPC

.NET Remoting

XML-RPC

Under Examination, though

File Transfer

!"#$%&'(#)#*#+%+

Don’t Work in the Cloud

Under Examination, though

Deployment

Intimate System Knowledge

Configuration

File Transfer

!"#$%&'(#)#*#+%+

Don’t Work in the Cloud

Scaling

Ideal: Message Queuing

Systems are “loosely-coupled”

MessageQueuing

Systems don’t know each other

Why Use it: Loose Coupling

BillingCatalogue

Shipping S-a-a-S Inventory

better position: Why do people

queuing? fonts

How do we connect them, without one outage or system change taking everything down like a pack of cards?

Why Use it: Loose Coupling

BillingCatalogue

Shipping S-a-a-S Inventory

better position: Why do people

queuing? fonts

How do we connect them, without one outage or system change taking everything down like a pack of cards?

Message Queuing lets Systems and Components exchangedata, events, commands and actions with one another

with no explicit knowledge or need for them to be online

It should be ideal for the cloud

!!!Until today, messaging is …

(ZeroMQ)

(SQS uses HTTP)

(MSMQ uses COM)

(memcached)

(Java JMSuses

Source)

(MQSeries / WebsphereMQ)

(Talarian, Rendezvous, etc)

Platform Restricted BespokeProprietary

!!!Until today, messaging is …

(ZeroMQ)

(SQS uses HTTP)

(MSMQ uses COM)

(memcached)

(Java JMSuses

Source)

(MQSeries / WebsphereMQ)

(Talarian, Rendezvous, etc)

Why is it Hell?

You need more staff And moneyYou need bridge technology

Why is it Hell?

You need more staff And moneyYou need bridge technology

However, there’s a solution

A-MQPAdvanced Message Queue Protocol

However, there’s a solution

However, AMQP fixes this

A common wire-level binary format and protocol

An explicit definition of a server (aka broker)’s

semantics

Open Means

Interoperable

That is good …

“AMQP will be to Messaging what HTTP was to the Web”

Clients run on any Platform Vendors are Interoperable

That is good …

“AMQP will be to Messaging what HTTP was to the Web”

Clients run on any Platform Vendors are Interoperable

From Anywhere to Anywhere

Quick Recap

For BeerThe fifth male memberof the A-Team, Frankie,

was played by

Question

Message Queuing connects systems and components. Is it ideal for the cloud?

Does Loose-Coupling make individualsystems more likely to suffer outages?

AMQP is Open.

This makes it suitable for programmingin C and Javascript?

Eddie Velez

Quick Recap

The fifth male memberof the A-Team, Frankie,

was played by

Question

Message Queuing connects systems and components. Is it ideal for the cloud?

Does Loose-Coupling make individualsystems more likely to suffer outages?

AMQP is Open.

This makes it suitable for programmingin C and Javascript?

Common Terms and Jargon

Jargon

Messaging: Which Jargon?

EMail(SMTP, POP3, IMAP)

VoIP(VoiceMail, XMPP)

Texting(SMS)

Instant Messaging(ICQ, MSN, Jabber)

Twitter

Enterprise Service Bus(ESB)

Dynamic OO Languages(eg Ruby)

Message Queuing(MQ)

What do wemean by

Messaging?

Essential Terms

Message Queue

Behaves like a Queue

Is First-In First-Put (FIFO)

Elements are Messages

Essential Terms

Message Queue What’s a Message?

Envelope

Payload

Header

Essential Terms

*Strictly speaking

a receiver polls for messages

a consumer has messages pushed

≣ ≊Send Publish Enqueue

≣ ≊ DequeueConsume*Receive

So how can you use it?

Concepts

Useful Book:Enterprise

Integration PatternsHohpe & Woolf

Store and Forward

Billing

“Could you send me a duplicate of my bill please”

Store and Forward

Billing

REST !

Store and Forward

Billing

StormMQMessaging

Billing

Store and Forward

Billing

StormMQMessaging

Billing

Store and Forward

Billing

StormMQMessaging

Billing

Fire and Forget

Shipping

“Too many orders. The website is running like a dog.”

SQL Push SQL Pull

Fire and Forget

Shipping

“Too many orders. The website is running like a dog.”

SQL Push SQL Pull

StormMQMessaging

Shipping

One-To-Many

“There’s a new widget in the Catalogue: Tell all the systems.”

CatalogueShipping

S-a-a-S Inventory

File Transfer

One-To-Many

“There’s a new widget in the Catalogue: Tell all the systems.”

CatalogueShipping

S-a-a-S InventoryStormMQMessaging

Publish-Subscribe (“Topics”)

“Shipments Sent, Delivered and Returned”

Shipping

S-a-a-S InventoryBilling

AnySentSent or Returned

StormMQMessaging

Publish-Subscribe (“Topics”)

“Shipments Sent, Delivered and Returned”

Shipping

S-a-a-S InventoryBilling

AnySentSent or Returned

Round-Robin

Billing

“How do we easily scale a massive batch job like Billing?”

EMail BillGenerate

StormMQMessaging

Round-Robin

Billing

“How do we easily scale a massive batch job like Billing?”

EMail BillGenerate

Getting Technical

Delving into AMQP

AMQP Client AMQP ServerTCP / IP Network

Connection Virtual Host

Connections and Channels

Connection

AMQP Client AMQP ServerTCP / IP Network

ConnectionVirtual Host

Connections and Channels

TLS “Shielding”

Channels

Each Channel is Independent:Effectively, a Virtual Connection

Basic AMQP: Connections

Open a Connection to a Virtual Host

Open a Channel

Send a Message

Receive a Message

Close Channel

Close Connection

You only need one channel!

How does this lookin Code?

// The API is styled similarly to Functionally Orientated Programming (FOP)// It makes extensive use of Constructor Injection and anonymous classes modelling 'blocks'// Closes are handled implicitly by design with exceptions thrown, ie try-finally is internal to the API

public void demonstrateConnection(){

final AmqpConnectionFactory amqpConnectionFactory = amqpConnectionFactoryWhichVerifiesStormMQ(“mycompany”, “mysystem”, “development”, “user”, “password”);

// Creates and closes a connectionamqpConnectionFactory . useConnection( // Creates and closes a channel on a connection

new ChannelCreatingConnectionUser( // Adapts a channel to make it convenient to use new ConvenientChannelCreatingConnectionUser

(new ConvenientChannelUser()(

public void use(final @NotNull ConvenientChannel convenientChannel){

// Send a message, or,// Receive a message (or both)

// Sending a Message

// ConvenientChannel has static convenience methods to create messageHeaders (meta-data)// Use them to signify a message's content’s MIME type or an unique id// Use meta data sparingly; think of it like email headers. More laterfinal BasicProperties messageHeaders = emptyBasicProperties();

// All messages are byte arraysfinal byte[] messageBody = "HelloWorld" . getBytes( forName("UTF-8") );

// Send the messagefinal String routingKey = "usually the destination message queue";final boolean mandatory = false;final boolean immediate = false;convenientChannel . basicPublish( "", routingKey, mandatory, immediate, messageHeaders, messageBody );

/* Notes * Mandatory (mandatory = true) messages are returned unless routed to a queue * Immediate (mandatory = true) messages are returned unless there is another channel consuming */

// Receiving a Message using Polling

final boolean noAck = false;convenientChannel . basicGet( "queue", noAck, DoNothingWhenMessageNotReceived, new MessageReceived() {

public void messageReceived(final @NotNull GetResponse message){

final byte[] messageBody = message . getBody();final BasicProperties messageHeaders = message . getProps();

// Acknowledge successfully received messages if noAck = false// If not acknowledged, they are eventually placed back on the queue for re-deliveryconvenientChannel . basicAck( message );

/* Notes * DoNothingWhenMessageNotReceived is a singleton of MesageNotReceived * Implement DoNothing to do something else, eg for debugging * Polling is thread-safe but inefficient; using consumers (“push”) for more efficiency… */

// Consuming Messages using Push

final boolean noAck = true;convenientChannel . basicConsume( "my queue", noAck, new ConvenientConsumer() {

// Consumers run on a different threadpublic void handleDelivery( final String consumerTag, final Envelope envelope, final BasicProperties messageHeaders, final byte[] messageBody ){

final byte[] messageBody = message . getBody();final BasicProperties messageHeaders = message . getProps();

// No need to acknowledge message as noAck = true

/* Notes * Consumers only exist as long as a channel is open * Add a threading primitive (eg volatile boolean) and reference it at * * It is possible to use multiple channels with several consumers… */

Channels

Complex ApplicationsConsume on several threads

Send on several threads

Why Use More than One Channel?

Recommendation: Keep it Simple!

Simple ApplicationsSend on a queue

Receive on another

Transactional ApplicationsNeed Transactions on a queue

Don’t Need Transactions on another

Exchanges Route Messages

Exchanges route Messages to Message Queues

Exchange

Exchanges Route Messages

Exchanges route Messages to Message Queues

You send messages to an exchange, not a message queue

Exchange

How do Exchanges Route?

Exchange

The Exchange finds a Bindingmatching the Routing Key#

A Binding connects a Routing Keyto one or more Message Queues

$Every sent message has a

Routing Key"

The Exchange delivers theMessage to the Message Queue%

A message queue can bebound more than once to

one or more exchanges(

“string”

Types of Exchange

More rarely used exchange types include amq.headers and extensions

Message Queue Name

direct

Like a ‘Map’: All MQs bound with the

routing key receive copies of the message

“” (blank)amq.direct

fanout

Empty String

All MQs bound to the Exchange receive

copies of the message

amq.fanout

Dotted

Bindings useglobbing expressions (wildcards) to route messages to MQs

amq.topic

A message queue can be bound more than once to an exchange;A message queue can be bound to more than one exchange

But a message queue will only receive a message sent once

Routing Key

RoutingBehaviour

Default Definitions

Point – to – Point One – to – Many Publish – SubscribeTypical Use

POLL VS PUSH

// Declaring Exchanges

final boolean passivelyDeclare = false;final boolean durable = true;final boolean autoDelete = false;

// Just change the ExchangeType to one of direct, fanout or topicconvenientChannel . exchangeDeclare( "my exchange A", direct, passivelyDeclare, durable, autoDelete );convenientChannel . exchangeDeclare( "my exchange B", fanout, passivelyDeclare, durable, autoDelete );convenientChannel . exchangeDeclare( "my exchange C", topic, passivelyDeclare, durable, autoDelete );

// A passive declaration asserts that an exchange exists, causing a channel exception if it does not convenientChannel . exchangeDeclare( "my exchange A", direct, true, durable, autoDelete );

/* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as exchanges should be long-lived * Declaring an exchange twice is not an error; if parameters differ, new values are used if possible * Passive declaration is interesting but typical deployment practices mitigate its usefulness */

// Declaring and Binding Message Queues

final boolean passivelyDeclare = false;final boolean durable = true;final boolean autoDelete = false;final boolean exclusive = false;

// A declared queue is bound to the default ("") exchange with the routing key the queue name ("my queue")convenientChannel . queueDeclare( "my queue", passivelyDeclare, durable, exclusive, autoDelete );

// A message queue can be bound to another exchange with whatever key you chooseconvenientChannel . queueBind( "my queue", "another exchange", "some routing key" );

// When binding to a fanout exchange the routing key is the empty string ("")convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.fanout", "" );

// When binding to a topic exchange, set the routing key as a wildcard// Publish messages with specific routing keys eg "BONDS.USD.FTSE", "BONDS.USD.NYSE", "BONDS.GBP.FTSE"convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.topic", "BONDS.USD.*" );

/* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as message queues should be long-lived * Declaring a message queue twice is not an error; if parameters differ, new values are used if possible * Exclusive message queues are for advanced usages and often aren’t needed */

AMQP: Introducing Users

StormMQ enforces that the sameUsers and Permissions are in all of a System’s Environments

Passwords are different

Passwords are securely generatedby StormMQ and are 512 bit keys

Configuration Managers can prevent revelation of passwords to different

system users, eg developers vs sysadmins

Aim: Separation of Environments

A Connection can use several authentication mechanisms

which are per Virtual Host

Commonest is User - Password

andand

Aim: Separation of Responsibility

Users are intended to be system accounts (robots), not sysadmins

Users have read, write and create permissions using a wildcard syntax: important to name queues, etc well

// Example of create-system.json used with API call stormmq-create-system{

"companyName" : "usuallyYourUserName","systemName" : "anythingAlphabetic","environments" :[{

"environmentName" : "development",// Different StormMQ clusters have different characteristics (and costs)"clusterName" : "free-1",

// These are the StormMQ user accounts to which passwords for AMQP users (below) can be revealed for this environment (development)"permittedStormMQUserNames" : [ "developerPeter", "developerJames", "developerJohn" ]

// User permissions operate on Message Queues and Exchanges"amqpUserPermissions" :{

"queueReader" : { "create" : "^$", "read" : ".*", "write" : "^$" },"queueWriter" : { "create" : "^$", "read" : "^$", "write" : ".*" },"queueCreator" : { "create" : ".*", "read" : "^$", "write" : "^$" },"onlyProcessingObjects" : { "create" : "^processing.*", "read" : "^processing.*", "write" : "^processing.*" }

Configuring AMQP

“*Our REST API has Java, Ruby, PHP and .NET bindings”

AMQP StormMQ

✓ ActiveMessage Queues

Exchanges

Bindings

User Permissions

Entire Systems

! Extra

✓ Active

✓ REST

✓ REST*

Programmatic Config

✓ Same

AMQP: Getting Statistics

“*Password revelation is restricted to specific individuals”

AMQP StormMQ

Message Queues

Exchanges

Bindings

Generated Passwords

Usage Patterns

Entire Systems

! Extra

✓ REST

Programmatic Info

! Extra ✓ REST

! Extra ✓ REST*

! Extra ✓ REST

// Getting Statistics from StormMQ using the REST API in Java

// If you need to use proxies or experience the pain of Java SSL choose a different method in ApiConfigurationfinal ApiConfiguration apiConfiguration = directConnectionIfHavingProblemsWithSslVerification();

// If your Magic Secret Key is in an unusual place choose a different method in Apifinal Api api = apiByGuessingUsingOsConfigurationFiles( apiConfiguration, stormMqUserName );

final List<AmqpQueue> amqpQueues = api . listQueues( companyName, systemName, environmentName );

final List<AmqpExchange> amqpExchanges = api . listExchanges( companyName, systemName, environmentName );

final List<AmqpBinding> amqpBindings = api . listBindings( companyName, systemName, environmentName );

final Map<AmqpUserName, AmqpUserPassword> passwords =api . listAmqpUserNamesAndPasswords( companyName, systemName, environmentName );

final CompanyDetails companyDetails = api . describeCompany( companyName );

// Finding “what we’ve got”final List<CompanyName> companyNames = api . listCompanies();final List<ClusterName> clusterNames = api . listClusters( companyName );final List<SystemName> systemNames = api . listSystems( companyName );final SystemDetails systemDetails = api . describeSystem( companyName, systemName );final Set<Environment> environments = systemDetails . getEnvironments();

/* Notes * Also consider the command-line tools stormmq-list-queues, stormmq-list-exchanges and stormmq-list-bindings * Further examples are in com.stormmq.api.Example (part of the url signer jar) */

Virtual Hosts

Virtual Hosts: Think Apache!

Virtual Hosts isolate Message Queues

Virtual Hosts isolate User Groups*

Virtual Hosts isolate Exchanges*

Virtual Hosts isolate Bindings*

Virtual Hosts isolate

Virtual Hosts: StormMQ

Virtual Hosts isolate Your Company

Virtual Hosts isolate Your Systems

Virtual Hosts isolate Your Environments

Enable Configuration Management

Virtual Hosts stop Data Accidents

Widgets Ltd /

Invoicing System /

Testing

Development

Production

Message Properties

Server Understood Good Practice

“This is just the tip of messaging best practice with AMQP”

Persistent = 1Priority = 0Timestamp = 4568965345MessageId = 567A-GH-20100709

Priority Queues are per-Message

Control life of a Message

Persistence is per-Message

Timestamps expire Messages

Completely Optional

MIME Content-Type

Unique MessageId

Schema & Version

// Message Properties: Examples of Good Practice

// Being explicit is good practicefinal BasicProperties messageHeaders = new BasicProperties();{{

DeliveryMode . NonPersistent . setOn( this );Priority . Zero . setOn( this );setMessageId( "My unique ID, eg a GUID or ascending value" );

// Explicit MIME type including charset; use application/octect-stream for trully binary datasetContentType("application/json; charset=utf-8");

// Use compression for larger textual data; use the same values as HTTP headers; use "identity" for nonesetContentEncoding( "gzip" );

// Timestamp each messagesetTimestamp( new Date( System . currentTimeMillis() ) );

// Schema and version, perhaps using a XML namespace definitionsetType( "syslog-message_1.0.4" );

// Optional: Use an expiration to discard old messages that are stale (milliseconds, example is 5 minutes)setExpiration( Integer . toString( 5 * 60 * 1000) );

// Optional: Identify the sender for later debuggingsetAppId( "sending application name" );setUserId( "role of sender, or, user of machine" );

// Optional: Identify domain or environment (consumer could error if working in different one)setClusterId( "production" );

And there is more!

Transactions

More Message Properties

Custom Message Properties

Immediate Delivery

Additional Exchange Types

Auto Deletion

Fine-Grained User Permissions

QueuePurging

Useful Book:Enterprise

Integration PatternsHohpe & Woolf

An Introduction to AMQP with Code Samples

Technology