Easy enterprise application integration with RabbitMQ and AMQP

transcript

Emile Joubert

December 2010

The Advanced Message Queuing Protocol (AMQP) is an application layer

protocol for messaging. This talk introduces RabbitMQ, itsprominent multi-lingual open-source implementation.

Disclaimer

This session may contain product features that are currently under development.

This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.

"THESE FEATURES ARE REPRESENTATIVE OF FEATURE AREAS UNDER DEVELOPMENT. FEATURE COMMITMENTS ARE SUBJECT TO CHANGE, AND MUST NOT BE INCLUDED IN CONTRACTS, PURCHASE ORDERS, OR SALES AGREEMENTS OF ANY KIND. TECHNICAL FEASIBILITY AND MARKET DEMAND WILL AFFECT FINAL."

About me...

Emile studied computer science and mathematics before following a career that included electronic publishing; joining LShift in 2008 and SpringSource in 2010.

Outline

• Messaging• AMQP• Publishing walk-through• RabbitMQ and the benefits of Erlang• Large scale sample uses

Messaging Use Cases

Decoupling

Messaging Use Cases

Bidirectional Decouplinge.g. Remote Procedure Call

Messaging Use Cases

Pipelining and Decoupling

Messaging Use Cases

Work-distribution and Decouplinge.g. duplication and round-robin

Messaging Use Cases

Work aggregation, distribution and decoupling

Why Messaging?

SQL : : Databases : : FilesystemAMQP : : Messaging : : Network

• Logical decoupling• Physical decoupling• Temporal decoupling

• An open, wire-level binary protocol for

Message Orientated Middleware• Created by users and technologists

working together, to satisfy real needs• A straight-forward and complete

solution for business messaging: SMTP for business

Transport

Framing

AMQP Wire Format

AMQP Model

Network-friendly

Network-optimized

Multi-vendorinteroperability

Flexible routing/queuing parameters

AMQP Model

AMQP Specification Sample

AMQP Model

• Producers• Exchanges• Bindings• Queues• Consumers

Publishing: Setup

Exchanges, queues & bindings are created dynamically by producers & consumers

Publishing

Publishing a second message

Consuming

Consuming second message

Publishing Multiple Messages

...to several consumers on the same queue

Publishing to Multiple Queues

Sending

Messages go to every queue bond to a fanout exchange

Queues can be consumed from at different rates

Publishing: Topic Routing

* matches 1 word

# matches 0 or more words

Publishing: Topic Routing

AMQP Model Elements

Exchange Properties•passive•durable•auto-delete

Exchange Types•direct•fanout•topic•headers

Queue Properties•passive

•durable

•exclusive

•auto-delete

Message Properties•routing key

•immediate

•delivery-mode

•priority

•expiration

Prominent AMQP Implementations

RabbitMQ (Erlang)

OpenAMQ (C++)

Apache Qpid (Java and C++)

RedHat MRG (Messaging, Realtime & Grid) (C++)

Why Erlang?

• Concise (binary matching)• Sane concurrency (no shared state)• Built-in distributed database (Mnesia)• Multi-platform (Windows, Unix)

Erlang binary matching

decode_method('basic.publish', <<F0:16/unsigned, F1Len:8/unsigned, F1:F1Len/binary, F2Len:8/unsigned, F2:F2Len/binary, F3Bits:8>>)

->F3 = (F3Bits band 1) /= 0,F4 = (F3Bits band 2) /= 0, #'basic.publish'{ticket = F0, exchange = F1, routing_key = F2, mandatory = F3, immediate = F4};

What is RabbitMQ?

RabbitMQ consists of the broker (server) and several clients (Java, .net and others)

They speak the AMQP protocol to each other

Anyone can write other clients which also speak AMQP and work with RabbitMQe.g. Ruby, Java, Spring, Python, C, C#, Perl,

Erlang, Lisp, Haskell, PHP, <your language here>

It's in many popular Linux distributions and can easily be installed on OS X

Released under MPL

What is RabbitMQ?

17k lines of Erlang code in the brokerSupports clustering for increased

scalabilityCan work with Pacemaker and

associated tools to provide High Availability

Supports several extension points via plug-ins, e.g. exchange types and backing queue storage

AMQP compared to JMS

AMQP has a wire-level binary protocol, instead of an API

AMQP offers multi-language interoperability

AMQP message payloads are opaqueUse Spring-AMQP to migrate

Protocols

AMQP (SSL)STOMPXMPP (Rabbiter)0MQREST

Real-world features

•Unroutable messages•Client acknowledgements•Persistence•Transactions•Flow control•Security

Management & Monitoring

RabbitMQ plugin

• Inspect and manage resources

• Load & save configuration

• HTTP API

• Commandline tool

Cloud Ready

• Messaging service hosted in cloud• Cloud infrastructure services

• e.g. Nasa Nebula• Providing a service in the cloud

• e.g. SQS on Amazon

Nebula Architecture

Unique Identification Authority of India (UIDAI) will issue a12-digit for all residents

•One of the largest IT projects ever•Largest online identity DB ever•200 million messages per day peak•600 million residents over 4 years

• Spring• Tomcat• Hibernate• MySQL• Hadoop• Mule• Apache httpd• RabbitMQ

The future

• AMQP 1.0• Connect-a-thon

• <Your suggestion here>• Get in touch!• http://www.rabbitmq.com/

This presentation will be available at

http://www.springsource.com/webinars

from Monday 13 Deccember 2010

Easy enterprise application integration with RabbitMQ and AMQP

Technology