+ All Categories
Home > Documents > Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ......

Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ......

Date post: 09-Mar-2018
Category:
Upload: trinhhanh
View: 221 times
Download: 2 times
Share this document with a friend
24
Using AMQP, and RHM (messaging) to solve real problems Carl C. Trieloff May 10, 2007
Transcript
Page 1: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Using AMQP, and RHM (messaging) to solve real problems

Carl C. TrieloffMay 10, 2007

Page 2: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

What is all the fuss about?Common complaints in deploying distributed /highly scaled systems

● Cost – is most cases the deployment typology is dictated by licenses not architecture

● Features – most deployments have to create a set of services before they can start, these include routing, replay, IVQ, LVQ SoWQ etc..

● Openness ­ No way to create open market exchanges● Performance ­ Many have created proprietary solutions to meet 

throughput requirements● Pure standards model failures ­ Web Services has not delivered at 

the transport level● Scope ­ Need to be able to use Messaging from the OS up to the 

application level● Management ­ Need a better way to manage it...

Page 3: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

AMQP born out of users frustration ...John O'Hara from JPMC started working on defining AMQP out of frustration of 

seeing money aimlessly wasted on patching around the core issues● Messaging to simply solve 80% of enterprise use cases

● Pub/sub patterns● Large message transfer (including file)● Consumer patterns● Eventing patterns● Dynamic configuration . Wiring patterns● Task queue patterns

● High volume use cases● Fan out, and multicast● High through put

● Entirely open

Page 4: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

AMQP in action

AMQP

Server

AMQP

Client

AMQP

Client

AMQP

Client

AMQP

Client

Point­to­point messaging over dynamic queues

Request/Reply Publish/Subscribe 

Topic Hierarchies

Header BasedRouting

AMQP

Server

AMQP

Server

Intelligent WAN RoutingIntelligent WAN Routing

New York Hub London Hub Tokyo Hub

AMQP Deployed today at JPMC

Page 5: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

What is AMQPAn Open Standard for Middleware:

● Middleware: software that connects other software together.  Middleware connects islands of automation, both within an enterprise and out to external systems. 

Why it is different:● A straight­forward and complete solution for business messaging● Cost effective for pervasive deployment● Totally open (developed in partnership)● Created by users and technologists(Messaging, OS, and Network) working 

together● Made to satisfy real needs (IVQ, LVQ, Replay, ...)

Went public/open on June 20th 2006

Page 6: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

What does it do?Messaging Middleware should…

● Provide event notification, messaging, file transfer● Deals with business transaction processing● Technology agnostic (there is more than Java)

Meet real­world requirements of mission­critical systems

Be Trustworthy● Robust, available, scalable, secure, resilient● Aims to be stable over the long run● Provide a common infrastructure for the enterprise

AMQP meets these needs in one protocol● Usually provided by 3 different proprietary products● One solution reduces costs, increases efficiency and simplifies 

management

File Transfer

Messaging

Publish/Subscribe

detect

transact

report

Page 7: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

AMQP – an open practical standardAMQP is a specification for how commodity messaging & middleware works

● It is a complete specification; if it doesn’t contain all the information necessary to implement, we consider that a bug in the specification

● Anyone may use the AMQP specification to create useful implementations without being charged for the IP rights to do so

AMQP aims to be technology and language­neutral● C, C++, Java, .NET, C#, Ruby, Python, Perl and future languages● Older languages should be feasible also (COBOL)● Requires IP, and can be used with TCP, UDP, SCTP, Infiniband, ...

Products complying with AMQP are interoperable● AMQP is a Wire­Level protocol based on the ubiquitous IP● Wire level compatibility means it can be embedded in the network● Applications written to Product X will plug into servers running Product Y

Page 8: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

AMQP Protocol Layering

Transport

Framing

AMQP Wire Format

AMQP Model

Network­friendly

Network­optimized

Multi­vendorinteroperability

Flexible routing/queuing parameters

Messaging (incl. JMS), File Transfer, Transactions, etc.

TCPSCTPUDP   

IP

Infrastructure datais binary

Independent of JMS

“Exchanges” define flexible routing rules

Ethernet Infiniband

Page 9: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Understanding the AMQP ModelThe AMQP Architecture specifies modular components and rules as

building blocks

Connected into “ processing chains,” these create the desired messaging / middleware functionality, which is very extensible...

Exchanges

● The “ Exchange” receives messages from publisher applications and routes these to queues, based on arbitrary criteria—typically topic & message headers

Queues

● The “ Queue” stores messages until they can be safely processed by a consumer application (or multiple applications)

Bindings

● The “ Binding” defines the relationship between a queue and an exchange and provides the message routing criteria

Page 10: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

AMQP Working Group, Governance, feedback

Apache QpidASLv2

RHM/RHEM LGPL – from Qpid

AMQP Working Group Open Source and Closed Implementations

AMQ Protocol Working Group Controls the Specification

Diverse Products Implement the Specification

Review comments only- carefully preserve open

status of AMQP specification

WG Members

29 WestCisco

Credit SuisseEnvoyIONAiMatix

JPMorganRabbit

Red HatTWIST

Industry and Academic Reviewers

Provide Feedback under

RLA

Special Interest Groups create revisions

SIG's comprise WG members who are expert

practitioners

OpenAMQGPLv2

RabbitMQMPL

Page 11: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

How is Red Hat Messaging built?Apache Qpid: Upstream AMQP messaging code base

Red Hat / JBoss Community Repositories: Develop additional open source messaging components, assemble open source messaging distribution

Fedora/RHEL/Red Hat Messaging: Linux­native messaging distributions,Red Hat Enterprise Linux and non­Linux supported distributionsDistributions

Community integration repository

Currently in the Apache Incubator

Page 12: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

How does it fit together / what is it used for?Now becoming available (performance, determinism, value) ­­ combine

● AMQP middleware and● Linux Real Time Kernel and● Gigabit Ethernet or Infiniband networking, Async durable IO

   ...offers a compelling solution for mission critical or large scale deployment

Applicable to most lines of business Initiative started in Finance, but interest and applicable to Telcom, Government, 

Healthcare and e­commerce / Retail.Creating open exchanges Business applications

Value bearing transactions OS Services

Grids Eventing

Think what you can do if every OS has messaging, and you can deploy             based on optimum solution, not some random constraint

Page 13: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

What is the Red Hat MessagingRed Hat Messaging implements AMQP (Qpid upstream) and ....

● Provides a stable tested and supported distribution● Will be supported in multiple languages and platforms (non­Linux also)

Provides the fastest reliable­messaging platform● Adds optimizations and new IO layers for Linux● Can take advantage of realtime Linux work for predictability

Adds QoS features● Direct write and persistence alternatives● Integrates with OS Clustering facilities

Becomes standard facility● Can utilize, virt/Xen, OS security● Forms the basis for many other new distributed OS services

All work is open source !!

Page 14: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

RHM ComponentsThe Red Hat messaging project includes all the Qpid components and also adds 

persistence options and integration with OS facilities 

C++ and Java Brokers●  Both brokers will support Clustering, High availability, FT, Durable queues/ 

exchanges, redirection, security, transactions, persistence, ...● Integrates with existing PMS systems, TPMs

Clients● Java, includes JMS API and extensions for AMQP● C++ client on any platform● Dynamic languages Ruby & Python (<50k clients)● C# client for .NET● Perl requested ...

Page 15: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

RHM Performance now & goalsCurrent performance

●  300K+ (Java) msgs/second per node ­ 85% network saturation● C++ durable IO rates can max out a SANs block rate

Targets 07●  1+ Million msgs /second per cluster ­ transient● 500k+ msgs / second per cluster ­ durable

Targets 08● 1.9 Million msgs/ second per cluster ­ transient● 800k+ msgs / second per cluster ­ durable

Goal is to combine the high throughput with the predictability of Real­time natively into the Linux OS, but also support messaging on other platforms.

Page 16: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

How do I play with it / resources? An early version is included in Fedora Core 7 rpms are posted at 

● rhm.et.redhat.com/download Visit the Qpid project pages and mailing lists to speak with engineers

● incubator.apache.org/qpid● qpid­[email protected]

General information on RHM at● rhm.et.redhat.com

Page 17: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Is it prime time?Current: Preview builds / specific project support – AMQP is being used today 400Million messages per day in 7 hour window

End Q1­Q2 /07: Messaging Preview for RHEL 4/5 and C++ broker / clients in Fedora 7

Working with partners and strategic adopters

Will provide support to a limited set of early adopters, for developmentand deployment

Provide open subscription support

Q4 /Q1: Targeted mass support for Red Hat Messaging, update release based on adopter support

Page 18: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Realtime & Messaging go hand­in­handMost of the projects that drive realtime requirements include messaging

● Requirements cover: predictability, high speed, reliability, multi­vendor interoperability,  security and scalability.

Realtime has advantages for any message system, but unique advantages when combined with RHM – as RHM (AMQP) can take kernel advantages others can't.

Page 19: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Clustering

When deploying a cluster in a messaging system there, any configuration will trade off throughput, latency, and connection management.● Using RHEL 5 technology● Handle commodity and central storage scenarios

Throughput

ConnectionsLatency

q1q3fa­2

q2q1­1

q3faq1­2

q3fbq3fa­1

q3fb­1

q3fb­2

q2­1

q2­2

Deployments trade offs Cluster config examples

Page 20: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

DTX support ­ Most Common Use Case1.  start distributed transaction T1 2.  consume a message from Queue X3.  write to relational database4.  produce a new message on Queue Y (which may or may not be 

on the same broker process)5.  commit or rollback T1

T1

     

AMQPserver

AP

AMQPRMstub

AMQP protocol

TM XA

Native

AP/TM process

Page 21: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Example – Python (Publish / Get)client = connect()

chan = client.channel(0)

chan.channel_open()

chan.exchange_declare(0, "test", "direct")

chan.queue_declare(queue="test-queue")

chan.queue_bind(queue="test-queue", exchange="test", routing_key="key")

reply = chan.basic_consume(queue="test-queue")

print "consumer: %s" % reply.consumer_tag

queue = client.queue(reply.consumer_tag)

BODY = "Hello World!"

chan.basic_publish(exchange="test", routing_key="key", content=Content(BODY))

msg = queue.get()

print "received: %s" % msg.content.body

assert msg.content.body == BODY, "bad message body: %s" % msg.content.body

chan.basic_ack(msg.delivery_tag, True)

Page 22: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

ExampleC++ (publish / consume)

Exchange exchange("MyExchange", Exchange::TOPIC_EXCHANGE);

Queue queue("MyQueue", true);

Connection con(verbose); Channel channel;

string host("localhost");

con.open(host, 5672, "guest", "guest", "/test");

con.openChannel(channel);

channel.declareExchange(exchange);

channel.declareQueue(queue);

qpid::framing::FieldTable args;

channel.bind(exchange, queue, "MyTopic", args);

Monitor monitor; string tag("MyTag");

SimpleListener listener(&monitor);

channel.consume(queue, tag, &listener);

channel.start();

Message msg; string data("MyMessage");

msg.setData(data);

channel.publish(msg, exchange, "MyTopic");

Monitor::ScopedLock l(monitor);

monitor.wait();

channel.close();

con.close();

Page 23: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

ExampleC++ (topic listener)

Connection connection(args.getTrace());

connection.open(args.getHost(), args.getPort(), "guest", "guest", "/test");

Channel channel(args.getTransactional(), args.getPrefetch());

connection.openChannel(channel);

//declare exchange, queue and bind them:

Queue response("response");

channel.declareQueue(response);

Queue control;

channel.declareQueue(control);

qpid::framing::FieldTable bindArgs;

channel.bind(Exchange::STANDARD_TOPIC_EXCHANGE, control, "topic_control", bindArgs);

//set up listener

Listener listener(&channel, response.getName(), args.getTransactional());

string tag;

channel.consume(control, tag, &listener, args.getAckMode());

cout << "topic_listener: Consuming." << endl;

channel.run();

connection.close()

Page 24: Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Questions ....


Recommended