Date post: | 10-May-2015 |
Category: |
Technology |
Upload: | stormmq |
View: | 3,477 times |
Download: | 5 times |
What’s Messaging?
An Introduction
History of Message Queuing
Manual Telegraphy Machine Assisted Telegraphy
1920s1911 - 192019th Century 1900s
Telegrams sent using“Store and Forward”
1900
1930s
History of Message Queuing
Electronic Telegraphy
1950s1940s1950s1940s
Electronic Telegram Machines, eg Plan 55-A
1948
IBM System/360 with BTAM & QTAM
Message Switching
Machine Assisted Telegraphy
1920s1911 - 1920 1930s
History of Message Queuing
Telcos UseElectronic Telegraphy
1950s1940s 1960s
IBM System/360 with BTAM & QTAM
Message Switching
1964
First Electronic Mail Solutions
1965
Banking Users
1970s
IBM TCAM which is the first true solution
Retired 1990!
1971
Machine Assisted Telegraphy
History of Message Queuing
First Electronic Mail
Financial Trading UsersBanking Users FMCG & Utilities
1980s 1990s
Growth of SMTPOrigins of Tibco in
Stock Price Messaging
1980s
IBM Launch MQSeries(now WebsphereMQ)
1992
1970s
IBM TCAM which is the first true solution
Retired 1990!
1971
Machine Assisted Telegraphy
History of Message Queuing
First Electronic Mail
FMCG & Utilities
1990s
IBM Launch MQSeries(now WebsphereMQ)
1992
Machine Assisted Telegraphy Corporates Large Websites YOU
Noughties Today
Sun Release Java JMS, Reinvigorating
Enterprise Messaging
2001
AMQP Working Group Formed by Investment Banks
2006
Cloud Enables and Drives StormMQ
Adoption
2009
The Integration Tag Soup
WS – File Transfer
!"#$%&'(#)#*#+%+
Message QueuingSOAP
SMTP
FTP
RSYNC
NTFS
!','-
!).$%&'/$.0%&1$%+
234%5)6!%$7%$
SOAMessage
OrientatedMiddleware
JMS
MQ
MQSeries
AMQP
Amazon SQS
Tibco
StormMQ
RabbitMQ
HTTP
TCP/IPDCE / RPC
CORBA
DCOM
REST
RMIXML-RPC
.NET Remoting
Remote Procedure Call
XML-RPC
The Integration Tag Soup
WS – File Transfer
!"#$%&'(#)#*#+%+
Message QueuingSOAP
SMTP
FTP
RSYNC
NTFS
!','-
!).$%&'/$.0%&1$%+
234%5)6!%$7%$
SOAMessage
OrientatedMiddleware
JMS
MQ
MQSeries
AMQP
Amazon SQS
Tibco
StormMQ
RabbitMQ
HTTP
TCP/IPDCE / RPC
CORBA
DCOM
REST
RMIXML-RPC
.NET Remoting
Remote Procedure Call
XML-RPC
Under Examination, though
File Transfer
!"#$%&'(#)#*#+%+
Remote Procedure Call
Don’t Work in the Cloud
Under Examination, though
Deployment
Intimate System Knowledge
Configuration
Admin
File Transfer
!"#$%&'(#)#*#+%+
Remote Procedure Call
Don’t Work in the Cloud
Scaling
Ideal: Message Queuing
Systems are “loosely-coupled”
MessageQueuing
Ideal: Message Queuing
Systems are “loosely-coupled”
MessageQueuing
Ideal: Message Queuing
Systems are “loosely-coupled”
MessageQueuing
Systems don’t know each other
Why Use it: Loose Coupling
BillingCatalogue
Shipping S-a-a-S Inventory
better position: Why do people
queuing? fonts
How do we connect them, without one outage or system change taking everything down like a pack of cards?
Why Use it: Loose Coupling
BillingCatalogue
Shipping S-a-a-S Inventory
better position: Why do people
queuing? fonts
How do we connect them, without one outage or system change taking everything down like a pack of cards?
Message Queuing lets Systems and Components exchangedata, events, commands and actions with one another
with no explicit knowledge or need for them to be online
It should be ideal for the cloud
It should be ideal for the cloud
!!!Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMSuses
Source)
(MQSeries / WebsphereMQ)
(Talarian, Rendezvous, etc)
Platform Restricted BespokeProprietary
!!!Until today, messaging is …
(ZeroMQ)
(SQS uses HTTP)
(MSMQ uses COM)
(memcached)
(Java JMSuses
Source)
(MQSeries / WebsphereMQ)
(Talarian, Rendezvous, etc)
Why is it Hell?
You need more staff And moneyYou need bridge technology
Why is it Hell?
You need more staff And moneyYou need bridge technology
However, there’s a solution
A-MQPAdvanced Message Queue Protocol
However, there’s a solution
However, AMQP fixes this
A common wire-level binary format and protocol
An explicit definition of a server (aka broker)’s
semantics
Open Means
Interoperable
That is good …
“AMQP will be to Messaging what HTTP was to the Web”
MRG
Clients run on any Platform Vendors are Interoperable
That is good …
“AMQP will be to Messaging what HTTP was to the Web”
MRG
Clients run on any Platform Vendors are Interoperable
64K
From Anywhere to Anywhere
From Anywhere to Anywhere
From Anywhere to Anywhere
From Anywhere to Anywhere
From Anywhere to Anywhere
Quick Recap
For BeerThe fifth male memberof the A-Team, Frankie,
was played by
Question
Message Queuing connects systems and components. Is it ideal for the cloud?
Does Loose-Coupling make individualsystems more likely to suffer outages?
Yes
No
AMQP is Open.
This makes it suitable for programmingin C and Javascript?
Eddie Velez
Quick Recap
✓
The fifth male memberof the A-Team, Frankie,
was played by
Question
Message Queuing connects systems and components. Is it ideal for the cloud?
Does Loose-Coupling make individualsystems more likely to suffer outages?
Yes
No
AMQP is Open.
This makes it suitable for programmingin C and Javascript?
!
✓
Common Terms and Jargon
Jargon
Messaging: Which Jargon?
EMail(SMTP, POP3, IMAP)
VoIP(VoiceMail, XMPP)
Texting(SMS)
Instant Messaging(ICQ, MSN, Jabber)
Enterprise Service Bus(ESB)
Dynamic OO Languages(eg Ruby)
Message Queuing(MQ)
Message Queuing(MQ)
What do wemean by
Messaging?
Essential Terms
Message Queue
Behaves like a Queue
Is First-In First-Put (FIFO)
Elements are Messages
Essential Terms
Message Queue What’s a Message?
Envelope
Payload
Header
Essential Terms
*Strictly speaking
a receiver polls for messages
a consumer has messages pushed
≣ ≊Send Publish Enqueue
≣ ≊ DequeueConsume*Receive
So how can you use it?
Concepts
Useful Book:Enterprise
Integration PatternsHohpe & Woolf
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST !
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST
StormMQMessaging
Billing
!
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST
StormMQMessaging
Billing
!!
Store and Forward
Billing
“Could you send me a duplicate of my bill please”
REST
StormMQMessaging
Billing
!
Fire and Forget
Shipping
“Too many orders. The website is running like a dog.”
SQL Push SQL Pull
Fire and Forget
Shipping
“Too many orders. The website is running like a dog.”
SQL Push SQL Pull
StormMQMessaging
Shipping
One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
CatalogueShipping
S-a-a-S Inventory
File Transfer
ETL
One-To-Many
“There’s a new widget in the Catalogue: Tell all the systems.”
CatalogueShipping
S-a-a-S InventoryStormMQMessaging
Publish-Subscribe (“Topics”)
“Shipments Sent, Delivered and Returned”
Shipping
S-a-a-S InventoryBilling
AnySentSent or Returned
StormMQMessaging
Publish-Subscribe (“Topics”)
“Shipments Sent, Delivered and Returned”
Shipping
S-a-a-S InventoryBilling
AnySentSent or Returned
Round-Robin
Billing
“How do we easily scale a massive batch job like Billing?”
EMail BillGenerate
StormMQMessaging
Round-Robin
Billing
“How do we easily scale a massive batch job like Billing?”
EMail BillGenerate
Getting Technical
Delving into AMQP
AMQP Client AMQP ServerTCP / IP Network
Connection Virtual Host
Connections and Channels
Connection
AMQP Client AMQP ServerTCP / IP Network
ConnectionVirtual Host
Connections and Channels
TLS “Shielding”
Channels
Each Channel is Independent:Effectively, a Virtual Connection
Basic AMQP: Connections
Basic AMQP: Connections
Open a Connection to a Virtual Host
Open a Channel
Send a Message
Receive a Message
Close Channel
Close Connection
You only need one channel!
"
#
$
%
&
'
How does this lookin Code?
// The API is styled similarly to Functionally Orientated Programming (FOP)// It makes extensive use of Constructor Injection and anonymous classes modelling 'blocks'// Closes are handled implicitly by design with exceptions thrown, ie try-finally is internal to the API
public void demonstrateConnection(){
final AmqpConnectionFactory amqpConnectionFactory = amqpConnectionFactoryWhichVerifiesStormMQ(“mycompany”, “mysystem”, “development”, “user”, “password”);
// Creates and closes a connectionamqpConnectionFactory . useConnection( // Creates and closes a channel on a connection
new ChannelCreatingConnectionUser( // Adapts a channel to make it convenient to use new ConvenientChannelCreatingConnectionUser
(new ConvenientChannelUser()(
public void use(final @NotNull ConvenientChannel convenientChannel){
// Send a message, or,// Receive a message (or both)
});
))
)}
// Sending a Message
public void use(final @NotNull ConvenientChannel convenientChannel){
// ConvenientChannel has static convenience methods to create messageHeaders (meta-data)// Use them to signify a message's content’s MIME type or an unique id// Use meta data sparingly; think of it like email headers. More laterfinal BasicProperties messageHeaders = emptyBasicProperties();
// All messages are byte arraysfinal byte[] messageBody = "HelloWorld" . getBytes( forName("UTF-8") );
// Send the messagefinal String routingKey = "usually the destination message queue";final boolean mandatory = false;final boolean immediate = false;convenientChannel . basicPublish( "", routingKey, mandatory, immediate, messageHeaders, messageBody );
}
/* Notes * Mandatory (mandatory = true) messages are returned unless routed to a queue * Immediate (mandatory = true) messages are returned unless there is another channel consuming */
// Receiving a Message using Polling
public void use(final @NotNull ConvenientChannel convenientChannel){
final boolean noAck = false;convenientChannel . basicGet( "queue", noAck, DoNothingWhenMessageNotReceived, new MessageReceived() {
public void messageReceived(final @NotNull GetResponse message){
final byte[] messageBody = message . getBody();final BasicProperties messageHeaders = message . getProps();
// Acknowledge successfully received messages if noAck = false// If not acknowledged, they are eventually placed back on the queue for re-deliveryconvenientChannel . basicAck( message );
}});
}
/* Notes * DoNothingWhenMessageNotReceived is a singleton of MesageNotReceived * Implement DoNothing to do something else, eg for debugging * Polling is thread-safe but inefficient; using consumers (“push”) for more efficiency… */
// Consuming Messages using Push
public void use(final @NotNull ConvenientChannel convenientChannel){
final boolean noAck = true;convenientChannel . basicConsume( "my queue", noAck, new ConvenientConsumer() {
// Consumers run on a different threadpublic void handleDelivery( final String consumerTag, final Envelope envelope, final BasicProperties messageHeaders, final byte[] messageBody ){
final byte[] messageBody = message . getBody();final BasicProperties messageHeaders = message . getProps();
// No need to acknowledge message as noAck = true
}
});}
/* Notes * Consumers only exist as long as a channel is open * Add a threading primitive (eg volatile boolean) and reference it at * * It is possible to use multiple channels with several consumers… */
*
!
✓
Channels
Complex ApplicationsConsume on several threads
Send on several threads
Why Use More than One Channel?
Recommendation: Keep it Simple!
Simple ApplicationsSend on a queue
Receive on another
Transactional ApplicationsNeed Transactions on a queue
Don’t Need Transactions on another
✓
Exchanges Route Messages
Exchanges route Messages to Message Queues
Exchange
MQ
A
MQ
B
Exchanges Route Messages
Exchanges route Messages to Message Queues
You send messages to an exchange, not a message queue
Exchange
MQ
A
MQ
B
How do Exchanges Route?
Exchange
MQ
A
The Exchange finds a Bindingmatching the Routing Key#
A Binding connects a Routing Keyto one or more Message Queues
$Every sent message has a
Routing Key"
The Exchange delivers theMessage to the Message Queue%
A message queue can bebound more than once to
one or more exchanges(
“string”
Types of Exchange
More rarely used exchange types include amq.headers and extensions
Message Queue Name
direct
Like a ‘Map’: All MQs bound with the
routing key receive copies of the message
“” (blank)amq.direct
fanout
Empty String
All MQs bound to the Exchange receive
copies of the message
amq.fanout
topic
Dotted
Bindings useglobbing expressions (wildcards) to route messages to MQs
amq.topic
A message queue can be bound more than once to an exchange;A message queue can be bound to more than one exchange
But a message queue will only receive a message sent once
Routing Key
RoutingBehaviour
Default Definitions
Point – to – Point One – to – Many Publish – SubscribeTypical Use
POLL VS PUSH
// Declaring Exchanges
public void use(final @NotNull ConvenientChannel convenientChannel){
final boolean passivelyDeclare = false;final boolean durable = true;final boolean autoDelete = false;
// Just change the ExchangeType to one of direct, fanout or topicconvenientChannel . exchangeDeclare( "my exchange A", direct, passivelyDeclare, durable, autoDelete );convenientChannel . exchangeDeclare( "my exchange B", fanout, passivelyDeclare, durable, autoDelete );convenientChannel . exchangeDeclare( "my exchange C", topic, passivelyDeclare, durable, autoDelete );
// A passive declaration asserts that an exchange exists, causing a channel exception if it does not convenientChannel . exchangeDeclare( "my exchange A", direct, true, durable, autoDelete );
}
/* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as exchanges should be long-lived * Declaring an exchange twice is not an error; if parameters differ, new values are used if possible * Passive declaration is interesting but typical deployment practices mitigate its usefulness */
// Declaring and Binding Message Queues
public void use(final @NotNull ConvenientChannel convenientChannel){
final boolean passivelyDeclare = false;final boolean durable = true;final boolean autoDelete = false;final boolean exclusive = false;
// A declared queue is bound to the default ("") exchange with the routing key the queue name ("my queue")convenientChannel . queueDeclare( "my queue", passivelyDeclare, durable, exclusive, autoDelete );
// A message queue can be bound to another exchange with whatever key you chooseconvenientChannel . queueBind( "my queue", "another exchange", "some routing key" );
// When binding to a fanout exchange the routing key is the empty string ("")convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.fanout", "" );
// When binding to a topic exchange, set the routing key as a wildcard// Publish messages with specific routing keys eg "BONDS.USD.FTSE", "BONDS.USD.NYSE", "BONDS.GBP.FTSE"convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.topic", "BONDS.USD.*" );
}
/* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as message queues should be long-lived * Declaring a message queue twice is not an error; if parameters differ, new values are used if possible * Exclusive message queues are for advanced usages and often aren’t needed */
AMQP: Introducing Users
but
StormMQ enforces that the sameUsers and Permissions are in all of a System’s Environments
Passwords are different
Passwords are securely generatedby StormMQ and are 512 bit keys
Configuration Managers can prevent revelation of passwords to different
system users, eg developers vs sysadmins
Aim: Separation of Environments
A Connection can use several authentication mechanisms
which are per Virtual Host
but
Commonest is User - Password
andand
Aim: Separation of Responsibility
Users are intended to be system accounts (robots), not sysadmins
Users have read, write and create permissions using a wildcard syntax: important to name queues, etc well
// Example of create-system.json used with API call stormmq-create-system{
"companyName" : "usuallyYourUserName","systemName" : "anythingAlphabetic","environments" :[{
"environmentName" : "development",// Different StormMQ clusters have different characteristics (and costs)"clusterName" : "free-1",
// These are the StormMQ user accounts to which passwords for AMQP users (below) can be revealed for this environment (development)"permittedStormMQUserNames" : [ "developerPeter", "developerJames", "developerJohn" ]
}],
// User permissions operate on Message Queues and Exchanges"amqpUserPermissions" :{
"queueReader" : { "create" : "^$", "read" : ".*", "write" : "^$" },"queueWriter" : { "create" : "^$", "read" : "^$", "write" : ".*" },"queueCreator" : { "create" : ".*", "read" : "^$", "write" : "^$" },"onlyProcessingObjects" : { "create" : "^processing.*", "read" : "^processing.*", "write" : "^processing.*" }
}}
Configuring AMQP
“*Our REST API has Java, Ruby, PHP and .NET bindings”
AMQP StormMQ
✓ ActiveMessage Queues
Exchanges
Bindings
Users
User Permissions
Entire Systems
! Extra
! Extra
! Extra
✓ Active
✓ Active
✓ REST
✓ REST
✓ REST*
Programmatic Config
✓ Same
✓ Same
✓ Same
AMQP: Getting Statistics
“*Password revelation is restricted to specific individuals”
AMQP StormMQ
Message Queues
Exchanges
Bindings
Generated Passwords
Usage Patterns
Entire Systems
! Extra
! Extra
! Extra
✓ REST
✓ REST
✓ REST
Programmatic Info
! Extra ✓ REST
! Extra ✓ REST*
! Extra ✓ REST
// Getting Statistics from StormMQ using the REST API in Java
// If you need to use proxies or experience the pain of Java SSL choose a different method in ApiConfigurationfinal ApiConfiguration apiConfiguration = directConnectionIfHavingProblemsWithSslVerification();
// If your Magic Secret Key is in an unusual place choose a different method in Apifinal Api api = apiByGuessingUsingOsConfigurationFiles( apiConfiguration, stormMqUserName );
final List<AmqpQueue> amqpQueues = api . listQueues( companyName, systemName, environmentName );
final List<AmqpExchange> amqpExchanges = api . listExchanges( companyName, systemName, environmentName );
final List<AmqpBinding> amqpBindings = api . listBindings( companyName, systemName, environmentName );
final Map<AmqpUserName, AmqpUserPassword> passwords =api . listAmqpUserNamesAndPasswords( companyName, systemName, environmentName );
final CompanyDetails companyDetails = api . describeCompany( companyName );
// Finding “what we’ve got”final List<CompanyName> companyNames = api . listCompanies();final List<ClusterName> clusterNames = api . listClusters( companyName );final List<SystemName> systemNames = api . listSystems( companyName );final SystemDetails systemDetails = api . describeSystem( companyName, systemName );final Set<Environment> environments = systemDetails . getEnvironments();
/* Notes * Also consider the command-line tools stormmq-list-queues, stormmq-list-exchanges and stormmq-list-bindings * Further examples are in com.stormmq.api.Example (part of the url signer jar) */
Virtual Hosts
Virtual Hosts: Think Apache!
Virtual Hosts isolate Message Queues
Virtual Hosts isolate User Groups*
Virtual Hosts isolate Exchanges*
Virtual Hosts isolate Bindings*
Virtual Hosts isolate
Virtual Hosts: StormMQ
Virtual Hosts isolate Your Company
Virtual Hosts isolate Your Systems
Virtual Hosts isolate Your Environments
Enable Configuration Management
Virtual Hosts stop Data Accidents
Widgets Ltd /
Invoicing System /
/
Testing
Development
Production
Message Properties
Server Understood Good Practice
“This is just the tip of messaging best practice with AMQP”
Persistent = 1Priority = 0Timestamp = 4568965345MessageId = 567A-GH-20100709
Priority Queues are per-Message
Control life of a Message
Persistence is per-Message
Timestamps expire Messages
Completely Optional
MIME Content-Type
Unique MessageId
Schema & Version
// Message Properties: Examples of Good Practice
// Being explicit is good practicefinal BasicProperties messageHeaders = new BasicProperties();{{
DeliveryMode . NonPersistent . setOn( this );Priority . Zero . setOn( this );setMessageId( "My unique ID, eg a GUID or ascending value" );
// Explicit MIME type including charset; use application/octect-stream for trully binary datasetContentType("application/json; charset=utf-8");
// Use compression for larger textual data; use the same values as HTTP headers; use "identity" for nonesetContentEncoding( "gzip" );
// Timestamp each messagesetTimestamp( new Date( System . currentTimeMillis() ) );
// Schema and version, perhaps using a XML namespace definitionsetType( "syslog-message_1.0.4" );
// Optional: Use an expiration to discard old messages that are stale (milliseconds, example is 5 minutes)setExpiration( Integer . toString( 5 * 60 * 1000) );
// Optional: Identify the sender for later debuggingsetAppId( "sending application name" );setUserId( "role of sender, or, user of machine" );
// Optional: Identify domain or environment (consumer could error if working in different one)setClusterId( "production" );
}};
And there is more!
Transactions
More Message Properties
Custom Message Properties
Immediate Delivery
Additional Exchange Types
Auto Deletion
Fine-Grained User Permissions
QueuePurging
QoS
Useful Book:Enterprise
Integration PatternsHohpe & Woolf