EnterpriseMessagingFoundations
JeremyDeanehttp://jeremydeane.net
Agenda
v EnvironmentSetup
v AsynchronousHelloWorld!
vMessagingFoundations
v Exercises
vMessagingNetworks
v AdvancedMessaging
v ExtensibleMessaging
v Exercises
v AdditionalEnvironmentSetup
v HelloBunny!
v AMQPFundamentals
v Exercises
v RabbitMQ Architecture
v RabbitMQ Security&Operations
v AMQPIntegrations
v Exercises
EnvironmentSetup§ JDK1.8+Pre-Requisite($JAVA_HOMEinpath)§Maven3.3.3+Pre-Requisite($MAVEN_HOMEinpath)
§ DownloadandunzipActiveMQ – Version5.14.0+
§ Downloadhawtio – Version:1.4.56+§ renamehawtio-app.jar
§ Download(Clone)CodeFromGithub§ https://github.com/jtdeane/magic-supplies§ https://github.com/jtdeane/message-client§https://github.com/jtdeane/camel-standalone-router
§WindowsInstallActiveMQ asService:./$ACTIVEMQ_HOME/win65/InstallService.bat
AsynchronousHelloWorld!1. StartActiveMQ
2. BuildMagicSuppliesProject
3. Optional importintoIDE
4. ExecuteMessageProducer– RunasJUnit Test
5. ViewActiveMQ Console
6. ExecuteMessageConsumer– RunasJUnit Test
cd $ACTIVEMQ_HOME/bin./activemq start OR ./activemq.bat
cd magic-suppliesmvn clean install
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
mvn test -Dtest=JMSProducerFunctionalTest
mvn test -Dtest=JMSConsumerFunctionalTest
ActiveMQWebConsoleAlternative: hawtio1. Starthawtio
2. OpeninWebBrowser
3. SelectConnectàLocal
4. ConnecttoActiveMQ
5. ClicknewAgentURLtolaunchbrowsing
6. SelectActiveMQ
java -jar hawtio-app.jar --port 8090
http://localhost:8090/hawtio
ActiveMQ Message Broker<execution environment>
Spring-Boot Application<execution environment>
REST Controller
Message Listener
TopicQueue
MagicSuppliesApplication
§ SpringBootWebApplication§ SpringBoot1.4.0§ JMSListeners- ActiveMQ
MagicSuppliesWeb Services1. StartActiveMQ *
2. StartWebApplication
3. OpenBrowser
4. ViewHealthCheckResponse
*WebApplicationrequiresActiveMQ started
cd $ACTIVEMQ_HOME/bin./activemq start OR ./activemq.bat
mvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m”ORcd magic-supplies/targetjava -jar magic-supplies-1.0.0.jar
http://localhost:8080/info
{"build":{"description":"Magic SuppliesWebApplication","name":"Magic SuppliesWebAppication","version":"1.0.0","artifact":"magic-supplies"}}
P2PHidden Costs
BottomLineSOAbyMarcRix
WebservicesAwebservicedoesNOT trulydecoupletheconsumerandprovider
P2PIntegrationsThecostofmaintainingP2Pintegrationsincreasesexponentiallyasthenumberoftheconnectionsincreases
#ofconnections
$
gain
value
cost
loss
MessagingFoundations
Enterprise IntegrationPatterns*
*A.K.A. MessageExchangePatterns(MEP)
Publish Subscribe Domain (Topics)
Point-to-Point Domain (Queues)
Enterprise IntegrationPatterns
Channel Message FIlterSquareTriangle ChannelSquare SquareSquare
GuaranteedMessageDelivery
FilterMessages
AsynchronousRequestReply
Message OrientedArchitecture(MOA)
MessageSender Receiver
Broker
TopicQueue
createsprocesse
s
Delivers
Connects/ to Connects/ to
Hosted//by
JavaMessageService(JMS)*
*Alternative– AdvancedMessageQueuingProtocol(AMQP)
Message
JMS & Custom Properties
Payload
Properties(a.k.a.Headers)
§ JMS*(e.g.JMSType,JMSCorrelationID,JMSDeliveryMode,JMSExpiration)
§ Custom(e.g.MimeType,Token)
Payload(a.k.a.Body)
§ Text§ Object§Map§ Bytes§ Stream
JavaMessageService(JMS)
Producer
Publisher
Message Broker<JMS Providert>
Topic
Queue
Subscriber
Consumer<Listener>
Subscriber
Message Store
JMS Selector
Channel
Channel
ApacheActiveMQ
IntegrationOptions
JavaMessageService(JMS)AdvancedMessageQueuingProtocol(AMQP)
DeploymentFlexibility
Stand-aloneEmbedded
AdvancedTopologies
Master-SlaveHighAvailability(HA)FederatedNetwork
Support
ActiveOpenSourceCommunityCommercial24X7Options
Languages- Transport• Java-Scala– TCP/NIO• Ruby,Perl,Python– Stomp• C#(NMS)–TCP/NIO
MessagingExercises
Exercise:JMSSender1. StartActiveMQ
2. BuildClient
3. ExecuteJAR
4. ViewMessagefromActiveMQ orhawtio (test.alchemy)
java -jar message-client-1.0.0.jar
mvn clean installcd target
cd $ACTIVEMQ_HOME/bin./activemq start OR ./activemq.bat
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
Spring Application<execution environment>
ActiveMQ Standalone Broker<execution environment>
Alchemy Queue
MessageSender
Channel
message-client/src/main/java/cogito/online/application/MessageClient:sendAlchemyAlert(..)
Demo:JMSListeners
magic-supplies/src/main/java/cogito/online/application/MagicSuppliesConfigurationmagic-supplies/src/main/java/cogito/online/messaging/SingleOrderProcessingMessageListenermagic-supplies/src/main/java/cogito/online/messaging/OrderProcessingMessageListener
Srping-Boot Application<execution environment>
MessageListener
Configuration
Producer <Message Client>
Message Broker<JMS Providert>
Queue
Message Store
Channel
Exercise:JMSListener1. StartActiveMQ
2. StartWebApplication
3. ExecuteJAR
4. ViewActivityviaActiveMQ orhawtio
5. ViewMagic-SuppliesConsoleLog
java -jar message-client-1.0.0.jar magic.orderjava -jar message-client-1.0.0.jar magic.orders
cd $ACTIVEMQ_HOME/bin./activemq start OR ./activemq.bat
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
cd magic-supplies/targetjava -jar magic-supplies-1.0.0.jar
Demo:JMSSubscribers
magic-supplies/src/main/java/cogito/online/application/MagicSuppliesConfigurationmagic-supplies/src/main/java/cogito/online/messaging/AlertProcessingTopicListener.java
Srping-Boot Application<execution environment>
MessageSubscriber
Configuration
Producer <Message Client>
Message Broker<JMS Providert>
Topic
Channel
Exercise:JMSSubscribers1. StartActiveMQ
2. StartMagicSupplies
3. ExecuteJAR
4. ViewActivityviaActiveMQ (orhawtio)
5. ViewMagic-SuppliesConsoleLog
java -jar message-client-1.0.0.jar magic.alerts
cd $ACTIVEMQ_HOME/bin./activemq start OR ./activemq.bat
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/topics.jsp
cd magic-supplies/targetjava -jar magic-supplies-1.0.0.jar
MessagingNetworks
ActiveMQ Broker Topologies
ActiveMQinAction byBruceSnyder,Dejan Bosanac,andRobDavies
§ EmbeddedBroker(VMTransport)
§ StandaloneBroker(a.k.a.Client-Server)
§ LoadBalancedBrokers
§ NetworkedBrokers(a.k.a.StoreandForward)
failover:(tcp://BrokerA:61616,tcp://BrokerC:61616)?randomize=true
ActiveMQ HighAvailabilityandDisasterRecovery
ActiveMQinAction byBruceSnyder,Dejan Bosanac,andRobDavies
§MasterSlave– FileSystem§ KahaDB§ SAN-NFS4
§MasterSlave– JDBC§ RDBMS(e.g.PostgreSQL)
§MasterSlave– Replicated§LevelDB &Zookeeper
§ FailoverProtocol(usedomainaliases)
failover:(tcp://PrimaryBroker:61616,tcp://SecondaryBroker:61616)?randomize=false
ActiveMQ HighAvailabilityExample
failover:(tcp://PrimaryBroker:61616,tcp://SecondaryBroker:61616)?randomize=false
Linux VM
Queue
<Execution Environment>ActiveMQ (Master)
Topic
Linux VM
Queue
<Execution Environment>ActiveMQ (Slave)
Topic
<NFS Mount>
Kaha DB
ActiveMQ SimpleHighAvailabilityExercise1. Createthefollowingdirectorystructure
2. UnpackActiveMQ twicerenamingtomasterandslave
3. UpdateMaster../conf/activemq.xml:Name,Management,Persistence,andNetworkConfigurations
../cluster
../cluster/data
../cluster/master
../cluster/slave
<!-- update --><broker xmlns="http://activemq.apache.org/schema/core" brokerName="master" dataDirectory="${activemq.data}”><!-- update --><managementContext>
<managementContext createConnector="true" connectorPort="1099"/> </managementContext><!-- update --><persistenceAdapter>
<kahaDB directory="<<your location>>"/cluster/data"/> </persistenceAdapter><!-- add--><networkConnectors>
<networkConnector uri="static:(tcp://localhost:61617)"/> </networkConnectors>
ActiveMQ HighAvailabilityExercise4. UpdateSlave../conf/activemq.xml:Name,Management,
Persistence,Network,TransportConfigurations
5. UpdateSlave../conf/jetty.xml:jettyPort
<!-- update --><broker xmlns="http://activemq.apache.org/schema/core" brokerName="slave" dataDirectory="${activemq.data}">
<!-- update à<managementContext>
<managementContext createConnector="true" connectorPort="2099"/> </managementContext><!-- update à<persistenceAdapter>
<kahaDB directory="<<your location>>"/cluster/data"/> </persistenceAdapter><!-- add--><networkConnectors>
<networkConnector uri="static:(tcp://localhost:61616)"/> </networkConnectors><!– update – also remove other transports à<transportConnector name="openwire" uri="tcp://0.0.0.0:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<property name="port" value="8162"/>
ActiveMQ HighAvailabilityExercise6. StarttheMasterandthentheSlaveBroker
7. OpenMasterWebConsole– QueuesPage
8. CreateaQueueandSendaMessage
9. StoptheMasterBroker
10. OpenSlaveWebConsole– QueuesPage
11. ViewtheQueueandpersistedMessage
cd ../cluster/<<master || slave>>/bin./activemq start OR ./activemq.bat
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
Open http://localhost:8162/admin/ {admin/admin}Open http://localhost:8162/admin/queues.jsp
cd ../cluster/<<master || slave>>/bin./activemq stop OR Ctrl+C
ActiveMQ FederatedTopologyExampleLinux VM
Queue
<Execution Environment>ActiveMQ (Master)
Topic
Linux VM
Queue
<Execution Environment>ActiveMQ (Slave)
Topic
<NFS Mount>
Kaha DB
Linux VM
Queue
<Execution Environment>ActiveMQ (Master)
Topic
Linux VM
Queue
<Execution Environment>ActiveMQ (Slave)
Topic
<NFS Mount>
Kaha DB
Linux VM
Queue
<Execution Environment>ActiveMQ (Master)
Topic
Linux VM
Queue
<Execution Environment>ActiveMQ (Slave)
Topic
<NFS Mount>
Kaha DB
WebNetworkSegment
ServiceNetworkSegment
DataNetworkSegment
ActiveMQ ErrorHandling
§ DeadLetterQueue(DLQ)§ Review(WebConsole)§ Redeliver(Plug-in)§ Remediate(Policy)
§MonitorDLQDepth§ Alerts(Splunk Example)
§ FailedMessageTransactions§ Rollback§ DLQ
§ ThroworLogException
ActiveMQ Security
§ AuthenticationandAuthorization§WebConsole§ JMXConnector(BindtospecificPort#)§ QueuesandTopics
§ Auditing§MonitorLogs§ PeriodicallyAuditAccess
§ Confidentiality§ Transport(SSL)§Messages(Encryption)§ KahaDB Files(Encryption)§ LogFiles(Encryption)
AdvancedMessaging
ApacheCamel– EnterpriseIntegrationLibrary
CamelInAction byClausIbsenandJonathanAnstey
§ DomainSpecificLanguages(DSL)– Java,Scala,SpringDSL
§ RouteBuilders– createEndpoints(i.e.from&to)usingComponents(e.g.protocol)andProcessors(e.g.Mediation,Enrichment
§ RouteEngine– LoadsandexecutesRoutes
ApacheCamelMessage
JMS
JMS & Custom Properties
Payload
Camel (IN)
Camel Headers
Body
Camel (OUT)
Camel Headers
Body
HTTP Request
HTTP Headers
Body
ApacheCamelRouteProcessing/** Splitter - xpath expression*/
from("activemq:emagic.orders").split(splitXPath).parallelProcessing().
to("activemq:emagic.order");
Camel Route Engine
Camel Route
<JMS Component>Endpoint (From)
<EIP Processor>Some Processing
<DB Component>Endpoint (To)
CamelMessage
Configurations
/** Content Based Routing - Inpatient-Outpatient Events* http://camel.apache.org/jsonpath.html*/
from("activemq:event.ingestion").process(new TrackingIdProcessor()).choice().
when().jsonpath("$[?(@.class==inpatient)]").to("activemq:event.inpatient").
otherwise().to("activemq:event.outpatient").
end().to("activemq:event.audit");
Spring-Boot&ApacheCamelJava Runtime Environment (JRE)
Spring IOC Container
Camel Route Engine
Camel Route
<JMS Component>Endpoint (From)
<EIP Processor>Some Processing
<DB Component>Endpoint (To)
CamelMessage
Configurations
/** Spring Boot – Camel Route Application */@Configuration@ImportResource("classpath:camel-route-spring.xml")public class MagicRouterApplication {
public static void main(String[] args) {SpringApplication.run(MagicRouterApplication.class, args);
}}
Spring Boot Application<Executable JAR>
Message Broker<JMS Provider>
Queue<magic.order>
Content Router
Queue<priority.orders>
Queue<emagic.orders>
Camel Routes
Queue<emagic.order>
Splitter
Producer <Message Client>
Srping-Boot Application<execution environment>
MessageSubscriber
Configuration
MessageListener
Producer <Console>
Exercise:ContentBasedRouting
Exercise:ContentBasedRouting1. BuildprojectandimportintoIDE(e.g.Eclipse,IntelliJ)
2. ViewMagicRouteBuilder.java &MagicRouterApplication.java
3. Viewcamel-route-spring.xml
4. ExecuteSpringBootApplication
5. ExecuteMessageClient
cd camel-standalone-routermvn clean installmvn eclipse:eclipse
mvn spring-boot:run ORjava –jar ./target/camel-standalone-router-1.0.0.jar
java -jar message-client-1.0.0.jar emagic.orders
Exercise:ContentBasedRouting6. OpenActiveMQWebConsoleandviewpriority.order queue
7. ViewMagicSupplieslogforordersprocessed(magic.order)
8. ViewCamelStandalone– JConsole
9. ConnecttoCamel– hawtio
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
jconsole {pid}
Camel Unit Testing@OverrideprotectedCamelContext createCamelContext()throwsException{
CamelContext context=super.createCamelContext();
//setuproutestoMockEndpointscontext.addRoutes(new RouteBuilder(){
publicvoidconfigure(){
//testxpath splitfrom("direct:split").
split(new XPathBuilder (MagicRouteBuilder.splitXpath)).to("mock:order");
});
returncontext;}
@TestpublicvoidtestSplitLogic()throwsException{
//SetexpectationsMockEndpoint order=getMockEndpoint("mock:order");order.expectedMessageCount(3);
//readinasmallbatchStringxml=IOUtils.toString(this.getClass().
getResourceAsStream("batch.xml"),"UTF-8");
//executethetesttemplate.sendBody("direct:split",xml);
//shouldbethreeordersinthebatchorder.assertIsSatisfied();
}
MagicRouteBuilderTest.java
Camel FunctionalTesting1. DownloadandinstallApacheJMeter
2. Copy$ACTIVEMQ_HOME/activemq-all-5.13.1.jarto$JMETER_HOME/lib
3. StartJMeter - ./jmeter.sh orjmeter.bat
4. Opencamel-magic-router/src/test/jmeter/Mock CamelPublisher.jmx
5. Runtest
Otherpossibleframeworks
• RobotFramework
• SOAPUI
Message Broker<JMS Provider>
Producer <JMeter>
Queue<emagic.dead>
Queue<emagic.orders>
Camel Routes
Splitter Dead Letter Channel
Demo:DeadLetterChannel
errorHandler(deadLetterChannel("activemq:emagic.dead").maximumRedeliveries(1).redeliveryDelay(1000));
java -jar message-client-1.0.0.jar emagic.bad
ExtensibleMessaging
ActiveMQ Master Broker<execution environment>
ActiveMQ Slave A Broker<execution environment>
ActiveMQ Slave B Broker<execution environment>
Spring Boot Application 1<Executable JAR>
Camel Routes
Spring Boot Application 2<Executable JAR>
Spring Boot Application 3<Executable JAR>
Camel Routes Camel Routes
Spring-Boot&Camel– ActiveMQ Integration
PrimaryBenefits:§ Resiliency– CamelRoutefailuredoesnottakedownBroker§ Extensibility– SwapoutMessageBrokerorEIPFramework§ ContinuousDevelopment– NoBrokerorWebServerdowntime
CompetingConsumers
ActiveMQ Broker<execution environment>
Spring Boot App 1<Executable JAR>
Camel Routes
Spring Boot App 2<Executable JAR>
Camel Routes
Spring Boot App 3<Executable JAR>
Camel Routes
Queue<emagic.order>
GoldenPath – Processedonceinorder
Bad Producer
Redis<execution environment>
Key-Value Store
Idempotent CompetingConsumers
Spring-Boot&Camel– DeploymentOptions
Linux VM
Camel Route
<Execution Environment>Spring-Boot Camel
Camel Route
Properties Log
Linux VM
Camel Route
<Execution Environment>Spring-Boot Camel
Camel Route
Properties Log
Port: 2100
Camel Route
<Execution Environment>Spring-Boot Camel
Camel Route
Properties Log
Port: 2101
§ SingleApplicationperVM
§MultipleApplicationsperVM
§ IaaS Container(e.g.Docker)
§ PaaS Container(e.g.CloudFoundry)
Exercise:EnrichmentandTransformation1. Update:ReplaceSimpleChoiceRoutewithMediationRoute
2. StopSpringBoot,BuildProject,andStartSpringBoot
XmlJsonDataFormat xmlJsonFormat = new XmlJsonDataFormat();xmlJsonFormat.setForceTopLevelObject(true);
from("activemq:emagic.order").choice().
when().simple("${in.body} contains 'Houdini'").process(new Processor() {public void process(Exchange exchange) {
exchange.getIn().setHeader("VIP", "true");}
}).to("activemq:priority.order").
otherwise().marshal(xmlJsonFormat).transform(body().regexReplaceAll("@", "")).to("activemq:magic.order");
mvn clean installmvn spring-boot:run ORjava –jar ./target/camel-standalone-router-1.0.0.jar
Exercise:EnrichmentandTransformation3. ExecuteMessageClient
4. OpenActiveMQWebConsoleandviewpriority.order queue
6. OpenActiveMQWebConsoleandviewpriority.order queue
7. ViewMagicSupplieslogforordersprocessed(magic.order)
8. ConnecttoCamel&ActiveMQ – hawtio
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jspOpen http://localhost:8161/admin/topics.jsp
java -jar message-client-1.0.0.jar emagic.orders
Open http://localhost:8161/admin/ {admin/admin}Open http://localhost:8161/admin/queues.jsp
Exercise:Wire-Tap1. ReplaceRoutingw/:Wire-TaptoSplitterandEvaluationRoute
2. StopSpringBoot,BuildProject,andStartSpringBoot
3. ExecuteMessageClient&ViewResults
from("activemq:emagic.order").wireTap("direct:ministry").
to("activemq:magic.order");
from("direct:ministry").choice().
when().simple("${in.body}contains'Elder'").log("ILLEGALMAGICALERT").to("activemq:topic:magic.alerts").
otherwise().log("...offintotheether");
mvn clean installmvn spring-boot:run ORjava –jar ./target/camel-standalone-router-1.0.0.jar
java -jar message-client-1.0.0.jar emagic.orders
Open http://localhost:8161/admin/queues.jspOpen http://localhost:8161/admin/topics.jsp Wire Ttap
BonusExercise:IdempotentConsumer(In-Memory)
?
Idempotent Consumer
1. ViewIdempotentConsumerRouteBuilder inCamel-Standalone-Router
2. StartCamel-StandaloneRouter
3. ExecuteMessageClient
4. OpenActiveMQWebConsole(orviewviahawtio)
5. ViewMagicSupplieslogforordersprocessed(magic.order)• Only two of three sent messages should be present.
mvn spring-boot:run ORjava –jar ./target/camel-standalone-router-1.0.0.jar
java -jar message-client-1.0.0.jar unique.orders
BonusExercise:IdempotentConsumer(Redis)
?
Idempotent Consumer
1. DownloadInstallRedis &Start• Mac:brew install redis• Linux:yum install redis
• Windows:https://github.com/rgl/redis/downloads
2. Updatecamel-route-spring.xml – Uncomment
3. ExecuteMessageClient
<!-- Redis Configuration--> <bean id="redisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="hostName" value="localhost"/> <property name="port" value="6379"/>
</bean>
<bean id="redisTemplate" class="org.springframework.data.redis.core.RedisTemplate"> <property name="connectionFactory" ref="redisConnectionFactory"/>
</bean>
BonusExercise:IdempotentConsumer(Redis)1. UpdateIdempotentConsumerRouteBuilder – Comment:
2. UpdateIdempotentConsumerRouteBuilder - Uncomment
3. StopSpringBoot,BuildProject,andStartSpringBoot
4. RepeatIn-MemorySteps3-5
from("activemq:unique.order").idempotentConsumer(header("uniqueId"),MemoryIdempotentRepository.memoryIdempotentRepository(200)).
to("activemq:magic.order");
..@AutowiredRedisTemplate<String, String> redisTemplate;..from("activemq:unique.order").
idempotentConsumer(header("uniqueId"), RedisIdempotentRepository.redisIdempotentRepository(redisTemplate, "camel-repo")).
to("activemq:magic.order");
mvn clean installmvn spring-boot:run ORjava –jar ./target/camel-standalone-router-1.0.0.jar
Messaging2.0
AdditionalEnvironmentSetup§ DownloadandinstallRabbitMQ (3.6.2+)
§MacStandalone orHomebrew§WindowsStandalone
§ Add$RABBITMQ_HOME/sbin toPATH
§ Commands(WindowsasaService)§ Start:rabbitmq-server -detached§ Status:rabbitmqctl status
§ Stop:rabbitmqctl stop
§ InstallManagementPlugin
§ VerifyManagementConsole(guest/guest):
rabbitmq-plugins enable rabbitmq_management
http://localhost:15672/
Mac&LinuxO/S– Update/etc/hosts127.0.0.1 <<name of your laptop>>
AdditionalEnvironmentSetup§ Download(Clone)CodeFromGithub
§ TestHarness-https://github.com/jtdeane/amqp-client§ AMQPConsumer- https://github.com/jtdeane/amqp-consumer§ AMQPProducer- https://github.com/jtdeane/amqp-producer§ AMQPComplexEventProcessor(CEP)- https://github.com/jtdeane/amqp-cep
Hello Bunny!
1. StartRabbitMQ
2. Buildandimportamqp-client intoIDE
3. SendAMQPMessage– RunasJUnit Test
4. ViewExchange,Queue,andMessageinManagementConsole
5. ReceiveAMQPMessage– RunasJUnit Test
mvn clean install
mvn test -Dtest=ProducerFunctionalTest
mvn test -Dtest=ConsumerFunctionalTest
http://localhost:15672/#/exchanges/%2F/test.direct
http://localhost:15672/#/queues/%2F/magic.greetings
amqp-client/src/test/java/ws/cogito/magic/messaging/ProducerFunctionalTest.javaamqp-client/src/test/java/ws/cogito/magic/messaging/ConsumerFunctionalTest.java
AdvancedMessagingQueueing Protocol(AMQP)
RabbitMQ implement0.9.1ofAMQPStandard
AMQPMessageMessage
Properties{Standard or Broker Defined}
Headers{Custom}
Payload{Stream of bytes}
Property Purpose
ContentType ThecontenttypeofthemessageasaMediaType.
ContentEncoding Theencodingto andfrombytes(UTF-8)
RoutingKey UsedbyanExchangeformessagerouting
DeliveryMode Persistent(2)ornon-persistent(1)
Reply To FormessagingforwardingorRequest-ReplyPattern
Expiration Time-to-Live (TTL)inMilliseconds
Priority MovemessageupQueue(defaultis0)
CustomHeaderExample:TrackingID
AMQPVirtualHosts, Exchanges&RoutingKeys§ VirtualHostsprovidelogicalcontainersforExchanges&Queues
§ ReferredtoasVHost§ DefaultVirtualHost(/)– Donotdelete!§ SuggestedNamingScheme:<owner>.<domain || purpose>.vhost
§ ExchangesroutemessagesbasedonaRoutingKey§ DirectExchange:Point-to-Point(P2P)§ TopicExchange: Publish-Subscribe§ Fanout Exchange: Broadcast
§DefaultExchanges(AMQP)– Donotdelete!§ SuggestedNamingScheme:<owner>.<domain || purpose>.<Exchange Type>
§ RoutingKeysrepresentahierarchicalnamespace§ ws.cogito.magic.sales.orders§ ws.cogito.magic.finance.bills
Message Broker
Virtual Host
Publisher ExchangeMessage
AMQPQueues andBindingKeys§ QueuesareassociatedwithExchangesviaBindingKeys
§ Durable orTransient§ SuggestedNamingScheme:<owner>.<hierarchy>.<entities>
§ acme.magic.finance.invoices§ acme.magic.sales.orders
§ BindingKeysoftenmatchQueueNamesexactlyforDirectExchanges§ acme.magic.finance.invoices
§ BindingKeysoftenusespecialcharactersforTopicExchanges§ * (star)cansubstituteforexactlyoneword§ # (hash)cansubstituteforzeroormorewords
§ acme.magic.finance.*§ acme.magic.#
RabbitMQ SimulatorDemo
http://tryrabbitmq.com/
AMQP Connections,ChannelsandAcknowledgements§ Connections arelonglivedTCPconnectionswhileChannelsmultiplexedinteractionswithinaConnection.
§MessageAcknowledgementsareautomaticbydefaultbutcanbemanual
§Messagescanalsoberejected orre-queued orsenttoDeadLetterQueue(DLQ)
AMQPDeclarations
ConflictingDeclarationswillfail!
AMQPExercises
AMQPExercisesSetup1. StartRabbitMQ
2. CreateUser(client/client)– AdminTag
3. CreateVirtualHost(VHost)– cogito
4. AssociateUserclient toVHost cogito&/
5. Buildandbootamqp-client
6. ViewdeclaredExchanges
http://localhost:15672/#/users
http://localhost:15672/#/vhosts
http://localhost:15672/#/vhosts/cogito
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
http://localhost:15672/#/exchanges/cogito/magic.directhttp://localhost:15672/#/exchanges/cogito/magic.topic
Code:amqp-client/src/main/java/ws/cogito/magic/utilities/AmqpDeclarationsConfiguration.java
AMQPExercisesSetup1. CreateQueuemagic.tests *
2. CreateQueueDirect&TopicBindings**
3. CreateUsers)– AdminTag• consumer/consumer• producer/producer• evaluator/evaluator
4. AssociateUserstoVHost cogito&/
http://localhost:15672/#/queues
Exchange: magic.direct, Queue & Routing Key: magic.tests
Exchange: magic.topicTopic: magic.testing.eventsRouting Key: magic.testing.#
Knuffle-Bunny-Cautionary*Shortenedfortesting,shouldbews.cogito.magic….etc**Normallythereshouldbeonequeueperconsumer
AMQPSender
Note:normallythereshouldbeonequeueperconsumer;thisisjustfortesting
http://tryrabbitmq.com/
AMQPSender1. IssueHTTPRequestviacurlorRESTclient(e.g.Postman)- Direct
2. ViewmessageinQueueandthenpurgeQueue
3. IssueHTTPRequestviacurlorRESTclient(e.g.Postman)- Topic
4. ViewmessageinQueueandthenpurgeQueue
5. ViewCode
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1125, "customer":"Houdini", "item":"Cards", "amount":6}' http://localhost:9999/amqp/magic.direct/magic.tests
http://localhost:15672/#/queues/cogito/magic.tests
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1126, "customer":"Teller", "item":"Dice", "amount":4}' http://localhost:9999/amqp/magic.topic/magic.testing.foo
http://localhost:15672/#/queues/cogito/magic.tests
/amqp-client/src/main/java/ws/cogito/magic/utilities/web/AmqpController.java
AMQPListener&Subscriber
http://tryrabbitmq.com/
1. StartRabbitMQ
2. Buildandbootamqp-consumer
3. ViewQueuesandBindings
4. ViewDeclarationCode
5. Bootamqp-client
AMQP Listener
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
http://localhost:15672/#/queues/cogito/magic.eventshttp://localhost:15672/#/queues/cogito/magic.sales.orders
/amqp-consumer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
6. SendOrderMessageviaamqp-client
7. Viewamqp-client andamqp-consumer consolelogs
6. ViewOrderListenerCode
7. Stopamqp-consumerandsendanotherOrderMessage
8. ViewmessageviaManagementConsole
9. Bootamqp-consumerandviewconsolelogs
AMQP Listener
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1127, "customer":"Blain", "item":"Marbles", "amount":2}' http://localhost:9999/amqp/magic.direct/magic.sales.orders
/amqp-consumer/src/main/java/ws/cogito/magic/messaging/OrdersMessageListener.java/amqp-consumer/src/main/java/ws/cogito/magic/MessageLogging.java
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1128, "customer":"Blackstone", "item":"Scarf", "amount":4}' http://localhost:9999/amqp/magic.direct/magic.sales.orders
1. SendEventMessageviaamqp-client
2. Viewamqp-client andamqp-consumer consolelogs
6. ViewEventListenerCode
7. SendanotherEventMessageviaamqp-client
8. Viewamqp-client andamqp-consumer consolelogs
AMQP Subscriber
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":21, "location":"Las Vegas", "title":"Penn and Teller", "date":"2016-12-31"}' http://localhost:9999/amqp/magic.topic/magic.events.shows
/amqp-consumer/src/main/java/ws/cogito/magic/messaging/EventsMessageListener.java
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":22, "location":"Clearwater", "title":"ArchConf", "date":"2016-12-12"}' http://localhost:9999/amqp/magic.topic/magic.events.conferences
RabbitMQ Architecture
RabbitMQ HighAvailabilityTopologyRabbitMQ Server
Message BrokerVirtual Host
RabbitMQ Server
Message BrokerVirtual Host
Mirrored Queues
Exchange Exchange
Queue A
Queue B
Queue A
Queue B
§ RabbitMQ ServersareClustered – SharedSecret– Erlang Cookie
§ RabbitMQ HAPolicy:rabbitmqctl set_policy ha-all "" '{"ha-mode":"all"}'
§ ClientsconnecttosetofaddressesorLoadBalancercanproxyClients(e.g.HAProxy)
RabbitMQ Federated Topologies
RabbitMQ Server RabbitMQ Server
Finance Cluster
RabbitMQ Server
RabbitMQ Server RabbitMQ Server
Sales Cluster
RabbitMQ Server
Federation Plugin
Shovel Plugin
§ FederationPlugin– Both“upstream”and“downstream”Consumersreceivemessage
§ ShovelPlugin– ForwardsMessagefrom“upstream”Queuetoremote“downstream”Exchange
RabbitMQ Security§ UsershaveVHost Read,Write,andConfigurepermissions
§ LDAPPlugin – Authentication-Authorization
§ TransportLevelSecurity(TLS)– Port5671§ TLSConnectionFactoryGist
§EncryptionatRest(Mnesia DB&Logs)– e.g.VormetricRabbitMQ Server
Message BrokerVirtual Host
DB
User Security
Logs
Transport Level Security
Encryption at Rest
RabbitMQ Operations§WebConsole
§ HTTPRESTAPI (guest/guest)– ListoutVirtualHosts
§ NewRelic &Zabbix Plugins
§ CommandLineInterface(CLI)&Tool
1. DownloadCommandLineToolin$RABBITMQ_HOME/sbin
2. Mayneedtoupdatepermissionsonfile
3. ListoutVirtualHosts
http://localhost:15672/cli/
chmod 777 rabbitmqadmin
rabbitmqadmin list vhosts
http://localhost:15672/api/vhosts
AdvancedAMQPPatterns
AMQPComplexEventProcessing
http://tryrabbitmq.com/
1
3
2
AMQP– ComplexEventProcessing1. StartRabbitMQ
2. Buildandbootamqp-cep
3. ViewQueuesandBindings
4. ViewDeclarationCode
5. Bootamqp-consumer &amqp-client
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
http://localhost:15672/#/queues/cogito/magic.evaluationshttp://localhost:15672/#/queues/cogito/magic.alerts
/amqp-cep/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
AMQP– ComplexEventProcessing6. PublishEventviacurlorRESTclient(e.g.Postman)
7. Viewamqp-client, amqp-consumer, amqp-cep consolelogs
8. ViewtheEvaluationMessageListener
9. PublishEventviacurlorRESTclient(e.g.Postman)
10. Viewamqp-client, amqp-consumer, amqp-cep consolelogs• Notehowanalertisnotraisedbecauseofthelocation
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":21, "location":"Las Vegas", "title":"Penn and Teller", "date":"2016-12-31"}' http://localhost:9999/amqp/magic.topic/magic.events.shows
/amqp-cep/src/main/java/ws/cogito/magic/messaging/EventsMessageListener.java
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":22, "location":"Clearwater", "title":"ArchConf", "date":"2016-12-12"}' http://localhost:9999/amqp/magic.topic/magic.events.conferences
AMQP– WhynotSpringIntegrationsorApacheCamel?@Beanpublic IntegrationFlow amqpFlow() {
return IntegrationFlows.from(Amqp.inboundGateway(this.rabbitConnectionFactory, queue())).transform("hello "::concat).transform(String.class, String::toUpperCase).get();
}@Beanpublic IntegrationFlow amqpOutboundFlow() {
return IntegrationFlows.from("amqpOutboundInput").handle(Amqp.outboundAdapter(this.amqpTemplate)
.routingKeyExpression("headers.routingKey")).get();
context.addRoutes(new RouteBuilder() {public void configure() {
from("rabbitmq://localhost?exchange=direct_logs&exchangeType=direct&bindingKey=ERROR").setBody().simple("ERROR: ${in.body}").to("stream:out");
from("rabbitmq://localhost?exchange=direct_logs&exchangeType=direct&bindingKey=WARN").setBody().simple("WARN: ${in.body}").to("stream:out");
}});
SpringIntegrations– Immature- poorlydocumentedDSL
ApacheCamel– ImmatureAMQP&RabbitMQ Components
AMQPAsynchronousRequest-Reply– TempQueue
http://tryrabbitmq.com/
1
2
3
StartswithPOSThttp://localhost:9000/priority/orders
4
AMQP– Asynchronous Request-Reply – TempQueue1. StartRabbitMQ
2. Buildandbootamqp-producer
3. ViewQueueandBinding
4. ViewDeclarationCode
5. Bootamqp-consumer
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
http://localhost:15672/#/queues/cogito/magic.priority.orders
/amqp-producer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
AMQP– Asynchronous Request-Reply – TempQueue6. IssuePriorityOrderRequestviacurlorRESTclient(e.g.Postman)
7. ViewthePriorityOrderController:processOrder
8. Viewamqp-consumer andamqp-producer consolelogs• NotetheTrackingIDandReplyTo Queue
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1129, "customer":"Penn", "item":"Rabbit", "amount":6}' http://localhost:9000/priority/orders
/amqp-consumer/src/main/java/ws/cogito/magic/web/PriorityController.java
“Beware of time-outs & downtime”
AMQPAsynchronousRequest-Reply– CorrelationID
http://tryrabbitmq.com/StartswithPOSThttp://localhost:9000/priority/vip/orders
12
3
4
AMQP– Asynchronous Request-Reply – CorrelationID1. StartRabbitMQ
2. Buildandbootamqp-producer
3. ViewQueueandBinding
4. ViewDeclarationCode
5. Bootamqp-consumer
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
http://localhost:15672/#/queues/cogito/magic.priority.orders
/amqp-producer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java
mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"
AMQP– Asynchronous Request-Reply– CorrelationID1. StartRabbitMQ
2. Buildandbootamqp-producer and amqp-consumer
3. IssueVIPOrderRequestviacurlorRESTclient(e.g.Postman)
4. ViewthePriorityOrderController:processOrderVip
5. View amqp-consumer andconsolelogsamqp-producer• NotetheTrackingID,CorrelationID,andReplyTo Queue
6. ViewthePriorityMessageListener
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1130, "customer":"Angel", "item":"quarters", "amount":6}' http://localhost:9000/priority/vip/orders
/amqp-consumer/src/main/java/ws/cogito/magic/web/PriorityController.java
/amqp-producer/src/main/java/ws/cogito/magic/messaging/PriorityMessageListener.java
Questions&FeedbackQuestions&Feedback
MyContactinformation:
[email protected]://jeremydeane.net
https://github.com/jtdeane/magic-supplies
https://github.com/jtdeane/message-client
https://github.com/jtdeane/camel-magic-router
https://github.com/jtdeane/camel-standalone-router
https://github.com/jtdeane/amqp-client
https://github.com/jtdeane/amqp-consumer
https://github.com/jtdeane/amqp-producer
https://github.com/jtdeane/amqp-cep