AMQPAdvanced Message Queueing Protocol
Caveats
Event LoggingLegacy Architecture & Problem
Event Logging
ServerClient
Client
Client
MongoDB
Connection Issues
ServerClient
Client
Client80
110
65
MongoDB
ServerClient
Client
Client
255
MongoDB
80
110
65
Connection Issues
Server MIA
ServerClient
Client
Client
0
0
0
0
MongoDB
Server
Network Segmentation
0
0
0MongoDB
Client
Client
Client
Total data loss
DATA LOSS?!
Why AMQP?
• Stable and mature
• Designed for SX markets
• Widely supported (good interop)
• Modular, Fast & Flexible
Service ModelWire
AMQP Overview
AMQP
Advanced Message Queueing Protocol
AMQ Model Wire
AMQP Overview
AMQP
Server-side service model
defined components
rules for wiring
AMQ Model Wire
AMQP Overview
AMQP
AMQ Model Wire
AMQP Overview
AMQP
Network wire-level protocol
command architecture
connections, channels, etc
ExchangeQueuesBindingAMQ Model
AMQ Model
QueuesExchange Binding
AMQ Model
AMQ Model
Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Publisher Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Publisher Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
?Message
Publisher Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
Publisher Consumer
?
Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
Publisher
Queue: critical_error_queue
Exchange: errors_exchange
Filter: *.error.critical
Consumer
Virtual Host
QueuesExchange
Binding
AMQ Model
Queue: critical_error_queue
Exchange: errors_exchange
Filter: *.error.critical
Message
Router
Publisher Consumer
QueuesExchange Binding
Important notes
Publisher Consumer
‣ creates a relationship between queues and exchanges ‣ contains criteria and properties ‣ can be created at runtime by consumers
‣ store messages ‣ named ‣ bound-able to exchange ‣ criteria ‣ can be created at runtime by consumers
‣ routes messages based on criteria ‣ doesn’t store messages ‣ can inspect message content ‣ can be created at runtime consumers
‣ can create exchanges and queues
‣ can create exchanges and queues
Exchange
The Exchange
FanoutTopicHeadersSystem
Exchange
Types (routing algo)
The Exchange
Fanout Topic Headers System
Exchange
Types (routing algo)
Direct
The Exchange
Fanout Topic Headers System
Exchange
Types (routing algo)
Direct
The Exchange
Exchange ‘events’
Direct Type Exchange
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
Direct Type Exchange
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Direct Type Exchange
Exchange ‘events’
Binding
➡Queue: ‣ user_click_queue
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Direct Type Exchange
Exchange ‘events’
Binding
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Direct Type Exchange
Exchange ‘events’
Binding
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
➡Routing Key: ‣ event.user.click
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Direct Type Exchange
Exchange ‘events’
Binding
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
➡Routing Key: ‣ event.user.click
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
user_click_queue
Direct Type Exchange
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
user_view_queue
user_share_queue
Direct Type Exchange
Yeah, so?
Exchange ‘events’
What if…
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Exchange ‘events’
What if…
user_interaction_queue
user_shares_queue
Message
Message
Message
event.user.click
event.user.view
event.user.share
Exchange ‘events’
What if…
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Exchange ‘events’
Topic Exchange
user_interaction_queue
Binding
Message
Message
Message
event.user.click
event.user.view
event.user.share
Exchange ‘events’
user_interaction_queue
Binding
➡Queue: ‣ user_interaction_queue
➡Exchange: ‣ events
➡Routing Key: ‣ event.user.*
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
Topic Exchange
Message
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
event.user.*
event.user.share
Topic Exchange
Can be done at runtime
Message
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Topic Exchange
Message
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Messageclick_share_queue
Message
event.user.share event.user.click
Topic Exchange
Message
Exchange ‘events’
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queueMessage
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Message event.staff.view
MessageMessage
Staff vs user views?
Topic Exchange
I’m still pissed
New Event LoggingExplorations & Implementation
Revisiting Our Logging
Client
Client
Client
MongoDB
Event Logging Revised
Client
Client
ClientMQ
MQ
MQ
MongoDB
Event Logging Revised
Client
Client
ClientMQ
MQ
MQ
MongoDB
Event Logging Revised
Client
Client
ClientMQ
MQ
MQ
MongoDB
No data loss!
SRSLY?!
MongoDB
Event Logging Revised
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
MongoDBMQClient
Client
Client
80
110
65
MQ
MQ
MQ
Event Logging Revised
MongoDBMQClient
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Event Logging Revised
MongoDB
MQ
Client
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Event Logging Revised
Message
Exchange ‘events’
Remember Topic Exchange?
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queueMessage
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Message event.staff.view
MessageMessage
MongoDB
Provides flexibility
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Reporting SQL
Notification
• durable
• passive
• no-wait
• exclusive
• auto-delete
• qos
• ack
• message recovery
Not Covered
A quick demo of AMQPwith RabbitMQ
Thank [email protected]
