Date post: | 17-May-2015 |
Category: |
Data & Analytics |
Upload: | andrea-giuliano |
View: | 2,978 times |
Download: | 2 times |
A S Y N C H R O N O U S D ATA P R O C E S S I N G
A N D R E A G I U L I A N O@bit_shark
# C A K E F E S T
T H E P R O B L E M
P R O D U C T C ATA L O G G E N E R AT O R
• 30k+ records in db representing products
• dispose records in pages (with a certain order)
• generate a 300+ pages PDF catalog
T H E P R O B L E M
H E AV Y O P E R AT I O N S T O D O H E R E
T H E P R O B L E M
U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M
U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M
A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O
G E T S T U C K !
T H E P R O B L E M
S Y N C H R O N Y
time
process A
process B
Request
Response
blocked
T H E N E E D S
P R E V E N T T I M E O U TT H E N E E D S
E V E N T U A L LY D E L I V E R YT H E N E E D S
“sooner or later your job will be processed.”
“ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N
T H E N E E D S
T H E N E E D S
A S Y N C H R O N Y
time
process A
process B
Request
I N T E R O P E R A B I L I T YT H E N E E D S
A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L
T H E S O L U T I O N
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange
Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
ExchangeRoutes
Consumes
B R O K E R
H O W D O E S I T W O R K
A M Q P
P R O D U C E R C O N S U M E R
Produce
Exchange Queue
Routes
Consumes
B R O K E R
T H R O U G H C O M P O S E R
I N S TA L L A M Q P L I B R A R Y
{ "require": { "videlalvaro/php-‐amqplib": "@stable", ... } }
$ composer.phar install
D I F F E R E N T S C E N A R I O S
P R O D U C E R / C O N S U M E RS C E N A R I O
S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R C O N S U M E R
Q U E U E
S C E N A R I O
P R O D U C E R / C O N S U M E R
use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel();
S E T T I N G U P C O N N E C T I O N
C R E AT E A C H A N N E L
S C E N A R I O
P R O D U C E R / C O N S U M E R
P R O D U C E R
Q U E U E
passive durable exclusive auto-‐delete $channel-‐>queue_declare('catalog', false, false, false, false); foreach ($catalog-‐>getPages() as $page) { $message = new AMQPMessage($page); $channel-‐>basic_publish($message, '', 'catalog'); } $channel-‐>close(); $connection-‐>close();
S C E N A R I O
P R O D U C E R / C O N S U M E R
C O N S U M E R
Q U E U E
$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>queue_declare('catalog', false, false, false, false); $callback = function($msg) { $msg-‐>body-‐>generatePdf(); }; $channel-‐>basic_consume('catalog', '', false, true, false, false, $callback); while(count($channel-‐>callbacks)) { $channel-‐>wait(); } $channel-‐>close(); $connection-‐>close();
S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E RQ U E U E
C O N S U M E R
…
S C E N A R I O
M U LT I P L E C O N S U M E R S
P R O D U C E R
C O N S U M E RQ U E U E
C O N S U M E R
…
parallelize work
easy scalability
C O N S U M E R S C A N D I E
M E S S A G E A K N O W L E D G E M E N T
S A F E T Y
M E S S A G E A C K S
$callback = function($msg){ $msg-‐>body-‐>generatePdf(); $msg-‐>delivery_info['channel']-‐>basic_ack($msg-‐>delivery_info['delivery_tag']); }; !$channel-‐>basic_consume('catalog', '', false, false, false, false, $callback);
S W I T C H A C K O N
D O N ’ T F O R G E T T O S E N D A C K S
T H E B R O K E R C A N D I E
D U R A B I L I T Y
D U R A B I L I T Y
M A R K T H E Q U E U E A N D T H E C H A N N E L
$channel-‐>queue_declare('catalog', false, true, false, false);
In order to achieve durability
$message = new AMQPMessage($data, #the message is now persistent array('delivery_mode' => 2) );
the queue must be declared durable
the message must be marked as persistent
Q O S P O L I T I C S
Q O S P O L I T I C A
B E FA I R
C O N S U M E R
C O N S U M E R
for certain instances of the Round Robin dispatching
B R O K E R
#1#3
#5
#2#4#6
Q O S P O L I T I C S
B E FA I R
$channel-‐>basic_qos(null, 1, null);
C O N S U M E R
C O N S U M E R
the broker sends messages only when it receives acks
B R O K E R
#1
#3
#3 #2#4
P U B L I S H / S U B S C R I B ES C E N A R I O
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
FA N O U T E X C H A N G E
$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>exchange_declare('chatRoom', 'fanout', false, false, false);
Exchange
…
Setting up the connection and declare the fanout exchange
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
$data = getAMessageToSendInTheRoom(); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom'); $channel-‐>close(); $connection-‐>close();
P R O D U C E R
Produce
Publishing a message to the exchange
N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Q U E U E B I N D I N GExchange
…
//connection setted up !list($queue_name, ,) = $channel-‐>queue_declare("", false, false, true, false); $channel-‐>queue_bind($queue_name, 'chatRoom');
Bind the Queue on the Exchange
amq.gen-A7d
bind
bind
amq.gen-Sb4
S C E N A R I O
P U B L I S H / S U B S C R I B E
chatRoom
Exchange
…
amq.gen-A7d
$channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();
C O N S U M E R
Consumes
amq.gen-Sb4
R O U T I N GS C E N A R I O
S C E N A R I O
R O U T I N G
chatRoom
Exchange type=direct
…
amq.gen-A7d
amq.gen-Sb4
Consumer1
ConsumerN
P R O D U C E R
Produce
routing_keys = friends
routing_keys = friends, colleagues
$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false);
S C E N A R I O
R O U T I N G
chatRoom
Exchange type=direct
P R O D U C E R
Produce
Producing messages//connection setted up !$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false); $data = getAMessageToSendInTheRoom('friends'); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom', 'friends'); $channel-‐>close(); $connection-‐>close();
S C E N A R I O
R O U T I N G
//connection setted up //exchange setted up //$queue_name is a system generated queue name !$rooms = array('friends', 'colleagues'); foreach($rooms as $room) { $channel-‐>queue_bind($queue_name, 'chatRoom', $room); } $channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();
Bind a Consumer on different routing keys and consuming messages
chatRoomamq.gen-Sb4
Consumer
routing_keys = friends, colleagues
T O P I CS C E N A R I O
G O F U R T H E R
T O P I C
$channel-‐>exchange_declare('vehicle', 'topic', false, false, false);
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<type>.<vehicle>.<colour>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A Y E L L O W S P O R T C A R
T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A Y E L L O W S P O R T C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<sport>.<car>.<yellow>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A R E D R A C E M O T O R B I K E
T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A R E D R A C E M O T O R B I K E
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<motorbike>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A R E D R A C E C A R
T O P I C
A R E D R A C E C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
T O P I C
A R E D R A C E C A R
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<race>.<car>.<red>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
A B L U E C I T Y VA N
T O P I C
A B L U E C I T Y VA N
vehicle
Exchange type=topic
amq.gen-A7d
amq.gen-Sb4
Consumer1
Consumer2
P R O D U C E R
Produce
routing_keys = *.car.*
routing_keys = race.#
<city>.<van>.<blue>
routing_keys = *.*.red
D O T S D E L I M I T E D R O U T I N G K E Y
D I S C A R D E D
R E M E M B E R
P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY
E V E N I N P H P
C A N M A K E Y O U R ( D E V )L I F E H A P P I E R
Y O U R A P P L I C AT I O N S T R O N G E R
Andrea Giuliano @bit_shark andreagiuliano.it
joind.in/11608Please rate the talk!
C R E D I T S
• https://www.flickr.com/photos/rayofsun/9401226342
• https://www.flickr.com/photos/thereeljames/11376085194
• https://www.flickr.com/photos/ollily/359817111
• https://www.flickr.com/photos/legofenris/4004170937
• https://www.flickr.com/photos/bsabarnowl/10993445723
• https://www.flickr.com/photos/jpott/2984914512
• https://www.flickr.com/photos/kalexanderson/6231391820
• https://www.flickr.com/photos/a2gemma/1448178195
• https://www.flickr.com/photos/dubpics/5619966355
• https://www.flickr.com/photos/umbertofistarol/5747425870
• https://www.flickr.com/photos/streetcarl/6888965017
• https://www.flickr.com/photos/infomastern/12407730413
• https://www.flickr.com/photos/giuseppemilo/11817936944
• https://www.flickr.com/photos/avardwoolaver/7137096221