Asynchronous data processing

transcript

A S Y N C H R O N O U S D ATA P R O C E S S I N G

A N D R E A G I U L I A N O@bit_shark

# C A K E F E S T

T H E P R O B L E M

P R O D U C T C ATA L O G G E N E R AT O R

• 30k+ records in db representing products

• dispose records in pages (with a certain order)

• generate a 300+ pages PDF catalog

T H E P R O B L E M

H E AV Y O P E R AT I O N S T O D O H E R E

T H E P R O B L E M

U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M

A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O

G E T S T U C K !

T H E P R O B L E M

S Y N C H R O N Y

process A

process B

Request

Response

blocked

T H E N E E D S

P R E V E N T T I M E O U TT H E N E E D S

E V E N T U A L LY D E L I V E R YT H E N E E D S

“sooner or later your job will be processed.”

“ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N

T H E N E E D S

A S Y N C H R O N Y

process A

process B

Request

I N T E R O P E R A B I L I T YT H E N E E D S

A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L

T H E S O L U T I O N

H O W D O E S I T W O R K

A M Q P

P R O D U C E R C O N S U M E R

Produce Consumes

B R O K E R

A M Q P

Produce

Exchange

Consumes

B R O K E R

A M Q P

Produce

ExchangeRoutes

Consumes

B R O K E R

A M Q P

Produce

Exchange Queue

Routes

Consumes

B R O K E R

T H R O U G H C O M P O S E R

I N S TA L L A M Q P L I B R A R Y

{ "require": { "videlalvaro/php-‐amqplib": "@stable", ... } }

$ composer.phar install

D I F F E R E N T S C E N A R I O S

P R O D U C E R / C O N S U M E RS C E N A R I O

S C E N A R I O

P R O D U C E R / C O N S U M E R

Q U E U E

S C E N A R I O

use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; $connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel();

S E T T I N G U P C O N N E C T I O N

C R E AT E A C H A N N E L

S C E N A R I O

P R O D U C E R

Q U E U E

passive durable exclusive auto-‐delete $channel-‐>queue_declare('catalog', false, false, false, false); foreach ($catalog-‐>getPages() as $page) { $message = new AMQPMessage($page); $channel-‐>basic_publish($message, '', 'catalog'); } $channel-‐>close(); $connection-‐>close();

S C E N A R I O

C O N S U M E R

Q U E U E

$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>queue_declare('catalog', false, false, false, false); $callback = function($msg) { $msg-‐>body-‐>generatePdf(); }; $channel-‐>basic_consume('catalog', '', false, true, false, false, $callback); while(count($channel-‐>callbacks)) { $channel-‐>wait(); } $channel-‐>close(); $connection-‐>close();

S C E N A R I O

M U LT I P L E C O N S U M E R S

P R O D U C E R

C O N S U M E RQ U E U E

C O N S U M E R

S C E N A R I O

M U LT I P L E C O N S U M E R S

P R O D U C E R

C O N S U M E RQ U E U E

C O N S U M E R

parallelize work

easy scalability

C O N S U M E R S C A N D I E

M E S S A G E A K N O W L E D G E M E N T

S A F E T Y

M E S S A G E A C K S

$callback = function($msg){ $msg-‐>body-‐>generatePdf(); $msg-‐>delivery_info['channel']-‐>basic_ack($msg-‐>delivery_info['delivery_tag']); }; !$channel-‐>basic_consume('catalog', '', false, false, false, false, $callback);

S W I T C H A C K O N

D O N ’ T F O R G E T T O S E N D A C K S

T H E B R O K E R C A N D I E

D U R A B I L I T Y

M A R K T H E Q U E U E A N D T H E C H A N N E L

$channel-‐>queue_declare('catalog', false, true, false, false);

In order to achieve durability

$message = new AMQPMessage($data, #the message is now persistent array('delivery_mode' => 2) );

the queue must be declared durable

the message must be marked as persistent

Q O S P O L I T I C S

Q O S P O L I T I C A

B E FA I R

C O N S U M E R

for certain instances of the Round Robin dispatching

B R O K E R

#2#4#6

Q O S P O L I T I C S

B E FA I R

$channel-‐>basic_qos(null, 1, null);

C O N S U M E R

the broker sends messages only when it receives acks

B R O K E R

#3 #2#4

P U B L I S H / S U B S C R I B ES C E N A R I O

S C E N A R I O

P U B L I S H / S U B S C R I B E

chatRoom

FA N O U T E X C H A N G E

$connection = new AMQPConnection(HOST, PORT, USER, PASSWORD); $channel = $connection-‐>channel(); $channel-‐>exchange_declare('chatRoom', 'fanout', false, false, false);

Exchange

Setting up the connection and declare the fanout exchange

S C E N A R I O

chatRoom

Exchange

$data = getAMessageToSendInTheRoom(); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom'); $channel-‐>close(); $connection-‐>close();

P R O D U C E R

Produce

Publishing a message to the exchange

N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N

S C E N A R I O

chatRoom

Q U E U E B I N D I N GExchange

//connection setted up !list($queue_name, ,) = $channel-‐>queue_declare("", false, false, true, false); $channel-‐>queue_bind($queue_name, 'chatRoom');

Bind the Queue on the Exchange

amq.gen-A7d

amq.gen-Sb4

S C E N A R I O

chatRoom

Exchange

amq.gen-A7d

$channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();

C O N S U M E R

Consumes

amq.gen-Sb4

R O U T I N GS C E N A R I O

S C E N A R I O

R O U T I N G

chatRoom

Exchange type=direct

amq.gen-A7d

amq.gen-Sb4

Consumer1

ConsumerN

P R O D U C E R

Produce

routing_keys = friends

routing_keys = friends, colleagues

$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false);

S C E N A R I O

R O U T I N G

chatRoom

Exchange type=direct

P R O D U C E R

Produce

Producing messages//connection setted up !$channel-‐>exchange_declare('chatRoom', 'direct', false, false, false); $data = getAMessageToSendInTheRoom('friends'); $msg = new AMQPMessage($data); $channel-‐>basic_publish($msg, 'chatRoom', 'friends'); $channel-‐>close(); $connection-‐>close();

S C E N A R I O

R O U T I N G

//connection setted up //exchange setted up //$queue_name is a system generated queue name !$rooms = array('friends', 'colleagues'); foreach($rooms as $room) { $channel-‐>queue_bind($queue_name, 'chatRoom', $room); } $channel-‐>basic_consume($queue_name, '', false, true, false, false, 'readMessage'); $channel-‐>close(); $connection-‐>close();

Bind a Consumer on different routing keys and consuming messages

chatRoomamq.gen-Sb4

Consumer

routing_keys = friends, colleagues

T O P I CS C E N A R I O

G O F U R T H E R

T O P I C

$channel-‐>exchange_declare('vehicle', 'topic', false, false, false);

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

A Y E L L O W S P O R T C A R

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

A R E D R A C E M O T O R B I K E

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

A R E D R A C E C A R

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

T O P I C

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

A B L U E C I T Y VA N

T O P I C

A B L U E C I T Y VA N

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

D I S C A R D E D

R E M E M B E R

P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY

E V E N I N P H P

C A N M A K E Y O U R ( D E V )L I F E H A P P I E R

Y O U R A P P L I C AT I O N S T R O N G E R

Andrea Giuliano @bit_shark andreagiuliano.it

joind.in/11608Please rate the talk!

C R E D I T S

• https://www.flickr.com/photos/rayofsun/9401226342

• https://www.flickr.com/photos/thereeljames/11376085194

• https://www.flickr.com/photos/ollily/359817111

• https://www.flickr.com/photos/legofenris/4004170937

• https://www.flickr.com/photos/bsabarnowl/10993445723

• https://www.flickr.com/photos/jpott/2984914512

• https://www.flickr.com/photos/kalexanderson/6231391820

• https://www.flickr.com/photos/a2gemma/1448178195

• https://www.flickr.com/photos/dubpics/5619966355

• https://www.flickr.com/photos/umbertofistarol/5747425870

• https://www.flickr.com/photos/streetcarl/6888965017

• https://www.flickr.com/photos/infomastern/12407730413

• https://www.flickr.com/photos/giuseppemilo/11817936944

• https://www.flickr.com/photos/avardwoolaver/7137096221

Asynchronous data processing

Data & Analytics