+ All Categories
Home > Data & Analytics > Asynchronous data processing

Asynchronous data processing

Date post: 17-May-2015
Category:
Upload: andrea-giuliano
View: 2,978 times
Download: 2 times
Share this document with a friend
Description:
Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating information behind the scenes. There are different message queuing systems designed to manage and process data asynchronously. Using simple messages, it becomes possible to define many types of queue patterns, from the straightforward solution to the more complex one like routing, publisher/subscriber and topic. The purpose of the talk is to show how to approach the different scenarios in php, adding value to your application.
Popular Tags:
70
ASYNCHRONOUS DATA PROCESSING ANDREA GIULIANO @bit_shark #CAKEFEST
Transcript
Page 1: Asynchronous data processing

A S Y N C H R O N O U S D ATA P R O C E S S I N G

A N D R E A G I U L I A N O@bit_shark

# C A K E F E S T

Page 2: Asynchronous data processing

T H E P R O B L E M

Page 3: Asynchronous data processing

P R O D U C T C ATA L O G G E N E R AT O R

• 30k+ records in db representing products

• dispose records in pages (with a certain order)

• generate a 300+ pages PDF catalog

T H E P R O B L E M

Page 4: Asynchronous data processing

H E AV Y O P E R AT I O N S T O D O H E R E

T H E P R O B L E M

Page 5: Asynchronous data processing

U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M

Page 6: Asynchronous data processing

U S E R S D O N ’ T W A N T T O W A I TT H E P R O B L E M

A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O

G E T S T U C K !

Page 7: Asynchronous data processing

T H E P R O B L E M

S Y N C H R O N Y

time

process A

process B

Request

Response

blocked

Page 8: Asynchronous data processing

T H E N E E D S

Page 9: Asynchronous data processing

P R E V E N T T I M E O U TT H E N E E D S

Page 10: Asynchronous data processing

E V E N T U A L LY D E L I V E R YT H E N E E D S

“sooner or later your job will be processed.”

Page 11: Asynchronous data processing

“ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N

T H E N E E D S

Page 12: Asynchronous data processing

T H E N E E D S

A S Y N C H R O N Y

time

process A

process B

Request

Page 13: Asynchronous data processing

I N T E R O P E R A B I L I T YT H E N E E D S

Page 14: Asynchronous data processing

A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L

T H E S O L U T I O N

Page 15: Asynchronous data processing

H O W D O E S I T W O R K

A M Q P

P R O D U C E R C O N S U M E R

Produce Consumes

B R O K E R

Page 16: Asynchronous data processing

H O W D O E S I T W O R K

A M Q P

P R O D U C E R C O N S U M E R

Produce

Exchange

Consumes

B R O K E R

Page 17: Asynchronous data processing

H O W D O E S I T W O R K

A M Q P

P R O D U C E R C O N S U M E R

Produce

ExchangeRoutes

Consumes

B R O K E R

Page 18: Asynchronous data processing

H O W D O E S I T W O R K

A M Q P

P R O D U C E R C O N S U M E R

Produce

Exchange Queue

Routes

Consumes

B R O K E R

Page 19: Asynchronous data processing

T H R O U G H C O M P O S E R

I N S TA L L A M Q P L I B R A R Y

{          "require":  {                  "videlalvaro/php-­‐amqplib":  "@stable",                    ...          }  }  

$ composer.phar install

Page 20: Asynchronous data processing

D I F F E R E N T S C E N A R I O S

Page 21: Asynchronous data processing

P R O D U C E R / C O N S U M E RS C E N A R I O

Page 22: Asynchronous data processing

S C E N A R I O

P R O D U C E R / C O N S U M E R

P R O D U C E R C O N S U M E R

Q U E U E

Page 23: Asynchronous data processing

S C E N A R I O

P R O D U C E R / C O N S U M E R

use  PhpAmqpLib\Connection\AMQPConnection;  use  PhpAmqpLib\Message\AMQPMessage;      $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);  $channel  =  $connection-­‐>channel();

S E T T I N G U P C O N N E C T I O N

C R E AT E A C H A N N E L

Page 24: Asynchronous data processing

S C E N A R I O

P R O D U C E R / C O N S U M E R

P R O D U C E R

Q U E U E

                                                                       passive    durable  exclusive  auto-­‐delete  $channel-­‐>queue_declare('catalog',  false,    false,    false,        false);        foreach  ($catalog-­‐>getPages()  as  $page)  {          $message  =  new  AMQPMessage($page);          $channel-­‐>basic_publish($message,  '',  'catalog');  }      $channel-­‐>close();  $connection-­‐>close();

Page 25: Asynchronous data processing

S C E N A R I O

P R O D U C E R / C O N S U M E R

C O N S U M E R

Q U E U E

$connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);  $channel  =  $connection-­‐>channel();      $channel-­‐>queue_declare('catalog',  false,  false,  false,  false);      $callback  =  function($msg)  {          $msg-­‐>body-­‐>generatePdf();  };      $channel-­‐>basic_consume('catalog',  '',  false,  true,  false,  false,  $callback);      while(count($channel-­‐>callbacks))  {          $channel-­‐>wait();  }      $channel-­‐>close();  $connection-­‐>close();  

Page 26: Asynchronous data processing

S C E N A R I O

M U LT I P L E C O N S U M E R S

P R O D U C E R

C O N S U M E RQ U E U E

C O N S U M E R

Page 27: Asynchronous data processing

S C E N A R I O

M U LT I P L E C O N S U M E R S

P R O D U C E R

C O N S U M E RQ U E U E

C O N S U M E R

parallelize work

easy scalability

Page 28: Asynchronous data processing

C O N S U M E R S C A N D I E

Page 29: Asynchronous data processing

M E S S A G E A K N O W L E D G E M E N T

Page 30: Asynchronous data processing

S A F E T Y

M E S S A G E A C K S

$callback  =  function($msg){          $msg-­‐>body-­‐>generatePdf();          $msg-­‐>delivery_info['channel']-­‐>basic_ack($msg-­‐>delivery_info['delivery_tag']);  };  !$channel-­‐>basic_consume('catalog',  '',  false,  false,  false,  false,  $callback);

S W I T C H A C K O N

Page 31: Asynchronous data processing

D O N ’ T F O R G E T T O S E N D A C K S

Page 32: Asynchronous data processing

T H E B R O K E R C A N D I E

Page 33: Asynchronous data processing

D U R A B I L I T Y

Page 34: Asynchronous data processing

D U R A B I L I T Y

M A R K T H E Q U E U E A N D T H E C H A N N E L

$channel-­‐>queue_declare('catalog',  false,  true,  false,  false);

In order to achieve durability

$message  =  new  AMQPMessage($data,                #the  message  is  now  persistent                array('delivery_mode'  =>  2)                  );

the queue must be declared durable

the message must be marked as persistent

Page 35: Asynchronous data processing

Q O S P O L I T I C S

Page 36: Asynchronous data processing

Q O S P O L I T I C A

B E FA I R

C O N S U M E R

C O N S U M E R

for certain instances of the Round Robin dispatching

B R O K E R

#1#3

#5

#2#4#6

Page 37: Asynchronous data processing

Q O S P O L I T I C S

B E FA I R

$channel-­‐>basic_qos(null,  1,  null);

C O N S U M E R

C O N S U M E R

the broker sends messages only when it receives acks

B R O K E R

#1

#3

#3 #2#4

Page 38: Asynchronous data processing

P U B L I S H / S U B S C R I B ES C E N A R I O

Page 39: Asynchronous data processing

S C E N A R I O

P U B L I S H / S U B S C R I B E

chatRoom

FA N O U T E X C H A N G E

$connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);  $channel  =  $connection-­‐>channel();      $channel-­‐>exchange_declare('chatRoom',  'fanout',  false,  false,  false);

Exchange

Setting up the connection and declare the fanout exchange

Page 40: Asynchronous data processing

S C E N A R I O

P U B L I S H / S U B S C R I B E

chatRoom

Exchange

$data  =  getAMessageToSendInTheRoom();  $msg  =  new  AMQPMessage($data);      $channel-­‐>basic_publish($msg,  'chatRoom');      $channel-­‐>close();  $connection-­‐>close();  

P R O D U C E R

Produce

Publishing a message to the exchange

Page 41: Asynchronous data processing

N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N

Page 42: Asynchronous data processing

S C E N A R I O

P U B L I S H / S U B S C R I B E

chatRoom

Q U E U E B I N D I N GExchange

//connection  setted  up  !list($queue_name,  ,)  =  $channel-­‐>queue_declare("",  false,  false,  true,  false);      $channel-­‐>queue_bind($queue_name,  'chatRoom');

Bind the Queue on the Exchange

amq.gen-A7d

bind

bind

amq.gen-Sb4

Page 43: Asynchronous data processing

S C E N A R I O

P U B L I S H / S U B S C R I B E

chatRoom

Exchange

amq.gen-A7d

$channel-­‐>basic_consume($queue_name,                    '',                    false,                    true,                    false,                    false,                    'readMessage');      $channel-­‐>close();  $connection-­‐>close();  

C O N S U M E R

Consumes

amq.gen-Sb4

Page 44: Asynchronous data processing

R O U T I N GS C E N A R I O

Page 45: Asynchronous data processing

S C E N A R I O

R O U T I N G

chatRoom

Exchange type=direct

amq.gen-A7d

amq.gen-Sb4

Consumer1

ConsumerN

P R O D U C E R

Produce

routing_keys = friends

routing_keys = friends, colleagues

$channel-­‐>exchange_declare('chatRoom',  'direct',  false,  false,  false);

Page 46: Asynchronous data processing

S C E N A R I O

R O U T I N G

chatRoom

Exchange type=direct

P R O D U C E R

Produce

Producing messages//connection  setted  up  !$channel-­‐>exchange_declare('chatRoom',                      'direct',                      false,                      false,                      false);        $data  =  getAMessageToSendInTheRoom('friends');  $msg  =  new  AMQPMessage($data);      $channel-­‐>basic_publish($msg,  'chatRoom',  'friends');      $channel-­‐>close();  $connection-­‐>close();  

Page 47: Asynchronous data processing

S C E N A R I O

R O U T I N G

//connection  setted  up  //exchange  setted  up  //$queue_name  is  a  system  generated  queue  name  !$rooms  =  array('friends',  'colleagues');      foreach($rooms  as  $room)  {          $channel-­‐>queue_bind($queue_name,  'chatRoom',  $room);  }      $channel-­‐>basic_consume($queue_name,  '',  false,  true,  false,  false,  'readMessage');      $channel-­‐>close();  $connection-­‐>close();  

Bind a Consumer on different routing keys and consuming messages

chatRoomamq.gen-Sb4

Consumer

routing_keys = friends, colleagues

Page 48: Asynchronous data processing

T O P I CS C E N A R I O

Page 49: Asynchronous data processing

G O F U R T H E R

T O P I C

$channel-­‐>exchange_declare('vehicle',  'topic',  false,  false,  false);

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<type>.<vehicle>.<colour>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 50: Asynchronous data processing

A Y E L L O W S P O R T C A R

Page 51: Asynchronous data processing

T O P I C

A Y E L L O W S P O R T C A R

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<sport>.<car>.<yellow>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 52: Asynchronous data processing

T O P I C

A Y E L L O W S P O R T C A R

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<sport>.<car>.<yellow>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 53: Asynchronous data processing

A R E D R A C E M O T O R B I K E

Page 54: Asynchronous data processing

T O P I C

A R E D R A C E M O T O R B I K E

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<race>.<motorbike>.<red>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 55: Asynchronous data processing

T O P I C

A R E D R A C E M O T O R B I K E

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<race>.<motorbike>.<red>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 56: Asynchronous data processing

A R E D R A C E C A R

Page 57: Asynchronous data processing

T O P I C

A R E D R A C E C A R

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<race>.<car>.<red>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 58: Asynchronous data processing

T O P I C

A R E D R A C E C A R

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<race>.<car>.<red>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 59: Asynchronous data processing

A B L U E C I T Y VA N

Page 60: Asynchronous data processing

T O P I C

A B L U E C I T Y VA N

vehicle

Exchange type=topic

amq.gen-A7d

amq.gen-Sb4

Consumer1

Consumer2

P R O D U C E R

Produce

routing_keys = *.car.*

routing_keys = race.#

<city>.<van>.<blue>

routing_keys = *.*.red

D O T S D E L I M I T E D R O U T I N G K E Y

Page 61: Asynchronous data processing

D I S C A R D E D

Page 62: Asynchronous data processing
Page 63: Asynchronous data processing

R E M E M B E R

Page 64: Asynchronous data processing

P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY

Page 65: Asynchronous data processing

E V E N I N P H P

Page 66: Asynchronous data processing

C A N M A K E Y O U R ( D E V )L I F E H A P P I E R

Page 67: Asynchronous data processing

Y O U R A P P L I C AT I O N S T R O N G E R

Page 68: Asynchronous data processing

Andrea Giuliano @bit_shark andreagiuliano.it

Page 69: Asynchronous data processing

joind.in/11608Please rate the talk!

Page 70: Asynchronous data processing

C R E D I T S

• https://www.flickr.com/photos/rayofsun/9401226342

• https://www.flickr.com/photos/thereeljames/11376085194

• https://www.flickr.com/photos/ollily/359817111

• https://www.flickr.com/photos/legofenris/4004170937

• https://www.flickr.com/photos/bsabarnowl/10993445723

• https://www.flickr.com/photos/jpott/2984914512

• https://www.flickr.com/photos/kalexanderson/6231391820

• https://www.flickr.com/photos/a2gemma/1448178195

• https://www.flickr.com/photos/dubpics/5619966355

• https://www.flickr.com/photos/umbertofistarol/5747425870

• https://www.flickr.com/photos/streetcarl/6888965017

• https://www.flickr.com/photos/infomastern/12407730413

• https://www.flickr.com/photos/giuseppemilo/11817936944

• https://www.flickr.com/photos/avardwoolaver/7137096221


Recommended