Asynchronous data processing

transcript

A S Y N C H R O N O U S D ATA P R O C E S S I N G

Andrea Giuliano@bit_shark

B u l g a r i a P H P CONFERENCESeptember 27th, 2015

Andrea Giuliano@bit_shark

$WHOAMI

COMMUNICATION

SYNCHRONOUS COMMUNICATION

system A

system B

Request

Response

blocked

Pros Cons

• easy to reason about• immediate feedback

• client can get stuck• impractical in low latency env

SYNCHRONOUS COMMUNICATION

ASYNCHRONOUS COMMUNICATION

system A

system B

Request

• useful for long-running jobs• systems decoupled• scales well (event-based)

• additional system requirements

ASYNCHRONOUS COMMUNICATION

Pros Cons

INTEROPERABILITY

ADVANCED MESSAGE QUEUING PROTOCOL

• message orientation

• queuing

• routing

• reliability

• security

P R O D U C E R C O N S U M E R

Produce Consumes

B R O K E R

ENTITIES INVOLVED

Produce

Exchange

Consumes

B R O K E R

THE BROKER

Produce

Exchange

Routes

Consumes

B R O K E R

THE BROKER

Produce

Exchange Queue

Routes

Consumes

B R O K E R

THE BROKER

MESSAGING PATTERNS

{ "require": { "videlalvaro/php-amqplib": "@stable", ... }}

$ composer.phar install

PRODUCER/CONSUMER

PRODUCING MESSAGES

CONSUMING MESSAGES

SCALING UP

P R O D U C E R

C O N S U M E RQUEUE

C O N S U M E R

CONSUMERS CAN DIE

MESSAGE ACKNOWLEDGEMENT

MESSAGE ACKS

switch on acks

send ack

QOS POLICY

C O N S U M E R

ROUND ROBIN DISPATCHING

C O N S U M E R

FAIR DISPATCHING

C O N S U M E R

FAIR DISPATCHING

C O N S U M E R

FAIR DISPATCHING

C O N S U M E R

FAIR DISPATCHING

C O N S U M E R

FAIR DISPATCHING

C O N S U M E R

FAIR DISPATCHING

BROKER CAN DIE

DURABILITY

declare the queue persistent

PUBLISH/SUBSCRIBE

P U B L I S H E R

S U B S C R I B E R

S U B S C R I B E R S U B S C R I B E R

PUBLISH/SUBSCRIBE

M A I L S E R V I C E

A U T H S E R V I C E

C O U P O N S E R V I C E

B R O K E R

tell me when a user is created

PUBLISH/SUBSCRIBE

U S E R S E R V I C E

B R O K E R

username: agiulianopassword: ******mail: ***@gmail.com

UserCreated

PUBLISH/SUBSCRIBE

B R O K E R

UserCreated

PUBLISH/SUBSCRIBEU S E R

S E R V I C E

B R O K E R

PUBLISH/SUBSCRIBE

B R O K E R

PUBLISH/SUBSCRIBEU

B R O K E R

ROUTING

B R O K E R

[UserCreated, UserUpdated]

UserCreated

ROUTINGU S E R

S E R V I C E

B R O K E R

routing key: updated

ROUTING

B R O K E R

routing keys: [created, updated]

ROUTING

B R O K E R

routing keys: [created, updated]

B R O K E R

[user. *]

[user. created][user. created, *.purchased]

B R O K E R

[user. *]

[user. created]

[user. created, *.purchased]

<user. created>

B R O K E R

[user. *]

[user. created]

<user. created>

B R O K E R

[user. *]

[user. created]

<user. updated>

B R O K E R

[user. *]

[user. created]

<user. updated>

B R O K E R

[user. *]

[user. created]

S T O R E S E R V I C E

<item.purchased>

B R O K E R

[user. *]

[user. created]

<item.purchased>

B R O K E R

[user. *]

[user. created]

<item.created>

B R O K E R

[user. *]

[user. created]

<item.created>

SUMMARY

PERFORMANCE

DECOUPLED SYSTEM

INTEROPERABILITY

SCALABILITY

joind.in/14878Please rate the talk

joind.in/14878

CREDITS• www.flickr.com/photos/rayofsun/9401226342• www.flickr.com/photos/kalexanderson/6231391820• www.flickr.com/photos/dhammza/2891991931• www.flickr.com/photos/kevlewis/15372562472• www.flickr.com/photos/hohumhobo/6349729535• www.flickr.com/photos/107346206• www.flickr.com/photos/berlin_steve/8414515260• www.flickr.com/photos/s3a/4816966416• www.flickr.com/photos/thatguyfromcchs08/2300190277• www.flickr.com/photos/toniblay/52445415• www.flickr.com/photos/bobsfever/8226802321• www.flickr.com/photos/activars/6590201161• www.flickr.com/photos/michaeljzealot/6485009571

Asynchronous data processing

Technology