www.mouritech.com

Data flow among Microservices

Data flow among Microservices

At MOURI Tech, we’ve started development of a new product with clear requirements and

microservices architecture. It is a great start but the next question is what about data flow among

services?

Know your requirements, understand and proceed!!

Our requirement is to accept the request in order, enrich data with the required information and send

it to the next service. If the service is unable to handle the request then try again. This process should

be asynchronous.

Ways of data flow among services

• Traditional API relationship: Send request and receive a response. If service goes down there is

always a chance of losing request-response. This is a tightly-coupled synchronous mechanism.

• PUB-SUB relationship: Publisher sends a message to a topic, subscriber creates the subscription

and receives a message from it. This is a loosely-coupled asynchronous mechanism.

PUB-SUB message flow fits our requirement as it handles a large number of events asynchronously.

Publisher

Message (1)

•Message Storage (2)Topic

Subscription (3)

•Ack (5)Message (4)

Subscriber

www.mouritech.com

Data flow among Microservices

1. A publisher/producer application sends messages to the topic. A message contains a payload and

optional attributes that describe the payload content.

2. A queue is a buffer that stores all messages.

3. Pub/Sub forwards messages from a topic to its subscriptions individually. It depends on the way in

which the subscriber has subscribed to the topic.

4. The subscriber receives pending messages from his subscription and acknowledges each one to the

Pub/Sub service.

5. When a message is acknowledged by the subscriber, it is removed from the subscription's message

queue.

There are numerous message-oriented middleware available such as ActiveMQ/Apollo, RabbitMQ,

ZeroMQ, Kafka, IronMQ, Apache Qpid and so on. Most of these follow the same or similar flow.

FEATURE Active MQ/Apollo

RabbitMQ ZeroMQ Kafka IronMQ Apache Qpid

Brokerless/

Decentralized

No No Yes Distributed Distributed/

Cloud-based

No

Clients C++, Java,

Others

C++, Java, Ruby,

Node JS, Others

C++, Java,

Others

C++, Java,

Others

C++, Java,

Others

C++, Java,

Others

Transaction Yes Yes No No, but can be

implemented

with a plugin

No Yes

Persistence/

Reliability

Yes

(Configurable)

Yes (Built-in) No

(Persistence

requiring

higher

layers to

manage

persistence)

Yes (Build-in

File System)

Yes (Built-in) Yes

(Additional

plugin

required)

Routing Yes (Easier to

implement)

Yes (Easier to

implement)

Yes

(Complex to

implement)

No (Can be

implemented)

No No

Users CSC etc. Mozilla, AT&T,

UIDAI etc.

No LinkedIn,

Twitter, Mozilla

etc.

No No

Failover/HA Yes Yes No Yes Yes Yes

Unlimited

Queue

Yes Yes Yes Yes Yes Yes

www.mouritech.com

Data flow among Microservices

Scalability Yes Yes Yes Yes Yes Yes

License/

Community

Apache (open

source)

SpringSource

(Licensed under

Mozilla public

license)

iMatrix -

Licensed

Apache (open

source)

Iron.io

(Commercial)

Apache

(open

source)

Table: Comparison of various features in different middleware

Finally, we decided to go with RabbitMQ. Why?

RabbitMQ is a traditional messaging broker which comes with open source

In-built simple management UI & helps in monitoring and managing message queue easily

Easy to understand and implement on different languages

Supports messages over multiple protocols

Easy to trace messages and transactional history

Easy to deploy on Cloud

No message partitions - sends messages on prioritized order

This is the most important factor in our requirement. For instance, there are 2 producers (A

and B) and 1 consumer (C). Events from A has to be picked up first then events from B by C.

This is possible in RabbitMQ with priority (1-255, a larger number indicates higher priority).

Source: CloudAMQP (https://www.cloudamqp.com/img/blog/exchanges-topic-fanout-direct.png)

How to implement? Happy Coding!!

1. Create your instance on https://customer.cloudamqp.com/instance/create

a. Signup on URL.

www.mouritech.com

Data flow among Microservices

b. Follow the steps and setup instance. Once an instance is up, we can manage all in

RabbitMQ dashboard.

2. Manage Vhost, Users and permissions on the Admin page.

3. Create an application on any language such as C++, Java, Ruby, Node js, Python, PHP etc. We

have used Node js. Follow the detailed steps mentioned below or go to

https://www.rabbitmq.com/tutorials/tutorial-three-javascript.html

4. Use amqplib to establish a connection with RabbitMQ with amqp connection string.

Ex connection string: amqps://{user}:{userpassword}@{hostname}/{vhost}

const conn = await amqp.connect(connectionString, { servername: hostname, clientProperties: { connection_name: connectionname } });

conn.createChannel()

www.mouritech.com

Data flow among Microservices

5. Create exchanges, queues and bind queues:

6. Now we are ready to publish and consume events.

a. Producer: On initialization connect and set up RabbitMQ with channel, exchange,

queues and bind queue. Then keep publishing an event to consumer’s queue.

b. Consumer: On initialization connect and set up RabbitMQ with channel, exchange,

queues and bind queue. Then keep consuming incoming events.

const entity = { name: ‘FirstService’, exchangeType: 'direct', exchangeOptions: { durable: true }, queueOptions: { exclusive: false , maxPriority: 10 } } await channel.assertExchange( entity.name, entity.exchangeType, entity.exchangeOptions ); await channel.assertQueue( entity.name, entity.queueOptions ); await channel.bindQueue( entity.name, entity.name, "" );

// similarly create other service exchanges, queues and bind

queues to exchanges.

const msg = ("Hello Word" await channel.publish(targetExchange, "", Buffer.from(msg))

channel.consume(q.queue, function(msg) { if(msg.content) { console.log(" [x] %s", msg.content.toString()); } }, { noAck: true });

www.mouritech.com

Data flow among Microservices

We are thus able to achieve our requirement with the help of RabbitMQ.

References

https://microservices.io/

https://www.rabbitmq.com

https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-

rabbitmq.html

Contact for further details

Lalitha Chukka

Technology Specialist

lalithac.in@mouritech.com

MOURI Tech