NSQ decentralized messaging

Guillaume Arnaud @guillarnaud

voyages-sncf.com

1

2

P publisher

C consumer

A Message-oriented Middleware (MoM)just a queue ?

3

ActiveMQ RabbitMQ Nats NSQ ZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto

Beanstalk Apollo Nanomsg …

JMS Stomp MQTT AMQP XMPP …

Flume Redis Logstash Syslog …

Brokers Protocol/API Almost

https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/

4

ActiveMQ RabbitMQ Nats NSQZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto

Beanstalk Apollo Nanomsg …

JMS Stomp MQTT AMQP XMPP …

Flume Redis Logstash Syslog …

Brokers Protocol/API Almost

https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/

MoM.diversity

5

NSQ.id

6

parents: nsq.io age: v0.3.8 (mar. 2016) license: open source native language: go fluently spoken: go, python, javascript, http(!) zodiacal sign: syslog++

7

P

topic: server_start_eventnsqd

message A

8

P

topic: server_start_eventnsqd

topic: server_start_eventnsqd

P

message Bmessage A

9

P

C

topic: server_start_eventnsqd

channel: server_subscribe

topic: server_start_eventnsqd

channel: server_subscribe

P

consumer: server_subscribe

tcptcp

message A message B

10

P

C

topic: server_start_eventnsqd

channel: server_subscribe

topic: server_start_eventnsqd

channel: server_subscribe

P

consumer: server_subscribe

tcptcp

message A message B

11

P

C

topic: server_start_eventnsqd

topic: server_start_eventnsqd

P

consumer: server_subscribe

C

message A message B

12

P

C

topic: server_start_eventnsqd

topic: server_start_eventnsqd

P

consumer: server_subscribe

C

message A message Bmessage A message B

13

P

C

topic: server_start_eventnsqd

topic: server_start_eventnsqd

P

consumer: server_subscribe

Cmessage A

message B

message A message B

14

C

topic: server_start_eventns

qdnsqlookupd

topic: server_start_eventns

qd topic: server_start_eventns

qd

server 1 server 2 server 3

subscribessubscribessubscribes

server 1 server 2 server 3declare server:

15

server

nsqd

server

nsqd

server

nsqd

server

nsqd

push

pull

NSQ.service discovery

16

1. application (container, tomcat, nodejs…) starts 2. application sends its information about who is it to admin 3. admin process receives and validates new configuration 4. admin fills a new haproxy configuration from a template 5. admin sends to haproxy its new configuration 6. haproxy sends confirmation of the reload to the admin

17

nsqd

nsqd

nsqd

haproxydaemon

admin

tomcat

new_ip_port1

nsqd

tomcat

18

nsqd

nsqd

nsqd

haproxydaemon

admin

tomcat

new_ip_port1

2

nsqd

tomcat

19

nsqd

nsqd

nsqd

haproxydaemon

admin

tomcat

new_ip_port1

2

3

nsqd

tomcat

20

nsqd

nsqd

nsqd

haproxydaemon

admin

tomcat

new_ip_port1

2

3

4

nsqd

tomcat

21

nsqd

nsqd

nsqd

haproxydaemon

reload

admin

tomcat

new_ip_port1

2

3

4

nsqd

tomcat

5

22

nsqd

nsqd

nsqd

haproxydaemon

reload

admin

tomcat

new_ip_port1

2

3

5

6

nsqd

tomcat

5

23

nsqd

nsqd

nsqd

haproxydaemon

reload

admin

tomcat

new_ip_port1

2

3

5

6

7

nsqd

tomcat

5

24

nsqd

nsqd

nsqd

haproxydaemon

admin

tomcat

nsqd

tomcat

my.really.hype.application.com

192.128.0.12:5123

NSQ.ops

25

$ tree bin/ bin/ ├── nsq_pubsub ├── nsq_stat ├── nsq_tail ├── nsq_to_file ├── nsq_to_http ├── nsq_to_nsq ├── nsqadmin ├── nsqd ├── nsqlookupd └── to_nsq

NSQ.ops

26

$ nsqd —help -e2e-processing-latency-percentile value … -max-bytes-per-file int -max-heartbeat-interval duration -max-msg-size int -max-msg-timeout duration -max-output-buffer-size int -max-output-buffer-timeout duration -max-rdy-count int -max-req-timeout duration -mem-queue-size int -msg-timeout string -snappy …

NSQ.admin

27

http://nsq.io/components/nsqadmin.html

your MoM is a part of your application

28

your application is a part of your distributed system

29

Questions ?

30