Post on 11-Apr-2017
transcript
NSQ decentralized messaging
Guillaume Arnaud @guillarnaud
voyages-sncf.com
1
2
P publisher
C consumer
A Message-oriented Middleware (MoM)just a queue ?
3
ActiveMQ RabbitMQ Nats NSQ ZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
4
ActiveMQ RabbitMQ Nats NSQZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
MoM.diversity
5
NSQ.id
6
parents: nsq.io age: v0.3.8 (mar. 2016) license: open source native language: go fluently spoken: go, python, javascript, http(!) zodiacal sign: syslog++
7
P
topic: server_start_eventnsqd
message A
8
P
topic: server_start_eventnsqd
topic: server_start_eventnsqd
P
message Bmessage A
9
P
C
topic: server_start_eventnsqd
channel: server_subscribe
topic: server_start_eventnsqd
channel: server_subscribe
P
consumer: server_subscribe
tcptcp
message A message B
10
P
C
topic: server_start_eventnsqd
channel: server_subscribe
topic: server_start_eventnsqd
channel: server_subscribe
P
consumer: server_subscribe
tcptcp
message A message B
11
P
C
topic: server_start_eventnsqd
topic: server_start_eventnsqd
P
consumer: server_subscribe
C
message A message B
12
P
C
topic: server_start_eventnsqd
topic: server_start_eventnsqd
P
consumer: server_subscribe
C
message A message Bmessage A message B
13
P
C
topic: server_start_eventnsqd
topic: server_start_eventnsqd
P
consumer: server_subscribe
Cmessage A
message B
message A message B
14
C
topic: server_start_eventns
qdnsqlookupd
topic: server_start_eventns
qd topic: server_start_eventns
qd
server 1 server 2 server 3
subscribessubscribessubscribes
server 1 server 2 server 3declare server:
15
server
nsqd
server
nsqd
server
nsqd
server
nsqd
push
pull
NSQ.service discovery
16
1. application (container, tomcat, nodejs…) starts 2. application sends its information about who is it to admin 3. admin process receives and validates new configuration 4. admin fills a new haproxy configuration from a template 5. admin sends to haproxy its new configuration 6. haproxy sends confirmation of the reload to the admin
17
nsqd
nsqd
nsqd
haproxydaemon
admin
tomcat
new_ip_port1
nsqd
tomcat
18
nsqd
nsqd
nsqd
haproxydaemon
admin
tomcat
new_ip_port1
2
nsqd
tomcat
19
nsqd
nsqd
nsqd
haproxydaemon
admin
tomcat
new_ip_port1
2
3
nsqd
tomcat
20
nsqd
nsqd
nsqd
haproxydaemon
admin
tomcat
new_ip_port1
2
3
4
nsqd
tomcat
21
nsqd
nsqd
nsqd
haproxydaemon
reload
admin
tomcat
new_ip_port1
2
3
4
nsqd
tomcat
5
22
nsqd
nsqd
nsqd
haproxydaemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
nsqd
tomcat
5
23
nsqd
nsqd
nsqd
haproxydaemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
nsqd
tomcat
5
24
nsqd
nsqd
nsqd
haproxydaemon
admin
tomcat
nsqd
tomcat
my.really.hype.application.com
192.128.0.12:5123
NSQ.ops
25
$ tree bin/ bin/ ├── nsq_pubsub ├── nsq_stat ├── nsq_tail ├── nsq_to_file ├── nsq_to_http ├── nsq_to_nsq ├── nsqadmin ├── nsqd ├── nsqlookupd └── to_nsq
NSQ.ops
26
$ nsqd —help -e2e-processing-latency-percentile value … -max-bytes-per-file int -max-heartbeat-interval duration -max-msg-size int -max-msg-timeout duration -max-output-buffer-size int -max-output-buffer-timeout duration -max-rdy-count int -max-req-timeout duration -mem-queue-size int -msg-timeout string -snappy …
your MoM is a part of your application
28
your application is a part of your distributed system
29
Questions ?
30