Microservices with Docker Swarm and Consul

Agenda

- Something about Microservices

- Implement Microservices with Docker Swarm and Consul

“You shouldn't start a new project with microservices, even if you're sure your application will be big enough to make it worthwhile”(Martin Fowler)

Some SuggestionsPrograming: Python, NodeJS, Golang*

Orchestration: Docker, Docker Swarm, Consul, Consul-Template, Registrator, CoreOS*

Databases & Caches: PosgreSQL, Cassanda, ElasticSearch, Redis

Load Balancer & API Gateway: Nginx

Queues: Kafka

CI/CD: Jenkins, Ansible

Monitors: Grafana + InfluxDB + (CollectD, cAdvisor)

Logs: Kibana + ElasticSearch + FluentD

Data Center: AWS EC2, AWS S3, AWS Float IP, Cloudfront

CDN & DNS & DDOS Protection: Cloudflare

Documents: Swagger, Markdown, mkdocs

Backend System Suggestions

An Example

Node Gateway will be used a API Gateway and Load Balancer

Node Agent One, Node Agent Two will be used to run services that are able to scale.

Docker & Docker SwarmDocker is a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible.

Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts

#install

$ docker pull swarm

#run joiner

$ docker run -d swarm join \ --addr=172.20.20.11:2375 \ token://acdb9dfa3ea6da0b0cfb2c819385fcd3

#run manager

$ docker run -d -p 12375:2375 \swarm manage \token://acdb9dfa3ea6da0b0cfb2c819385fcd3

Consul and Consul-TemplateConsul

- Service Discovery

- Key/Value Storage

- Failure Detection

- Multiple Datacenter

Consul-Template

- Listen/Query the updates from Consul

- Updates Configuration File by the templates provided

- Executes commands

#run a consul agent$ consul agent -data-dir /tmp/consul -node=agent-one \ -bind=172.20.20.11 -client=0.0.0.0 \

-config-dir /etc/consul.d \ -retry-join 172.20.20.10

#run consul-templateexec consul-template \ -consul 127.0.0.1:8500 \ -template "/build/gateway/consul-template/nginx.ctmpl:/etc/nginx/sites-available/default:service nginx reload" \ -retry 30s

RegistratorRegistrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online. Registrator supports pluggable service registries, which currently includes Consul, etcd and SkyDNS 2.

#run registrator

$ docker run -d \ --name=registrator --net=host \ --volume=/var/run/docker.sock:/tmp/docker.sock \ gliderlabs/registrator:latest \ consul://localhost:8500

A Scenario

At the beginning, Service #1 only runs in Node 2.

After that Service #1 scales to Node 2

Via Docker API, Registrator will know the appearance of Service #1 on Node 2.

Registrator will register Service #1 with Consul.

In the meanwhile, Consul-Template in listening Consul. So Consul-Template will know the new service is up in Node 2. Finally, Consul-Template will trigger a reload command to Nginx add the new service into load balancer.

DEMOhttps://github.com/thanhson1085/microservices-swarm-consul

MICROSERVICES

DEVOPS

References

http://martinfowler.com

http://microservices.io/patterns/

https://www.sequoiacap.com/article/build-us-microservices/

https://sonnguyen.ws/microservices-with-docker-swarm-and-consul/