Core os dna_oscon

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Inserting CoreOS DNA for Creating Docker Clusters.

Advanced Technology GroupJuly 22, 2015


Inserting CoreOS DNA for Creating Docker Clusters.

Why? What’s the goal of this presentation?


About the speaker● Patrick Galbraith ● HP Advanced Technology Group● Has worked at Blue Gecko, MySQL AB,

Classmates, Slashdot, Cobalt Group, US Navy, K-mart

● MySQL projects: memcached UDFs, DBD::mysql, Ansible HP switch drivers

● federated storage engine● Family● Outdoors

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

Enterprise workloads are migrating towards Docker

Server

Host OS

Docker

bins+libs

Workload B

Workload A

bins+libsDocker

Server

Host OS

Hypervisor

Guest OSGuest OS

bins+libs

Workload B

Workload A

bins+libsVirtual

Machine

Docker is Great• Lightweight “container” technology.• Intelligent Packaging – Docker

Images• Intelligent Deployment – Docker

Containers• Rapidly evolving ecosystem.• Linux IS the API.

Docker has some gaps• Scalability: Docker is a host

application.• Not Multi-Tenant.• No comprehensive Host-to-Host

networking.• Host OS maintenance is not included.• No workload (Docker Container)

distribution.


Clustering Docker – What’s Important

Coordination

Configuration + discovery for the base cluster, and applications.

Deployment Deploy a Docker Image to some node in the cluster.

Scheduler Place Docker Images efficiently on to the cluster.

NetworkInter-Host networking is obscured in the default Docker model.

The default model uses an internal only bridge.

Maintenance

Install & update the base system in a scalable and effective way.

Note: Docker provides its own system: Images & Containers.


Existing Approaches to Clustering Docker

• Coreos• Kubernetes• Swarm• Docker Machine• Project Atomic• Apache Mesos• RancherOS


CoreOS DNA

The Clustered Docker Proof of

Concept


Take the clear winners now. Ensure room for the remaining winners later.

Insert CoreOS DNA in Debian

Coordination etcd Rapid growth in both use and popularity combined with a rapidly growing ecosystem.

Deployment fleetIntegrated with etcd. Technologically very similar to geard. Both utilize systemd for local container start/stop.

Scheduler — Kubernetes

Network — Flannel most popular (ubiquity)Weave, SocketPlane (SDN)

Maintenance — RancherOS

Linux Distribution

Debian-based

Chosen based on the direction of HP Helion and hLinux. The critical component is systemd.


CoreOS DNA

Architecture


Use Cases:• In Cloud• Moonshot• Bare

metal

Single Node – CoreOS DNA

Linux Kernel

etcd

fleetd

Base System

CoreOS DNA Node

dockerd

Docker Containers

systemd

App Container 1

App Container …

Port: 4001

Fleetctl interacts with fleetd by directly changing values in etcd.

etcdctl

Client(s)

fleetctl


CoreOS DNA Cluster

Cluster Configuration – CoreOS DNA

CoreOS DNA NodeDocker Containers etcd

fleetddockerd

systemd


fleetddockerd

systemd


fleetddockerd

systemd…

etcd Discovery Server

etcd

http://discovery.etcd.io

Docker Registry

Images

Discovery

Control Node(jump box)

etcdctl

fleetctl

ssh

1

2

3

1. Cluster Start (etcd discovery)

2. Container Start ( fleetctl )

3. Docker Download


Networking as deployed – CoreOS DNA

eth0

Linux Kernel

etcd

fleetd

CoreOS DNA Node

dockerd

Docker Containers

Container

Container

Container

Container

docker0

172.x.x.xiptables

port mapping

Netfilter

CoreOS DNA Node

CoreOS DNA Node

CoreOS DNA Node

Neutron Router

10.x.x.x

Public Interne

t

15.x.x.x

NAT

172.x.x.x - Docker Internal10.x.x.x - Host (private) Network15.x.x.x - Public (NAT’ed) Addresses


ELK Stack + Sinatra Worker Agents (ELK - Elasticsearch, Logstash, Kibana)

Example Application – CoreOS DNA

CoreOS DNA Cluster


fleetddockerd

systemd


fleetddockerd

systemd


fleetddockerd

systemd…

Agent “@5001”

Agent “@5003”

Agent “@5002”

Agent “@5000”Nginx Logstash

Elasticsearch +

KabannaDockerImages

Fleet Mapped(scheduled into)

DockerContainers


Building the POC – Lessons Learned


Building the cluster Using Ansible

Ansible Modules• Nova_compute – to launch instances• Nova_facts – used to build inventory files for launched instances• Docker and docker_facts – used to run containers outside of fleet (testing)

and verification• Docker_pull – pre-pull images on instances for faster launch by systemd (via

fleet)

Using Ansible to provision etcd and build clusters• query discovery URL • write URL to local file ad set as a variable• render etcd service file with the variable • Build, configure, and run etcd • Build, configure and run Fleet


The Special Sauce

Etcd• Integral to cluster functioning.• Fleet communicates with etcd to obtain key/values from etcd.• Etcd also used by the sample ELK app to store key/value pairs used by confd

to render config files upon running containers (boot).

Confd• Stored in each Docker container.• Keeps an eye on files rendered.• Can use etcd key/value pairs to interpolate what it rendered.• Automatically keeps config files up to date with etcd information.


Sample App unit files

Systemd unit files• Unit file directives

– ExecStartPre – pull image– ExecStart – run container– ExecStartPost– set IP of the container in etcd– ExecStopPost – remove IP in etcd upon container stop

• ElasticSearch – sets its own public and private IPs in etcd for discovery by logstash

• Logstash – sets its own IP in etcd for discovery by logstash agents• Sinatra app – sets title of app in etcd as well as IP addresses for discovery by

nginx to generate nginx conf using confd


Sample App container Dockerfiles

Docker file functionality• ElasticSearch – install confd, install and configure elasticsearch, install kopf

and kibana plugins,expose port 9200, launch• Logstash – Install confd, Install and configure logstash, run boot script• Sinatra – Install sinatra, confd, place logstash agent, expose port 5000, run

boot.sh• Nginx – Install nginx, confd, run boot.sh


Sample App container CMD scripts

Boot Script• Logstash – render logstash config (confd –onetime), generate SSL private key

and cert, stores in etcd, then run logstash• Sinatra – render app.rb, SSL cert and keys, logstash forwarder config, start

logstash forwarder, start sinatra app (foreman)• Nginx – render nginx.conf (-onetime) and start confd to check and update

conf every 10 minutes, start nginx, tail nginx logs


Confd – resource file (nginx)

[template]

keys = [ "app/server", "elasticsearch/host" ]

owner = "nginx"

mode = "0644"

src = "nginx.conf.tmpl"

dest = "/etc/nginx/sites-enabled/docker_dns.conf"

check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"

reload_cmd = "/usr/sbin/service nginx reload"


Confd – template (nginx)

upstream app {

{{ range $server := .app_server }}

server {{ $server.Value }};

{{ end }}

}

…

upstream elasticsearch {

server {{ .elasticsearch_host }}:9200;

keepalive 15;

}


Looking at etcdubuntu@dod-01:~$ etcdctl ls --recursive

/elasticsearch

/elasticsearch/host

/elasticsearch/hostpublic

/logstash

/logstash/ssl_certificate

/logstash/ssl_private_key

/logstash/host

/app

/app/title

/app/server

/app/server/5000

/app/server/5001

/app/server/5002

/app/server/5003

ubuntu@dod-01:~$ etcdctl get /app/server/5000

10.0.0.58:5000


The CoreOS DNA Cluster + ELK Stack


ELK Stack + Sinatra Worker Agents (ELK - Elasticsearch, Logstash, Kibana)

Example Application – Application Architecture

HTTP

LogstashService

HTTP

Nginx

Port: 80

Key

Docker Container

HTTP

LogstashPublic

Internet

Worker “@5001”Worker

“@5002”Worker

“@5003”

…

Worker “@5000”Sinatra

Service:“Hello World”Logstash Agent

HTTP

etcd

/logstash/logstash/host…/app/server/5000/app/server/5001…/elasticsearch/host…

etcd(federated

)

ElasticsearchKiban

aKopfPlugins:


Demohttps://youtu.be/pRtQ0AXYe6M


TODO

• Use etcd2 setup• Set up flannel and inter-host container networking• Add Kubernetes components• Other Linux distributions


Questions, Comments & Feedback?

Dod Ansible Repository https://github.com/HPATG/DeCore

Sample app code https://github.com/HPATG/sample_app

Marcel De Graaf’s blog post

http://marceldegraaf.net/2014/05/05/coreos-follow-up-sinatra-logstash-elasticsearch-kibana.html


Thank You

Advanced Technology Group

Eric Gustafson [email protected]

Yazz Atlas [email protected]

Patrick Galbraith [email protected]

Special Thanks

Marcel De Graaf http://marceldegraaf.net/

Kelsey Hightower https://github.com/kelseyhightower

Date post:	14-Aug-2015
Category:	Software
Upload:	patrick-galbraith
View:	30 times
Download:	0 times

Core os dna_oscon

Software