+ All Categories
Home > Technology > Taking Control of Chaos with Docker and Puppet

Taking Control of Chaos with Docker and Puppet

Date post: 08-May-2015
Category:
Upload: puppet-labs
View: 17,426 times
Download: 6 times
Share this document with a friend
Description:
"Taking Control of Chaos (with Docker and Puppet)" by Tomas Doran (@bobtfish) of Yelp at Puppet Camp London 2014. Find the video here: http://puppetlabs.com/community/puppet-camp
46
Transcript
Page 1: Taking Control of Chaos with Docker and Puppet
Page 2: Taking Control of Chaos with Docker and Puppet

Taking control of chaos

(with Docker and puppet) - Tomas Doran

@bobtfish

18/11/2013

Page 3: Taking Control of Chaos with Docker and Puppet

• Isolation – Run each daemon in it’s own container

• Security – Containers are fully independent

– Build a new container for a new application version!

• Ease of development – Same containers in Vagrant on laptop as in production!

– Build pipeline for whole application environment

• Ease of deployment – Deploy from your own private registry

– Roll back by just deploying the last version

Docker - why should you care?

Page 4: Taking Control of Chaos with Docker and Puppet

• My personal server

• Lots of responsibilities – Postfix – Dovecot – DNS – Irc + tmux – tor – A zillion web apps (ruby, perl, php, python,

nodejs!)

• 0 time spent on it

Taking control of chaos

Page 5: Taking Control of Chaos with Docker and Puppet

My personal server

Page 6: Taking Control of Chaos with Docker and Puppet

Several years later….

$864

Label for your image (if needed)

Page 7: Taking Control of Chaos with Docker and Puppet

Reality

$864

Label for your image (if needed)

Page 8: Taking Control of Chaos with Docker and Puppet

This year’s model

$864

Label for your image (if needed)

Page 9: Taking Control of Chaos with Docker and Puppet

Half the answer

$864

Page 10: Taking Control of Chaos with Docker and Puppet

• Exactly one run – Should be building machines clean every time

– Doing exactly one puppet run

• Always rebuild – Unless you rebuild regularly

– You don’t know you can rebuild

• Immutable instances – Never change config on a server, replace instance!

• Hard if you only have 1 server!

Convergence and immutability

Page 11: Taking Control of Chaos with Docker and Puppet

The other half

$864

Page 12: Taking Control of Chaos with Docker and Puppet

$864

Compartmentalization

Page 13: Taking Control of Chaos with Docker and Puppet

Every service?

$864

Page 14: Taking Control of Chaos with Docker and Puppet

Yes

$864

Postfix

Page 15: Taking Control of Chaos with Docker and Puppet

Really

$864

DovecotPostfix

Page 16: Taking Control of Chaos with Docker and Puppet

Every

$864

DovecotPostfix Postgrey

Page 17: Taking Control of Chaos with Docker and Puppet

Single

$864

DovecotPostfix Postgrey Spamassasin

Page 18: Taking Control of Chaos with Docker and Puppet

Service

$864

DovecotPostfix Postgrey Spamassasin Mysql

Page 19: Taking Control of Chaos with Docker and Puppet

Independently!

$864

DovecotPostfix Postgrey Spamassasin Mysql irssi

Page 20: Taking Control of Chaos with Docker and Puppet

For real.

$864

DovecotPostfix Postgrey Spamassasin Mysql irssi playground

(sshd)

Page 21: Taking Control of Chaos with Docker and Puppet

• Build puppet code for installing service on the old server

• Run same puppet code inside a container to install packages / build config

• Add shims to start service inside container

A cunning plan

Page 22: Taking Control of Chaos with Docker and Puppet

• All mutable data is an lvm volume mounted from the host

• All lvm volumes also get bind mounted read only

• Share unix domain sockets this way • Server for socket creates • Clients mount ro version

Data management

Page 23: Taking Control of Chaos with Docker and Puppet

Compartmentalization

$864

Page 24: Taking Control of Chaos with Docker and Puppet

Containers and volumes

Page 25: Taking Control of Chaos with Docker and Puppet

What’s inside a container?

supervisord: • The ‘real’ process • mcollective • sshd

Page 26: Taking Control of Chaos with Docker and Puppet

• profile::dovecot – All the things needed to run dovecot

– Parameterizeable as needed for two different deploy environments.

• container::dovecot – Main entry point when building the container

– Delegates most of real setup to profile::dovecot – Adds all the container specific overrides

– Adds supervisor service(s) for this container

• run_container::dovecot – Wraps docker::run {} – Manages the associated lvm volumes

– Adds firewall rules

Code structure

Page 27: Taking Control of Chaos with Docker and Puppet

!– All the things needed to run dovecot

– Parameterizeable as needed for two different deploy environments.

profile::dovecot

Page 28: Taking Control of Chaos with Docker and Puppet

!– Main entry point when building the container

– Delegates most of real setup to profile::dovecot – Adds all the container specific overrides

– Adds supervisor service(s) for this container

container::dovecot

Page 29: Taking Control of Chaos with Docker and Puppet

!– Wraps docker::run {} – Manages the associated lvm volumes

– Adds firewall rules

run_container::dovecot

Page 30: Taking Control of Chaos with Docker and Puppet

• profile::docker::build_container – define - writes out Dockerfile + support files

– Runs docker build .

– Manages dependencies so base containers get built first

• profile::docker::with_socket::mysql – /socket/mysql

– Symlink into /var/lib/mysql

• profile::docker::with_supervisord – Setup supervisor with default (mcollective + ssh) tasks

– Default /start script to invoke supervisord

Building containers

Page 31: Taking Control of Chaos with Docker and Puppet

!– define - writes out Dockerfile + support files

– Runs docker build .

– Manages dependencies so base containers get built first

profile::docker::build_container

Page 32: Taking Control of Chaos with Docker and Puppet

!– define - writes out Dockerfile + support files

– Runs docker build .

– Manages dependencies so base containers get built first

profile::docker::build_container

Page 33: Taking Control of Chaos with Docker and Puppet

!– define - writes out Dockerfile + support files

– Runs docker build .

– Manages dependencies so base containers get built first

profile::docker::build_container

Page 34: Taking Control of Chaos with Docker and Puppet

– /socket/mysql

– Symlink into /var/lib/mysql

container::with_socket::mysql

Page 35: Taking Control of Chaos with Docker and Puppet

!– Setup supervisor with default (mcollective + ssh) tasks

– Default /start script to invoke supervisord

container::with_supervisord

Page 36: Taking Control of Chaos with Docker and Puppet

• Drop facts – /etc/facter/facts.d/is_container.txt

– /etc/facter/facts.d/container_name

• Copy in code – ADD support/puppet /etc/puppet

• Run puppet – Masterless

– No real manifest, just:

if $::is_container {..

The Dockerfile

Page 37: Taking Control of Chaos with Docker and Puppet

– /etc/facter/facts.d/is_container.txt

!!!!

Drop facts

Page 38: Taking Control of Chaos with Docker and Puppet

– /etc/facter/facts.d/is_container.txt

!!!!!!

– /etc/facter/facts.d/container_name.txt

Drop facts

Page 39: Taking Control of Chaos with Docker and Puppet

– ADD support/puppet /etc/puppet

– RUN bundle exec rake puppet

Puppet code

Page 40: Taking Control of Chaos with Docker and Puppet

– Masterless – No node manifest, just:

if $::is_container {..

site.pp

Page 41: Taking Control of Chaos with Docker and Puppet

• Docker is kinda buggy – Just went 1.0, being fixed fast! – No sane exit status to docker build

– AUFS 42 layer limit

• Forge modules + packages assume service management

– No upstart inside containers - fails everywhere

• Debian packages are inconsistent – Lots of packages don’t use invoke-rc.d

Issues

Page 42: Taking Control of Chaos with Docker and Puppet

It’s still awesome

$864

Page 43: Taking Control of Chaos with Docker and Puppet

• HAProxy all the things! – Currently just bind containers to local ports

– Container replacement is not invisible

– Run haproxy on the real host

– Dynamically regenerate its config based on running containers

• Registry – Build containers in Vagrant, push up to prod

• PAAS web stuff – www all still lives on old server :(

– Gonna fix this real soon

Loads of TODOs

Page 44: Taking Control of Chaos with Docker and Puppet

• Open source all the things! – garethr++

– (I owe you patches)

– Forked and changed a million modules

– Will cleanup and upstream some changes

• My profile::docker code – Don’t know how to make this really generically

reusable

– We need sub-modules?

– Happy to share chunks

Open sores!

Page 45: Taking Control of Chaos with Docker and Puppet

Sorry!

Page 46: Taking Control of Chaos with Docker and Puppet

Questions?

$864

http://www.yelp.com/careers?jvi=ogVTXfwL

http://containercats.tumblr.com/


Recommended