@bridgetkromhout

Docker in Production Reality, Not Hype

Bridget Kromhout

@bridgetkromhout

Bridget Kromhout

Operations Engineer @DramaFeverMinneapolis, Minnesota

@devopsdays@devopsdaysMSP

@arresteddevops

bridgetkromhout.com

@bridgetkromhout

K-dramas since 2009. Docker in prod since October 2013.

@bridgetkromhout

Why Docker?

consistent development repeatable deployment

How?

not: a tutorial

but: repeatable

@bridgetkromhout

private registry: the official party line

https://github.com/docker/docker-registry#quick-start

S3 is a storage engine optionbut… a central registry server didn’t scale well for us

@bridgetkromhout

private registry: dramafever

@bridgetkromhout

# this goes in /etc/default/docker to control

docker's upstart config

DOCKER_OPTS="--graph=/mnt/docker --insecure-

registry=local-repo-alias.com:5000"

● local-repo-alias.com in DNS with A record to 127.0.0.1● OS X /etc/hosts: use the boot2docker host-only network IP

registry upstartdocker pull public_registry_image

docker run -p 5000:5000 --name registry \

-v /etc/docker-reg:/registry-conf \

-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml \

public_registry_image

@bridgetkromhout

config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true

@bridgetkromhout

what even is flate?!Pulling repository local-repo-alias.com:5000/www4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671d497ad3926c8: Error downloading dependent layers2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671

@bridgetkromhout

registry rewrite coming!

DOCKER_OPTS="--graph=/mnt/docker --insecure-registry=local-repo-alias.com:5000 -e STORAGE_REDIRECT=true"

...until we get to the promised go lan(d|g), there’s a workaround for the flate errors we’re seeing:

@bridgetkromhout

Achievement unlocked:distributed privateDocker registry

@bridgetkromhout

@bridgetkromhout

Next up:build pipeline

starringeveryone’s

favorite butler

weekly base builds

FROM local-repo-alias.com:5000/www-base

● include infrequently-changing dependencies○ ubuntu packages○ pip requirements○ wheels

● other builds can start from these images (so they’re faster):

@bridgetkromhout

sudo docker build -t="a12fbdc" .

sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 --

name test.a12fbdc a12fbdc py.test -s

sudo docker tag a12fbdc local-repo-alias.com:

5000/www:'dev'

sudo docker push local-repo-alias.com:5000/www:'dev'

@bridgetkromhout

www-master build

2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705-b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67-init: no such file or directorymake: *** [build-django] Error 1Build step 'Execute shell' marked build as failure

@bridgetkromhout

breaking builds

https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

@bridgetkromhout

@bridgetkromhout

Retry the build…...only if a specific regex appears

@bridgetkromhout

useful for unattended base builds

need to change how it reports to Slack

@bridgetkromhout

tag for staging

tag for prodout of ELBrestart upstartback in ELB

Ship it!

What about local development?

@bridgetkromhout

before summer 2014Vagrant for local development

chef-solo provisioner

17 minutes to install everything

@bridgetkromhout

now: boot2docker

devs pull down images built on jenkinsmysql image is built with fixturescan run master or qa image (or even prod)

can build new local images from Dockerfiles

@bridgetkromhout

local registry for devdocker run -d -p 5000:5000 --name

docker-reg -v ${DFHOME}:${DFHOME} -e

DOCKER_REGISTRY_CONFIG=${DFHOME}

/config/docker-registry/config.yml

public_registry_image

@bridgetkromhout

$ boot2docker ssh date -u

Mon Nov 24 16:09:02 UTC 2014

$ date -u

Tue Nov 25 01:43:49 UTC 2014

@bridgetkromhout

time is what turns kittens into cats

S3 requires clock sync$ docker pull local-repo-alias.com:5000/mysqlPulling repository local-repo-alias.com:5000/mysql2014/11/24 19:44:31 HTTP code: 500

$ boot2docker ssh sudo date --set \"$(env TZ=UTC date '+%F %H:%M:%S')\"

@bridgetkromhout

Devs can use their preferred editing environment:

-v ${DFHOME}/www:/var/www

We still want logs, too, so we expose those for the dev here:

-v ${DFHOME}/www/run:/var/log

volume mounting & our fork

@bridgetkromhout

Until 1.3 we ran a forked boot2dockerWe needed to mount local files into the VM

containerizing front-enduseful for building front-end apps on Jenkinsalso allows consistent testing

RUN apt-get install -y nodejs nodejs-legacy npmRUN npm install -g grunt-cli@0.1.13RUN npm install -g bower@1.3.8RUN npm install -g phantomjs@1.9.7-14ADD bower.json /var/www/dependencies/bower.jsonRUN cd /var/www/dependencies && bower install --allow-root --config.interactive=false --force

@bridgetkromhout

@bridgetkromhout

django: image: local-repo-alias.com:5000/www:dev ports: - "8000:8000" links: - mysql - redis environment: - PYTHONPATH=/var/local - DJANGO_ENVIRON=LOCAL - DB_PORT_3306_TCP_ADDR=mysql command: /var/local/config/local/start-django-local volumes: - ${DFHOME}/www/run:/var/log - ${DFHOME}/www:/var/local

mysql: image: local-repo-alias.com:5000/mysql:dev expose: - "3306:3306"

for persistent instances# remove stopped containers

@daily docker rm `docker ps -aq`

# remove images tagged "none"

@daily docker rmi `sudo docker images | grep none

| awk -F' +' '{print $3}'`

@bridgetkromhout

failure modes

cron zombiesout of memory errors

race conditions

@bridgetkromhout

what isolation?-v /var/log/containers:/var/log

@bridgetkromhout

Host instances moving into

through a container darkly: monitoring

@bridgetkromhout

containers building (lighter) containers

easier with statically linked binaries

go microservicesandroid apk

@bridgetkromhout

$ docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB

local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB

local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB

google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB

public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB

scratch latest 511136ea3c5a 18 months ago 0 B

$

ever-smaller images

@bridgetkromhout

@bridgetkromhout

www.dramafever.com/company/careers.html

Thank you!(and we’re hiring!)