Docker  at  MediaflyBryan  Murphy

2014.03.12

1  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

Bryan  Murphy

Mediafly,  Inc.

Senior  Technical  Architect  (7  Years)

Specialize  in  back-­end  services,  video  processing,scaling  and  architecture.

Mobitrac,  Inc.

Senior  Developer  (2  Years)

Specialized  in  travelling  salesman  problem,  routing  algorithms,  and  visualizationfor  logistics  software.

RBC/Centura  Mortgage

Lead  Web  Developer  (5  years)

Created  and  maintained  online  customizable  loan  officer  hosting  platform  and  ratesearch  engine.

HCST,  Inc.  (ISP  in  Dayton,  OH)

Developer/Administrator  (3  years)

Web  and  systems  development,  server  and  network  administration.

2  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

What  do  we  do?

Content  Management  System

3  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

What  do  we  do?

Secure  Content  Delivery  Platform

4  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

What  do  we  do?

Document  Conversion

DOC,  PDF,  PPT,  XLS

Video  Processing

AVI,  F4M,  F4V,  M3U8,  MKV,  MOV,  MP4,  MXF,  WMV

Customizable  UI

Cross  device  HTML5+JavaScript  UI.

Secure  Delivery

Encryption,  DRM,  Real-­Time  Watermarking

5  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

What  makes  us  interesting?

Multi-­Device

Multi-­Tenant

Major  customers  are  deployed  to  their  own  isolated  environment.

Service  Oriented

Multiple  services  coordinated  using  messaging  and  http  APIs.

Distributed

Asynchronous  message  driven  achitecture  using  RabbitMQ.

6  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

A  Sampling  the  Technologies  Use

Programming  Languages

Bash,  C#,  Java,  Python,  Ruby,  PHP,  BrightScript,  Objective-­C

Platforms

Android,  Flash,  iOS,  JDK6,  JDK7,  Mono,  .NET,  Python,  Roku,  Ruby

Application  Servers

Flask,  IIS,  Tomcat,  Rails

Operating  Systems

Ubuntu  10.04  /  12.04  /  14.04,  CentOS,  Windows  2008,  Windows  2012

Software

Apache,  Nginx,  IIS,  Memcached,  RabbitMQ,  PostgreSQL,  Salt,  Puppet,  FFmpeg,SoX

And  more!

7  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

Some  NumbersMillions  of  API  requests/day

Thousands  of  cloud  servers  provisioned/day

Thousands  of  video  transcode  jobs/day

Thousands  of  document  processing  jobs/day

100's  of  Millions  of  files  stored  on  S3

Hundreds  of  Terabytes  of  online  video  storage

8  /  34

Mediafly

Who  am  I?

What  do  we  do?

What  makes  usinteresting?

Multitenant  Secure  Distributed  AsynchronousService  Oriented  Polyglot  Architecture  Under  Load

What  have  we  gotten  ourselves  into??

9  /  34

Enter  the  Docker

10  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Developer  Friendly

Simple  Syntax

Mediafly  Monitoring  Web  Site:

FROM mediafly-ubuntu-12.04-base-flask

ADD ./src/web.py /srv/web.py

EXPOSE 8888

WORKDIR /srv

CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG

Source  Control

Branch?  Merge?  No  problem!

$ hg commit -m "updated database cluster configuration" Dockerfile

Infrastructure  as  Code

RUN add-apt-repository ppa:nginx/development

RUN apt-get update

RUN apt-get install -y nginx-extras==1.1.19-1ubuntu0.6

11  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Iterate

Fast  Start  Time

$ time docker run -i -t ubuntu:12.04 date

real 0m0.177s

user 0m0.000s

sys 0m0.036s

Incremental  Builds

$ docker build -t elasticsearch .

...snip...

Step 9 : RUN chown -R root.root /opt/elasticsearch-1.0.0

---> Using cache

---> 8bb1225ea753

Successfully built 8bb1225ea753

real 0m0.132s

user 0m0.020s

sys 0m0.024s

Consistent  Environment  (Repeatability)

$ docker run -i -t ubuntu:12.04 touch /tmp/file.txt

$ echo $?

0

$ docker run -i -t ubuntu:12.04 test -f /tmp/file.txt

$ echo $?

1

Faster  Development

Don't  waste  time  waiting!

12  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Testing

Test  Your  Code  in  Context

$ docker run -i -t appserver python unittests.py

$ echo $?

0

Test  Your  Infrastructure

$ docker run -i -t appserver test -e /usr/sbin/nginx

$ echo $?

0

Test  Your  Configuration

$ docker run -i -t appserver grep -q "root /var/lib/appserver;" \

/etc/nginx/conf.d/appserver.conf

$ echo $?

0

Deploy  What  Was  Tested

$ export VERSION=r2014.03.12b01

$ docker build -t appserver . \

&& docker run -i -t appserver python unittests.py \

&& docker run -i -t appserver test -e /usr/sbin/nginx \

&& docker run -i -t appserver grep -q "root /var/lib/appserver;" \

/etc/nginx/conf.d/appserver.conf \

&& docker tag appserver registry:443/appserver:$VERSION \

&& docker push registry:443/appserver:$VERSION

$ echo $?

0

13  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Dependency  Management

Encapsulation

Everything  you  need  in  a  single  package.

FROM ubuntu:12.04

RUN apt-get update

RUN apt-get install libapache2-mod-php5 mysql-server supervisor

ADD ./conf /etc/supervisor/conf.d

CMD supervisord -n

$ docker build -t lampstack .

$ docker run -d lampstack

14698a1366c

$ lxc-attach -n 14698a1366c9 /bin/ps -- x

PID TTY STAT TIME COMMAND

1 ? S 0:00 /bin/sh -c supervisord -n

7 ? S 0:00 /usr/bin/python /usr/bin/supervisord -n

8 ? S 0:00 /bin/sh /usr/sbin/apachectl -k start -X

14 ? S 0:00 /usr/bin/mysqld

38 ? R+ 0:00 /bin/ps -x

Mutual  Exclusion

Run  multiple  versions  of  the  same  program.

$ lsb_release -r

Release: 12.04

$ docker run -i -t s3cmd-v1.1.0-beta2 s3cmd --version

s3cmd version 1.1.0-beta2

$ docker run -i -t s3cmd-v1.5.0-alpha3 s3cmd --version

s3cmd version 1.5.0-alpha3

14  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Sharing

Jane  Doe's  Development  Environment

Dockerfile:

FROM ubuntu:12.04

RUN apt-get update

RUN apt-get install -y mono-complete

RUN apt-get install -y openjdk-6-jdk

RUN apt-get install -y python-flask

RUN apt-get install -y postgresql

RUN apt-get install -y rake

etc.

Sharing:

$ docker build -t registry:443/dev .

$ docker push registry:443/dev

or:

$ hg commit -m "updated dev environment" dev/Dockerfile

$ hg push

Jim  Smith's  Development  Environment

Build  and  Test  Application:

$ docker pull registry:443/dev

$ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake build

$ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake test

15  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Standardized

Apache

$ docker run -i -t -p 80 -p 443 apache

Nginx

$ docker run -i -t -p 80 -p 443 nginx

Postgres

$ docker run -i -t -p 5432 postgres

ElasticSearch  +  Logstash  +  Redis  +  Kibana

$ docker run -i -t -p 5544 -p 8080 -p 9200 elasticsearch

What  does  this  mean?

Standardized  execution  environment.

Standardized  port  mapping.

Standardized  volume  management.

Standardized  monitoring.

16  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Isolation

Isolate  Services

$ docker run -d -p 80 -p 443 nginx

f50c4854b7d8

$ lxc-attach -n f50c4854b7d8 /bin/ps -- ax

PID TTY STAT TIME COMMAND

1 ? S 0:00 nginx: master process nginx

7 ? S 0:00 nginx: worker process

8 ? S 0:00 nginx: worker process

9 ? S 0:00 nginx: worker process

10 ? S 0:00 nginx: worker process

21 ? R+ 0:00 /bin/ps ax

Minimize  Exposed  Surface  Area

mediafly-ubuntu-12.04-base 394mb

mediafly-ubuntu-12.04-base-extras 882mb

mediafly-ubuntu-12.04-base-extras-mono-2.10 1,074mb

mediafly-ubuntu-12.04-base-extras-mono-2.10-java6 1,369mb

mediafly-ubuntu-12.04-base-flask 403mb

mediafly-ubuntu-12.04-base-java6 1,172mb

mediafly-ubuntu-12.04-base-nginx 410mb

etc.

17  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Right  Sized  Deployments

Only  build  what  changed

$ docker history mediafly-monitor

IMAGE CREATED SIZE

397b4009c926 5 seconds ago 0 B

39afabb5721c 6 seconds ago 0 B

5f470d30018a 6 seconds ago 587.5 kB

fd05e65cd81d 12 days ago 0 B

0c9b69626be3 12 days ago 9.414 MB

d3d6f1d5b7b0 12 days ago 0 B

d29aa2f4f869 12 days ago 0 B

960ee306f74e 12 days ago 2.226 MB

c10f048ae9f7 12 days ago 33.14 MB

57f2e95ecead 12 days ago 213 B

3db83c8f02f4 12 days ago 232.3 MB

9e52e7b4a0ee 4 weeks ago 149 B

02930da590e1 4 weeks ago 0 B

72e10143e54a 6 weeks ago 125.9 MB

b74728ce6435 11 weeks ago 0 B

511136ea3c5a 8 months ago 0 B

Only  deploy  what  changed

397b4009c926 5 seconds ago 0 B

39afabb5721c 6 seconds ago 0 B

5f470d30018a 6 seconds ago 587.5 kB

18  /  34

Why?

DeveloperFriendly

Iterate

Testing

DependencyManagement

Sharing

Standardized

Isolation

FasterDeployments

InfrastructuralFreedom

Infrastructure  Freedom

What  host  can  your  service  run  on?

On  host  ip-­10-­244-­202-­48:

$ docker run -i -t postgres pg_dump

On  host  ip-­10-­204-­151-­29:

$ docker run -i -t postgres pg_dump

On  host  ip-­10-­2-­1-­15:

$ docker run -i -t postgres pg_dump

What  host  does  your  service  run  on?

Who  cares!

19  /  34

20  /  34

How?

Standardize

Describe

Deploy

Discover

Standardize

Everything  is  a  Docker  Container  (except  Windows):

$ docker build -t mediafly-cms .

$ docker build -t mediafly-jenkins-master .

$ docker build -t mediafly-jenkins-worker .

$ docker build -t mediafly-monitor .

$ docker build -t mediafly-postgres .

$ docker build -t mediafly-rabbitmq .

$ docker build -t mediafly-servicelocator .

$ docker build -t mediafly-transcoder .

etc.

Always  Build  All  Dependent  Containers

$ docker pull ubuntu:12.04

$ docker build -t mediafly-ubuntu-12.04-base

$ docker build -t mediafly-ubuntu-12.04-base-flask

$ docker build -t mediafly-monitor

Always  EXPOSE  Ports

EXPOSE 8888

Always  Specify  Default  CMD

CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG

Prefer  One  Service/Container

21  /  34

How?

Standardize

Describe

Deploy

Discover

Describe

Create  Container  Descriptor

$ cat <<EOF > appcluster01/containers/mediafly-elasticsearch.json

{

"tag": "mediafly-elasticsearch",

"version": "r2014.03.12b01",

"url": "containers/mediafly-elasticsearch:r2014.03.12b01.json",

"enabled": true,

"ports": [

{

"host": 9200,

"container": 9200

}, {

"host": 9300,

"container": 9300

}

],

"volumes": [

{

"host": "/mnt/elasticsearch",

"container": "/mnt/elasticsearch"

}

]

}

EOF

Why?

Provide  additional  information  about  container  such  as  where  to  deploy  it,monitoring  requirements,  volume  and  port  mappings,  etc.

22  /  34

How?

Standardize

Describe

Deploy

Discover

Deploy

Deploy  Images  to  S3

$ container-export --image mediafly-elasticsearch:r2014.03.12b01

23  /  34

How?

Standardize

Describe

Deploy

Discover

Deploy

Example  Container  Manifest

$ cat mediafly-elasticsearch:r2014.03.12b01.json

{

"id": "e8e2c810d9af",

"layers": [

{

"id": "511136ea3c5a",

"url": "s3://bucket/layers/511136ea3c5a.json"

},

{

"id": "6170bb7b0ad1",

"url": "s3://bucket/layers/6170bb7b0ad1.json"

},

{

"id": "9cd978db300e",

"url": "s3://bucket/layers/9cd978db300e.json"

}

etc.

]

}

Example  Layer  Manifest

$ cat 511136ea3c5a.json

{

"id": "511136ea3c5a",

"key": "s3://bucket/layers/511136ea3c5a.tar.xz.key.enc",

"url": "s3://bucket/layers/511136ea3c5a.tar.xz.enc"

}

24  /  34

How?

Standardize

Describe

Deploy

Discover

Discover

Register  All  Containers  with  Service  Locator

25  /  34

How?

Standardize

Describe

Deploy

Discover

Discover

Example  Service  Descriptor

$ cat mediafly-elasticsearch.json

{

"mediafly-elasticsearch": {

"9200": [

"ec2-12-34-56-2.compute-1.amazonaws.com:9200",

"ec2-12-34-56-2.compute-1.amazonaws.com:9200",

"ec2-12-34-56-3.compute-1.amazonaws.com:9200"

],

"9300": [

"ec2-12-34-56-2.compute-1.amazonaws.com:9300",

"ec2-12-34-56-2.compute-1.amazonaws.com:9300",

"ec2-12-34-56-3.compute-1.amazonaws.com:9300"

]

}

}

26  /  34

27  /  34

Lessons

LayerManagement

OperationsSupport

Areas  of  Friction

Docker  is  not  Git

Layer  Management

Container  Layering

Frequent  and  smaller  changes  at  the  top.

Infrequent  and  larger  changes  at  the  base.

28  /  34

Lessons

LayerManagement

OperationsSupport

Areas  of  Friction

Docker  is  not  Git

Layer  Management

Test  Caching  Behavior

Make  sure  all  of  your  layers  cache  properly.

Clean  Up  After  Yourself

Clean  up  your  apt  packages!  You  can  easily  double  the  size  (or  worse)  of  yourcontainers  if  you  don't  clean  up  after  yourself.

Verify  the  Size  of  Your  Layers

Use  the  docker history  command  to  see  the  size  of  your  layers.  Check  the  filesystem!

Don't  Build  Layers  Manually

Don't  build  containers  manually  (only  do  that  to  better  understand  how  Dockerworks).

Favor  Building  Images  over  Pulling  from  Registry

The  registry  is  great  for  sharing  and  deployment,  but  you  can  easilybranch/merge/diff  Dockerfiles.

29  /  34

Lessons

LayerManagement

OperationsSupport

Areas  of  Friction

Docker  is  not  Git

Operations  Support

Connect  to  a  running  container

$ lxc-attach -n CONTAINER /bin/bash

Access  a  container's  filesystem

$ cd /var/lib/docker/aufs/mnt/CONTAINER

Access  a  containers  log

$ tail -f /var/lib/dockers/containers/CONTAINER/CONTAINER-json.log

Clean  up  old  containers

$ docker ps -a | grep Exit | awk '{print $1}' | xargs docker rm

Clean  up  old  images

$ docker images | grep IMAGENAME | awk '{print $3}' | xargs docker rmi

Stop  all  running  containers

$ docker ps -q | xargs docker stop -t 0

30  /  34

Lessons

LayerManagement

OperationsSupport

Areas  of  Friction

Docker  is  not  Git

Areas  of  Friction  (as  of  03/2014)I  fully  expect  most  of  these  issue  to  be  addressed  in  future  versions.

Docker  Registry

We  chose  not  to  use  the  registry  because  of  the  following:

buggy,  slow,  scaling  bottleneckadditional  point  of  failureincreases  support  burden  and  infrastructure  costshard  to  secure

Supplementary  Services

There's  no  standard  for  handling  supplementary  services  such  as  logrotate,sendmail,  and  syslog.

Init  Systems

Integration  with  init  systems  such  as  upstart  or  systemd  is  a  work  in  progress.

Creating  Containers

You  can't  create  a  container  (docker run)  without  running  it.

Volume  Management

Stateful  data  is  your  problem.  There  is  no  standardized  solution  for  permanent  diskstorage,  backups,  and  replication.

31  /  34

Lessons

LayerManagement

OperationsSupport

Areas  of  Friction

Docker  is  not  Git

Docker  is  not  GitDocker  is  not  Git,  but  you  can  do  some  Git-­like  things  with  it.  Here's  an  alternateanalogy  to  help  you  understand  it.

What  is  Docker  like?

It's  like  an  incremental  build  system  with  shared  network  storage.

What's  the  souce  code?

Dockerfile's  are  source  code.

What  are  compiled  artifacts?

Layers  are  like  binaries  compiled  from  the  source  code  (e.g.  jars).

What  are  compiled  applications?

Images  are  like  compiled  applications  (.e.g  like  executables).

Installed  Executables

Containers  are  installed  executables  (running  or  stopped).

Shared  Network  Storage

The  registry  is  shared  network  storage  for  binaries  (layers  and  images).

32  /  34

Any  Questions?

33  /  34

twitter.com/bryanmurphy

bmurphy@mediafly.com

34  /  34