1 Orange Restricted

“DEVOPS-DDAY”

Docker@DFY : REX

Marseille, November 20, 2015

2 Orange Restricted

DISCLAIMER

If you want to learn Docker or important information about it

If you want to watch sexy girls/boys

If you expect a technology scoop

If you think you’ll learn magical recipes to do whatever you wish

3 Orange Restricted

Docker@DFY : Synopsis Who (and where) are we ?

– just a bunch of IT guys

Why Docker ?– context– reasonable choice ?

The bet we took : we need a significant project to go to production – our production – what’s at stake– challenge accepted !

Implementation choices : get closer of standards we’re aiming to reach– principles– advantages to be carefully conserved – examples

What we’ve learned– a lot of work– ROI is here as long as you dedicated means to reach it

What’s next : bets we’re taking– Docker cluster– Mesos cluster

4 Orange Restricted

Who are we Orange/OF/DTSI/DSI/DFY/HBX/GOT/NOH

– Not long ago in a galaxy far, far away …

4 guys, geeks and unconventional– Including the manager– Technologies are tools we love to use !

Motivated– Strong opinions, beliefs, maybe faith …

Thinking IT is more than machines and software– Humans ?

5 Orange Restricted

Docker@DFY : Why Docker Old school Technology mastery

– Permanent craftmanship– Losing value : bad TTM, expensive maintenance– Pet vs Cattle : very poor efficiency of our hardware capex

Exceptional geographic and organizational position– We have think/build AND run under the same hierarchical roof– Why don’t we rethink the way we address our hosting services

So why don’t we think about fancies technologies : [I|P]aaS– Still a cloudware in the house : changing from Cloudstack to Openstack …– We’re the production ! Difficult to push to PaaS without a global commitment– And Ubuntu LTS 2014 has been released … With Docker in it !

Docker was obviously (already) offering new values– A good complement with our virtual and bare metal offers– Easy to set up, learn and use – Fancy technology : good argument for geeks

Easy marketing : a lot of buzzwords to sell to our teams (and management)– Same components from dev to prod– Scalability, elasticity … Cloud ?– 12 factors app, infra as a code … microservices – Agile, Devops– Hardware Capex efficiency improvements, separation of concerns …– And so on …

6 Orange Restricted

We manage We operate We garantee QoS

We are here to make technology operable

Not (only) to play with it

But we’re the production team !

So …

7 Orange Restrictedinterne Orange7

OBJECTIVES PRODUCTION

1. Find a partner • A development team crazy

enough• And a significant project to

to production with it2. Convince (real) OPS

• To accept this challenge (and risks)

3. Convince management we’ll do the job …

To do list

8 Orange Restricted

What’s at stake ? Project : Advise as a service

– A breakthrough (for us)– High visibility : just what we needeed

Delay : 4 month– So what ?

Code refactoring : make it fit into containers Collect “OPS” constraints

– And make them as light as possible– Still give them confidence …

Hosting and system operations : what has to be done (@minimum)– On what infra ?– Logs– Monitoring and supervision : Inside and outside containers– Deployment, orchestration … of containers and software pieces in it– Build chain ?

What’s technically at stake ?

9 Orange Restricted

The contract

Alerting in legacy supervision system Reuse what already exists

– When possible …– If it respects the technology philosophy

Operating components:– logs management– Monitoring– Supervision– minimal orchestration

4 MONTHS …

10 Orange Restricted

Lightweight container

Containers Life cycle

Automating Docker image build

Security

Application configuration

Easy deployment

Metrics

Alerting in legacy supervision system (yes it’s important)

Reuse what exists

Logs: only files

Containerize everything !

Implementation choices

11 Orange Restrictedinterne Orange11

Reminder : Basic Docker Vocabulary

Dockerfile: text file, composed of various commands (instructions) to build an image

Image: an image, is a set of layers as described in a Dockerfile

Container: an instance of an image is called container

Registry: server that stores and lets you distribute Docker images

12 Orange Restricted

Ligthweight container

1 container 1 service

Easy to scale (up and down)

Only scale what needs to be scale

Only build / rebuild what is needed

Fast and safe deployment

Buzz word: Micro services

13 Orange Restricted

Life cycle

14 Orange Restricted

Automating Docker image build

Build docker image from Dockerfile

Extract the list of installed packages

Verify installed packages

Checkpoint for images before being pushed in registries

Use Jenkins

15 Orange Restrictedinterne Orange15

Automating Docker image build / SecuritySCM

(Gforge, Orange Forge, Gitorious,

etc)

Security Preriquisitesatisfied

1931 - 2015

Files to build(Dockerfile &

co)

Image

Build

Data

Extractions

Security

Check

If security preriquisiteAre not satisfied…

Repository

16 Orange Restricted

Configuration

Staging configuration:– outside the container– or use environment variables

With configuration outside the container

– you deploy the same container– in different environments

You can generate / deploy configuration with chef, ansible, puppet .. whatever

17 Orange Restrictedinterne Orange17

Deployment Maestro-ng

– immediate availability – effectiveness – simplicity of implementation– manual launch

Description of deployment in a yaml file: – ships : target machines – registries : registries used to retrieve images – services: services to deploy ( set of instances )– …

Healthcheck

Deployment order : dependencies managed

Deploy all your containers in an command line(in a static way)

18 Orange Restrictedinterne Orange18

Application metrics

Fluentd:– data collector written in Ruby, easy to patch

Heka:– high performance data gathering, analysis, and routing, with a lua sandbox, written in Go

InfluxDB– time series and metrics database, written in Go

Riemann:– streams events processor, written in … Clojure

Grafana– Gorgeous metric viz, and dashboards

Metrics collected with Fluentd sent to Heka and processed by Riemann, stored in InfluxDB and displayed in Grafana

19 Orange Restricted

Application metrics

20 Orange Restricted

Docker metrics – same player shoots again

Read docker metrics on disk (file) :– Cpu– Ram– I/O …

Metrics collected with Fluentd sent to Heka, processed by Riemann and stored in InfluxDB …

21 Orange Restricted

Docker metrics

22 Orange Restricted

Metrics

23 Orange Restrictedinterne Orange23

Containerize EVERYTHING

For 1 application in a container, you have:

– 1 docker to collect Docker metrics

– 1 docker to collect Application logs

– 1 docker Heka to route logs/events

– 1 docker Riemann to process events

– 1 docker Riemann dash for infra alerting

– 1 docker influxdb to store metrics

– 1 docker grafana to display metrics

24 Orange Restricted

Tools … again … we need tools … and we don’t have them all

Inventory tools :– Where are my containers (discovey)– On what machine (at what time)– How many (and what kind of) containers by host– What kind of containers

Inspection tools :– What network interface attached to which

container Graphical tools

– Registry– Provisionning …

Reporting, dashboarding, convincing, fu…ng … tools

25 Orange Restrictedinterne Orange25

We started from nothing– We wanted to !

We do pets not cattle– Our production system

granularity : the machineAnd clearly organization is not helping us out In 4 month all has been rebuilt :– Logs : collection and routing,

graphing– Monitoring and supervision

(threshold)– Orchestration

what we’re doing has Because

26 Orange Restrictedinterne Orange26

Technology break … or breakthrough ?

Finally… we didn’t go to production in 4 month– It wasn’t that easy for both IT guys (all of them not just us) and developpers– We were ready though … but other problems prevented development team to deliver– We’re close now from production : but final tests are not a piece of cake !

But we did it in 4 month– Cleary still wet (and not drying)– Still changing (a lot) and already too old– But it works

We proved Docker can let us go far, very far– Lego building blocks– Pop infrastructure as a code– CI to CD’s

Revolution is on his way* … at least it begins to move a bit at Orange*But it’s slow and it takes time

27 Orange Restrictedinterne Orange27

The famous …. REX

28 Orange Restrictedinterne Orange28

Lessons learned … if any What is important

Techno is a mean not an end

it ain’t over till it’s over :

Using well buzzwords I don’t appreciate so much

1. The team : all is about giving sense2. The team : all is about pride

3. The team : all is about fun and interest

– a game you play … to win … money or time (which is the same)– techno goes fast … don’t try to go as fast (linearly) … take steps (and

breathe slowly)

– never stop communicate … get your chakras opened always

– be agile and devops … mouarf– it could be a long story …

– all has to be integrated : loose coupling is a dream not easy to catch

– and discussions never stop …

29 Orange Restricted

DEVOPS – BUILD & RUN – DESIGN & PRODUCTION– isn’t it only building walls ??? underlining this bloody dichotomy we want to avoid

DEVOPS : what [I|we] learned – are small pizza teams product and value oriented : best definition of devops ever : when developers got the pager in

their pocket– run what you build : be responsible, be aware of others constraints, be empathic !– none is trained to be a devops tech … you become one … it’s a mindset before competencies– still you need skill … a lot of skill : especially with cloud and disruptive technologies : multidisciplinary– I’m bored of hearing devops … so bored … like agile or cloud … I love precise words describing clear concepts …

Doing Devops … is nearly completely impossible (at least for the moment)– Silos ? You know silos ? Organization too ?– And management ? I’m sure you do know management ! And have an idea about it …

Same for ‘Agile’– We did what we did with officially no machines– Three different projects to (not) achieve it (yet)

Devops Dday … DEVOPS ! DEVOPS ? DEVOPS …what about that, Rex

30 Orange Restrictedinterne Orange30

AGILE

SHA

RIN

G

My two cents vision about it

1AGILE / LEAN

2DEVOPS

3CLOUD

1CULTURE

2ORGA

3AUTOMATION

4MEASURES

31 Orange Restricted

We’re not thereTrust and confidence are lackingBut we’ve got all to achieve it …And I believe … (For sure I can’t fly …)

Once upon a time in a far far away …

32 Orange Restrictedinterne Orange32

33 Orange Restrictedinterne Orange33

Keep cool buddy

We need it a lot– Pressure on TTM,– Budgets …– with reliability …

We’re still looking for have chosen solutions

– CloudFoundry and Mesos – With what organization and

governance ? Docker seems to be « central »

– Automated, orchestrated– A nice build chain– With a managed configuration

And good urbanised repositories– Release management is a key

feature

34 Orange Restrictedinterne Orange34

We need a « tour de PaaS PaaS »

IaaS

PaaS

CaaS

CFY

Cloudstack

Not c

ompu

lsory

SwarmCompose

Mesos DUCP

Bare Metal aaS

35 Orange Restrictedinterne Orange35

Quiete a lot of work to do

• MVP approach• Nothing like a full automated PaaS as CFY• Still a lot of difficulties

• Technical … Network … multi tenant … security …• Most important : every one has to be transformed … at once …

• Does not seem right to me

• But, still alive and kicking : having faith and hope (a lot will be needed)

36 Orange Restrictedinterne Orange36

37 Orange Restricted

Thank you