The future of System The future of System AdministrationAdministration

Kris Buytaert

Apache CloudStack Collaboration Summit

Amsterdam 2013

@krisbuytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops

Why Me ?Why Me ?

Why Me ?Why Me ?

The future is here, The future is here,

it's just not evenly distributedit's just not evenly distributed

yetyet

Why we study history ?Why we study history ?● Because Because I`m a grumpy old frustrated sysadminI`m a grumpy old frustrated sysadmin

● Because IBecause I`m an old opiniated guy`m an old opiniated guy

● Because history repeatsBecause history repeats

● We need to learn from our mistakesWe need to learn from our mistakes

What has changedWhat has changed● Software was Software was

delivered on CD's, delivered on CD's, then DVD's, then DVD's,

● Then online , Then online , periodicallyperiodically

● Then internally , more Then internally , more frequentlyfrequently

● ContinuouslyContinuously

Deploying an InfrastructureDeploying an Infrastructure

● 1996 : Manual Installations1996 : Manual Installations

● 2001 : Mondo rescue2001 : Mondo rescue

= Reproducable single instances= Reproducable single instances

Deploying an InfrastructureDeploying an Infrastructure

● 1996 : Manual Installations1996 : Manual Installations

● 2001 : Mondo rescue2001 : Mondo rescue

● 2003 : SystemImager2003 : SystemImager

• Reproducable Infrastructure , with Reproducable Infrastructure , with “OVERRIDES”“OVERRIDES”

• Fast Multicast Image deployments\Fast Multicast Image deployments\

• Image Sprawl (thank you VMware)Image Sprawl (thank you VMware)

Deploying an InfrastructureDeploying an Infrastructure

● 1996 : Manual Installations1996 : Manual Installations

● 2001 : Mondo rescue2001 : Mondo rescue

● 2003 : SystemImager2003 : SystemImager

● 2005 : Dreaming of Jeos + IAC (Cfengine)2005 : Dreaming of Jeos + IAC (Cfengine)

• Kickstart / FAI Kickstart / FAI

Deploying an InfrastructureDeploying an Infrastructure

● 1996 : Manual Installations1996 : Manual Installations

● 2001 : Mondo rescue2001 : Mondo rescue

● 2003 : SystemImager2003 : SystemImager

● 2005 : Dreaming of Jeos + IAC (CFengine)2005 : Dreaming of Jeos + IAC (CFengine)

● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC

● 2010 : Vagrant for development 2010 : Vagrant for development

What has changedWhat has changed● End users End users

● 1 user using their own PC,1 user using their own PC,

● 500 users using the intranet application500 users using the intranet application

● 500000 users the cloud / saas 500000 users the cloud / saas (.eu)(.eu)

Physical to VirtualPhysical to Virtual

WARNING : Adoption vs IntroductionWARNING : Adoption vs Introduction

● Up till 2000 , Virtualization was a desktop Up till 2000 , Virtualization was a desktop thing, running emulation on your desktop (Plex, thing, running emulation on your desktop (Plex, BochsBochs

● Early 2000's Paravirtualization , Xen starts and Early 2000's Paravirtualization , Xen starts and gains adoptiongains adoption

● August 2006 : Amazon announced EC2 August 2006 : Amazon announced EC2

● 2009 : US started adopting EC2, Europe never 2009 : US started adopting EC2, Europe never really didreally did

● 2012 – now : Private clouds2012 – now : Private clouds

Virtualization should be CheapVirtualization should be Cheap

● 1 host with way too much services1 host with way too much services

• No isolation, dependency hell , management No isolation, dependency hell , management hellhell

● 1 host multiple vm's , same CAPEX1 host multiple vm's , same CAPEX

• VM's are cheap !VM's are cheap !

● Public clouds => multiple vm's , growing OPEXPublic clouds => multiple vm's , growing OPEX

● Private clouds => multiple vm's same CAPEX Private clouds => multiple vm's same CAPEX

What has changedWhat has changed● End users working 9-5End users working 9-5

● Local users using your application during Local users using your application during daytimedaytime

● A global economy, no downtime an be A global economy, no downtime an be toleratedtolerated

High Availability & ScaleoutHigh Availability & Scaleout

● KISSKISS

● Clustered Storage is finally in it's 10'sClustered Storage is finally in it's 10's

● Most apps still don't get itMost apps still don't get it

• Stateless vs StatefullStateless vs Statefull

For years we've tolerated humans to to make For years we've tolerated humans to to make structural manual changes to the infrastructure structural manual changes to the infrastructure our critical applications are running on.our critical applications are running on.

Whilst at the same time demanding those critical Whilst at the same time demanding those critical applications to go trough rigid test scenarios.applications to go trough rigid test scenarios.

Who let this happen ?Who let this happen ?

Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code

● Development best practicesDevelopment best practices

• Model your infrastructureModel your infrastructure

• Version your cookbooks / manifestsVersion your cookbooks / manifests

• Test your cookbooks/ manifestsTest your cookbooks/ manifests

• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra

● Model your infrastructureModel your infrastructure

● A working service = automated ( Application Code + A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring )

● Think Puppet, Chef, Cfengine, ....Think Puppet, Chef, Cfengine, ....

Testdriven InfrastructureTestdriven Infrastructure

Jenkins Pipeline Jenkins Pipeline

A pipelineA pipeline● Checkout codeCheckout code

● SyntaxSyntax

● StyleStyle

● Code CoverageCode Coverage

● TestsTests

● BuildBuild

● More TestsMore Tests

● Package Package

● Upload to RepoUpload to Repo

● Deploy on TestDeploy on Test

● Check PuppetrunsCheck Puppetruns

● Check IcingaCheck Icinga

● Promote to UATPromote to UAT

Repository ManagementRepository Management

Continuous Delivery or Deployment Continuous Delivery or Deployment of Infrastructureof Infrastructure

OrchestrationOrchestration● Distributed ssh++Distributed ssh++

● Trigger actions on selected nodesTrigger actions on selected nodes

● Think : Mcollective, Ansible, Salt, RundeckThink : Mcollective, Ansible, Salt, Rundeck

● (& Discuss :))(& Discuss :))

Orchestration 2Orchestration 2ndnd gen gen● While ....While ....

● First install XFirst install X

● When it is ready configure YWhen it is ready configure Y

● Then notify ZThen notify Z

● Think : Noah , Zookeeper, Serf , JujuThink : Noah , Zookeeper, Serf , Juju

Juju Juju

Serf Serf Serf is a decentralized solution for service Serf is a decentralized solution for service discovery and orchestration that is lightweight, discovery and orchestration that is lightweight, highly available, and fault tolerant.highly available, and fault tolerant.

Gossip based, Detecting Failure, etc.Gossip based, Detecting Failure, etc.

By HashiCorpBy HashiCorp

Deploying an InfrastructureDeploying an Infrastructure

● 1996 : Manual Installations1996 : Manual Installations

● 2001 : Mondo rescue2001 : Mondo rescue

● 2003 : SystemImager2003 : SystemImager

● 2005 : Dreaming of Jeos + IAC (CFengine)2005 : Dreaming of Jeos + IAC (CFengine)

● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC

● 2010 : Vagrant 2010 : Vagrant

● 2014: Stateless , Immutable infrastructure ?2014: Stateless , Immutable infrastructure ?

Are Docker, Packer , next ?Are Docker, Packer , next ?

Stateless Stateless

ImmutableImmutable

#monitoringsucks -> #monitoringlove#monitoringsucks -> #monitoringlove

● 1996 : BigBrother1996 : BigBrother

● 2001: Netsaint => Nagios + OpenNMS2001: Netsaint => Nagios + OpenNMS

● 2009: Shootout => Nagios (& Forks) clear 2009: Shootout => Nagios (& Forks) clear winnerwinner

● 2011: @lusis #monitoringsucks2011: @lusis #monitoringsucks

● 2012: @ulfmansson #monitoringlove2012: @ulfmansson #monitoringlove

● AutodetectionAutodetection

● MultiplexingMultiplexing

● Trend ForecastingTrend Forecasting

I love CheckMKI love CheckMK

• Autodetection ?Autodetection ?

• Service,Service,

• FunctionalitiesFunctionalities

• eg. vhosts etceg. vhosts etc

• Single Source of TruthSingle Source of Truth

I hate CheckMKI hate CheckMK

Monitoring & MetricsMonitoring & Metrics• Oculus , Skyline, Riemann, Esper, Oculus , Skyline, Riemann, Esper,

• FlapJack (2nd incarnation)FlapJack (2nd incarnation)

• BPM & MonitoringBPM & Monitoring

• Creating Information out of this data Creating Information out of this data

• Big dataBig data

• Machine LearningMachine Learning

I`m scared of the future !I`m scared of the future !● JuniorsJuniors

● Lack of PassionLack of Passion

● Broken education Broken education

Our role tomorrow:Our role tomorrow:● Teacher (for both collegues and suppliers)Teacher (for both collegues and suppliers)

● StudentStudent

● Math ExpertMath Expert

● DeveloperDeveloper

● TroubleshooterTroubleshooter

● EvangelistEvangelist

● ShrinkShrink

● Gap BridgerGap Bridger

Work to be done..Work to be done..● #releasemanagement#releasemanagement

● #Backupsucks#Backupsucks

● #printingsucks#printingsucks

Every 18 months, automate yourselve out of your Every 18 months, automate yourselve out of your job job

Someone at Google, longtime agoSomeone at Google, longtime ago

CfgMgmtCamp.eu CfgMgmtCamp.eu

Gent Gent

3 & 4 February 20143 & 4 February 2014

Ansible, Chef, Puppet, Cfengine, Rudder, Ansible, Chef, Puppet, Cfengine, Rudder, Juju,Saltstack, Foreman & You ?Juju,Saltstack, Foreman & You ?

ContactContactKris Buytaert Kris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/bhttp://www.krisbuytaert.be/blog/log/http://www.inuits.be/http://www.inuits.be/

InuitsInuits

Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221