The future of System The future of System AdministrationAdministration
Kris Buytaert
Apache CloudStack Collaboration Summit
Amsterdam 2013
@krisbuytaert
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops
Why Me ?Why Me ?
Why Me ?Why Me ?
The future is here, The future is here,
it's just not evenly distributedit's just not evenly distributed
yetyet
Why we study history ?Why we study history ?● Because Because I`m a grumpy old frustrated sysadminI`m a grumpy old frustrated sysadmin
● Because IBecause I`m an old opiniated guy`m an old opiniated guy
● Because history repeatsBecause history repeats
● We need to learn from our mistakesWe need to learn from our mistakes
What has changedWhat has changed● Software was Software was
delivered on CD's, delivered on CD's, then DVD's, then DVD's,
● Then online , Then online , periodicallyperiodically
● Then internally , more Then internally , more frequentlyfrequently
● ContinuouslyContinuously
Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
= Reproducable single instances= Reproducable single instances
Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
• Reproducable Infrastructure , with Reproducable Infrastructure , with “OVERRIDES”“OVERRIDES”
• Fast Multicast Image deployments\Fast Multicast Image deployments\
• Image Sprawl (thank you VMware)Image Sprawl (thank you VMware)
Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : Dreaming of Jeos + IAC (Cfengine)2005 : Dreaming of Jeos + IAC (Cfengine)
• Kickstart / FAI Kickstart / FAI
Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : Dreaming of Jeos + IAC (CFengine)2005 : Dreaming of Jeos + IAC (CFengine)
● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC
● 2010 : Vagrant for development 2010 : Vagrant for development
What has changedWhat has changed● End users End users
● 1 user using their own PC,1 user using their own PC,
● 500 users using the intranet application500 users using the intranet application
● 500000 users the cloud / saas 500000 users the cloud / saas (.eu)(.eu)
Physical to VirtualPhysical to Virtual
WARNING : Adoption vs IntroductionWARNING : Adoption vs Introduction
● Up till 2000 , Virtualization was a desktop Up till 2000 , Virtualization was a desktop thing, running emulation on your desktop (Plex, thing, running emulation on your desktop (Plex, BochsBochs
● Early 2000's Paravirtualization , Xen starts and Early 2000's Paravirtualization , Xen starts and gains adoptiongains adoption
● August 2006 : Amazon announced EC2 August 2006 : Amazon announced EC2
● 2009 : US started adopting EC2, Europe never 2009 : US started adopting EC2, Europe never really didreally did
● 2012 – now : Private clouds2012 – now : Private clouds
Virtualization should be CheapVirtualization should be Cheap
● 1 host with way too much services1 host with way too much services
• No isolation, dependency hell , management No isolation, dependency hell , management hellhell
● 1 host multiple vm's , same CAPEX1 host multiple vm's , same CAPEX
• VM's are cheap !VM's are cheap !
● Public clouds => multiple vm's , growing OPEXPublic clouds => multiple vm's , growing OPEX
● Private clouds => multiple vm's same CAPEX Private clouds => multiple vm's same CAPEX
What has changedWhat has changed● End users working 9-5End users working 9-5
● Local users using your application during Local users using your application during daytimedaytime
● A global economy, no downtime an be A global economy, no downtime an be toleratedtolerated
High Availability & ScaleoutHigh Availability & Scaleout
● KISSKISS
● Clustered Storage is finally in it's 10'sClustered Storage is finally in it's 10's
● Most apps still don't get itMost apps still don't get it
• Stateless vs StatefullStateless vs Statefull
For years we've tolerated humans to to make For years we've tolerated humans to to make structural manual changes to the infrastructure structural manual changes to the infrastructure our critical applications are running on.our critical applications are running on.
Whilst at the same time demanding those critical Whilst at the same time demanding those critical applications to go trough rigid test scenarios.applications to go trough rigid test scenarios.
Who let this happen ?Who let this happen ?
Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code
● Development best practicesDevelopment best practices
• Model your infrastructureModel your infrastructure
• Version your cookbooks / manifestsVersion your cookbooks / manifests
• Test your cookbooks/ manifestsTest your cookbooks/ manifests
• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Infrastructure Code + Security + Monitoring )
● Think Puppet, Chef, Cfengine, ....Think Puppet, Chef, Cfengine, ....
Testdriven InfrastructureTestdriven Infrastructure
Jenkins Pipeline Jenkins Pipeline
A pipelineA pipeline● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● Package Package
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● Check PuppetrunsCheck Puppetruns
● Check IcingaCheck Icinga
● Promote to UATPromote to UAT
Repository ManagementRepository Management
Continuous Delivery or Deployment Continuous Delivery or Deployment of Infrastructureof Infrastructure
OrchestrationOrchestration● Distributed ssh++Distributed ssh++
● Trigger actions on selected nodesTrigger actions on selected nodes
● Think : Mcollective, Ansible, Salt, RundeckThink : Mcollective, Ansible, Salt, Rundeck
● (& Discuss :))(& Discuss :))
Orchestration 2Orchestration 2ndnd gen gen● While ....While ....
● First install XFirst install X
● When it is ready configure YWhen it is ready configure Y
● Then notify ZThen notify Z
● Think : Noah , Zookeeper, Serf , JujuThink : Noah , Zookeeper, Serf , Juju
Juju Juju
Serf Serf Serf is a decentralized solution for service Serf is a decentralized solution for service discovery and orchestration that is lightweight, discovery and orchestration that is lightweight, highly available, and fault tolerant.highly available, and fault tolerant.
Gossip based, Detecting Failure, etc.Gossip based, Detecting Failure, etc.
By HashiCorpBy HashiCorp
Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : Dreaming of Jeos + IAC (CFengine)2005 : Dreaming of Jeos + IAC (CFengine)
● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC
● 2010 : Vagrant 2010 : Vagrant
● 2014: Stateless , Immutable infrastructure ?2014: Stateless , Immutable infrastructure ?
Are Docker, Packer , next ?Are Docker, Packer , next ?
Stateless Stateless
ImmutableImmutable
#monitoringsucks -> #monitoringlove#monitoringsucks -> #monitoringlove
● 1996 : BigBrother1996 : BigBrother
● 2001: Netsaint => Nagios + OpenNMS2001: Netsaint => Nagios + OpenNMS
● 2009: Shootout => Nagios (& Forks) clear 2009: Shootout => Nagios (& Forks) clear winnerwinner
● 2011: @lusis #monitoringsucks2011: @lusis #monitoringsucks
● 2012: @ulfmansson #monitoringlove2012: @ulfmansson #monitoringlove
● AutodetectionAutodetection
● MultiplexingMultiplexing
● Trend ForecastingTrend Forecasting
I love CheckMKI love CheckMK
• Autodetection ?Autodetection ?
• Service,Service,
• FunctionalitiesFunctionalities
• eg. vhosts etceg. vhosts etc
• Single Source of TruthSingle Source of Truth
I hate CheckMKI hate CheckMK
Monitoring & MetricsMonitoring & Metrics• Oculus , Skyline, Riemann, Esper, Oculus , Skyline, Riemann, Esper,
• FlapJack (2nd incarnation)FlapJack (2nd incarnation)
• BPM & MonitoringBPM & Monitoring
• Creating Information out of this data Creating Information out of this data
• Big dataBig data
• Machine LearningMachine Learning
I`m scared of the future !I`m scared of the future !● JuniorsJuniors
● Lack of PassionLack of Passion
● Broken education Broken education
Our role tomorrow:Our role tomorrow:● Teacher (for both collegues and suppliers)Teacher (for both collegues and suppliers)
● StudentStudent
● Math ExpertMath Expert
● DeveloperDeveloper
● TroubleshooterTroubleshooter
● EvangelistEvangelist
● ShrinkShrink
● Gap BridgerGap Bridger
Work to be done..Work to be done..● #releasemanagement#releasemanagement
● #Backupsucks#Backupsucks
● #printingsucks#printingsucks
Every 18 months, automate yourselve out of your Every 18 months, automate yourselve out of your job job
Someone at Google, longtime agoSomeone at Google, longtime ago
CfgMgmtCamp.eu CfgMgmtCamp.eu
Gent Gent
3 & 4 February 20143 & 4 February 2014
Ansible, Chef, Puppet, Cfengine, Rudder, Ansible, Chef, Puppet, Cfengine, Rudder, Juju,Saltstack, Foreman & You ?Juju,Saltstack, Foreman & You ?
ContactContactKris Buytaert Kris Buytaert [email protected]@inuits.be
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/bhttp://www.krisbuytaert.be/blog/log/http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221