CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT

SAM BASHTON, BASHTON LTD

CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT

MASTERLESS PUPPET: WHYAND HOW

SAM BASHTON, BASHTON LTD

ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies

OUR FULLY MANAGEDENVIRONMENTS

Primarily transactional websites (e-commerce)Majority (70%+) on Amazon Web Services (AWS)Majority using CentOS

HOW WE WORKSimple is better than complexComplexity is worth adding only if it provides obviousfunctional benefits

Re-usabilityResilience

WHY DID WE PICK AWS?Featureset and toolset massively in advance of anyother cloud provider, public or private#1 customer reason for switching to AWS? The ability toscale on demand

TOOLS WE USE FORBUILDING AND MANAGINGDo one thing and do it well

CloudFormation - Amazon tool to manageinfrastructurePuppet - Manage system configurationPulp - centralised repository, manages packagerevisionsJenkins

HOW WE USE PUPPETNo PuppetmasterPuppet manifests, hieradata and modules distributedto all machines via RPMAll machines boot with a common, blank image and getconfigured at first boot

WHAT'S WRONG WITHMASTER BASED PUPPET?

Pets vs Cattle

Puppet designed for a world of servers as petsWe do not live in that world

PUPPET DESIGNED FOR PETSMany assumptions in Puppet presume that yourservers are petsSome of these work against us when managing a herd

MANUAL CERTIFICATESIGNING

Clearly unsuitable when machines are automaticallyprovisioned

POTENTIAL WORKAROUNDS:AutosignUse/write another automated certificate generationmechanism

Possibly tied in with autoscaling

NO MECHANISM FORCLEANING OLD HOSTS

Likely to have host-names reused, causing machines tofail to configurePuppetmaster will fill with certificates for machines thatran for a few hours and went away again

POTENTIAL WORKAROUNDS:Use UUID certificatesAgree not to look in the certificate directoryWrite mechanism for cleaning up old certificates

HOSTS CONFIGURED BASEDON HOSTNAME

Our machines have names like ip-172-26-5-123How does Puppet know what type of machine this is?

POTENTIAL WORKAROUNDSUse an external node classifierUse some mechanism for giving a better hostname, egweb-172-26-5-123 and use regex for node names

PUPPETMASTER IS A SINGLEPOINT OF FAILURE

If the Puppetmaster fails, we can no longer autoscaleupIn particular, this could be a problem if there isavailability zone failure

POTENTIAL WORKAROUNDSClustered Puppetmasters

WORKAROUND RECAPUse/write alternative certificate management softwareWrite an external node classifier / mechanism forsetting hostname appropriatelyCluster multiple Puppetmasters

WHAT WE DID INSTEADDecided using a Puppetmaster was trying to fit a squarepeg into a round holeInstead, decided to run Puppet without a master

APPLYING LOCAL PUPPETMANIFESTS

puppet apply --modulepath=/etc/puppet/modules example.pp

DISTRIBUTING MANIFESTSUse RPMDistribute full set of manifests/modules to eachmachineApply only the manifest relevant to that machine

PACKING PUPPETMANIFESTS IN RPM

Build an RPM containing everything under /etc/puppetMake files readable only by root

APPLY PUPPET MANIFESTSHave an RPM %postinst command apply the Puppetconfig

This isn't as straightforward as running the puppetapply from %postinstPuppet needs to install packages via yum, but yum isrunning installing the Puppet packageInstead, we work around with a dirty hack: have the%postinst create an at script which checks if yumhas finished and then runs the puppet apply

RPM INSTALLATION ANDMANAGEMENT

How do we get these RPMs on our machines?

PULPWe were already using PulpProvides yum repository managementUsed for managing security updates and deployingapplication code

http://pulpproject.org/

WHAT IS PULPRepository managerAllows us to easily audit what packages and versionsare installed whereAllows us to push package installations

Uses qpid message queue

Has concept of 'content distrubtion servers' for easyreplication and clustering

HOW WE USE PULPPuppet contains details of what packages should beinstalledPulp manages which version of the package should beinstalledPulp allows us to clone repos and copy packagesbetween them for easy qa->stage->live environmentmanagement

DEPLOYINGCONFIGURATION AS CODEAllows us to reuse our existing code deploymentinfrastructureManage configuration deployment from Jenkins

HOW WE DEPLOY CODEEverything managed via the Jenkins continuousintegration serverJenkins uses Pulp to install code on remote machines

DETAILS ON HOW WEDEPLOY CODE

Jenkins fetches code from source control (git)An RPM is builtTests are runIf tests pass, the RPM is added to the relevant Pulprepository RPM installed on the target machine(s)

DEPLOYMENT LIFE-CYCLEJenkins also manages deployment life-cycleRPMs are installed on stagingPromoted Builds plugin then used to install the sameRPMs on live once testing is complete

PUPPET DEPLOYMENTPROCESS

Puppet manifests are checked into gitLint tests via Jenkins pulls in modules with librarian-puppet, thenbuilds an RPMDeployment to test environments, functional tests forwider code-base run

Jenkins Warnings plugin

PUTTING IT INTOPRODUCTION

Once suitable tests (automated and manual) have beencarried out, we promote Puppet config into productionWe use the Jenkins 'Promoted Builds' plugin for this

JENKINS: PROMOTION

EXCEPT..How does a machine get from a bare image to the statewhere we can push packages to it from Pulp?How does a machine know what type of machine it is?How do we find other resources, eg databasehostname?

CLOUDFORMATIONAmazon tool for specifying infrastructureEverything* we provision inside AWS is provisioned viaCloudFormationJSON templates

* Everything except for the things Amazon doesn't exposevia CloudFormation..

CLOUD-INITWorks with multiple cloud typesSorts out things like SSH keys, allows us to configurehost namesAlso allows us to provide a bash script to run on startup

PROVISIONING A BAREINSTANCE

cloud-init automatically manually adds the pulp repowhich contains Pulp, Puppet and our Puppetmanifests/modulesInstalls appropriate RPMsPuppet runs, subscribing the machine to the relevantPulp repos, and installing packages in the usual Puppetway

HOW DOES IT KNOW WHATTYPE OF MACHINE IT IS?

We tell it!Use an environmental variable $HOSTTYPESimply run

puppet apply \--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp

EXTRA FACTSCustom facter factsAlso specified in an environmental variable

Data comes from within the CloudFormation templateOn our list of things to look at:

FACTER_HOSTENVIRONMENT=liveFACTER_STACKNAME=customer-web-live

https://github.com/fanduel/hiera-cloudformation

OTHER RESOURCESWe either:

Provide details as a facter factF̀ACTER_DBHOST=xyz

Also use this approach to limit distribution ofsecure details, eg DB passwords

Discover via the EC2 APIEg Varnish servers discover web backends bycalling API and finding hosts tagged appropriately

FREE WINS!

FREE WINS!Greater control over the timing of Puppet runsImproved visibility - for ops and devsConfiguration changes now have to be deployed totesting/staging first

MORE FREE WINS!Puppet configs now have a versionEasy to find config version on the machine itselfConfig changelogs accessible on every machine

(Git changelog added to RPM)

THE DOWNSIDESPuppet manifests and modules on all machines

Potentially a security issue?Mitigated by CloudFormation holding most sensitivedata

ALTERNATIVEIMPLEMENTATIONS

Don't want to use Pulp?Could do basically the same thing with yum s3 plugin

Use mcollective to push package updateshttps://github.com/jbraeuer/yum-s3-plugin

FUTURE IMPROVEMENTSBuild AMIs using Packer instead of configuring at boottime

Decrease time to autoscaleWould probably still need to run Puppet at first bootto configure machine specific settings

QUESTIONS? COMMENTS?Sam Bashton

sam@bashton.com

Twitter: @bashtoni

(Psst.. )http://www.bashton.com/jobs/