SNEAK PEEK The Puppet Environment at Fairfax Media / Domain Group
Dejan Golja // Gonzalo Servat
Introduction
Dejan GoljaSenior Systems Engineer // Fairfax Media
Twitter @dejangolja
Introduction
Gonzalo ServatSenior DevOps Engineer // Domain Group
Twitter @gservat
Who are we?
Our Puppet History
☁ Early Puppet adopters … since version 0.25.X
☁ Large scale environment with distributed team
☁ We made every possible mistake
We’re on our 3rd major Puppet implementation!
Lessons Learned
☁ Do NOT make changes directly on the Puppet Master
☁ DO your testing - lint, code validation, etc.
☁ Do NOT pollute your downloaded official modules
☁ DO define and document the standard workflow
☁ Do NOT store sensitive data into modules
Example of ugliness (old site.pp)
Standardised process for deploying changes
Easily repeatable for multiple
environments
Puppet “V3” Requirements
High Availability
Bring order to our modules
Ingredients
☁ Packer
☁ Puppetlabs R10k
☁ Custom Ruby Service - “Captain Hook”
☁ Notification Systems - Slack/Flowdock
EC2 ASG ELB
SNS SQS RDS
CloudFormation
☁ Module preferences☁ Puppetlabs supported☁ Vendor supported☁ Puppet approved
☁ Create module guidelines else it gets very messy!☁ Each module lives in a separate Git repository
Puppet Modules
Git Flow
Feature Branch
Workflow
Architecture
Puppet Client
Puppet CA
Puppet Masters
Puppet DB
Architecture
$ aws cloudformation create-stack \
--stack-name puppetmaster-staging \
--templatebody file://infrastructure-puppetmaster.template \
--parameters \
ParameterKey=Environment,ParameterValue=staging \
ParameterKey=InstanceType,ParameterValue=m3.large \
ParameterKey=MinScalingGroupSize,ParameterValue=2 \
ParameterKey=MaxScalingGroupSize,ParameterValue=4
CloudFormation Example
Awesome, so we have a neat-looking HA/FT set-up…
…but how do we manage pushing changes to Puppet Masters?!
☁ Puppet masters can “come and go” randomly
☁ Keep the modules up to date per environment
☁ K.I.S.S. - Tame the learning curve for the team
☁ Avoid reinventing the wheel
Challenges
OUR SOLUTION
P3 Workflow
1. Make changes and commit/push to git server2. Git server triggers post-commit hook (POST) to Captain Hook server3. Captain Hook server validates the payload & creates new message:
☁ Full refresh ➩ r10k deploy environment -p☁ Light refresh ➩ r10k deploy environment☁ Module refresh ➩ r10k deploy module <name>
4. Captain hook server pushes new message to SNS5. Captain hook client polls & reads messages ( ➩ r10k)6. … and we get notified in slack/flowdock/dashboard
Typical Puppet Workflow
DEMO
☁ Fairfax is a large scale complex environment
☁ Many systems engineers are constantly pushing changes
☁ Puppet architecture designed for HA and fault tolerance
☁ Puppet workflow helps us deploy changes to multiple
masters safely and easily
Summary
Our Blogshttp://engineering.fairfaxmedia.com.au/
http://tech.domain.com.au/
Our Codehttp://github.com/fairfaxmedia
Your Feedback or QuestionsYou can contact us on twitter: @dejangolja @gservat
Before you go ...