Date post: | 20-Aug-2015 |
Category: |
Technology |
Upload: | peter-leschev |
View: | 10,450 times |
Download: | 4 times |
• Build platform & services used internally within the company• 90k builds per month• 43k automated tests just for JIRA• Developers expect a reliable infrastructure & fast CI feedback
Build Engineering today @ Atlassian
• 1000 build agents (own hardware + EC2 instances)• include SCM clients, JDKs, JVM build tools, databases, headless
browser testing, python builds, NodeJS, installers & more
• Maintain 20 AMIs of various build configurations• 8 Bamboo Servers• maven.atlassian.com / 6 Nexus instances • Monitoring - opsview / graphite / statsd
Build Engineering today @ Atlassian
• Automated style checking• Setup automated build that runs checks & posts results• Setup ratchet build to detect regressions
Puppet Lint https://github.com/rodjek/puppet-lintTim Sharpe
@rodjek
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence of Change
Dev Code review Rollout Soak in Prod
initial + Code review
• Coding on Puppet Master• Culture of manually modifying
production - Configuration Drift
• Impact on Builds
Using Staging for Development
puppetmaster
build agentsbuild agents
staging puppet environment
• Easily spin up Infrastructure locally on your laptop• Reproducible / disposable environments• Machine provisioning via Virtual Box / VMWare / AWS• Configuration applied via Shell Scripts / Puppet / Chef• Develop and test infrastructure changes locally
Vagrant http://www.vagrantup.com/Mitchell Hashimoto
@mitchellh
Vagrant
Vagrantfile
vagrant basebox
http://www.vagrantup.com/Mitchell Hashimoto
@mitchellh
Vagrant
Spins up a local VM to a known state
Destroy the VM when done
Make some puppet changes and then run:
to apply your changes
SSH into your VM using:
to check your changes
http://www.vagrantup.com/Mitchell Hashimoto
@mitchellh
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence of Change
Dev Code review Rollout Soak in Prod
initial + Code review + Vagrant
• Vagrant basebox differences with production machines• Originally using publicly available vagrant baseboxes
• Installed packages biggest differences
• Generating a basebox manually was a painful process
Vagrant != Production
Packer http://packer.ioMitchell Hashimoto
@mitchellh
Vagrant box for Virtualbox
packer template JSON
Vagrant box for AWS
• Latest basebox generated in CI & published to fileshare• No need to generate baseboxes locally
Basebox generation via CI
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Code review Rollout Soak in Prod
initial + Code review + Vagrant + Packer
• Behaviour Driven Development
Cucumber https://github.com/cucumber/aruba
Cucumber & Vagrant
Vagrant
Custom Provisioner
Virtual Box
VM
puppet apply
cucumber *.features
via ssh
• Requires cucumber dependencies to be installed on tested VM
• Tests run within the VM making testing firewall rules harder
Disadvantages
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Code review Rollout Soak in Prod
initial + Code review + Vagrant + Packer + Cukes
The Cattle: you give them numbers. When
they get ill, you shoot them
The Pets: you give nice names,
you stroke them, and when they get ill,
you nurse them back to health,
taking a long time over it
– Tim Bell, CERN”
“
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Code review CI & Rollout Soak in Prod
initial + Code review + Vagrant + Packer + Cukes + CI
Spread out CI
provision VM #1
Moved from sequential to parallel provisioning
provision VM #2
provision VM #3
provision VM #4
provision VM #1
provision VM #2 provision VM #3
provision VM #4
Profiling Puppet Runs
Add “--evaltrace” to puppet apply
+ =Collect and show the longest occurrences of: “Evaluated in ([\d\.]+) seconds”
Profiling Cucumber runs
http://itshouldbeuseful.wordpress.com/2010/11/10/find-your-slowest-running-cucumber-features/
• Provision locally & for CI• Faster & different class of problems found• Matches production state
Delta Provisioning
‘from scratch’ provision delta provision
provision VM
export VM fileshare
import VM box
provision VMon success
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Code review CI & Rollout Soak in Prod
initial + Code review + Vagrant + Packer + Cukes + CI+ Delta CI
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Branch CI Code review CI & Rollout Soak in Prod
initial + Code review + Vagrant + Packer + Cukes+ CI + Delta CI + Branch CI
• MacPros no longer required• They were limited in supply & old
• 2x speed improvement• Only limited by our credit card limit
Vagrant-AWS https://github.com/mitchellh/vagrant-aws
Catalog Diff
puppet master --logdest console --compile HOSTNAME
Step 1: Generate a hash of a node’s catalog
HOSTNAME.json
- Sort elements - Remove timestamps - Generate shasum
f50db91e6461f5bdcb56769a8f77da1fac26943d
Catalog Diff
Step 2: Compare the hash of master versus your branch to avoid unnecessary provisioning
Example 1:
f50db91e6461f5bdcb56769a8f77da1fac26943d f50db91e6461f5bdcb56769a8f77da1fac26943d=
Hash is the same, no build required
master branch
Example 2:
f50db91e6461f5bdcb56769a8f77da1fac26943d 18033e4d21b78bab6deb3ae1ff3c147ade5a37ca!=
Hash is different, build required
master branch
Catalog Diff
Step 3: Profit!Reduction in
feedback time +
$$$ saved
Images: http://pixabay.com/p-30984/ https://www.flickr.com/photos/williamnyk/3598113750/
• Puppet runs impacted running builds• Disabling all the build agents
• Performing the roll out
• git clone / librarian-puppet / symlink update on puppetmaster
• Manually kick off puppet on all the build agents
• Enabling all the build agents
• Set of Puppet environments for every bamboo server
Painful Puppet Rollouts
Graceful Service restarts
+Bamboo Agent JVM process watches for touch file & shutdowns when Idle (written as a Bamboo Plugin)
• BEFORE - Multiple puppet envs for each Bamboo Server• jbac_staging
• jbac_production
• cbac_staging
• cbac_production
• etc
• AFTER - Changed to use ‘staging’ & ‘production’ only
Puppet Environments
• BEFORE: Manually on puppetmaster• git clone the puppet tree
• run librarian-puppet to pull external modules
• Update staging / production symlink
• AFTER: Bamboo build which performs the above steps automatically
Updates on Puppetmaster
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Branch CI Code review CI & Rollout Soak in Prod
initial + Code review + Vagrant + Packer+ Cukes + CI + Delta CI + Branch CI+ Frequent Releases
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Branch CI Code review CI & Rollout Soak in Prod
initial + Code review + Vagrant + Packer+ Cukes + CI + Delta CI + Branch CI+ Frequent Releases + Notification
HIGH
NONE
Lifecycle of an infra change
confidence
Confidence in Change
Dev Branch CI Code review CI & Rollout Soak in Prod
before after
• https://forge.puppetlabs.com/atlassian/nexus_rest • Configure Nexus using Custom Puppet Provider Types
rather than XML files
Puppet Module for Sonatype Nexus