Post on 10-May-2015
transcript
So Continuous. Much Delivery.
Very Chef. Wow.
A Case Study on using Chef to start building a Continuous Delivery Pipeline
• George Miranda
• Sr Consultant at Chef Software, Inc.
• Unix guy (15+ years)
About Me
Minimum Viable Pipeline
• Step 1: Develop a new change
• Step 2: ???
• Step 3: Production!
• MOAR FASTERZ
What we know
• Must utilize existing tools within the company
• Git for SCM
• Jenkins approved for use
• Working in a static VM environment
• Just migrated to single cookbook repos
• Starting with infrastructure cookbooks
• Want a manual go-to-production button (ugh!)
Case Study: Requirements
• Git PR model: branch from master for any new feature
• 4-person team, only 3 active at any time
• Code review done manually and informally
• Simple communication/reqs (makes it easy!)
Case Study: Code Review Model
• How are developers expected to work locally?
• When do they push to remote? How do we verify their work?
• Code Review criteria: what does it mean to be ready to merge?
• How do we go from merged code to artifact?
• How do we get that artifact all the way to Production?
Figuring out new workflow
• New branch for every feature
• Create a failing test
• Write a resource to pass the test
• Local commits
• Test-Kitchen + guard
• Once local tests passed, push to remote
Local Development Work
• Open a Pull Request (new branch to master)
• Triggers a build via Jenkins GHPRB plugin
Push to remote
The Verify Build Job
• Verify syntax (knife cookbook check)
• Foodcritic Rules
• Test-Kitchen w/ BATS busser
Push to remote
@test "My directory is created" {! test -d /foo/bar!}!!@test "A basharific test" {! if [ foo != bar ]; then! skip "foo isn't bar"! fi!! run foo! [ "$status" -eq 0 ]!}!!• https://github.com/sstephenson/bats • Super low learning curve (but also very limited)
BATS: Simple Unit Tests
• If failed, notify
• Another commit to the same branch triggers another Verify Build Job
• Super easy to track, comment, and approve
• If passed, let’s go to Human Code Review
Push to remote
• Only one change per one cookbook at one time
• Must have test for feature that changed
• One for one: resource unit tests
• Consider the smoke test
Human Code Review Rules
• Unit tests: small, fast, check one single concern
• In this context: checking Chef resources
• Smoke tests: test multiple things in the course of one concern
• In this context: check the intent of a recipe
• Note: that was testing for this use case
Unit Test vs Smoke Test
• Only 3 active team members at any given time
• Submitter cannot approve
• Merge approval requires 2 approvals
• Code review can happen at any time, but only merge when you’re ready to fix it.
When are we ready to merge?
• Freeze your cookbooks!
• Semantic versioning: Major.Minor.Patch
• You own Major.Minor
• The Pipeline owns .Patch
• No one gets to knife upload
No one.!
Ever.!
• "git merge" is the new "knife upload"
Merged code to artifact
• Bumps Cookbook version
• Re-commits to master
• Upload frozen cookbook (via berks)
• Pin that new cookbook to the Integration environment
• Converge all nodes that use that cookbook
The Integration Job
• First sign that things may be broken
• These nodes also run smoke tests
• serverspec, minitest, etc
The Integration Job
• We survived! Trigger the next job(s)
• The Jenkins Build Pipelines Plugin allows upstream/downstream definitions to string together jobs
• From here out, it’s all the same Promote Job*
• After the Integration job, we just run X number of Promote Jobs
The Integration Job
* (mostly)
• Pin cookbook to new Chef Environment
• Converge all nodes using this cookbook
• Run Tests
Promote Jobs
#!/opt/chef/embedded/bin/ruby !require 'chef/environment' require 'chef' Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb") !def pin_env(env, cookbook_versions) to = Chef::Environment.load(env) cookbook_versions.each do |cb, version| puts "Pinning #{cb} #{version} in #{env}" to.cookbook_versions[cb] = version end to.save end !cookbook_data = Array.new !if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))) metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')) File.read(metadata_file).each_line do |line| if line =~ /^name\s+["'](\w+)["'].*$/ cookbook_data << $1 end if line =~ /^version\s+["'](\d+\.\d+\.\d+)["'].*$/ cookbook_data << "= #{$1}" end end end !cookbook_versions = Hash[*cookbook_data] !pin_env(ARGV[0], cookbook_versions)
Pin the cookbook to Env
$ berks apply <environment>
Pin the cookbook to Env
$ knife ssh "recipes:mycookbook AND chef_environment:promote-environment” 'sudo chef-client'!
… OR …
Pushy!
Converge Nodes
• Most testing frameworks have a Report Handler to automatically run tests
• chef-serverspec-handler
• minitest-handler
• Deploy to your nodes by adding ‘chef_handler’ to their run_list
• Many community cookbooks are already packaged with tests
Run Tests
• In this particular use case:
• Build job: BATS (unit tests)
• Integration & Promote jobs: serverspec (smoke tests)
• UAT: also ran Cucumber tests (acceptance)
Run Tests
• Can string together N number of promotions
• UAT
• Production A
• Production B
• etc
Promoting to more environments
• In production monitoring is the test
• Could not queue up changes reliably anyway
• There is no spoon
Push to Production
• Small incremental deployments led to greater confidence
• TDD was pushed to the forefront of priorities
• Commitment from Dev group to write application deployment cookbooks
• But the biggest lesson learned…
Results
• Continuous Delivery is a practice, not a tool
• Small incremental changes in code
• Small incremental changes in workflow
• Small incremental changes in tooling
• You will constantly improve your code, your workflow, your tools, your team, and your skills.
Let’s Go Devop with a CD tool
RECAP
• Step 1: Develop a new change
• Step 2: ???
• Step 3: Production!
• MOAR FASTERZ
What We Wanted
• (Pre-req) Test Driven Development
• 2A. Establish development workflow before submitting changes *
• 2B. Auto verification of submission before humans look at it
• 2C. Humans Apply Code Review Criteria *
• 2D. Don’t merge unless you mean it *
• 2E. Merge kicks off an Integration Job
• 2F. Followed by a series of Promotion Jobs
• 2G. There is no spoon *
Wait… what was Step 2?
• Step 1: Develop a new change
• Step 2:
!
!
• Step 3: Production!
• Step 4: Level Up. This is great!
• Step 5: MOAR THINGS! Wait. This is hard!
• Go to Step 1
What We Got
(Pre-req) Test Driven Development
2A. Establish development workflow before submitting changes *
2B. Auto verification of submission before humans look at it
2C. Humans Apply Code Review Criteria *
2D. Don’t merge unless you mean it *
2E. Merge kicks off an Integration Job
2F. Followed by a series of Promotion Jobs
2G. There is no spoon *
• Test Kitchen — http://kitchen.ci/
• Guard Plugin for Test Kitchen — https://github.com/test-kitchen/guard-kitchen
• Foodcritic — http://acrmp.github.io/foodcritic/
• Berkshelf — http://berkshelf.com/
Key Chef Ecosystem Tools
• git
• github
• build-pipeline-plugin
• ghprb
• warnings
• mailer
Helpful Jenkins Plugins
I want to hear from you! !
@gmiranda23 gmiranda@getchef.com