Post on 28-Nov-2014
description
transcript
Chef at Etsy
@jonlives
Jon Cowie
Sr Operations Engineer
30 Million Members
4
1 Million Active Shops
20 Million Items Listed
5
60 Million Monthly Unique Visitors
@jonlives
We Love Chef!
@jonlives
Absorb what is useful.
Discard what is useless.
@jonlives
“I am not smart enough to build an ontology … that
can encompass all the variations in infrastructure.
Nobody is, the world moves too fast.”
@jonlives
There is no magic pill.
@jonlives
You are the expert.
@jonlives
Chef at Etsy
• Chef Server 11.1.4
• ~2000 Nodes
• CentOS, some Mac OS X
@jonlives
Beginning of 2010 Today
@jonlives
Chef at Etsy
@jonlives
Evolution of Chef
@jonlives
2010: The Beginning
• ~250 Nodes (Ubuntu & CentOS
• The first cookbooks
• Out of the box workflow
@jonlives
2011: Growth
• ~400 Nodes (CentOS)
• Chef still pretty specialised knowledge
• Handlers added
@jonlives
2012: A big year
• ~800 Nodes (CentOS & MacOS X) • More in-house Chef expertise • Workflow tooling • Debugging tooling • Monitoring
@jonlives
2013: Chef at Etsy
• ~1500 Nodes • Workflow tooling enhancements • Feature flags in Chef • Chef performance - Chef 11 upgrade
@jonlives
2014: Chef at Etsy
• ~2000 nodes • Consolidation • CI with Chef • Omnibus • Work-in-Progress tooling
@jonlives
Patterns & Workflows
@jonlives
Cookbook Workflow
@jonlives
$> review -r jcowie --cc ops
@jonlives
knife-spork
• https://github.com/jonlives/knife-spork • Workflow tool • Helps multiple chefs avoid clashing • Visibility into changes • Plugins
@jonlives
knife-spork
• knife spork bump • knife spork upload • Test change
@jonlives
Test Change
• https://github.com/jonlives/knife-flip
• knife node flip foo.etsy.com testing
• knife role flip MyRole testing
@jonlives
Test Change
• https://github.com/mrtazz/knife-wip • Uses node tags <irccat> CHEF: bburry started work cent7 package bugfixing on deploy01.ny5.etsy.com
@jonlives
knife-spork
• knife spork bump • knife spork upload • Test change • knife spork promote --remote • git commit and push
@jonlives
Monitoring & Debugging
@jonlives
knife-spork & CI Job
<irccat> CHEF: Jon Cowie uploaded pentaho@0.1.8 <irccat> CHEF: Jon Cowie promoted pentaho@0.1.8 to production <snip> <irccat> Git PUSH -> Sysops/chef <snip> <Jenkins> Starting build #5649 for job chef-server-git-sync <Jenkins> Project chef-server-git-sync build #5649: SUCCESS in 2 min 36 sec: http://ci.etsycorp.com/job/chef-server-git-sync/5649/
@jonlives
IRC Handler<irccat> Chef run failed on officebackup01.office.etsy.com gist failed, see /var/log/chef/client.log on the host !
<irccat> Still Failing on dbnest01.ny4.etsy.com since 2 days ago https://github.etsycorp.com/gist/656d8914fbef5a6bd9aa
@jonlives
Lastrun Data
• https://github.com/jgoulah/knife-lastrun
• knife node lastrun foo.bar.com
@jonlives
Lastrun Data% knife node lastrun dbnest01.ny4.etsy.com Status failed Elapsed Time 29.055892 Start Time 2014-‐10-‐06 12:54:51 +0000 End Time 2014-‐10-‐06 12:55:20 +0000 !<snip> !Exception <snip> Installed package backupd-‐1.4-‐1.365657d.el5.centos is newer than candidate package backupd-‐1.2-‐1.99ddb8e.el5
@jonlives
Dashboards
@jonlives
Dashboards
@jonlives
Dashboards
@jonlives
Monitoring & Debugging
• https://github.com/etsy/chef-handlers • https://github.com/etsy/dashboard • https://github.com/jgoulah/knife-lastrun • https://github.com/bmarini/knife-inspect
@jonlives
Feature Flags
@jonlives
Downsides of Existing Approach
• Holding cookbook in testing is blocking • Accidental promotions • Testing env affects all cookbooks • “Upgrade” envs often used • How to make it more “Etsy”?
@jonlives
@jonlives
chef-whitelist
• https://github.com/etsy/chef-whitelist • Databag driven • Cookbook library • Feature flags!
@jonlives
chef-whitelist{ "id": "php-5-5-17", "patterns": [ "statsd*.ny5.etsy.com", "deploy*.ny5.etsy.com", <snip> ] }
@jonlives
chef-whitelist
if node.is_in_whitelist? "php-5-5-17" package "php-pecl-opcache" do action :remove end end
@jonlives
Configuration Data
@jonlives
Keep cookbooks:• Simple • Modular • Scalable • Maintainable
@jonlives
Environments
• Cookbook version constraints
@jonlives
Roles
• Group-level config • Syslog-ng • Iptables • Sudoers
@jonlives
Roles - iptables“firewall": { "ports": { "11211": { "subnet_group": "prod_subnets" }, <snip> } }
@jonlives
Roles - Syslog-ng"syslog": {
"web": {
"web_apache_access_log": {
"source": "/var/log/httpd/access_log",
"source_program_override": "APACHEACCESS: ",
"destination": "/data/syslog/current/web/access.log",
"destination_filters": [
"host('^(web0|dlweb)')",
"match('APACHEACCESS')"
]
}
}
@jonlives
Data Bags
• Global / Datacenter specific Config • Ganglia • Cobbler • VOIP
• Data Storage
@jonlives
Data Bags - Ganglia{
"id": "config_se5",
"grid_name": "EtsySE5",
"authority": "http://gangliase5.etsycorp.com",
"trusted_hosts": <snip>,
"groups": {
"Utilities": "239.2.11.71",
<snip>
}
<snip>
}
@jonlives
Data Bags - Cobbler{
"id": "config_corp",
"cobbler_server": "corpking02.corp.etsy.com",
"dns_servers": [ “10.x.x.x", “10.x.x.x" ],
"dhcp_ranges": {
"10.100.x.0": {
"routers": "10.x.x.1",
"mask": "255.255.255.0",
"range": "10.x.x.11 10.x.x.250"
}
}
}
@jonlives
Write cookbooks you’ll thank yourself for.
@jonlives
!
http://jonliv.es/book !
Discount Code: AUTHD !
40% off Print 50% off Digital
@jonlives
Thanks! Questions?
!
@jonlives / http://jonliv.es / jon@etsy.com