Continuous Delivery at Netflix: Powered by Open Source
Dianne MarshSan Francisco, May 6, 2014
http://www.linkedin.com/in/diannemarsh
Building Continuous Delivery
Netflix GoalsHigh Availability
But … Move fastTools encourage Best Practices
But … Freedom to do the right thing
Teams Deploy Their Own CodeRun What You Wrote• Rapid Innovation• Rapid Detection• Rapid Response= Freedom + Responsibility
Nebula/Gradle• Netflix Build Language• Gradle plugins• Supports internal and open source
builds
https://github.com/nebula-plugins
BUILD
BUILD Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code
BUILD BAKE
Aminator• Create AMI from Base AMI• Image contains service and
everything needed to run it• Unit of Deployment for Test and
Prod• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
BAKE
BUILD DEPLOYBAKE
DEPLOY
Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push
http://www.infoq.com/presentations/asgard
CANARY ANALYSISTest, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue
Asgard Developer Portal
GLISTEN Extending Asgard’s WorkflowAutomated Red/Black Push
Run canary/analysisScale up new instancesTurn on traffic/Run Tests/Turn off trafficWait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
One Button Deployment?
Regional IsolationLimit Impact of Human Error• Stagger deployments?• Canary testing per region?
Know your service!
Multi-Region ConsistencyBuild Tooling to:• Schedule Deployments• Prefer off peak• Choose next available
region automatically• Provide high visibility
per region
BUILD
DEPLOY
BAKE
Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey (and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
Send in the Conformity MonkeyHave deployments diverged? Balance regional consistency
with regional isolation Provide meaningful thresholds Build best practices into
tooling and reporting
Clean up with the Janitor Monkey• Disassociate unused EIPs• Delete unassociated
Amazon EBS volumes• Delete older Amazon
EBS snapshots• Leverage Amazon S3
Object Expiration
Key Elements for Netflix• Value Self-service • Test Everywhere• Build Awareness of Multiple Regions• Avoid peak times• Roll back quickly and easily• Be Cloud Native
Netflix OSS
More at netflix.github.com
