Date post: | 08-May-2015 |
Category: |
Technology |
Upload: | matt-graham |
View: | 1,686 times |
Download: | 0 times |
Continuous Deploymentof Architectural Change
Matt GrahamCore Engineer @ EtsyContinuous Deployer
#iasanySeptember 27, 2011
A Brief History of Deployment
The Internet
Web Applications
Agility
Continuous
Database Deployment @ Etsy
Thursday
What it's all about
● Broadly Applicable● Tools Help● Enables the Unfeasible
MTTR vs MTTF
MTTR vs MTTF
● Cheaper for electrons
MTTR vs MTTF
● Cheaper for electrons
● Cheaper for protons & neutrons
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Cheaper for protons & neutrons
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Necessitates low MTTD
● Cheaper for protons & neutrons
The Case of Electrons
● Consider a project that has 6 bugs● continuous deployment testing misses 4& deploys fixes in 4 hours
● monthly release testing misses only 2& deploys fixes in 24 hours
● 16 hours of broken < 48 hours of broken
Good Excuses
● Infrequent Changes
Good Excuses
● Infrequent Changes● Infrequent Executions
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment
Deployment Tools @ Etsy
Culture Before Tools
● No planned deploys; only on demand
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
● Software is stable & supported
Jenkins
● Unit Tests
Jenkins
● Unit Tests● Functional Tests
Jenkins
● Unit Tests● Functional Tests● Manual Testing
tail -f | grep
IRC
Key Tools
Communication Tools
Graphs
Feature Flags
Feature Flags
Deploy != Product Launch
Dark Launch
def get_feature_a_link():
return ...
Dark Launch
def get_feature_a_link():
if enabled('NewFeatureA'):
return new_feature_a_link()
else:
return old_feature_a_link()
Dark Launch
application_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on
AB Testing
● Prove success of interface changes
● Prove interest in new features
Graphs!!!
Ganglia
Graphite
Event Overlay
StatsD
github.com/etsy/statsd
if ($success) { StatsD::timing('feature.runtime', $time);} else { StatsD::increment('feature.failure');}
Community Communication
Forums / Message Boards
etsystatus.com
twitter.com/etsystatus
twitter.com/etsystatus
Deployment is First Class
Deployment is aFirst Class Feature
Engineers are Users Too
Enable Architectural Change
Architecture is Relative
Organic Architecture
Premature Architecture
Premature Architecture
What now?
Passing Time => Change
● Scale● Product● Technology● Engineering Team
Passing Time => Change
● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes
Examples
● Images From Twisted to PHP● From PostgreSQL to MySQL Shards
From Twisted to PHP
● Run Apache/PHP on a new port
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version
PostgreSQL to MySQL Shards
● Migrate table by table
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables
Continuous Deploy Pattern
● Change in small steps
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture
Review
● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication
● Enables Architectural Change● It's really fun!!!
Continuous Deploymentof Architectural Change
Matt Grahamhttp://twitter.com/lapsu
http://lapsu.tv
Core Engineer @ EtsyContinuous Deployer
http://codeascraft.etsy.comWe're Hiring!
http://bit.ly/etsywantsawesome