Continuous Delivery in Practice (extended)

transcript

Continuous Delivery In Practice

Lessons from Kenshoo’s RTB project

Who, What, Where

Tzach Zohar:● System Architect● tzach.zohar@kenshoo.com

Kenshoo: ● Founded 2006● Online Marketing Technology● >500 employees● 12 World Wide locations

Agenda

● Continuous Delivery: What? Why?● RTB Project● How: 10 Field Tested Tips● The Process● Appendices

Continuous Delivery: Definition(s)

“Continuous Delivery (CD) is a design practice …blah blah blah… Techniques such as

automated testing, continuous integration …blah blah blah... resulting in the ability to rapidly, reliably and

repeatedly push out enhancements ...blah blah blah.”

- Wikipedia

“Continuous delivery is a set of principles and practices to reduce the cost, time, and

risk of delivering incremental changes to users.”

- Jez Humble

“Continuous Delivery is a software development discipline where you build

software in such a way that the software can be released to production at any time”

- Martin Fowler

Continuous Delivery: Why bother?

“Our highest priority is to satisfy the customerthrough early and continuous delivery

of valuable software”

First principle of the Agile Manifesto

Continuous Delivery: Why bother?

Better suited productResponsiveness

Less wasteHigher quality

Simplicity

Recommended Further Reading on ThoughtWorks

Background: RTB Project

● ~1.5 years ● ~3 developers, 1 PM, 0.5 Ops (no QA)● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability

Frontend ClusterHighly available, high throughput ~20 node cluster

BackendSingle node, internal APIs

FBXFacebook RTB API

Reporting ClusterElastic Map Reduce (EMR) on-demand 16-node cluster

Cassandra ClusterHighly available, high throughput ~24 node cluster

S3Raw traffic logs

~5-10 deployments / week

1.The Obvious

● Single branch (details later)● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously

Deploying

● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade

2. Four-Layer Test Suite

Staging: verify compatibility of new build with other components’ production builds

2. Four-Layer Test Suite

3. Keep Builds Stable

Do not overlook a test that “sometimes fails”, trusting build status is crucial

3. Keep Builds Stable

● Random data tests● Asynchronous tests● Integration tests

Be suspicious of:

4. Master Is Always Shippable

On every commit? Not QuiteWe follow the “GitHub Flow”:

Local Master

Local Feature Branch

Master

Feature Branch

1. pull

3. push

2. checkout

4. Merge

4. Master Is Always Shippable

“Merge” == Build and Deploy

credit: tal.salmona@kenshoo.com

5. Rigorous Code Reviews

● Because “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!

5. Rigorous Code Reviews

https://github.com/tzachz/github-comment-counter

6. Real-Time Feedback

Detect issues immediately and visually

7. Keep Upgrade in Mind (1)

Use the “Parallel Change” pattern when changing cross-node APIs / Data

1.Write: oldRead: both

2.Write: new Read: both

3.Write: new Read: new

deploy deploy

8. Keep Upgrade in Mind (2)

Verify backward compatibility in tests

9. A/B Testing

Apply new features to a limited user-group Measure business results per-group

(Not by branching)

9. A/B Testing

Splitting into groups correctly is important

9. A/B Testing

It’s easy to mess up (neglecting biases, wrong grouping, wrong comparison

methods)

This excellent talk by LivePerson’s Shlomo Lahav helped us a lot

10. Own It

Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am

10. Own It

That’s It.

The Process

● Greenfield? That’s easy:○ Start with deployment and build○ Deploy a Hello World application○ Every new feature is test-covered

The Process (RTB)

1.Increase Unit+Integration CoverageCreate naive deployment AutomationCreate monitoringManual Staging tests

2.Automated stagingDowntime eradicatedManual (but often) deployment trigger

3.Autopilot - deploy upon commit

~ 9 Months

~ 3 Months

Appendix A. Partial Tool List

Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo

Code Review: GitHubProvisioning: Puppet

Deployment: Fabric, botoMonitoring: Metrics, Graphite

Appendix B. Are You Ready?

Unit Coverage > 90%?

Good Staging Tests?

Informative Monitors?

Builds Are Kept Green?

No API Breaking Changes?

Rigorous Code Reviews?

Support Has Your Phone Number?

Do You Own it?

Not Ready

No Yes

credit: tal.salmona@kenshoo.com

Thanks. Questions?

Continuous Delivery in Practice (extended)

Software