Introduction to Continuous Delivery

André Meurer@themeurer

Head of Software Engineering @ Olympic

How long would it take to get aone line code change

into production using the current process?

Idea

Funding

Analysis

Test

Deploy

Water-scrum-fall

The last mile

The “tail period” or “last mile”

• Time from code freeze to release– Beta & regression testing– Code integration & integration testing– Documentation– Bug fixing– Deployments

Continuous Delivery is aboutreducing risk of releases

and making deployments boring

Reduce risk of releases

Fast, automated feedback on the production readiness of your application every time there is a

change

Constant flow of small increments into production

Software always production ready

Key Principles

• Constant flow of small incremental changes into prod• Software always production ready• Automate almost everything• Build, deploy, test, release, repeat• Use humans for high value stuff• Manual testing, approvals• Keep everything in source control• If it hurts, do it more often, and bring the pain forward• Done means released in production• Continuous improvement – don’t try to do it all at once

Build the right thing

• Every business idea is just a hypotesis• Most features are rarely or never used• Don’t ask customers what they want: try it out• Release a minimum viable product• Can respond quickly to market demands• Hypotesis – MVP – Analyse – Repeat

Failure will happen

check in

ci fast

ci slow

regression tests

performance tests

deployment

check in

ci fast

ci slow

regression tests

performance tests

deployment

check in

ci fast

ci slow

regression tests

performance tests

deployment

check in

ci fast

ci slow

regression tests

performance tests

deployment

check in

ci fast

ci slow

regression tests

performance tests

deployment

Faster feedback

Delivery Pipeline

Testing & promoting

Continuous integration

• Early integration• Everyone’s changes integrated frequently in the same place• Build & test at every check-in• Rapid feedback• It’s a practice, not a tool

The software testing pyramid

GUI

Acceptance tests

Unit & component tests

More business-facingLess stableMore realistic

Lower costEasier maintenance

More stableFaster feedback

Exploratory/manual

The software testing cupcake anti-pattern

Trunk based development

• Check-in or merge into trunk at least once per day• Feature branches should be short lived• Branches discourage refactoring

– People don’t know what code other team members are touching, so they are hesitant to refactor for risk of impact

• Branches delay integration and hide risk• Merging wastes time and is tedious

Inside out development

• Build from the bottom– DB – Business Logic – UI

• Feature not accessible until full development is complete• Small slices/stories to enable fast feedback

Feature toggles/switches

• Deploy incomplete features switched off– Be aware of feature switch madness– Switches need to be deleted as the feature is in production

• Control who can see the new feature– Test it with a small group of users– Ensure it handles the load– Gradually increase group of users if appropriate

• Separate deployments from releases

Feature switches

Dev

Smaller

Many

Short

Operations

Business

Larger

Few

Long

Granularity

Numbers

Lifetime

Branch by abstraction

• Parallel routes• Make architectural changes

– Interface to create abstraction– Abstraction layers with two working implementations– Swap bit by bit of the code to the new architecture until complete– Old implementation can be removed once no longer used

• DB changes– No breaking schema changes until code is ready to deal with new

schema

Roll Forward

• Rollback strategy is complex & risky• Easier to fix something and get it through the pipeline

Canary Releases

• Release features to smaller groups/clusters• Test/verify stability/performance/etc.• Roll out to more/rest of users

Blue Green deployments

• Two instances of the production environment– Current PROD (blue)– Upgrade environment (green)

• Start routing small groups/clusters of people to Green environment

• Swap them

Quality

• QA inspects quality• Everyone is responsible for quality• QA represents the user

IT ops team close to dev team

• Sit together: Dev, QA, and Ops• Have sysadmin as part of the development team • Deploy to all environments the same way we deploy to

production

Delivery Pipeline Tools

• Orchestration/CI– Team City– Jenkins– Bamboo

• Delivery Manager– LiveRebel– OctopusDeploy

• Infrastructure Automation– Chef– Puppet– Bcfg2– CFEngine

Where do I start?

• Why do you want to do it?• It will take time, start now• CI is first step• Get the architecture right• Culture of continuous improvement • It will need investment

It’s a journey

• Fundamental change to the way most organisations ship software

• Releases tied to business needs, not IT constraints• Minimise lead time from idea to live (concept to cash)• IT no longer the bottleneck• Buy-in from client is critical

Resources

• Jez Humble: Continuous Delivery • Patrick Kua: Implementing Continuous Delivery• Facebook: Moving fast at scale• John Allspaw: 10 deploys per day, dev and ops cooperation at Flickr• David Cramer: Practicing Continuous Delivery • Sam Newman: The path to continuous delivery• Rolf Russell: Introduction to Continuous Delivery