Designing aRelease Pipeline

Jose Luis Soria

jlsoriat@gmail.com - @jlsoriat

What is a Release Pipeline?

• Automated manifestation of your delivery process.

• Feedback mechanism.

• Detection of unfit release candidates.

• Pull system.

• Useful for CD or any other delivery model.

Pipeline design considerations

• Emergent design. No BDUF.

• Start early.

• Start simple and evolve with the system.

• Begin with the most valuable assets.

• Address the bottlenecks.

#1 Define Components

What is a Component?

• A set of artifacts (binaries, dynamic code, configuration files, other supporting files) that can be deployed and verified together without affecting other areas of the application.

Tips for choosing components

• Deploy and test the smallest independent entity.

• Rely on the architecture:• Logical / physical.

• Layers / tenants.

• See the whole.

What do we need to know?

For each component:

• Meaningful name.

• Description.

• Priority / order (when to address it?)

• Source (most likely, version control.)

• Target (where it gets deployed.)

• Pre-requisites.

• Dependencies.

• Configuration tokens.

Configuration tokens

• Make a list of environment-dependent information.

• Tokenize it in the configuration.

• Gather the values for all the environments.

Component sheet

Activity: defining components

„Our application consists on:

A web site built on top this technology stack:

• MVC framework.

• Client-side logic (HTML5, JavaScript.)

• Entity model mapped to the DB using a ORM.

• Data model residing in a DB.‰

Activity: defining components

„Our application consists on:

A web site built on top this technology stack:

• MVC framework.

• Client-side logic (HTML5, JavaScript.)

• Entity model mapped to the DB using a ORM.

• Data model residing in a DB.

Key business logic resides on a web services layer.

We also maintain a mobile client for two platforms.‰

Activity: defining components„Our application consists on:

A web site built on top this technology stack:

• MVC framework.

• Client-side logic (HTML5, JavaScript.)

• Entity model mapped to the DB using a ORM.

• Data model residing in a DB.

Key business logic resides on a web services layer.

We also maintain a mobile client for two platforms.

For some operations we make calls to third-party web services.‰

Example: component sheet

Implementation notes

• It should be possible to independently build, deploy and test anything defined as a component.

• You should decide how dependencies will be made available:

• Source Control.

• Artifact repositories (NuGet, Maven⁄)

• Deployed artifacts.

• Etc.

#2 Identifysub-pipelines

Single pipeline

• A single pipeline servicing all the components and teams.

• May be able to detect which component has changed and operate only on that one.

One pipeline per component

• Each component has its own pipeline.

• Different pipelines may have different designs.

• Individual pipelines may fan-in to a system pipeline.

• More flexible but more complex.

One pipeline per team

• Each team has its own pipeline.

• Different pipelines may have different designs.

• Individual pipelines may fan-in to an integration pipeline.

Mixed approach

• Different teams building different components.

• Keep it simple!

Implementation notes

• It is better to use a tool that allows to define sub-pipelines, fan-in, fan-out, etc.

#3 Define Stages & Orchestration

What is a Stage?

• A set of steps or activities that are performed on a release candidate.

• I lets any release candidate advance towards production, or discards it.

• When a release candidate passes through a stage, our confidence on it is increased.

• It is a source for feedback.

What is Orchestration?

• It is the way we arrange the stages so release candidates flow through them, in their way to production.

Tips for stages & orchestration

• Feedback is the key. Arrange stages and orchestration based on the feedback we need.

• Stages are filters. The orchestration should be arranged to stop the pipeline if a stage fails.

• Stages can contain both manual and automated steps.

• Stages can be manually or automatically triggered (approvals.)

• Automate as much as possible. Including approvals.

More tips for stages & orchestration

• Grow your pipeline wide, not long http://bit.ly/1jsNGP5.

• Build only once.

• Use environment-agnostic binaries.

• Version everything.

What do we need to know?

For each stage:

• Meaningful name.

• Clear goal.

• Does it need a manual approval to be triggered?

• Does it need a manual verification when it has finished?

• Sources.

• Flow (orchestration.)

Pipeline-level orchestration (examples)

Commit Acceptance testing Manual testing Release

Minimum pipeline

Commit Acceptance testing Release

Exploratory testing

Capacity testing Security testingUser Acceptance

testing

Complex pipeline

Fully automated

Partially automated, or manual

Legend:

Stage-levelorchestration(example)

About sources

• Version control.

• Artifact repositories.

• Environment libraries.

Which stages do I need?

• Think about the kind of feedback you need.

• Think about what should stop a release candidate to get to production.

• Create a Value Stream Map.

Value Stream Map (example)

Assessment Approval Planning

UAT

Release

Acceptance

testsCodeSpecification

2 days 1 dayValue-

added

time

Wait

time

Development: cycle time ~ ?

3 days 3 days ? ?

?

?

4 days 1 day 2 days 2 weeks ? ?

Delivery: lead time ~ ?

Prevalent stages: the Commit stage

• Eliminate early release candidates that are unfit for production.

• Close to (or the same as) a CI build.

• Quick validations: build, unit testing, static analysis, etc.

• Packaging.

• For Continuous Delivery, it runs on each commit (no branches – feature toggles.)

• For other models, decide when it gets triggered (for example, on each merge to trunk.)

Prevalent stages: the Commit stage

http://bit.ly/1jsSkwA

Prevalent stages: the Automated Acceptance Test stage

http://bit.ly/1jsSkwA

Prevalent stages: the Manual Test stage

http://bit.ly/1jsSkwA

Prevalent stages:non-Functional Testing stages

http://bit.ly/1jsSkwA

Activity: defining stages & orchestration

„We have a basic suite of automated acceptance tests that we plan to grow along with the system.‰

„The team does (manual) functional testing.‰

Activity: defining stages & orchestration

„We have a basic suite of automated acceptance tests that we plan to grow along with the system.‰

„The team does (manual) functional testing.‰

„We need to support 2,000 concurrent users.‰

Implementation notes

• Choose a tool that allows to easily model and visualize the flow.

• Choose a tool that supports what you need for orchestration:

• Approvals.

• Validations.

• Parallelization.

• Alerts.

• Etc.

#4 Define environments

What is an Environment?

• A set servers, devices or any other resources we need in order to run and validate a release candidate in its way to production.

Tips for defining environments

• Prepare for deployment automation.

• Lock down environments. Restrict access.

• Different stages could target the same environment if needed.

• Prepare for auto-provision.

• Make environments disposable. DonÊt turn them into bottlenecks.

• Environments may not be tied to stages. It should be easy to point any stage to any environment.

Activity: defining environments

„We have a basic suite of automated acceptance tests that we plan to grow along with the system.‰

„The team does (manual) functional testing.‰

„We need to support 2,000 concurrent users.‰

Implementation notes

• Use virtualization.

• Use cloud-based environments.

• Use tools for managing templates, configuration, auto-provision, etc.

#5 Define Steps

What is a Step?

• Any activity that is done in the context of a stage, that allows us to get feedback and prove the fitness of the release candidate.

• Examples:• Deploy a component.

• Run automated tests.

• Run manual tests.

• Update metrics.

• Alert the user of some event.

• Etc.

Tips for defining steps

• Consider:• The goal of the stage.• The kind of feedback you need.• Sources.• Targets (environments.)

• Build and package only in the Commit stage.

• Most times, deployment is present, but not always.

• (Automated) Smoke Testing should follow any deployment.

• Think about both automated and manual steps.

Activity: defining steps

„We want to filter out anything producing static analysis warnings.‰

„We want to try exploratory testing.‰

„We may use the same environment for load testing and security testing‰

#6 Define automation & tooling

Tips for step automation

• Automate everything.

• Automate everywhere (for all the environments.)

• Preference for automation:• Fully automated steps.

• Manually triggered automatic steps.

• Manual steps.

• Build only once.

• Version everything.

• Have environment lockdown in mind.

Deployment automation considerations

• Deploy the same way to every environment. The target environment should be a (implicit) parameter for the automations.

• Set up (tool-agnostic) one-click deployments.

• Treat configuration tokens as parameters for the automations.

• Prepare for rollbacks.

Database deployment considerations

• Database deployment is not the same as database development.

• Decide about the deployment strategy:• Schema & Data compare.

• Delta scripts (better for Continuous Delivery.)

• ORM tools (schema update, migrations, etc.)

Test automation considerations

• Q2 tests are not necessarily run through the UI.

• Smoke tests may be run through the UI.

• Frequently, non-functional testing can be automated.

• Leave environments and data in a known state.

• A few things canÊt be automated (UAT & Q3 testing.)

What do we need to know

For each step to be automated:

• Automation tool or technology.

• Execution model.

• Parameters (at least youÊll have the configuration tokens.)

• Source / target.

About execution models

• Native OS tool.

• Agent.

• Remote execution.

Activity: defining automation & tooling

„Production environment is in an isolated network‰

„Our operations people wonÊt allow us to install anything there‰

#7 Define execution model, monitoring and metrics

Continuous Delivery flow model

• Pipeline instances are created on each commit. Any commit is a release candidate.

• One-piece continuous flow model.

• There is no way back. Any error makes the release candidate to be discarded.

• Fixes are treated as new release candidates. They are run through the entire pipeline from the beginning.

Other flow models

• Pipeline instances are created as needed. A release candidate might comprise several commits.

• Decide on the batch size. Larger batches are cheaper but limit feedback.

• Errors might be fixed in the context of the stage where they arise.

Monitoring the pipeline

• Transparency.

• Rely on a proper tool.

• Set up alerts for key events.

• Use a Cumulative Flow Diagram (CFD.)

• Gather metrics and act on them.

Primary metrics (direct evidence)

• Cycle time.

• Mean Time Between Failures (MTBF.)

• Mean Time To Recover (MTTR.)

Secondary metrics (circumstantial evidence)

• Test coverage.

• Duplication of code.

• Coupling.

• Compilation warnings.

• Code churn.

• Build frequency

• Etc.

#8 Plan for future enhancements

Examples – DevOps culture

• Improve branching model if needed.

• Debugging optimization, symbol servers.

• Canary releases.

• Blue/green deployments.

• A/B Testing.

• Preventive profiling.

• Telemetry, analytics, Application Performance Monitoring (APM.)

• Proactive resiliency enablement (Simian army http://nflx.it/SPeTGj)

Example: Advanced pipeline

#9 Inspect & adapt

Continuously improve the pipeline

• Component or architectural changes.

• New skills in the team.

• New resources, tools, environments.

• Reserve time, and make the team accountable for improvement.

Summary

1. Define components.

2. Identify sub-pipelines.

3. Define Stages & Orchestration.

4. Define Environments.

5. Define Steps.

6. Define Automation & Tooling.

7. Define execution model, monitoring and metrics.

8. Plan for future enhancements.

9. Inspect & adapt.

Thanks!

Jose Luis SoriaContinuous Improvement Manager at Ria Financial

jlsoriat@gmail.com - @jlsoriat

http://geeks.ms/blogs/jlsoria

Slides: http://www.slideshare.net/jlsoria

http://aka.ms/releasepipeline

Get a book! #xp2014 @jlsoriat