A Continuous Delivery Journey - RHAD · •We were to start continuous delivery in September 2015....

transcript

Copy r i g ht © S A S I nst i t ut e I nc. A l l r i g ht s r e se r v e d.

A Continuous Delivery Journeyand Scaling our Culture

SHOBHA SUBRAMONIANOCTOBER 18, 2018

Picture source: QuizzClub.com

INTRODUCTION

• A Case Study in moving to Continuous Delivery

• One organization’s journey: How we made the transition

• Practices that may work for others

C o p yright © SAS In sti tute In c. Al l r igh ts res erved.

AGENDA

• WHERE WE WERE

• OUR NEW DESTINATION

• CHALLENGES

• TRANSITION PLAN

• HYBRID APPROACH

• START OF THE JOURNEY

• ROCKY ROAD

• MINDSET SHIFT

• CODE DELIVERY MODEL

• WHERE WE ARE NOW

• WHERE WE WANT TO GET TO

• DEVOPS

*******************

• SCALING CULTURE

WHERE WE WERE

• On premise deployments

• Releases ~ once a year

• Long development cycle

• Agile sprints: Theoretically release-ready at end of each sprint

• However we were not actually release-ready!

• Big difference between being actually ready to release versus theoretically done

WHERE WE WERE (Continued)

………Release Planning, Scoping

Development & Test Iterations Last BuildSignoffRelease/Ship

OUR NEW DESTINATION

• Hosted environment on the cloud (using Amazon Web Services)

• Multi-tenant

• Single code base for all customers

• Short development cycle

• Continuous delivery

• Release every 4 weeks

• 4 week Agile sprints

• Goal: be release-ready at end of each sprint

Application(Source code, OS, Hardware)

AWSCustomer A

Customer B Customer C

OUR NEW DESTINATION (Continued)

Customer A’s data

Customer B’s data

Customer C’s data

OUR NEW DESTINATION (Continued)

Release Planning, Scoping

Development & Test

Release

CHALLENGES

• Typically the key to continuous delivery or release is:

• But Automation is not achieved in a day

CHALLENGES (Continued)

What is the automation we are looking for ?

• Automation of Unit testing

• Automation of Functional / End-to-End testing

• Automation of Regression testing

• Automation of Builds

• Automation of Deployment

• Automation of Deployment validation / Health checks

• Automated promotion and deployment to Stage

• Automated promotion and deployment to Production

CHALLENGES (Continued)

Other challenges:

• New leading-edge Cloud technology

• New Amazon Infrastructure Services and Platform Services

• Design of Architecture with the above

• Setup of various Instances / Environments

• Design of Code promotion process

• Development practices for Cloud, Multi-tenancy

• Security, Stability, Performance

• Extensive training; Some Hiring of new talent

12 Slide from

TRANSITION PLAN

1. Release an Early Adopter / Beta version in the cloud• ~10 months to build and deploy

• Allowed time for setup of new delivery framework

• Resulted in an operational product in the cloud

• With 2 Early Adopter customers

2. Make the leap to continuous delivery• Bringing on additional customers

• Move from Beta to full Production mode

• For this we arrived at a Hybrid approach.

HYBRID APPROACH

• We would have a release every 4 weeks

• However, every release would follow a 3-sprint (12-week) cycle

• 3 Sprints / Phases:• Story Development and Testing

• Integration Testing

• Release Candidate and Production Deployment

• Over time, we would reduce the length of this cycle

15C o p yright © SAS In sti tute In c. Al l r igh ts res erved.

HYBRID APPROACH (Continued)

• Dev & Test phase• Majority of work for release done in this phase

• Common sprint in Jira for all teams

• Cut over to Integration phase• Small team focused on Integration phase tasks

• Cut over to Release Candidate phase• Even Smaller team focused on RC phase tasks

START OF THE JOURNEY

• We were to start continuous delivery in September 2015.

• And put out a release every 4 weeks after that.

• We had a complex product suite that had:• 11 teams (most in Cary, NC; some in India, UK, and other locations)• Every team had 6-12 members (Dev + Test)• 4 project managers

• We set standards across the program: • Sprint process• Story template and Workflow• Code delivery

ROCKY ROAD

• Most teams had been together only a few months

• Still going through Storming, Norming - Transitioning to Performing

Source: Okpalad, based on Tuckman and Jensen

ROCKY ROAD (Continued)

• First continuous release did not come in September

• … or in October …

• Our scope was too large for one sprint

• Could not complete testing which was still largely manual

• We continued work to complete most of the scope

• And addressed build, deploy and other challenges

• We finally delivered in December – 3 months beyond plan

• Retrospectives to see how to reign in the scope

• Scope for next release slashed significantly

• Minimized dependencies between teams

• Scrum of scrums held for close inter-team collaboration

• Share point and Jira process standardization for consistent use

• Incorporation of Early Adopter feedback

• In 2016, we missed 4 out of 13 planned releases.

• In 2017, we only missed 1 out of 12 planned releases.

• (We did not do December releases anymore.)

MINDSET SHIFT

• Meet the need to get new features to customers quickly• One of the biggest benefits of continuous delivery

• Plan significantly smaller scope to finish in a sprint

• Deliver basic functionality first – additions/embellishments after

• Deliver few “done” stories instead of many half-finished stories

• Chunk up stories so they are “deliverable” in sprint

• Code to be production ready at end of sprint

• Design of scope with the above in mind

CODE DELIVERY MODEL

• Several build processes experimented

• Automated tools but still manual handoffs

• Monolith code base… but moving now to Microservices

• Goals:• Code coverage target – 80%

• Sprint completion target – 80%

• Defect rate below .1 defects per Story Point

• Weekly and monthly reports to track all this

CODE DELIVERY MODEL (Continued)

INTEGRATION

MASTER / DEV

Features A & B complete - merged into MASTER

Feature C incomplete -moves to next sprint

Features C, D, E merged to MASTER

MASTER (Sprint 01 content) branched off as Release 01

Production

Rel 01 moves from INT to RC phase

Rel 01 deployedto Prod

Production release dateEnd of cycle for Rel 01

Start of cycle

RC (Release Candidate)

Sprint 01 Sprint 02 Sprint 03

Rel 02 moves from INT to RC phase

MASTER (Sprint 02 content)branched off as Release 02

CODE DELIVERY MODEL (Continued)

• Developers only push code to Feature branches

• When ready to merge to MASTER, they send a request to Merge Contacts.

• Merging code into INTEGRATION requires a higher level of care • Needs approval from manager

• Merging code into RC requires extreme care • Needs approval from manager and director

WHERE WE ARE NOW

• We are now delivering a release reliably every 4 weeks

• We frequently achieve 80% of planned scope delivery

WHERE WE WANT TO GET TO

• Ability to deploy once a day in production automatically• Especially for defect fixes

• Ability to deploy without any downtime / outage

• Ability to complete releases in 1 sprint versus 3 sprints

• Ability to deploy only a component versus monolith

• Requires Automation of all testing and deployment pipeline

• Elimination of manual handoffs

DEVOPS

DevOps is:

• The collaboration between Development and Operations

• To identify efficiencies and techniques

• To Speed up the delivery of the release to the customer

DEVOPS (Continued)

SCALING CULTURE

Our program has:

• 15 teams (3 in India)

• ~ 100 developers

• ~ 40 testers

• ~ 10 Product Managers

• ~ 4 Project Managers

SCALING CULTURE (Continued)

How do we ensure everyone is pulling in the same direction…

…and that our practices scale across the entire org

• Our scaling methodology is home-grown

• Not designed from SAFe (Scaled Agile Framework)

• But naturally aligns with some of the SAFe concepts

• Alignment to a common schedule

• Alignment to a common code delivery model

• Alignment to a common process

Picture Source: Scrum Framework.png –Wikimedia Commons

ALIGNMENT TO A COMMON SCHEDULE:

• All teams have 4 week sprints

• Common Sprint in JIRA for entire program

• Same start date/time and end date/time

• Common merge deadlines for Release

ALIGNMENT TO A COMMON CODE DELIVERY MODEL:

• All teams merge code to Master

• Via a Team Stage stack

• That has to pass Lev 0 and Lev 1 tests

• Release branch is cut at end of sprint

ALIGNMENT TO COMMON PROCESS:

• Overall: One Release each sprint

• Overall: High level Release Plan ~6 weeks prior to sprint start

• Teams: Prioritize Stories per High level Plan

• Teams: Pre-Planning; Sprint Planning; Scrums

• Teams: Team End-of-sprint demo

• Teams: Retrospectives

• Overall: Solution End-of-sprint Demo

• Overall: Metrics collected and published

ALIGNMENT TO A COMMON PROCESS:

• Project managers collaborate to standardize practices

• Ops engaged throughout for close DevOps collaboration

• All teams follow similar Agile practices

• Priorities set by product management

• Product managers engaged throughout

• Dev and Test work closely together

Some key mindset shifts that we are working to enforce:

• Stories more granular

• Merge to Master more frequently

• Keep new code Feature Flagged until ready for release

• Fix defects earlier in the pipeline

• Virtual services framework for testing

Source: https://www.scaledagileframework.com/iteration-execution/

Questions?

Contact info: Shobha.Subramonian@sas.com

linkedin.com/in/shobha-subramonian-1967016

A Continuous Delivery Journey - RHAD · •We were to start continuous delivery in September 2015....

Documents