A Continuous Delivery Journey - RHAD · •We were to start continuous delivery in September 2015....

Copy r i g ht © S A S I nst i t ut e I nc. A l l r i g ht s r e se r v e d.

A Continuous Delivery Journeyand Scaling our Culture

SHOBHA SUBRAMONIANOCTOBER 18, 2018

Picture source: QuizzClub.com

2

INTRODUCTION

• A Case Study in moving to Continuous Delivery

• One organization’s journey: How we made the transition

• Practices that may work for others

C o p yright © SAS In sti tute In c. Al l r igh ts res erved.

3

AGENDA

• WHERE WE WERE

• OUR NEW DESTINATION

• CHALLENGES

• TRANSITION PLAN

• HYBRID APPROACH

• START OF THE JOURNEY

• ROCKY ROAD

• MINDSET SHIFT

• CODE DELIVERY MODEL

• WHERE WE ARE NOW

• WHERE WE WANT TO GET TO

• DEVOPS

*******************

• SCALING CULTURE


4

WHERE WE WERE

• On premise deployments

• Releases ~ once a year

• Long development cycle

• Agile sprints: Theoretically release-ready at end of each sprint

• However we were not actually release-ready!

• Big difference between being actually ready to release versus theoretically done


5

WHERE WE WERE (Continued)

………Release Planning, Scoping

Development & Test Iterations Last BuildSignoffRelease/Ship


6

OUR NEW DESTINATION

• Hosted environment on the cloud (using Amazon Web Services)

• Multi-tenant

• Single code base for all customers

• Short development cycle

• Continuous delivery

• Release every 4 weeks

• 4 week Agile sprints

• Goal: be release-ready at end of each sprint


7

Application(Source code, OS, Hardware)

AWSCustomer A

Customer B Customer C

OUR NEW DESTINATION (Continued)

Customer A’s data

Customer B’s data

Customer C’s data


8

OUR NEW DESTINATION (Continued)

Release Planning, Scoping

Development & Test

Release


9

CHALLENGES

• Typically the key to continuous delivery or release is:

• But Automation is not achieved in a day


10

CHALLENGES (Continued)

What is the automation we are looking for ?

• Automation of Unit testing

• Automation of Functional / End-to-End testing

• Automation of Regression testing

• Automation of Builds

• Automation of Deployment

• Automation of Deployment validation / Health checks

• Automated promotion and deployment to Stage

• Automated promotion and deployment to Production


11

CHALLENGES (Continued)

Other challenges:

• New leading-edge Cloud technology

• New Amazon Infrastructure Services and Platform Services

• Design of Architecture with the above

• Setup of various Instances / Environments

• Design of Code promotion process

• Development practices for Cloud, Multi-tenancy

• Security, Stability, Performance

• Extensive training; Some Hiring of new talent


12 Slide from

13

TRANSITION PLAN

1. Release an Early Adopter / Beta version in the cloud• ~10 months to build and deploy

• Allowed time for setup of new delivery framework

• Resulted in an operational product in the cloud

• With 2 Early Adopter customers

2. Make the leap to continuous delivery• Bringing on additional customers

• Move from Beta to full Production mode

• For this we arrived at a Hybrid approach.


14

HYBRID APPROACH

• We would have a release every 4 weeks

• However, every release would follow a 3-sprint (12-week) cycle

• 3 Sprints / Phases:• Story Development and Testing

• Integration Testing

• Release Candidate and Production Deployment

• Over time, we would reduce the length of this cycle


15C o p yright © SAS In sti tute In c. Al l r igh ts res erved.

16

HYBRID APPROACH (Continued)

• Dev & Test phase• Majority of work for release done in this phase

• Common sprint in Jira for all teams

• Cut over to Integration phase• Small team focused on Integration phase tasks

• Cut over to Release Candidate phase• Even Smaller team focused on RC phase tasks


17

START OF THE JOURNEY

• We were to start continuous delivery in September 2015.

• And put out a release every 4 weeks after that.

• We had a complex product suite that had:• 11 teams (most in Cary, NC; some in India, UK, and other locations)• Every team had 6-12 members (Dev + Test)• 4 project managers

• We set standards across the program: • Sprint process• Story template and Workflow• Code delivery


18

ROCKY ROAD

• Most teams had been together only a few months

• Still going through Storming, Norming - Transitioning to Performing

Source: Okpalad, based on Tuckman and Jensen


19

ROCKY ROAD (Continued)

• First continuous release did not come in September

• … or in October …

• Our scope was too large for one sprint

• Could not complete testing which was still largely manual

• We continued work to complete most of the scope

• And addressed build, deploy and other challenges

• We finally delivered in December – 3 months beyond plan


20


• Retrospectives to see how to reign in the scope

• Scope for next release slashed significantly

• Minimized dependencies between teams

• Scrum of scrums held for close inter-team collaboration

• Share point and Jira process standardization for consistent use

• Incorporation of Early Adopter feedback


21


• In 2016, we missed 4 out of 13 planned releases.

• In 2017, we only missed 1 out of 12 planned releases.

• (We did not do December releases anymore.)


22

MINDSET SHIFT

• Meet the need to get new features to customers quickly• One of the biggest benefits of continuous delivery

• Plan significantly smaller scope to finish in a sprint

• Deliver basic functionality first – additions/embellishments after

• Deliver few “done” stories instead of many half-finished stories

• Chunk up stories so they are “deliverable” in sprint

• Code to be production ready at end of sprint

• Design of scope with the above in mind


23

CODE DELIVERY MODEL

• Several build processes experimented

• Automated tools but still manual handoffs

• Monolith code base… but moving now to Microservices

• Goals:• Code coverage target – 80%

• Sprint completion target – 80%

• Defect rate below .1 defects per Story Point

• Weekly and monthly reports to track all this


24

CODE DELIVERY MODEL (Continued)

INTEGRATION

MASTER / DEV

Features A & B complete - merged into MASTER

Feature C incomplete -moves to next sprint

Features C, D, E merged to MASTER

MASTER (Sprint 01 content) branched off as Release 01

Production

Rel 01 moves from INT to RC phase

Rel 01 deployedto Prod

Production release dateEnd of cycle for Rel 01

Start of cycle

RC (Release Candidate)

Sprint 01 Sprint 02 Sprint 03

Rel 02 moves from INT to RC phase

MASTER (Sprint 02 content)branched off as Release 02


25

CODE DELIVERY MODEL (Continued)

• Developers only push code to Feature branches

• When ready to merge to MASTER, they send a request to Merge Contacts.

• Merging code into INTEGRATION requires a higher level of care • Needs approval from manager

• Merging code into RC requires extreme care • Needs approval from manager and director


26

WHERE WE ARE NOW

• We are now delivering a release reliably every 4 weeks

• We frequently achieve 80% of planned scope delivery


27

WHERE WE WANT TO GET TO

• Ability to deploy once a day in production automatically• Especially for defect fixes

• Ability to deploy without any downtime / outage

• Ability to complete releases in 1 sprint versus 3 sprints

• Ability to deploy only a component versus monolith

• Requires Automation of all testing and deployment pipeline

• Elimination of manual handoffs


28

DEVOPS

DevOps is:

• The collaboration between Development and Operations

• To identify efficiencies and techniques

• To Speed up the delivery of the release to the customer


29

DEVOPS (Continued)


30

SCALING CULTURE

Our program has:

• 15 teams (3 in India)

• ~ 100 developers

• ~ 40 testers

• ~ 10 Product Managers

• ~ 4 Project Managers


31

SCALING CULTURE (Continued)

How do we ensure everyone is pulling in the same direction…

…and that our practices scale across the entire org


32


• Our scaling methodology is home-grown

• Not designed from SAFe (Scaled Agile Framework)

• But naturally aligns with some of the SAFe concepts


33


• Alignment to a common schedule

• Alignment to a common code delivery model

• Alignment to a common process


Picture Source: Scrum Framework.png –Wikimedia Commons

34


ALIGNMENT TO A COMMON SCHEDULE:

• All teams have 4 week sprints

• Common Sprint in JIRA for entire program

• Same start date/time and end date/time

• Common merge deadlines for Release


35


ALIGNMENT TO A COMMON CODE DELIVERY MODEL:

• All teams merge code to Master

• Via a Team Stage stack

• That has to pass Lev 0 and Lev 1 tests

• Release branch is cut at end of sprint


36


ALIGNMENT TO COMMON PROCESS:

• Overall: One Release each sprint

• Overall: High level Release Plan ~6 weeks prior to sprint start

• Teams: Prioritize Stories per High level Plan

• Teams: Pre-Planning; Sprint Planning; Scrums

• Teams: Team End-of-sprint demo

• Teams: Retrospectives

• Overall: Solution End-of-sprint Demo

• Overall: Metrics collected and published



37


ALIGNMENT TO A COMMON PROCESS:

• Project managers collaborate to standardize practices

• Ops engaged throughout for close DevOps collaboration

• All teams follow similar Agile practices

• Priorities set by product management

• Product managers engaged throughout

• Dev and Test work closely together



38


Some key mindset shifts that we are working to enforce:

• Stories more granular

• Merge to Master more frequently

• Keep new code Feature Flagged until ready for release

• Fix defects earlier in the pipeline

• Virtual services framework for testing


39



Source: https://www.scaledagileframework.com/iteration-execution/

https://www.scaledagileframework.com/iteration-execution/

40



Source: https://www.scaledagileframework.com/iteration-execution/

https://www.scaledagileframework.com/iteration-execution/

41

Questions?


42

Contact info: [email protected]

linkedin.com/in/shobha-subramonian-1967016


https://www.linkedin.com/in/shobha-subramonian-1967016

Date post:	17-Jun-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

A Continuous Delivery Journey - RHAD · •We were to start continuous delivery in September 2015....

Documents