DW1 DevOps Transformation Wednesday, November 7th, 2018 11:30 AM
Applying Systems Thinking to DevOps Practices at Scale
Presented by:
Jared Speno Nationwide Insurance
‘
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Jared Speno As enterprises begin to adopt DevOps practices, there can be a tendency for these transformations to err in one of two ways. One model is where everything is driven locally by teams trying new concepts and tools in order to accelerate their delivery. Another model is where a very top-‐down program is initiated, driven by what leaders believe are the right things for teams to do. Jared Speno will show how Nationwide Insurance applied DevOps concepts at scale to accelerate delivery for the business. Learn how the organization created a structure and culture that empowers teams to own their journey while identifying areas where they can improve. You'll learn how to apply systems thinking to take local improvements and use them to optimize enterprise delivery capability, why it is necessary to treat the delivery pipeline as a product, and how to architect the pipeline for speed of delivery. You'll leave this talk with lessons learned from a successful DevOps implementation at a large company, as well as ideas for kickstarting your own initiative.
11/7/18
1
Nationwide’s Journey to Accelerated Delivery
November 7, 2017
About Nationwide
16+ MILLION
POLICIES $400 MILLION
CONTRIBUTED TO NONPROFITS
AND COMMUNITIES SINCE 2000
$209.8 BILLION IN ASSETS
7th
LARGEST COMMERCIAL
INSURER
34,000 Nationwide associates serve customers in every state
$17 BILLION
#68 ON THE
FORTUNE 500 LIST
PAID IN CLAIMS AND BENEFITS IN 2016
11/7/18
2
NATIONWIDE IT
#27: COMPUTERWORLD BEST PLACES TO WORK IN IT
250 AGILE / LEAN STANDING TEAMS >2500 APPLICATIONS
21 BUSINESS UNITS
5500 IT ASSOCIATES
OF
AND
BALANCE
INNOVATION
DISCIPLINE DEVELOPERS & TESTERS
>2700
65 PRIMARY
TECHNOLOGIES
Our Journey
11/7/18
3
4
Why Speed Matters
Based on experiments at MicrosoK, 60%-‐90% of ideas do not improve the metrics they were intended to improve.
All ideas tested were thought to be good ones, but neither intui:on nor expert opinion are good gauges of the value our ideas have for users.
Source: Lean Enterprise: How High Performance OrganizaWons Innovate at Scale
11/7/18
4
What is DevOps?
DevOps Model Lines
Issues/Blockers
DevOps LeadershipTeam
Goals, Scope, Systems Thinking
DevOps Platform Team
Team 1Backlog
Team 1 Team 2A/B Tests
DevOps Product Backlog
Team 2Backlog
Platform Backlog
Governance Team
11/7/18
5
Guiding Principles
Version Control Everything
Fast and Reliable Automated Tests
Continuous Integration to the Trunk
Automated Infrastructure
Automate/Enable Low-Risk Releases
Architect for Low-Risk Releases
Monitor Everything
• Code Merge / GitHub features incl. Pull Requests : 8 Hours to 45 Minutes using GitHub
• Lead Wme for data changes 68 Hours to 1 Hour, reduced hand-‐offs with Liquibase
Automated DB Deployments
Version Control Everything
11/7/18
6
• On-‐demand PT tesWng – reduced lead Wme from 90 days to 2 hours by self-‐service enabling the process
• WriWng staWc analysis checks to catch coding issues known to cause producWon outages – 5 emergency releases in 2016 cosWng 5 person weeks to fix, down to 0 in 2017
• Pull quality checks leK – unit tests, code coverage and staWc analysis run and published to pull requests
Fast and Reliable Automated Tests
• Dark launching all code into producWon – reduced lead Wme from 97 – 45 days
• Deployment chat bot – automated all build/deployment steps in Rocket.Chat, so steps can be accomplished by simply saying, “build and deploy master into ST”
• Release when ready – Started doing deploys during the day using blue/green deployments. Implemented 50 producWon releases over the course of 3 months, vs 3 in Wme-‐based approach
Automate/Enable Low Risk Release
11/7/18
7
• Build health monitor – improved visibility for broken builds allowing the team to address much quicker (within minutes, instead of hours)
Monitor Everything
11/7/18
8
15
Hyper-‐local DevOps hack days
Seeding change agents on lines to accelerate progress
Book clubs drive culture
Data deployments as a first-‐class ciWzen
SELF-‐SERVICE DEVOPS
11/7/18
9
Platform team
PPM
• Demand, Resource, Porkolio Management
Release Planning
• Agile Lifecycle Management
• Release Management
Design / Develop
• IDEs • StaWc code analysis
• Source Code Management
• Build • Dependency management
Integrate
• ConWnuous IntegraWon
• Bots • Packaging • ArWfact Management
Readiness CerWficaWon
• Quality management
• Performance management
• Security management
Deploy
• Infrastructure AutomaWon
• ConWnuous Delivery
Monitor & Feedback
• APM • Log management
• Incident & event management
Developer CollaboraWon, DevOps Tool IntegraWon
Release Planning – a porJon of Monitor
11/7/18
10
Gamification Schedule
April Most mature Splunk adoption
March New developer setup time
June Most mature GitHub adoption
May Most mature New Relic adoption
August Best quality left/automation in CI pipeline
July Most mature chatbot/Rocket.Chat adoption
October Shortest lead time
September Automation of a manual process
December Lowest change failure rate
February Number of community contributions
November Shortest mean time to recover (MTTR)
January Highest rate of deployments
High volume of flooded cars
Business ask: opWmize online claims process
IT response: collaboraWve design and implementaWon in 8 hours
Results: streamlined online claims process by 40%
HURRICANE HARVEY
11/7/18
11
“There is surely nothing quite so useless as doing with great efficiency what should not be done at all”
-‐ Peter Drucker
11/7/18
12
Thank you