© Copyright 2015 EMC Corporation. All rights reserved. 1© Copyright 2015 EMC Corporation. All rights reserved.
© Copyright 2015 EMC Corporation. All rights reserved.
DevOps is a ReorgHow To Accelerate The Change
Don Demcsak
East Region – DevOps Process Lead
Twitter - @DonDemcsak
© Copyright 2015 EMC Corporation. All rights reserved.
CHANGE IS THE NEW BUSINESS NORM3© Copyright 2015 EMC Corporation. All rights reserved.
© Copyright 2015 EMC Corporation. All rights reserved.
AGILITY – BAKING CHANGE INTO IT
Organization Design
OrgAgility
Business Agility
People Process Tools
ITAgility
© Copyright 2015 EMC Corporation. All rights reserved. 5© Copyright 2015 EMC Corporation. All rights reserved.
STRATEGIES FOR MANAGING CHANGE
© Copyright 2015 EMC Corporation. All rights reserved.
• From Silos to Delivery Teams– Outcomes over Activities
• This change isn’t “solved” with a technology change
• This change is evolved through:
People + Process + Technology
Culture!
THE FOCUS OF IT NEEDS TO CHANGE
© Copyright 2015 EMC Corporation. All rights reserved.
PEOPLE + PROCESS + TECHNOLOGIES
Aligning ALM and Cloud strategies
© Copyright 2015 EMC Corporation. All rights reserved.
UNDERSTANDING CHANGES IN IT
• Fictional story about today’s IT challenges
• Explains how Operational and Cultural changes save the day.
• Based on “The Goal: A Process of Ongoing Improvement” book
• Mark Burgess, a pioneer of system configuration management. Creator of CFEngine.
• What Science tell us about Information Infrastructure video (Video)
© Copyright 2015 EMC Corporation. All rights reserved.
BI-MODAL or TRI-MODAL IT?
TRI-MODAL (SIMON WARDLEY)BI-MODAL (GARTNER)
BUSINESS
IT MANAGEMENT
LEGACYAPPLICATIONS
NEW APPS
© Copyright 2015 EMC Corporation. All rights reserved.
• Eliminate non-value-added action (Waste/Muda)
• Pull over Push
• Kaizen (Continuous Improvement)
• Kaikaku (Disruptive Change)
• Small Batch + Experimentation
• Kanban (Try www.trello.com )
• Iterative vs. Incremental
ADOPTION OF LEAN PRACTICES
http://en.wikipedia.org/wiki/Lean_startup
© Copyright 2015 EMC Corporation. All rights reserved.
BUILD PROJECTS ITERATIVE VS. INCREMENTAL
Painting of the Mona Lisa
Woman in Pastoral Settings
1 2 3
Incremental
Iterative
© Copyright 2015 EMC Corporation. All rights reserved.
the “WHAT” – defines outcomes; setsenterprises standards to maintain qualityand provide a consistent work environmentacross the enterprise
EMPLOY A TOP DOWN & BOTTOMS UP CHANGE MODEL
SUCCESS REQUIRES A SHARED VISION
CHANGE ADOPTION
LEADERSHIP TEAM
DELIVERY TEAM
the “HOW” – determines the bestapproach for achieving the standardsdefined at the enterprise level
© Copyright 2015 EMC Corporation. All rights reserved.
• Reduce cycle time by 300%
• Reduce recovery/rollback time to less than 4hrs
• Improve process efficiency by 10%
• Reduce break/fix workload from 50% to 30%
• Solution is extensible and parameterized
SUCCESS CRITERIA & MEASURESDEFINE THE OUTCOMES OF TRANSFORMATION
© Copyright 2015 EMC Corporation. All rights reserved.
Release Candidate Pipelines
DELIVERY PIPELINES
VERSIONCONTROL
COMMITCREATECHANGE
• New software• Enhancements/fixes• Upgrades• Patches• Configurations
TEST UAT STAGE
PROD
Development & Build Process
1.1
1.2
1.3
1.4
1.5
1.2
Automated Tool Chain
© Copyright 2015 EMC Corporation. All rights reserved.
START SMALL AND SCALE WITH CONFIDENCE
BUILDING PIPELINES - INCREMENTAL METHODOLOGY
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APPAPP
APP
APPAPP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APPAPP
APP
APPAPP
APP
APP
Single Application(Project Team)
Dependent Applications(Program Teams)
10-100s Applications(Line-of-Business Portfolio)
100-1000s Applications(Enterprise)
PH
AS
E
1
PH
AS
E
2P
HA
SE
4
PH
AS
E
3
GOAL
GOAL
GOAL
GOAL
Goal+ +
+ +
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
InfrastructureProduction-like environments can be prepared
thru version controlled scriptsHeterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery team
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
Infrastructure
App Architecture
Production-like environments can be prepared thru version controlled scripts
Heterogeneous, managed by operations
Production-Standardized PaaS or IaaS provided by delivery team
Everything deployed together
Single packet per product/web property
Strict SOA with forward/backward compatibility
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
Infrastructure
App Architecture
Testing
Production-like environments can be prepared thru version controlled scripts
Heterogeneous, managed by operations
Production-Standardized PaaS or IaaS provided by delivery team
Everything deployed together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that owns functional test automation
Fast, comprehensive automated tests at unit and functional level organized in a deployment pipeline
Developers & Testers maintain automated functional test together
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
Infrastructure
App Architecture
Testing
Databases
Production-like environments can be prepared thru version controlled scripts
Heterogeneous, managed by operations
Production-Standardized PaaS or IaaS provided by delivery team
Everything deployed together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that owns functional test automation
Fast, comprehensive automated tests at unit and functional level organized in a deployment pipeline
Developers & Testers maintain automated functional test together
Manual migrations
Delta scripts for db, rollbacks practiced
Apps are forward/backward compatible with dbversions by design (using expand/contract)
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
Infrastructure
App Architecture
Testing
Databases
Releases
Branch Model
Production-like environments can be prepared thru version controlled scripts
Heterogeneous, managed by operations
Production-Standardized PaaS or IaaS provided by delivery team
Everything deployed together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that owns functional test automation
Fast, comprehensive automated tests at unit and functional level organized in a deployment pipeline
Developers & Testers maintain automated functional test together
Manual migrations
Delta scripts for db, rollbacks practiced
Apps are forward/backward compatible with dbversions by design (using expand/contract)
Release trains
Dark launching, Blue-green deployment, canary releasing
Developers push their own changes
Develop on branchesMerge to release branches
Then branch again after release
Develop on trunkBranch for Release
Develop on trunkRelease from trunk
Pull Requests toa Release Branch
© Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENTRelease Frequency
1 Release Every 100 Days
1 Release Every 10 Days
Daily Release 10 Releases a Day
100 Releases a Day
Infrastructure
App Architecture
Testing
Databases
Releases
Branch Model
Production-like environments can be prepared thru version controlled scripts
Heterogeneous, managed by operations
Production-Standardized PaaS or IaaS provided by delivery team
Everything deployed together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that owns functional test automation
Fast, comprehensive automated tests at unit and functional level organized in a deployment pipeline
Developers & Testers maintain automated functional test together
Manual migrations
Delta scripts for db, rollbacks practiced
Apps are forward/backward compatible with dbversions by design (using expand/contract)
Release trains
Dark launching, Blue-green deployment, canary releasing
Developers push their own changes
Develop on branchesMerge to release branches
Then branch again after release
Develop on trunkBranch for Release
Develop on trunkRelease from trunk
Pull Requests toa Release Branch
© Copyright 2015 EMC Corporation. All rights reserved.
WHAT DO WE DEFINE AS INFRASTRUCTURE?
© Copyright 2015 EMC Corporation. All rights reserved.
© Copyright 2015 EMC Corporation. All rights reserved.
© Copyright 2015 EMC Corporation. All rights reserved.
© Copyright 2015 EMC Corporation. All rights reserved.
• Deploy, monitor and connect together all the pieces needed to run services for the organization
• Processes and tools for faster end-to-end delivery of quality services
• Automation comes built-in
WHY DEFINE INFRASTRUCTURE AS CODE?
© Copyright 2015 EMC Corporation. All rights reserved.
• Desired state specified in text files
• Autonomic (self-corrects to desired state)
• State should be known through monitoring
• Remove snowflake servers
HOW DO WE DO THIS?
© Copyright 2015 EMC Corporation. All rights reserved.
• Easy to read and edit
• Shareable
• Can use standard version control like Git or SVN
• Becomes executable documentation
WHY STORE THEM IN TEXT FILES?
© Copyright 2015 EMC Corporation. All rights reserved.
node 'www2' {
class { 'apache': } # use apache module
apache::vhost { 'awesomewebsite.com': # define vhost
port => '80',
docroot => '/var/www/html'
}
}
SIMPLE EXAMPLE
© Copyright 2015 EMC Corporation. All rights reserved.
user { root:
ensure => present,
password => '$ecretP@ssw0rd',
}
CHANGE ROOT PASSWORD
© Copyright 2015 EMC Corporation. All rights reserved.
MANUALLY CONFIGURED ENVIRONMENTS ARE LIKE A HOUSE
OF CARDS IN A CHINA SHOP
- NEAL FORD
© Copyright 2015 EMC Corporation. All rights reserved.
WHAT CAN HAPPEN IF YOU DON'T TREAT
INFRASTRUCTURE AS CODE?
© Copyright 2015 EMC Corporation. All rights reserved.
UP TO 60% OF FAILURES ARE CAUSED BY HUMAN ERROR, NOT
HARDWARE FAILURE
© Copyright 2015 EMC Corporation. All rights reserved.
EXAMPLE - KNIGHT CAPITAL GROUP
https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg
© Copyright 2015 EMC Corporation. All rights reserved.
• Manual deployment of new trading software
• 7 of 8 servers correctly updated
• Old function still alive on the 8th server led to…
KNIGHT CAPITAL GROUP - WHAT HAPPENED?
© Copyright 2015 EMC Corporation. All rights reserved.
$440 MILLION LOSSIN 45 MINUTES
© Copyright 2015 EMC Corporation. All rights reserved.
SO
© Copyright 2015 EMC Corporation. All rights reserved.
TREAT YOUR INFRASTRUCTURE AS
CODE