W4 DevOps Wednesday, October 17th, 2018 10:15 AM
The Test Expert's Role in DevOps
Presented by:
Mike Sowers
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Mike Sowers Coveros' Executive Vice President of Training, Mike Sowers has more than twenty-‐five years of practical experience as a global leader of internationally distributed test teams across multiple industries. Mike is a senior leader, skilled in working with both large and small organizations to improve their software development, testing, and delivery approaches. He has worked with companies -‐ including Fidelity Investments, PepsiCo, FedEx, Southwest Airlines, Wells Fargo, ADP, and Lockheed -‐ to improve development approaches, increase software quality, reduce time to market, and decrease costs. With his passion for helping teams deliver software faster, better, and cheaper, Mike has mentored and coached senior software leaders, small teams, and direct contributors worldwide.
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1
Agility. Security. Delivered.
The Test Experts Role in DevOps
Michael D. Sowers@michaelsowers4
Coveros, Inc.
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 2
Mike Sowers is Executive VP for Coveros and has more than twenty-five years of practical experience as a global leader of internationally distributed test teams across multiple industries. Mike is a senior leader, skilled in working with both large and small organizations to improve their agile software development, testing, and DevOps delivery approaches.
He has worked with companies—including Fidelity Investments, PepsiCo, FedEx, Southwest Airlines, Wells Fargo, ADP, and Lockheed—to improve development approaches, increase software quality, reduce time to market, and decrease costs. With his passion for helping teams deliver software faster, better, and cheaper, Mike has mentored and coached senior software leaders, project teams, and direct contributors worldwide.
Michael Sowers@MichaelSowers4
Trainer
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 3
About Coveros
• Coveros helps organizations accelerate the delivery of secure, reliable software using agile methods
• Coveros Services• Agile transformations & coaching• Agile development and testing• DevOps implementations• Software security analysis
• Agile, DevOps & Security training
• Software Products• SecureCI – open source CI/CD toolchain• Selenified – open source automated testing
framework
Areas of Expertise
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 4
Agenda
• DevOps?• Why DevOps• What is DevOps
• CI Testing• Process and Tasks
• CD Testing• Process and Tasks
• Summary
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 5
DevOps is a software development method that stresses communication, collaboration and integration between software developers, testers, and operations engineering.
DevOps extends the principles of Agileinto Operations
Like Agile, DevOps can be usedoutside software development toimprove IT operations in general
DevOps is NOT a tool or a particularprocess
Definition
What is DevOps?
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 6
• Systems Thinking – Optimize the entire system
• Amplify Feedback Loops – shorter, incremental feedback
• Culture of Experimentation and Learning From Gene Kim, Co-author of The Phoenix Project
DevOps Principles – “The 3 Ways”
What is DevOps?
https://itrevolution.com/the-three-ways-principles-underpinning-devops/
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 7
• Repeatable and Reliable Software Delivery• Increased Deployment Frequency
• Reduced Lead time for Changes
• Reduced Mean time to Recover (MTTR)
• Reduced Failure Rates
• Collective Ownership • Collaboration• Continuous Communication• Unified Processes and Tooling
• Continuous Improvement “Kaizen”
DevOps Goals
What is DevOps?
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 8
DevOps Mindset
Fundamental conflict - Dev vs. Ops
• Operations• Risk Minimization - Stability, SLAs,
Uptime, Response Time
• Scripts and procedures to install production system components
• Development• Rapid introduction of change
(delivery of features)
• Scripts and procedures to install development system components
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 9
Successful DevOps Methods and Practices
What is DevOps?
Continuous Deployment
Continuous Integration
Lean Product Management
Continuous Testing
Continuous Monitoring
Comprehensive Change Management
Dev Ops
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 10
What is DevOps?
DevOps Terminology
• Build automation – Automating the compilation, linking, and packaging of software applications
• Continuous integration – Leveraging build automation capabilities to integrate and test software on a regular basis to thwart integration and quality issues earlier in the process
• Check-in builds – Builds performed whenever code is changed. Often includes unit tests and code quality checks.
• Nightly builds – Integration builds performed nightly to assure nothing changed during the day breaks existing functionality. Regression testing included. Platform TBD.
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 11
What is DevOps?
DevOps Terminology
• Continuous delivery – A series of practices to assure code can be rapidly and safely deployed to downstream environments (QA, staging, production, etc.) by delivering every change to downstream environments while ensuring functionality works as expected through automated and manual testing.
• Continuous deployment – A series of practices to assure code can be rapidly and safely deployed to production by delivering every change to a production (or production-like) environment while ensuring functionality works as expected through automated testing.
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 12
What is DevOps?
Environments
• While continuously testing and building software is important for a robust final product, it is difficult to continuously release software while creating it.
• Having multiple environments allows for continuous development and low level testing to continue, while having more static versions of the software available for testers.
• Typically, a set of 4 or 5 environments will exist• Development• Dev Integration (CI)• QA• Staging• Production
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 13
What is DevOps?
Environments (cont.)
• Development• Local development / test environments where new code is created and
locally tested prior to checking changes into a source code repository
• Development Integration / CI• Updated continuously, always with the most up to date code that
passed all static analysis, unit tests, and code coverage criteria
• QA• Should be updated at least nightly, always with the last code to pass all
continuous integration and nightly regression testing
• Staging• Mimics production• Used for any testing that must be performed on production-like
environments to be valid
• Production• Updated for new releases / patches only, always with stable fully tested
code
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 14
DevOps Delivery Pipeline
Process of taking a code change from developers and getting it deployed
into production or delivered to the customerthrough a series of testing environments
• Key DevOps metric:• How long does it take for a change in
the code to get into production?
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 15
Delivery Pipeline (example)
Requirement
Code
Check-in
Unit Level, CI,
Regression
Typical Testing Activities with appropriate quality gates
Trigger
Deploy to QA/Test
Story & Use Case
LevelDeploy to Staging
System Integration
& Non-functionals
Deploy to Pre-Prod
Compliance &
Final AcceptanceDeploy to Prod
Early Rapid Feedback
No Late Surprises
A Balancing Act
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 16
Testing in DevOps
Testing activities are done across environments where appropriate
Testing here is:
Code focused Quality focused Delivery focused
In Sprints
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 17
Quality Gates
• Acceptance criteria that determines whether an application is ready to move to the next stage in a delivery pipeline
• A “good” quality gate does the following:• Enforces best practices
• Ensures team norms and expectations are enforced
• Continuously changes with continuous team improvements
• Finds a balance between enforcing quality, but not becoming too restrictive
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 18
Quality Gates and the Definition of Done
• Agile teams define a “Definition of Done” for User Stories, Sprints, and Releases as part of Initial Planning.
• Necessary amounts of testing for units, stories, integration of stories, use cases, etc.
• Code quality and security standards
• Performance standards
• Your Definition of Done needs to be part of your quality gates!
• Where specific ‘Done’ criteria reside in your delivery pipeline depends upon your environments and delivery process
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 19
“When we compared high performers to low performers, we found that high performers are doing significantly less manual work, and so have automated:
• 33 percent more of their configuration management.• 27 percent more of their testing.• 30 percent more of their deployments.• 27 percent more of their change approval processes." State of DevOps 2017
Benefits of Automation
Automation DOES make a difference
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 20
Continuous Integration is a software development practice where software changes are continuously integrated together and tested to
make sure a software application is always stable and software developers receive rapid feedback if there is a quality problem
What is Continuous Integration?
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 21
Continuous Integration within DevOps
Continuous Integration Continuous Delivery
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 22
Automate Test as Part of Continuous Integration
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 23
Typical Continuous Integration Pipeline
Local
Development
Source code control
Jenkins
Check-in Builds
Integration
To
Continuous
Delivery
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 24
CI Step 1: Local build and test
To
Continuous
Delivery
Jenkins
Check-in Builds
IntegrationLocal
Development
Source code control
Typical activities1. Checkout code2. Modify code3. Unit test4. Merge code5. Run local tests
- unit- acceptance- security- performance
4. Check in code
Test Experts Role/Tasks:• Help define success criteria
for local• Assist with environment
design/configuration• Contribute to Unit Test
Design• Participate in Reviews• Design, Implement &
Automate Story Acceptance Test
• Assist with dependency management
• Ensure critical Non-Functionals are being tested
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 25
CI Step 1: Local build and test
To
Continuous
Delivery
Jenkins
Check-in Builds
IntegrationLocal
Development
Source code control
Typical activities1. Checkout code2. Modify code3. Unit test4. Merge code5. Run local tests
- unit- acceptance- security- performance
4. Check in code
Testing Tip: The more production-like our local environments are, the earlier we canbegin testing non-functional requirements like security and performance
Testing Tip: Have the developers integrate their code locally prior to check in and runstory acceptance tests to validate stories as they go. Test with them too.
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 26
CI Step 2: Validate code quality & integrate
To
Continuous
Delivery
Local
DevelopmentIntegration
Source code control
Jenkins
Check-in BuildsTypical activities1. Jenkins detects
code changes2. Compiles code3. Runs quality gate
tests- unit & coverage- acceptance- code analysis- security
4. Packages app ifall quality gatespass
Test Experts Role/Tasks:• Help define success
criteria for check-in• Understand Static
Analysis Results (risks)• Analyze Code Coverage• Participate in Reviews• Design, Implement &
Automate Story Acceptance Test
• Validate definition of done
• Contribute to regression strategy
• Ensure critical Non-Functionals are being considered
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 27
CI Step 2: Validate code quality and integrate
To
Continuous
Delivery
Local
DevelopmentIntegration
Source code control
Jenkins
Check-in BuildsTypical activities1. Jenkins detects
code changes2. Compiles code3. Runs quality gate
tests- unit & coverage
- acceptance- code analysis- security
4. Packages app ifall quality gatespass
Testing Tip: Check-in builds should take no longer than 10 minutes as developersshould be waiting for feedback before beginning new work
Testing Tip: Security testing must be lightweight to be useful during check-ins. Don’ttry to run comprehensive scans at this step
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 28
CI Step 3: Validate code integration
To
Continuous
Delivery
Local
Development
Source code control
Jenkins
Check-in Builds
IntegrationTypical activities1. Provision envs2. Install / conf
application3. Run smoke test4. Run quality gate
tests- acceptance- lightweight
security- performance
5. Lightweightregression
Test Experts Role/Tasks:• Help define success
criteria for environments• Analyze critical coverages
(design, requirements, function, interface, etc.)
• Define, Implement and Automate Smoke Tests
• Design, Implement & Automate Story Acceptance Test
• Validate definition of done
• Design, Implement & Automate regression tests
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 29
CI Step 3: Validate code integration
To
Continuous
Delivery
Local
Development
Source code control
Jenkins
Check-in Builds
IntegrationTypical activities1. Provision envs2. Install / conf
application3. Run smoke test4. Run quality gate
tests- acceptance- lightweight
security- performance
5. Lightweightregression
Testing Tip: All testing during CI must be automated
Testing Tip: If you are able to use cloud-based services, consider parallelizing yourCI process to enable more testing during your CI window (hourly, nightly)
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 30
CI Step 4: Package release candidate
Local
Development
Source code control
Jenkins
Check-in Builds
IntegrationIf all quality gates pass:1. package (and tag) application as a
release candidate2. Destroy integration environment3. Enter continuous delivery process
If any quality gates fail:1. Notify development team of errors2. Keep integration envs available for
debugging
To
Continuous
Delivery
Test Experts Role/Tasks:• Help define success
criteria for quality gates
• Help define, implement and maintain team quality dashboard
• Ensure appropriate configuration management
• Contribute to root cause analysis and continuous improvement
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 31
Continuous delivery – A series of practices to assure code can be rapidly and safely deployed to downstream environments (QA,
staging, production, etc.) by delivering every change to downstream environments while ensuring functionality works as expected
through automated & manual testing.
Continuous deployment – A series of practices to assure code can be rapidly and safely deployed to production by delivering every change to a production (or production-like) environment while ensuring functionality works as expected through automated
testing.
What are Continuous Delivery & Deployment?
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 32
Continuous Integration within DevOps
Continuous Integration Continuous Delivery
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 33
Automate Test as Part of Continuous Delivery
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 34
CD Step 5: Test release candidateFrom
Continuous
Integration
Staging
ProductionQA
Jenkins
Typical activities1. Choose release candidate
to test2. Provision environments3. Install / conf application4. Run smoke test5. Run QA tests
- Acceptance / Explore- Performance- Secure code scan- Full security test
6. If quality gates pass,mark release ‘ready forstaging’
Test Experts Role/Tasks:• Help define success
criteria for release candidate
• Help define environment and application configurations to be tested
• Track coverages • Define, Implement and
Automate Smoke Tests• Design, Implement &
Automate Story Acceptance Test
• Run Non-Functional Tests• Design, Implement &
Automate regression tests
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 35
CD Step 5: Test release candidateFrom
Continuous
Integration
Staging
ProductionQA
Jenkins
Typical activities1. Choose release candidate
to test2. Provision environments3. Install / conf application4. Run smoke test5. Run QA tests
- Acceptance / Explore- Performance- Secure code scan- Full security test
6. If quality gates pass,mark release ‘ready forstaging’
Testing Tip: While there are applications wherein the CD testing process can becompletely automated, it is the exception and NOT the rule
Testing Tip: Consider integrating your QA and Staging Smoke Tests into CI ifenvs are elastic as it will identify issues earlier and save testers time
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 36
CD Step 6: Test against production envFrom
Continuous
Integration
QA Production
Jenkins
StagingTypical activities1. Compliance testing2. Full load test3. Red team /
penetration test4. Rollback procedure
test OR A/B switchover test
5. If quality gates pass, production ready release candidate
Test Experts Role/Tasks:• Help define success
criteria for compliance• Run Full Non-Functional
Tests• Validate
Rollback/Recovery• Test Feature Switches• Run full Story Tests• Run full Regression Test• Tests all critical
configurations• Validate any
dependencies• Validate monitoring
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 37
CD Step 6: Test against production envFrom
Continuous
Integration
QA Production
Jenkins
StagingTypical activities1. Compliance testing2. Full load test3. Red team /
penetration test4. Rollback procedure
test OR A/B switchover test
5. If quality gates pass, production ready release candidate
Testing Tip: Many downstream testing activities performed in Staging becomeredundant as you gain confidence in the quality of release candidates
Testing Tip: It is critical to effectively test your rollback / recovery processes
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 38
CD Step 7: Deploy to productionFrom
Continuous
Integration
QA
Staging
Production
Jenkins
Typical activities1. Automated / push
button deployment2. Deployment of
reliability monitoring3. Deployment of
security monitoring
Test Experts Role/Tasks:• Help define success
criteria for production deployment
• Run Production Implementation tests
• Validate data/data migrations
• Validate monitoring
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 39
Wrap-Up
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 40
Questions?
Michael D. Sowers
@MichaelSowers4
Coveros, Inc.