Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
The Mega Actionable Continuous Testing Workshop
Client Services Consultant – DevOps
Surya Suravarapu
2 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
• Introduction – What You’ll Learn
• The CT Mega Actionable Report
• Continuous Testing – What & Why
• Segment 1: The CT Framework Revealed– People
– Process
– Technology
• Segment 2: Making Continuous Testing Mega Real
• Conclusion, Raffle and Wrap-up
The Mega Actionable CT Workshop - Today’s Agenda
3 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Defining the end to end Value Stream - DevOps
Ideation
Customer
Experience
Continuous Testing
Test Data /Synthetic Data
Test CaseCreation
PerformanceTesting
Mocking& Virtual Services
Continuous Integration Operations
CapacityManagement
WIP
BuildCode
Synthetics
ServiceManagement
Monitoring
Transformation Intelligence
Business
Strategy
XLHigh Level
Planning
Backlog
Features
Stories
BehaviorModeling
Test Scripting
Test Management
Continuous Delivery
Release
Orchestration
CD Automation
API Governance &Management
Quality Management
Test Planning
Tracking
OKRs
Platform / Cloud Management (Kubernetes)
Platforms
4 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Reality: DevOps Success Is Difficult and Elusive
CEOs not happy
with the speed of
software delivery
78
%
Lost revenue and inability to
respond to market opportunities
=
Accelerate innovation
Delivery Takes Too Long
CEOs that say
software quality
needs to
improve
77
%
Bad software can cost billions
and result in irreparable
damage
+
Reduce operational risk
Quality Still Unpredictable
Very successful
at quick
development and
implementation
Neutral
Unsuccessful at
quick
development and
implementation
Only 1/10 organizations have mastered the balance between innovation and risk
Source: Freeform Dynamics 2018, How Agile and DevOps enable digital readiness and transformation
Source: Harvard Business Review Analytic Services 2019, Competitive Advantage Through DevOps: Improving Speed, Quality, and Efficiency in the Digital World
5 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Common Challenges With DevOps Success
Massive Complexity Abundant Bottlenecks Low Visibility Enable or Protect
6 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Key Challenges: Balancing Velocity and Risk at Scale
Traditional testing delays
More than any other part of the DevOps process
Automation needs intelligence
To provide quick feedback on new code changes
Need to proactively assess deployment risks
Lack of insight leads to compromising velocity for stability
Are we going in the right direction?
We are moving fast – but is it delivering the business outcomes we expect?
Inability to Assess Risk
Automation Can Help,
But…
55% more tests than they need
Testing Bottleneck
Align Velocity with Value
56% gaps in test
case coverage
Enablethe business
Protectthe business.
7 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Solution: Modern, Intelligent DevOps Process and Pipeline
1
Intelligent Automation
to improve overall visibility,
velocity, and quality
Wasted
Time
AI-Powered, Intelligent Delivery Pipeline
Ris
k
Data-Driven, Predictive Insights, Risk Remediation
Testing
“Delays”
Modern Testing with Shift Left / Shift Right Practices
2
Embed quality
From requirements through
production to eliminate existing
QA roadblocks
3
Continuous
Improvement through ML
for business processes and
and site reliability
8 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
What is Continuous Testing?
The practice of testing
across every activity in the SDLC to uncover and fix
unexpected behaviors as soon as they are injected.
9 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Delivery Velocity is key for High Performers
HIGH PERFORMERS
2018 State of DevOps Report
Deploy frequency (# of deploys per year)
LOW PERFORMERS
Leaders are deploying more frequently than laggards
2019 Continuous Testing Report
2%
11%
18%
29%
28%
11%
Several times per hour
Hourly
Daily
Weekly
Monthly
Less often than monthly
The frequency at which teams would like to deploy new builds
10 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
But QA Is THE Bottleneck.....
10
http://www.computing.co.uk/digital_
assets/634fe325-aa28-41d5-8676-
855b06567fe2/CTG-DevOps-
Review-2017.pdf
Where are the main hold-ups in
the software production
process?
32%
16%22%
63%
21% 23%30%
Plan code Pre-test Build Test/QA Release Deploy Review/Improve
“
63% agree
Traditional Testing Tools Are
Too Complex & Expensive
70%Of Testing Today Is
Still Manual*
*Bloor Research
Today’s Testing Center
Of Excellence Is A
Bottleneck To Speed(Not built for Agile approach)
11 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Traditional SDLC Process
Service Management
APIs
Customer
ProductionApplications
Security
Test Data/ Synthetic Data
Test CaseCreation
Functional & Performance
Testing
Virtual Service
DEVELOPMENT
Code BuildPortfolio & Project
Prioritization
Build Business
Case
Requirements Definition & Management
PLANNING
Monitoring
TESTING
OPERATIONS
Testing as a time-boxed event, in a linear
process.
Ideas
12 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Testing Embedded Throughout SDLC
TESTING
Test CaseCreation
Test Data/ Synthetic Data
Functional & Performance
Testing
Virtual ServiceMonitoring
Continuous Feedback & Collaboration
Service Management
APIs
Customer
ProductionApplications
Security
DEVELOPMENT
Code BuildPortfolio & Project
Prioritization
Build Business
Case
Requirements Definition & Management
PLANNING
OPERATIONS
Ideas
Continuous Testing is the embedding of testing as a fundamental and ongoing aspect of every activity though the software delivery cycle
13 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
The goal: in-sprint Agile Testing
With Continuous Testing, testing keeps pace with Agile development.
Release Release Release Release Release
Traditional testing presents barriers to speed with quality.
Source: Dimensional Research, Continuous Testing – Companies Don’t Know What They Don’t Know, 2018
14 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Testing Key Pillars
Requirements Quality
Pipeline Quality
Code and application
quality
Customer Experience
• Can we deploy & release apps across environments rapidly and correctly?
• Do we have the right tests defined and automated as part of the pipeline?
• Did we build it right?• Are we testing and validating all
along the way?• Quality?
• Are users perceiving value in the application delivered?
• Are we building the right things?• Are we testing the right things?
Testing
Lifecyle
Automation
15 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
General Approach to implementations
Think BIGStart Small
Learn fastScale thru success
Transform
16 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Testing Maturity Model Summary
MA
TU
RIT
Y
Initial
Basic
Automated
Integrated &
Orchestrate
d
Intelligent
BUSINESS VALUE
LEVEL 0 LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4
?
Continuous
Optimization
End-to-End
Orchestration
Consistency &
Collaboration
Heroes &
Heavy Lifting
17 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Business Impact and Metrics Summary
L0 Initial
Unpredictable quality, high costs
• Manual testing, minimal
automation
• High QA cost
• Depend on SME,
individuals to determine
quality
• Un-scientific measures
of test and data
coverage
• Poor/unpredictable
quality
L1 Basic
Reduced cost; improved code quality
• Requirements Ambiguity ↓
• Test design effort ↓
• Test maintenance effort,
change analysis effort ↓↓
• Test coverage ↑↑
• Unit test coverage ↑
• Code quality ↑↑
• Defect leakage from Dev
to Test ↓
• Test data coverage
• Test environment setup
time ↓
• Start implementing Shift
Left activities
L2 Automated
Improved dev/test agility; improved app quality
• Embrace Shift Left
activities
• Test automation level ↑↑
• Well defined
Automation
Frameworks
• Test execution time ↓↓
• Test feedback response
time to Dev ↓↓
• Defect leakage from
Integration ↓↓
• Test data provision time
↓↓
• Test environment setup
time ↓↓
L3 Orchestrated
Improved release agility
• Automate Everything
Test automation level ↑↑↑
• Test execution time ↓↓↓
• Continuous Testing a key
part of DevOps pipeline
• Defects due to
environment mismatch ↓↓
• Release quality: defect
leakage to production ↓↓↓
L4 Intelligent
Optimization and predictability
• Defect prevention
• Test scoping and
maintenance effort ↓↓↓
• Test prioritization
• Dynamic test selection
Maturity Level
Cap
ab
ilit
ies
Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Segment 1, Part 1: The People, Organization and Culture
19 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
The organizational shift to true Agile Continuous Testing
New org models require new collaboration skills with multiple personas
How does the COE fit into Agile
Center of Excellence or Enablement - Shift Left
+ + +
Produ
ct
Owne
r
Devel
opers
Tester
s
Produ
ct
Owne
r
Devel
opers
+
Tester
s
Agile
Team
Diverse goals Aligned goals Common goals
Aspiring Agile Testing– Product Owner +
Developers +
Independent Testers
Traditional Agile Testing– Product Owner + (Developers +
Embedded Testers)
20 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
People – Changing the Status Quo
• Change takes time, not always embraced, but it is possible
• Personalities are the biggest challenge
• Leaders & Early Adopters, buy in help drive changes
• Have to address Negatives…can’t let them grow
• Start with the WHY
• Desire to adopt a Continuous Testing mindset
• Company, Team and Individual benefits
• The HOW
• Developers and testers can work better together
• Bring both teams together on the same floor or workspace, if possible
• Regular meetups to ensure collaboration
• Define Team Goals, dependencies on each other
• The WHAT
• Deliver quality code more quickly, reduce defects
• Customer / End User benefits is the main goal
21 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
People – The Start of Your Continuous Testing Journey
• Acknowledge this is a transformation and expect it to be
uncomfortable for people, the goal is to adapt and thrive, find
the people that are up to the challenge
• Empower developers with testing capabilities
• Developers and agile teams need to understand testing is
part of their responsibility
• Improve the relationship between tester and each developer
• Use your technically aware testers to train the developers to
build relationships and to transfer good testing skills
• Keep the teams small – easier to build those relationships
• Team goals, rewards not individual
• Encourage inter-team collaboration
• Make reports easy to access and share online
22 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Who is Software Development Engineer in Test (SDET)
• Tester who is deep skilled in software development
• Has deep development skills
• Works closely with developers/development team
• Addresses quality at the code/component level
• Primarily focuses on testing frameworks and automation
• Focus on “Build to Run/Deploy/Perform vs Test to….”
23 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Typical adoption challenges• People / Process / Cultural challenges
• Documented value – even if compelling -- is not sufficient to drive adoption
• Lack of agreed transformation approach and roadmap
• Lack of sense of urgency
• Org and alignment issues
• Goals and metrics issues
• Skills/People issues
• Don’t have skills in process/tools/integrations, etc
• Technical/integration complexity
• Most customer already have some DevOps tools and tool-chains
• Some new tools are sometimes difficult to integrate – from process/data perspective
• Some new tools are difficult and complex to use
• Different teams want to use different tools
• Existing assets not easy to migrate
• Value not documented
• Biz value not documented despite project level success
24 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - People
Question #1: Does your entire organization universally understand the definition of continuous testing? (select one)
A. Some of our team understands the definition of CT but not all of them
B. Most of our team understands the definition of CT
C. All of our team understands the definition of CT
D. There is still confusion about what CT is exactly
25 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your People – Survey #1
26 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - People
• Question #2: How well do your developers do their own testing? (select one)
• A. Our developers perform complete and thorough unit testing
• B. Our developers occasionally do unit testing of high risk code
• C. Our developers are starting to do more unit testing
• D. Our developers don’t do much testing and the burden of quality is on QA
27 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your People – Survey #1
28 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - People
Question #3: How technically aware are your testers? (select all that apply)
• Both our testers and developers can code an automated test
• Our testers can collaborate with our programmers
• Our testers can articulate to their managers why an approach the development team is taking is not the best option
• None of the above
29 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your People – Survey #1
30 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - People
Question #4: How Agile is your team? (select all that apply)
A. Your developers and QA team members work together during the current sprint to develop and test a feature
B. Your developers code a new feature in short increments and then “throw it over the wall” for testers
C. Your testers hurriedly try to keep up with each sprint usually only having days to test a new feature within a deadline
D. Our teams are practicing test driven development (TDD), writing failing unit tests before writing code
31 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your People – Survey #1
32 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - People
Question #5: How well do your developers and testers communicate? (select all that apply)
A. Our testers and developers use a communication tool such as Slack
B. Our testers and developers are co-located in the same area of the building
C. We incorporate group activities and team building exercises to break down communication barriers.
D. We consistently emphasize the common goal of Continuous Testing to both developers and testers
E. Our testers and developers do not routinely communicate
33 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your People – Survey #1
Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Segment 1, Part 2: The Process and Technique
35 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Testing Shifting Left, Right…Up and Down
• Architecture testing
• Design and Code quality
• TDD, BDD
• Synthetic Virtual Svs and Test
data
• Unit Performance Security –
static and binary scans
• Env config validation
• Test change impact
• Hybrid VSE and Test
Data with masking
• Performance and
Security Test of every
build
• Monitoring in test
• Subset/mask test data from
prod
• Rollback testing
• Deployment validation
• Monitoring in test
• Test Performance and
Security Test based on ops
insight
• A/B tests & canaries
• Tie monitoring data to tests
and VSE
• CX Monitoring & Analytics
SHIFT LEFT SHIFT RIGHT
• Visual Model-based Requirements
• Better collaboration with biz/BAs
• CX-based requirements validation
• Test the idea
SHIFT UP
SHIFT DOWN
• Predictive Analytics – defect/failure prediction
• Adaptive testing – dynamic test optimization
• Intelligent Automation
36 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Understand Current State - Value Stream Mapping Real vs Perceived
Assessment of the Development, Testing & Release Processes
Development
Testing
Release/
Operations
▪ Goals & Initiatives▪ Organization, Team Structure▪ Processes▪ Development Tools▪ Code Quality ▪ Testing Expectations
▪ Testing Goals & Initiatives▪ Testing Organization & Team Structure▪ Processes & Maturity▪ Testing Tools ▪ Test Creation & Maintenance▪ Test Artifacts, Data & Automation ▪ Testing Environments & Constraints▪ API, UI, Functional, Performance,
Regression
▪ Release Train management▪ Operations team structure review▪ AS-IS Release Process review
▪ Key System Integrations ▪ Approval & Code Promotion Review▪ Current Toolset ▪ Challenges & KPIs
Organizational▪ Main Driver for change - $, Efficiency, Speed, Quality,
Risk ▪ Enterprise IT Organization Structure Review▪ Application Architecture Review – Integrations▪ Ownership & Team Responsibilities
Planning
▪ Agile Process Maturity▪ Challenges▪ Traceability▪ Key integrations with Development,
Testing & Release processes
Outcomes▪ Current State & Desired State ▪ Mutual Activity Plan▪ Prioritized Actions – High Value ideas, Low Hanging Fruit▪ Value Metric Measurement Process
37 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Example DevOps Value Stream Assessment– Typical High-Level
Elapsed time:
Process Time:
Wait Time:
Effort:
WIP/Size:
Quality:
Elapsed time:
Process Time: Wait Time: Effort:
WIP: Quality:
Elapsed time:
Process Time: Wait Time: Effort:
WIP: Quality:
Elapsed time:
Process Time: Wait Time: Effort:
WIP: Quality:
Elapsed time:
Process Time: Wait Time: Effort:
WIP: Quality:
Requirements Development & Int Test Staging/Pre-prod Release to Prod
DevOps Process Efficiency = Σ(Process Time)/Σ(Lead Time)
Effective Flow Rate: WIP Delivered/Σ(Lead Time)
Process Flow Rate: WIP Delivered/Σ(Process Time)
Impediment Rate: Σ(Impediment Time)/WIP delivered
Feedback Process Time: Wait Time: WIP: Quality:
Deployment Process Time: Wait Time: RW Time:WIP: Quality:
Feedback Process Time: Wait Time: WIP: Quality:
Deployment Process Time: Wait Time: RW Time: WIP: Quality:
Feedback Process Time: Wait Time: WIP: Quality:
Deployment Process Time: Wait Time: ReWork Time:WIP: Quality:
Feedback Process Time: Wait Time: WIP: Quality:
Deployment Process Time: Wait Time: RW TimeWIP: Quality:
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Wait Time
Process Time
Incident Remediation Feedback Loop
38 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Improving the Continuous Testing Process
• Make automation a priority
• Do not remove all manual testing tasks, but adopt an
“automated first” mindset
• Automate everything when possible
• Code scanning upon check-in, build creation and
deployments
• Remove “repetitive work” -- Focus that on areas that will be
run repeatedly
• Set up the plumbing first so that you can let the water flow
later without any worry or concern about leaks
• Map out your SDLC and identify automation opportunities
• Where is your team? Are they ready for change?
Implement a “ You build it you own it” mentality, testing becomes a critical aspect of development
39 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Our PoV on Continuous Testing Lifecycle Automation
Test creation and execution
automation
Test process automation
Test environment automation
CI/CD integration
Testing
Lifecyle
Automation
Automate test generation and maintenance
� Automated model-based test generation and
optimization
� Automated change impact analysis
� Automated automation
� Automated test execution
� API + GUI: Functional + perf, security
Automate the testing process
� Automated test process orchestration
� Automated analytics and dashboards
� Defect management automation
� Traceability automation
� Automated analytics and test insights
Automate test environments
� Test environment setup
� Env provisioning
� App/data deployment
� Continuous test data
� Continuous virtualization
Automate integration with CI/CD
� TDD/BDD automation
� Automated selection of tests
� Automated trigger of tests from CI/CD engines
� In-sprint automation
40 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
The Benefits of Automated Testing
• Find issues earlier in the SDLC because developers are helping with testing
• Automated testing saves a huge amount of time and allows for faster release cycles
• Code quality is improved. You can test everything on every build
• Save precious time
• Less risk because you know your code is good with every build
41 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
General Approach to solutions implementations
Think BIGStart Small
Learn fastScale thru success
Transform
42 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Process Maturity -- Summary View
Mostly manual testing process
• Manual requirements definition and tracking
• Manual test design and execution
• Manual test data management
• SCM, manual builds, no CI
• Hand-coded virtual services
• Manual test env provisioning and deployments
• Manual release mgmt.
• Classic QA/manual tester skillset
Focus on test optimization
• Requirements modeled in MBT tool
• Automated test design
• Automated unit tests
• Some TDD/BDD
• Some GUI/API test automation (F/P/S) ~20%
• Scripted test data; synthetic test data
• Code scans
• Algorithmic performance analysis (profiling)
• Recorded virtual services
• Scripted env provisioning and deployments (incl. test data and VSEs)
• Basic CI – automated builds
• Model-based tester(MBT) and some QE skills
Focus on CI integration
• Automated TDD (BDD, ATDD)
• Automated test automation from models – GUI/APIs (F/P/S) ~40%
• Automated testing tied to CI cycles.
• Focus on In-sprint automated testing (i.e. unit, integration, app-level system tests)
• Automated test data generation – sub-setting and masking
• Data rich VSEs
• Modeled test env. definitions; automated provisioning & deployments tied to CI
• Automated release promotion within release train
• QE/SDET skills
Focus on CD Integration
• Greater level of test automation ~60%
• Full automated traceability --Requirements models tied to releases, code, test data, virtual services and performance scripts
• Full test process/lifecycle automation and integration with CD processes
• Focus on e2e automated release testing (i.e. inclusive of x-app system testing, canary and A/B testing)
• Model-driven test environment management and deployment of assets
• Deployment and release coordination across complex release trains
• SDET/SRE skills
Focus on predictive capabilities and optimization
• Full testing/CD lifecycle data collection, integration and x-lifecycle data correlation in data marts/warehouses
• Use of advanced analytics, data correlation and ML
• Adaptive test (and automation) optimization using ML on lifecycle data
• Self-healing tests assets (e.g. automation)
• Predictive testing capabilities, e.g. defect prediction, adaptive test optimization
• Self-identifying A/B scenarios
• DET, Test Data Science/Analytics skills
Maturity Level
Cap
ab
ilit
ies
L0 Initial
L1 Basic
L2 Automated
L3 Orchestrated
L4 Intelligent
43 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Process
Question #1: Have you mapped out your SDLC to identify automation opportunities? (select one)
A. Yes, we have mapped out our SDLC to identify areas of automation
B. No, we have not mapped out our SDLC to identify areas of automation
C. We started the process but are having trouble identifying which areas of automation to tackle first
D. We started the process and know exactly what areas we should automate
44 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Process – Survey #2
45 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Process
Question #2: How mature is your test automation? (select one)
A. Very high degree of in-sprint automation (> 80%)
B. Medium degree of in-sprint automation (< or = 50%)
C. Automation lags development sprints
D. Limited automation (< 30%)
46 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Process – Survey #2
47 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Process
Question #3: How mature is your unit testing? (select one)
A. Very mature with TDD and BDD
B. Mature with automated functional, performance and security testing
C. Somewhat automated with code coverage around 50%
D. Limited unit testing with no clear measure of code coverage
48 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Process – Survey #2
49 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Process
• Question #4: How would you describe your test environment management? (select one)
A. We have virtual environments and test data on demand
B. Some level of automation in test environment setup
C. Manually created environments and test data
D. Ad hoc test environment management (i.e. dependent on ISP)
50 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Process – Survey #2
51 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Process
Question #5: How would you rate your current Continuous Testing Process progress? (select one)
A. We have no Continuous Testing Process in place
B. We are interested in adopting a Continuous Testing Process in the next 6 months
C. We are currently implementing a Continuous Testing Process
D. We have adopted a Continuous Testing Process and believe we are on the right track
52 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Process – Survey #2
Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Segment 1, Part 3: The Technology
54 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Key CT Technology Components
CT Platform
Requirements Modeling
Functional, Perf and Sec Automation –
API/GUI
Requirements & Test
Management
Service Virtualization
Test Data Management
Test Asset Management
Test and Release
Orchestration
Other CI/CD tools
(integrations)
Typical Components of CT Platform
55 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Technology
• The right technology is key to achieving Continuous Testing
• Manual testing approaches must give way to newer, faster technology-based solutions
• Automate Everything possible
• Companies often struggle with which tools to implement
• No one tool suite will work–Focus on Integration Platform
–Open Source and Vendor tools
56 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Some Guidelines for Technology
• Provide testing solutions that developers will use
• Look for tools that will help you develop, test and analyze continuously
• Pick best of breed tools that work together that works in your environment
• Review available tools that support your process and prioritize implementation (open source & vendor solutions should co-exist)
• Make the best decision for the team and commit to it…
• Let’s review some tools available
57 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Transforming the traditional agile backlog grooming & Test Case creation/maintenance with model-based Requirements
Product Owners
Developers
Testers
Product Owners
Product Owners
Happy Path:
Create User ID
Product Owners
Agile Requirements
Designer
58 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Model Based Testing with ARDShift-left testing efforts into building a model
1. Improve Collaboration - build better requirements,
enhance communication & requirements validation
2. Automate Test Creation and Optimization – dynamic
generation based on testing goals/scenario (functional,
regression, smoke, etc.)
3. Analyze Test Coverage & Risk programmatically
4. Automatically handle the impact of requirement
changes – automated test suite maintenance
5. Identify & define data requirements via powerful data
expressions to support both positive and negative testing
6. Generate automation assets needed to execute the
generated tests in any language for any framework
7. Integration with Agile tools – Rally, Jira, TFS, ALM…
59 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
How ARD Fits into the Continuous Testing Ecosystem
REQUIREMENTS & TEST CASE
MANAGEMENT TOOLS
AUTOMATED MODELS
CA ARD leverages selenium recorder to record tests
and generate models based on those recordings
Creating
Unambiguous
Models
TEST AUTOMATION
(NOT LIMITED TO BELOW)
Create & Maintain
Automated Test
Scripts
Integrates with…
ENVIRONMENT AUTOMATION
Service
Virtualization
Load/Append into
Existing/Running
Virtual Service
DATA AUTOMATION
Test Data
Manager
Define Test Data
Attributes & Find
Existing Data
Create NEW Data /
Synthetic Data
Dynamic Test Case
Creation & Maintenance
always Up-To-Date
Provide Traceability
Coverage statistics
Agile
Requirements
Designer
60 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Model-based Collaborative Test Design
OVERLAYING ALL LAYERS ROLES AND BENEFITS
Removes all ambiguities
MODELER (PO/BA/TESTER)
Draws and maintains business process flows
Baselines understanding across the team
TEST DATA ENGINEER
Match test data to each test case
Automatically generate synthetic data
Find and reserve data automatically
SERVICES ENGINEER
No dependency on external interfaces
Accelerate Code Development
TEST AUTOMATION ENGINEER
Automatic creation of scripts
Automatic impact analysis
Business Flows based on User Stories
Alternate Scenarios
Negative Scenarios
MODEL
Data Requirements
Data Parameters
Test Datamart connection
TEST DATA
External Interfaces Requirements
API Definitions
Service Virtualization
SERVICES
Framework linkage
Keyword linkage
Actions on the app
TESTAUTOMATION
SDET
61 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Dependency Challenges in Distributed Systems Development
“I can’t do anything until I have everything…
and I never have everything!”
ESB
!
!
! ! !
System Unavailable
x x x
x
Invalid data Access Fees
Incomplete
Development
62 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Addressing Constraints with Virtual Services
“I have full system access when I need it”
ESB
Agile-
Parallel
Developme
nt
Service
Virtualization
Service Virtualization
the Wind tunnel of Software
63 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Hybrid Test Data Management
Enterprise Wide Test Data Management
Dev/Test
Controlled
(Consumers)
Model-
based
specFill in the
gaps
(Synthetic &
Cloning)
Profile,
Subset,
Mask
Allocation
and
Reservation
Provision
on Demand
Ops Controlled
(Providers)
Model, synthesize,
virtualize
Profile, subset,
mask
Test Data
Management
64 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Differentiating between CI and CD
CI != CDCode & unit
test
Commit build, int.
test
Backlog
Design
System Test
• Deploy, test, promote
Pre-prod
• Deploy, test, promote
Prod
• Deploy, go-live, monitor
PackageCI Process
CD Process
Artifact
repo/
Containers
Artifact
repo/
Containers
65 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Enterprise CD requirements are different than that for CI
Pipeline/App Env Config
Tool Chain Integration
Process Orchestration
Complex Deployments
Systems/user testing
Rollbacks Approval
ManagementRelease
ManagementVisibility
Governance, Compliance, Audits
66 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Challenges doing enterprise CD with CI tools
• Everything is scripted or a plug-in
• Dependency on buggy open-source codeScripting
• Difficult to perform complex pipeline tasks like rollbacks, b/g deployments
• Difficult to manage pipeline inter-dependencies in complex release trains
Difficult to support ARA
• Debugging across 100s of scripts -- complex pipelines often brittle
• Requires Dev skills vs functional SME skills Maintainability
• App meta data (dependencies, config, properties, etc.) has to be scripted
• Pipeline does not auto-adapt based on changes. E.g.: no testing intelligence
Lack of app meta-data & Reporting
• Lack of integrated dashboards, no integrated data lake
• Difficulty in supporting compliance, identity management, audits
Lack of visibility and governance
67 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Deployment and Release Management
QA/TEST PRE-PROD PRODUCTIONDEV
AUTOMATED
DEPLOYMENT
Continuous Delivery DirectorOPTIMIZE
PIPELINE
Mo
del-
based
Rele
ases:
CA
CD
D
PLAN & MANAGE
RELEASES
Mo
del b
ased
D
ep
loym
en
ts:
CA
AR
A
68 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
CI & CD solutions can co-exist
CI: As-code
• Developer friendly
• Less complexity
• Lower controls and governance
CD: Low-code
• Functional specialist/management friendly
• High complexity
• Higher controls and governance
DevOps Value Stream Management Platform (CDD)
Continuous Testing
69 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Testing Platform considerations▪ Functional Testing
▪ API, UI, Security, Regression ….
▪ Scriptless Automation enable everyone to automate
▪ Record & generate script
▪ Performance & Load testing basic levels and extreme levels
▪ Test data available for execution
▪ Virtual Services/Mock Services for constraints as needed
▪ API Monitoring for modern architectures
▪ Shared Platform for execution
▪ Reporting & History
▪ Compare executions
▪ Drive Load internal & external (cloud) & Cloud Migrations
70 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Continuous Testing Platform
Quality Analytics & Insights
• Shift-Left, Intelligent Test Automation
360 Degree API Test,
Reuse Scripts for
Monitoring
Eliminate Test Env Delays
with Integrated Test Data
and Virtual Services
Scriptless
Automation
Generate Tests
& Automation
Functional UI,
API, Load
Testing
Built-On Open Source
• Evolution of BlazeMeter, Industry’s #1 Shift-Left Performance Testing SaaS Solution
• UI/Functional Testing, Service Mocking and Virtualization, API & GUI Testing, Monitoring
• Available as SaaS & Private Cloud
• Multi-Cloud Support (AWS, Azure, GCP)
• Integrated Service Virtualization & Mock Services
• Script Conversion – shiftleft.blazemeter.com
71 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Defining the end to end Value Stream - DevOps
Ideation
Customer
Experience
Continuous Testing
Test Data /Synthetic Data
Test CaseCreation
PerformanceTesting
Mocking& Virtual Services
Continuous Integration Operations
CapacityManagement
WIP
BuildCode
Synthetics
ServiceManagement
Monitoring
Transformation Intelligence
Business
Strategy
XLHigh Level
Planning
Backlog
Features
Stories
BehaviorModeling
Test Scripting
Test Management
Continuous Delivery
Release
Orchestration
CD Automation
API Governance &Management
Quality Management
Test Planning
Tracking
OKRs
Platform / Cloud Management (Kubernetes)
Platforms
72 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
DevOps & Continuous Testing | Reference Architecturefit within a broader DevOps ecosystem
Continuous Feedback & Collaboration
Continuous Integration
Monitoring
Portfolio & Project
Prioritization Service Management
Security
Build Business
Case Requirements
Modeling & Management
Ideas
UI
Performance Testing
User Acceptance
Testing
Test Data
API
APIs
Customer
ProductionApplications
Continuous Deployment
Test CaseCreation
CodeBuild
Security
Virtual Service
Continuous Delivery
Planning & Requirements
OperationsCollaboration Continuous
DeploymentContinuous Integration
Continuous Testing Development
API Management & Testing
Source Code Repository
Artifact Repository
Project & Portfolio Planning
Requirements Management
Deployment Tools
Requirements Modeling
Testing Efficiency Tools
Container Orchestration
Environment Configuration
Clarity
PPM
Agile
Requirements
Designer
Continuous Delivery Director
Service
Virtualization
DXI
APMTest Data
Management
Planning
Operations
Development
Continuous Testing
Performance & Functional Testing
Endevor
Automic
Automation
Shift LeftContinuous Testing
Runscope &
API Management
DevOps IntelligenceAI-Powered Predictive Insights & Feedback
73 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Technology
Question #1: Which tools are you currently using to help you achieve continuous testing? (select all that apply)
A. Requirements definition and modelling (CA, Tricentis, etc.)
B. Service Virtualization (CA, Smartbear, Tricentis, etc.)
C. Test data management (CA, Delphix, Tricentis, IBM, Informatica, etc.)
D. Pipeline orchestration tools (CA, Xebia, etc.)
74 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Technology – Survey #3
75 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Technology
Question #2: What tools do you feel are absolutely vital to achieving continuous testing at your company? (select all that apply)
A. Requirements planning and modelling
B. Test data management
C. Service Virtualization
D. Pipeline orchestration
E. Performance testing
F. UI testing
G. API testing
H. Other (please specify)
76 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Technology – Survey #3
77 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Technology
Question #3: Which continuous testing tools are you currently evaluating? (select all that apply)
A. Requirements modelling
B. Service Virtualization
C. Test data management
D. Pipeline orchestration
E. Performance testing
F. API testing
G. Other (please specify)
78 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Technology – Survey #3
79 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Technology
Question #4: What do you feel are the biggest obstacles for your team to adopt new continuous testing tools? (select all that apply)
A. Budget constraints
B. Cultural challenges that don’t permit us to change
C. Lack of desire by senior management
D. Can’t teach an old dog new tricks (developers don’t want to test…)
80 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Technology – Survey #3
81 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Survey Questions - Technology
Question #5: What do you consider the best way for your team to learn about continuous testing tools? (select all that apply)
A. Webinars and webcasts
B. White papers and data sheets
C. Live product demonstrations
D. Industry analysts
E. Other (please specify)
82 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Your Technology – Survey #3
83 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Technology Maturity Summary View
Ad-hoc tools with some integration
• Requirements definition tools (e.g. CA-Agile Central, Jira)
• Spreadsheets/docs for test design, test data and test env specification
• Mocking/stubbing tools
• IDE, SCM tools and build tools
• Spreadsheet-based release tracking
Integrated Test Platform
• Requirements modeled in CA-ARD, tied to AC/Jira
• Unit testing tools such as: Junit
• Other test automation tools such as Selenium, UFT
• Synthetic test data using CA-TDM and CA-ARD
• Basic virtual services with CA-SV, CA-SVCode
• API testing CA-DevTest, SOAP-UI etc.
• Perf testing using jMeter, CA-BZM, etc
• Test Environment manifests defined in config tools (Puppet etc) tied to deployment tools (CA-CDA)
• CI tools like Jenkins
• App release tracking in CA-CDD
Test Platform Integrated with CI platform
• TDD tools such as: Gherkin, Cucumber
• Tie CA-ARD models to automation tools (e.g. Selenium)
• Integration w/ APIM (such as CA-APIM) tools; API monitoring tools such as CA-Runscope
• Sub-setting and masking of test data using CA-TDM
• Data rich virtual services (CA-SV) with dynamic data handling (CA-TDM)
• All test tools integrated into CI platform
• CA-CDA deployments of dependencies within release train
• CA-CDD release tracking across apps in release train
Test Platform Integrated with CD platform
• Tie CA-ARD with SCM systems like Git, CA-SV, CA-TDM, CA-BZM
• A/B testing tools like Optimizely
• X-release train artifact tracking using CDD and CDA
• All test tools integrated into CD platform
Test Platform Integrated with Analytics/ML platform
• Use CDD/CDI for data aggregation across lifecycle, use of intelligent analytics
• Predictive analytics and ML tools and frameworks
• Machine learning for test optimization (ARD) based on lifecycle data (CDD)
• All test tools integrated into analytics/ML platform with automated centralized data collection
Maturity Level
Cap
ab
ilit
ies
L0 InitialL1 Basic
L2 Automated
L3 Orchestrated
L4 Intelligent
84 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
CT Capability Detail by Maturity LevelKey Capability Level 0 Initial Level 1 Basic Level 2 Automated Level 3 Orchestrated Level 4 Intelligent
Requirements
management
Manual requirements
definition and tracking
Model-based requirements
definition; manual traceability
mapping to stories/features, tests,
defects
Automated traceability management to
test assets; model versioning
Automated traceability management to
CD lifecycle assets (e.g. code, builds,
environments, releases, etc)
Automation requirements verification
based on CX analytics
Test design and
management
Manual test design;
manual tracking in
spreadsheets
Automated test design and
optimization from model
Some automated tracking; change-
impact testing; some risk-based
prioritization
Fully automated test tracking tied to CI
processes. Test model versioning;
automated change impact testing tied to
code changes
In-sprint testing
Full model-based integration of all test
assets.
Fully automated test tracking tied to
CD processes.
Integration of tests with monitoring
Adaptive test optimization based on ML
on past data
Test automation Mostly manual test
execution
Automated unit tests
Basic GUI/API test automation (<
20%)
API structural testing
Automated test automation (~40%);
execution integration with CI processes;
in-sprint test automation; automated API
monitoring
Full test process Lifecyle
automation(~60%) integrated with CD
processes; release lifecycle test
automation (incl.x-app system tests,
canary and A/B tests)
Self-healing automation using ML,
Test data
management
Manual test data
definition
Scripted test data generation;
synthetic test data
Automated test data with sub-setting,
masking, reservations; integrated with CI
processes; virtual test data
Automated test data provisioning
integrated with CD processes and tools
Predictive test data computation using
analytics
Virtual Services Basic mocking/
stubbing
Recorded virtual services
Synthetic virtual services created
with synthetic data
Data rich virtual services
integration w/ API Registry
Automated VSE provisioning integrated
with CD processes and tools;; VSE
updates based on production usage
Self-healing VSEs
Code quality Manual unit tests; code
reviews;
Automated basic unit tests, code
quality scans, SAST/DAST; code
profiling
Automated TDD/BDD/ATDD tied to CI
processes; automated unit perf tests.
Model versions tied to code versions
Full support for dark deployments Code promotion based on predictive
analytics; predict failures based on code
quality analytics
CI/CD/Release
Management
Integration
No CI. Use of SCM
and manual builds
Basic CI integration, e.g. Automated
builds tied to automated unit tests
Full CI lifecycle integration within release
train. In-sprint tests triggered from
successful builds. Monitoring during perf
tests
Full integration into CD lifecycle
processes with orchestration across
release trains; full monitoring in all test
environments
Automated release risk/quality prediction
Test
environment
and deployment
management
Basic – statically
allocated
environments; manual
deployments
Scripted provisioning/updates to
env. Scripted deployments
Model-driven test environments (with test
data, VSEs, test suites) and deployment
rules; integrated with CI process
automation
Integrated with CD automation; support
for dark deployments; A/B and canary
tests
Self-healing/sizing test environments
Test analytics
and intelligence
Basic metrics tracking;
trend analysis
Automated descriptive analytics of
metrics in silos
Some diagnostic analytics using manual
correlation of data
Automated diagnostics analytics using
automated correlation
Full testing/CD lifecycle data
warehousing; prescriptive analytics and
ML support
Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Segment 2: MegaCorp Case Study
86 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaAction Corp – A Case Study
• Where MegaAction Corp is at in their journeyIdentifying
• Goals, gaps, bottlenecks across the end to end process/organizations
Value Stream Mapping exercise
• For MegaAction testers and developersBuilding a game plan
• Of the MegaAction Corp CT implementationMeasuring success
• With MegaAction Corp senior managementSharing results
• From MegaAction Corp’s journeyKey learnings
87 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp Case Study
Assess people, process,
technology
Develop mission/vision/goals
statement
Build a COE for a solution
framework – P/P/T
Start with pipeline automation,
then quality improvement
Continuous “Shift Left” Testing
Continuous Improvement
SOLUTION APPROACH
1-2 week release cycle plus
on-demand releases
85% test coverage
75% reduction in regression
test effort
Cost savings >.5M in 1 year
RESULTS
• Manual testing (4 hrs for
regression 30K+ hrs overall
testing)
• Limited test coverage
(<44%)
• Limited non-functional testing
• Low release velocity
(monthly)
• Unstable environments
• Manual release process
• Limited collaboration
• One Click to Production Goal
CHALLENGES
88 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp: Looking at People, Process, Technology
89 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp – Bringing People Together
90 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaAction Corp – Focus Areas
• Automate the Deployment & Release pipeline
• Automate Environment Builds
• Enable Model Based Testing
• Automate Test scenarios
• Automate Test Data provisioning for builds
• Integrate Test execution into Pipeline
91 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp – The Process and Technology
92 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp State 1 year later
93 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp Results
94 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
MegaCorp Results
Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
Wrap-up
96 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|
Thank You for Joining us Today