Date post: | 21-Jan-2018 |
Category: |
Engineering |
Upload: | ievgenii-katsan |
View: | 59 times |
Download: | 3 times |
QA Management in BIG Agile teams
Volodymyr Prymakov
Speaker info
Volodymyr Prymakov, UkraineSenior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
•14 years in QA
• 45 projects experience
• Certified: ISTQB Advanced Test Analyst &
Manager, ICAgile CP, SAFE
/in/vladimirprimakov/
Agenda
1. Big Team, What is it?
2. Process (Development and QA)
3. Infrastructure
4. Collaboration
5. Transparency
6. QA Team organization
7. Automation
8. Questions
What is a big team?
What is a big team?
Team
What is a big team?
Team Big team? - Expected!
What is a big team?
Team Big team? in Reality….
What is a big team?
Most of the problems in a big team
relates to
cross-team work and
common dependencies
Big agile team is a team of >50 people
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
PO Manager
Dev Manager
Architectors
QA Manager, Leads
SMSMSM
Program Manager
Release Manager
Dev. Teams
Leadership Team Release train
Products
1. Big (A lot of functionality or sub-products)
1. Complex (Architecture and infrastructure)
1. (Quite often) Monolete architecture or
interdependable components
1. A lot internal and external integrations.
2. A lot of functionality interdependencies.
High Regression risk!
3. Many end-users (sometimes, in many
countries)
PROCESS
Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
STABLE BRANCH
Process - Overlapped Releases
Supported by the same teams
STABLE BRANCH
MASTER BRANCH
Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug Hotfixes
STABLE BRANCH
Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug Hotfixes
STABLE BRANCHRegression Problems
Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams
Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams
Overlappin
g a
ctivitie
s
Process - Overlapped Releases
!!!More Delay
Supported by the same teams
Overlappin
g a
ctivitie
s
Overlappin
g a
ctivitie
s
Process - Overlapped Releases
!!!More Delay
SHIFTED RELEASES
Supported by the same teams
Overlappin
g a
ctivitie
s
Overlappin
g a
ctivitie
s
Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching
Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching
● Complicated branching
strategy and risky hot-fixing
process
● Not compatible with
Continuous Delivery
Alternative Process - Overlapped Releases
Regression and release support by Another team
PROBLEMS:
● Lack of new functionality
(implementation) knowledge
● Knowledge Transfer required
● Not enough expertise for bug
fixing
● Extra Collaboration needed
● Not Enough Capacity
● Motivation problem
System Dev & QA Team
STRAITFORWARD Release Pipeline
MASTER BRANCH
NOW
STRAITFORWARD Release Pipeline
MASTER BRANCH
FUTURE
STRAITFORWARD Release Pipeline
BENEFITS:
● SIMPLE and STRAIGHTFORWARD
● NO WASTE of TIME FOR
KNOWLEDGE TRANSFER
● NO CONTEXT SWITCHING
● RELEASES EVERY 2 WEEKS
● COMPATIBLE with CONTINUOUS
DELIVERY
Process Effect on QualityNEW PROCESS
Bug Rate
Story
Points
Delivered
OLD PROCESS
STILL GOOD VELOCITY
30-40% LESS BUGS
STABILIZATION Period
● No Feature Merging in
master or release
branch
● Only Blocker and
Critical bugs fixing
● Regression testing
● Bug Retest.
Release management FOR OTHER PRODUCTS
Monitor Release
Weight Impact
Prepare in advance
Test and Retest it!
Releases for Integrated Systems (internal and 3-rd parties)
Late Code Merging
Late code merging leads to bugs, no time for testing and release delays!
Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing
Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing
● Communicate, manage, and revert risky merges.
Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets
Regression testing approach (e.g.)
Fast
Feedback about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1
Mostly p1, p2
bugs
Regression testing approach (e.g.)
Fast
Feedback about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs
Regression testing approach (e.g.)
Fast
Feedback about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Regression Cycle 3
Validate
Quality after
Bug-fixesverifications
Scope:
P1, P2 test
cases
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs
p1, p2, p3 or lower
priority bugs
Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 2 (IE)
Cross Platform Testing approach - Sprint N
Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Cross Platform Testing approach - Sprint N + 1
Regression testing approach (e.g.)
Type of Test Scope Outcome
Smoke End-to-end flow (most
popular booking)
Blocker bugs
Acceptance Alternative End-to-End
flows
P1 and P2 bugs
UAT User Acceptance Test
scenarios
P1, P2, P3
Some bugs may be missed
Regression P1 P1 test cases P1 and P2 bugs
Regression P2 P2 test cases P3 bugs
Exploratory Exploratory test cases P1, P2, P3
Testing Order!
System Intergration Testing
● Obligatory test integrations with real life internal and 3d party services
● Do confirmation testing for the rest of functionality if needed (Exploratory, Cross-platforms, or more wider testing)
Cross team Bug Escalation
Priority
p3, p4, less
QA Coordinator /
Release Coordinator
Product
Backlog
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!Is this bug produced by the team?
Does the team has expertise to fast fix it?
Or does the team has capacity for bug fixing?
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersion
Which team produced the bug?
Which team has better expertise to fix it?
Which team has more capacity for bug fixing?
TESTER
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
Which team has more capacity for bug
fixing?
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
Which team has lower priority functionality
which can wait for the sake of bug fixing?
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
FACILITATE
Align QA approaches across teams
Align QA approaches across teams
• DoD for test cases
• DoD for Automation scripts
• Test suite Structure
• Test plan and run structure in test management
system
• Regression approach
• Bug Reporting
• Release readiness reporting
• and many others
Agreeing project process/approach
Agree cross-team approaches/processes on all levels
Agreeing project process/approach
Agree cross-team approaches/processes on all levels
Involve all concerned parties in the discussions
Agreeing project process/approach
Agree cross-team approaches/processes on all levels
Involve all concerned parties in the discussions
Align cross-team procedures
Agreeing project process/approach
Agree cross-team approaches/processes on all levels
Be tolerant and persistent
Involve all concerned parties in the discussions
Align cross-team procedures
Agreeing project process/approach
Agree cross-team approaches/processes on all levels
Be tolerant and persistent
Document agreements
Involve all concerned parties in the discussions
Align cross-team procedures
Infrastructure
Infrastructure problems
2h of infrastructure
downtime for the team in
size of 50 Devs and QAs
may cost 12.5 man days…
(several thousands USD)
Test Environments Requirements
SIMILAR to PRODUCTION
AUTOMATED and CONTROLLED
FAST and POWERFUL
LESS INTERDEPENDENT
ROBUST and STABLE
Typical Infrastructure/environment problems
Problem
Essential difference between environments (configuration, integrations, data) leading to leaks of bugs to production
Uncontrolled changes and manual interventions
Configuration problems.
Slow and overloaded environment. Usage conflicts.
Integration issues (external and internal).
Services and DB compatibility.
Unreliable or absent test data.
Unpredictable downtimes. Connectivity and access issues.
Solution
Minimize the difference. At lease 1 test environment should be 99% identical to production.
Automate Everything. Apply change control procedure. Limit unauthorized access. Notify about changes in advance!
Increase Capacity. Build several test environments.
Don’t save on the environment. Use more powerful network.
Controlled 3d party releases. Component decoupling. Mocks usage. Backward compatibility.
Mocking Data, Standard data sets usage, Agreeing test data provision from 3-d parties, etc.
Plan B: Backups, alternative suppliers etc.
Environments (e.g.)
Testing Owners
Integrations
Hardware capacity
DB
Devops should be part of teams
● ~1 Devop for 2 teams
● Open communication. Participation at
daily scrums and other sync-ups
● Proactive resolution of ongoing
infrastructure and environment
problems
● Devops backlog prioritization for sake
of teams’ needs
Cross team Collaboration
Strategic Planning
Plans are hidden from teams
Strategic Planning
Strategic Planning
Strategic planning helps to build the right product
Product increment and other initiatives for a Quarter/Half-a-year
Sprint Planning (Devs)
Teams
Grooming
Sprint Planning (Devs)
Teams Cross-Team
GroomingTech-leads
Preplanning
Sprint Planning (Devs)
Teams Cross-Team
Grooming
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning
Sprint Planning (Devs)
Teams Cross-Team Teams
GroomingSprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning
Sprint Planning (Devs)
Teams Cross-Team Teams
GroomingSprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Sprint
Start
Tech-leads
Preplanning
Sprint Planning (QAs)
Cross-Team
QA Leads
Preplanning
Sprint Planning (QAs)
Sprint Planning (QAs)
Cross-Team
Additional sprint
testing scope
identified
QA Leads
Preplanning
Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
QA Leads
Preplanning
Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
Start
QA Leads
Preplanning
Sprint Planning (QAs)
Cross-Team Teams
QA Leads
Preplanning
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
StartCross-Team
QA Sprint
Scope
Review
Next Slide =>
QA Sprint Scope Review & Planning
Participants
QA Manager,
QA Leads,
1 QA from
every team
QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
QA Manager,
QA Leads,
1 QA from
every team
QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
-------------------------------------------------
1. Discuss QA Automation coverage
increment and other achievements
2. QA automation scope adjusting
QA Manager,
QA Leads,
1 QA from
every team
QA Sprint Scope Review - Outcomes
Regular Team & Scrums
Teams Cross-Team
ScrumScrum of
Scrums
● Cross-team
Dependencies,
impediments, and help
needed
● Environmental Issues
● Overall Sprint Scope
covering (once)
● Are we ready for Code
Freeze?
● Discussing release
stoppers: Blocker and
critical bugs
Auditory:
Release Manager,
Scrum Masters,
Tech Leads,
QA Leads,
Devops Lead
Regular Cross Team Syncups
Dev Manager,
QA Manager,
Tech Leads,
DevOps Leads,
Release Coordinator
● Strategic Plans,
● Common
Approaches,
● Important Ongoing
Activities,
● Other Common
Questions and
Problems
Readiness Checkpoints
Ready for Regression?
Ready for Stage Testing?
Ready for UAT by Business?
Ready For Production?
Regularly Sync-ups
on Product Readiness!
Release Manager,
Scum Masters,
PO Manager,
QA Leads/QA Coordinator,
(POs),
(Tech Leads)
Lessons Learned
Teams
Team
Retro
Sprint/Release
Finishes
Lessons Learned
Teams Cross-Team
Team
RetroQAs
Retro
Sprint/Release
Finishes
Lessons Learned
Teams Cross-Team
Team
RetroQAs
Retro
Release
Retro
Sprint/Release
Finishes
Cross-Team
Knowledge sharing
Spread Knowledge among the project!
!!!Record videos and/or document the stuff
● Business Domain
● Product Functionality
● Overall Architecture
● Component Technology
● 3-d party components
● Common approaches, etc.
Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo
Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo
!!!Avoid long
meetings at
essential
sprint/release
phases
Transparency
Scope understanding
Initiative/Epic level
Know and
align with
the
project
roadmap
Scope understanding
Sprint/Release level
Use Fix Version for
features and bugs
in advance to
track overall
release scope
(cross-team)
Realtime quality boards
Monitor Quality on daily bases or more often
1. Dynamic Jira Dashboards 2. Kanban Boards for Blocker and Critical bugs
Testing reporting
1. Report/Escalate on Blocker and
Critical bugs by a necessity.
2. Report regression testing results
on daily bases.
3. Involve all concerned parties in
the report.
4. Build quality
awareness/transparency on the
project!
Quality trends
Monitor and analyze Quality trends.
Apply Corrective actions if needed.
QA Team
Typ
ica
l QA
Te
am
Str
uc
ture
QA Manager
QA Manual Lead
Team 1
Senior/Middle QA
Middle/Junior QA
Team 2
Senior/Middle QA
Middle/Junior QA
Team 3
Senior/Middle QA
Middle/Junior QA
QA Manual Lead
Team 4
Senior/Middle QA
Middle QA/Junior QA
Team 5
Senior/Middle QA
Middle/Junior QA
Team 6
Senior/Middle QA
Middle/Junior QA
QA Manual Lead
Team 7
Senior/Middle QA
Middle/Junior QA
Team 8
Senior/Middle QA
Middle/Junior QA
Team 9
Senior/Middle QA
Middle/Junior QA
QA Automation Lead
Core Automation Team
Senior/Middle Auto QA
Middle Auto QA
Middle/Junior Auto QA
QA Team Typical Roles
Defines and aligns QA Approaches on the project,
Strategic Planning and Reporting,
Monitor and adjust ongoing QA activities,
Resource planning, recruitment, competences, etc.
Escalation questions
QA Manager
QA Approaches implementation,
Cross team ongoing QA work planning and coordination in a sprint (release),
Regular QA reporting,
People management in their teams (Mood, PDPs), etc. QA Lead
Building and aligning QA automation approaches and expertize on the project, Organizing the corresponding Knowledge Sharing,
Leading automation Framework development and support,
Automation team management.
QA Automation
Lead
QA Component Lead role
1. Main knowledge holder / expert in the product area.
2. Responsible for the quality of the product area. Regularly
monitor its quality. Escalate problems if needed.
3. Define dependencies and quality risks in the area. Inform
others about them.
1. Defines manual and automation testing scope for the product
area.
2. Monitor and analyze automation testing results for the
product area. Execute automation tests if needed.
1. Plans and organizes regression and other kind of testing for
the product area.
QA Coordinator Role
1. Organize QA syncups and planning meetings.
2. Plans and organizes cross team testing activities
a. regression testing.
b. testing at production.
c. etc.
3. Monitor overall product quality state.
4. Escalate Blocker and Critical bugs if needed.
5. Prepare and send quality reports.
6. Play a representative role in release readiness
meetings.
Coordinate cross-team testing activities in a release.
Cycle the role
between QA leads.
Delegation on All levels
Other roles:
● QA Product Market Lead
● Platform statistics
management
● Devices Management
● A/B Testing management
● Payment Card and refund
Management
● Etc.
Automation
Why Automation?
Cross team
work
Regression Risk Automationprevents the risk
Requirements to QA automation
All Levels Correct order Cross Platform
Requirements to QA automation
Sufficient
Coverage
Stable
and
Trustful
High Speed
Requirements to QA automation
FrequentExecuted on all
environments
Aligned with Manual QA
Manual QAAutomation QA
Aligned with Manual QA
Integration with Test Management system
Non-Functional Automation
Parting Words
Questions