Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | hunter-davies |
View: | 218 times |
Download: | 3 times |
_experience the commitment TM
Why is Test Planning so Hard?Elizabeth D’Angelo, PhD
CGI Group Inc.
September, 2008
© CGI GROUP INC. All rights reserved
2
Confidential
Agenda
• Background• Test Planning: Three Issues• Test Planning: Three Solutions• Practical Tips to Help you in Test Planning• Conclusions• Q&A
3
Confidential
Reasons Why Test Planning is Difficult
Planning Expertise
Application Expertise
Forecasting Expertise
Testing Expertise
4
Confidential
Test Planning Issue 1: Knowledge (1)
• Right Mix is Critical!
• Requirements• Requirements: Implicit vs. explicit knowledge (people vs. docs)• Expertise
• Thorough knowledge of applications and flows• Breath vs. depth
Testing Domain
Application Domain
5
Confidential
Test Planning Issue 2: Project Management (1)
• Test Planning is Project Management within Project Management!
• It is NOT optional• Be prepared to articulate WHY is needed
• Plan – Prepare – Execute motto
• Take time to do proper planning • “Vanilla flavour” plan is useless
6
Confidential
Test Planning Issue 2: Project Management (2)
Environment
Data
Tests
7
Confidential
Test Planning Issue 3: The “Gruyere” paradigm
• How many holes are in this cheese?• What is actually delivered vs. what was planned to be
delivered?• Are all the features truly complete?
8
Confidential
Test Planning Solution 1: The Expert (1)
• KNOW the WHAT…• Industry Knowledge• Application/Environment Knowledge• Testing Knowledge
9
Confidential
Test Planning Solution 1: The Expert (2)
• Become the testing guru• Help your team to become the testing experts• Some Ideas:
• Formal training/certification• Attend conferences• But there are also cheaper ways:
• Lunch and Learn sessions • Organize workshops with presentations
• Test Planning• Risk-based approaches• Test terminology• Test case design• Metrics• Test tools
• Create a testing library
10
Confidential
Test Planning Solution 1: The Expert (3)
• Build your credibility by becoming the industry/application expert• Read ALL documentation
• Review the requirements, design, architecture
• Talk to users, developers• Don’t be afraid of asking questions
• Ensure to communicate your status properly• Assume responsibility when needed
11
Confidential
Test Planning Solution 1: The Expert (4)
• And there is more… Soft Skills!• Communications (defect reporting)• Detective• Negative thinking or “What can go wrong”• Time management• Discipline and Tenacity• Creativity
12
Confidential
Test Planning Solution 1: The Planner (1)
• KNOW the HOW• Testing scope• Discuss under/over testing• Who will be doing what• What needs to be in place before test starts• Test case design guidelines• Tool usage• Metrics to be collected• Entry/Exit criteria• …
13
Confidential
Test Planning Solution 1: The Planner (2)
• You will reap what you sow
• Planning is the most essential activity!PLAN, PLAN, PLAN ….
You cannot prepare what you have not planned for
You cannot execute what you have not prepared for!
14
Confidential
Test Planning Solution 1: The Planner (3)
• Make your Master Test Plan the Bible• Document ALL your assumptions• Think of ALL risks• Use ALL information available to you• Establish in/out scope INCLUDING
• Test boundaries
• Brainstorm and define what will be tested when!• Phased approach
• Don’t forget • Environment• Test Data• Access/Security
15
Confidential
Test Planning Solution 3: The Weather-Man (1)
• Look at signs to forecast quality issues• Previous history (application, business function, etc.)• Get to know the team• Collect and store metrics you will need them!
• Clearly communicate deviations from plan based on unexpected quality issues
16
Confidential
Test Planning Solution 3: The Weather-Man (2)
• Establish and Communicate Forecast for your Plan:• Failure rate• SLAs for defect fixes• Environment availability
Confidential
Practical Tips to Help you in Test Planning
18
Confidential
ALWAYS Use Risk-Based Approach! (1)
• Spend your effort where is worth it,…Unless you have unlimited time/budget/resources
• Risk-based approach will allow you to test• More comprehensibly what really matters• First what is more important• If needed, know where to cut testing
• NOT an isolated task, you need to socialize approach and get agreement on risk areas
19
Confidential
ALWAYS Use Risk-Based Approach! (2)
• Technical risks assessment should include:• New and immature technology, • Brand new system or enhancement• New and difficult to understand business requirements, • Complexity of business logic, • Complexity of technology, • Size of the project, • Skill mix on the project,• Number of users (including customer facing or back office
considerations) historical data when appropriated (the risk is higher for modified functions that have frequently failed in the past, i.e., Do you have any historical data of defects by area?)
20
Confidential
ALWAYS Use Risk-Based Approach! (3)
• Business Impact assessment should include:• Importance of features from client/customer perspective• Criticality of features from the business perspective• Frequency of use
21
Confidential
ALWAYS Use Risk-Based Approach! (4)
• As a rule of thumb:• High Impact:
• Calculation• Validation• Legal• Front end (end user)• High frequency, High volume of users
• Medium Impact: • Data changes• Wrong information displayed• Medium frequency and groups of users
• Low Impact: • Display• Visual appearance• Used by a few users• Back office
22
Confidential
Risk-Based Approach Process
1. Organize test risk assessment workshop with business and technical stakeholders
2. Identify and list requirements, business functions
3. Evaluate technical and business impact (rate 1-3)
4. Evaluate probability of failure (rate 1-3)
You might want to read: http://www.satisfice.com/articles/hrbt.pdf
Risk Factor Testing Priority Testing Extent
7-9 High Extensive
4-6 Medium Comprehensive
1-3 Low Smoke
23
Confidential
Risk-Based Approach: Business/Requirements Map
High
Low
Low HighImpact
Lik
eliho
od
1 32
1
2
3
Accounting
Finance
Product Filing
Req n-7
Licensing
DistributionCommunications
Req n-4
Req n-1
Administration
Extensive Testing
High Degree of Testing
Less Testing Required
24
Confidential
Cover all aspects of testing
• When planning… make sure to cover the overall process:
Test Preparation Test ExecutionPlanning Reporting
Test Results
Problem / Defect Reports
Test Plan (s)
Update Test Project Plan
Create Regression Test Suite
Test Environment set up
Test Data set up
Test Cases
Create Automation Test Suite
Test Exit Report
Lessons Learned - Recommendations
Final Test Report
Strategy
Test Strategy
Test Project Plan (Schedule)
Test Coordination
25
Confidential
Schedule Templates Detailing All Activities (1)
• Have an overall schedule template that you can easily fill with specific data
Activity
Work-Effort
(in days)
Start Date End Date# of Res
Resource Details
Review Business, Technical Requirements 0 YY/MM/DD YY/MM/DD 0 Test Manager, Test Leads, Test Analysts
Create Test Strategy/Master Test Plan/Traceability Matrix 0 YY/MM/DD YY/MM/DD 0 Test Manager, Test Leads, BAs
Review Test Strategy/Master Test Plan 0 YY/MM/DD YY/MM/DD 0 Project Team, Stakeholders
Sign-off Strategy 0 YY/MM/DD YY/MM/DD 0
Create detailed Test Schedule 0 YY/MM/DD YY/MM/DD 0 Test Manager
Review Test Schedule 0 YY/MM/DD YY/MM/DD 0 Project Team
Sign-off Test Schedule 0 YY/MM/DD YY/MM/DD 0
Creation of System Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team
Review Test Cases 0 YY/MM/DD YY/MM/DD 0 BAs, BSAs
Update Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team
Create Test Data 0 YY/MM/DD YY/MM/DD 0 Data Analyst
Sign-off System Test Cases 0 YY/MM/DD YY/MM/DD 0
Review Regression Test Bed 0 YY/MM/DD YY/MM/DD 0 Test Team
Update Regression Test Bed 0 YY/MM/DD YY/MM/DD 0 Test Team
26
Confidential
Schedule Templates Detailing All Activities (2)
ActivityWork-Effort
(in days)Start Date End Date # of Res Resource Details
Create/Select UAT Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps
Review Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Leads
Update Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps
Create UAT Data 0 YY/MM/DD YY/MM/DD 0 Data Analyst
Sign-off UAT Test Cases 0 YY/MM/DD YY/MM/DD 0
Execute System Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team
Review exit criteria and produce System Test Report 0 YY/MM/DD YY/MM/DD 0 Test Manager
Sign-off System Test Results 0 YY/MM/DD YY/MM/DD 0
Execute UAT Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps
Support UAT Execution 0 YY/MM/DD YY/MM/DD 0 Test Team
Produce UAT Test Report 0 YY/MM/DD YY/MM/DD 0 Business Lead, Test Manager
Sign-off UAT Results 0 YY/MM/DD YY/MM/DD 0
27
Confidential
Systematic Entry/Exit Criteria
Entry Criteria Exit CriteriaUnit/Integration Testing exit criteria has been fulfilled and approval provided
System Test Plan is approved or Approach is documented in the Overall Test Strategy/Plan
All system test cases as per plan have been executed
System scenarios, conditions and cases are complete and approved
All detected fixed defects have been re-tested
Test data has been created and is readily available in system test environment
No outstanding Severity 1 or 2 defects
Test environment is set up and operational
System Test Report has been completed, reviewed and approved
Plan
Test Cases/Data Readiness
Environment Readiness
Planned Activities
Quality
Communication & Sign-off
Preconditions
28
Confidential
Forecasting with Metrics – Failure Rate/Env
• Document what is the expected failure rate• Existing application/delivery team: use previous data• Be as granular as possible
• By Functional Area• By Application
• If you don’t have data, rule of thumb is around 30% (i.e., 30% of test cases will fail during test execution)
• Incorporate retesting activities to schedule
# TestersPlanned
# TestersActual
Env # TestsPlanned
# TestsActual
Passed Failed FailureRate
DefectsNew
DefectsFixed
ExecutionVariance
RetestingVariance
Total Variance
Cycle 1 5 5 100% 50 50 30 20 40.00% 12 0 0 -20Day 1 5 3 90% 50 30 25 5 16.67% 4 4 -20 -5Day 2 5 7 100% 50 75 60 15 20.00% 10 10 25 -15Day 3 5 50Day 4 5 50Day 5 5 50Day 6-retesting 5 50Day 7-retesting 5 50
TOTALS 400 155 115 40 26 14 5 -40 -35
29
Confidential
Forecasting with Metrics – Defect Age SLAs
• Negotiate and document SLAs for defect fixing, then measure• How long defects remain in New state (i.e., not assigned?)• How long defects remain in Assigned state (Are too many defects assigned to
the same resource and therefore there is no progress?• How long defects remain in Fixed state (i.e., before being closed by Test
Team)? (Is testing team following up on defect fix verification? Are there competitive priorities (execution vs. verification)?
• A high number of defects in Need Info state could indicate that there are concerns with the requirements, lack of details on the defect (to reproduce).
30
Confidential
Conclusions
• DO• PLAN, PLAN, PLAN …. AND THEN PLAN!• Get buy-in into risk-based approach• Create templates to help you be efficient• Be systematic• Use historical information/metrics to predict future• Share your documents, status and metrics so stakeholders understand
challenges• Hope for the best, but plan and prepare for the worst!
• DON’T• Work in isolation• Over complicate your methodology• Think you don’t need to understand domain• Do ad-hoc testing (unless it is planned)• Be inconsistent• Keep issues, departures from assumptions to yourself• Be naïve and plan for the best!
_experience the commitment TM
THANKS!!
Questions??
32
Confidential
Contact Information
Elizabeth D’Angelo, M.Sc. PhDDirector, Quality COE150 Commerce Valley Drive WestMarkham, Ontario(905) 695-6513(416) 459-1257 [email protected]