Date post: | 10-May-2015 |
Category: |
Software |
Upload: | ho-chi-minh-city-software-testing-club |
View: | 395 times |
Download: | 1 times |
Value-Based Software Testing: An Approach to Prioritizing Tests
STC 2014
VU NGUYEN
Faculty of Information Technology University of Science, VNU-HCMC
Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
2
Introduction - 1
3
This release is very important for our users, we want your team to test it within 5 days
The release includes many changes and new features. We have to test 2000 test cases, and that takes at least 10 days to complete
Well, we understand that. But we cannot release later than 5 days
As a QA engineer, what should you do?
Your client
You
Introduction - 2
• Pareto Principle or 80-20 rule
– 80% of the land in Italy owned by 20% of population
– 80% of the wealth and income controlled by 20% of population
– 80% of accidents committed 20% motorists
– 80% of Facebook posts made by 20% users
4
Introduction - 3
• In software development
– 80% of MS Office users use only 20% of its features
– 80% of the business value come from 20% of the features
– 80% of the defects come from 20% of the modules
– 80% of the defects come from 20% of the test cases
5
Introduction - 4
• Value-based solution to the test request above
6
% of
value
from
testing
Test cases
100
80
60
40
20
500 1000 1500
Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
7
Value-neutral software practices
• Value-neutral software practices
– Every requirement, use case, module, test case is equally important
• Value-neutral practices are risky because
– project is constrained by time, budget, staff, quality
– software lifecycle is ever reduced
8
Value-based SE
• Value considerations are important to software project success
• Value
– An equivalent in goods, services, or money
– Relative worth, utility or importance
• Non-monetary values are important
– Fairness, customer satisfaction, trust
• Values vary by stakeholders
9
Value-based SE - 2
• VBSE: software development practices take into account value considerations
• Examples
– Value-based requirements • Defining objectives by taking into account stakeholder
value propositions
• Feature prioritization based on business value, cost, risk
– Value-based testing • Test case prioritization based on business value, cost, risk
10
Value-based SE - 3
• Some important VBSE practices
– Expectations management • Manage stakeholders’ expectations focusing on their value
propositions
– Risk management • Manage most important risks (eg., top ten risks)
– Prioritization • Prioritize requirements, use cases, test cases, etc. based
on value
11
Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
12
Value-based testing practice
• Test management and execution based on value
• A value-based testing approach
1. Derive test cases from prioritized requirements
2. Determine risk exposures and risk reduction leverage related to each test case
3. Prioritize test cases using risks
13
Risk management
• Risk is a potential problem
• Risk Exposure (RE) = Loss x Probability
– Loss: the value lost if the risk occurs
– Probability: the chance of the risk to occur
• Risk Reduction Leverage (RRL)
RRL = (REbefore – REafter)/Cost
Cost: can be the effort spent for testing
REbefore, REafter: Risk Exposure before and after testing is done
14
Test case prioritization - 1
• For each test case, determine RE using
– Business value
– Defect criticality
– Defect-prone components and developers
RE = Size of Loss x Probability of Loss
– E.g., Size = 20 person-hours, Probability = 20% RE = 20 * 20% = 4
15
Size of Loss
Probability of Loss
Test case prioritization - 2
• Value-based goal: maximizing the value gained in testing
– by reducing risk associated with tests
– meaning, maximizing Risk Reduction Leverage (RRL)
• Greedy approach: prioritize test cases based on RRL (focusing on tests with high RRL first)
RRL = (REbefore – REafter)/Cost
16
Research evidence
• Qi Li et al, 2012
– Results from experiment at Galorath Inc.
17
74.07%
77.78%
83.95%
90.12% 93.83% 95.06%
100.00%
4.94% 6.17%
9.88%
16.05%
22.22%
25.93%
58.02%
35.80%
39.51%
45.68%
51.85%
58.02% 61.73%
87.65%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
8 10 12 14 16 18 20 22 24 26 28 30Stop Testing
Value-based
Value-neutral
Value-inverse
Tool support
• Calculations can be done easily in Excel
• Test cases are prioritized/sorted according to RRL
18
Making it simple to use
• It can be time-consuming to estimate risks for all test cases
• We can approximate the process by
– ignoring risks after testing • prioritizing based on REbefore/Effort
– prioritizing features/functions to test instead of test cases
19
Making it simple to use - 2
• Ignoring risks after testing
– Assuming we get the same risk reduction rate after testing
20
Making it simple to use - 3
• Prioritizing at feature level
– Prioritizing features instead of test cases
– Link test cases with features
– Run test cases according to prioritized features
21
Conclusions
• Value considerations important to success of software projects
• Values dependent on stakeholders
– Monetary, satisfaction, importance, cost, trust, happiness, etc.
• Value-based testing: prioritizing tests using risks is important to maximize values of testing
22
References
• Qi Li, Barry Boehm, “Value-Based Software Test Prioritization”, Annual Research Review, CSSE, USC, Mar 2012 (http://csse.usc.edu/csse/event/2012/ARR/presentations/QLi_ARR_March2012.pdf)
• Barry Boehm, “Value-Based Software Engineering”. ACM Software Engineering Notes, 2003; 28(2)
• Barry Boehm, and Victor Basili, "Software Defect Reduction Top 10 List," Computer, vol. 34, no. 1, pp. 135-137, Jan. 2001 (http://csse.usc.edu/csse/TECHRPTS/2001/usccse2001-510/usccse2001-510d.doc)
23
© 2014 HCMC Software Testing Club
THANK YOU