Date post: | 08-Jan-2017 |
Category: |
Software |
Upload: | ho-chi-minh-city-software-testing-club |
View: | 1,057 times |
Download: | 1 times |
Analytical:
Risk-based and Specification-
based Testing
SPEAKER: TAM BUI
About me
• B. A in English in 1996 – University of Hue.
• B. Eng in Computer Science in 2005 – Hanoi
University of Science and Technology.
• ISTQB – Advanced Level – Test Manager in USA in
2013.
• Over 14 year experiences in software testing.
• Strong knowledge and experiences in automation
testing, performance testing and security testing.
• Great passion in software testing: learning and
sharing knowledge and experiences.
2
Objectives
• What is a Test Strategy?
• What is about Analytical Test Strategy?
• Risk-based Testing.
• Specification-based Testing.
• Benefits of Analytical Test Strategy.
3
What are Test Objectives?
• Finding Defects.• Preventing Defects.• Providing Quality Information.• Gaining confidence about the level of quality.• Satisfying the needs of stakeholders: Scope, Time, and
Budget.• Satisfying the Business/System Requirement
Specification.
4
What is a Test Strategy?
A Test Strategy is a document to be developed to inform project managers, testers, developers how to achieve test objectives.
5
Types of Test Strategies
AnalyticalModel-based
MethodicalStandard-compliant
Dynamic ConsultativeRegression-
averse
6
Test objectives in real projects
Project 1You are a Test Manager. You are assigned to work with a new project with some difficult objectives: large scope with short time and resource shortage.
Project 2You are a Test Manager. You are assigned to work with a new project to develop a safety critical system which will be used in the surgery field.
Which Test Strategy is selected?
Analytical
7
What is about Analytical
Strategy?
All test planning activities are based on data and analysis of data
Risk-based TestingSpecification-based Testing
8
Risk-based Testing
• What is Software Risk?
• Risk Management.
9
What is Software Risk?
A risk is something that has not happened yet and it may never happen. It is a potential problem.
Risk is the possibility of a negative or undesirable outcome.
A risk has some likelihood between 0% and 100%.
10
Software Risk Areas
Functional complexity
Performance
Safety
Data Selections
Recoverability
New Technology
Scalability and Reliability
Scope of Use
Environment
Security
Reliability
Usability
Interface Complexity Technical Complexity
11
Risk Management
Risk Identification
Risk Control, Mitigation
Risk Analysis
12
Risk Identification
Expert interviews
Independent assessments
Use of risk templates
Lessons learned• Metrics and data from past projects.• Develop a risk database or repository
13
Risk Identification (Cont.)
Formal analysis methods• Failure modes and effects analysis (FMEA)• Hazard analysis, Cost of failure, Other
Risk workshops
Brainstorming
Checklists
Calling on past experience
14
Risk Analysis – Likelihood
(Technical Factors)
Complexity of technology
Personnel issues
Trainings
Intra-team and inter-team issues
Supplier and Vendor contractual problems
Geographical distribution of the development organization
New Technologies and designs
Bad quality of tools and technologies used
Bad management and technical leadership
Time, resource, and management pressure
Late testing and bad quality assurance
Requirements, designs, code changes
High defect rate
The frequency of use of the affected feature
15
Risk Analysis – Impact (Business
Factors)
Potential damage to the company image
Loss of Customers and Business
Financial and social losses or liability
Civil or criminal legal sanction
Loss of licenses or permits
The lack of reasonable workarounds
The visibility of failure and the associated negative publicity
16
Risk Analysis – Risk Priority
Number• Apply 5-point-scale for Likelihood and Impact for
each risk:
– 1 = Very High
– 2 = High
– 3 = Medium
– 4 = Low
– 5 = Very Low
Risk Priority Number = Likelihood * Impact
17
Risk Analysis – Extent of
Testing1-5: Extensive: run large number of tests, both broad and deep, combine and vary interesting conditions, use all relevant techniques with strong coverage criteria.
6-10: Broad: run medium number of tests, exercise many different interesting conditions use most relevant different interesting conditions, use most relevant techniques with medium coverage criteria.
11-15: Cursory: run small number of tests, sample most interesting conditions, use efficient techniques with weak interesting conditions, use efficient techniques with weak coverage criteria
16-20: Opportunity: leverage other tests or activities to test 1-2 interesting conditions, investing very little time and effort, using reactive techniques especially.
21-25: Report bugs only: allocate only a small amount of extra time to report and manage these accidental bugs.
18
ExampleRISK LIST WITH EXTENT OF TESTING
No Quality Risk Likehood Impact Risk Pri. # Extent of Testing Tracing
1 Functional Risks
1.1 Function 1 2 5 10 Broad 100-100-100
1.2 Function 2 1 1 1 Extensive 100-100-101
1.3 Function 3 3 5 15 Cursory 100-100-102
1.4 Function 4 4 5 20 Opportunity 100-100-103
1.5 Function 5 5 5 25 Report Bugs 100-100-104
2 Performance Risks
2.1 Function 6 1 1 1 Extensive 100-100-105
2.2 Function 7 2 4 8 Broad 100-100-106
2.3 Function 8 3 3 9 Broad 100-100-107
2.4 Function 9 2 2 4 Extensive 100-100-108
2.5 Function 10 5 5 25 Report Bugs 100-100-109
3 Usabilty Risks
3.1 Function 11 4 4 16 Opportunity 100-100-110
3.2 Function 12 2 3 6 Broad 100-100-111
3.3 Function 13 3 1 3 Extensive 100-100-112
3.4 Function 14 1 1 1 Extensive 100-100-113
3.5 Function 15 5 5 25 Report Bugs 100-100-114
19
Risk Control
Mitigate: Take steps in advance to reduce the possibility and impact of the risk.
Contingency: Have a plan in place to reduce the possibility of the risk to become an outcome.
Transfer: Convince some other member of the team or project stakeholder to reduce the probability or accept the impact of the risk.
Ignore: Ignore the risk, which is usually a good option only when there is little that can be done or when the possibility and impact of that risk are low in the project.
20
Bottom line of Risk-based
Testing
Plan Testing Activities such as design tests,
implement tests, execute tests, report test results according to Risk
Priority Number
21
Specification-based Testing
• What is a Specification?
• What are implicit Specifications?
• How to learn from the Specification.
• Clarify and fix ambiguities – Review techniques.
• Prioritize Software Requirement specification.
• Bottom line of Specification-based Testing.
22
What is a Specification?
A detailed description of work to be done or materials to be used in a project
- Marriam-Webster
A software requirements specification (SRS) is a description of a software system to be developed. It lays out functional and non-functional requirements and may include a set of use cases that describe user interactions that the software must provide.
- Wikipedia
23
What kinds of Specification can
be tested?
Software Requirement Specification
Software Design Specification
User Interface Description
User Manual
24
What are implicit
Specifications?
Characteristics of the product are not mentioned in the
specification.
25
What are implicit Specifications
(Cont.)
Business Culture
Technical Norms
Design/Code Standards
Guidelines
Regulations
Marketing/Sale Presentations
Memos (Software changes)
26
What are implicit Specifications
(Cont.)
Competing products
Related products
Email discussions within the project
Customer comments
Bug Reports
Test Results
Prototypes
27
How to learn from the
Specification?
SQ3R Reading Strategy
Skim Question Read
Respond Review
28
Clarify and Fix Ambiguities –
Review Techniques
Walkthrough
Technical ReviewPeer Review
Inspection
29
Prioritize Requirement
Specifications
MoSCoW
MUSTMandatory
SHOULDHigh Priority
COULDDesired but not necessary
WOULDCan be delayed and proposed
for future releases
30
Example
SOFTWARE REQUIREMENT LIST
Req. ID Requirement Requirement Description Priority
1 Funtionality Requirement
1.1 Requirement 1 Description of Requirement 1 Must
1.2 Requirement 2 Description of Requirement 2 Could
1.3 Requirement 3 Description of Requirement 3 Would
1.4 Requirement 4 Description of Requirement 4 Must
1.5 Requirement 5 Description of Requirement 5 Should
1.6 Requirement 6 Description of Requirement 6 Must
1.7 Requirement 7 Description of Requirement 7 Could
1.8 Requirement 8 Description of Requirement 8 Must
1.9 Requirement 9 Description of Requirement 9 Would
1.10 Requirement 10 Description of Requirement 10 Must
2 Performance Requirements
2.1 Requirement 11 Description of Requirement 11 Should
2.2 Requirement 12 Description of Requirement 12 Could
2.3 Requirement 13 Description of Requirement 13 Should
2.4 Requirement 14 Description of Requirement 14 Must
2.5 Requirement 15 Description of Requirement 15 Should
31
Bottom line of Specification-
based Testing
Study Specifications which include implicit Specifications
Review Specifications and update them if necessary
Specifications are prioritized
Plan testing activities such as design tests, implement tests, execute tests and report test results based on prioritized Specifications
32
Benefits of Analytical Strategy
Risk-base Testing Specification-based Testing
• Exhausted testing is impossible. Important features/functions will be addressed early. Important problems can be discovered early.
• Defects/bugs can be prevented by running static testing.
• Test Managers properly distribute test efforts, schedule time, use budget to mitigate risks and make contingency plan when risks occur.
• Prioritized features/functions will be addressed early. Quality will be improved early.
• Defects/bugs can be prevented by running static testing.
• Tests can be derived easily.
• Customer’s needs and expectations can be satisfied correctly.
• Training new team members is easily.
33
Summary
• Understand common test strategies and select appropriate test strategy for each project.
• Comprehend risk-based testing and how to plan risk-based testing activities.
• Comprehend specification-based testing and how to plan specification-based testing activities.
34
Reference
• Lessons Learned in Software Testing – Cem
Kaner, James Bach, Bret Pettichord.
• Specification-based Testing – Testing Education
and BBST – Cem Kaner.
35
36
37