Date post: | 21-Dec-2015 |
Category: |
Documents |
View: | 215 times |
Download: | 0 times |
Topic 12 Summer 2003 1
ICS 52: Introduction to Software Engineering
Lecture Notes for Summer Quarter, 2003
Michele RousseauTopic 12
Partially based on lecture notes written by Sommerville, Frost, Van Der Hoek, Taylor & Tonne. Duplication of course material for any commercial purpose
without the written permission of the lecturers is prohibited
Topic 12 Summer 2003 2
Verification and Validation
Informal Requiremen
ts
Informal Requiremen
ts
Formal Specificatio
n
Formal Specificatio
n
Software Implementati
on
Software Implementati
on
Validation
Verification
Verification: is implementation consistent with requirements specification?Validation: does the system meet the customer’s/user’s needs?
Topic 12 Summer 2003 3
Software Quality: assessment by V&V
Software process must include verification and validation to measure product qualities • correctness, reliability, robustness• efficiency, usability, understandability • verifiability, maintainability• reusability, portability, interoperability,• real-time, safety, security, survivability, accuracy
Products can be improved by improving the process by which they are developed and assessed
Topic 12 Summer 2003 4
Testing Terminology
Failure: Incorrect or unexpected output, based on specifications• Symptom of a fault
Fault: Invalid execution state• Symptom of an error• May or may not produce a failure
Error: Defect or anomaly or “bug” in source code• May or may not produce a fault
Topic 12 Summer 2003 5
1: input A,B
2: A>0?
3: C :=0 4: C := A*B
5: B>0?
6: X := C*(A+2*A) 7: X := A+B
8: output X
Examples: Faults, Errors, and Failures
Suppose node 6 should be X:= C*(A+2*B)
• Failure/Fault-less error: - Suppose the inputs are
(A=2,B=1) – the executed path
will be (1,2,4,5,7,8) which will not reveal this fault because 6 is not executed
- Suppose the inputs are (A=-2,B=-1) – the executed path will be (1,2,3,5,6,8) which will not reveal the fault because C = 0
Need to make sure proper test cases are selected
• the definitions of C at nodes 3 and 4 both affect the use of C at node 6- executing path (1,2,4,5,6,8) will reveal the failure, but only if B != 0 - (e.g. Inputs (A=1,B=-2) )
Topic 12 Summer 2003 6
Software Testing
Exercising a system [component] • on some predetermined input data• capturing the behavior and output data• comparing with test oracle• for the purposes of
» identifying inconsistencies » verifying consistency between actual results and specification
to provide confidence in consistency with requirements and measurable qualities
to demonstrate subjective qualities
» validating against user needs Limitations
• only as good as the test data selected• subject to capabilities of test oracle
Topic 12 Summer 2003 7
Definition to execution (Levels of testing)
System Testing • Defined at Requirements -> Run after integration testing
Integration Testing• Defined at Design -> Run after Module Testing
Module Testing• Defined at Design -> Run after Unit Testing
Unit Testing• Defined at Implementation -> Run after Implementation of
each unit Regression Testing (testing after Change)
• Defined throughout the process -> Run after modifcations
Topic 12 Summer 2003 8
Requirements Analysis
Architectural (Preliminary)
Design
Algorithmic (Detailed)
Design
Implemen- tation
Requirements Specification
Module Interface
Specification
Module Design
Specification
Source Code
System Testing
Integration Testing
Module Testing
Unit Testing
Problem Definition
System Test Plan
Integration Test Plan
Module Test Plan
Unit Test Plan
Behavior verification
against oracle
MaintenanceRS/MIS/ MDS/SC
Modification
Regression Testing
Regression Test Plan
Testing-oriented Life Cycle Process
Topic 12 Summer 2003 9
How to make the most of limited resources?
Fundamental Testing Questions
Test Criteria: What should we test? Test Oracle: Is the test correct? Test Adequacy: How much is enough? Test Process: Is our testing effective?
Topic 12 Summer 2003 10
Practical Issues
Purpose of testing• Fault detection• High assurance of reliability• Performance/stress/load• Regression testing of new versions
Conflicting considerations• safety, liability, risk, customer satisfaction,
resources, schedule, market windows and share
Test Selection is a sampling technique• choose a finite set from an infinite domain
Topic 12 Summer 2003 11
Test Criteria
Testing must select a subset of test cases that are likely to reveal failures
Test Criteria provide the guidelines, rules, strategy by which test cases are selected• actual test data• conditions on test data• requirements on test data
Equivalence partitioning is the typical approach• a test of any value in a given class is equivalent to a
test of any other value in that class• if a test case in a class reveals a failure, then any
other test case in that class should reveal the failure• some approaches limit conclusions to some chosen
class of errors and/or failures
Topic 12 Summer 2003 12
Test Adequacy
Coverage metrics• when sufficient percentage of the program
structure has been exercised Empirical assurance
• when failures/test curve flatten out Error seeding
• percentage of seeded faults found is proportional to the percentage of real faults found
Independent testing• faults found in common are representative of
total population of faults
Topic 12 Summer 2003 13
Functional and Structural Testing
Functional Testing• Test cases selected based on specification• Views program/component as black box
Structural Testing• Test cases selected based on structure of
code• Views program /component as white box
(also called glass box testing)
Topic 12 Summer 2003 14
Black Box vs. White Box Testing
SELECTED INPUTS
RESULTANT OUTPUTS
INTERNAL BEHAVIOR
DESIRED OUTPUT
SOFTWARE DESIGN
“BLACK BOX” TESTING
“WHITE BOX” TESTING
SELECTED INPUTS
RESULTANT OUTPUTS
DESIRED OUTPUT
Topic 12 Summer 2003 15
Structural (White Box) Testing
Use source code to derive test cases Build flow graph of program State test requirements in terms of graph
coverage Various types of coverage identified Choose test cases that guarantee different types of
coverage• All-Statements coverage• All-Branches coverage• All-Edges• All-Paths
Topic 12 Summer 2003 16
Graph representation of control flow and data flow relationships
Flow Graphs
Control Flow• The partial order of statement execution, as
defined by the semantics of the language Data Flow
• The flow of values from definitions of a variable to its uses
Topic 12 Summer 2003 17
123456789
101112131415
function P return INTEGER isbegin
X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop
X := X – 10;exit when X = 10;
end loop;if (Y < 20 and then X mod 2 = 0) then
Y := Y + 20;else
Y := Y – 20;end if;return 2*X + Y;
end P;
A Sample Program to Test
Topic 12 Summer 2003 18
2,3,4 5
6
9´
10
12
14
T T
F
F 9 T
F
7
TF
9a 9b
P’s Control Flow Graph (CFG)
function P return INTEGER isbegin
X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop
X := X – 10;exit when X = 10;
end loop;
12345678
9101112131415
if (Y < 20 and then X mod 2 = 0) then
Y := Y + 20;else
Y := Y – 20;end if;return 2*X + Y;
end P;
Topic 12 Summer 2003 19
All-Statements Coverage
Select test cases such that every node in the graph is visited• Also called node coverage
» Guarantees that every statement in the source code is executed at least once
Selects minimal number of test cases
1 3 7 82 4 5 6 9 10
Topic 12 Summer 2003 20
2,3,4 5
6 10
12
14
TF
9
T
F
7
TF
At least 2 test cases needed
Example all-statements-adequate test set:(X = 20, Y = 10)
<2,3,4,5,6,7,9,10,14>(X = 20, Y = 30)
<2,3,4,5,6,7,9,12,14>
All-Statements Coverage of P
Topic 12 Summer 2003 21
All-Branches Coverage
Select test cases such that every branch in the graph is visited
» Guarantees that every branch in the source code is executed at least once
More thorough than All-Statements coverage• More likely to reveal logical errors
1 3 7 82 4 5 6 9 10
Topic 12 Summer 2003 22
2,3,4 5
6 10
12
14
TF
9
T
F
7
TF
At least 2 test cases needed
Example all-branches-adequate test set:(X = 20, Y = 10)
<2,3,4,5,6,7,9,10,14>(X = 15, Y = 30)
<2,3,4,5,6,7,5,9,12,14>
All-Branches Coverage of P
Topic 12 Summer 2003 23
All-Edges Coverage
Select test cases such that every edge in the graph is visited
» Takes complex statements into consideration – treats them as separate nodes
More thorough than All-Branches coverage• More likely to reveal logical errors
Topic 12 Summer 2003 24
2,3,4 5
6
9b
10
12
14
T T
F
F9a
T
F
7
TF
At least 3 test cases needed
Example all-edges-adequate test set:(X = 20, Y = 10)
<2,3,4,5,6,7,9a,9b,10,14>(X = 5, Y = 30)
<2,3,4,5,9a,12,14>(X = 21, Y = 10)
<2,3,4,5,6,7,5,6,7,5,9a,9b,12,14>
All-Edges Coverage of P
Topic 12 Summer 2003 25
All-Paths Coverage
Path coverage• Select test cases such that every path in the
graph is visited• Loops are a problem
» 0, 1, average, max iterations
Most thorough… …but is it feasible?
Topic 12 Summer 2003 26
2,3,4 5
6
9b
10
12
14
T T
F
F9a
T
F
7
TF
Infinitely many test cases needed
Example all-paths-adequate test set:(X = 5, Y = 10)(X = 15, Y = 10)(X = 25, Y = 10)(X = 35, Y = 10)…
All-Paths Coverage of P
Topic 12 Summer 2003 27
2,3,4 5
6
9b
10
12
14
T
F
9aT
FY
X
X
Y
Y X
X
Y
YXX
X
TF
7
TF X
X
P’s Control and Data Flow Graph
Topic 12 Summer 2003 28
Subsumption of Criteria
C1 subsumes C2 if any C1-adequate testT is also C2-adequate• But some T1 satisfying C1 may detect fewer
faults than some T2 satisfying C2
Topic 12 Summer 2003 29
all-statements
all-edges
boundary-interiorloop testing
min-maxloop testing
all-paths
all-defs
all-uses
all-DU-paths
all-p-uses all-c-uses
C1 C2
subsumes all-branches
Structural Subsumption Hierarchy
Topic 12 Summer 2003 30
Equivalence partitioning
Identify the set of all inputs Identify possible bases for subdividing the input Domain:
•size, order, structure•correctness•stated requirements & your smarts
Divide input set into (sub)domains using the basis•“Equivalence partitions”• Subdomains may overlap (may not be a partition)
Select representative test cases from each subdomain•one test case may suffice, more is better
Topic 12 Summer 2003 31
Example
1 float homeworkAverage(float[] scores) {
2 float min = 99999;
3 float total = 0;
4 for (int i = 0 ; i < scores.length ; i++) {
5 if (scores[i] < min)
6 min = scores[i];
7 total += scores[i];
8 }
9 total = total – min;
10 return total / (scores.length – 1);
11 }
Topic 12 Summer 2003 32
Possible Bases
Array length• empty array• one element• 2 or 3 elements • 4 or more elements
Input domain: float[]Basis: array length
one
small
emptylarge
} subdomains, eq. partitions
Topic 12 Summer 2003 33
Possible Bases
Position of minimum score• Smallest element first• Smallest element in middle• Smallest element last
Input domain: float[]Basis: position of minima
somewhere in middlefirst last
Topic 12 Summer 2003 34
Possible Bases
Number of minima• Unique minimum• A few minima• All minima
Input domain: float[]Basis: number of minima
all data equal1 minimum2 minima
Topic 12 Summer 2003 35
Equivalence partitions & white-box testing
The basis - equivalence partition - test case approach works for structural testing, too.
A basis could be nodes, edges, or paths.
Each node, edge, or path defines a partition of the input.
Topic 12 Summer 2003 36
Challenges
Structural testing can cover all nodes or edges without revealing obvious faults• No matter what input, program always returns 0
Some nodes, edges, or loop combinations may be infeasible• Unreachable/unexecutable code
“Thoroughness”• A test suite that guarantees edge coverage also
guarantees node coverage…• …but it may not find as many faults as a
different test suite that only guarantees node coverage
Topic 12 Summer 2003 37
Limitations of Structural Testing
Interactive programs Listeners, event-driven programs Concurrent programs or tasks; threads Exceptions Self-modifying programs (assembly language) Objects loaded over the network Super-class constructors Asynchronous garbage collection
Topic 12 Summer 2003 38
Summary of Structural Testing
Conceptually simple Relies on access to source Not applicable in all situations Acts in ignorance of specifications
Topic 12 Summer 2003 39
Ensure that changes made during maintenancedo not destroy existing functionality
Regression Testing
Permanent suite of test cases• Saves effort creating test cases• Provides record of existing functionality
Add new test cases and delete obsolete ones when necessary
Topic 12 Summer 2003 40
Cost reduction:Select minimal subset of
regression test suite that tests the changes
Selective Regression Testing
Analyze relationship between the test cases and the software entities they cover
Use information about changes to select test cases for new version
Topic 12 Summer 2003 41
= modified entity
System Under Test
Test 2Test 1 Test 3
= exercises relation
= entity
Simple Example
Topic 12 Summer 2003 42
System Under Test
Test 3
= exercises relation= entity
= modified entity
Selective Regression Testing
Topic 12 Summer 2003 43
Cost of running necessary tests
Cost of running necessary tests
RetestAll
Cost-IneffectiveSelective
Retest
Cost of running necessary tests
Cost
Cost-EffectiveSelective
Retest
Cost of running unnecessary tests
Cost of additional analysis
Cost of additional analysis
The Cost-Effectiveness Tradeoff
Topic 12 Summer 2003 44
The Reality of Software Testing
One of the most widely-used, practical class of techniques for revealing faults in software
An area of software engineering showing the greatest gap between theory and practice