49
Discover the world at Leiden University Testing Object Oriented Software Marcello Bonsangue LIACS, Fall 2017
Discover the world at Leiden University
Testing Object Oriented Software
Marcello Bonsangue LIACS, Fall 2017
Discover the world at Leiden University
�Marcello BONSANGUE
�Hui FENG ([email protected])
Course Information
Date Hours Class
September 11, 18, 25October 2, 9, 16, 23, 30November 6, 13, 20, 27
11:00 – 12:45 Huygens 207
December 4 11:00 – 12:45 Snellius B01
December 18 9:00 – 14:0014:30 – 17:00
Snellius 174
Mini Hackathon
☺
Discover the world at Leiden University
Course programme
Introduction to the foundation of software testing
1. Fundamentals of testing
a. What is testing
b. Testing throughout the software life cycle
2. Coverage techniques
3. Generating test cases
4. Testing object-oriented programs
5. Design and programming assignment of a small tool support for testing an object-oriented program
Grade = 40% in-class question + 50 % assignment + 10% participation
Discover the world at Leiden University
What is testing?
11/27/2017
The origins of computing
• 1801: J.M. Jacquard uses punch cards to control a sequence of operations for textile industry
• No real computations, but inspired Charles Babbage in 1820 for storing programs in his Analytical Engine
- Sequential control, branching, looping
What is testing?
11/27/2017What is testing?
Software may contains errors (1843)
“an analyzing process must equally havebeen performed in order to furnish theAnalytical Engine with the necessaryoperative data; and that herein may alsolie a possible source of error. Grantedthat the actual mechanism is unerring inits processes, the cards may give it wrongorders”
Ada Byron, Countess of Lovelace (notes onBabbage’s Analytical Engine)
11/27/2017
The first bug
“It has been just so in all of my inventions.The first step is an intuition, and comeswith a burst, then difficulties arise—thisthing gives out and [it is] then that'Bugs'—as such little faults and difficultiesare called—show themselves and monthsof intense watching, study and labor arerequisite …” – Thomas Edison, 1878
What is testing?
11/27/2017
The first actual bug is found
A moth trapped in a relay. – Grace Hopper, 1947
What is testing?
11/27/2017
Why we do tests?
To avoid failures in software
• NASA Mars Polar Lander crash due to a software integration fault(misunderstanding in unit measures)
• Arianne 5 explosion due to a fault exception-handling
• Therac-25 radiation machine: people died due to a faultinterface
We want programs to be reliable!!
What is testing?
11/27/2017
Do You Trust Your System?
The real wonder is that the system works
as well as it does
(Peterson, 1996)
but remember that software systems provide theinfrastructure in virtually all industries today:
air traffic control
Automotive systems
water level management
energy production and distribution
…
What is testing?
11/27/2017
Airbus 319 Safety Critical Software Control
October 2005, BA Flight London – Bucarest
More than 2 minutes flight in degraded conditions
- Loss of autopilot - Loss of radio power
- Loss of most navigation displays - No MAYDAY possible !
What is testing?
11/27/2017
How do We Build Trust in Our Systems?
Any engineering process consists of
- construction activities
- techniques to check intermediate and final products
Testing is one technique to increase our confidence in the correctness of a software system.
Whenever we use software, we incur some risk
- Risk may be small and consequences unimportant
- Risk may be great and the consequences catastrophic
What is testing?
11/27/2017
Some statistics
o 30-85 errors are made per 1000 lines of source code
o Extensively tested software contains 0.5-3 errors per 1000 lines
of source code
o The later an error is discovered, the more it costs to fix it.
o Error distribution: 60% design, 40% implementation.
o 66% of the design errors are not discovered until the software
has become operational.
What is testing?
11/27/2017
Relative cost of error correction
What is testing?
1
2
5
10
20
50
100
Spec Design Code Test Operational
11/27/2017
Validation and Verification
Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage
- Are we building the right system?
Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase
- Are we building the system right?
What is testing?
11/27/2017
Validation and Verification
What is testing?
Are we building the right system? Are we building the system right?
11/27/2017
Testing, verification, and validation
What is testing?
11/27/2017
Software verification
Software has some characteristics that make verification difficult:
• Many different quality requirements
• Evolving structure
• Uneven distribution of faults
If an elevator can safely carry a load of 1000 kg, it can also safely carry any smaller load
If a procedure correctly sorts a set of 256 elements, it may fail on a set of 255 or 53 or 12 elements, as well as on 257 or 1023
What is testing?
11/27/2017
Specification and program
What is testing?
11/27/2017
Specification and program
What is testing?
Static analysis
11/27/2017
Specification and program
What is testing?
Dynamic Analysis
11/27/2017
Specification and program
What is testing?
Test value generation
11/27/2017
Specification and program
What is testing?
Run-time verification
11/27/2017
Specification and program
What is testing?
Learning specification
11/27/2017
Specification and program
What is testing?
Visualization
11/27/2017
Software Faults, Errors & Failures
Patient describes
symptoms(failures)Doctor must find
cause of symptoms
(fault)
It looks for anomalous conditions,
eg. high blood pressure, or high cholesterol(errors)
What is testing?
11/27/2017
Software Faults, Errors & Failures
Software fault: A static defect in the software, the root of the failures. They are design mistakes.
Software error: An incorrect internal state that is the manifestation of some fault
Software failure: External, incorrect behavior with respect to the requirements or other description of the expected behavior
What is testing?
11/27/2017
An example
Public static int numZero (int[ ] x) {
// return the number of occurrences of 0 in array x
int count = 0
for (int i=1; i < x.length; i++) {
if (x[i] == 0) {
count ++
}
}
return count
}
Input x = [2,7,0]
Output count =1
Input x = [0,2,7]
Output count = 0
What is testing?
11/27/2017
An example
Public static int numZero (int[ ] x) {
// return the number of occurrences of 0 in array x
int count = 0
for (int i=1; i < x.length; i++) {
if (x[i] == 0) {
count ++
}
}
return count
}
Input x = [2,7,0]
Output count =1
Expected count = 1
Input x = [0,2,7]
Output count = 0
Expected count = 1
fault
Both executions result in an error (the fault is executed)
but only the second is a failure
What is testing?
11/27/2017
Testing Central Issue
Given a fault, not all inputs will trigger the fault into a failure.
Problem: finding inputs that cause a software failure
What is testing?
11/27/2017
Fault and Failure Model
Three conditions necessary for a failure to be observed
1. Reachability: The location or locations in the program that contain the fault must be reachable
2. Infection: The subsequent states of the program must be incorrect
3. Propagation: The infected state must propagate to cause some output of the program to be incorrect
What is testing?
11/27/2017
What is testing?
Testing is not the same as debugging or troubleshooting
Debugging: the process of efficiently finding a fault given a failure
Troubleshooting: The process of solving a fault given a software failure
What is testing?
11/27/2017
Testing can only show the presence of failures, and not the correctness of a program
What if there are no failures?
- Good software or bad tests
But correctness via testing is impossible to achieve
- Edsger Dijkstra: Program testing can be used to show the presence of bugs, but never to show their absence!
What is testing?
What is testing?
11/27/2017
Testing: the process of systematic evaluation of software by observing its execution possibly identifying some of the failures
- A failure is any variance between actual and expected observable results.
Testing is one way to increase quality of software.
It is part of verification and validation process in which testers and developers work together to reduce the software risk.
What is testing?
What is testing?
11/27/2017
Testing process
What is testing?
oracle
Prog
Progtest
strategycompare
subset ofinput
subset ofinput
expectedoutput
realoutput
testresults
Objective 1: find as many faults as possible
Objective 2: increase confident that the software works
When have we tested enough?
How do we find input
values?
11/27/2017
Observability and Controlability
Software observation: observing the behavior of a program in terms of its outputs, effects on the environment and other hardware or software components
Software control: providing a program with the needed inputs, in terms of values, operations, and behaviors, to obtain a certain output
What is testing?
11/27/2017
Test Cases
Test case values: the input necessary to complete an execution of a program
Expected results: The result that will be producedwhen executing the test if the program satisfies its intended behavior
Test case: The test case values, expected results, any other inputs necessary to start and conclude the execution
Test cases are used to determine if a program satisfies a test requirement
What is testing?
11/27/2017
Test Suite
Test suite: a set of test cases
Executable test script : A test case that is prepared in a form to be executed automatically on the test software and produce a report
What is testing?
11/27/2017
Constructing a Test Suite
Test coverage: define a model of the software, then find ways to cover it
Test requirements : specific things that must be satisfied or covered during testing
- Requirements that cannot be satisfied are called infeasible
Coverage Criterion : A collection of rules and a
process that define test requirements on a test suite
What is testing?
11/27/2017
An example: Jelly beans
• Lemon, Apricot Yellow
• Orange,Tangerine Orange
• Pistachio Green
• Pear White
• Test requirement
- All flavors {Lemon, Orange, Tangerine, Apricot, Pistachio, Pear}
- All colors {Yellow, Orange, Green, White}
- Infeasible test requirement {Orange, Purple}
• Coverage Criterion : A strategy for selecting jelly beans
What is testing?
11/27/2017
Adequacy of a test suite
• Ideally – exhaustively test everything
• Practically impossible
• Test coverage criteria are measures of adequacy to increase the confidence that we have tested enough
What is testing?
11/27/2017
Advandages of coverage criteria
Gives testers a “stopping rule” … when testing is finished
Can be used for test case generator: look for a test which exercises some statements not covered by the tests so far
Can be used as measurement: a test suite that covers 80% of a criteria is better than one which covers 70%
Can be well supported with powerful tools
Provide traceability from software artifacts to tests
- Source, requirements, design models, …
Make regression testing easier
What is testing?
11/27/2017
Coverage
Given a set of test requirements TR for coverage criterion C, a testsuite T satisfies C coverage if and only if for every test requirementtr in TR, there is at least one test t in T such that t satisfies tr
A coverage criterion C1 subsumes C2 if and only if every
test set that satisfies C1 also satisfies C2
What is testing?
11/27/2017
Static and Dynamic Testing
•Static Testing: Testing without executing the program- This include software inspections and some forms of static
analyses
- Very effective at finding certain kinds of problems –especially “potential” faults, that is, problems that could lead to faults when the program is modified
•Dynamic testing: Testing by executing the program with real inputs
What is testing?
11/27/2017
Manual Test Techniques
� reading, informal versus peer review
� walkthrough and inspections
Example: Fagan Inspection
• Going through the code, statement by statement
• Team with ~4 members, with specific roles:
- moderator: organization, chairperson
- code author: silent observer
- (two) inspectors, readers: paraphrase the code
• Uses checklist of well-known faults
• Result: list of problems encountered
What is testing?
11/27/2017
Example checklist
• Wrong use of data: variable not initialized, dangling pointer, array index out of bounds, …
• Faults in declarations: undeclared variable, variable declared twice, …
• Faults in computation: division by zero, mixed-type expressions, wrong operator priorities, …
• Faults in relational expressions: incorrect Boolean operator, wrong operator priorities, ….
• Faults in control flow: infinite loops, loops that execute n-1 or n+1 times instead of n, ...
What is testing?
11/27/2017
White and Black Box Testing
Black-box testing: Deriving tests from external descriptionsof the software
- Requirements, specification, design
White-box testing: Deriving test case from the source code internals
- Conditional, statements, internal state
What is testing?
11/27/2017
Stress Testing
Stress testing: Tests that are at the limit of the software’s expected input domain
- Very large numeric values (or very small)
- Very long strings
- Very large files
- Many users making requests at the same time
- Invalid values
What is testing?
11/27/2017
Monkey Testing
Monkey testing: A human sits at the keyboard, wiggles the mouse and bangs the keyboard
- No automation
- No theory
- Minimal training required
What is testing?
