Best Practices in Software Development:TestingAnsgar FehnkerBased on slides by: Arend Rensink, Christoph Bockisch, Lodewijk Bergmans
Testing Agile changes the role of testing Traditional After full integration resembles final product most...
During integration interfaces and integration are error-prone...
When unit is ready to ensure proper quality is delivered
On each implemented feature test as soon as it’s ready requires small-grained test cases
Agile approach Implement tests before functionality
Best Practices in Software Development 2
However, principles of testing remain
Testing PreliminariesSoftware quality assessment divide into two categories:
Static analysis It examines the code and reasons over all behaviors that might
arise during run time Examples: Code review, inspection, and algorithm analysis
Dynamic analysis Actual program execution to expose possible program failure One observe some representative program behavior, and reach
conclusion about the quality of the system
Best Practices in Software Development 3From Naik
and Tripathy
Testing PreliminariesVerification Evaluation of software system that help in determining whether the
product of a given development phase satisfy the requirements established before the start of that phase Building the product correctly
Validation Evaluation of software system that help in determining whether the
product meets its intended use Building the correct product
Best Practices in Software Development 4From Naik
and Tripathy
Testing PreliminariesFailure, Error, Fault and Defect Failure A failure is said to occur whenever the external behavior of a
system does not conform to that prescribed in the system specification
Error An error is a state of the system. An error state could lead to a failure in the absence of any
corrective action by the system Fault A fault is the adjudged cause of an error
Defect It is synonymous of fault It a.k.a. bug
Best Practices in Software Development 5From Naik
and Tripathy
Testing Preliminaries
Input vector An input vector is a collection of all data entities read by the routine
whose values must be fixed prior to entering the routine. Members of an input vector can be as follows. Input arguments to the routine Global variables and constants Files Contents of registers (in Assembly language programming) Network connections Timers
6
Testing PreliminariesWhite-box and Black-box Testing White-box testing a.k.a.
structural testing Examines source code with
focus on: Control flow: refers to flow of
control from one instruction to another
Data flow: refers to propagation of values from one variable or constant to another variable
It is applied to individual units of a program
Software developers perform structural testing on the individual program units
Black-box testing a.k.a. functional testing
Examines the program that is accessible from outside
Applies the input to a program and observe the externally visible outcome
It is applied to both an entire program as well as to individual program units
It is performed at the external interface level of a system
It is conducted by a separate software quality assurance group
Best Practices in Software Development 7From Naik
and Tripathy
Testing PreliminariesTesting Terminology Unit testing Individual program units, such
as procedure, methods in isolation
Integration testing Modules are assembled to
construct larger subsystem and tested
System testing Includes wide spectrum of
testing such as functionality, and load
Acceptance testing Customer’s expectations from
the system
Best Practices in Software Development 8From Naik
and Tripathy
Unit Testing
The environment of a unit is emulated and tested in isolation
The caller unit is known as test driver
A test driver is a program that invokes the unit under test (UUT)
It provides input data to unit under test and report the test result
The emulation of the units called by the UUT are called stubs
It is a dummy program
The test driver and the stubs are together called scaffolding
The low-level design document provides guidance for selection of
input test data
Best Practices in Software Development 9
Unit Testing Unit Testing Environment
Best Practices in Software Development 10
Unit Tests (JUnit)
Beste Practices in Software Development 11
Test suite
• A class annotated with@RunWith(Suite.class)@SuiteClasses({TestClass.class, …})
• Groups tests for one system or component
Test case
• A class (e.g., TestClass)• Groups tests for one test item (unit)• Defines common set-up and tear-down operations• @BeforeClass, @AfterClass: before/after all tests• @Before, @After: before/after each test
Test
• A method annotated with@Test
• Implement usage of test item according to specification• Assert that the outcome is correct• Possibly specify timeout and expected exception for test
Techniques to select test dataControl flow testing Identify paths in the CFG to satisfy given selection criteria Derive the path predicate expression from the selection paths Solving the path predicate expression generate the data Data flow testing Use a data flow graph (DFG) from a program unit and then follow
the procedure described in control flow testing. Domain testing Domain errors are defined and then test data are selected to catch
those faults Functional program testing Input/output domains are defined to compute the input values that will
cause the unit to produce expected output values
Best Practices in Software Development 12
Test Selection and Coverage
Path selection criteria
Select paths to achieve complete statement coverage.
Select paths to achieve complete branch coverage.
Select paths to achieve predicate or condition coverage.
Select paths to achieve data flow coverage.
Select all paths.
13
Test Selection and Coverage
1 input(Y)2 if (Y<=0) then3 Y := −Y4 end_if5 while (Y>0) do6 input(X)7 Y := Y-18 end_while
Beste Practices in Software Development 14
2
5
3
6,7
nodes
edges
Pseudo Code and CFG
Beste Practices in Software Development 15
2
5
3
6,7One test case:input(y) -1input(x) don’t care
Test Selection and CoverageHow many test cases required for Statement Coverage?
1 input(Y)2 if (Y<=0) then3 Y := −Y4 end_if5 while (Y>0) do6 input(X)7 Y := Y-18 end_while
2
5
3
6,7
Beste Practices in Software Development 16
Test Selection and CoverageHow many test cases required for Branch Coverage?
1 input(Y)2 if (Y<=0) then3 Y := −Y4 end_if5 while (Y>0) do6 input(X)7 Y := Y-18 end_while
Two test cases:1. input(Y) -1
input(X) don’t care2. input(Y) 1
input(X) don’t care
Test Selection and CoverageDoes Branch Coverage Imply Statement Coverage? Normally, yes… In the absence of dead code
Beste Practices in Software Development 17
Dead code is not reachable via any executable program path.
Under the (reasonable) assumption of the absence of dead code, Branch Coverage implies Statement Coverage.
Test Selection and CoverageCondition Coverage A branch predicate may have more than one condition
1 input(Y)2 if (Y<=0) or (X=0) then3 Y := −Y4 end_if5 while (Y>0) and (not EOF) do6 input(X)7 Y := Y-18 end_while
Condition Coverage requires that each condition has been true at least once has been false at least once.
Beste Practices in Software Development 18
Test Selection and CoverageData Flow A program unit accepts inputs, performs computations, assigns new
values to variables, and returns results. One can visualize of “flow” of data values from one statement to
another. A data value produced in one statement is expected to be used later. Example Obtain a file pointer ……. use it later.
If the later use is never verified, we do not know if the earlier assignment is acceptable.
Two motivations of data flow testing The memory location for a variable is accessed in a “desirable” way. Verify the correctness of data values “defined” (i.e. generated) – observe
that all the “uses” of the value produce the desired results.
Best Practices in Software Development 19
Test Selection and CoverageOccurrences of variables Definition: A variable gets a new value. i = x; /* The variable i gets a new value. */
Undefinition or kill: This occurs if the value and the location become unbound.
Use: This occurs when the value is fetched from the memory location of the variable. There are two forms of uses of a variable. Computation use (c-use) Example: x = 2*y; /* y has been used to compute a value of x. */
Predicate use (p-use) Example: if (y > 100) { …} /* y has been used in a condition. */
20
Test Selection and CoverageExample data flow criterion: All du-paths: Include all paths from a definition to a use (p-use or c-
use)
Best Practices in Software Development 21
int binsearch(int X, int V[], int n){int low, high, mid;low = 0;high = n - 1;while (low <= high) {
mid = (low + high)/2;if (X < V[mid])
high = mid - 1;else if (X > V[mid])
low = mid + 1;else
return mid;}return -1;
}
123456789101112131415
Test Selection and Coverage Path Coverage requires that all program paths have been traversed at least once
Often described as the “strongest” form of code coverage Usually impossible to achieve when loops are present Other criteria: Loop coverage, Bock coverage
Tool support
Tools exist to determine coverage given a citerion Tools exist to generate input vectors to achieve coverage
Beste Practices in Software Development 22
Test Selection and CoverageMutation Testing
Used to “test” if the tests cover the code Modify a program by introducing a single small change to the code.
The modified program is called mutant Determine the quality of a set of test, by how many mutants it
discovers A mutant is said to be killed when the execution of test case cause
it to fail. A mutant is an equivalent to the given program if it always produce
the same output as the original program A mutation score for a set of test cases is the percentage of non-
equivalent mutants killed by the test suite New test need to be added to ensure all mutants are detected.
Best Practices in Software Development 23
Test Selection and CoverageConsider the following program
main(int argc, char *argv) {
int r = 1;
for(int i = 2;i< 3;i++)
{
if (argv[i] > argv[r])
{
r = i;
}
}
cout<<“Result: ” <<r;
exit(0);
}
Test Case 1: input: 1 2 3 output: Result: 3
Test Case 2: input: 1 2 1 output: Result: 2
Test Case 3: input: 3 1 2 output: Result: 1
Best Practices in Software Development 24
Mutant 1: Change line 7 to r = 1;• Programs will not pass the test suite (fail case 1 and 2)• Mutant 1 is killed by current test suite.
Test Selection and CoverageConsider the following program
main(int argc, char *argv) {
int r = 1;
for(int i = 2;i< 3;i++)
{
if (argv[i] > argv[r])
{
r = i;
}
}
cout<<“Result: ” <<r;
exit(0);
}
Test Case 1: input: 1 2 3 output: Result: 3
Test Case 2: input: 1 2 1 output: Result: 2
Test Case 3: input: 3 1 2 output: Result: 1
Best Practices in Software Development 25
Mutant 3: Change line 5 to if (argv[i] >= argv[r])• Programs will pass the test suite• However, mutant 3 is killable. Add one more test (2 2 1)
Test Selection and CoverageConsider the following program
main(int argc, char *argv) {
int r = 1;
for(int i = 2;i< 3;i++)
{
if (argv[i] > argv[r])
{
r = i;
}
}
cout<<“Result: ” <<r;
exit(0);
}
Test Case 1: input: 1 2 3 output: Result: 3
Test Case 2: input: 1 2 1 output: Result: 2
Test Case 3: input: 3 1 2 output: Result: 1
Best Practices in Software Development 26
Mutant 3: Change line 3 to for(int i = 1; …)• Programs will pass the test suite• Mutant 3 is equivalent, i.e., is not killable.
Behavior-Driven Test Implementation Behavioral specification of tests: user story Title Narrative (introduction) Scenario (acceptance criteria)
Documentation uses specific keywords
Narrative: In order to <achieve some business value> As a <stakeholder> I want to <effect to achieve>
Beste Practices in Software Development 27
Behavior-Driven Development Scenario <description> Given <prerequisite> And <additional prerequisite> When <what triggers the scenario> Then <expected outcome> And <additional expected outcome>
Typical structure of test code Making this structure explicit helps documenting tests Be as descriptive as possible!
Supported by tools, e.g., JBehave, Hamcrest
Beste Practices in Software Development 28
Unit Testing in eXtreme Programming Three laws of Test Driven development (TDD)
1. One may not write production code unless the first failing unit test is written
2. One may not write more of a unit test than is sufficient to fail 3. One may not write more production code than is sufficient to make
the failing unit test pass
Pair programming: In XP code is being developed by two programmers working side by
side One person develops the code tactically and the other one inspects it
methodically by keeping in mind the story they are implementing
Best Practices in Software Development 29
Unit Testing in eXtreme Programming
1. Pick a requirement, i.e., a story 2. Write a test case that will verify
a small part of the story and assign a fail verdict to it
3. Write the code that implement particular part of the story to pass the test
4. Execute all test 5. Rework on the code, and test
the code until all tests pass 6. Repeat step 2 to step 5 until the
story is fully implemented
Best Practices in Software Development 30
Integration Testing
A software module is a self-contained element of a system Modules are individually tested commonly known as unit testing Next major task is to put the modules, i.e., pieces together to construct
the complete system Construction of a working system from the pieces is not a
straightforward task because of numerous interface errors
Interface errorsInterface errors are those that are associated with structures existing outside
the local environment of a module, but which the module uses
31
Integration Testing
The objective of system integration testing (SIT) is to build a “working” version of the system Putting modules together in an incremental manner Ensuring that the additional modules work as expected without disturbing
the functionalities of the modules already put together Integration testing is said to be complete when The system is fully integrated together All the test cases have been executed All the severe and moderated defects found have been fixed
32
System Tests
Basic tests provide an evidence that the system can be installed, configured and be brought to an operational state
Functionality tests provide comprehensive testing over the full range of the requirements, within the capabilities of the system
Robustness tests determine how well the system recovers from various input errors and other failure situations
Inter-operability tests determine whether the system can inter-operate with other third party products
Performance tests measure the performance characteristics of the system, e.g., throughput and response time, under various conditions
Scalability tests determine the scaling limits of the system, in terms of user scaling, geographic scaling, and resource scaling
33
System Tests
Stress tests put a system under stress in order to determine the limitations of a system and, when it fails, to determine the manner in which the failure occurs
Load and Stability tests provide evidence that the system remains stable for a long period of time under full load
Reliability tests measure the ability of the system to keep operating for a long time without developing failures
Regression tests determine that the system remains stable as it cycles through the integration of other subsystems and through maintenance tasks
Documentation tests ensure that the system’s user guides are accurate and usable
34
Acceptance Testing
Acceptance testing is a formal testing conducted to determine whether a system satisfies its acceptance criteria
There are two categories of acceptance testing: User Acceptance Testing (UAT) It is conducted by the customer to ensure that system satisfies the
contractual acceptance criteria before being signed-off as meeting user needs.
Business Acceptance Testing (BAT) It is undertaken within the development organization of the supplier to
ensure that the system will eventually pass the user acceptance testing.
35
Acceptance Testing
Three major objectives of acceptance testing:
Confirm that the system meets the agreed upon criteria
Identify and resolve discrepancies, if there is any
Determine the readiness of the system for cut-over to live operations
36
Beta Testing
Beta testing is conducted by the potential buyers prior to the official release of the product
The purpose of beta testing is not to find defects, but to obtain feedback from the field about the usability of the product
Three kinds of beta: Marketing beta Technical beta Acceptance beta
37
Test Design Process
One test case is created for each test objective
Figure 11.6: State-transition diagram of a test case. 38
Test Results
Figure 11.7: State transition diagram of a test case results
39
Modeling Defects
40
Testing PreliminariesThe Objectives of Testing It does work It does not work Reduce the risk of failures Reduce the cost of testing Reduce the cost of development
…
Best Practices in Software Development 41