Software TestingPart II

March, 2001

Fault-based Testing Methodology(white-box)

2

Mutation Testing

3

Klaidos įvedimas

Automatinis programos testavimas Klaidos įvedimas Automatinio testavimo pakartojimas

4

Strong mutation testing Called “fault-based” because it’s directed

at the possible faults that can occur in a program

Provides a measure of test-data adequacy The technique begins by generating test

data by some means; it does not really matter how it is generated (can be randomly generated)

5

Strong mutation testing (cont)

Approach: generate variations of the original program, called “mutants”

The original test data is run through both the original program and the mutant

If there is a difference in the output, then the mutant is dead

6

Strong mutation testing (cont) If the mutant is not killed, then the test data is

not adequate to reveal the fault and distinguish the mutant from the original

If this is the case, then the test data must be augmented to reveal the fault and kill the live mutant

The obvious problem: too many simple faults that can be introduced, O(n^2) for an n line program

7

List of possible faults, Budd (1983)

constant replacement scalar replacement scalar variable for

constant replacement constant for variable array reference for

constant replacement array reference for

scalar variable

constant for array reference replacement

scalar variable for array reference replace

array reference for array reference replace

constant replacement data statement

alteration

8

List of possible faults, Budd (1983)

comparable array name replacement

arithmetic operator replacement

relational operator replacement

logical connector replacement

absolute value insert unary operator insertion statement analysis statement deletion return statement replace goto label replace do statement end

replace

9

How effective is mutation testing?

“Test data that distinguishes all programs differing from a correct one by only simple errors is so sensitive that it also finds complex errors”, DeMillo (1978)

mutation testing makes the “competent programmer” assumption: “programmers write programs that are nearly correct”!

10

Example program:

string searching program: the program prompts the user for a positive integer in the range 1 to 20 and then for a string of characters of that length. The program then prompts for a character and returns the position in the string at which the character was first found or a message indicating that the character was not present in the string. The user has the option to search for more characters.

11

main() {(1 ) char a[20], ch, response = ‘y’;(2) int x, i;(3) bool found;(4) cout << “Input an integer between 1 and 20: “;(5) cin >> x;(6) while ( x < 1 || x > 20) {(7) cout << “Input an integer between 1 and 20: “;(8) cin >> x; }(9) cout << “input “ << x << “ characters: “;(10) for (int i = 0; i < x; ++i) cin >> a[i];(11) cout << endl;

12

(12) while ( response == ‘y’ || response == ‘Y’) {(13) cout << “input character to search: “;(14) cin >> ch;(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;(19) if (found) cout << ch << “ at: “ << i << endl;(20) else cout << ch << “ not in string” << endl;(21) cout << “search for another character? [y/n]”;(22) cin >> response; }

13

mutation testing string example

Assume we have branch tested the program

consider lines 15 to 18

(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

14

Test data for branch testing

x input string search char response expectedoutput

34 wrong input3 abc c pos = 3

yk not in string

n

15

RIP Since the output of the mutant program

differs from the output of the original program, mutant #1 is dead!

It is important to make only one small change so that faults don’t cancel each other

16

mutation testing string example

Assume we have branch tested the program,

create another mutant:(15) found = false;(16) i = 0;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

17

Mutant #2 lives!

The output of the mutant is the same as the original program ==> mutant #2 is alive

Analysis of the program reveals that the test data is not adequate to reveal the fault; add a test case to search first position

18

RIP

x input string search char response expectedoutput

34 wrong input3 abc c pos = 3

ya pos = 1

yk not in string

n

19

Super mutants!

Sometimes it’s impossible to kill a mutant because no test data can be found that will reveal the newly introduced fault

an absurd example:

main() { int x = 3; int y = x*3; if (y < 10) cout << “less”;}

20

Suppose the string processing program had contained a fault!

(15) found = false;(16) i = 1;(17) while (!found && i < x) (18) if (a[i++] == ch) found = true;

21

Strengths of strong mutation testing

Shows the absence of particular faults; contrary to Dijkstra’s comment! By introducing a fault into the program and then showing that the test data reveals it, we know that the fault cannot be there!

Forces the programmer to scrutinize ( kruopščiai ištirti) and analyze the program under test to think of test data to expose the newly introduced fault, i.e., kill the mutant!

22

Weaknesses of strong mutation testing

Computationally expensive! Huge number of mutants can be

generated There are systems that can help

generate mutant programs and test data; but much work must be done by hand.

23

Weak mutation testing Less demanding that strong mutation take some construct of the program and

create some mutants of this construct consider: if (A && B || C), possible mutants:

if (A || B || C) if (A && B && C) if ((A && B) || C) etc.

24

Weak mutation testing (cont) Howden (1981) identified five candidate constructs:

data access (or variable reference) data storage (or variable assignment) arithmetic expression arithmetic relation boolean expression: same as multiple condition

coverage Howden developed reliable tests for the five

constructs

25

Consider data access or variable reference

We’re considering the r-value here. Aim is to ensure that the correct variable is

being accessed. consider the variables in string processing

program, type-by-type integer variables are referenced on lines 9, 10, 17,

19 and 20; make sure we have adequate test data to ensure that they are getting correct values

bool variable is referenced on line 15

26

Consider arithmetic relations:

they occur on lines 6 and 17; test data that ensures that x has values 0, 1, 2, 1, 20 and 21 would test x and i

27

Strengths of weak mutation

Like strong mutation testing, it promotes a thorough analysis of the program under test

computationally cheaper not necessary to physically generate the

mutants

28

weaknesses of weak mutation

Unlike strong mutation, it is not reliable for the program as a whole; adequacy of the data is local to the construct being mutated

Girgis and Woodward (1986) found that weak mutation was outperformed by data flow testing; they also found that different faults were found by each technique and promoted a combination of testing techniques