Applied Software Testing

A crash course in applying SW testing strategies

Birgitta Lindström

2

Overview

Introduction exercise (30 min)

Test Strategies (50 min) White-box Black-box

Assignment introduction (10 min)

3

Exercise

Implement a program that classifies triangles given

Input: 3 parameters that represent the length of the edges Example : Triclass 10 10 10

Output: One of four classes Equilateral Triangle (liksidig) Isosceles Triangle (likbent) Triangle Error

10 10

10

4

You should

Write the program on a paper Use pseudo code

Similar to some well-known sequential programming language…

Challenge a friend Exchange programs Try to come up with a set of test cases

that makes your friends program fail

5

Recap : Purpose of testing

Find faults! Provoke failures

Deviations from Specification or expected behavior

Gain confidence in product quality Remember : Testing is NOT proving

6

Testing methodology STEP 1 : Test Case Generation

Select test cases according to strategy

Test case is input and expected outcome

Input is e.g., input values, abstract parameters, sequences of button pushes, flag values, DBS contents, etc…

Outcome includes output (external behavior) but often also internal behavior (e.g., traced states)

Expected outcome is always taken from “specification”

STEP 2: Test Case Execution

Run the program with selected test-cases (i.e., the test suite)

STEP 3: Analysis

Compare expected outcome with actual outcome

Decide whether actual outcome is correct or not

7

Test criteria

We cannot test everything ! Unfortunately…

Test requirements Specific demands that must be satisfied or

investigated during testing Execute a specific source code statement Check if a particular error code can be produced

Test criterion Specifies class of test requirements

Execute all source code statements Check if all error codes can be produced

8

Test coverage

Expressed in context of test criterion

Example

Total #test requirementsCoverage =

Fulfilled #test requirements

#Executed statements = 922Total #statements = 1000Statement Coverage = = 92,2%

9

Testing strategies

Structural strategies (white-box) Implementation based

Statement coverage Decision coverage MC/DC

Functional strategies (black-box) Specification based

Equivalence partitioning Boundary-value analysis Combination strategies

10

Statement coverage

Each statement should be executed by at least one test case

Minimum requirement for code coverage

int silly_func(int a, int b){ int i=4; if (a<b) { a=a*14; } if (a<=b){ b=42; } else { i=a-b; } return i*b; }

silly_func(1,2)

silly_func(1,17)

11

Decision coverage

Also called ‘branch coverage’

All decisions with true and false value

“Subsume” all statements coverage

int silly_func(int a, int b){ int i = 4; if (a<b) { a=a*14; } if (a<=b){ b=42; } else { i=a-b; } return i*b; }

silly_func(1,2)silly_func(1,17)

true

false

true

true

silly_func(5,1)

false

false

12

Modified Condition/Decision Coverage (MC/DC)

For each variable x in a logical expression B, let x decide the value of B try x=false try x=true

Example:

if(x1 and x2) {S

}

Used for safety critical applications

x1 x2 x1 and x2

true true true

false true false

true true true

true false false

B

(x1 and true)

x1 decides B :

(true and x2)

x2 decides B :

13

Using white-box test strategies1 Test Case Generation (e.g. statement coverage)

1 Choose initial input data (e.g., random)

2 Execute and check coverage

3 Select a requirement (e.g. a statement) you have not yet covered

4 Select input data that will fulfill the uncovered requirement (e.g. execute that statement)

5 Iterate from 2 until desired code coverage level is reached

6 Minimize test suite (if possible)

7 For each test case, derive expected outcome from specification

2 Test Case Execution and analysis (see slide 6)

14

Using black-box strategies

Input parameter modeling Identify input parameters

Input values, sequences of button pushes, mouse movements, sensor signals, etc

System state, flags, DBS content, cache content, etc Abstract parameters based on functionality and behavior

Equivalence Partitioning Divide each parameter into classes

Boundary Value Analysis Define parameter values for closer analysis

Generic representation of test problems Combine classes with a combination Strategy

All Combinations Each Choice Pair-Wise Base Choice

ClassParam

1 2 3

A

B

C

15

Equivalence partitioningFor each parameter Identify classes of values based on

specification Assumption: if one input from class s leads

to a failure, then any input from class s will lead to the same failure

Chose a representative value from each class (e.g., by random)

Form test cases with the representative values according to a criterion (e.g., pair-wise)

16

Example : Partitioning a parameterA command-line program : SquareInput : A numberOutput : The square roots of that number

10^6 to 10^999 1 to 10^60

Class 1Negative

Class 2Zero

Class 3Small int

Class 4Large int

-10^999 to -1

17

Example: Partitioning with several parameters

Type Integer Real String

Size Small Large Zero

Negative Yes No

Square Yes No

18

Example : Representative valuesA command-line program : SquareInput : A numberOutput : The square roots of that number

17676500 001 420

Class 1NegativeIntegers

Class 2Zero

Class 3Small integers

Class 4Large integers

-5003

Test-CasesSquare -5003Square 0Square 42Square 17676500001…

19

Boundary value analysis

Identify boundaries in input and output

For each boundary value : Select one value from each side of

boundary (as close as possible)

Form test cases with the chosen values

20

Example : Boundary value analysis

10^999 0-10^999

Class 1NegativeReal

Class 2Zero

Class 3Small real

Class 4Large real

Test-CasesSquare -10^999Square -0,0000001Square 0Square 0,0000001…

10^6

21

Combination strategies

Equivalence partitioning and boundary analysis give representative values for input parameters

Test case often contain more than one input parameter

Combination strategies help to combine parameter values!

22

Example : A direct combination of input parameters

Triclass( , , )

100 001 420

-1257.0089 1876.897897896786875812343.089876

Class 1 Class 2 Class 3 Class 4

Class 5 Class 6 Class 7

-1532

23

Example : A direct combination of input parameters

Triclass( , , )

100 001 420

-1257.0089 1876.897897896786875812343.089876

Class 1 Class 2 Class 3 Class 4

Class 5 Class 6 Class 7

-1532

24

Example : A direct combination of input parameters

Triclass( -1532 , , )

100 001 420

-1257.0089 1876.897897896786875812343.089876

Class 1 Class 2 Class 3 Class 4

Class 5 Class 6 Class 7

-1532

Class 1

25

Example : A direct combination of input parameters

Triclass( -1532 ,43.089876, )

100 001 420

-1257.0089 1876.897897896786875812343.089876

Class 1 Class 2 Class 3 Class 4

Class 5 Class 6 Class 7

-1532

Class 1 Class 6

26

Example : A direct combination of input parameters

Triclass( -1532 ,43.089876, 100 001)

Class 1 Class 4Class 6

Sometimes we write this as

Denoting the equivalence classes rather than the particular values

Test case

A B C

TC1 1 6 4

27

Generic test case representation

Test case

A B

TC1 1 1

TC2 1 2

TC3 1 3

TC4 1 4

This allows us to define tests sets that aregeneral for all testing problems that have the same number of parameters and values

For example, two parameters with three and four classes always give following twelve test cases,assuming all combinations

Test case

A B

TC5 2 1

TC6 2 2

TC7 2 3

TC8 2 4

Test case

A B

TC9 3 1

TC10 3 2

TC11 3 3

TC12 3 4

28

Decoding generic test cases

Print_strX Input : String and Integer Output : A string containing each letter in the

original string X times

Class # Parameter class Representative value

1 Empty string “”

2 String with white-space “w h i t e space”

3 String without white-space “yeaha”

Test case

A B

TC1 1 3print_strX(“”,)

29

Decoding generic test cases

Print_strX Input : String and Integer Output : A string containing each letter in the

original string X times

1230002994210

Class 1NegativeIntegers

Class 2Zero

Class 3Small integers

Class 4Large integers

-64566

Test case

A B

TC1 1 3print_strX(“”, 421 ) print_strX(“”,

30

Testing Criteria for Combination Strategies

Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values

For example : my_func(bool A, int B, string C);

Parameter Value 1 Value 2 Value 3

A true false

B Positive Integers Negative Integers

C Empty string String without white-space

String with white-space

31

Testing Criteria for Combination Strategies

Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values

For example : my_func(bool A, int B, string C);

Parameter Value 1 Value 2 Value 3

A true false

B Positive Integers Negative Integers

C Empty string String without white-space

String with white-space

TC : 2, 2, 1 my_func(false,-21,””)

32

Combination strategy : All Combinations

Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values Gives

2*2*3 = 12 TC Grows quickly

E.g 3*3*4 = 36

A B C

TC1 1 1 1

TC2 1 1 2

TC3 1 1 2

TC4 1 2 1

TC5 1 2 2

TC6 1 2 3

TC7 2 1 1

TC8 2 1 2

TC9 2 1 3

TC10 2 2 1

TC11 2 2 2

TC12 2 2 3

Problem : Very expensive !

33

Combination strategy : Each choice

Each parameter value (class) should occur in at least one test case

Test case

A B C

TC1 1 1 1

TC2 2 2 2

TC3 any any 3

my_func(true,341,””);

my_func(false,-4622,”hej”);

my_func(false,4622,”hej Monika”);

Problem : Few test cases / combinations poor testing…

34

Combination strategy: Pair-wise

Each pair of chosen values occurs in at least one test case

Efficiently generated by ‘Latin squares’

Can detect failures due to specific pairs of input values

1 2 3

3 1 2

2 3 1

Latin square

ARow

BCol

CValue

TC1 1 1 1

TC2 1 2 2

TC3 1 (3) 3

TC4 2 1 3

TC5 2 2 1

TC6 2 (3) 2

TC7 (3) 1 2

TC8 (3) 2 3

TC9 (3) (3) 1

Problem : What if we have more than 3 parameters ?

Row=2

Column=1

Value=3

35

A mathematical property that is exploited is “orthogonality”. Two Latin Squares are orthogonal if the contents in each intersection is unique. Orthogonal Latin Squares preserve the pair-wise coverage criterion.

1 2 3

1

12

2

3

3

1 2 3

3

23

1

1

2

11 22 33

13

1223

21

31

32

Combination strategy: Pair-wise

36

Combination strategy: Pair-wise

Each pair of chosen values occurs in at least one test case

Efficiently generated by latin squares

Can detect failures due to pairs of input values

1 1 2 2 3 3

3 2 1 3 2 1

2 3 3 1 1 2

2 Orthogonal Latin squares

ARow

BCol

CValue 1

DValue 2

TC1 1 1 1 1

TC2 1 2 2 2

TC3 1 3 3 3

TC4 2 1 3 2

TC5 2 2 1 3

TC6 2 3 2 1

TC7 3 1 2 3

TC8 3 2 3 1

TC9 3 3 1 2

37

Combination strategy : Base Choice

For each parameter, define a base choice, i.e., the most likely value

Let this be base test case and vary one value at a time until all other classes has been chosen

In the example, let (1, 2, 2) be base choices

A B C

TC1 1 2 2

TC2 2 2 2

TC3 1 1 2

TC4 1 2 1

TC5 1 2 3

38

Equivalence partitioning and combination strategy problems

If equivalence partitioning are based on independent input parameters A lot of functionality may never be tested

For example : What is probability that the triclass function

finds an error in classifying a equilateral triangle ?

Success of combination strategies depend a lot on how equivalence partitioning is done Parameters must sometimes be more abstract Combine properties on combined inputs instead of

independent classes of input values !!

39

Example : Combination testing of Triclass

“Parameter” “values”

a+b>c True False

# sides of equal length 3 2 0

Value Ranges 0< >0 &&<=65536

>65536

Number of decimals 0 >0 &&< 5

>5

a b

c

40

Example : Combination testing of Triclass

“Parameter” “values”

a+b>c True False

# sides of equal length 3 2 0

Value Ranges 0< >0 &&<=65536

>65536

Number of decimals 0 >0 &&< 5

>5

a b

c

TCn = (1,2,2,1) e.g. Triclass(5,5,9)

41

Example : Combination testing of quickSort

“Parameter” “values”

File exists True False

Size of file 0kb 1Mb 2gb

Type of rows Numbers Strings Numbers +Strings

Number of similar signs from beginning

0 5 all

TCn = (1,2,2,1) e.g. quickSort testfile1221.txt

A command-line program : quickSortInput : A text file with rows ending with ;Output : A file with sorted rows

42

Using black-box test strategies

1. Test Case Generation1. Do equivalence partitioning or use some other

method to find good parameters values

2. Select a combination strategyE.g. Pair-wise or base-choice

3. Generate test cases using the selected methodIn the generalized format

4. Select input data within the specified classes for each parameter value

5. For each test-case, derive expected output from specification

2. Test Case Execution (see slide 6)