T.Y. Chen Swinburne University of Technology, Australia

transcript

T.Y. Chen

Swinburne University of Technology, Australia

T.H. Tse and Zhiquan Zhou

The University of Hong Kong

Semi-Proving: an Integrated Method Based on Global Symbolic Evaluation and Metamorphic Testing

(speaker)

Presentation Outline

Conventional Program Testing and Proving Metamorphic Testing Our method: Semi-Proving Summary.

Conventional Program Testing and Proving

Given a bijective function f ;

A Program: F_Sort (a1, a2, ..., an), n 2

Output: (a1’, a2’, ..., an’), such that

1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)

2. f (a1’) f (a2’) ... f (an’).

Given a bijective function f ;

A Program: F_Sort (a1, a2, ..., an), n 2

Output: (a1’, a2’, ..., an’), such that

1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)

2. f (a1’) f (a2’) ... f (an’).

Testing

1. Design test cases: e.g. (2, 6, 3) for n=3

2. Run: F_Sort (2, 6, 3) = (6, 3, 2)

3. Check: f (6) < f (3) < f (2) ?

1. Design test cases: e.g. (2, 6, 3) for n=3

2. Run: F_Sort (2, 6, 3) = (6, 3, 2)

3. Check: f (6) < f (3) < f (2) ?

Proving correctness

1. F_Sort terminates for any valid input;

2. The output is correct.

1. F_Sort terminates for any valid input;

2. The output is correct.

Proving properties

F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’) F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’)

Permutation.

Metamorphic Testing

Employing relationships between different executions

Fact: different permutations will produce same output

F_Sort (a1, a2, a3)

Fact: different permutations will produce same output

F_Sort (a1, a2, a3) F_Sort (a3, a1, a2) = “ Metamorphic Relation ” ·

Metamorphic Testing

Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}

Metamorphic Testing:

1. F_Sort (2, 6, 3) = (6, 3, 2)

Metamorphic Testing:

1. F_Sort (2, 6, 3) = (6, 3, 2)

No matter whether an oracle is available or not;Very useful when the oracle cannot be found.

2. F_Sort (3, 2, 6) = (6, 3, 2)| |

Metamorphic Testing

Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}

Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2)

2. F_Sort (3, 2, 6) = (3, 6, 2) Failure.| |

Conventional Program Testing and Proving Metamorphic Testing Semi-Proving: Verifying Metamorphic

Relations Summary.

Semi-Proving: Verifying Metamorphic Relations

Objective:

If the program does not satisfy a metamorphic relation on some inputs, locate these inputs;

Otherwise prove the satisfaction of the metamorphic relation over all inputs.

Why called “Semi”?

Proving necessary properties, which may not be sufficient for program correctness

Characteristics of Semi-Proving

Multiple symbolic executions

Testing and proving.

double GetMid (double x1, double x2, double x3){ double mid;

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

mid = x2;else if (x1 > x3)

mid = x1; return mid;

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

Specification

“GetMid (X, Y, Z)” returns the median of (X, Y, Z)

E.g. GetMid (3, 4, 1): “3”.

Verifying “GetMid” by Semi-Proving

Identify a Metamorphic Relation

GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )

any numbers any permutation

Purpose: to verify

Basic concepts

Transposition

• simple permutation that exchanges two elements

(1, 2, 3)

......... 1

(1, 2, 3) (1, 3, 2) ......... 2

(2, 1, 3)

A tuple (1, 2, 3)

A permutation (2, 3, 1)

(1, 2, 3)

A tuple (1, 2, 3)

A permutation (2, 3, 1)

(1, 2, 3) (2, 3, 1)1 (2, 1, 3) 2

Basic concepts

Composition of Transpositions

Result from Group Theory

Any permutation of (X, Y, Z) can be achieved by compositions of transpositions (X, Z, Y) and (

Y, X, Z).

Purpose

Only need to verify:

Any permutation.

• GetMid (X, Y, Z) = GetMid (X, Z, Y)

• GetMid (X, Y, Z) = GetMid (Y, X, Z)

Purpose

Only need to verify:

• GetMid (X, Y, Z) = GetMid (X, Z, Y)

• GetMid (X, Y, Z) = GetMid (Y, X, Z)

Global Symbolic Evaluation on GetMid (X, Y, Z)

Execute all the possible paths.

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

C1: (Y X < Z) OR (Z < X Y)

Path Conditions C2: (X < Y < Z) OR (Z Y < X)

C3: (Y < Z X) OR (X Z Y)

X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true

Z when C3 is true

?GetMid (X, Z, Y)

?X when C1 is true

GetMid (X, Y, Z) = Y when C2 is trueZ when C3 is true

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

?GetMid (X, Z, Y)

?X when C4 is true

= Z when C5 is true

Y when C6 is true

Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

Contradiction

C1: (Y X < Z) OR (Z < X Y) &

GetMid (X, Z, Y)?

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

C1: (Y <= X < Z) OR (Z < X <= Y) &

X=Y<Z OR Z<Y=X

?GetMid (X, Z, Y)

C4: (Z X < Y) OR (Y < X Z)

C5: (X < Z < Y) OR (Y Z < X)

C6: (Z < Y X) OR (X Y Z)

Z when C3 is true

X when C4 is true

= Z when C5 is true

Y when C6 is true

C1: (Y <= X < Z) OR (Z < X <= Y) &

Yes. X=Y

X=Y<Z OR Z<Y=X

GetMid (X, Z, Y)

Z when C3 is true

GetMid (X, Z, Y)

verified

Z when C3 is true

Conclusion

GetMid (X, Z, Y)

Z when C3 is true

Conclusion

GetMid (X, Z, Y)

Z when C3 is true

Conclusion

GetMid (X, Z, Y)

Z when C3 is true

Conclusion

GetMid (X, Z, Y)

Composition of transpositions

GetMid (X, Y, Z) = GetMid ( Permute(X, Y, Z) )

GetMid (Y, X, Z)

Any Any.

Detecting Program Faults ·

Semi-Proving: Detecting Program Faults

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

mid = x3;if (x2 < x3)

if (x1 < x2)mid = x2;

else {if (x1 < x3)

mid = x1;}

elseif (x1 > x2)

Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)

when Y X < Z

when (Z < Y X ) OR (Y Z AND X Z) AND

when Y X < Z

(Y=X<Z) OR (Y<X<Z)

when Y X < Z

(Y=X<Z) OR (Y<X<Z)

failure

Failure-causing inputCan identify all the

failure-causing inputs.

Summary A proving technique: all the paths A testing technique:

failure-causing inputs selected path(s)

Characteristics Metamorphic relations Multiple symbolic executions Employing global symbolic evaluation and constraint

solving.

Questions are welcome

T.Y. Chen Swinburne University of Technology, Australia

Documents