Post on 26-Jan-2017
transcript
Graph Methods for GeneratingTest Cases with Universal and
Existential Constraints
Sylvain Hallé, Edmond La Chanceand Sébastien Gaboury
Université du Québec à Chicoutimi, Canada
http://www.liflab.ca
November 23rd, 2015
Example:
How to test the system in such a way that we cover all possible interactions between two inputs?
can each be in position 1, 2 or 3, ,
Naïve method: enumerate all combinations of values
All combinations of values for any two inputs are present
How many test cases are there?
1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3
1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3
1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3
In fact, this table verifies all combinations of 3 inputs exactly once... but combinations of 2 appear more than once
Can we observe them all by doing fewer tests?
1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3
1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3
1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3
Here's how:
Each combination of values for 2 given parameters appears the same number of times (here, once)
We call this an orthogonal array (OA)
3 2 1 2 1 3 1 3 2
1 2 3 1 2 3 1 2 3
1 1 1 2 2 2 3 3 3
How many testcases this time?
Pairwise testingTesting strategy that identifies interactions between two input parameters, and tests all combinations of values for the selected pairs
Intuition: some problems occur only through an interaction between many (e.g. 2) parameters
By varying each parameter individually, it isunlikely to generate the pair causing the error
We can use orthogonal arrays to get the test casesto try
Exemple of an interaction:
The error occurs only when altitude_adj == 0 AND volume < 2.2
if (altitude_adj == 0) { // do something if (volume < 2.2) { faulty code! BOOM! } else { good code, no problem }} else { // do something else}
10 effects, each with 2 possible values (on/off)
All combinations:210 = 1024 tests
How many tests to coverall interactions of 3effects?
There are = 120 triplets of effects
For each triplet, we have to test 23 = 8 combinations of values. We need a maximum of 960 tests.
But each test exercises three triplets; hence we need no more than 320 test to cover everything...
0 1 1 0 1 0 1 0 1 0
...and in fact, each test covers much more than three triplets
103( )
Quel est le nombre minimum detests?
Actually, only 13 tests are necessary!
Is it an orthogonalarray?
0 0 0 0 0 0 0 0 0 01 1 1 1 1 1 1 1 1 11 1 1 0 1 0 0 0 0 11 0 1 1 0 1 0 1 0 01 0 0 0 1 1 1 0 0 00 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 1 1 1 01 1 0 1 0 0 1 0 1 00 0 0 1 1 1 0 0 1 10 0 1 1 0 0 1 0 0 10 1 0 1 1 0 0 1 0 01 0 0 0 0 0 0 1 1 10 1 0 0 0 1 1 1 0 1
Covering arrayAn array CA(N, k, v, t) such that:
The previous array is CA(13, 10, 2, 3)
The array contains N lines
We consider k parameters
Each parameter can have v values
Each combination of values for t parameters occurs at least once in the array
t-way test case generation
Back to our example:
But...
can each be in position 1, 2 or 3, ,
must always be greater than
and can never be at position
1 at the same time
Back to our example:
But...
can each be in position 1, 2 or 3, ,
must always be greater than
and can never be at position
1 at the same time
Universal constraints
Universal constraintConstraint on the possible values for a test case
Every test case must follow the constraint to be considered valid
We wish to find the smallest covering array CA(N, k, v, t), for a given set of constraints
May lead to a smaller or a larger covering array, depending on the constraints
Example: mutually exclusive parameters (subscript/superscript)
Back to our example:
Also...
The test suite shouldinclude a case where allinputs are different
At least one test should try and at the same position
can each be in position 1, 2 or 3, ,
Back to our example:
Also...
The test suite shouldinclude a case where allinputs are different
At least one test should try and at the same position
can each be in position 1, 2 or 3, ,
Existential constraints
Existential constraintConstraint on the possible values for a test case
At least one test case must follow the constraint for the test suite to be considered valid
We wish to find the smallest covering array CA(N, k, v, t), for a given set of constraints
May lead to a smaller or a larger covering array, depending on the constraints
Same as before
Why use existential constraints?
Three scenarios (at least):
See the paperb
Extending an existing test suite (add onecondition for each existing test)Relaxed t-way (Mone combination sufficesM)MC/DC testing
Let D1, D2, ..., Dk be the domains for k parameters named p1, p2, ..., pk
Let Φ = {φ1, φ2, ..., φm} be a set of m arbitrary Boolean expressions, whose ground terms are of the form pi = d, or pi = pj, for d ∈ Di
Let Σ ⊆ D1 × D2 × ... × Dk be a set of value assignments for each parameter
We call Σ a Φ-way covering if, for every φ ∈ Φ, there exists an assignment σ ∈ Σ such that σ |= φ
Φ is the set of existential constraintsΣ is the set of test cases
Back to our example:
Also...
The test suite shouldinclude a case where allinputs are different
At least one test should try and at the same position
can each be in position 1, 2 or 3, ,
Back to our example:
Also...
The test suite shouldinclude a case where allinputs are different
At least one test should try and at the same position
D1 = D2 = D3 = {1,2,3}
Back to our example:
Also...
The test suite shouldinclude a case where allinputs are different
p2 = p3
D1 = D2 = D3 = {1,2,3}
Any t-way problem can be converted into a Φ-way problem...
Let I = {S ⊆ [1,k] : |S| = t}
Let DS = Di
Define Φ as:∩
S ∈ I {d1,...,dn} ∈ DS
∏i ∈ S
∩
pi = di∧i ∈ S
How do we find a Φ-way covering?
Two solutions based on reductions to graph problems
Graph colouring
Hypergraph vertex covering
Reduction to graph colouring
Create the graph G = <V,E> such that:
Let κ : V → [1,n] be a colouring of G with n colours. Define σκ,i as any variable assignment satisfying:
Then Σ = σκ,i is a Φ-way covering (theorem)
V = ΦE = { (φ,φ') : φ ∧ φ' is unsatisfiable }
φ∧φ ∈ κ-1(i)∩
i ∈ [1,n]
Informally...
Create one vertex per constraint
Link pairs of vertices that cannot be true at the same time
Find a colouring for this graph
Create a test case for each colour (take the conjunction of all conditions for nodes ot this colour)
Example with
a = 0 ∧ b = 0, a = 0 ∧ b = 1, a = 1 ∧ b = 0,a = 1 ∧ b = 1, a = 0 ∧ c = 0, a = 0 ∧ c = 1,a = 1 ∧ c = 0, a = 1 ∧ c = 1, b = 0 ∧ c = 0,b = 0 ∧ c = 1, b = 1 ∧ c = 0, b = 1 ∧ c = 1 }
Φ = {
a=0 ∧ b=0
a=1 ∧ b=0
a=1 ∧ b=1a=1 ∧ c=0
a=1 ∧ c=1
a=0 ∧ b=1
b=1 ∧ c=0
b=1 ∧ c=1
b=0 ∧ c=0
b=0 ∧ c=1a=0 ∧ c=0
a=0 ∧ c=1
Pairwise testing for a, b, c ∈ {0,1}
a=1,b=0,c=1
a=1,b=0,c=0
a=1,b=1,c=0
a=1,b=1,c=1
a=0,b=1,c=1
Testcases
Some remarks:
The conjunction of all conditions of the same colour must be satisfiable. Counter-example:
Classical t-way problems (and many others) always satisfy this constraint
The minimum number of tests is the chromatic number of G
⇒ "Guaranteed" lower bound
a=0 ∨ b=0 a=1 b=1
1.
2.
3.
Reduction to hypergraph vertex covering
Create the hypergraph G = <V,E> such that:
Let Σ ⊆ V be a vertex covering of G. Then Σ is a Φ-way covering (theorem)
V = D1 × D2 × ... × Dk
E = {S ⊆ V : there exists φ ∈ Φ such that s ∈ S iff s |= φ}
Works all the time!
Can even deal with universalconstraints (remove verticesthat violate any of them)
Example with
a = 0, b = 0, a ≠ 0 ∨ b ≠ 0 }Φ = {
a=0,b=0
Testcases
a=0, b=0
a=0, b=1
1
a=1, b=0
2
3
a=1, b=1
3
3
a=1,b=1
a : 0, 1, 2b : 0, 1c : 2, 3, 4, 5
Once a != bOnce a < b
Always !(a > b) || c == 2
Both techniques implemented in a test case generation tool
Only known tool to handle existential AND universal constraints
Uses a simple extension of the input format used by PICT
Domains
Existential constraints
Universalconstraints
5
10
15
20
25
30
35
40
45
50
55
2 3 4 5 6 7 8 9
Test
siz
e
n
Test size with t:2, D:3
QICTColouring
AllPairsTCases
Jennyhypergraph
For pairwise test case generation...
With t = 3...
20
30
40
50
60
70
80
90
100
110
120
3 4 5 6 7 8
Test
siz
e
n
Test size with t:3, D:3
ColouringTCases
JennyHitting
Fewer toolsremain...
With universal constraints...
With existential constraints...
Still fewertools...
Hypergraph6060118
TCases158203254
n567
Hypergraph6566110
n567
Last onestanding!
Existential constraints in test case generation arise in a variety of situations, but are not addressed by existing tools/research
Finding an optimal test suite can be reduced to finding the optimal solution of two well-known graph problems
Empirically, existing heuristics for these problems perform at least as well as avariety of existing tools
https://bitbucket.org/ sylvainhalle/gcases
Take-home points...