Combinatorial Interaction Testing for TestSelection in Grammar-Based Testing
Elke Salecker, Sabine Glesner
Technical University of Berlin, Germany
Workshop on Combinatorial Testing (CT) 2012Montreal, Canada
Introduction Background Approach Evaluation Conclusions
Motivation
Grammar-based Testing
• black-box testing approach• test data derived from context-free grammar
à automatic generation of test data
Problem
• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus
New Approach
à test selection based on combinatorial interaction testing
à automatic generation of CIT test specification
Elke Salecker, Sabine Glesner 1
Introduction Background Approach Evaluation Conclusions
Motivation
Grammar-based Testing
• black-box testing approach• test data derived from context-free grammar
à automatic generation of test data
Problem
• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus
New Approach
à test selection based on combinatorial interaction testing
à automatic generation of CIT test specification
Elke Salecker, Sabine Glesner 1
Introduction Background Approach Evaluation Conclusions
Motivation
Grammar-based Testing
• black-box testing approach• test data derived from context-free grammar
à automatic generation of test data
Problem
• exhaustive testing of test sets infeasible• controlled/random-based enumaration process• rule combination coverage not in focus
New Approach
à test selection based on combinatorial interaction testing
à automatic generation of CIT test specification
Elke Salecker, Sabine Glesner 1
Introduction Background Approach Evaluation Conclusions
Outline
1 Introduction
2 CIT and Grammars
3 Test Set Generation
4 Evaluation
5 Conclusions
Elke Salecker, Sabine Glesner 2
Introduction Background Approach Evaluation Conclusions
Combinatorial Interaction Testing
CIT specification:parameters, values
P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3
test case (a1, b1, c1, d2)
test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)
constraints ¬(b1 ∧ c2)
Elke Salecker, Sabine Glesner 3
Introduction Background Approach Evaluation Conclusions
Combinatorial Interaction Testing
CIT specification:parameters, values
P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3
test case (a1, b1, c1, d2)
test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)
constraints ¬(b1 ∧ c2)
Elke Salecker, Sabine Glesner 3
Introduction Background Approach Evaluation Conclusions
Combinatorial Interaction Testing
CIT specification:parameters, values
P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3
test case (a1, b1, c1, d2)
test set P1 × P2 × P3 × P4, 24 test cases
coverage criterion (t=2)
constraints ¬(b1 ∧ c2)
Elke Salecker, Sabine Glesner 3
Introduction Background Approach Evaluation Conclusions
Combinatorial Interaction Testing
CIT specification:parameters, values
P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3
test case (a1, b1, c1, d2)
test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)
1 a3 b1 c1 d22 a2 b2 c2 d23 a1 b2 c1 d24 a1 b1 c2 d15 a2 b1 c1 d16 a3 b2 c2 d1
constraints ¬(b1 ∧ c2)
Elke Salecker, Sabine Glesner 3
Introduction Background Approach Evaluation Conclusions
Combinatorial Interaction Testing
CIT specification:parameters, values
P1 P2 P3 P4a1 b1 c1 d1a2 b2 c2 d2a3
test case (a1, b1, c1, d2)
test set P1 × P2 × P3 × P4, 24 test casescoverage criterion (t=2)
1 a3 b1 c1 d22 a2 b2 c2 d23 a1 b2 c1 d24 a1 b1 c2 d15 a2 b1 c1 d16 a3 b2 c2 d1
constraints ¬(b1 ∧ c2)
Elke Salecker, Sabine Glesner 3
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}
rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �derivation sequence of rule applications
< add, addimm, r2c >
r Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �
derivation sequence of rule applications< add, addimm, r2c >
r Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �derivation sequence of rule applications
< add, addimm, r2c >
r
Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �derivation sequence of rule applications
< add, addimm, r2c >
r Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �derivation sequence of rule applications
< add, addimm, r2c >
r Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Grammars
nonterminals N = {�, r, imm}terminals Σ = {Const, ObjAddr, Content, Plus, Assign}rules def: �→ Assign((ObjAddr a), r)(lhs → rhs) add: r → Plus(r, r)
addimm: r → Plus(r, imm)use: r → Content(ObjAddr a)r2c: r → Const cr2i: r → immi2c: imm → Const c
start symbol �derivation sequence of rule applications
< add, addimm, r2c >
r Plus
r r
Plus
Plus
r imm
r
Plus
Plus
Const 2 imm
r
Elke Salecker, Sabine Glesner 4
Introduction Background Approach Evaluation Conclusions
Outline
1 Introduction
2 CIT and Grammars
3 Test Set Generation
4 Evaluation
5 Conclusions
Elke Salecker, Sabine Glesner 5
Introduction Background Approach Evaluation Conclusions
Symmetry in Derivations
r2c r → Const ci2c imm → Const cadd r → Plus(r, r)addimm r → Plus(r, imm)
derivation:
add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c
Plus
Plus
Const 2 Const 2
Const 4
derivation:
add ⇒ r2c ⇒ addimm ⇒ r2c ⇒ i2c
Plus
Const 4 Plus
Const 2 Const 2
Elke Salecker, Sabine Glesner 6
Introduction Background Approach Evaluation Conclusions
Symmetry in Derivations
r2c r → Const ci2c imm → Const cadd r → Plus(r, r)addimm r → Plus(r, imm)
derivation:
add ⇒ addimm ⇒ r2c ⇒ i2c ⇒ r2c
Plus
Plus
Const 2 Const 2
Const 4
derivation:
add ⇒ r2c ⇒ addimm ⇒ r2c ⇒ i2c
Plus
Const 4 Plus
Const 2 Const 2
Elke Salecker, Sabine Glesner 6
Introduction Background Approach Evaluation Conclusions
Principle of Specification Generation
Derivations with Fixed Length1 2 3 4 5 6
1 def add add r2c r2c r2c2 def add use add r2c r2c3 def add r2c add r2c r2c. . .6 def add addimm r2c i2c use7 def add addimm use i2c use. . .30 def add use add use use
P1 P2 P3 P4 P5 P6derivation length = number of parametersvalue of parameter i = rule can be applied in step iconstraints for invalid combinations
Elke Salecker, Sabine Glesner 7
Introduction Background Approach Evaluation Conclusions
Principle of Specification Generation
Derivations with Fixed Length1 2 3 4 5 6
1 def add add r2c r2c r2c2 def add use add r2c r2c3 def add r2c add r2c r2c. . .6 def add addimm r2c i2c use7 def add addimm use i2c use. . .30 def add use add use use
P1 P2 P3 P4 P5 P6derivation length = number of parametersvalue of parameter i = rule can be applied in step iconstraints for invalid combinations
Elke Salecker, Sabine Glesner 7
Introduction Background Approach Evaluation Conclusions
CIT Specification Generation
1 Generate compact representation of derivations
2 Calculate value sets
3 Calculate constraints
Elke Salecker, Sabine Glesner 8
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
Recursive Construction:
Base Case derivation length = 1store for terminal ruleslhs nonterminal, rule identifier
Recursion derivation length > 1
check remaining rulessplit overall length for number of nonterminals in rulefind smaller derivationsstore lhs nonterminal, rule identifier, extended pattern
Elke Salecker, Sabine Glesner 9
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
Recursive Construction:
Base Case derivation length = 1store for terminal ruleslhs nonterminal, rule identifier
Recursion derivation length > 1check remaining rulessplit overall length for number of nonterminals in rulefind smaller derivationsstore lhs nonterminal, rule identifier, extended pattern
Elke Salecker, Sabine Glesner 9
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅
2 r r2i < (1, imm) >
� def < (1, reg) >
3 r addimm < (1, reg), (1, imm) >
add < (1, r), (1, r) >
� def < (2,r) >
4 add < (1, r), (2, r) >
< (2, r), (1, r) >
Elke Salecker, Sabine Glesner 10
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, reg) >
3 r addimm < (1, reg), (1, imm) >
add < (1, r), (1, r) >
� def < (2,r) >
4 add < (1, r), (2, r) >
< (2, r), (1, r) >
Elke Salecker, Sabine Glesner 10
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
def:�→Assign((ObjAddr a),r)add:r → Plus(r, r)addimm:r → Plus(r, imm)use:r → Content(ObjAddr a)r2c:r → Const cr2i:r → immi2c:imm → Const c
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, reg) >
3 r addimm < (1, reg), (1, imm) >
add < (1, r), (1, r) >
� def < (2,r) >
4 add < (1, r), (2, r) >
< (2, r), (1, r) >
Elke Salecker, Sabine Glesner 10
Introduction Background Approach Evaluation Conclusions
Generate Value Sets
Recursive Construction:
Base Case derivation length = 1collect rules for length 1
Recursion derivation length > 1iterate over rules with considered nonterminalextend parameter with rulesfor each rule iterate over alternativesdescend for each nonterminal
Elke Salecker, Sabine Glesner 11
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
P1P2P3P4
Elke Salecker, Sabine Glesner 12
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
P1 defP2P3P4
Elke Salecker, Sabine Glesner 12
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
P1 defP2 addimmP3P4
Elke Salecker, Sabine Glesner 12
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
P1 defP2 addP3 use, r2cP4
Elke Salecker, Sabine Glesner 12
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
P1 defP2 addimmP3 use, r2cP4 i2c
Elke Salecker, Sabine Glesner 12
Introduction Background Approach Evaluation Conclusions
Generate Constraints
Recursive Construction:
Base Case derivation length = 1disjunction of all rules for length 1
Recursion derivation length > 1iterate over rules with considered nonterminal
construct disjunctionfor each rule iterate over alternatives
construct disjunctionfor each alternative
construct conjunction
Elke Salecker, Sabine Glesner 13
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
Elke Salecker, Sabine Glesner 14
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
def −→ f
Elke Salecker, Sabine Glesner 14
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
def −→(addimm ∧ f )
Elke Salecker, Sabine Glesner 14
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
def −→(addimm ∧((use ∨ r2c) ∧ f))
Elke Salecker, Sabine Glesner 14
Introduction Background Approach Evaluation Conclusions
Generate Compact Representation of Derivations
L NT Rule RHS Pattern
1 r use ∅r2c ∅
imm i2c ∅2 r r2i < (1, imm) >
� def < (1, r) >
3 r addimm < (1, r), (1, imm) >
add < (1, r), (1, r) >
� def < (2,reg) >
4 r addimm < (2, r), (1, imm) >
add < (1, r), (2, r) >
< (2, r), (1, r) >
� def < (3,r) >
def −→(addimm ∧((use ∨ r2c) ∧ i2c))
Elke Salecker, Sabine Glesner 14
Introduction Background Approach Evaluation Conclusions
Outline
1 Introduction
2 CIT and Grammars
3 Test Set Generation
4 Evaluation
5 Conclusions
Elke Salecker, Sabine Glesner 15
Introduction Background Approach Evaluation Conclusions
Size of Parameters
P1 P2 P3 P4 P5 P6
G1 Spec3 5 17 9
Spec4 10 38 26 13
Spec5 10 43 38 26 13
Spec6 12 55 47 38 26 13
G2 Spec3 5 21 13
Spec4 10 44 30 20
Spec5 10 53 47 32 20
Spec6 12 88 67 54 32 20
G186 rules12 initial rulesG2110 rules12 initial rules
à generation within seconds
Elke Salecker, Sabine Glesner 16
Introduction Background Approach Evaluation Conclusions
Size of Test Sets
G1 G2 G3 G4
L 5 (1) 6 (2) 6 (1) 7 (2)
6 Spec 1 23 42 24 42 1 22 3 52 23 3 52
Derivations 64 128 144 288
TCases(ACTS) 16 25 13 22
8 Spec 1 23 44 24 44 1 22 3 54 23 3 54
Derivations 640 1280 2160 4320
TCases(ACTS) 22 32 17 28
10 Spec 1 23 46 24 46 1 22 3 56 23 3 56
Derivations 7168 14336 36288 -
TCases(ACTS) 32 38 24 30
• significant reduction• generation failed for case study grammars
Elke Salecker, Sabine Glesner 17
Introduction Background Approach Evaluation Conclusions
Size of Test Sets
G1 G2 G3 G4
L 5 (1) 6 (2) 6 (1) 7 (2)
6 Spec 1 23 42 24 42 1 22 3 52 23 3 52
Derivations 64 128 144 288
TCases(ACTS) 16 25 13 22
8 Spec 1 23 44 24 44 1 22 3 54 23 3 54
Derivations 640 1280 2160 4320
TCases(ACTS) 22 32 17 28
10 Spec 1 23 46 24 46 1 22 3 56 23 3 56
Derivations 7168 14336 36288 -
TCases(ACTS) 32 38 24 30
• significant reduction• generation failed for case study grammars
Elke Salecker, Sabine Glesner 17
Introduction Background Approach Evaluation Conclusions
Conclusion and Future Work
Conclusion
• new approach for grammar-based testing• criterion guarantees t-wise rule coverage• test selection based on combinatorial interaction testing• CIT specification automatically generated
• very encouraging results regarding test set size
Future Work
• enhancement of constraint generation• combination with alternative approach for grammar-based
testing
• comprehensive case study for compiler back ends
Elke Salecker, Sabine Glesner 18
Introduction Background Approach Evaluation Conclusions
Conclusion and Future Work
Conclusion
• new approach for grammar-based testing• criterion guarantees t-wise rule coverage• test selection based on combinatorial interaction testing• CIT specification automatically generated
• very encouraging results regarding test set sizeFuture Work
• enhancement of constraint generation• combination with alternative approach for grammar-based
testing
• comprehensive case study for compiler back ends
Elke Salecker, Sabine Glesner 18
IntroductionCIT and GrammarsTest Set GenerationEvaluationConclusions