ICT
Testing Product Lines of Industrial Size:Advancements in Combinatorial Interaction Testing
Martin Fagereng Johansen
PhD Thesis Defense, 2013-11-05
ICT
INDUSTRIAL MOTIVATION
2
ICT
TOMRA's Reverse Vending Machines
Finale's Financial Reporting Systems
ABB's Configurable Safety Module
Eclipse IDEs – Free and Open Source
3
ICT
About the Eclipse IDE
Initiated and funded by IBM Widely used by software engineers to develop software
Major competitor to Microsoft Visual Studio Many third-party extensions
4
ICT 5
ICT
Eclipse IDE – v3.7.0 (Indigo) – An Example of a Software Product Line
6
The problem: Can we gain confidence that any product will work?
ICT
Which products are possible?
7
356,352 possibilities!
→ model its features and their relationships in a→ feature model:
ICT
Today: A Test Suite for Each Feature
8
http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/testResults.php
http://wiki.eclipse.org/Eclipse/Testing
ICT
OVERVIEW
9
ICT 10
ICT
SAMPLING
11
ICT
Faulty Features
12
Unit tests may find faults inside a single feature. n test suites required for a product line with n features.
What about faulty cooperation between features? What if they interact incorrectly?
ICT
Interaction Faults
13
2-wise interaction fault reproducible by including 2 specific features the others do not matter
ICT
Interaction Faults
14
3-wise interaction fault reproducible by including 3 specific features the others do not matter
ICT
Kuhn et al. 2004: Almost all bugs can be attributed to the interaction of a few features.
Empirics Show:
15
ICT
Only a few products are needed to cover all simple interactions. i.e. testing a few well-selected products might reveal almost all bugs
Examples (2-wise testing):
For the "e-shop product line" with 287 features: 21 products For the Linux kernel with almost 7,000 features: 480 products
Covering Arrays
16
ICT 17
?
ICT
Configuring Feature Models
Feature models can be solved by configuration:
…or by satisfying the corresponding Boolean formula: R (A R) (B R) (C A) (D A) (C D) ¬(C D) ∧ ⇒ ∧ ⇒ ∧ ⇒ ∧ ⇒ ∧ ∨ ∧ ∧
(E B) (F B) (E F) (D E) ∧ ⇒ ∧ ⇒ ∧ ∨ ∧ ⇒ e.g. R = 1, A = 1, B = 1, C = 0, D = 1, E = 1, F = 0 The SAT problem.
18
ICT
State of the art argument
SAT is the classic NP-complete problem. worst-case analysis (Cook 1971)
Configuring basic feature models i.e., finding a single product of a product line SPLE-SAT – Software Product Line Engineering Boolean SATisfiability
Includes only feature models that occur in SPLE.
Argument SPLE-SAT = SAT, and SAT is NP-complete i.e., SPLE-SAT is NP-complete i.e., SPLE-SAT is impractical
(unless P=NP, due to Cobham's thesis) i.e. because sampling involves SPLE-SAT, sampling is impractical.
19
ICT
Our Argument
If SPLE-SAT is impractical: Configuring a feature model is impractical. i.e., testing product lines is of no concern.
If we cannot find any products, why care about their quality? However, if we have a product line with products:
Finding them were practical. We care about their quality. i.e., SPLE-SAT is practical.
Also: If a feature model is too hard to configure then it cannot serve its
purpose as an SPLE artifact. A customer cannot use it to customize a product to their needs.
i.e., SPLE-SAT is practical.
20
ICT
Empirical Investigation: SAT time
21
SPLE-SAT is very quick. Even for the largest
models. E.g. The Linux Kernel
Routinely configured by hand.
ICT
Conclusions as Venn Diagrams
State of the Art Conclusion:
Our Conclusion:
22
SAT = SPLE-SAT
Hard SAT
ICT 23
?
ICT
?
24
ICT
Sampling: Impractical in Practice
25
ICT
A New, Efficient Algorithm: ICPL
ICPL
26
2
ICT
What makes ICPL quick?
Based on a greedy polynomial time approximation algorithm (PTAS) for the set covering problem (SCP) Chvátal's algorithm (Chvátal 1979)
We know SPLE-SAT is quick. Strategically run SPLE-SAT often and infer as much as possible.
Utilize modern hardware. large amounts of memory (128 GB) truly parallel processing (64 concurrent executions)
Separate out data-parallel sub-algorithms. ++
27
ICT
Comparison
28
State of the art:
Our new algorithm (ICPL):
ICT
Comparison
Case Features Clauses in Constraint
t SotA (s) ICPL (s) Improvement in speed
Violet 101 90 3 244 32 86.9%
Eshop 287 22 2 366 5 98.6%
Ecos 1,244 2,768 2 ? 185 ?
FreeBSD 1,396 17,352 2 ? 240 ?
Linux 6,888 187,193 2 ? 33,702 ?
29
ICT 30
?
ICT 31
ICT
MARKET-FOCUSED SAMPLING
32
ICT
Industrial Context
TOMRA's Reverse Vending Machines:
33
ICT
Feature Model of TOMRA RVM
34
435,808 possibilities!
ICT
The 12 Products in Their Test-lab
35
ICT
Full Sampling was Too Costly
The problem Too many test-products
Their Need: Optimize the selection of
12 products. Our answer:
1. Model the market situation.
2. Select the most relevant products according to that model.
36
1 2 30
10
20
30
40
50
60
ICT
Our Model of the Market Situation:"Weighted sub-product lines"
37
ICT
Better Coverage with 12 Products
38
1 2 30%
20%
40%
60%
80%
100%
t-setweight
t
coverage
All Inter-actionsInteractions of market
ICT 39
?
?
ICT
Interactions
40
TestCSV succeeds for both.
CSV works with and without Web Tools.
Does CSV work without GEF?
Does CSV work with CDT?
Etc…
With a 3-wise covering array, we get a few products with:
With a 2-wise covering array, we get a few products with:
ICT 41
What Eclipse Tests Today: 2-Wise Covering Array:
ICT 42
Test Results – Pair-wise Testing
ICT
Possible Causes
Two (or more) features … access the same resource have overlapping GUI
elements SWTBot tests
have dependencies that interact wrongly
wait for each other (deadlock)
+++
43
ICT
Potential Faults Found using Existing Test Cases Strategic application of
existing tests revealed potential faults. Relatively inexpensive to
apply. Raises confidence on
success. Such a large scale, fully
reproducible and documented application of a product line testing technique is not found in the existing literature.
44
ICT
Also Applied to the ABB-case
45
Two bugs identified with 5 test cases.
ICT 46
?
?
ICT 47
SPLCATool
SPLCAToolSPLCATool
SPLCATool
ICT
Future Work
Further empirical study of faults in software product lines. Complete application to the Eclipse IDE
With test cases for all features; it is possible today! A good source of further empirics. A good basis of further improvements.
Even quicker algorithms for covering array generation. Less memory usage. Higher degree of parallelism.
Improved test allocation. Based on specification, model or implementation. Based on meta-data such as versions.
48
ICT
Summary
SPLE-SAT was investigated. Realistic feature models are readily configurable. Encourages the investigation into faster algorithms.
A fast algorithm for sampling. Enables the use of sampling for product line testing.
Theory and algorithms for market-focused sampling. One approach for automatic allocation of test cases.
Enables the production of a test report from (1) an implementation, (2) a test case collection and (3) feature model.
An automatic and scalable technique for software product line testing supported by free, open source tooling. SPLCATool
49
ICT 50