© 2012 IBM Corporation

IBM Haifa Research Laboratory

Combinatorial Test Design (CTD)

© 2012 IBM Corporation��

How can we thoroughly test complex systems and avoid errors of omission?

� Testing is challenging at every level

– Exhaustive testing of complex systems is impossible: many billions of potential tests

– Thorough testing of individual features is difficult: for example, even search functions have multiple options that may have complex interactions

� When people create tests by hand, they tend to fall into a pattern that produces similar tests, most of which add little value in discovering defects. Manually-built test plans tend to have many gaps.

� Combinatorial Test Design (CTD) is a smarter approach that helps us identify the tests that are likely to expose defects. CTD takes a systematic approach to modeling the things that need to be tested, then uses advanced mathematics to dramatically reduce the number of test cases while ensuring coverage of conditions and interactions.

�

End-to-end and system-level testing

Feature-level testing

© 2012 IBM Corporation���

� Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html

The root cause analysis of many bugs shows they depend on a value of one variable (20%-68%)

Most defects can be discovered in tests of the interactions between the values of two variables (65-97%)

Do we really need to test all combinations?

© 2012 IBM Corporation�

Combinatorial Test Design (CTD)

� To balance cost and risk, we select a subset of tests that covers all the interactions of variables at some level of interaction (pairs, three-way, etc.)

� A combinatorial test design (CTD) algorithm finds a small test plan that covers 100% of a given interaction level

© 2012 IBM Corporation�

Combinatorial Test Design is a technique that takes all possible points of variation and generates a small subset of tests based on levels of interaction.

Model the:

Input space

Environment

Scenarios

Generate test set

Generate test configurations

Generate

Combinatorial

Design

Attributes and Values

Test Descriptions

Test set

Test configurations

© 2012 IBM Corporation�

Combinatorial test design is an approach that can be applied to different phases and scopes of testing

End-to-end and system-level testing

Feature-, service- and API-level testing

CTD models at this level typically include

Business transactionsBusiness data and stateSystem implementation

Testing scenarios

CTD models at this level typically

includeInput selections

Input dataNavigation

Error conditions

© 2012 IBM Corporation�

Existing Tests

IBM FoCuS/ CTD solution

The CTD model can be used to optimize tests in multiple ways, taking existing tests and data into account.

CTD model

Find gaps, generate tests to

close

Generate complete set of

tests

Remove overlapping tests Test selection

Test enhancement

Test generation

Test data Select data subset

Distilled test data

Existing Tests

Subset of existing Tests

New Tests to close gaps

New Tests for complete coverage

Test data selection