January 27, 2002 ECEN5033 University of Colorado -- Class Testing

1

Specifying interactions Remainder of slides assume

Operations defined by a class are specified by preconditions, post conditions, and class invariants

contract design approach

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

2

Contract vs. defensive approach & testing Contract More responsibility on

human designer Less class-level testing

due to fewer paths due to less error-checking code

MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?)

“illegal” for test cases to violate preconditions

Defensive More responsibility on

error-checking code More class-level testing

due to increased paths due to more code

Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met

Appropriate to violate preconditions to see if receiver catches that

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

3

Sampling Exhaustive testing: every possible test case

covering every combination of values Not reasonable much of the time

Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is

as good as we can do A sample is a subset of a population (e.g. all

possible test cases) that has been selected based on some probability distribution

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

4

Use profile (operational profile) Gave us a way to associate each use of the

system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the

probability of selection

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

5

Operational profile – stratified sample Stratified sample is a set of samples in which each

sample represents a specific subpopulation Example

Select test cases that exercise each component of the architecture

Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component.

Sample on each subset independent of the others Need a basis for stratifying

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

6

A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor

in the use cases Each actor uses some subset of the possible uses with

some frequency; rank by frequency Stratifying the test case samples by each actor provides a

way to increase the reliability of system Running selected tests

uses the system the way it will be used in typical situations;

finds defects likely to be found in typical use gives largest possible increase in reliability with least

effort

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

7

Test data generation for a range of values Generate test data first using the specification as

basis Select values based on boundary values

just within and just on the boundaries if contract just outside the boundaries also if defensive

Select values within intervals by sampling

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

8

Test data generation for a range of values, cont. Consider a random function over range 0.. N

such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over

many iterations Need to log actual values used in case of failure

so that the failed test case can be re-created. A randomly chosen value causing failure is

explicitly added to test suite and used to test the repaired software

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

9

Systematic sampling (+ boundary value testing)

Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling

over ranges We never omit boundary value test data

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

10

Issues about sampling for interaction testing

Sampling issues Different states can cause two objects from the

same class to behave differently A class family is a subset of a larger family

Possibility of combinatorial explosion in the number of test configurations

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

11

Impact of inheritance hierarchy on s.t.d.’s Each member of the family may have different states

that can cause two objects from the same class to behave differently.

In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there

will be the same number of states or more states in the derived class as in the base class

should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

12

Class families as subsets of others

If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

13

Substitution Principlea reminder

Only the following changes are allowed in defining the behavior associated with a new subclass: Preconditions for each operation must be the

same or weaker – less constraining – than those of the superclass

Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass

Class invariant – must be the same or stronger; add more constraints

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

14

Orthogonal array testing applied to OO Orthogonal array testing is a sampling

technique to limit the explosion of test cases Define pair-wise combinations of a set of

interacting objects because most faults result from interactions due to two-way interactions

An orthogonal array is an array of values in which each column represents a factor.

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

15

Why PAIR-wise Goal: choose a fairly small subset of test cases

that finds a great many defects. (duh!) How? Test all pairs of whatever it is that is

varying. Pairs are fewer: Suppose 4 input parameters,

each of which could take on 3 values. Number of combinations is 3^4 = 81. Can cover all pairwise input combinations in 9 tests

13 input params, each of which as 3 values; number of comb.’s is 3^13. Can cover all pairwise input combinations in 15 tests.

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

16

More on pairwise AT&T wanted to test a LAN-based electronic mail

system. Original plan was to develop and execute 1500 test cases. Switched to pairwise testing – 50 percent less effort and found 28% more defects.

NIST reviewed 15 years of defect data on recalled medical devices. They concluded: 98% of the reported software flaws would have been detected by testing all pairs of parameter settings.

Why does it work so well?

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

17

Orthogonal array testing applied to OO An orthogonal array is an array of values in

which each column represents a factor. A factor is

a variable in an experiment; in our case, a specific class family in the software system

Or states of that class family Each variable takes on a certain set of values

called levels (rows); in our case, specific classes or states of those classes

Each cell is an instance of the class, that is, a specific object or a particular state

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

18

What are orthogonalarrays

Consider numbers 1 and 2 – how many pair combinations are there?

An orthogonal array is a 2-dimensional array of numbers that has this property: Choose any two columns of the array; all the pairwise combinations of its values will occur in every pair of columns.

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

19

Example

Subscript is number of rows in the array. Superscript is number of columns. 2 says each value is a 1 or a 2.

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

20

another

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

21

Mixed arrays

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

22

How to use Identify the variables Determine the number of choices for each variable Locate an orthogonal array with a column for each

variable and values within the columns corresponding to the number of choices.

Map the test problem onto the array Construct test cases with these combinations

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

23

Sample – see excerpt

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

24

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

25

When mapped to all of array Each row becomes a test case Decode the level numbers for a row in the array

back to the individual lists for each factor Sometimes, you don’t need all of the columns

We can ignore them because we don’t need them.

Or we may decide we can include one more class family and its states in the future

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

26

What about mismatches? When there is a difference in the number of levels,

we can use a column that matches or exceeds the maximum

What about a class with 2 states but there can be a 3 in the array we picked Interpret the 3 as if it were either 1 or 2. What

do you want to emphasize in the testing? Arbitrary assignment High frequency High risk

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

27

Other what-ifs? See the Copeland excerpt on Class Lecture

materials page of web site

January 27, 2002 ECEN5033 University of Colorado -- Class Testing

28

OATS adequacy criteria Ability to vary how completely the software under

test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base

classes from each hierarchy are tested Random – tester haphazardly selects cases from

several of the classes; not statistically random Representative – uniform sample ensures every

class is tested to some level Weighted representative – adds cases to the

representative approach based on relative importance or risk associated with the class