Minimization of Randomized Unit Test Cases

Yong Lei, James H. AndrewsDept. of Computer Science Univ. Of Western OntarioLondon, Ontario, Canada

ISSRE 2005, 10 Nov 2005

I apologize in advance for not being able to speak more loudly.

3

Contents

• Randomized unit testing• Minimization• Case study• Experiment• Related work

4

Unit Testing

• Testing methods, groups of methods, classes

• Key practice of TDD

5

Conventional Unit Testing

• Write test cases (sequences of method calls)• Execute and verify each test case• Record % successes and failures

6

Randomized Unit Testing

• Write test fragments• Execute many random sequences of

fragments• Keep failures, throw away successes• Potential: lots of automatically generated

test input, “unexpected” sequences

7

Example Unit Under Test: “Course” Class

• Student object contains Courses taken• Course object contains prerequisite Courses• Course class contains addStudent method

– Succeeds (returning true) iff Student has taken all prerequisites

– Otherwise returns false

8

Course: ConventionalUnit Testing

• Initialization: create Course c with prerequisites

• Test cases:– TC1: create Student with prerequisites; add to c– TC2: create Student with only some

prerequisites; try to add to c– TC3: create Student with no prerequisites; try

to add to c

9

Course: Randomized Testing

• Initialization: create Course c, Student s• Test fragments:

– Add/delete random prerequisite from Course c– Add/delete random course taken from Student s– Try to add s to c

• Randomized test case = random sequence of test fragment calls

10

Example Test Case

• addPrereq “cs 210”• addStudent• addTaken “cs 210”• addTaken “cs 212”• addPrereq “cs 211”• addStudent• …

11

Randomized Unit Test Cases

• Testing engine can execute longer and longer randomized test cases

(TC2)

(TC1) (TC3)

(Failure)

…

12

Previous Results

• TSE July 2003 paper: for lab-built units– Randomized testing covered more, was more

effective than thorough black-box test suite– Randomized testing more efficient than

effective black+white-box test suite• ASE 2004 paper: for Sourceforge units

– Building test fragments and driver reasonably fast

– Randomized testing effective

13

Issue: Failing Test Case Length

• Randomized testing can find failing test cases

• These test cases tend to be long– Length = number of test fragments (“lines”)– Execute many irrelevant test fragments in

addition to ones on failing path• Long test cases difficult to use in debugging

14

Solution: ddmin

• Can we apply Zeller & Hildebrandt’s ddmin algorithm to minimize them?

• How much smaller do they get?• What benefits do we obtain?• How efficient is the process?

15

ddmin overview

• Start by cutting out large blocks, preserving failure

• Cut out smaller and smaller blocks, preserving failure

• Result: failing test case s.t. deleting any line will result in a successful test case

16

Case Study: Sourceforge Units

• 3 faulty units with 2 faults each, 827 SLOC• Found 100 failing test cases for each; minimized

them• Example: htable.c fault 1

– Mean size of original failing test case: 51.69– Mean size of minimized test case: 5.12

• Average ratio minimized/original: 0.11• Average time to find and minimize: 1.8 CPU sec• Minimized test cases focused debugging

17

18

Experiment: Lab Data Structures

• 4 implementations of Dictionary ADT– Add, delete, find functions– Linked list, BST, AVL, B-tree impls– 681 NLOC

• 945 mutants (faulty versions) generated• Test fragment = 1 call to add, delete or find• Found 10 failing test cases for each;

minimized them

19

20

21

22

23

24

25

Related Work

• Randomized testing: e.g. McKeeman• Jartege (Oriat)• Recent work on model checking +

randomized testing