Date post: | 06-Jan-2018 |
Category: |
Documents |
Upload: | dennis-jackson |
View: | 216 times |
Download: | 0 times |
Minimization of Randomized Unit Test Cases
Yong Lei, James H. AndrewsDept. of Computer Science Univ. Of Western OntarioLondon, Ontario, Canada
ISSRE 2005, 10 Nov 2005
I apologize in advance for not being able to speak more loudly.
3
Contents
• Randomized unit testing• Minimization• Case study• Experiment• Related work
4
Unit Testing
• Testing methods, groups of methods, classes
• Key practice of TDD
5
Conventional Unit Testing
• Write test cases (sequences of method calls)• Execute and verify each test case• Record % successes and failures
6
Randomized Unit Testing
• Write test fragments• Execute many random sequences of
fragments• Keep failures, throw away successes• Potential: lots of automatically generated
test input, “unexpected” sequences
7
Example Unit Under Test: “Course” Class
• Student object contains Courses taken• Course object contains prerequisite Courses• Course class contains addStudent method
– Succeeds (returning true) iff Student has taken all prerequisites
– Otherwise returns false
8
Course: ConventionalUnit Testing
• Initialization: create Course c with prerequisites
• Test cases:– TC1: create Student with prerequisites; add to c– TC2: create Student with only some
prerequisites; try to add to c– TC3: create Student with no prerequisites; try
to add to c
9
Course: Randomized Testing
• Initialization: create Course c, Student s• Test fragments:
– Add/delete random prerequisite from Course c– Add/delete random course taken from Student s– Try to add s to c
• Randomized test case = random sequence of test fragment calls
10
Example Test Case
• addPrereq “cs 210”• addStudent• addTaken “cs 210”• addTaken “cs 212”• addPrereq “cs 211”• addStudent• …
11
Randomized Unit Test Cases
• Testing engine can execute longer and longer randomized test cases
(TC2)
(TC1) (TC3)
(Failure)
…
12
Previous Results
• TSE July 2003 paper: for lab-built units– Randomized testing covered more, was more
effective than thorough black-box test suite– Randomized testing more efficient than
effective black+white-box test suite• ASE 2004 paper: for Sourceforge units
– Building test fragments and driver reasonably fast
– Randomized testing effective
13
Issue: Failing Test Case Length
• Randomized testing can find failing test cases
• These test cases tend to be long– Length = number of test fragments (“lines”)– Execute many irrelevant test fragments in
addition to ones on failing path• Long test cases difficult to use in debugging
14
Solution: ddmin
• Can we apply Zeller & Hildebrandt’s ddmin algorithm to minimize them?
• How much smaller do they get?• What benefits do we obtain?• How efficient is the process?
15
ddmin overview
• Start by cutting out large blocks, preserving failure
• Cut out smaller and smaller blocks, preserving failure
• Result: failing test case s.t. deleting any line will result in a successful test case
16
Case Study: Sourceforge Units
• 3 faulty units with 2 faults each, 827 SLOC• Found 100 failing test cases for each; minimized
them• Example: htable.c fault 1
– Mean size of original failing test case: 51.69– Mean size of minimized test case: 5.12
• Average ratio minimized/original: 0.11• Average time to find and minimize: 1.8 CPU sec• Minimized test cases focused debugging
17
18
Experiment: Lab Data Structures
• 4 implementations of Dictionary ADT– Add, delete, find functions– Linked list, BST, AVL, B-tree impls– 681 NLOC
• 945 mutants (faulty versions) generated• Test fragment = 1 call to add, delete or find• Found 10 failing test cases for each;
minimized them
19
20
21
22
23
24
25
Related Work
• Randomized testing: e.g. McKeeman• Jartege (Oriat)• Recent work on model checking +
randomized testing