1
James A. JonesCollege of Computing
Georgia Institute of Technology
Automation to Assist SoftwareDebugging
Supported by Tata Consultancy Services, NSF, NASABoeing Commercial Airplanes
CERCS IAB Workshop April 2007
Overview
TTTTT
TTTTT
Passing Failing
test
2
CERCS IAB Workshop April 2007
Debugging Phases
• Make changes to the program• Test• Observe failure• Locate bugs• Identify possible fixes• Choose best fix• Implement fix
CERCS IAB Workshop April 2007
Debugging Phases
• Make changes to the program• Test• Observe failure• Locate bugs – Fault Localization• Identify possible fixes• Choose best fix• Implement fix
3
CERCS IAB Workshop April 2007
Current Practice
CERCS IAB Workshop April 2007
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
Technique usesDynamic information
• statements executed• outcome (pass/fail)
Statistical analysis• computes suspiciousness of each statement
Tarantula: Fault Localization
Intuition: Statements that areprimarily executed by failedtest cases are moresuspicious than statementsthat are primarily executed bypassed test cases
4
CERCS IAB Workshop April 2007
••••
••
•
Tarantula
mid() { int x,y,z,m;1: read(“Enter 3 numbers:”,x,y,z);2: m = z;3: if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y; // bug8: else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13: print(“Middle number is:”, m); }
3,3,
5
1,2,
33,
2,1
5,5,
5
5,3,
42,
1,3
P P P P P F
•••••
•
•••
•••
•
•••
••
•
•
••••
•
•
••••
••
•
Pass Status
safe suspicious
CERCS IAB Workshop April 2007
Coloring statements
3,3,
5
1,2,
33,
2,1
5,5,
5
5,3,
42,
1,3
P P P P P F
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
•
•
••••
••
•
Test Cases
Pass Status
mid() { int x,y,z,m;1: read(“Enter 3 numbers:”,x,y,z);2: m = z;3: if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8: else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13: print(“Middle number is:”, m); }
safe suspicious
5
CERCS IAB Workshop April 2007
Tarantula
CERCS IAB Workshop April 2007
0 20 40 60 80 100% of program to be examined to find bug
% o
f bug
s
0
20
40
60
80
100
Tarantula on Space (~6000 LOC)
Results
6
CERCS IAB Workshop April 2007
Overview
TTTTT
TTTTT
Passing Failing
test
CERCS IAB Workshop April 2007
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
3,3,
51,
2,3
3,2,
25,
5,5
1,1,
45,
3,4
P P P P F
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
••
•
••••
•
•
Pass/fail Status
3,2,
12,
1,3
5,4,
25,
2,6
•••
•••
•
••••
••
•
••••
••
•
•••
•••
•P P F FF
Tarantula: Fault Localization
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
//bug;correct m=y
7
CERCS IAB Workshop April 2007
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
3,3,
51,
2,3
3,2,
25,
5,5
1,1,
45,
3,4
P P P P F
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
••
•
••••
•
•
Pass/fail Status
3,2,
12,
1,3
5,4,
25,
2,6
•••
•••
•
••••
••
•
••••
••
•
•••
•••
•P P P FP
Tarantula: Fault Localization
//fixed
CERCS IAB Workshop April 2007
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
3,3,
51,
2,3
3,2,
25,
5,5
1,1,
45,
3,4
P P P P F
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
••
•
••••
•
•
Pass/fail Status
3,2,
12,
1,3
5,4,
25,
2,6
•••
•••
•
••••
••
•
••••
••
•
•••
•••
•P P P FP
Tarantula: Fault Localization
//bug;correct:m=x;
//fixed
8
CERCS IAB Workshop April 2007
••••
••
•
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = x;8:else9: if (x>y)10: m = y;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
3,3,
51,
2,3
3,2,
25,
5,5
1,1,
45,
3,4
P P P P P
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
••
•
••••
•
•
Pass/fail Status
3,2,
12,
1,3
5,4,
25,
2,6
•••
•••
•
•••
•••
•P P P PP
Tarantula: Fault Localization
//fixed
//fixed
••••
••
•
CERCS IAB Workshop April 2007
Debugging Process
PExecute Debug
testsfail P’
alltestspassExecute Debug
testsfail P’’
Pi isfailure-free
Execute …
• Are all failing tests caused by the same fault?• Are all failing tests caused by the same fault?• Can we associate groups of tests with different faults?
• Are all failing tests caused by the same fault?• Can we associate groups of tests with different faults?• Can we search for different faults simultaneously?
9
CERCS IAB Workshop April 2007
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
3,3,
51,
2,3
3,2,
25,
5,5
1,1,
45,
3,4
P P P P F
••••
••
•
•••••
•
•••
•••
•
•••
••
•
•
••••
••
•
••••
•
•
Pass/fail Status
3,2,
12,
1,3
5,4,
25,
2,6
•••
•••
•
••••
••
•
••••
••
•
•••
•••
•P P F FF
Tarantula: Fault Localization
mid() { int x,y,z,m;1:read(“Enter 3 integers:”,x,y,z);2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”, m); }
CERCS IAB Workshop April 2007
Debugging Process
PExecute Debug
testsfail P’
alltestspassExecute Debug
testsfail P’’
SequentialDebugging
testsfailExecute
P Debug
Debug
P’Execute
alltestspass
ParallelDebugging P’’ is
failure-freeP’ is
failure-free
Execute …
…
10
CERCS IAB Workshop April 2007
Debugging Process
PExecute Debug
testsfail P’
alltestspassExecute Debug
testsfail P’’
SequentialDebugging
testsfailExecute
P Debug
Debug
P’Execute
alltestspass
ParallelDebugging P’’ is
failure-freeP’ is
failure-free
Execute …
…
Potential benefits:• Reduced time to failure-free program
Potential cost:Multiple debuggers +
Overhead to partition test cases
Potential benefits:• Reduced time to failure-free program• Better utilization of developer effort• Fewer tests executed• Early indication of the number of faults• Less “noise” in locating each fault
Crucial problem:• Partitioning failed tests into groups
of similar behavior—focus ondifferent faults
• fault-focusing clusters
CERCS IAB Workshop April 2007
Fault-focusing Clusters—Overview
t06
t07
t09
t08
t10
t08t07
t04
t09
t02
t05
t01 t03
t10
Fault-focusing clusters:• Clusters of failing test cases• Clusters failing in similar way• Each cluster targeting a different fault
Test Cases
11
CERCS IAB Workshop April 2007
Fault-focusing Clusterst07
t09
t08
t10
t06
t04t02
t05
t01 t03
Test Cases
t06
t04t02
t05
t01 t03
Specialized Test Suites
Developer 1
Developer 2
Specialized test suites:Fault-focusing clusters
combined withpassing test cases
Specialized test suites:Fault-focusing clusters
combined withpassing test cases
Give each specializedtest suite to differentdeveloper (debugger)
CERCS IAB Workshop April 2007
mid() { int x,y,z,m;1:read(“Enter 3 integers:”…2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”… }
//bug
//bug
Visualization of Each Result
mid() { int x,y,z,m;1:read(“Enter 3 integers:”…2:m = z;3:if (y<z)4: if (x<y)5: m = y;6: else if (x<z)7: m = y;8:else9: if (x>y)10: m = z;11: else if (x>z)12: m = x;13:print(“Middle number is:”… }
//bug
//bug
12
CERCS IAB Workshop April 2007
Fault-focusing Clusters
ExecutionClustering
FaultLocalization
failed test casesexecution
information
specializedtest suites
fault locationprediction
Clustering Technique 1: Less aggressive
Clustering Technique 2: More aggressive
CERCS IAB Workshop April 2007
D
0
5
10
15
20
25
30
35
40
D
D 36.26046842 31.14062524 26.16479859
S CT1 CT2
FF
0
5
10
15
20
25
30
35
40
FF
FF 36.26046842 18.28856692 20.78618781
S CT1 CT2
• Parallel less expensive even if debugging sequential• CT1 if time to delivery is most important (low salaries, high urgency)• CT2 if developer cost is most important (high salaries, low urgency)
Summary of ResultsD FF
13
CERCS IAB Workshop April 2007
Conclusion• Technique and visualization to provide support
for finding faults• New approach to debugging—parallelized
debugging• Technique to get fault-focused clusters to
create specialized test suites• Benefits, supported by studies
• less time to failure-free program• better utilization of developer effort• less running of the test suite• indication of the number of faults• less “noise” in locating each fault
CERCS IAB Workshop April 2007
DebuggingTarantula: Fault Localization
• Jones, Harrold, Stasko, Workshop on SoftwareVisualization 2001
• Jones, Harrold, Stasko, International Conference onSoftware Engineering 2002
• Jones, Orso, Harrold, Software Visualization 2003• Jones, ICSE 2004 Doctoral Workshop• Jones, Orso, Harrold, Journal of Information
Visualization 2004• Jones, Harrold, Automated Software Engineering
2005• Jones, Bowring, Harrold, International Symposium
on Software Testing and Analysis, 2007