Practical Techniques for Improving theEfficiency and Usability of Mutation Analysis
for Java Programs
Gregory M. Kapfhammer†
Department of Computer ScienceAllegheny College
http://www.cs.allegheny.edu/∼gkapfham/
University of Sheffield – February 3, 2012†Joint with Rene Just and Franz Schweiggert (University of Ulm) and Jonathan Miller Kauffman (Allegheny College)
Introduction Mutation Analysis Empirical Evaluation Conclusion
Important Points
Accessing the Presentation
Scan this QR Code with your smartphone!
... or, visit this Web site:
http://is.gd/rekiwo
... or, ask me for a USB drive!
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Important Points
Presenter Introduction: Gregory M. Kapfhammer
test
testing
softwaresuites
prioritization
components
coverageem
piric
allyregressionsuiteapplications
evaluating
algo
rithm
analysis
data
effectiveness
empirical
genetic
perf
orm
ance
understanding
using
approachcommercialofftheshelf
comparison
comprehensive
constrainedcreation
databasecentric envi
ronm
ents
execution
findi
ng
framework
identifying
interactive
javamethods
multiplots
mutation
party
prioritized
reduction
relational
study
third
timeaware
towards
105
adequacy
appr
oach
es
array
automatically
building
callchallenges
chapter
communication
compare
compressing
com
pute
r
conditional
cost
cots
covering
creating
criteria
database
databaseaware
data
base
driv
en
databases
declarativedependable
detection
devices
distributed
distributing
duringsearchbased
dynamic
efficiency
efficient
engi
neer
ing
environment
evaluate
examination
executing
experimental
family
flow
forward
frameworks
free
generation
greedy
gui
hamiltonian
handbook
heaps
implementation
impr
ove
incorporating
incr
ease
information
initial
intranode
invariant
javaspace
javaspacebased
kernel
knapsack
linux
measurement
memory
method
monitoring
operators
paths
poster
potential
preliminary
primitives
prio
ritiz
atio
ns
prio
ritiz
ers
prioritizing
problematic
receive
remote
reports
resource
resourceconstrained
results
role
runtime
science
searchbased
selectionsolutions
solvers
space
studies
stud
yingsupported
synthetic
techniques
transmission
transparently
trees
tuple
unstructured
wrappers
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
Input
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
Input Output
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
TestVerdict
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
TestVerdict
ExpectedOutput
Output
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
TestVerdict
ExpectedOutput
Output
TestVerdict
The test case passes and the code is correct!
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
TestVerdict
ExpectedOutput
Output
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Case?
MethodUnder Test
TestSet Up
Input Output
TestClean Up
TestOracle
ExpectedOutput
TestVerdict
ExpectedOutput
Output
TestVerdict
The test case fails and a defect is found!
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
B2B2B2B2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
B2B2B2B2B2B2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
B2B2B2B2B2B2B2B2B2B2
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
How Good is Test Suite T ?
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
How Good is Test Suite T ?
Coverage Analysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Software Testing
What is a Test Suite?
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
Test Suite T = 〈T1,T2, . . . ,T9,T10〉
R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2
Requirements R = {R1, . . . ,R6}, Features F = {F1, . . . ,F4}, Bug Fixes B = {B1,B2}
How Good is Test Suite T ?
Coverage Analysis Mutation Analysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Methodicallyinject smallsyntacticalfaults into
the programunder test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Methodicallyinject smallsyntacticalfaults into
the programunder test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Methodicallyinject smallsyntacticalfaults into
the programunder test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Methodicallyinject smallsyntacticalfaults into
the programunder test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
MutationOperator
MutationOperator
MutationOperator
MutationOperator
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
The test suitecannot kill themutant – either
a test suiteweakness oran equivalent
mutant!
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Overview of Mutation Analysis
Test Case T1 Test Case T2 Test Case T3 Test Case T4
Execute thetest suite after
enabling asingle mutantin the program
under test
Repeat thisprocess forall of the
test casesand mutants– calculate
mutation scorewhen finished
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Syntax TreeTransformation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Syntax TreeTransformation
Expressionsand Statements
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Syntax TreeTransformation
Expressionsand Statements
CompilerIntegrated
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Syntax TreeTransformation
Expressionsand Statements
CompilerIntegrated
ComprehensiveEmpirical Study
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Contributions of this Presentation
EfficientMutationAnalysis
Challenges
Solutions
Conditional Mutation
Syntax TreeTransformation
Expressionsand Statements
CompilerIntegrated
ComprehensiveEmpirical Study
Efficient Technique - Fully Integrated into the Java 6 SE Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
Methodicallyinject smallsyntacticalfaults into
the programunder test
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
• y = a - x;
• y = a + x;
• y = a / x;
• if(b < a)
• if(b != a)
• if(b == a)
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
Unbiasedand powerfulmethod forassessing
oracles andinput values
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Understanding Mutation Analysispublic int eval(int x){
int a=3, b=1, y;
y = a * x;
y += b;return y;
}
public int max(int a, int b){int max = a;
if(b>a){
max=b;}
return max;}
=⇒
=⇒
Unbiasedand powerfulmethod forassessing
oracles andinput values
Useful methodfor fault seeding
during theempirical study
of testingtechniques
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
MutationAnalysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
MutationAnalysisTests
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
MutationAnalysisTests Results
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
MutationAnalysisTests Results
Individually Executing theMutants is Too Expensive
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Mutation Analysis Challenges
MutantGeneration
MutationOperators
Program
Mutants
Often Yields aSubstantial Num-
ber of Mutants
High Time Over-head for Generation
MutationAnalysisTests Results
Individually Executing theMutants is Too Expensive
PriorSolutions?
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer
Sampling Selection
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer Do Smarter
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer Do Smarter
Distributed Weak Mutation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer Do Smarter
Do Faster
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer Do Smarter
Do Faster
CompilerIntegrated
BytecodeTransformation
MutantSchemata
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Fundamental Concepts
Prior Work in Mutation Analysis
Improving Mutation Analysis Offutt andUntch
Do Fewer Do Smarter
Do Faster
Higher OrderMutation
Jia andHarman
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Practical Mutation Analysis
Practical (adjective):
1 Of or concerned with the actual doing or use of somethingrather than with theory and ideas
2 (of an idea, plan, or method) Likely to succeed or beeffective in real circumstances; feasible
3 Suitable for a particular purpose
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Practical Mutation Analysis
Practical (adjective):
1 Of or concerned with the actual doing or use of somethingrather than with theory and ideas
2 (of an idea, plan, or method) Likely to succeed or beeffective in real circumstances; feasible
3 Suitable for a particular purpose
What are the practical techniques that MAJOR employs forimproving the efficiency and usability of mutation analysis?
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Conditional Mutation
Conditional Mutation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Conditional Mutation
Conditional Mutation
Encapsulates allmutants withinthe same block
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Conditional Mutation
Conditional Mutation
Encapsulates allmutants withinthe same block
Can be inte-grated withinthe compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Conditional Mutation
Conditional Mutation
Encapsulates allmutants withinthe same block
Transforms theabstract syntax
tree (AST)
Can be inte-grated withinthe compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Conditional Mutation
Conditional Mutation
Encapsulates allmutants withinthe same block
Transforms theabstract syntax
tree (AST)
Stmt → Conditional Stmt(if-then-else, switch)
Expr → Conditional Expr(conditional operator ?:)
Can be inte-grated withinthe compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Transforming the AST
public int eval(int x){int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
⇓
=⇒
=⇒
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Transforming the AST
public int eval(int x){int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
⇓
ASSIGN
IDENT
y
BINARY
∗
a x
=⇒
=⇒
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Transforming the AST
public int eval(int x){int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
⇓
ASSIGN
IDENT
y
BINARY
∗
a x
=⇒
=⇒
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Transforming the AST
public int eval(int x){int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
⇓
ASSIGN
IDENT
y
BINARY
∗
a x
=⇒
=⇒
ASSIGN
IDENT
y
COND-EXPR
THEN
BINARY
+
a x
COND
(M NO ==2)
ELSE
COND-EXPR
THEN
BINARY
-
a x
COND
(M NO ==1)
ELSE
BINARY
∗
a x
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==1)? a - x :a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==2)? a + x :(M_NO==1)? a - x :
a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :
a * x ;
y += b;return y;
}
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Source Code View of Inserting Mutantspublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :
a * x ;
y += b;return y;
}
Mutants that are not ex-ecuted cannot be killed
1 Define mutation operators MOP(x ∗ y) = {x − y , x + y , x/y}
2 Determine whether current expression or statement isaffected by mutation
3 Apply mutation operators
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Collecting and Using Mutation Coveragepublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :
a * x ;
y += b;return y;
}
Mutants that are not ex-ecuted cannot be killed
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Collecting and Using Mutation Coveragepublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :
(M_NO==0 &&
COVERED(1,3))?
a * x : a * x ;
y += b;
return y;}
Mutants that are not ex-ecuted cannot be killed
Determine coveredmutants with addi-
tional instrumentation
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Collecting and Using Mutation Coveragepublic int eval(int x){
int a=3, b=1, y;
y = (M_NO==3)? a / x :(M_NO==2)? a + x :(M_NO==1)? a - x :
(M_NO==0 &&
COVERED(1,3))?
a * x : a * x ;
y += b;
return y;}
Mutants that are not ex-ecuted cannot be killed
Determine coveredmutants with addi-
tional instrumentation
Only execute and investi-gate the covered mutants
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Enhanced StandardJava Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Enhanced StandardJava Compiler
Source Files
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Enhanced StandardJava Compiler
Source Files
CommonCompiler Options
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Enhanced StandardJava Compiler
Source Files
CommonCompiler Options
Domain SpecificLanguage
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Compiler
MAJOR’sCompiler
Enhanced StandardJava Compiler
Source Files
CommonCompiler Options
Domain SpecificLanguage
Bytecode withEmbedded
Mutants
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Integration into the Java Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Integration into the Java Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Integration into the Java Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Integration into the Java Compiler
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};
// Define replacement list
BIN(+)<"org"> -> {-,*};
BIN(*)<"org"> -> {/,%};
// Define own operator
myOp{
BIN(&&) -> listCOR;
BIN(||) -> listCOR;
COR;
LVR;
}
// Enable built-in operator AOR
AOR<"org">;
// Enable operator myOp
myOp<"java.lang.System@println">;
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};
// Define replacement list
BIN(+)<"org"> -> {-,*};
BIN(*)<"org"> -> {/,%};
// Define own operator
myOp{
BIN(&&) -> listCOR;
BIN(||) -> listCOR;
COR;
LVR;
}
// Enable built-in operator AOR
AOR<"org">;
// Enable operator myOp
myOp<"java.lang.System@println">;
Specify mutationoperators in detail
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};
// Define replacement list
BIN(+)<"org"> -> {-,*};
BIN(*)<"org"> -> {/,%};
// Define own operator
myOp{
BIN(&&) -> listCOR;
BIN(||) -> listCOR;
COR;
LVR;
}
// Enable built-in operator AOR
AOR<"org">;
// Enable operator myOp
myOp<"java.lang.System@println">;
Specify mutationoperators in detail
Define own mutationoperator groups
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
MAJOR’s Domain Specific Language// variable declarationlistCOR={&&, ||, ==, !=};
// Define replacement list
BIN(+)<"org"> -> {-,*};
BIN(*)<"org"> -> {/,%};
// Define own operator
myOp{
BIN(&&) -> listCOR;
BIN(||) -> listCOR;
COR;
LVR;
}
// Enable built-in operator AOR
AOR<"org">;
// Enable operator myOp
myOp<"java.lang.System@println">;
Specify mutationoperators in detail
Define own mutationoperator groups
Enable operators fora specific package,
class, or method
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Optimized Mutation Analysis Process
1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Optimized Mutation Analysis Process
1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Optimized Mutation Analysis Process
1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis with MAJOR
Optimized Mutation Analysis Process
1 Embed and compile all mutants2 Run test suite on instrumented program3 Sort tests according to their runtime4 Perform mutation analysis with reordered test suite
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Mutant Generation and Compilation
1
2
3
4
5
6
7
8
9
10
11
12
0 20000 40000 60000 80000 100000 120000 140000
Co
mp
iler
run
tim
e in
se
co
nd
s
Number of mutants
apache antjfreechart
itextjava pathfindercommons mathcommons lang
numerics4j
Overhead for generating and compiling mutants is negligible
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Mutant Generation and Compilation
1
2
3
4
5
6
7
8
9
10
11
12
0 20000 40000 60000 80000 100000 120000 140000
Co
mp
iler
run
tim
e in
se
co
nd
s
Number of mutants
apache antjfreechart
itextjava pathfindercommons mathcommons lang
numerics4j
Overhead for generating and compiling mutants is negligible
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption
original instrumented original instrumentedwcs wcs+cov
aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303
itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149
numerics4j 5,869 1.2 1.3 1.6 73 90
• Runtime overhead is application dependent• Larger for CPU-bound applications
• Small for I/O-bound applications
• Even for large projects, applicable on commodity workstations
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption
original instrumented original instrumentedwcs wcs+cov
aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303
itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149
numerics4j 5,869 1.2 1.3 1.6 73 90
• Runtime overhead is application dependent• Larger for CPU-bound applications
• Small for I/O-bound applications
• Even for large projects, applicable on commodity workstations
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption
original instrumented original instrumentedwcs wcs+cov
aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303
itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149
numerics4j 5,869 1.2 1.3 1.6 73 90
• Runtime overhead is application dependent• Larger for CPU-bound applications
• Small for I/O-bound applications
• Even for large projects, applicable on commodity workstations
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption
original instrumented original instrumentedwcs wcs+cov
aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303
itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149
numerics4j 5,869 1.2 1.3 1.6 73 90
• Runtime overhead is application dependent• Larger for CPU-bound applications
• Small for I/O-bound applications
• Even for large projects, applicable on commodity workstations
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Compilation Efficiency
Time and Space OverheadApplication Mutants Runtime of test suite Memory consumption
original instrumented original instrumentedwcs wcs+cov
aspectj 406,382 4.3 4.8 5.0 559 813apache ant 60,258 331.0 335.0 346.0 237 293jfreechart 68,782 15.0 18.0 23.0 220 303
itext 124,184 5.1 5.6 6.3 217 325java pathfinder 37,331 17.0 22.0 29.0 182 217commons math 67,895 67.0 83.0 98.0 153 225commons lang 25,783 10.3 11.8 14.8 104 149
numerics4j 5,869 1.2 1.3 1.6 73 90
• Runtime overhead is application dependent• Larger for CPU-bound applications
• Small for I/O-bound applications
• Even for large projects, applicable on commodity workstations
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis Efficiency
Evaluating and Improving Mutation Analysis
0
20
40
60
80
100
120
140
160
180
0 20 40 60 80 100
Nu
mb
er
of
kill
ed
mu
tan
ts
Runtime in seconds
optimized order (using coverage information)
random order (using coverage information)
original order (using coverage information)
optimized order (without coverage information)
random order (without coverage information)
original order (without coverage information)
• Mutation analysis is not feasible without coverage information• Reordering the test suite significantly speeds up the process,
especially if runtimes of tests differ by orders of magnitudeKapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis Efficiency
Evaluating and Improving Mutation Analysis
0
20
40
60
80
100
120
140
160
180
0 20 40 60 80 100
Nu
mb
er
of
kill
ed
mu
tan
ts
Runtime in seconds
optimized order (using coverage information)
random order (using coverage information)
original order (using coverage information)
optimized order (without coverage information)
random order (without coverage information)
original order (without coverage information)
• Mutation analysis is not feasible without coverage information• Reordering the test suite significantly speeds up the process,
especially if runtimes of tests differ by orders of magnitudeKapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Mutation Analysis Efficiency
Evaluating and Improving Mutation Analysis
0
20
40
60
80
100
120
140
160
180
0 20 40 60 80 100
Nu
mb
er
of
kill
ed
mu
tan
ts
Runtime in seconds
optimized order (using coverage information)
random order (using coverage information)
original order (using coverage information)
optimized order (without coverage information)
random order (without coverage information)
original order (without coverage information)
• Mutation analysis is not feasible without coverage information• Reordering the test suite significantly speeds up the process,
especially if runtimes of tests differ by orders of magnitudeKapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Revisiting Practical Mutation Analysis
Practical (adjective):
1 Of or concerned with the actual doing or use of somethingrather than with theory and ideas
2 (of an idea, plan, or method) Likely to succeed or beeffective in real circumstances; feasible
3 Suitable for a particular purpose
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Revisiting Practical Mutation Analysis
Practical (adjective):
1 Of or concerned with the actual doing or use of somethingrather than with theory and ideas
2 (of an idea, plan, or method) Likely to succeed or beeffective in real circumstances; feasible
3 Suitable for a particular purpose
The evidence suggests that MAJOR is “likely to succeed or beeffective” in real-world software testing circumstances
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Reviewing MAJOR’s Contributions
MutationAnalysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Reviewing MAJOR’s Contributions
MutationAnalysis
Efficiency: MAJOR has ac-ceptable time and space over-heads and scales to large,real-world programs
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Reviewing MAJOR’s Contributions
MutationAnalysis
Efficiency: MAJOR has ac-ceptable time and space over-heads and scales to large,real-world programs
Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Retrospective
Reviewing MAJOR’s Contributions
MutationAnalysis
Efficiency: MAJOR has ac-ceptable time and space over-heads and scales to large,real-world programs
Usability: MAJOR’s inte-gration into the Java SEcompiler makes it a no-hassle, drop-in tool
We will release MAJOR as free and open source software
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Conclusions and Future Work
Conclusion
Key Concepts and Features:
• Compiler-integrated solution• Conditional mutation with the abstract syntax tree• Furnishes its own domain specific language• Collects and leverages mutation coverage information
Characteristics of MAJOR:
• Fast and scalable technique• Configurable and extensible mutation tool• Enables an optimized workflow for mutation analysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Introduction Mutation Analysis Empirical Evaluation Conclusion
Conclusions and Future Work
Conclusion
Key Concepts and Features:
• Compiler-integrated solution• Conditional mutation with the abstract syntax tree• Furnishes its own domain specific language• Collects and leverages mutation coverage information
Characteristics of MAJOR:
• Fast and scalable technique• Configurable and extensible mutation tool• Enables an optimized workflow for mutation analysis
Kapfhammer Allegheny College
Practical Techniques for Improving the Efficiency and Usability of Mutation Analysis for Java Programs
Practical Techniques for Improving theEfficiency and Usability of Mutation Analysis
for Java Programs
Gregory M. Kapfhammer
Department of Computer ScienceAllegheny College
http://www.cs.allegheny.edu/∼gkapfham/
Thank you for your attention!I welcome your questions and comments.