Oop 2015 – Mutation Testing

© Computas AS 26.01.15

Mutation Testingwith Mutant and PIT

Filip van Laenen & Markus SchirpOOP 2015

2015-01-26

2 © Computas AS 26.01.15

Agenda

• Basics of mutation testing• Relation to other testing techniques• Example• Mutation testing techniques• Mutation testing tools• Personal experiences and recommendations


Basics ofMutation Testing


Mutation Testing in a Nutshell

Seeking The Summoner @ The Daily WTFhttp://thedailywtf.com/Articles/Seeking-The-Summoner.aspx


Mutation Testing in a Nutshell (cont'd)





• Unit tests guard the source code• But who guards the guardians?

• Do the unit tests cover all source code?• Lines?• Branches?• Paths?• Semantics?

• Do the unit tests test the right things?

Mutation testing tests the tests!















int max(int a, int b) { return (a < b) ? b : a;}

int max(int a, int b) { return (a <= b) ? b : a;}



“


Does Mutation Testing Work?

In practice, if the software contains a fault, there will usually be a set of

mutants that can only be killed by a test case that also detects that fault.

Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992

“


Does Mutation Testing Work? (cont'd)

Complex faults are coupled to simple faults in such a way that a test data set

that detects all simple faults in a program will detect most complex faults.

K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995


Does Mutation Testing Work? (cont'd)

• “Generated mutants are similar to real faults.”• Andrews, Briand, Labiche, ICSE 2005

• “Mutation testing is more powerful than statement or branch coverage.”• Walsh, Ph.D. Thesis, State University of New York at

Binghampton, 1985• “Mutation testing is superior to data flow

coverage criteria.”• Frankl, Weiss, Hu, Journal of Systems and Software,

1997


Relation to OtherTesting Techniques


Relation to Other Testing Techniques

• Unit tests• Test-Driven Development (TDD)• Test coverage• Static code analysis• Fuzz testing


Relation to Other Testing Techniques


Is Mutation Testing New?

• R. Lipton, “Fault Diagnosis of Computer Programs,” 1971

• R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978

• Historical obstacles:• No unit testing• No TDD• Inefficient implementation

• Hence time-consuming• No integration with IDEs


Practical Exampleof Mutation Testing


Code Along…

https://github.com/computas-fvl/oop2015

git clone \https://github.com/computas-fvl/oop2015.git

mvn clean site:site \ org.pitest:pitest-maven:mutationCoverage


Mutation TestingTechniques


Mutation Testing Techniques

• Three aspects:• Mutation injection• Mutation types• Unit test selection per mutant

• Key properties:• Efficiency• Performance• Coverage


Mutation Injection

• Source code mutation• Intermediate code mutation• Binary code mutation• Caveats:

• De-mutation (for reporting)• Compilation errors• Invalid binary code• Leaked state


Mutation Types

• Some mutations never change behaviour• Constants reused by unit tests• Log messages• Exception messages

• Some mutations can change behaviour• Switching between < and ≠• Switching between < and ≤

• Some mutations always change behaviour• Switching between < and ≥


Mutation Types (cont'd)

int max(int a, int b) { return (a < b) ? b : a;}

int max(int a, int b) { return (a <= b) ? b : a;}

int max(int a, int b) { return (a >= b) ? b : a;}


Mutation Types (cont'd)

for (int i = 0; i < 10; i++) …

for (int i = 0; i != 10; i++) …

for (int i = 0; i >= 10; i++) …


Mutation Types Guaranteed to Change Behaviour

• Negation of the comparison• Switching between = and ≠• Switching between < and ≥• Switching between > and ≤

• Negation of boolean conditions• Adding a ¬, ! or ~

• Shortcutting boolean conditions• Replacement with True or False

*


Unit Test Selection

• Goal: find the unit test that “kills” the mutant• Selection aids:

• Hints• Name/package matching• Code coverage tools• Incremental mutation testing• Other heuristics


Unit Test Selection (cont'd)

• System:• 50 classes• 20 unit tests per class• 1 ms per unit test• Unit testing time: 50 × 20 × 1ms = 1s

• 10 mutants per class:• Brute-force: 10 × 50 × 1s = 6m 20s• Educated: 10 × 50 × 20 × 1ms = 10s


Unit Test Selection (cont'd)

• System:• 500 classes• 20 unit tests per class• 1 ms per unit test• Unit testing time: 500 × 20 × 1ms = 10s

• 10 mutants per class:• Brute-force: 10 × 500 × 10s = 13h 53m 20s• Educated: 10 × 500 × 20 × 1ms = 1m 40s


Complexity

• f: Number of function points• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × f²• Educated force: τ × μ × φ × f• Ideal: τ × μ × f


Complexity (cont'd)

• c: Number of classes• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²• Educated force: τ × μ × φ² × c• Ideal: τ × μ × φ × c


Loops

for (int i = 0; i < 10; i++) …

for (int i = 0; i < 10; i--) …


Infinite Loops

• Terminate mutants that take too long to run• What's “too long”?

• Ruins the performance• Can be hard to predict


Other Problems

• Recursion:• Stack overflows• Out of memory exceptions

• Syntax errors• Leaked state• Segmentation faults


Mutation TestingTools


Mutation Testing Tools

• Java:• PIT

• Ruby:• Mutant• Heckle

• PHP• Humbug

• C#:• NinjaTurtles


PIT

• Java• Junit & TestNG• Maven or command-line• Operates on byte code• Large set of mutators

• Also possibly equivalent mutators available• Highly configurable• Sensible defaults

• http://pitest.org/


PIT Mutators

• Conditionals Boundary• Negate Conditionals• Remove Conditionals*• Math• Increments• Invert Negatives• Inline Constant*• Return Values• Void Method Calls• Non Void Method Calls*• Constructor Calls*


PIT Sample Report


Jester

• Java• JUnit• Usually run from the command-line

• Grester for Maven2• Operates on source code• Runs all unit tests on all classes• Reporting and documentation could be better

• http://jester.sourceforge.net/• http://sourceforge.net/projects/grester/


Jester Sample Report Overview


Jester Sample Detailed Report


Pester and Nester

• Pester• Jester for Python• PyUnit

• Nester• Port of Jester for C#• NUnit• Integrated with Visual Studio• But outdated…• http://nester.sourceforge.net/


Nester Sample Report


Jumble

• Java• JUnit• Run from the command-line• Operates on byte code• Runs unit tests on a class• Reporting and documentation could be better• Claims to be faster than Jester

• http://jumble.sourceforge.net/index.html


Jumble Sample Report

Mutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s..M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67%


Mutant

• Ruby 1.9 up to 2.2• Rspec2 and Rspec3• Usually run from the command-line• Good to-the-point reporting• Good performance• OK documentation• Active project

• https://github.com/mbj/mutant


Heckle

• Ruby 1.8• Doesn't work on Ruby 1.9

• Test::Unit and rSpec• Usually run from the command-line• Runs a set of unit tests on a class or a method• Good to-the-point reporting• Acceptable performance• Virtually no documentation

• http://rubyforge.org/projects/seattlerb/• http://docs.seattlerb.org/heckle/


Heckle Mutations

• Booleans• Numbers• Strings• Symbols• Ranges• Regexes• Branches (if, while, unless, until)


Humbug

• PHP 5.4 or greater• PHPUnit• Usually run from the command-line• OK documentation• Active project

• https://github.com/padraic/humbug


NinjaTurtles

• .Net:• C#• Visual Basic/VB.NET• Any other .NET language code

• Doesn't seem much alive

• http://www.mutation-testing.net/


NinjaTurtles Mutations

• Sequence point deletion• Arithmetic operator substitution (*, /, +, -, %)• Bitwise operator substitution (&, |, ^)• Branch substituion (condition, always and never

branch)• Conditional boundary substition (< and <=, >

and >=)• Substitution of reads from variables,

parameters and fields of the same type• Substitution of writes to variables of the same

type


Personal Experiencesand Recommendations


Experiences and Recommendations

• Use mutation testing from day 1• Start on a small code base

• Keep things simple• Have small interfaces

• Select a good tool• Configurable• Flexible• One that can output the mutant


Experiences and Recommendations (cont'd)

• Believe the tool• Or try to proof that the tool is wrong

• Fix the problem• Don't turn mutation testing off

• Embrace the “more than 100%” test coverage• Semantic coverage• Less code• More unit tests• More intelligent unit tests


Improvements


Improvements

• Integration with more unit testing frameworks• Better unit test–source code mapping

• Better heuristics• Parallellisation• Better reporting• IDE integration• Building tool integration

Date post:	17-Jul-2015
Category:	Software
Upload:	filip-van-laenen
View:	389 times
Download:	0 times