From jUnit to Mutationtesting

prepare for…

Mutation Testing..from jUnit to Mutation-Testing

@SvenRuppert has been coding java since 1996

Fellow / Head of R&D

reply Group

Germany - Munich

2


3


Projects in the field of:•Automobile-industry•Energy•Finance / Leasing•Space- Satellit-•Government / UN / World-bank

Where?

•Europe•Asia - from India up to Malaysia

3

4

Save harbor statement

4

Save harbor statement

The following is intended for information purposes only. Ican not be held responsible for the overuse of effects andanimations in this presentation. If any person in this roomhas a medical condition that is triggered by fast movingobjects on the screen and/or explosions, he/she shouldprobably better leave now…

(I got carried away by the topic.)

5

The Environment @SvenRuppert

Codebase is > 13 years old

5


Codebase is > 13 years old no test coverage

5



no dedicated refactoring budget

5



no dedicated refactoring budgetdecrease complexity

5



no dedicated refactoring budgetdecrease complexity

but… lets start with the basics

6

TDD with jUnit @SvenRuppert

6


are you using jUnit?

6



assume that the following would make sense.. ;-)

6




public class Service { public int add(int a, int b){ if(a<2){ return (a+b) * -1; } else { return a+b; } } }

6





How many tests

you will need ?

6





How many tests

you will need ?

it depends ;-)

7


public class Service { public int add(int a, int b){ if(a<2){ return (a+b) * -1; } else { return a+b; } }}

How many tests

you will need ?

it depends ;-)

7



How many tests

you will need ?

it depends ;-)

for line 100% coverage

7



How many tests

you will need ?

it depends ;-)

for line 100% coverage 2

7



How many tests

you will need ?

it depends ;-)


but will this be enough?

7



How many tests

you will need ?

it depends ;-)


but will this be enough? No

8



How many tests

you will need ?



it depends ;-)

8



How many tests

you will need ?



how to find out, what will be enough?

it depends ;-)

8



How many tests

you will need ?



how to find out, what will be enough?how to find the right tests?

it depends ;-)

9



How many tests

you will need ?

9



How many tests

you will need ?

@Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); }

9



How many tests

you will need ?

@Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); } @Test public void testAdd002() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); }

10

Mutation Testing @SvenRuppert

10


Mutation Testing is a structural testing method

10



we want to find a way to write "good" tests

10



we want to find a way to write "good" testshow to find "good" tests?

10




let the machine find the targets

10




let the machine find the targets

let´s mutate it... but how?

11

Mutation Testing - the Idea @SvenRuppert

11


a mutation is a small change in the code

11


a mutation is a small change in the code.. small enough to be a small defect

11



P will be the program

11



P will be the programT will be the collection of all tests / Test Suite

12



12



we will create a sequence of mutations / P1,P2,P3...

12




.. Px will have only one mutation compared to P

12





running all tests from T against Px

12






green: T will kill the mutation

12






green: T will kill the mutation.. at leased one test from T will fail

12






green: T will kill the mutation.. at leased one test from T will fail

red: if all tests are green

13


13


if we kill k out of n mutants

13


if we kill k out of n mutants -> we are not good enough ;-)

13



we are perfect enough if we are reaching : k == n

13




how to create all versions of Px ?

13





.. the good thing..

13





.. the good thing..

we could almost generate/

automate everything

14


practical TDD with Mutation Testing

14


generating the mutants and


14




running all junit tests

14




running all junit testscheck the reports

14





write more / better tests

14





write more / better tests

loop until quality target reached

15


mutants are a good approach / model to estimate the default rate of defects per 1k lines of the P

15



estimate that:

15



estimate that: the defects are independent

15



estimate that: the defects are independent normaly ;-)

16


16


no need to know that Mutation Testing will be done, independend creation of T

16



for K==n we need a high number of mutants ( P1, P2, .., Px)

16




.. mostly it will lead into exponential numbers of Px

16





.. how to find YOUR barrier you

have to reach?

16





.. how to find YOUR barrier you

have to reach?

but.. what is a mutation?

17

Mutation Testing - Kinds of Mutation @SvenRuppert


Value Mutationchanging constants,

loop bounds (adding/subtracting values)

18



Value Mutation

Decision Mutation for example < will be changed to <=

19



Value Mutation Decision Mutation

for example swapping/deleting/duplicating lines of code

Statement Mutation

20



Value Mutation Decision MutationStatement Mutation

20




for Java you could think about more language spec. mutations

20





.. changing modifiers

20





.. changing modifiers.. changing between static / non static

20






.. delete member initialization

20






.. delete member initialization .. delete this.

20






.. delete member initialization .. delete this... argument order change

21

Mutation Testing - in short words @SvenRuppert

21


mutation testing is an add on to normal jUnit TDD

21



tools are supporting it well

21




generating and running all tests are time consuming

21





but most important

21





but most important

will effect your project structure

22

Mutation Testing - Frameworks @SvenRuppert

22


muJava

22


muJava2003. First released as JMutation (Java Mutation System).2004. The name was changed to MuJava (Mutation System for Java).2005. Software Copyright Registration, ALL RIGHTS RESERVED.2005. Version 2 released with several fault fixes and modified mutation operators.2008. Version 3 released with minimal support for Java 1.5 and 1.6.2013. Version 4 released to support JUnit tests and Java 1.6 language features, including generics, annotations, enumerations, varargs, enhanced for-each loops, and static imports.2015. Additional and improved error messages. Bug fixes for OpenJava. Licensing changed to the Apache license.

22



https://cs.gmu.edu/~offutt/mujava/

https://github.com/jeffoffutt/muJava/graphs/contributors

22



https://cs.gmu.edu/~offutt/mujava/

https://github.com/jeffoffutt/muJava/graphs/contributors

inactive

23


23


2012. started around 2012 with a small codebase.2014. very active since 2014

23



http://pitest.org/

23



http://pitest.org/

active ;-)

24

Mutation Testing - Hello World @SvenRuppert

http://pitest.org/

24


http://pitest.org/assume the following would make sense ;-)

24


http://pitest.org/assume the following would make sense ;-)

public class Service { public int add(int a, int b){ if (a<2) { return (a+b) * -1; } else { return a+b; } }}

25



25


public class Service { public int add(int a, int b){ if (a<2) { return (a+b) * -1; } else { return a+b; } }} how many test you will need for..

25



100% Line Coverage… and…

25



100% Line Coverage… and… to be save ?

25




2 for Line Coverage

25




2 for Line Coverage we will see ;-)

26




26




we have one if statement

26




we have one if statement with an else branch

26




we have one if statement with an else branch

this will lead to 2 jUnit Tests to get 100 %

27


public class Service { public int add(int a, int b){ if (a<2) { return (a+b) * -1; } else { return a+b; } }} 100% Line Coverage… and…

27


public class Service { public int add(int a, int b){ if (a<2) { return (a+b) * -1; } else { return a+b; } }} 100% Line Coverage… and…@Testpublic void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0);}

27


public class Service { public int add(int a, int b){ if (a<2) { return (a+b) * -1; } else { return a+b; } }} 100% Line Coverage… and…@Testpublic void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0);} @Test

public void testAdd002() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3);}

28


final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0);


28




28




29


final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3);

29



we got 100% Line Coverage

29



we got 100% Line CoverageHow good these tests are?

29




How to measure if these test are good?

29




How to measure if these test are good?

How to find the good tests?

30


final int add = new Service().add(0, 0);


How to find the good tests?

30




How to find the good tests?let´s generate a the mutation report

30




How to find the good tests?let´s generate a the mutation reportwith maven : pitest: mutationCoverage

30





>> Generated 54 mutations

30





>> Generated 54 mutations Killed 3 (6%)

30






org.pitest……mutators.ConditionalsBoundaryMutator

Killed 3 (6%)

30






org.pitest……mutators.ConditionalsBoundaryMutatororg.pitest……mutators.IncrementsMutator

Killed 3 (6%)

30






org.pitest……mutators.ConditionalsBoundaryMutatororg.pitest……mutators.IncrementsMutatororg.pitest……mutators.ReturnValsMutator

Killed 3 (6%)

30






org.pitest……mutators.ConditionalsBoundaryMutatororg.pitest……mutators.IncrementsMutatororg.pitest……mutators.ReturnValsMutatororg.pitest……mutators.MathMutator

Killed 3 (6%)

30






org.pitest……mutators.ConditionalsBoundaryMutatororg.pitest……mutators.IncrementsMutatororg.pitest……mutators.ReturnValsMutatororg.pitest……mutators.MathMutatororg.pitest……mutators.NegateConditionalsMutator

Killed 3 (6%)

31


final int add = new Service().add(0, 0);final int add = new Service().add(3, 0);


31




31




32



>> Generated 54 mutations Killed 3

32




32




Killed 3

33




Killed 3

33



>> Generated 54 mutationsfinal int add = new Service().add(2, 0);

33




Killed 4

33




Killed 4

33




Killed 4

33




Killed 4

33




Killed 4

33




Killed 4

33



Killed 4

34




Killed 4

34




34




Killed 5

34




Killed 5

34




killed 9:1

Killed 5

35




killed 9:1

Killed 5

35




killed 9:1


35




killed 9:1


Killed 6

35




killed 9:1


Killed 6

35




killed 9:1


Killed 6

35




killed 9:1


Killed 6

35



final int add = new Service().add(2, 0);final int add = new Service().add(1, 1);final int add = new Service().add(2, 2);

killed 11:1

Killed 6

35




killed 11:1

Killed 6

36




Killed 6

36




37

Mutation Testing - Lesson Learned @SvenRuppert

37


mutation tests are often leading to

37



…cleaner code compared to jUnit only

37



…cleaner code compared to jUnit only… smaller modules (shorter mutation runtime)

37



…cleaner code compared to jUnit only… smaller modules (shorter mutation runtime)

and something nice… helps to find useless code

38

Example of useless Code @SvenRuppert

38


38


39

Mutation Testing - How to start @SvenRuppert

39


you need jUnit - to generate the reference

39


you need jUnit - to generate the reference add the pitest-plugin to the build section

39



configure the plugin

39



configure the plugingenerate the reference -> clean , install

39




run pitest: mutationCoverage

39




run pitest: mutationCoveragereport will be under target/pit-reports

40


pom.xml - example - build<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats> <targetClasses> <param>org.rapidpm.*</param> </targetClasses> <targetTests> <param>org.rapidpm.*</param> <param>junit.org.rapidpm.*</param> </targetTests> </configuration> </plugin>

41


pom.xml - example - reporting<reporting> <plugins> <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <reportSets> <reportSet> <reports> <report>report</report> </reports> </reportSet> </reportSets> </plugin> </plugins> </reporting>

42

Mutation Testing - practical usage @SvenRuppert

42


Start with some tests

42


Start with some testsgenerate the pitest report

42



write more tests to kill mutations

42



write more tests to kill mutationsif you have time, eliminate useless tests

42




do it one by one

42


Mutation 001Mutation 002

Mutation 003Mutation 004 Survived

SurvivedSurvived

Survived



do it one by one

42


Mutation 001Mutation 002

Mutation 003Mutation 004 Survived

Killed

SurvivedSurvived

SurvivedKilled

KilledKilled



do it one by one

Summary

43

@SvenRuppert

If you are interested…

have a look at GITHUB

ProxyBuilder

Dynamic-Dependency-Injection

Java-Microservice

or contact me ;-) @SvenRuppert

Summary

43

@SvenRuppert

If you are interested…

have a look at GITHUB

ProxyBuilder

Dynamic-Dependency-Injection

Java-Microservice

or contact me ;-) @SvenRuppert

Thank You !!!

Date post:	13-Apr-2017
Category:	Software
Upload:	sven-ruppert
View:	208 times
Download:	0 times