Mutants, tests and zombies

Alexander Todorovhttp://atodorov.org

@atodorov_TestCon Moscow 2018

S

Is my test suite good enough

?

What is a mutation ?

- if age > 18 then

+ if age < 18 then

buy(beer)

Федеральный закон о государственном регулировании

производства и оборота этилового спирта, алкогольной

и спиртосодержащей продукции says 18+

- had_lunch = True

+ had_lunch = False

while True do

- break

+ continue

Definition

A mutation is a controlled change in the SUT

which produces a valid syntax but changes

program behavior!

Mutation testing can be used to evaluate the

quality of existing test suites!

It can find missing or inadequate tests or dead

code!

Algorithm for mutation testing

for operator in mutation-operators:

for site in operator.sites(code):

for mutant in operator.mutate(site):

run_tests(mutant)

https://github.com/sixty-north/cosmic-ray

3 rules to kill mutants !

run_tests(vanilla_code) MUST PASS

if run_tests(mutated_code) == FAIL:

mutant dies

if run_tests(mutated_code) == PASS:

zombie

class Desk < ApplicationRecord

... skip ...

def upcase_language_code

language_code&.upcase

end

end

@@ -1,4 +1,4 @@

def upcase_language_code

- language_code&.upcase

+ nil

end

еxpect(

desk.upcase_language_code

).to_not be_nil

@@ -1,4 +1,4 @@

def upcase_language_code

- language_code&.upcase

+ self

end

еxpect(

desk.upcase_language_code

).to be_instance_of(String)

@@ -1,4 +1,4 @@

def upcase_language_code

- language_code&.upcase

+ language_code

end

desk.language_code = 'ru'

expect(

desk.upcase_language_code

).to eq('RU')

@@ -1,4 +1,4 @@

def upcase_language_code

- language_code&.upcase

+ language_code.upcase

end

desk.language_code = nil

expect(

desk.upcase_language_code

).to be_nil

Is my test suite good enough

?

http://kaner.com/pdfs/negligence_and_testing_coverage.pdf

Mutation testing vs. coverage

●http://bit.ly/GTAC2015mutation

●http://bit.ly/GTAC2016coverage

●http://bit.ly/MutationVsCoverage

SUT: pelican-ab, v0.2.1

AB_EXPERIMENT="control" make github

AB_EXPERIMENT="123" make github

AB_EXPERIMENT="xy" make github

●100% branch & mutation coverage

●Bug when

DELETE_OUTPUT_DIRECTORY=True

SUT: pelican-ab, v0.2.1

●Recursion loop bug when class is inherited

- super(self.__class__, self).....

+ super(PelicanAbExperimentWriter, self)....

●See Pylint #1109 for more info

100 % of any metric

is useless

How slow is it ?

https://github.com/rhinstaller/pykickstart/

●Full mutation test run: > 6 days

●After divide and conquer & fail fast: 6.5 hrs

When to use

mutation testing ?

Mutation testing tools

●Python – sixty-north/cosmic-ray

●Ruby – mbj/mutant

●Java – hcoles/pitest

●JavaScript - stryker-mutator/stryker

●PHP - padraic/humbug

●LLVM based: C, C++, Rust, etc – mull-

project/mull

https://mutation-testing.slack.com/

СПАСИБО