Mutation-Based Testing

Mutation-Based TestingDavid PryorMutation-Based TestingSame basic goal as Code CoverageEvaluate the testsDetermine how much code exercisedMutation testing goes beyond checking which lines of code were executedGoal: Distinguish statements that are simply executed from those that are fully testedHistoryTheory published in 1971 by a studentComputationally InfeasibleTechnological advances: 90s presentStill not mainstream, but gaining popularity2Basic ProcedureRequirementsComplete working programTests written and passingMake a small change(mutation) to the source code of the programRun the testsIf a test fails:The test killed this mutantIf all tests still pass:Redundant/Unnecessary codeTests are incomplete

3Example test functionsThis test achieves 100% code coveragetestAbs1 will never failEven if abs(-3) == -3Mutation testing can detect thistestAbs1 will initially passIt will still pass on mutated codeIt failed to kill the mutantThe test is inadequate

4Example test functionstestAbs2 passes initiallyOn the mutated function, it failsabs(-5) != 5Test is supposed to failIt shows the test is robust enough to catch errors such as this mutationIt killed the mutant

5Mutation OperatorsTypes of mutations to be appliedTypes of changes to make to the codeDefined by the testing frameworkChosen by the userGoalsIntroduce errorsSimulate common coding bugsEnsure the testing of all possible circumstancesTraditional Mutation OperatorsSimpleCommonIncluded in most mutation frameworks6Dropped StatementRemoves a single statement from the programUnnecessary codeNeeds to be selectiveMany possible mutants

7Arithmetic Operator ReplacementSwaps arithmetic operators+, -, *, /, %Some frameworks have set translations+ always becomes *Others allow any/random swaps

8Boolean Relation ReplacementSwaps boolean relations==, !=, =Tightly constrained formOnly mutates to/from similar relations< to to >=== to !=

9Boolean Expression ReplacementReplaces an entire expression with true or falseUnnecessary CodeCode paths that arent sufficiently tested

10Variable ReplacementReplaces a variable reference with a different variable referenceThe new variable must be accessible and defined in the same scopeNot trying to create compiler errorsFalse positivesUnnecessary/Duplicate variables

11Non-Traditional OperatorsLots of Operators out thereBit operation / Binary operatorsShift/Rotate replacementIncrements / DecrementsInvert NegativesShould be able to define your own for customized testingIdeally:Minimize false positivesReasonable number of created mutants12Replace inline constantsReplaces an inline constantNumeric or stringNon-DeterministicTests for responses more than behaviorRequires many tests that may not be helpfulUsed in Heckle(Ruby)

13Object-Oriented OperatorsEncapsulationChanging access modifiersInheritanceHiding variablesMethod overridingParent / super actionsPolymorphismLots of operators hereBasic Idea: change something between the parent and the child in the usage of an object14Concurrency OperatorsModify sleep/wait callsMutual exclusion and semaphoresChange boundaries of the critical sectionChange synchronization detailsSwitch concurrent objectsOthersGoal is the sameEvaluate the adequacy of the tests15Computation TimeThe biggest problem/roadblock with Mutation testingTheory has existed for 40+ yearsComputationally infeasible for use in industry for a long timeFrameworks can automate almost everything, but:Every mutant has to be run against the entire test suiteMany mutants and a large test suite cause immense testing times16Computation Time EstimationT = Time to run test suite against the code baseM = # of Mutation Operators in use (3-20+)N = # of Mutants per Operator (depends on code base size)Mutation testing causes time to increase from T to T*M*NMinimum time increase of a factor of 30For small code base and few operatorsTotal time increases very quicklyOnly time needed to run tests, not compilationIf T = 1 minute, T*M*N can become hours or daysIf T = 1 hour, T*M*N can become weeks or longer

17Addressing Computation TimeNeed to spend less time on mutation testingVariety of methods that fall into three categoriesDo LessTest fewer mutants and mutation operatorsNeed to be carefulFewer may result in poor tests slipping throughDo FasterIncrease the speed of executionDo SmarterEliminate mutants and mutation operators that do not provide meaningful results18Source code or Byte code?Can perform mutations on the source code itself, but:Large code bases result in lots of slow disk readsHave to compile EVERY mutantInstead, compile the original source onceMutate the compiled byte codeMuch fasterCan be difficult to back-trace the byte code to the source code to show the mutants that were created19Weak vs. Strong TestingTwo conditions for killing a mutantTesting data should cause a different program state for the mutant than for the originalFor example, a test results in:valid = falsedone = trueThe difference in state should result in a difference in output and be checked by the testIn this case: the test should check resultWeak Testing: Only satisfy the first conditionStrong Testing: Both conditions

20Weak vs. Strong TestingWeak assures that the tests cause a differenceNot assured that they check the differenceNot as thoroughStrong is idealComputationally expensiveMust always run to the endWeak can stop as soon as it detects a difference in state21Incremental AnalysisCurrently experimentalMost useful for long-term projects with large code bases, which use mutation testing over and overBasic Idea: save state and results of tests and codeOnly re-run those tests and mutants for which relevant code has changedDecide which to skip based on changes madeNot perfected yet can be tricked by odd behavior22Selective MutationGoal: Eliminate some mutants or operators that are not necessaryMutantsRemove duplicates caused by multiple operatorsRemove likely duplicates those that will probably cause duplicate resultsSome amount of error hereMutation OperatorsSome pairs of operators might produce many of the same mutantsAn operator might produce a subset of mutants from a different operatorDetection can be difficult23Coverage Based Test SelectionTypical mutant only affects a single statement / lineTypical test suite has many tests that do not execute this lineNo need to run these tests on the mutantOnly run tests that exercise the mutated code

Optimize the running order of tests24Other Problems and Design ConsiderationsEquivalent MutantsThis mutant is functionally equivalent to the originalNo test that calls this code could ever distinguish the twoSome mutants cant be killedCan sometimes be detected automatically and filtered outNot all can be detectedRequires human effort to determine if equivalentNot always an easy task

25Other Problems and Design ConsiderationsMutant Infinite LoopsSome mutations can cause infinite loops in the mutantsStatement deletion removed the only way out of this loopsSolution: Time the un-mutated codeIf the mutant takes significantly longer than this timeProbably an infinite loopTimeout after the un-mutated time, plus some padding

26Other Problems and Design ConsiderationsComplex BugsMutation only makes small changesWhat if the test cases miss a large, complex error?Mutation doesnt create complex mutantsCoupling EffectHypothesis Tests that detect simple errors are sensitive enough to detect more complex errorsSupported by empirical dataTesting for simple errors helps to find the more complex ones27Fuzz Testing / FuzzingCompletely unrelated to Mutation testingOften confused with MutationInvolves generating random data to use as input to a programTest security/vulnerabilitySee if the program crashesFuzzing modifies inputChecks program behaviorMutation modifies source codeChecks test case resultsDeterministic28Tools and EnvironmentsJavaMuJavaBacterioJavalancheJumblePITJesterC/C++Insure++FortranMothraPHPMutagenesisRubyHeckleMutantC#Nester

29Using Mutation Testing in IndustryUse Mutation from the beginning of a projectDont use it with dangerous methodsLook for high quality tools/environmentsSpeed optimizationsReporting/Coverage informationConfigurable Operators30Benefits of Mutation testingEvaluation of tests / test suiteMore than code coverage: are the tests adequate?Mutation Score: % of non-equivalent mutants killedEvaluation of codeFind unreachable or redundant codeFind bugs that were hidden through inadequate testsFuture: Automatic Test GenerationCreate dummy tests, use Mutation to reviseRepeat until all or most non-equivalent mutants killedStill experimental, but promising31Questions32ReferencesAlexander, R. T., & Bieman, J. M. (2002). Mutation of java objects. IEEE Int. Symp. Software Reliability Engineering, Retrieved from http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdfOffutt, A. J. (n.d.). A practical system for mutation testing: Help for the common programmer. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdfOffutt, A. J., & Untch, R. H. (n.d.). Mutation 2000: Uniting the orthogonal. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdfMa, Y. S., Kwon, Y. R., & Offutt, A. J. (n.d.). Mujava: An automated class mutation system. Retrieved from http://www.cs.gmu.edu/~offutt/rsrch/papers/mujava.pdfBradbury, J. S., Cordy, J. R., & Dingel, J. (n.d.). Mutation operators for concurrent java(j2se 5.0). Retrieved from http://www.irisa.fr/manifestations/2006/Mutation2006/papers/14_Final_version.pdfPit mutation testing. (n.d.). Retrieved from http://pitest.org/ 33