Tests antipatterns

transcript

TestsAntipatterns

by Maciej Przewoźnik

AgendaSlow TestOveruse of mocksObscure Test

Agenda (2)Manual TestFragile TestErratic TestEager Test

Agenda (3)Verbose TestMystery GuestTest Code DuplicationHigh Test Maintenance Cost

Slow Test

Slow TestStabilizing a very slow testIndividual slow testsExecuting all tests is too long

Slow Test - stabilizing a very slowtest

Don't steer a rover on Mars from EarthExecute a test ...8 minutes later: failureFix & rerun10 minutes later: failureFix & rerun10 minutes later: failure

What to do in the meantime?GTD: don't multitaskCPU cache vs "brain cache"Context switch cost

What to do in the meantime?Solution: use the fastest machineHave the machine in the companyOther tests on slower machines

What to do in the meantime?For performance or stress tests with bigamount of dataSolution: use smaller data sets forstabilization firstThen run with bigger datasets

Solution: use REPL environment tostabilize it fasterReal-eval-print loopA failure shouldn't invalidate previous results!

Solution: stabilize parts separatelyGiven a test:

Generate a big file.Consume it.

Don't remove the file if failure occurs in (2).(to be continued)

Slow Test - individual tests areslow

What is a slow test?a developer gets irritated or bored waiting forfinishis tempted to get a break, surf the Internet,chat, or walk around

What is a slow test?Sometimes more than a second, sometimes amillisecondNot long individually, but all tests execution istoo longOr a group of tests is too long

Impact:Developers stop running them after every codechangeDevelopers wait for a coffee break, lunch or ameeting to run themDelayed feedback, loss of "flow"Decreases productivity and job satisfaction

Cause: Slow dependent component (1)Solution: Fast Fake ObjectsReal DB → Fake DB (e. g. In Memory DB)Real Cloud → In memory cloudFilesystems: HDD, SSD → RAM disk:

mount t tmpfs o size=200m tmpfs /mnt/tmp

Cause: Slow dependent component (2)Example: Backup applicationsSolution: Recorded testRecord operations done by backupapplicationsWarning: Recorded test may be a Fragile Test

Cause: Slow environment or layerSolution: Separate tested logic fromenvironment:

Hardware/GUI/HTTP layerThreads, processesClouds, etc.Kernel code

Cause: Over-engineered fixtureLong setup of many complicated objectsEven if most of them are not neededOr using full product in testsSolution: General Fixture → MinimalFixture

Cause: Manual testsSlow by definitionEven slower if repeated"Please check that the system just works"before a releaseAutomated tests may get written anyway later

Cause: Manual testsOften under pressure or under "give-it-to-me-now driven-development"No future returnsMay be worthless after just a single commitLack of tests automation will slow testdevelopment later

"Just Say No to More End-to-End Tests"Testing pyramid

Manual tests

End-to-end tests

Integration tests

Component tests

Fast Unit Tests

Slow Tests - executing all tests istoo long

In some big IT companies:build time: one minuterelease to production: 8 minutesnot just a release – a release to production!

What problems may it address?

Impact: Developers context-switching,worse productivity

tests start ...15 min: a test failsfix & rerun20 min: a test fails

Impact: Integrating code slowerBigger release queueSlower features delivery, missing deadlines

Impact: Production big fixes aredelayed

A customer requires a fix, we need to test itRelease a quick fix or a well tested fix?

Cause: Too much overlap betweentests

Test1: [ DATA LOADING ][doSth][check]

Cause: Too much overlap betweentests

Use Shared Fixture?Shared setup for many testsWarning: may get closer to Erratic Tests,

High Test Maintenance Cost

Cause: Too Many TestsToo many tests run too frequentlySolution: subsets of tests can be run atthe timeAll tests within larger period of timePrerelease tests: all with fast fakesPostrelease tests: with real objects

Cause: Too Many TestsThe system is too largeSolution: break it into independentsubsystems

Cause: Slow MachinesOnly one integration serverToo slow integration serversInsufficient parallelization: look at "top","vmstat"

Overuse of (strict)mocks

Overuse of mocksImpact:Tests can be harder tounderstand

The extra code detracts from the tester authorsintent

Overuse of mocksImpact: Tests are harder to maintain

Focus: implementation vs public interfaceFocus: implementation vs behaviorExtra code for mock behavior neededOver-specified softwareFragile

Overuse of mocksImpact: Less assurance that the codeis working properly

It's hard to guarantee that the mocks behave asreal implementationsEspecially over time

Overuse of mocksWhen the real object cannot be used:

They are too slowTheir setup is too complexThey charge money, etc.

Overuse of mocksAlternatives:

Fake objectsHermetic local servers (for cash transactions,etc.)

Consider thefollowing bad code ...

class TeaHouse(object): def __init__(self, cash_service, tax_service): self.cash_service = cash_service self.tax_service = tax_service self.money_total = 0

def order_cup(self, credit_card): cup = Tea() self.cash_service.begin_transaction() result = self.cash_service.charge( credit_card, cup.price) if result: self.money_total += cup.price self.cash_service.commit_transaction() self.tax_service.register_transaction( cup.price ) else: self.cash_service.cancel_transaction()

def should_compute_doubled_price_for_two_cups(): cs = mock(CashService) cs.expect_call("begin_transaction").times(2) cs.expect_call("charge").times(2).repeatedly( Return(Cup()) ) cs.expect_call("end_transaction").times(2) ts.mock(TaxService) ts.expect_call("register_transaction").times(2) t = TeaHouse(cs, ts) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

Overuse of mocksWe actually don't care for interaction with cashservice and tax service here

Overuse of mocksAlternative: use a fake

def should_compute_doubled_price_for_two_cups(): t = TeaHouse( FakeInMemoryCashService(), FakeTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

Overuse of mocksAlternative: use a stub or a non-strictmock:

def should_compute_doubled_price_for_two_cups(): t = TeaHouse( StubCashService(always_ok=True), StubTaxService() ) c = CreditCard(money=usd(30)) t.set_tea_price(usd(10)) t.order_cup(c) t.order_cup(c) assert t.money_total == usd(20)

But in this particularsituation, even better

is ...

One function, oneresponsibility

Overuse of mocksclass TeaHouseOrders: def __init__(self): # no cash_service and no tax service! self.orders = [] def order_cup(credit_card): self.orders.append(TeaOrder()) return def total_price(): return sum([o.price() for o in self.orders])

Overuse of mocksclass TeaHouseCash: def charge(price): self.cash_service.begin_transaction() result = self.cash_service.charge(cc, price) if result: self.cash_service.commit_transaction() else: self.cancel_transaction() return self.cups

Overuse of mocksdef should_compute_doubled_price_for_two_cups(): t = TeaHouseOrders() t.set_tea_price(usd(10)) t.order_cup() t.order_cup() assert t.total_price() == usd(20)

Neither mocks nor fakes needed!Important concept: separate descriptionfrom execution

Obscure Test

Obscure TestImpact:

Harder to understandHarder to maintainDevelopers will not read them asdocumentationMay lead to Can result in a Buggy Test, and

then Production Bugs

Obscure TestGood Tests:

Tests should act as documentationAnd also a self-verifying executablespecification

Obscure TestIs it a good test?

def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw") f = File("a.txt", "rwxrwxrwx") fso = FileSystemOps(fs) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

Obscure TestIs it a good test?

def should_have_the_same_permissions_after_copy(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))

Obscure TestCause: Irrelevant information

which values affect the outcome?There can be a hundred constants in a testcode

Obscure TestCause: Too much informationSolution: introduce higher-level testlanguage

def file_copy_permissions_should_be_as_original(): given_a_filesystem() given_a_file_with_permissions("rwxrwxrwx") when_a_file_is_copied() then_the_new_file_has_permissions("rwxrwxrwx")

Obscure TestCause: Too much information, EagerTest

The test verifies too much functionality in asingleTest Method and therefore is hard tounderstand

def test_my_filesystem(): fs = FileSystem("rw", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000) assert(fs.isReadOnly() == false) f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz", "20161212", "20161212", "20161212") assert(f.getPermissions() == "rwxrwxrwx") assert(f.getCreationDate() == "20161212") fso = FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "b.txt") assert(fso.fileperm("a.txt") == fso.fileperm("b.txt")) assert(fso.size("a.txt") == fso.size("b.txt")

Often a Fragile TestAnd a High-Maintenance Test

Obscure TestCause: Too much information

Not caring about clean codeNo refactoring"These are only tests""just do it in-line" mentality

Obscure TestCause: Too little information – MysteryGuest

Cause and effect not clearSomething is hidden from the Test Method

Obscure TestCause: Too little information – MysteryGuestDoesn't exist because is copied to root?

def test123(): fs = createFileSystem() f = new File("a.txt", "rwxrwxrwx", 64*MB, "xyzxyz", "20161212", "20161212", "20161212" ) fso = new FileSystemOps(fs, true, false, 1024, 11) fso.copy(f, "/b.txt") assert(fso.exists("b.txt") == false)

Obscure TestCause: Too little information – MysteryGuestNo, because createFileSystem creates read-onlyfilesystem:

def createFileSystem(): return new FileSystem( "ro", "noauto", 620, false, true, true, 8096, 10*1024*1024*1024, 1000000 ))

Obscure TestCause: Setup-action-verify phases notclear

def test123(): fs = createFilesystem() f = createFile("a.txt") fso = createFso() f2 = fso.copy(f, "b.txt") s1 = fso.size("a.txt") s2 = fso.size("b.txt") assert(s1 == s2)

Do we test "size" here or "copy"?Solution: Given-when-then

Obscure TestCause: Setup-action-verify phases notclearSolution: Given-when-then:

given_filesystem() given_file() given_file_system_operations() when_the_file_is_copied() then_the_new_file_size_is_as_original()

Obscure TestCause: Overcomplicated Fixture

class TestFSFixture: def createRealOnlyFS(): def createRWFS(): def createJournalingFS(): def createTransactionalFS(): def createNFS(): def createHDFS(): def createNoAcFs(): def createTwoFilesystems(): def createPerfOptFS(): def createBuggyFS(): def createSlowFS():

Obscure TestCause: Overcomplicated FixtureThe fixture is too big and too complexTo understand the test, reading the fixture maybe necessarySolutions:

Divide a General Fixture into smaller FixturesUse Minimal FixtureUse Facades

Obscure TestCause: Indirect Testing

Testing one component through othercomponentsExample: Execute a product in debug mode tosee if log rotation worksRoot Cause: software not designed fortestability

SourcesGerard Meszaros - xUnit Test Patternstesting.googleblog.comPaul Chiusano and Rúnar Bjarnason - FunctionalProgramming in Scala

THE END

Tests antipatterns

Technology