WHO WILL TEST YOUR TESTS ?

Yahya Poonawala @meetykp Priti Biyani @pritibiyani

2 https://flic.kr/p/9uZhDS 2

TESTS Code works

Reduce Bugs

Reduce cost of change

Allow Refactoring

Faster development

Reduce fear

3

NOW WHEN YOU HEAR ….

4

Oh, it failed?

Don’t worry. Just re-trigger it!

5

Oh, it failed?

Strange; it passed on my machine!

6

Oh, it failed?

Run it on agent 007! It’s an agent issue!

7

Oh, it failed?

I know. It fails at 12AM!

8

9

Now, again,

WHY DO WE WRITE TESTS?

10

Tests prove that the code actually works

Tests prove that the code sometimes works

11

Tests reduce bugs

Tests itself might have bugs

12

Tests reduce the cost of change

Change? Are you kidding?

13

Tests helps in faster development.

Testing is slowing me down L

14

NON-DETERMINISTIC

TESTS

A.K.A “Flaky Tests”

15

16 http://www.trovatten.com/long-game/

17

18

http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg

19

ILLUSION OF SUCCESS!

20

ILLUSION OF FAILURE!

21

22

23

DO NOT DELETE YOUR FLAKY TESTS!!

24

CAUSES OF FLAKINESS

25

Thread 1:: !" " if( name is not Yahya ) { !" " " " " "… !" " " " " "fputs( name, file ) !" " " " " "… !" " " " "} !!Thread 2:: !" " Initialize name to Yahya!!

CONCURRENCY

27

Initialize lock; !!Thread 1:: !" " Acquire lock !" " " " "if( name is not Yahya) { !" " " " " ". . . !" " " " " "fputs( name, file ) !" " " " " ". . . !" " " " "} !" " " " "Release lock !!Thread 2:: !" " Acquire lock !" " " " " Initialize name to"Yahya!" " " " " Release lock !!!

TIME

29

http://imgs.xkcd.com/comics/bug.png

EXTERNAL SYSTEM CALLS

30

UNORDERED COLLECTIONS

31

32 https://www.vardot.com/sites/default/files/2.jpg

! !# pseudo-code ! !!! !makeAsyncCall; ! !sleep(2000); ! !readResponse; !

ASYNCHRONOUS WAITING

34

!!"make asynchronous call; !!"While (response received?) !" "if timeout then !" " " "throw Test Timeout Exception !"!" else !" " "sleep(10) ! . . . !!"read response !!!

RESOURCE LEAKS

36

# //pseudo-code !FILE_NAME = “content.txt”!!// Test 1:!!"FileReader.read(FILE_NAME).to be empty !!// Test2: Write to the file!!"data = “Priti” !"FileWriter.write(FILE_NAME, data) !"FileReader.read(FILE_NAME) == data !!// Test 3: Updates the file!!"append_data = “Biyani” !"complete_data = “Priti Biyani” ! FileWrite.append(FILE_NAME, append_data) ! FileReader.read(FILE_NAME) == complete_data !!

TEST ORDER DEPENDENCY

38

INPUT / OUTPUT

39

40

WHAT’S YOUR STRATEGY?

41

STOP CALLING YOUR BUILD FLAKY!!

42

Identify Flaky tests

Quarantine

Plan Fix Tests

Monitor

43

IT’S NOT ALWAYS YOUR TESTS

44

DO NOT GET OVERWHELMED BY FAILING TESTS

45

REMEMBER!

46

LOVE YOUR TEST CODE !

47

There is no such thing as FLAKY TEST!

If its flaky, its not a test ! 48

AND THE FUTURE…

49

REFERENCES • An Empirical Analysis of Flaky tests •  Eradicating Non-Determinism in Tests – Martin

Fowler

• No more flaky tests on the GO team - Pavan Sudarshan

•  Top 12 reasons to write unit tests

• http://tech.just-eat.com/2015/03/30/how-to-fix-flaky-tests/

50

THANK YOU yahya.poonawala@gmail.com

priti.biyani6@gmail.com