Monster Builds and How to Tame Them

Chris Mountfordchris@atlassian.com

1

Source Code

Working Software Build

2

x

Truck Build

Small

Monsteryouare here

3

The Monster Build Threat• Shipping

• Unit Testing

• Integration Testing

• Functional Testing

• Selenium Testing

• Synergy Harnessing

4

5

Just JIRA

6

Just JIRA(not the biggest)

7

Multi

PolandHosted

PluginsJust JIRA

8

Just JIRA

Multi

PolandHosted

Plugins

9

Wallboards

10

Build Duration• release build took 30-60 CPU hours

• supported platform matrix combinations

• 3.5 hour functional test runs

JIRA11

Long Builds Cause Low Velocity

• Context switching overhead

• Task entanglement

• Build breakage confusion

• Agile?

12

13

•optimization

•matrix culling

•maximize test power

•minimize wait time

Taming Monster Build Times

14

15

• measure first

• reduce network & disk

• RAM disks, proxies, colocation

• cut rework

• measure again

Optimization

16

•smarter admin navigation

•20% speedup

Example: JIRA Func Tests

17

•browsers

•operating systems

•databases

•libraries

Matrix Culling

18

• user based editions

• dropped ancient app servers

• dropped java 1.4

• release down to 18 hours

Example: JIRA Matrix Culling

19

Power to Wait Ratio

unit tests

functional tests

20

Be Selectively Continuous•common customer configurations

•branch vs trunk

•nightly builds

•sunday builds

21

testsfeatures

platformsbrowsersversions

ajax22

23

24

Herb Sutter http://www.gotw.ca/publications/concurrency-ddj.htm

25

26

27

Parallelism

28

Parallelism

TestingCheckout,Compile

time

agent

29

ParallelismTesting

time

agent

agentagentagentagentagentagentagentagent

agentagent

Checkout,Compile

30

The Embarrassingly Parallel Problem

vs.The Elastic Batches

of Bamboo!31

Inelegant Yet Working1532

• parametrize your test suite:

• number of batches

• current batch number

• make the suite only run subset of tests in the requested batch

Inelegant Yet Working

33

Inelegant Yet Working

34

Inelegant Yet Working

35

Need MoreComputers!

36

37

38

Reliability39

JIRA

maven 1proxy

maven 2repo

svn

nfs

release buildagent

internet

test dbjava

blogs

bamboo

license server

40

internet

test dbjava

blogs

bamboo

license server

41

$False Negatives

42

43

canary death

44

45

46

47

Maven vs. Bash

48

My technology is better

This ENDS NOW

49

• measure, optimize, measure

• use power to wait ratio

• elastic parallelization

• canary builds measure reliability

• careful tool evaluation

Monster Counter-tactics

50

Eternal Vigilance

51

chris@atlassian.com

Chris MountfordJIRA Dude

52