Large Scale Continuous Integration

Made by:Erik RomsonZenior

Presented by:Johannes BrodwallLead Software ArchitectBBS

(1)

“Wait a minute…”

Architect?

Reuse and cooperation

«delsystem»Delsystem1

«delsystem»Delsystem2

«delsystem»Delsystem3

«delsystem»Delsystem4

«delsystem»Delsystem5

Release strategy (one code line)

Development

When the code is stable, a branch is made

Trunk

Test Production

Development Test Production

Rel. 1.0.0 Rel. 1.0.1 Rel. 1.0.2 Rel. 1.0.3Rel. 1.0.4

Rel. 2.0.1

Ends when the new version goes into production

Principle: Everyone should be on one build server

(2)

Consequences

Size

0

50

100

150

200

250

300

350

400

450

500

Novem

ber

Desem

ber

Janu

ar

Febr

uar

Mar

sApr

il

Module count

KLOC (estimate)

Developers(estimate)

Build time

0

5

10

15

20

25

30

35

Novem

ber

Desem

ber

Janu

ar

Febr

uar

Mar

sApr

il

MinimumTypiskMaksimum

Error notification is delayed

TrunkCheckinCheckin CheckinCheckinCheckinCheckinCheckinCheckin

Skyldig

CheckinCheckinCheckinCheckin

Ändringar inkl. fel

Brutet byggBad checkin Buildstarts

Build breaks

Many developers

CruiseControlContinuous

BuildCVS

checks incode

all changes

checks incode

checks incode

checks incode

checks incode

Whodoneit?10 suspects

TrunkCheckinCheckin CheckinCheckinCheckinCheckinCheckinCheckin

Skyldig

CheckinCheckinCheckinCheckin

Ändringar inkl. fel

Brutet byggBad checkin Buildstarts

Build breaks

(3)

How does it look?

(disclaimer)

20:04 4,09m

20:15 6,15m

20:24 7,29m

20:35 8,37m

21:28 11,36m

21:53 10,41m

22:12 12,05m

(4)

How does it work?

CruiseControlContinuous

Build

CruiseControlIntegration

Build

CruiseControlDaily Build

CVS

checks incode

all changes

Uses compileand tested code

Triggers build

Installscompiled andtested code

Repository

Update tested and compiled code from repository

daily

Test ServerDeploys

applicationand run tests

foo depends onbar-1.1

Maven

Continuous build: Quickly after checkin

(should’ve caught all mistakes)

Integration build: In due time

(creates (and deploys some) EARs etc)

Daily build: Nightly

(generates project-sites)

CruiseControlContinuous

Build

CruiseControlIntegration

Build

CruiseControlDaily Build

CVS

checks incode

all changes

Uses compileand tested code

Triggers build

Installscompiled andtested code

Repository

Update tested and compiled code from repository

daily

Test ServerDeploys

applicationand run tests

Technology: Maven + Cruisecontrol + massive amounts of custom code

Implementation:Cruisecontrol triggers a custom maven plugin which uses beanshell to access a custom Java API that executes Maven goals for modules

(I think)

(5)

Lessons learned

What is hard?

Somebody has to care about the build time!

Culture for following up when the build breaks

This one is mine. I forgot to check in a file.

But sometimes, things go wrong…

Anyone looking into this one?

Could xxx please look here ASAP!

He’s gone home I will check it out.

Notification mails must be easy to interpret

Many dependencies

Buildserver

CVS Server

Mailserver

Databas

Artifactrepository

Refactoring central modules is expensive

(but doable, even in a system with several 100 KLOC)

Don’t leave the build red while you fix your problems

CVS update frequently!

Protect yourself with unit tests

(6)

The road goes ever on

Domain Driven Design build server

Module

Dependency

BuildAttempt

ChangeSet

BuildStep

*parent

*dependent

1*

*

(cruisecontrol = cron on steroids)

Smaller units of reporting

Concurrent build

Build farm

Buildconfiguration

BuildMaster

Build Worker

Build Worker

Build WorkerSource repository

Artifact repository

1. calculatechangeset

2. takework unit

3. get latest

4. publish artifact

Better visibility

Thank you for the attention

Johannes Brodwall<johannes.brodwall@bbs.no>