Improving Infrastructure Tools – Introduction to Continuous Integration (work in progress)

Improving Infrastructure Tools – Introduction to Continuous Integration (work in progress)

Mar 2nd, 2009

2

Continuous Integration - Background

• C.I. is the process of integrating work frequently

• Summary of Martin Fowler’s definitionA software development practice where

– developers integrate their work frequently

– usually each person integrates at least daily — leading to multiple integrations per day.

– Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible

• Continuous Integration assumes a high degree of automated tests

3


• Building a Feature with Continuous Integration

(1) Get latest code

(2) Develop feature, regularly running automated build (which runs automated tests)

(3) Update code

(4) Run private build, which runs automated tests

(5) If tests pass then commit changes, else go back to (2)

(6) Do an integration build (aka commit build), which runs automated tests

(7) If integration build succeeds, then you are done, otherwise go back to (2)

4


• The process does not have to be automated to qualify as C.I.

• But obviously automation of the process with a C.I. server is recommended

• A CI server runs an integration build whenever a change is committed to the version control repository.

• CI servers can also be hard-scheduled to build on a regular frequency, such as every night, or every hour

– This is not the same thing as a continuous build and isn't enough for continuous integration – The whole point of C.I. is to find problems as soon as you can. – Nightly builds mean that bugs lie undetected for a whole day.

5

• A Note on the Word "Continuous"

• Using it is technically incorrect

• "Continuous" implies that something kicks off once and never stops.

• This suggests that the process is constantly integrating, which is not the case in even the most intense CI environment.

• It should really be called "continual integration"


6


7

• Essential for C.I.: Keeping the Build Fast

• The whole point of Continuous Integration is to provide rapid feedback.

• Every minute you shave off the build time is a minute saved for each developer every time they commit

• Since CI demands frequent commits, this adds up to a lot of time.

• Most people regard a build that takes an hour totally unacceptable.

• XP guideline = 10 min

• TopOffice build > 45 min – 30 min checkout from Clearcase

– 15 min maven build

– Tests? TBD


8

• Essential for C.I.: Keeping the Build Fast

• Usual bottleneck: testing

• Most crucial step: set up a staged build = multiple builds done in sequence

• Simple example: two stage build

• Commit build has to be fast (XP guideline = 10 min)– does compilation

– runs tests that are more localized unit tests with the database completely stubbed out

• Secondary build (might take 2 hrs to run)– runs a different suite of tests that do hit the real database and involve

more end-to-end behavior.


9

• Simple Example: Two Stage build


• Unit Tests

• Component Tests

• System Tests

• Functional Tests

• Performance Tests

10

• Unit Tests– Verify behavior of small elements, most often a single class.

– Occasionally a unit test touches additional classes because the classes under test are tightly coupled.

– Some unit tests only require as outside dependencies simple classes with shallow object graphs.

– Some unit tests employ mocks

– key aspect: no reliance on outside dependencies such as databases, which increase test set up and execution time.


11

• Unit Tests

As Michael Feathers (http://www.objectmentor.com) puts it:

Unit Test RulzA test is not a unit test if:

• It talks to the database.• It communicates across the network.• It touches the file system.• It can't run correctly at the same time as any of your other unit tests.• You have to do special things to your environment (such as editing config files) to run it.

– Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness.

– However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.


12

• Unit Tests– Have little configuration cost, and the resource cost to run them is

negligible

– A true unit test should run to completion (successfully) in a fraction of a second.

– If a unit test takes longer, take a close look at it—it's either broken, or instead of being a unit test, it is really a component-level test.

– unit tests should be run:• As part of the private build, before check-in

• Each time someone checks in code (commit build)


13

• Component Tests (aka Subsytem Tests, or Integration Tests)

– verify that components in a portion of the system interact to produce the expected aggregate behavior.

– use more dependencies than unit tests, but still not necessarily as many as higher-level system tests

– typically require DB

– may also require file system, network, fully installed system

– don't always exercise a publicly preferable API

– E.g. • While a System test exercises a web app through web pages

• Component test exercises struts action classes: requires DB, but web container is mocked out


14

• Component Tests (aka Subsytem Tests, or Integration Tests)

– have a specific cost to them: dependencies have to be put in place and configured.

– exercise more code than unit tests, so take a bit longer to run

– may only take a few seconds; however, in the aggregate, this time adds up.

– should be run before committing code into a repository (in your private build)

– should be run:• as part of a secondary build, a more "heavyweight“ integration build

that follows the commit build • or periodically

– Some projects with lightweight component tests can get away with running them with every commit build.


15

• System Tests– exercise complete system, so require fully installed system, e.g.

web container and DB

– verify that external interfaces like Web pages, Web service end points, and GUIs work end to end as designed.

– have the tendency for lengthy runtimes in addition to prolonged set-up times

– fundamentally different from functional tests, which test a system much like a client would use the system

– E.g. while functional test would use a browser, a system test would mimic a browser by manipulating the site via HTTP


16

• System Tests– require a fully installed system, so take the longest to run.

– running system tests with every commit build could be a recipe for disaster

– complexity of configuring a fully functional system occasionally limits the full automation of these tests.

– sometimes these types of tests are run with secondary or periodic builds

– Otherwise, nightly (off-hour) runs are good for these tests.


17

• Functional Tests (aka acceptance tests)– test the functionality of an application from the viewpoint of a client,

so they mimic clients.

– frameworks like Selenium actually control a browser and enable it to interact with a Web site.

– Selenium tests are written in tabular forms, which represent a work flow, complete with commands and assertions.

– Should be run like system tests


18

Private build

Commit build

Secondary build

Periodic build

Nightly build

Unit

tests

Yes Yes Yes Yes

Component tests

ideally Yes alternatively Yes

System tests

sometimes sometimes otherwise


When do we run the various types of tests?

19

• Inspections– Testing is dynamic and executes the software in order to test the

functionality.

– Inspection analyzes the code based on a set of predefined rules

– Inspectors (or static and dynamic analysis tools) are directed by identified standards that teams should adhere to


20

• Automated Inspection Tools– Style & Defects: Checkstyle, FindBugs, PMD

– Key metrics: JavaNCSS• Cyclomatic complexity

• Non-commented lines of code

• commented lines of code

• Number of classes, methods, etc

– Code Coverage: Clover, EMMA, Cobertura

– Design quality metrics: Jdepend


21

Private build

Commit build

Secondary build

Periodic build

Nightly build

Coding standards

Maybe Yes Yes

Code coverage

Maybe Yes Yes

Metrics Yes


When do we run the various types of inspections? WIP

Date post:	31-Dec-2015
Category:	Documents
Upload:	lara-simon
View:	33 times
Download:	1 times

Improving Infrastructure Tools – Introduction to Continuous Integration (work in progress)

Documents