Rien van Vugt, Maurice SiteurEurostar Manchester, 23 November 2011

Houston we have a problem

And suddenly the project is Out of Control

2© 2011 Eurostar Vugt-Siteur. All rights reserved.

AGENDA

What is the problem? What do we need? Demo Future

3© 2011 Eurostar Vugt-Siteur. All rights reserved.

Testing seems so easy:We just need to perform X tests in Y amount of time

But when we detect defects:• We need to redo a fair amount of testing => high impact on required capacity• We do not know how many defects we will find !

WHAT IS THE PROBLEM?

© 2011 Capgemini. All rights reserved. 4

Testing seems so easy:We just need to perform X tests in Y amount of time

But when we detect defects:• We need to redo a fair amount of testing => high impact on required capacity• We do not know how many defects we will find !

Just a single hill to climb: or an endless struggle of hill after hill:

• Planning based on Baseline, ETC and Actuals does not work for testing :

Cycle 1 Cycle 2 Extra Cycle

Original deadline Start

0

50

100

150

200

250

300

WK35 WK36 WK37 WK38 WK39 WK40 WK41 WK42

# T

estc

ases

Planned vs Actual

Tests executed

Actual progress

Planned progress

Actual completion

Planned completion

5© 2011 Capgemini. All rights reserved.

WHAT IS THE PROBLEM?

0

50

100

150

200

250

300

350

400

450

Planned vs Actual

Tests executed

Actual progress

Planned progress (target)

Planned progress (original)

Actual completion

Planned completion (original)

What is The Problem?

Large project• Testing at multiple locations• How do I get the complete status at a moment?

Tracking takes all day

Lack of time for analysis of the collected information

Delayed monitoring & reporting of progress: e.g. on a weekly basis (or longer)

Figures made-up manually based on subjective information

(intentionally or unintentionally)

6© 2011 Eurostar Vugt-Siteur. All rights reserved.

Spreadsheets

Spreadsheets are generally not good enough for registration of the progress

(Test) Tools are good in registration but not so good in advanced graphic, presentation and managing the list of tests or defects

Presentation is the issue – a bowl of defects is confusing

A table with amounts is limited, a good picture paints a thousand words

Trends over time can provide valuable information but are often unavailable

We often resort back to e.g. Excel to make the results “presentable” on the fly

or run into our defect management meeting with a print-out of tests or defects

7© 2011 Eurostar Vugt-Siteur. All rights reserved.

“Houston we have a problem”; Rien van Vugt, Maurice Siteur; EuroStar 22 nov 2011

Tools for Monitoring Test & Defects

Defect management tools are in place most of the timeClearQuest, Mantis, Bugzilla, HP mercury, SourceForge, …..

Often set-up from a development point of viewoStatus only open/closed (out of the box)oLimited detail, more information is needed to monitor & control the workflowoPriority and severity often confused

Test (monitoring) tools are less abundant and may be expensiveFrom a test coordination & management point of view we just need “something” to manage a list of tests, much like a list of defects

8© 2011 Eurostar Vugt-Siteur. All rights reserved.

What Do We Need? Statistics & trends on tests as well as defects

Easy to set-up and generate with flexibility for presentation

much like a spreadsheet can do

Dashboards for:oTest coordination (typical daily)oProgress reporting at project level (typical weekly)oManagement reporting/summaries (weekly or monthly)oTest evaluation (by test level)

Capability & flexibility to present the information at a higher aggregated level

Embed them into the test process so that they:oAre up to dateoObjectiveoCan be retrieved at a moments notice

9© 2011 Eurostar Vugt-Siteur. All rights reserved.

Monitoring Tests - Examples

10© 2011 Eurostar Vugt-Siteur. All rights reserved.

Monitoring Defects - Examples

0

5

10

15

20

25

30

0 1 2 3 4 5 10 15 20 40 60 80 >

# D

efe

cts

Defect Age (max. days)

Open Defect Aging

Tested OK

In Test

Resolved

Additional

Low

Medium

High

Critical

Blocking

0

50

100

150

200

250

300

350

400

450

# D

efe

cts

Open Issue Trend

On hold

In Test

Resolved

Additional

Open prio 5

Open prio 4

Open prio 3

Open prio 2

Open prio 1

5 1320

2 00

51

37226

Open Issues

Open prio 1

Open prio 2

Open prio 3

Open prio 4

Open prio 5

Additional

Resolved

In Test

On hold

© 2011 Eurostar Vugt-Siteur. All rights reserved. 11

Get Into The Game Early

12© 2011 Eurostar Vugt-Siteur All rights reserved.

Quality GateDelivery

Test execution

Build

BusinessRequirements

FunctionalDesign

TechnicalDesign

IntegrationDesign

ComponentIntegration

Test

Unit Test

Test

Development

TestResponsibility

High level test design Detailed test design

High

Level

Test

Design

Detailed

Level

Test

Design

Technicalaspects

Interfaces

Functionality

SystemIntegration Test

(SIT)

System Test

(ST)

AcceptanceTest (UAT)

ProductionReview, Intake

&Knowledge Transfer

Requirements

Bus

sseni

Get Into The Game Early

13© 2011 Eurostar Vugt-Siteur All rights reserved.

Early Involvement of testing in the development life cycle:

1. Review and intake of:o Requirementso Integration and interfaceso Specifications (e.g. use cases)

2. Do not review just for the sake of itActual use of the information drives out issues and inconsistencies and can be used to:oPrevent defectsoComplete test designs early: “conform specs” & “fit for business”oSet-up traceabilityoDefine structure for metrics and different views for monitoring & control

3. Combine this with knowledge transfer on the business background

End Of Day Demonstration

14© 2011 Eurostar Vugt-Siteur. All rights reserved.

Test cases

Defects

Imported Information

End Of Day (Excel)Test Tooling

Extra info

Manual orDerived from

Import

Special EOD formulasTrend tables

Traceability tablesShortlists

CSV, XLS or APIExcel Graphics &

Tables

Dashboard(s) forCoordination

&Reporting

The Future

Integrated tooling?

15© 2011 Eurostar Vugt-Siteur. All rights reserved.

Conclusions

Get involved early Through effective reviews and test intakes Set-up different angles to view/monitor tests progress & defects

Embed advanced metrics in the test & defect process Instant grip on your testing Set-up and monitor trends in test execution Improvements in Test Tooling are needed

Act today to solve tomorrow’s problems

16© 2011 Eurostar Vugt-Siteur. All rights reserved.

Become a Pro-Active Test Manager

Questions?

The Tool is obtainable – Leave your card