What makes a successful Measurement Program? - a case study · 2019-05-29 · of Origin 8 ARLS...

DatePlace

Presenter

© Copyright Total Metrics Pty Ltd

What makes a What makes a successfulsuccessfulMeasurement Program?Measurement Program?

- a case study

September 2008ISMA - WashingtonPam Morris

Total Metricswww.totalmetrics.com


Presenter - Pam Morris

CEO - TOTAL METRICS o Certified IFPUG (IFPUG CPC, CFPS, CSMS - 3)

Consulting, Training Tools and Standardso Metrics Implementation, Function Point Analysiso SCOPE Project Sizing Software™

Committee Member of:o Australian Software Metrics Association (ASMA) – Executive (1991 - )o ISBSG – Vice President Executive (2000 - )o International Function Point User Group (IFPUG) (1993 - 2000)o COSMIC-FFP Core Committee – (1997 - )o International Standards Organisation (ISO) - WG12 (1993 – 2007)o Standards Australia - IT15 (1993 - )


Has anything changed?

Risk of Failure of Measurement Programs

050

100150200250300350400450

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991

Year of Implementation

Number of

MeasurementPrograms

(USA)

Failed Programs Successful Programs

• “80% of all measurement programs fail “Source : Howard Ruben Associates 1994

Overview of Topics

• Background • Measurement Process• Lessons Learned• Critical Success Factors

Background

• Australian Government Department• Large Legacy Application - ~14,000 fps• Mid-range – Cool:Gen, Java• 60 developers• Initial Objective : Verify improvements gained

by Re-factoring activity


Measurement Process– ISO/IEC 15939:2007

1. Establish and Sustain Measurement and Management Commitment• Management had clear stated objectives • 4 Year commitment• Buy in from CIO to Project Team Leaders

STEP 1


2. Plan the Measurement Process• Workshops to agree:

– KRA, KPIs– Report Templates – Data Collection Templates– Tools

Metrics Consultant Resource:• 1 consultant• Effort = 22 days• Duration = 1 Calendar Month

Client Resource:• 4 Management• Effort = 1 ½ day workshops +

Review 2 Drafts • Duration = 1 Calendar Month

STEP 2


Reporting Structures

√√Application CumulativeCostARLS Maintenance Intensity13

√√Project / 6 month snapshotCost / QualityARLS Rework Analysis - Detail12

√√Release / CumulativeCost / QualityARLS Rework Analysis – Summary11

√√Release / 6 month snapshotCost / QualityARLS Time Spent in Testing10

√√Project / 6 month snapshotQuality / CostARLS Development Stage Analysis9

√√Release / 6 month snapshotQualityARLS Analysis of Defects – by Source of Origin

8

√√Release / 6 month snapshotQualityARLS Analysis of Defects – by Severity

7

√√Project / 6 month snapshotQualityARLS Project Quality and Testing Effectiveness

6

√√Project / 6 month snapshotCost & QualityARLS Project Productivity and Quality5

Supplementary Reports

√√Application / CumulativeCost (Investment)ARLS Baseline Growth4

√√√Release / CumulativeQualityARLS Release Quality and Testing Effectiveness

3

√√Release / CumulativeCostARLS Productivity and Release Size2

√√Release / CumulativeCost & QualityARLS Productivity and Quality1

Main Reports

QC

M

ngment

TeamLeadersProjectB

oard

IT SteeringC

omm

ittee

NameNo

Target Audience

Report LevelKey Result Area

Report

Report Templates

• Each Report had agreed:– Purpose– Target Audience– Frequency / Level– Rules for Calculation– Description :

• How to read the report• What it was demonstrating• The types of decisions it would support


Report TemplatesARLS Release Test Effectiveness

4 2 %

2 0 %2 4 %

9 %13 %

18 % 17%

1%

6 0 %

76 %

6 2 %

9 7%

8 6 % 8 6 % 8 7%8 2 %

9 0 %

3 8 %

6 4 %

8 2 %79 %79 %

70 %6 7%

77%

6 5%

3530

4

13

3

3429

75

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

External Benchmark (Total M

etricsIndustry Figures)

FEB 2005 (Size =345 FPs)

MAY 2005 (Size =951 FPs)

AUG

2005 (Size =1461 FPs)

NO

V 2005 (Size =1175 FPs)

FEB 2006 (Size =695 FPs)

MAY 2006 (Size =562 FPs)

AUG

2006 (Size =993 FPs)

NO

V 2006 (Size =741 FPs)

0

10

20

30

40

50

60

70

80

90

100D

efects(Weighted)/1000FPs in M

1 in Production

Effectiveness of Product Stability Testing (PST)Effectiveness of System TestingEffectiveness of Integration TestingRelease Defect Density for First Month in Production (Weighted and normalised for Release size)

Note: Testing Effectiveness compares the number of defects found at a particular stage of testing against how many defects w ere actually le

• Eg Description• This report shows the overall quality of the ARLS development process

since the degree to which defects are released into production are a good indication of the maturity of software development.

Data Collection Templates• 5 Base Measures and Tools Agreed:

– Functional Size (fps)• IFPUG 4.2• SCOPE Project Sizing Software™

– Effort (hours)• ISBSG Definitions Level 2 • NIKU™

– Defects (number)• Origin, severity • ISBSG Customised • Test Track Pro ™

– Duration (Calendar Days )• ISBSG Definitions• NIKU™ Rules for Calculation

– Full-time Equivalents (people)• ISBSG Definitions


STEP 3

• Establish Baseline

Metrics Consultant Resource:• 1 consultant• Effort = 5 days• Duration = 1 Calendar week

Client Resource:• Project Teams • Effort = ? • Duration = 3 Calendar Months

Metrics Consultant Resource:• 1 consultant• Effort = 33 days• Duration = 2 Calendar Months

Client Resource:• 8 application experts• Effort = ~1/2 day each• Duration = 2 Calendar Months

• Ongoing Measurement– ~ 6 projects every 3 month Release (846fps)

3. Perform the Measurement Process


3. Perform Measurement Process

• Analysis of the Results – 49 KPIs

Metrics Consultant Resource:• 1 consultant• Effort = 10 - 15 days• Duration = 3 Calendar weeks

Client Resource:• Management Reviews• Effort = 1 days • Duration = 1 Calendar day


• Reporting the Results– Benchmark Report (6 monthly) – 100 pages

Client Resource:• 1 Metrics Analyst• Effort = 10 days • Duration = 1 Calendar Month


4. Feedback into Technical and Management Processes Release Level - Productivity and Quality Trends

17.616.918.0

11.6

15.2

12.6

21.922.9

29.0

74.7

3.4

34.0

12.9

3.6

30.235.1

0

5

10

15

20

25

FEB 2005 (Size =321FPs)

MAY 2005 (Size =951

FPs)

AUG

2005 (Size=1461 FPs)

NO

V 2005 (Size=1175 FPs)

FEB 2006 (Size =695FPs)

MAY 2006 (Size =562

FPs)

AUG

2006 (Size =993FPs)

NO

V 2006 (Size =741FPs)

Prod

uctiv

ity (H

Rs/

FP)

0

10

20

30

40

50

60

70

80

90

100 Defects (W

eighted) / 1000FPs in 1st Month in Production

PDR for Release in HRs/FP

Release Defect Density for First Month in Production (Weighted and normalised for Release size)

Trend for Defect Density across Releases

Trend for PDR across Releases

PDR Trend

Defect Density Trend

Defect Density Trend

STEP 4

• Product Quality

– Observations• Most defects originated in Build phase• Testing was introducing defects• Testing efficiency was below industry

standard• Time spent early life cycle was below

industry standard• Large variability between projects

4. Feedback into Technical and Management Processes

• Product Quality

– Improvements Introduced• Peer Reviews• Formal Unit Test process• Focus on System Testing• Formal Requirements Management and

Design Process



• Productivity– Observations

• Less productive than Industry• Small projects (<100fps):

– have lower productivity– Small projects behave unpredictably.

• Larger Projects (>250 fps) took longer users optimum 12 months

• FP size gave accurate early life cycle estimates

• Large variability between projects

Less Productive than Industry Median

81892845776842Number in sample

80.740.556.1Maximum Value

30.012.412.523.418.116.821.1Bottom 25% of

Productivity

14.46.79.110.112.112.2Median rate

6.53.76.87.5Top 25% of

productivity

1.80.92.7Minimum value

Case Tools4 GL ProjectsCool:GEN Nov-06Aug-06May-06Feb-06Nov-05Aug-05May-05Feb-05

Industry Values ( R10 - 2007 )Project Median PDR Comparison to Industry by Release

Position


Small Project are more unpredictable

PDR and Project Size

Industry Median COOL GEN Project Size, 209, PDR = 9.1

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

11.0

12.0

13.0

14.0

15.0

16.0

17.0

18.0

19.0

20.0

0 100 200 300 400 500 600

Project Size (FPs)

PDR

As projects get larger i.e.. >180 PDR Spread gets less i.e.. PDR is more predictable and more


FP Size has good correlation with effort

Effort Vs Functional Size (All Projects)

y = 8.6607x + 555.82R2 = 0.6668

0.0

1000.0

2000.0

3000.0

4000.0

5000.0

6000.0

7000.0

0 100 200 300 400 500 600 700Project Size (Function Points)

Effo

rt H

ours

Total Work Effort For ProjectLinear (Total Work Effort For Project)


Estimated FP Size Produced Accurate Effort Estimates

DRIVES Release Effort Estimates Vs Actual Release (Project) Effort in Hours

FEB 20065,969

NOV 200514,702

MAY 20068,426

AUG 200611,862

NOV 200610,366

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

17000

18000

19000

20000

0 4

Rel

ease

Eff

ort H

ours

(D

evel

opm

ent)

Actual Effort (excludes Release Management) (+10% error bars)

Predicted Effort hours using DRIVES (Release Level) median PDR

2nd Benchmark 3rd Benchmark 4th Benchmark


Only 33% of Projects delivering New functionality to the Business and Net Growth is decreasing

Release Functional Impact Analysis - Summary

880269

4

125

109347

402

687

341

426

218255

182

770

808525

477

307

020

0

4

26

0

Net Growth, 215

Net Growth, 109

Net Growth, 453

Net Growth, 334

Net Growth, 343

Net Growth, 835

Net Growth, 976

Net Growth, 400

0

200

400

600

800

1000

1200

1400

FEB 2005 MAY 2005 AUG 2005 NOV 2005 FEB 2006 MAY 2006 AUG 2006 NOV 2006

Tota

l Num

ber o

f Fun

ctio

n Po

ints

0

200

400

600

800

1,000

1,200

1,400

Net

Gro

wth

for R

elea

se (f

ps)

Fps Deleted

Fps Changed

Fps Added

Net Growth

Trend Line for Net Growth

Nett Grow th For Release = (Added Fps)+ (Changed After size - Changed Before Size) - (Deleted)


STEP 5

• Metrics Review Workshop – 2 hours


Client Resource:• Metrics Analyst + Training• Effort = 5 days • Duration = 1 Calendar month

Metrics Consultant Resource:• 1 consultant• Effort = 2 hours• Duration = 1 Calendar day

• Implementing Changes– Data Collection and Recording

Client Resource:• 5 Management team• Effort = 1 day• Duration = 1 Calendar day

5. Evaluate Measurement


Changes Introduced• Defects

– All defects now captured – early life cycle– Unit Testing defects now captured

accurately– Defects now allocated correctly to phase

☺4th Benchmark more defects being reported

• Effort– QC Effort now allocated to the project not

the Release overhead.☺ 4th Benchmark higher Project PDR being

reported


More Defects being Reported

DRIVES - Total number of Defects Found in Release(Weighted normalised defects per 1000 function points)

591

791

626

824

433

771

304

180

0

100

200

300

400

500

600

700

800

900

1000

Def

ects

(Wei

ghte

d N

orm

alis

ed )

/100

0 FP

s


Comparison ALL PROJECT PDR to Industry Medians

7.5

23.4

20.4

11.6

30.3

13.5

31.4

10.1

18.1

12.112.2

16.8

21.1

21.1

16.615.2

14.9

5

10

15

20

25

30

Benchmarking Release

Prod

uctiv

ity (H

Rs/

FP)

all P

roje

cts

in a

Rel

ease

Release Project MedianRelease Project AverageMedian for all Projects All Releases(ISBSG : Coolgen n=28) = 9.1 hrs/fp(ISBSG : 4GL n=89) = 6.7 hrs/fp(ISBSG : Case Tools n=81) = 14.4 hrs/fp

ARLS MEDIAN

Lower Project Productivity (higher PDR)Reported


Critical Success Factors

• Formal Process– Started at beginning– Clear Stated objectives– Vision – long term commitment– Adequate Budget and Resources– Used skilled Metrics personnel– Invested in a baseline functional size

• Management– Realistic expectations– All levels interested, results are shared– Acts on the results– Open to change– Sees bad news as an opportunity– Measurement is viewed as important

News Flash - May 2008

• Two other IT Areas want what they have got!

• 20 other Applications want to be involved and get what the ARLS team are getting!

True Measure of Success!

Total Metrics Pty Ltd667 Burke RoadCamberwellVictoria 3124 Australia

Phone: +61 3 9882 7611Fax: +61 3 9882 7633

[email protected]

Total Metrics Pty Ltd667 Burke RoadCamberwellVictoria 3124 Australia

Phone: +61 3 9882 7611Fax: +61 3 9882 7633

[email protected]

Contact Details

Presentation on Total Metrics WWW site:WWW.totalmetrics.com

Date post:	15-Aug-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

What makes a successful Measurement Program? - a case study · 2019-05-29 · of Origin 8 ARLS...

Documents