Beyond the Hype - QSM the... · 2011. 7. 29. · Comparison of Projects being Assessed to QSM...

Quantitative Software Management, Inc.2000 Corporate Ridge, Suite 900Mclean, VA 22102703.790.0055 ● 703.749.3795 (fax)

BEYOND THE HYPE: [email protected] ● www.qsm.com

EVALUATING AND MEASURING AGILE

© Quantitative Software Management, Inc.

DEVELOPMENT

Outline

• Some silver bullets. Is Agile one?Some silver bullets. Is Agile one?• What are the characteristics of successful

projects?• The promise of Agile• Agile staffing

A il ff t• Agile effort• Agile schedule• Agile productivity• Agile productivity• Agile quality• Measuring and tracking Agile projects

(#2)

Measuring and tracking Agile projects

7/14/2011

Introduction

• “There is no single development, in either technology or management technique which by technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity.”

- Frederick Brooks in “No Silver Bullet –Essence and Accidents of Software Engineering”Essence and Accidents of Software Engineering

• “There is no new thing under the sun”- Ecclesiastes 1.9

• “It has always been agile”- Philip G. Armour

(#3) 7/14/2011

Some Silver Bullets

• Structured programmingLif l th d l i• Lifecycle methodologies

• CASE tools, Code generators• 3gl 4gl languages• 3gl, 4gl, … languages• Object oriented programming• Graphical user interface (GUI)Graphical user interface (GUI)• ERP packages• CMMI• Service oriented architecture (SOA)• Cloud computing

(#4)

• Outsourcing7/14/2011

Silver Bullet Problems

• Neither individually nor in concert with others have the “silver bullets” produced more than have the silver bullets produced more than linear improvement in productivity, quality, or time to market

• Offer technical solutions to a non-technical problem Paradigm has been to transform custom artisan work Paradigm has been to transform custom artisan work

into assembly line production Software is not a manufacturing process. Solutions

designed to improve manufacturing are not applicable to designed to improve manufacturing are not applicable to software development

• Software: a knowledge acquisition process with a

(#5)

technical component7/14/2011

Characteristics of Successful Projects

• Case study Best projects vs. Worst projectsB t j t d fi d th th t th 1 Best projects defined as those that are more than 1σ(standard deviation) better than average for both time to market and cost/effortW t j t 1 th f b th Worst projects are 1σ worse than average for both time to market and cost/effort

Projects evaluated on 58 criteria in Tools & Methods, T h i l C l it P l d RTechnical Complexity, Personnel, and Re-use

(#6) 7/14/2011

Best Project/Worst Projects

Schedule vs Size100

S

Worst Projects

1

10S

chedule

10 100 1,000

Size (thousands)

1

Effort vs Size10,000

Best Projects

100

1,000

,

Effort M

ont

Worst Projects

10 100 1,000

Size (thousands)

10

100 hs

(#7) 7/14/2011

Best Projects

Differentiators

Average Value of Metrics

rPeople, Communication,

Management EfficiencyStaff Turnover

T Skill

s

Knowledge Team SkillMotivation

CohesivenessC i ti

yComplexity

Knowledge

Overall DifficultyC f

Communication

Tools

Customer InterfaceDocumentation Rqmts

Programming Tools Capability

(#8)

0 1 2 3 4 5 6 7 8 9 10Average

7/14/2011

Things that Don’t Matter

Average Value of Metrics

5.7

4.0

5.2

4 2

Data Complexity

Integration Complexity

9.0

8.4

4.2

8.7

g p y

Hardware Stability

System Software Stability

6.0

7.2

5.8

System Software Stability

Overall Tools Capability

C

0 1 2 3 4 5 6 7 8 9 10

5.9

7.0

6.8

6.5

Project Mgt Tools Capability

Development Standards Experience

(#9)

0 1 2 3 4 5 6 7 8 9 10Average

7/14/2011

The Promise of Agile: Agile Manifesto

• Individuals and Interactions over processes and tools

g

tools• Working Software over comprehensive

documentation• Customer collaboration over contract negotiation• Responding to change over following a plan• Key traits

Frequent delivery Business people and developers work together daily Business people and developers work together daily Face to face conversations

(#11) 7/14/2011

The Promise of Agile

• It appears that Agile development embraces the People Knowledge and Communication traits People, Knowledge, and Communication traits that were found in highly successful projects

• Agile is very focused on the social component of g y psoftware development

• So, how well do Agile projects compare to traditional development?traditional development?

(#12) 7/14/2011

Demographics

• 54 recently completed Agile projects12 diff t i• 12 different companies

• 87% business, 7% scientific applications, 6% system softwaresystem software

• Team size clustered in 5-10 and 20-50 ranges• Median size 42.9k lines of code• Median effort 47 staff months• Median staff 7.5• Median duration 6.1 months• Principally new development and major

enhancements

(#13)

enhancements

7/24/2009

Agile Staffing

Average Staff (People) vs Effective SLOC

100

1,000 Averag

1 10 100 1 0000.1

1

10

ge Staff (People)

1 10 100 1,000

Effective SLOC (thousands)

Comparison of Projects being Assessed to QSM BusinessAverage Staff vs Effective SLOC

C&T Average Staff (People) Values

Benchmark Reference Group: QSM BusinessComparison Data Set: Projects being AssessedDifference From Benchmark

at MinEffective SLOC:

5040

2.90

3.400.50

at 25% QuartileEffective SLOC:

18838

5.03

6.211.18

at MedianEffective SLOC:

42870

7.09

9.031.94


122444

10.99

14.583.59

at MaxEffective SLOC:

952614

25.90

37.1611.26

Comparison breakpoints based on min, max, median and quartile values for the data set: Projects being Assessed

The blue trend lines in this and subsequent graphs are the QSM business ith l & i 1 t d d d i ti Th d li i th A il

(#14) 7/14/2011

average with plus & minus 1 standard deviation. The red line is the Agile dataset average

Agile Staffing Observations

• The agile projects use slightly more staff than non agile business projects although the trend is non-agile business projects although the trend is very similar

(#15) 7/14/2011

Agile Effort

Effort Months vs Effective SLOC

100

1,000

Effort

1 10 100 1,000

Eff ti SLOC (th d )

1

10t M

onths


Comparison of Projects being Assessed to QSM BusinessEffort Months vs Effective SLOC

C&T Effort (PM) Values

at MinEff ti SLOC

at 25% QuartileEff ti SLOC

at MedianEff ti SLOC


at MaxEff ti SLOC


Effective SLOC:5040

9.99

8.63-1.36

Effective SLOC:18838

24.63

21.85-2.78


43.24

39.01-4.23


88.68

81.74-6.94


361.08

347.02-14.06


Agile and non-Agile projects use nearly the same amount of project effort for projects with similar amounts of delivered functionality

(#16) 7/14/2011

effort for projects with similar amounts of delivered functionality

Agile Schedule Length

Duration (Months) vs Effective SLOC100

Duratio

1 10 100 1,000

Eff ti SLOC (th d )

1

10

n (Months)


Comparison of Projects being Assessed to QSM BusinessDuration (Months) vs Effective SLOC

C&T Duration (Months) Values

at MinEff ti SLOC


at MedianEff ti SLOC


at MaxEff ti SLOC


Effective SLOC:5040

3.45

2.54-0.91


4.90

3.52-1.38


6.10

4.32-1.78


8.07

5.61-2.46


13.94

9.34-4.60


Agile projects complete much more rapidly

(#17) 7/14/2011

Agile Schedule Observations

• Agile projects complete much more quickly than non agile projects while expending about the non-agile projects while expending about the same amount of effort (Cost)

• Since schedule is frequently an important project q y p p jdriver, this is a significant advantage

(#18) 7/14/2011

Agile Productivity Index (PI)

PI vs Effective SLOC

25

30

35

1 10 100 1,0000

5

10

15

20 PI


Comparison of Projects being Assessed to QSM BusinessPI vs. Effective SLOC

PI Values

at Min at 25% Quartile at Median at 75% Quartile at Max



5040

13.50

15.381.88


18838

16.22

18.191.97


42870

17.92

19.932.02


122444

20.08

22.172.08


952614

24.32

26.532.21


Productivity indices for Agile projects were significantly higher than

(#19) 7/14/2011

y g p j g y gthe business average

Agile Quality

Defects Found in Testing

1,000

10,000

D

10 100 1,0001

10

100D

efects


Comparison of Projects being Assessed to QSM BusinessErrors (SysInt-Del) vs. Effective SLOC

Errors (SysInt-Del) Values

at Min at 25% Quartile at Median at 75% Quartile at Max



12240

44.08

39.64-4.44


28013

86.29

67.04-19.25


101274

244.75

151.55-93.20


254563

516.94

272.02-244.92


952614

1507.89

628.54-879.35


Agile projects produced fewer defects

(#20) 7/14/2011

In Summary

Agile Business IT Difference %DifferenceTypical Sized Agile and Business IT Projects

Size in SLOC 42,900 42,900Average Staff 9 7.1 1.9 26.8%Devel. Duration (Mths) 4.3 6.1 -1.8 -29.5%Effort Months 39 43 -4.0 -9.3%

• Agile projects outperform conventional

Defects (testing) 152 245 -93.0 -38.0%Productivity Index 19.93 17.92 2.0 11.2%

Agile projects outperform conventional development in Productivity, Quality, and Time to MarketSt ffi l l hi h b t ll ff t i • Staffing levels are higher; but overall effort is slightly lower while achieving significant schedule compression

(#21)

p

7/14/2011

Measuring and TrackingMeasuring and Tracking

Agile Projects

© Quantitative Software Management, Inc. 7/14/2011

Issues

• Agile definitely suffers from the “We’re not like other software development so we can’t be other software development so we can t be measured or tracked like them” syndrome

• Large projects may require formality in g p j y q ydocumentation and procedures that nullify Agile advantagesCurrent business practices often conflict with • Current business practices often conflict with Agile methods Outsourcing Splitting teams into onshore/offshore groups Multi-site development

• Agile methods are the key to the results we have

(#23)

• Agile methods are the key to the results we have seen7/14/2011

Overview

• Estimating size of Agile projectsSt i t i t & li f d Stories, story points, & lines of code

• Estimating Agile projects One project with multiple iterations (sprints)O e p ojec u p e e a o s (sp s) One project per iteration

• Tracking Agile projects

(#24) 7/24/2009

Estimating Size of Agile Projects

• Story PointsA l ti i A relative size measure

No standard criteria for definition

• Lines of Code Lines of code or their equivalent (implementation units)

are the basis for sizing in all major parametric estimation toolsestimation tools

Not intuitive and difficult to accurately estimate beforehand

(#25) 7/24/2009

Estimating Size of Agile Projects

• StoriesDi t f f ti lit Discreet groups of functionality

Sprints typically bundle a number of stories Stories that are not completed within the time frame of

the sprint are moved to another sprint Sprints are time boxed: additional sprints may be

added to a project; but a sprint will not be lengthened to complete work

• QSM has captured lines of code for completed sprints and has developed gearing factors for sprints and has developed gearing factors for stories These allow Agile projects to be estimated by SLIM

(#26) 7/24/2009

Stories and CodeCode and Stories Completed

StoriesCompleted

150

200

Sto

6543210

Both stories and code created

0

50

100oriesC

ompleted

code created were measured throughout the project. We

2 4 6 8 10 12 14 16 18 20 22 24 26 28 3007/08

'0607/22 08/05 08/19 09/02 09/16 09/30 10/14 10/28 11/11 11/25 12/09 12/23 01/06

'0701/20 02/03

Cum Eff SLOC1206543210

jwere able to determine gearing factors and refine our

20

40

60

80

100 SLO

C (thousands)

and refine our code estimates by the end of the second iteration

2 4 6 8 10 12 14 16 18 20 22 24 26 28 3007/08

'0607/22 08/05 08/19 09/02 09/16 09/30 10/14 10/28 11/11 11/25 12/09 12/23 01/06

'0701/20 02/03

0

Data from actual project monitored by QSM consultants

(#27) 7/24/2009

Data from actual project monitored by QSM consultants

Code & Story Data by Iteration

Iteration Code Per release Stories Completed Story Gearing

Iteration 0 14295 21 680.7Iteration 1 13165 22 598.4Iteration 2 20130 22 915.0Iteration 3 15794 27 585.0Iteration 4 13348 36 370.8Iteration 5 17940 35 512.6

Average 15779 27 610.4

The data in the table above was used to determine starting point gearing f t f t ifactors for stories

Low complexity 370Average complexity 610g p yHigh complexity 915

This process can be used to determine appropriate gearing factors in different environments

(#28) 7/24/2009

different environments

Estimating Agile Projects

• Entire Agile development effort may be modeled as one estimate with milestones for the iterations as one estimate with milestones for the iterations (see slide 30) Track progress and adjust schedule based on

performance (Are stories being deferred to future sprints?)

• Each Iteration (Sprint) can be an estimate( p ) These are combined to provide a program level view

(see slide 31)

(#29) 7/24/2009

EstimatingOne Project Multiple Iterationsj p

Avg Staff (people)<Solve for PI Wizard>

12543210

8

10Six IterationsIteration Development

6

Avg Staff (people

2

4

e)

Iteration Planning throughout the project

Go Live support

1 2 3 4 5 6 7 8 9Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

0

p j

(#30) 7/24/2009

'10 '11

EstimatingOne Estimate per Iterationp

Monthly Gantt Chart (L3)< Baselline Template >

PLANNING & ARCHITECTURE Architecture Design Iteration Planning Iteration Planning Planning CompleteAGILE SOFTWARE ITERATIONS iteration_001 Iteration 1 Complete iteration_002 Iteration 2 Complete iteration_003 Iteration 3 Complete

8/18/2007

8/28/2007

10/1/2007

11/1/2007

Estimates are combined in SLIM MasterPlan to Iteration 3 Complete

iteration_004 Iteration 4 Complete iteration_005 Iteration 5 Complete iteration_006 Iteration 6 Complete Iteration Integration & Test Development Complete

11/1/2007

12/6/2007

1/8/2008

2/9/2008

3/8/2008

MasterPlan to provide a program level view

Development CompleteCustomer AcceptanceTestEnd of ProgramPROJECT OVERHEAD Project Management Configuration Management Quality AssuranceDEPLOYMENT & TRAINING Site Installation

3/8/2008

4/18/2008

1 2 3 4 5 6 7 8 9 10 11Jul'07

Aug Sep Oct Nov Dec Jan'08

Feb Mar Apr May Jun

Site Installation T raining 1 T raining 2

(#31) 7/24/2009

Tracking Agile ProjectsCore Metrics View

FTE Staffing Total Project Effort

50

606543210Cumulative Effort Total Project Effort

50

60

Cum

6543210

0

10

20

30

40

FTE

0

10

20

30

40

m H

rs (thousands)

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

03 6 9 12 15 18 21 24 27 30

07/08'06

07/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13'07

02/03

0

StoriesCompleted2506543210

Defects Found1506543210

50

100

150

200 StoriesCom

pleted

25

50

75

100

125

Defects

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

0

50

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

0

25

Overall view of effort defects and stories completed

(#32) 7/24/2009

Overall view of effort, defects, and stories completed

Tracking Agile ProjectsStories

Count of Stories by Iteration2506543210

150

200

100

Count

50

2 4 6 8 10 12 14 16 18 20 22 24 26 28 3007/08

'0607/22 08/05 08/19 09/02 09/16 09/30 10/14 10/28 11/11 11/25 12/09 12/23 01/06

'0701/20 02/03

0

This slide shows the stories completed by iteration

(#33) 7/24/2009

p y

Tracking Agile ProjectsIteration 1 Dashborad

Cumulative Effort Iteration 1Based on a Plan of 23 Story Deliv erables

3,000

4,000

5,000C

um6543210

Effort Burn Rate Iteration 1Based on a Plan of 23 Story Deliv erables

1,000

1,500

Hou

6543210

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

0

1,000

2,000

m H

rs

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

-250

250500

urs

Stories Increment 1 Percentage CompleteFTE Staffing Iteration 1 Stories Increment 1 Percentage CompleteBased on a Plan of 23 Story Deliv erables

3 6 9 12 15 18 21 24 27 30020406080100120

% C

omplete

6543210

FTE Staffing Iteration 1Based on a Plan of 23 Story Deliv erables

3 6 9 12 15 18 21 24 27 30-10

0

10

20

30

40

FTE

6543210

07/08'06

07/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13'07

02/0307/08'06

07/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13'07

02/03

FTE Staffing ManageBased on a Plan of 23 Story Deliv erables

3

4

5

F

6543210

Defects Found Category TotalBased on a Plan of 23 Story Deliv erables

100125150

De

6543210

3 6 9 12 15 18 21 24 27 3007/08

'0607/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13

'0702/03

0

1

2

FTE3 6 9 12 15 18 21 24 27 30

07/08'06

07/29 08/19 09/09 09/30 10/21 11/11 12/02 12/23 01/13'07

02/03

0255075

fects

(#34) 7/24/2009

Here are tracking metrics for a single iteration within the project

QUESTIONS?QUESTIONS?

© Quantitative Software Management, Inc.

Date post:	25-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Beyond the Hype - QSM the... · 2011. 7. 29. · Comparison of Projects being Assessed to QSM...

Documents