Application of economic models on software maintenance

Anh Nguyen DucTrial lecture

Trondheim, Norway

April 10th 2015

2

Agendao Software maintenanceo Cost models of the maintenance phaseo Economic measures of maintenance activitieso Value driven prioritization of maintenance task

3

Significance of software maintenance

2 to 100 times: maintenance cost vs.

development cost

Figure from Floris and Harald, 2010

4

CorrectiveAdaptivePerfective Preventive

Software maintenance types

The totality of activities required to provide cost-effective support to software

5

Software maintenance process

6

Economic concerns in software maintenance

Busines

s leve

l

• Should we continue maintaining the system in the next X year or buying a new system?

• How much to invest on maintenance for the greatest earned value (saved cost, increased income)?

Produc

t leve

l

• How much people need to be assigned for maintaining the module Y?

• How much does maintenance cost in comparison to development cost?

Project leve

l

• How long does it take to fix a post-release software defect?

• Which features require the least cost but provide the greatest customer value?

7

Agendao Software maintenanceo Cost models of maintenance phaseo Economic measures of maintenance activitieso Value driven prioritization of maintenance task

8

History of parametric cost models

SDC PRICE S Putnam SLIMCOCOMO 81 ESTIMACS

Checkpoint

Belady and Lehman

65 72

77 79 81 83 85

SEER-SEM

97 00

COCOMO versionsTan and Mookerjee

03 05

95

• 65-85: the search for right parametric forms• 85-95: advances in size and complexity metrics • 95-05: proliferation of software development styles• 05-15: advances in prediction and classification models

Ahn’s SMPEEM

Albrecht FP

Niesink et al.’s MFP

98

year

2015

9

Belady and Lehman (1972)

• Effort: total maintenance effort• p: productive effort, including analysis, design,

code testing• d: degree of maintenance team familiarity with the

software• c: complexity caused by lack of structured design

and document• K: empirical constant, depends on the environment

Effort= p + K c-d

10

Effort (person-month) as a function of size (LOC)

Effort = b x Size c

• Organic: small teams develops software in known environment (b=2.4, c=1.05)

• Embedded: inflexible and constrained environment (b=3.6, c=1.20)

• Semidetached: varying levels of team experience working on larger projects (b=3.0, c=1.12)

Effort = b x Size c x EAF

• Intermediate model• b, c: calibrated factors• EAF: effort adjustment factor

COCOMO (1981)

11

Example:o A ERP module with original size of 10,000 KLOCo Development effort is 600 PMso Estimated 1,500 new and modified KLOC per yearo ACT = 1500/ 10000 = 15% per year

o Maintenance cost of a 10 year project:= 0.15 x 600 x 10 = 900 PMs

Maintenance effort = ACT x Development effort

• NNL: number of new LOCs• NML: number of modified LOCs• NOL: number of original LOCs

Annual change traffic (ACT) = (NNL + NML)/ NOL

COCOMO (1981) – Maintenance cost model

12

COCOMO (1997) – Post architecture level

Application composition

Early design

Post-architecture

o SF: cost drivers, i.e. development flexibility, architecture/risk resolution, team cohesion and process maturity

o EM: 17 effort multipliers, i.e. the required software reliability, database size, product complexity, required reusability, documentation, etc.

13

Function Point based estimation

• Function point (FP) as a consistent and early size metric

• EFP: enhanced project function point• ADD, CHG, DEL: added, changed and deleted FPs• CFP: unadjusted function points added by conversion• VAFa and VAFb: value adjustment factors of the application

after and before the project

EFP = (ADD + CHG + CFP) × VAFa + (DEL × VAFb)

14

Figure from Boehm et al. 2008

• Repeatable estimations• Easy to modify input• Easy to customize and refine formula

Advantages

• Subjective inputs• Unable to deal with exceptional

conditions• Mainly designed for waterfall• Needs historical data for calibration

Disadvantages

COCOMO suite of models (1981-2007)

15

Model calibration

• A systematic adjustment of model parameters to get an expected output with known inputs

• Proper calibration is important

COCOMO II 1997 calibration of 83 data points: accuracy within 20% of actual values in 46% of the times

COCOMO II 1998 recalibration of161 data points: accuracy within 30% of actual values in 75% of the times

16

Model evaluation

• Modest predicting results– “… (estimating in general varies) from as much as 85 - 610 % between

predicated and actual values …” (Kemerer 1993)– “…best models had MMRE (Mean Magnitude of Relative Error) of 60%,

… too inaccurate to be of any use …” (Jørgensen 1997)

• Combination with other estimation approaches– Expert judgments, case based reasoning, regression models, Bayesian

network …

• Milestone budget and schedule for project stakeholder negotiation and expectations management

• Basis of software maintenance planning and control

17

Agendao Economic aspects of software maintenanceo Economic measures of maintenance activitieso Cost models of maintenance phaseo Value driven prioritization of maintenance task

18

Value-based software maintenance

• The decision to perform maintenance at any time is based on cost benefit analysis and economic measures

• Earned-value system• A set of maintenance tasks• Planned Value (PV) in

money and time of the tasks

• Quantification of Earn Value (EV)

19

Return on Investment (ROI)

Example:o The benefit of the current system is 250,000 USDo The benefit of the adapted system is 300,000 USDo The estimated cost of change: 14,000 USD

o The benefit of the maintenance task is: 300,000 – 250,000 = 50,000 USD

o ROI = (50,000 – 14,000) / 14,000 = 2.57

• Benefit = Value of Changed System - Value of Original System• Cost = Adjusted size / productivity x Cost driver factors

ROI = (benefit – cost) / cost

20

Net present value (NPV)

• Development phase requires n1 years with a cost of c1 at the end of each year.

• Operation and maintenance (O&M) phase requires n2 years with a cost of c2 at the end of each year.

• Cost of capital is at a change rate of r %

21

Management of technical debts

• Metaphor indicating possibly of significant economic consequences in software maintenance

• Technical debts occur when making the change worked as quickly as possible with as few resources as possible

Increased productivity, lowered development costs of the current release

Increased maintenance costs, debt out of controls

22

Management of technical debts

• Principal = cost of paying off the debt – Maintenance task effort estimation

• Interest benefit of paying off – Interest probability x interest amount

• Which debts to pay off?• Applications of technical debts

– Project budget allocation– Prioritizing maintenance tasks

23

Agendao Economic aspects of software maintenanceo Economic measures of maintenance activitieso Cost models of maintenance phaseo Value driven prioritization of maintenance task

24

Focus on low-cost, high-value changes

Effort-to-change

Perceived value

Low High

High

Lowo Technical debt pay-offo Test case triageo Defect prioritizationo Feature reductiono …

25

Deciding customer´s perceived value

• Approaches– Multi criteria decision analysis (MCDA): Analytic hierarchy

process (AHP), PROMETHEE, etc– Agile: MoSCoW, planning game, quality house, hundred-

dollar test, etc

• Expert judgment, no model based approach

26

Triage and prediction models

Post-release defect triage

Issue resolution time estimation

Modification request (MR)

History of delta

Communication data

Project context

Regression+ machine learning

SIZE & COMPLEXITY

Change proneness

Defect proneness

Auto task assignment

27

Indicators of change effortCode metricso Cyclomatic complexityo Object oriented metricso Entropy of code changeso Churn metricso Code smell

Modification request (MR)

History of delta

Communication data

Project context

Regression+ machine learning

SIZE & COMPLEXITY

28

Indicators of change effortCommunication datao No. of commentso No. of commenterso Communication

frequencyProject contexto No. of locationso No. of developerso CMMI levelso Team experience levelso …

Modification request (MR)

History of delta

Communication data

Project context

Regression+ machine learning

SIZE & COMPLEXITY

29

Indicator of change effort

Figure from Zhang et al. 2013

Figure from Nguyen-Duc et al. 2011

30

Reflection on model building

• Local vs. global prediction• Implicit assumptions of models• Universal indicators of software changes• Data missing and preprocessing• Compounding impacts

31

Summary

• Cost-benefit concern of software maintenance for vendors and buyers

• Measuring economic value of software maintenance activities

• Limited applicability of traditional maintenance costs• Integration of code, process, human based metrics in

estimating and triaging maintenance tasks

32

Reference

BOOK1. Barry Boehm, Software Engineering Economics, Prentice-Hall, 19812. Guide to the Software Engineering Body of Knowledge – SWEBOK, IEEE Press,

20043. I. Sommerville. Software Engineering, 7th edition, Pearson Education, 20044. Shari Lawrence Pfleeger, Joanne M. Atlee, Software Engineering: Theory and

Practice, 3rd edition, Pearson Education, 2006. 5. Stefan Biffl, Aybuke Aurum, Barry Boehm, Hakan Erdogmus, Paul Grünbacher,

Value based software engineering, Springer, 2006

33

Reference

PAPER (1)• Basili, V., et al. (1996). Understanding and predicting the process of software maintenance release.

Proceedings of the 18th international conference on Software engineering, IEEE Computer Society.• Boehm, B. W. and R. Valerdi (2008). "Achievements and challenges in cocomo-based software

resource estimation." Software, IEEE 25(5): 74-83.• Chulani, S., et al. (1999). "Calibrating software cost models using bayesian analysis." IEEE

Transactions on Software Engineering: 573-583.• Čubranić, D. (2004). Automatic bug triage using text categorization. In SEKE 2004: Proceedings of the

Sixteenth International Conference on Software Engineering & Knowledge Engineering, Citeseer.• Curtis, B., et al. (2012). "Estimating the Principal of an Application's Technical Debt." IEEE

software(6): 34-42.• Duc, A. N., et al. (2011). Impact of stakeholder type and collaboration on issue resolution time in

OSS Projects. Open Source Systems: Grounding Research, Springer: 1-16.• Fioravanti, F. and P. Nesi (2001). "Estimation and prediction metrics for adaptive maintenance effort

of object-oriented systems." Software Engineering, IEEE Transactions on 27(12): 1062-1084.

34

Reference

PAPER (2)• Granja-Alvarez, J. C. and M. J. Barranco-García (1997). "A method for estimating maintenance cost in

a software project: a case study." Journal of Software Maintenance 9(3): 161-175.• Jorgensen, M. and M. Shepperd (2007). "A systematic review of software development cost

estimation studies." Software Engineering, IEEE Transactions on 33(1): 33-53.• Kemerer, C. F. (1987). "An empirical validation of software cost estimation models." Communications

of the ACM 30(5): 416-429.• Mockus, A., et al. (2003). Understanding and predicting effort in software projects. Proceedings of

the 25th International Conference on Software Engineering, IEEE Computer Society.• Niessink, F. and H. van Vliet (1997). Predicting maintenance effort with function points. Software

Maintenance, 1997. Proceedings., International Conference on, IEEE.• Sneed, H. M. (2004). A cost model for software maintenance & evolution. Software Maintenance,

2004. Proceedings. 20th IEEE International Conference on, IEEE.• Vienneau, R. L. (1995). "The present value of software maintenance." Journal of Parametrics 15(1):

18-36.• Zhang, H., et al. (2013). Predicting bug-fixing time: an empirical study of commercial software

projects. Proceedings of the 2013 International Conference on Software Engineering, IEEE Press.• • • • • •

35

Q&A