Read the corresponding blog post here http://www.ontechnicaldebt.com/blog/it-organizations-have-far-too-much-technical-debt/ New article published by IEEE discussing the estimation of technical debt, authored by experts at CAST Software
Steve McConnell deSCribed technical debt as including both inten-tional and unintentional violations of good architectural and coding prac-tice1—an expansion of Ward Cunning-ham’s original focus on intentional de-cisions to release suboptimal code to achieve objectives such as faster deliv-ery.2 By choosing debt as a metaphor, Cunningham engaged a set of financial concepts that can help executives think about software quality in business terms. Although the concept of tech-nical debt incorporates entities such as principal, interest, liabilities, and
opportunity costs, this article explores only the estimation of its principal.
the technical debt MetaphorIn embracing McConnell’s approach as the most comprehensive for com-municating the costs and risks of poor structural quality, we use the follow-ing definitions for constructs estimated in this article:
• Should-fix violations are viola-tions of good architectural or cod-ing practice (hereafter referred to
simply as “violations”) known to have an unacceptable probability of contributing to severe opera-tional problems (outages, security breaches, data corruption, and so on) or of contributing to high costs of ownership, such as excessive ef-fort to implement changes.
• Principal is the cost of remediat-ing should-fix violations in produc-tion code (hereafter referred to as “TD-principal”).
• Interest is the continuing costs at-tributable to should-fix violations in production code that haven’t been remediated, such as greater maintenance hours and inefficient resource usage.
• Technical debt is the future costs attributable to known violations in production code that should be fixed—a cost that includes both principal and interest.
For the technical debt metaphor to be useful, its constructs must be measurable or at least estimable from measurable elements of software. For-tunately, we can estimate the viola-tions underlying TD-principal via tech-niques such as static analysis of the software’s nonfunctional, structural characteristics.3 Violations of struc-tural quality are often difficult to de-tect through standard testing but are frequent causes of severe operational problems.4,5
Facing limited application bud-gets, IT organizations will never fix all violations in an application. Tech-nical debt estimates ought to only in-clude should-fix violations in produc-tion code. Nonetheless, the amount of should-fix problems sometimes exceeds the budget available for remediation. Consequently, IT management must estimate the amount of technical debt in its applications and then adjust the
Estimating the Principal of an Application’s Technical DebtBill Curtis, Jay Sappidi, and Alexandra Szynkarski, CAST Software
// A formula with adjustable parameters can
help in estimating the principal of technical
debt from structural quality data. //
FOCUS: Technical DebT
s6cur.indd 34 10/4/12 2:35 PM
NovEmbEr/dEcEmbEr 2012 | IEEE SoftwarE 35
parameters in its estimates to deter-mine how much of that debt can be re-duced within the available budget. This evaluation helps prioritize the problems to remediate, as well as provides infor-mation about the amount and types of risk remaining in an application.
A Method for estimating td-PrincipalThere’s no exact measure of an appli-cation’s TD-principal because its cal-culation should be based only on the should-fix violations, some of which might be undetected. However, mod-ern software analysis and measure-ment technology lets us estimate TD-principal from counts of detectable violations. Thus, we can estimate TD-principal as a function of three vari-ables—the number of should-fix vio-lations in an application, the hours to
fix each violation, and the cost of la-bor. We can measure or estimate each of these variables to develop a formula for computing TD-principal. The time to fix a violation, for example, could be available from historical effort data. The cost to fix violations can be set to the average burdened rate for the devel-opers assigned to the activity. Using the three variables, the equation for esti-mating TD-principal is
TD-principal = ((S high-severity violations) × (percentage to be fixed) × (average hours needed to fix) × (US$ per hour)) + ((S medium-severity violations) × (percentage to be fixed) × (average hours needed to fix) × ($ per hour)) + ((S low-severity violations) × (percentage to be fixed) × (average hours needed to fix) × ($ per hour)).
Estimates from this equation pro-vide managers with ballpark figures for assessing future maintenance costs, as well as for making investment decisions regarding structural quality improve-ments to reduce future costs and risks. Although we present several choices for parameter values here, IT organiza-tions should calibrate these parameters to their own experiences to obtain the most relevant estimates.
We used three different settings for these parameters (see Table 1) to ex-plore their effects on TD-principal esti-mates. Although burdened hourly rates can vary by experience and location, we found that a rate of US$70 to $80 per hour reflects the average costs for many IT organizations, especially where they have a mix of on- and offshore opera-tions. Consequently, we used the same hourly rate in all three estimates. In
Tab
le
1 Parameter values for three estimates of TD-principal.
each estimate, we varied the percentage of violations at each severity level that would be prioritized for remediation, addressing fewer severity levels in esti-mates 2 and 3.
The calculation in estimate 1 as-sumes that all violations would be fi xed within one hour. Based on industry data, this is an extremely conservative number, which we chose to provide a lower bound for TD-principal. In esti-mate 2, we varied the hours needed for fi xing within each severity category, assuming that high-severity violations would contain proportionately more vi-olations and require more hours to fi x. Because there are few published dis-tributions of hours to fi x, we modeled
the time to fi x in estimate 3 as a dis-tribution that might be more realistic based on data observed in several IT organizations.
the Sample and dataThe data reported here are drawn from the Appmarq benchmarking reposi-tory maintained by CAST Software.6 For this exploration, we drew a sam-ple from the repository of 700 applica-tions submitted by 158 organizations for the analysis and measurement of their structural quality characteristics. The applications in this sample cumula-tively contained 357 MLOC. We didn’t accept applications into the sample if they consisted of fewer than 10 KLOC.
The organizations that submitted these applications were primarily from the US, Europe, and India. Because there’s no rigorous characterization of the global population of business ap-plications, it’s impossible to assess the generalizability of results drawn from this sample. Nevertheless, these results emerge from what we believe to be the largest sample of applications to be statically analyzed for structural qual-ity characteristics across different lan-guages and technology platforms. The industries from which we received these applications included fi nancial services, insurance, telecommunications, manu-facturing, energy, IT consulting, retail, and government. Because of criteria
Evaluation of 1,200+ coding and
architectural rules
Application metadata
Expensive operation in loopStatic vs. pooled connections Complex query on big tableLarge indices on big table
Empty CATCH blockUncontrolled data accessPoor memory managementOpened resource not closed
SQL injectionCross-site scriptingBuffer over�owUncontrolled format string
Unstructured codeMisuse of inheritance Lack of commentsViolated naming convention
Highly coupled componentDuplicated codeIndex modi�ed in loopHigh cyclomatic complexity
Oracle PL/SQLSybase T-SQLSQL Server T-SQLIBM SQL/PSMC, C++, C#Pro CCobolCICSVisual BasicVB.NetASP.NetJava, J2EEJSPXMLHTMLJavaScriptVBScriptPHPPowerBuilderOracle FormsPeopleSoftSAP, ABAPNetweaverTibcoBusiness ObjectsUniversal analyzer for other languages
FiGURe 1. CAST Software’s Application Intelligence Platform. This technology analyzes all the source code of an application and reintegrates
the metadata across languages to detect violations of more than 1,200 rules of good architectural and coding practice. These violations are
aggregated into measures of several quality characteristics, which are provided to both management and the developers.
s6cur.indd 36 10/4/12 2:35 PM
NovEmbEr/dEcEmbEr 2012 | IEEE SoftwarE 37
used by most companies for submit-ting applications to analysis, we believe this sample is biased toward business- critical applications.
We analyzed these applications us-ing CAST’s Application Intelligence Platform (AIP),7 which analyzes an en-tire application using more than 1,200 rules to detect violations of good ar-chitectural and coding practice. We drew these rules from software engi-neering literature, repositories such as the Common Weakness Enumeration (CWE; cwe.mitre.org), online discus-sion groups of structural quality prob-lems, and customer experience as re-ported from defect logs and application architects. As an example, security- related violations would include SQL injection, cross-site scripting, buffer overflows, and other violations from the CWE.
The AIP begins by parsing an ap-plication’s entire source code at build time to develop a representation of the elements from which the application is built—its data flows, class hierarchies, transaction paths, calling relationships, and so on. The AIP includes parsers for the 28 languages listed in Figure 1, and a universal analyzer provides a par-tial parse of languages that don’t have dedicated parsers. The metadata pro-duced from this parsing is reintegrated across languages and platforms to pro-vide a full view of the application. The AIP uses several techniques to detect violations of its architectural and cod-ing rules (examples of which are under the Detected violations column in Fig-ure 1). Integrating the metadata lets us consider the full application context in detecting violations and reducing false positives. For instance, to determine whether a table being called by a loop with an expensive operation violates performance efficiency rules, we need to know the context of whether the ta-ble is large or small.
AIP scoring begins by detecting
the number of opportunities for a rule to be triggered and the percentage of times that rule was violated. Each vio-lation is weighted by its severity and aggregated to the application level. (Us-ers can adjust severity weights for each violation to match local priorities, but the results we present here are based on the original weights.) The AIP provides a series of management reports and a portal to guide developers to locations in the source code where specific viola-tions can be remediated. The manage-ment report aggregates violation scores into measures for the five quality char-acteristics shown in Figure 1:
• robustness, the stability or re-silience of an application and its ability to avoid outages or recover quickly from them;
• performance efficiency, the appli-cation’s responsiveness and its effi-cient use of resources;
• security, an application’s abil-ity to prevent unauthorized in-trusions and protect confidential information;
• transferability, the ease with which a new team can understand the ap-plication and quickly become pro-ductive in working with it; and
• changeability, an application’s abil-ity to be easily modified and avoid the injection of new defects.
We selected these characteristics after reviewing ISO/IEC 9126.8 How-ever, because the quality character-istics in 9126 and its successor, ISO/IEC 25010,9 aren’t defined to a level
that can be computed from the source code, some quality characteristic names used here differ based on the content analyzed and the meaningfulness of the names to management. The AIP evalu-ates anywhere from 176 to 506 rules for each quality characteristic, and some rules are evaluated for more than one characteristic.
Calculating td-PrincipalWe calculated three estimates of TD-principal for each of the 700 applica-tions using equations with the three sets of parameter values listed in Table 1. The first row of Table 2 shows de-scriptive statistics for the distribution of these estimates across the full sample for each of the three estimating equa-tions. The differences among means for these three estimates are large, rang-ing from $3.61 in estimate 1 to $15.62 in estimate 3. This large difference in mean values reveals how sensitive the resulting estimates of TD-principal are to values selected for the parameters. Although almost any result can be ob-tained by manipulating parameters, the critical lesson is the importance of adjusting parameters to fit local priori-ties for fixing violations and matching historical data regarding the times and
costs for doing so. Only then can such estimates provide useful information about TD-principal.
Table 2 also presents descriptive sta-tistics for the three estimating equa-tions for seven of the languages used in the 700 applications. Because mod-ern applications are frequently devel-oped in several languages, we split each
Integrating the metadata lets us consider the full application context in detecting violations and reducing false positives.
s6cur.indd 37 10/4/12 2:35 PM
38 IEEE SoftwarE | www.computEr.org/SoftwarE
FOCUS: Technical DebT
application into subsystems written in different languages. We produced estimates using each of the three esti-mating equations for each subsystem within the seven languages in Table 2.
The differences in TD-principal es-timates among different languages are
large. For instance, the mean for esti-mate 1 ranges from a low of $1.90 per LOC for Advanced Business Applica-tion Programming (ABAP) to a high of $21.16 per LOC for Oracle Forms (F = 10.63; p < .0001). Structural dif-ferences in the languages could partly
explain this large spread. However, it might also be affected by the different uses to which these languages are ap-plied, ranging from customizing an ex-isting vendor package with ABAP to developing an entire application with Java EE or C++. We also can’t rule out
Tab
le
2 Estimated US dollars per LOC of TD-principal by language.*
*Maximums, minimums, and quartiles for individual language distributions can be greater than or less than the figures for the total sample because the numerator and denominator change when applications are divided into language-specific subsystems.
**Because some applications contain multiple languages, the sum of the samples for the languages is greater than 700.
s6cur.indd 38 10/4/12 2:35 PM
NovEmbEr/dEcEmbEr 2012 | IEEE SoftwarE 39
sample-specific factors affecting these differences because other research, con-trary to our results, found technical debt to be the highest for ABAP.10
The variance in estimates within a single language category is quite large. For instance, among Java EE applica-tions, estimates of TD-principal using the parameters in estimate 2 ranged from $0.23 per LOC to $253.03 per LOC. The distributions are all positively skewed, with most languages having large interquartile ranges. Consequently, to be used effectively in management de-cisions regarding cost of ownership or investments in structural quality, IT or-ganizations should measure and analyze
TD-principal for each application indi-vidually—or, at most, for applications being developed under similar condi-tions for similar uses—rather than using an average estimate across all applica-tions within a language category. Identi-fying the factors behind these variances in operational environments can reveal opportunities for large reductions in software costs.
td-Principal by Quality CharacteristicThe violations of structural quality from which TD-principal is estimated can represent different types of threats to the business or costs to IT. To use
TD-principal estimates strategically, management must establish its struc-tural quality objectives and allocate remediation resources accordingly.11 Our data allow us to estimate the TD-principal associated with the violations that constitute each of the five quality characteristics we defined in the section called “The Data and Sample.”
The first row in Table 3 shows the percentage of total violations constitut-ing TD-principal for each of the five quality characteristics using violation parameters for each of the three esti-mates. For the full Appmarq sample, 70 percent of the TD-principal esti-mated in this sample was contained in
Tab
le
3 Percentage of technical debt associated with each quality characteristic.
the IT cost–related quality characteris-tics of changeability and transferability, whereas only 30 percent was contained in the business risk factors of robust-ness, performance efficiency, and secu-rity. We can’t determine from the data whether IT organizations are spending more time eliminating TD-principal re-lated to business risk or, alternatively, whether TD-principal is created most often in violations associated with IT cost–related factors.
The remaining rows in Table 3 indi-cate that the relative percentages of vio-lations from each of the quality charac-teristics constituting TD-principal were generally consistent across language categories and estimating parameters. However, there were two notable varia-tions. First, robustness violations ac-counted for relatively higher percent-ages of ABAP’s and Oracle Forms’ TD-principal; security and changeabil-ity violations accounted for relatively lower percentages. These differences could relate to how these languages are used in customizing their associated packaged software.
Second, as the estimating param-eters for violations shifted toward
high-severity violations in estimates 2 and 3, the percentage of viola-tions shifted from transferability to-ward other quality characteristics for Java EE and Oracle Forms. This shift likely resulted from the relatively high proportion of comment-related violations for these languages that were eliminated from TD-principal calculations as the estimating param-eters shifted toward higher-severity
violations. (We discuss this more in the next section.)
The quality characteristic results suggest that the analysis and measure-ment of TD-principal can be used in conjunction with structural quality priorities to guide management deci-sions about how to allocate resources for reducing business risk and IT cost. For many IT managers and executives, trying to make decisions about retir-ing TD-principal at a global level is overwhelming, and they struggle to visualize what the expected payoff will be. However, when managers can analyze TD-principal by its constitu-ent quality characteristics, they can set specific reduction targets based on strategic quality priorities with a bet-ter understanding of the cost or risk reduction benefits.
Analysis of violationsWe can gain deeper insight into struc-tural quality by investigating the types of violations that create TD-principal. Table 4 presents the five most frequent violations included in TD-principal and the frequency of their detection across applications for each of the seven lan-
guages we studied.These results are difficult to com-
pare at the application level because many violations are defined specific to each language. Nevertheless, sev-eral themes emerge from these results. First, if we divide the frequencies of the individual violations by the number of applications in which they were de-tected, we find that the number of oc-currences per application is large for
all of the violations. For instance, the average number of violations in Java EE for the high-severity violation of us-ing fields from other classes was 1,173 per application.
Second, a consistent problem across all the languages except Java EE is com-plexity represented as violations of ei-ther a module-calling structure with high fan-out to external components or a control flow with high internal complexity. These violations are often traces of the tradeoffs that led Cun-ningham to make his original observa-tion regarding technical debt.2 Under-standing the tradeoffs between these violations and either operational per-formance or maintenance costs are crit-ical to helping managers set strategic structural quality objectives.
Third, the large percentage of TD-principal accounted for by the cost-re-lated quality characteristics of change-ability and transferability in Table 3 is reflected in Table 4 by the frequen-cies of complexity-related violations, as well as violations of practices that aid comprehension. These cost-related violations are especially prevalent in Java, accounting in part for its higher TD-principal cost compared to other languages. Because these tend to be categorized as medium-severity viola-tions, managers must weigh the cost tradeoff between remediating these violations and the accruing interest from increased effort to understand the code.
e ven when measured with a con-servative formula, the amount of technical debt in most busi-
ness applications is formidable. For instance, even when applying the con-servative parameters in estimate 1, the average application is estimated to have $361,000 of TD-principal for each 100 KLOC. When the more realistic parameters of estimate 3 are applied,
Managers can set specific reduction targets based
on strategic quality priorities.
s6cur.indd 40 10/4/12 2:35 PM
NovEmbEr/dEcEmbEr 2012 | IEEE SoftwarE 41
executives will likely dismiss the esti-mated size of TD-principal as exces-sive. However, when large estimates result from accurate parameters for an organization’s hours to fix and cost per hour, then the percent of violations to be fixed can be varied to determine how many violations can be remediated within existing budgets and which vio-lations to prioritize.
We urge caution in interpreting the
estimates we present in this article as industry benchmarks. This explo-ration of estimates for TD-principal demonstrates that these estimates are extremely sensitive both to the as-sumptions made in parameterizing the equation and the different types of languages to which they are applied. These estimates could also shift with changes in the mix of application char-acteristics in each language category
as the number of applications grows in the Appmarq repository. Neverthe-less, these results provide a good start-ing point for exploring TD-principal, and one that can be adjusted based on different assumptions about the pa-rameters used. When developed with professional discipline, estimates of TD-principal can be a powerful tool to aid management in understanding and controlling IT costs and risks.
Tab
le
4 The top five violations contributing to TD-principal by language.
Language Violation Frequency
.NET
(n= 63)
Avoid uncommented methodsAvoid declaring public class fieldsAvoid artifacts with high fan-outAvoid classes with a high lack of cohesionAvoid instantiations inside loops
203,651152,97284,58056,48616,309
ABAP
(n = 72)
Avoid artifacts with a complex Select clauseAvoid artifacts with high internal complexityAvoid artifacts with high fan-outAvoid artifacts with high depth of codeAvoid artifacts with high fan-in
61,37661,18443,42820,06116,490
C
(n = 44)
Avoid undocumented functionsAvoid artifacts with high internal complexityAvoid functions with SQL statement including subqueriesNever use strcpy() function—use strncpy()Never use sprintf() function or vsprintf() function
56,02732,94330,15329,33221,608
C++
(n = 30)
Avoid undocumented functions, methods, constructors, destructorsAvoid data members that are not privateAvoid unreferenced methodsAvoid using global variablesAvoid artifacts with high internal complexity
267,861182,07673,88847,83418,065
Java EE
(n = 474)
Avoid methods missing JavaDoc commentsAvoid methods missing appropriate JavaDoc @param tagsAvoid methods missing appropriate JavaDoc @return tagsAvoid private fields missing JavaDoc CommentsAvoid using fields (nonstatic final) from other classes
4,028,7273,227,0143,018,1821,737,620
556,046
Oracle Forms
(n = 45)
Avoid objects without Comment propertyAvoid artifacts with high fan-outAvoid artifacts with high internal complexityAvoid artifacts with high fan-inUse based data blocks naming convention—represented table
1,717,61668,21332,41120,6169,866
Visual Basic
(n = 16)
Avoid undocumented functions and methodsAvoid using global variablesAvoid unreferenced functions and methodsAvoid direct usage of database tablesAvoid artifacts with high internal complexity
45,68032,25823,67512,885
7,143
s6cur.indd 41 10/4/12 2:35 PM
42 IEEE SoftwarE | www.computEr.org/SoftwarE
FOCUS: Technical DebT
The next step in our exploration of TD-principal is to provide individual ratings for the effort to fi x each of the 1,200+ violations. These effort ratings will be further adjusted by the number of components involved in fi xing the violation and the complexity of each component. This refi nement will make the calculation of TD-principal more granular and could provide better in-dicators of components most in need of refactoring.
references 1. S. McConnell, “Technical Debt,” blog, 1
2. W. Cunningham, “The WyCash Portfolio Management System,” ACMSIGPLANOOPSMessenger, vol. 4, no. 2, 1993, pp. 29–30.
3. B. Curtis, J. Sappidi, and A. Szynkarski, “Esti-mating the Size, Cost, and Types of Technical Debt,” Proc.3rdInt’lWorkshopManagingTechnicalDebt, IEEE CS, 2012, pp. 49–53.
4. D. Spinellis, CodeQuality:TheOpenSourcePerspective, Addison-Wesley, 2006.
8. ISO/IEC9126,SoftwareEngineering—ProductQuality, Int’l Org. for Standardi-zation, 2001.
9. ISO/IECStd.25010,SystemsandSoftwareEngineering—SystemsandSoftwareQualityRequirementsandEvaluation(Square)—Sys-temandSoftwareQualityModels, Int’l Org. for Standardization, 2011.
10. J. de Groot et al., “What Is the Value of Your Software?,” Proc.3rdInt’lWorkshopMan-agingTechnicalDebt, IEEE CS, 2012, pp. 37–44.
11. C. Sterling, ManagingSoftwareDebt:BuildingforInevitableChange, Addison-Wesley, 2011.
bill CUrtiS is senior vice president and chief scientist of CAST Soft-ware and heads CAST Research Labs. His research interests include software productivity and quality, empirical software engineering, organizational maturity models, and a formal proof that the Veer-T triple option can execute in linear time. Curtis received a PhD from Texas Christian University. He’s an IEEE Fellow. Contact him at [email protected].
JAY SAPPidi is the senior director of product marketing at CAST Software and a senior director in CAST Research Labs. His research interests include predictive analytics for software risk management through software analysis, measuring the impact of technical quality on developer productivity, and comparative analysis of application technical quality across technologies. Sappidi received an MBA from the MIT Sloan School of Management. Contact him at [email protected].
AleXAndrA SZYnKArSKi is a research associate in CAST Software’s Research Labs. Her interests include structural quality benchmarks and measuring software performance trends on the global application development community. Szynkarski received an MS in in-ternational business administration from the Institut Administration des Entreprises de Nice. Contact her at [email protected].
ab
OU
T T
he
aU
Th
OR
S
IEEE Software seeks practical, readable
articles that will appeal to experts and nonexperts
alike. The magazine aims to deliver reliable
information to software developers and managers to
help them stay on top of rapid technology change.
Submissions must be original and no more than 4,700
words, including 200 words for each table and fi gure.
