Portugal

Process Improvement in High Maturity

Environments

Improving the Efficiency of Code Reviews

André Ferreira

Software Quality Assurance Engineer

Critical Software S.A.

2013-10-18

2 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

Outline

1. SPI success - evaluation and measurement

2. CMMI High Maturity Processes Areas - a framework to drive

SPI initiatives

3. SPI initiative at Critical Software – Improving the efficiency of

Code Reviews

4. Recommendations/guidelines to improve the probability of

success of SPI initiatives.

3 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI success: reports on SPI Success.

Performance Results of CMMI-Based Process Improvement –Technical Report

CMU/SEI-2006-TR-004 ESC-TR-2006-004

Performance category

Medium Improvement

# Data points

Lowest Improvement

Highest Improvement

Cost 34% 29

3% 87%

Schedule 50% 22 2% 95%

Productivity 61% 20 11% 329%

Quality 48% 34 2% 132%

Customer Satisfaction

14% 7 -4% 55%

Return on Investment

[4.0:1] 22 [1.7:1] [27.7:1]

4 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI success: evaluation of SPI Success - I

Evaluation and validity of SPI success is questionable.

Hard to establish a clear relation of SPI initiative and SPI

results. Results are not questionable by themselves, but a clear

understanding of how success relates to SPI is unclear, mainly as result of:

1. Confounding factors – not considering relevant factors, validity threat

2. Incomplete context descriptions – context of SPI not fully described

3. Evaluation validity – threats to validity not discussed

4. Measurement validity – measurement definition inconsistencies

5. Measurement scope – missing the big picture

Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement -

A Systematic Literature Review. IEEE Transactions on Software Engineering

5 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI success: evaluation of SPI Success - I

Y(SPI sucess) = f ( f1, f2, f,3)

Causation

established

Presence of

confounding factors Y’(SPI sucess) = f’ (?, ?, f1, f2, f3, ?)

6 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

CMMI High Maturity Framework Overview

Support

Pro

cess

Managem

ent

Pro

ject

Managem

ent

CAR OPP

QPM

OPM BO

QPPOs PPBs

Defects

OIP

Corrective Actions Project QPPOs

Performance Issues

7 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative

Improving the Efficiency of

Code Reviews

8 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Opportunity Identification OPM.SP 1.1/OPM.SP1.2

1. Annual Review Business Objectives

Business objective: Improve organizational profit margins

QPPOs: Improve project profit margins – do the

same/better for less

2. Feedback from projects

Issues with review procedure – cumbersome - to much

time managing review records.

Lead to establish a performance baseline of total

appraisal cost for code reviews.

9 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Opportunity Identification OPM.SP1.2/OPM.SP 1.3

Analysis of performance data

Feedback from projects was confirmed to be an

issue – statistical analysis of a control chart.

Code review is a - fat process!

Opportunity identified:

Decrease appraisal cost of performing code

review procedure by:

Improving efficiency (less effort) but control

impact on effectiveness

10 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

(A) Appraisal and

(P) Preventive

costs

(I) Internal

and (E)

External

failure costs

Quality

Co

st

Low

High

100%

good

100%

defective

Total cost of

quality

𝐶𝑜𝑄 = 𝑠𝑢𝑚(𝐼 + 𝐸 + 𝐴 + 𝑃)

SPI Initiative – Potential Improvement I OPM.SP 1.3

Cost of Quality (CoQ) Management model

11 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement I OPM.SP 1.3

BO

QPPOs

Processes

Pro

cess M

anagem

ent

𝐶𝑜𝑄 = 𝑠𝑢𝑚(𝐼 + 𝐸 + 𝐴 + 𝑃)

Project Gross

Margin >= Target

EBT

Rework Verification/ Code reviews

Rationale of impact on QPPOs

Decrease on overall CoQ will

have a positive impact in the

ability of projects to meet

Project Gross Margin (QPPO)

12 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement II OPM.SP 1.3

Hypothesis1: Less effort spent performing code reviews will

contribute to a reduction in total appraisal cost.

Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Co

st

Low

High

100%

good

100%

defective

Total cost of

quality

13 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement II OPM.SP 1.3

Hypothesis1: Less effort spent performing each code review will

contribute to a reduction in total appraisal cost

Rationale of estimation of potential benefit

(1) Assuming a 10x decrease on managing

review records -> x hours saved per review

(2) y number of reviews performed each year

Total benefit (TH1)

TH1(hours) = x*y

TH1(€) = ~4K€/year

14 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement III OPM.SP 1.3

Hypothesis 2: Less effort per code review can be used to

perform additional code reviews (re-invest in Appraisal Cost)

Motivate an increase of # code reviews in projects with little

investment in code reviews

Motivate re-investment of effort saved to perform additional

code reviews (increase code coverage)

...with the final objective of decrease Total Cost of Quality by

decreasing considerably the Internal and External Failure

costs.

15 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement III OPM.SP 1.3

Hypothesis 2: Less effort per code review can be used to

perform additional code reviews (Increase Appraisal Costs)

Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Co

st

Low

High

100%

good

100%

defective

Total cost of

quality

16 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Potential Improvement III OPM.SP 1.3

Estimation of potential benefit Based on performance baselines it takes 5x more to fix a

defect at ST when compared to code reviews

Projects who invest in code reviews, on average, have less

defects/LOC found in ST (corroborates the model) with a

typical distribution of [74%/24%] distribution of code reviews

VS. ST defects

Projects who invest less in code reviews have [5%/94%]

defect distribution

In the scenario of these projects with lower investment in code review

shifting to a bigger investment in code reviews, having an approximation

to a ratio of [74%/24/] we estimated potential savings of 70K€ for a year

timeframe.

TH2(€) = 70K€

17 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – ROI OPM.SP 1.3

Estimation of potential benefit

Hypothesis 1

Tool license cost: 3.6K

Management of SPI initiative:

Deployment + Training: 3K€

ROI [ 0.4:1]

Hypothesis 2

Tool license cost: 3.6K

Management of SPI initiative:

Deployment + Training: 3K€

ROI [ 9.6:1]

ROI =𝑟𝑒𝑡𝑢𝑟𝑛 − 𝑖𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡

𝑖𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡

18 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Elicit Improvements OPM.SP2.1/SP2.2

A target was given to identify approaches that could

decrease in 10x the time to manage code review

records

Several tools were identified and analyzed in a DAR

initiative – ‘smoke test’.

Crucible came as the best option as it enabled to set-up

a review in less than 10 minutes among other relevant

factors related to infrastructure.

19 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Pilot OPM.SP2.3/SP2.4

Criteria of success in supporting H1.

Decrease review setup/manage records is

consistent below 10m/code review

Impact on defect detection ability (def/kloc) is

non negative

Pilot lasted 3 months and involved 5 project and a

workshop with SPA area members.

20 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Pilot Results OPM.SP2.4

Performance baseline with pilot results was created.

Review setup/manage records was under 10m

resulting in a 12x decrease on overall effort – change

in performance tested with statistical

significance.

Defect defection rate has not affected:

when compared to existing performance baseline

(44 defect/KLOC)

Overall, Crucible supported H1 and therefore a base to

approach H2 with a higher degree of confidence.

21 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Initiative – Deployment OPM.SP3.1/SP3.2/SP3.3

Phase currently in progress

– Evaluation of ROI will be over a 1 year period

– Monitoring cost of quality

– For the second year end beyond the NPV will

replace ROI for Cost/Benefit analysis of SPI

initiative

22 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

SPI Recommendations

Seek to establish your Governance Model (GM)

Use/create a GM that supports the rationale of your

network of Objectives vs. Processes

Start small, measure the least as possible, focus on

making the model reliable.

Only with a minimal statistical confidence on cause and

effect you are able to estimate ROI of SPI

Y(Repeatable SPI sucess) = f (f1,f2,f3)

Y’(Risky SPI initiative) = f‘(?, ?, f1, f2, ?)

23 © 2

013 C

MM

I P

ort

ug

al

Confe

rence S

eries –

All

Rig

hts

reserv

ed.

You can contact me at: al-ferreira@criticalsoftware.com

or

csw-quality@criticalsoftware.com

Questions?