Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1

transcript

Outsourcing Test activities

How to prove cost reductionsHarold van Heeringen, Metrics Consultant

Sogeti

Wednesday May 28, Milan, Italy

Outline

• Problem definition• Factors that influence Test

Performance• Model CASE• Results & Conclusions

Problem definition

• More and more outsourcing test activities

• Contracts - x% of cost reductions for the client in 1,2,3 years

• So far: Impossible to prove!>Compare bills ?>What about volume increase/decrease ?>What about defects found… and defects not

found?

Test activities

Functional design

Technical design

Coding + Unit test

Systems test

User Acceptance Test

Client

Sogeti

Client

Factors that influence Test Performance

• The most important factors:>Number of defects found (in relation to

number of defects present)Many defects found – did we test well?Few defects found – did we test badly?

>Number of hours spentWe have spent 1.000 hours testing this project. Is that

>Time interval availableBuild phase ended late, but end date is fixed. What

can we do in the time interval available?

The relation between these factors

• Extra short duration...>How many extra

hours do we need?

• Extra long duration...>How many extra

defects will we find?>How many defects

are not found?

Impossib

Unpractical

DurationD

When do we test well?

• More defects found than expected?• Less hours spent than expected?

• But how do we know the number of defects and hours expected??

• We need an estimation tool!

Defects found < expected

Defects found = expectated

Defects found > expected

hours spent < expected

? + ++hours spent = expected

- ± +hours spent > expected

-- - ?

QSM SLIM Estimate

• Project Estimation> Hours, duration, fte, costs, etc

> But also: defects !!!

Benchmark test performance

• Benchmark supplier performance against client performance – but how??>Every project is different >Duration is critical variable>Test hours and defects found are highly

dependent on the Build performance

• How can we know the client’s performance?

• Hours spent• Defects found• Defects not found

Model proposed: 3 stages

>1: Baseline analysis – assess the client situation

• PI (productivity duration)• Cost per test hour• Defect Tuning factor

>2: Operational execution – Estimate, measure and benchmark performance

• Simulate project with actual duration, but with client baseline parameters

• Measure actuals• Calculate Cost reductions

>3: Evaluation – Evaluate whether the cost reduction targets have been met

Model step 1: Baseline analysis

• Select 4 – 5 representative projects>Size the projects in COSMIC/FPA/slocs>Administrate results in SLIM Datamanager

• Results:>Detailed analysis per project>SLIM Estimate WBS>Average/median Productivity index>Defect tuning factor>Costs per test hour

CASE: Baseline analysis

• Contract: outsourcing systems testing • Target: 8% Cost reductions 1st year

• Baselinedata:

• 1 test hour = 80 euro> activities?> Fixed costs?> Waiting hours?

Project Size in

Hours S-test

Defects S-test

Defects A-test /1st M production

A 310 554 12 8

B 512 937 27 5

C 212 508 22 12

D 224 732 52 18

E 487 878 35 8

Model step 2: Operational execution

• Choice: Measure everything, or measure predefined set of projects1 – Measure size 2 – Estimate project with client parameter

set and actual duration3 – Compare client estimation with Sogeti

actual data4 – Calculate Test Performance5 – Calculate Cost Reduction

CASE: Operational execution

Step 1 - 5 projects selected to measureStep 1 - Size measurement in COSMICStep 2 - Estimation in SLIM, using client baseline parameters and actual durationProject Q

Scenario Defects S-test FS

Defects A-test + 1M Prod. FAP

Defects Total FT

Client baseline

30 31 61 0,49

Scenario Test hours TU

Hour rate R

Costs C (TU*R)

Client baseline

2.045 80 163.600

Step 3 – Compare Sogeti actuals against baseline scenarioProject Q

Scenario Test hours TU

Hour rate R

Costs C (TU*R)

Client baseline

2.045 80 163.600

Sogeti actuals

1.800 82 147.600

Scenario Defects S-test FS

Defects A-test + 1M Prod. FAP

Defects Total FT

Client baseline

30 31 61 0,49

Sogeti actuals

35 30 65 0,53

Step 4 – Assess Test Performance projects> Outsourcing Effectiveness

Did we test better than the client would have done?

> Outsourcing EfficiencyDid we test more efficiently than the client would have done?

Outsourcing Efficiency OF

Baseline C – Actual C Baseline C

Outsourcing Effectiveness OV

(Actual FS/FT) – (Baseline FS/FT) Baseline (FS/FT)

• Project Q>Outsourcing Effectiveness:

OV = (0.53-0.49)/0.49 ≈ 8,2 %

>Outsourcing Efficiency: OF = (163.600–147.600) / 147.600 ≈ 9,7%

Outsourcing effectiveness

Outsourcing efficiency

- 100 %

100 % 0 %

- 100 %

No Cost Reduction

Cost Reduction

• Step 5 – Calculate Cost Reduction>Maybe client rates OV higher than OF or

vice versa>Contract – agree on weight factors

OV = Outsourcing EffectivenessOF = Outsourcing Efficiencyw1 = weight factor OVw2 = weight factor OF

Cost Reduction CR

(w1*OV) + (w2*OF) w1+w2

Model step 3: Evaluation

• After contract period: Analyze results

Outsourcing effectiveness

Outsourcing efficiency

- 100 %

100 % 0 %

- 100 %

No Cost Reduction

Cost Reduction

CASE: Evaluation

• Evaluation client X, 1st year

• Total CR: € 129.143 / € 1.383.570 ≈ 9,3%• Substantial cost reductions, even more than

was promised!!

Project OV OF C Baseline CR % CR €

Q 9,7% 8,2% € 163.600 8,95% € 14.642

R 12,3% 6,1% € 312.150 9,2% € 28.718

S 8,2% 3,0% € 122.650 5,6% € 6.868

T 6,4% 13,3% € 718.120 9,85% € 70.735

U 15,6% 8,8% € 67.050 12,2% € 8.180

Total € 1.383.570 € 129.143

First results

• Clients reactions are very positive• Model should be evaluated in

practice and should be developed further

• Please help us further develop the model:>Metrieken.sogeti.nl>sec@sogeti.nl

Conclusions

• The model proposed is objective and can be used in all organizations and all projects

• The client can now see the real cost reductions they achieve by outsourcing

• The supplier can also use the model as a commercial instrument and is able to support claims of cost reduction based on experience

metrieken.sogeti.nl

Smef 2008 Harold Van Heeringen Outsourcing Test Activities How To Prove Cost Reductions V1

Documents