Defect Metrics

Defect Management & Metrics

Society for Software Quality

April 24, 2010

021209.01SSQ Presentation © 2010 – Holl Consulting

Who Is This Guy?• 30 years in software

and hardware/software Quality

• 19 years in Quality & Process Management

• 6 years as Director of Software Best Practices and Corporate Software Metrics Leader at Intuit

• Past SSQ PresidentQUALI YT

QSS


Primary Relevant Experience• Designed Intuit’s corporate defect

management process

• Assisted in rollout to ~1800 engineers

• Guided its evolution for six years

• As Corporate Software Metrics Leader, fostered quantitative project management and process improvement

• Made mistakes and learned from them


Agenda

• Summary of Previous Talk

• Defect Management Metrics

• Metrics for Project Management

• Metrics for Process Improvement

• Advanced Metric Models


Last time…

• We reviewed cultural issues related to defect management

• We went over a defect management process:

– Policies

– Workflow/States

– Fields


Data Collection

• At different states in the workflow, different data are collected

• Most data are collected at submit time

• Additional data are entered by the developer upon resolution

• And finally by the tester upon verification


Agenda

Summary of Previous Talk

• Defect Management Metrics





When Does A Defect Count?It depends on your goals• Defect Management: To track defects for resolution in

order to release products– Just log functional problems found in test; deviations from the

requirements that the customer will see; defects that you don’t want to fall through the cracks.

• Defect Analysis: To improve the process or predict– If your goal is to get better (learn) or predict the future, you need

to capture much more information.

– Without early lifecycle data (requirements, etc.) predictive models aren’t that reliable

World-class companies don’t just manage defects – they analyze information.


Data Collection Costs• Estimate Assumptions:

– 18,000 defects

– ½ the defects are corrected (9,000)

– Data collection time: 10 minutes (validated at HP)

– Average defect cost: 16 hours (data from Michael Fagan)

– Data collection reduces defects 5%

Time to record (18k *10m) 3,000 hrs

Time to correct (9k *16h) 144,000 hrs

Defect reduction (9k * 5%)

450 defects

Time savings (450 * 16h) 7,200 hrs

ROI 240%

Sampling is valid, butsampling has to be managed to get good random samples and to ensure that adequate (statistically significant) samples are being collected.

This doesn’t quantify the value of being able to predict quality and schedules


Simple Measures

• Defect cost =

• Fix rate =

• Duration =

Last two include queue time, so they depend on workloadIt’s helpful to understand the standard deviation tooBonus: By type, root cause, injection phase

* Excel NetWorkDays is a useful function

Σ (Total Resolution Effort)

Number of resolved defects


Period of time (week, month, etc.)

Σ((Closed Date) – (Submit Date))*



Relative Defect Costs

• If you record injection and discovery phases and effort, you can compute relative costs

• Example:

– Cost of a requirements defect = 1

– Cost of a design defect = 10

– Cost of a coding defect = 100

– Cost of a production defect = 1000


Similar to Phase Containment• Start with matrix of injection and discovery

• Enter the average rework (or duration) costs in each cell (resolved defects only)

Injection Phase

Discovery Phase

Req's Design Code Test Production

Requirements 2.4

Design 15.3 4.1

Code 22.7 9.5 1.2

Testing 43.8 18.5 4.3 1.2

Production 85.3 38.5 18.3 2.2 1


Similar to Phase Containment

• Normalize by dividing each cell by the Injection=Discovery value

Injection Phase

Discovery Phase

Req's Design Code Test Production

Requirements 1.00

Design 6.38 1.00

Code 9.46 2.32 1.00

Testing 18.25 4.51 3.58 1.00

Production 35.54 9.39 15.25 1.83 1.00


Logging Defects• This is not

How many defects leak?

• This is

What percent of known defects are recorded?

• If you only start logging defects in test, you’re missing over half of your data

• Try to capture data from reviews/inspections and developer unit tests


Prioritizing Defects• This is not about how to set priority, but…

• Priority means the most important defects get resources first

• To measure this, we need to know the queue time (from “submit” to “work starts”)

• Therefore, we need an “Accepted” state, or something to note when work began

• Why can’t we use Submit to Verify duration?


Aging Defects

• Time in Resolver queue

• Time in Verifier queue

• Alternately, use durations minus effort as a proxy

Average Queue Time

0

2

4

6

8

10

12

0-TestStopper

1 2 3 4 5

Priority

Day

s in

Qu

eue


Defect Backlog• Many groups don’t count defects left over from

prior releases; they only focus on defects found and not fixed in the current project

• This is dangerous because a product defect backlog can grow unmanaged

• ALL of the open defects are released, not just the new ones

• When producing the report of open defects, query for all open defects in the product


The CalculationWeek

1

2

3

4

Total number of defects

– Number closed before the week started

– Number that were opened after the week ended

= Number open in the week


Calculating Backlog in Excel

• Query Submit Date and Closed Date


Calculating Backlog in Excel

This uses named ranges, where

D.Num.CRs = Total number of CRs opened

D.T.Closed = Range of Closed Dates

D.Submit.Date = Range of Submit Dates

Array formula. <Shift-Ctrl-Return>


The Result

Product Defect Backlog

0

200

400

600

800

1000

12001/

7/20

06

2/7/

2006

3/7/

2006

4/7/

2006

5/7/

2006

6/7/

2006

7/7/

2006

8/7/

2006

9/7/

2006

10/7

/200

6

11/7

/200

6

12/7

/200

6

1/7/

2007

2/7/

2007

3/7/

2007

4/7/

2007

5/7/

2007

6/7/

2007

7/7/

2007

8/7/

2007

9/7/

2007

10/7

/200

7

11/7

/200

7

12/7

/200

7

1/7/

2008

2/7/

2008

3/7/

2008

4/7/

2008

# O

pen

Def

ects


Agenda


Defect Management Metrics





Some Project Management Defect Metrics

• Reviews & Inspections

• Defect Discovery & Closure

• Defect Density


A Project Axiom

• The quality of product in one phase is an indicator of the quality in later phases

• Why?

–There are two answers


Review Metrics

• Reviews & inspections have proven to be the most cost-effective way to detect and remove defects

• Inspect your key project artifacts (requirements, designs, code, tests, etc.)


Review/Inspection Data

• There are many data collected during a review or inspection

• The ones that tell us about quality are:

– Number of Operational defects discovered

– Number of Minor defects found

per changed line of inspected doc/code

• Number of reviews per doc/module


Project Defect Density

• Consistency in categorizing inspection defects is important

– Have clear guidelines for Operational and Minor defects

• Defect Density Project =

Operational Defects found by inspection

Delta SLOC (or LOT)


Projecting Defect Leakage• Inspected total of 5340 SLOC

• Found 47 operational defects

• Density = 47/5340 = 8.8 defects/KSLOC

• Project changed 18,463 SLOC

• Estimated defects = 18.463 * 8.8 = 162

• Remaining = 162 - 47 = 115

What should you do? Inspect more? Or Test for them?


Verification

• Use Defect Density to estimate expected yield

– Similar to estimate described with reviews

• Defect discovery & closure

• Use extrapolation to predict when defect exit criteria will be met


Defect Discovery

Time

Cum

ulat

ive

Def

ects

Fou

nd

Total Defects Opened

Ideal Curve!

Weekends


Defect Discovery Curve Assumptions

• Time is a proxy for “exposure”

– Remove weekends to smooth the curve

• Number of test cases run would be better, assuming

– Test coverage is good

– Not continuing to add new test case


Well-Behaved Curve

Time

Cum

ulat

ive

Def

ects

Fou

nd


Release Target Date


Add: Defect Closure

Time

Cum

ulat

ive

Def

ects

Total Defects Opened & Closed

Opened

Closed

Remaining Open


Add: To QA

Time

Cum

ulat

ive

Def

ects

Total Defects Opened, Closed, & To QA

Opened

To QA

Closed

Opened, waiting for development fix

Turned over from development to QA, awaiting verification


Finding The Balance

Time

Cum

ulat

ive

Def

ects

Total Defects Opened, Closed, & To QA

Opened

To QA

Closed

What’s happening

here?

QA is a bottleneck

Dev is a bottleneck


Add: Legacy & Target Release

Time

Cum

ulat

ive

Def

ects

Total Defects Opened & Closed

Total

Targeted

To QA

Closed

Inherited from previous history, targeted for or found in the project/release


Extrapolation to Release Date

Time

Cum

ulat

ive

Def

ects

Project Defects

Total

Targeted

To QA

Closed

Release Target Date

Should I be concerned?


Extrapolation to Release Date

Time

Cum

ulat

ive

Def

ects

Project Defects

Total

Targeted

To QA

Closed

Release Target Date

When will the project be done?


Computing in Excel

Query defects into Excel

Convenient: Use named ranges for data


Computing in Excel• In a second sheet, create a date column and count

the defects submitted and closed for each date.

Array formula. <Shift-Ctrl-Return>


Real Life ExampleAardvark 4.7 Defect Discovery & Closure

(With extrapolations)

0

50

100

150

200

250

300

350

400

450

5006/

17/0

8

6/24

/08

7/1/

08

7/8/

08

7/15

/08

7/22

/08

7/29

/08

8/5/

08

8/12

/08

8/19

/08

8/26

/08

To

tal C

Rs

0

10

20

30

40

50

60

70

80

90

100

Fo

un

d in

4.7

Σ Open

Σ Closed

Σ Found in 4.7

6/16 = Start of testing

Target Release

Date

2nd order polynomial fit


Release Criteria: Defects

• Goals:

– Provide good-enough quality to customers

• Don’t release any “bad” defects

• Don’t release “too many” defects

– Minimize support costs

• Customers will need support proportional to product quality


Defect Discovery

Time

Cum

ulat

ive

Def

ects

Fou

nd


Release Target Date

Should I release?


Bad Defects• Bad is a function of

– Severity = the consequences of the failure

– Probability that it will occur

– The number of customers likely to experience it

– May also include “important” customers or functions

• These are often complex to calculate, so most companies just use Severity, tempered by other factors


Quality Threshold• No more than the following number of open

defects:

– Where New = defects discovered in the current project, and

– Old = defects discovered in previous releases

Severity* New Old

Critical 0 0

Important 5 10

Moderate 30 100

Trivial 50 200

Release team (stakeholders) need to determine values based on history. The goals are:• Not to release any Critical defects• Not to introduce “too many” new defects• Manage the backlog


What’s Good Enough?

The two market leaders – Quicken TurboTax, produced by Intuit, and TaxCut, produced by H&R Block

It also offers better end-of-return reviews …

TurboTax offers better videos than TaxCut …


What’s Good Enough?TaxCut offers easier tagging of items you’d like to deal with later.

It also appears to be more scrupulously proofread than TurboTax: Twice in TurboTax’s screens in my tests, incorrect information was displayed, though the tax returns themselves were unaffected.


What’s Good Enough?

… if you’re choosing which program to go with for the first time this year, I’d go with TaxCut – solely because of the text errors in TurboTax.

I have a higher standard for accuracy in tax programs than I do for any other type of software, and while the errors we found didn’t affect the bottom line, they did affect my confidence in the program’s makers.

It’s unfortunate, because I felt TurboTax offered better features.


Detroit Free Press

> 1.7M Readers+ web site+ wire serviceThe article came from the Monterey Peninsula Herald


Predicting Defects• There are several models for predicting

defects

• The predictions of interest are:

– How many defects are going to be created?

– When will I find them?

– How long will it take to fix them?

– When will I be done?

– How good will the result be?


Using Defect Density• Phase Defect Density

DDPhase =

• At first this may not seem right, but it’s a reasonable predictor

• Which is more reliable?

– 67 specification defects per 200 pages of documentation (0.34 defects/page)

– 67 specification defects per 4300 SLOC (15.6 defects/KSLOC)

Defects InjectedPhase

Total Project Affected SLOC


Using Defect Density• Project/Process Defect Density

– Number of defects peraffected SLOC

– Affected SLOC can be difficult to get without a good tool.

– The best I’ve found is CodeReports (aka SourceROI)

– http://www.polariguana.com/sourceroi.php

Release A Release B

Defects 452 563Total SLOC 8,600 43,500Defect Density 52.6 12.9

Project Changed SLOC

5,400 4,350

Project Defect Density

83.7 129.4


Calculating DD per Phase

# Defects Injected Per Phase Project KSLOC

Number of Defects Injection Phase

Discovery Phase Req's Design Code Test Production

Requirements 46 0 0 0 0

Design 8 26 0 0 0

Code 83 19 83 0 0

Testing 33 22 37 2 0

Production 3 2 16 0 1

Total Injected 173 69 136 2 1

Defect Density 4.94 1.97 3.89 0.06 0.03

Project SLOC = 35,000Note: if this is only test defect data, you’re missing an opportunity to get better information.

DDPhase =


Next Project• Estimated to be 20,000 SLOC

Historical Defect

Density

Likely to

Inject

Phase Containment

Likely to Leak

Effort to Resolve*

Total Rework

Requirements 4.94 99 26.6% 26 85.3 2242.2

Design 1.97 39 17.3% 7 38.5 263.3

Code 3.89 78 62.1% 48 18.3 882.9

Testing 0.06 1 81.7% 1 2.2 2.1

Production 0.03 1 100.0% 1 1 0.6

Total 218 Grand Total 3391.0

* Worst case cost (defects discovered in test)


Defect Discovery & Closure

• Predictions can bemade in Excel by extrapolating andseeing where the discovery and close lines meet

Cumulative Defects Submitted

0

50000

100000

150000

200000

250000

300000

10/1

4/200

6

10/2

8/200

6

11/1

1/200

6

11/2

5/200

6

12/9

/200

6

12/2

3/200

6

1/6/

2007

1/20

/200

7

2/3/

2007

2/17

/200

7

3/3/

2007

3/17

/200

7

3/31

/200

7

4/14

/200

7

4/28

/200

7

5/12

/200

7

5/26

/200

7

6/9/

2007

6/23

/200

7

7/7/

2007

7/21

/200

7

8/4/

2007


Extrapolation

• This method is quick and in some cases surprisingly good

• But not in all cases


Other Excel Extrapolations

• Adding a trend line uses all the data

• You may only want to use some of the data

• Create a parallel column of the data you want to extrapolate and extrapolate only that curve


Example


Simple Linear Extrapolation

• To create a linear extrapolation on the last few data points, start by extending the date column:



• Select the cells that look linear

• Copy and PasteSpecial - Values



• Then drag the cells down to the last extrapolated date

• Excel willdo a linearextrapolationof the data


Extrapolation• In order for extrapolations to be reliable,

the discovery rate must have peakedWeekly Defects Submitted

0

10

20

30

40

50

60

1/1

/20

08

1/1

5/2

00

8

1/2

9/2

00

8

2/1

2/2

00

8

2/2

6/2

00

8

3/1

1/2

00

8

3/2

5/2

00

8

4/8

/20

08

4/2

2/2

00

8

5/6

/20

08

5/2

0/2

00

8

6/3

/20

08

6/1

7/2

00

8

7/1

/20

08

7/1

5/2

00

8

7/2

9/2

00

8

8/1

2/2

00

8

8/2

6/2

00

8

9/9

/20

08

9/2

3/2

00

8

10

/7/2

00

8


Done?

• What does it mean to be done (from a defect perspective)?

– No “bad” defects remain

– Predictable/manageable defects remain

• Two categories of defects remain

– Those you know

– Those you don’t know


Latent Defects

• Latent defects (unknown at release time) can be predicted using defect density and Rayleigh PDF

• What needs to happen before the predictions become reliable?

– The defect discovery rate must flatten

d f(t) dt

= 0


Discovery RateCumulative Defect Submitted

0

100

200

300

400

500

600

1/1/

2008

1/15

/200

8

1/29

/200

8

2/12

/200

8

2/26

/200

8

3/11

/200

8

3/25

/200

8

4/8/

2008

4/22

/200

8

5/6/

2008

5/20

/200

8

6/3/

2008

6/17

/200

8

7/1/

2008

7/15

/200

8

7/29

/200

8

8/12

/200

8

8/26

/200

8

9/9/

2008

9/23

/200

8

10/7

/200

8

Weekly Defects Submitted

0

10

20

30

40

50

60

1/1/

2008

1/15

/200

8

1/29

/200

8

2/12

/200

8

2/26

/200

8

3/11

/200

8

3/25

/200

8

4/8/

2008

4/22

/200

8

5/6/

2008

5/20

/200

8

6/3/

2008

6/17

/200

8

7/1/

2008

7/15

/200

8

7/29

/200

8

8/12

/200

8

8/26

/200

8

9/9/

2008

9/23

/200

8

10/7

/200

8


Fitting Data to a Model• Fitting data

to a curve helps us tell the rate is decreasing


Using Excel

• As before, create a linear extrapolation of the last <criteria> days:

• “Flattening” criteria example:

– The discover rate must less than 5 defects per day in the past two weeks of testing


Using Excel

• Add linear trend line

• Select “Display equation on chart”


Using Excel• This gives the slope of the line m,

where y=mx + b

– In this case, m = 2.2 defects/day

– Not good enough

Cumulative Defect Submitted

y = 2.163x - 85395

0

100

200

300

400

500

600

1/1/

2008

1/15

/200

8

1/29

/200

8

2/12

/200

8

2/26

/200

8

3/11

/200

8

3/25

/200

8

4/8/

2008

4/22

/200

8

5/6/

2008

5/20

/200

8

6/3/

2008

6/17

/200

8

7/1/

2008

7/15

/200

8

7/29

/200

8

8/12

/200

8

8/26

/200

8

9/9/

2008

9/23

/200

8

10/7

/200

8


Latent or Known Defect Risk• Probability of occurrence in production

– Risk = Severity * Probability

• Need to have a decent measure/estimate of probability

– Need a usage profile

• Also, frequency does not always equal value

– I may not use the “Restore” function very often, but it better work


And Finally…

• Track latent/released defects carefully

• They are your best indicator of Quality


Release Defect DensityFrom:

Software Quality Measurement:A Framework for Counting Problems and Defects

CMU/SEI-92-TR-022

http://www.sei.cmu.edu/pub/documents/92.reports/pdf/tr22.92.pdf


Agenda



Metrics for Project Management




A True Story

• A while back, I looked in the mirror and decided I needed to loose some weight.

• What is the first thing I’m going to do?


My First Step


My First Step – On A Scale

• If I don’t measure myself today, I won’t know

– If the changes I’m planning to make are taking me the right direction

– Which changes have the best affect

– How far I’ve come

– When I’m done

• It’s important to establish a baseline


What Are The Goals?

`Would you tell me, please, which way I ought to go from here?'

`That depends a good deal on where you want to get to,' said the Cat.

`I don't much care where--' said Alice.

`Then it doesn't matter which way you go,' said the Cat.


Closure Data Collection

Submit reason

• Test case failed• Pass criteria unclear• Tested wrong version• Setup/config wrong• Data-related error• Other

No

• Correction made• Duplicate• Change not justified

Injection Phase

Root Cause

Defect?

Resolution

Yes

Classification, Mode

Corrected Component(s)

This is where we record what we’ve learned

Total Resolution Effort

Resolved


Defect Classification Models

• Industry guidelines to standardize information collection

• The most prevalent are from:

– IEEE

– IBM

– Hewlett Packard


HP Defect Origins, Types, and Modes


Defect Leakage Reasons1. Environmental Dependency

2. Test Not Run

3. Test Case Error

4. Data Dependency

5. Inconsistent Symptoms

6. Test Case Success Criteria Unclear

7. Test Execution Error

8. Test Case Results Not Verified

9. Code Changed After Test Passed

10. No Test Case/Coverage

11. Configuration Management Problem

12. Requirements Missing/Changed


Using Leakage Data• For each defect found in production, log a leakage

reason

• Use the frequency (volume) to improve the related process

• Resist temptation to weight the frequency by something important

– This assumes cause & effect, and there is none

– This is different from root cause, where the cost to repair is related to the type of defect


Specific Process Improvement Metrics

• Two types of process metrics:

1. Metrics that monitor a process (tell you the effect of changes)

• Basic SPC-type charts

2. Metrics that tell you what to do

• Phase Containment

• Root Cause Analysis

• Test Coverage

• Defect Leakage Reasons

• Static Code Analysis - Complexity


Should all projects be measured?• Pros:

– More data points

– Evaluate everything we’re doing

• Cons:– Projects may use different processes

– Variation may be too great

• Suggestions:– Don’t measure projects – measure processes

– Only measure your best projects

– Tailoring is okay, but separate out the data


Hold Periodic Process Reviews

• Monthly Quality Reviews focused on

– Process Improvement initiatives

– Measures & Analysis/Interpretation

– Lessons Learned

• Rules of metrics etiquette:

– Preview data with stakeholders = no surprises


Phase Containment

• Identifies the lifecycle activities that leak defects

• Tells you where to improve defect detection efforts

Phase Containment

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Requirements Design Code Build Testing

% D

efe

cts

Co

nta

ine

d


Root Cause Analysis

• Pareto the types of errors being created, weighted by cost (effort)

Root Cause Total Hours

0

500

1000

1500

2000

2500

3000

Req

uire

men

ts A

mbi

guou

s/va

gue

Req

uire

men

ts M

issi

ng

Logi

c flo

w

Mis

sing

logi

c

Req

uire

men

ts C

hang

ed

Wro

ng v

aria

ble

used

Oth

er

Cas

e st

atem

ent

Typo

Spe

cific

atio

n M

issi

ng

Gar

bage

col

lect

ion

Exc

eptio

n ha

ndlin

g

Par

amet

er m

ism

atch

Req

uire

men

ts W

rong

Uni

nitia

lized

var

iabl

e

Bad

val

idat

ion

chec

k

Mis

sing

end

sta

tem

ent

Env

not

mat

ch t

est

Spe

cific

atio

n-re

quire

men

t m

ism

atch

Cod

ing

stan

dard

not

fol

low

ed

Spe

cific

atio

n A

mbi

guou

s

Env

cap

acity

Spe

cific

atio

n W

rong

Wro

ng m

etho

d ca

lled

Des

ign

logi

c w

rong

Des

ign-

spec

mis

mat

ch

Req

uire

men

ts I

ncon

sist

ent

Env

har

dwar

e fa

ilure

Doc

typ

o

Doc

vag

ue

Env

Ope

ratio

nal E

rror

Inst

all m

issi

ng f

iles

Req

uire

men

ts O

ther

Doc

info

mis

sing

Inst

all w

rong

con

figur

atio

n

Inst

all i

ncom

plet

e

Doc

wro

ng

Inst

all w

rong

ver

sion

Root Cause x Rework Hours


Agenda



Metrics for Project Management

Metrics for Process Improvement



Defect Probabilities• Weibull distributions are useful engineering

models, where the tail approaches zero asymptotically

• Weibull Probability Density Function (PDF) is:

Where m determines the shape of the curve, andc is the scale parameter

tc

mt

m

etc

m

-

f(t) =tc

mc

m-1

etc

m

-

=

(Wikipedia)


Different Weibull Distributions

Weibull Curves

0

1

2

3

4

0 1 2 3

m=0.5

m=1 m=2 m=4

m=10

Weibull curves with different shapes (values of m) can be used to model various systems (probability distributions)


Weibull Probability Distribution

Weibull Curvesm = 0.1 to 5.0 in .1 increments

0.0

0.5

1.0

1.5

2.0

0.0 0.5 1.0 1.5 2.0

Rayleigh (m=2)


Weibull Curves

Weibull Curvesm = 0.1 to 5.0 in .1 increments

0.0

0.5

1.0

1.5

2.0

0.0 0.5 1.0 1.5 2.0

Rayleigh (m=2)

(almost) Normal (m=3.4)


Rayleigh Probability Distribution Function

• Weibull distribution with shape (m) = 2

• Let x = 2t

tc

2t

2

etc

2

-

f(t) =2tc2

etc

2

-

=

xc2

ex2c

2

-

f(x) =xc2

ex2

4c2-

=


Rayleigh Probability Distribution Function

xc2

ex2

4c2-

f(x) =Rayleigh Curves

c = 0.1 to 0.5

0

1

2

3

4

5

6

7

8

9

0.0 0.5 1.0 1.5

c = 0.1

c = 0.2

c = 0.3


Excel Formula


Applying Rayleigh PDF• Let tm = time at which the curve reaches its peak

• To find the peak, we set = 0

• Solving gives tm =

• To scale the area under the curve, multiply by K

• Substitute c = tm 2

d f(t) dt

c

2


Fitting The Rayleigh PDF

2tc2

etc

2

-

f(t) = K 2t

(tm 2)2e

t(tm 2)

2

-

= K

2t2t2

me

- t2

2t2m= K

1tm

te- t2

2t2mf(t) = K

2

If we can estimate how many defects we’ll find (K) and when we’ve hit the peak (tm), we can create a PDF for our project!


Building Your Estimate


Predicting Latent Defects• At peak, the area under the Rayleigh curve is 39.35%

• This means 60.65% of the defects are yet to find

• Example:– At peak you found a cumulative total of 58 defects.

– Total = (58/39.5)*100=147.4

– 147 - 58 = 89 remaining.

Latent Defects = K dt 1tm

te- t2

2t2m

2

Release Date

1 year?


Easier that Calculus

• In Excel, run a column for cumulative total and a column for remaining defects

• From your release date you can easily estimate latent defects


Cumulative Rayleigh Model

tc

2

-

CDF = 1 - e


Rayleigh Assumptions

• The defect rate observed during the development process is positively correlated with the defect rate in the field.

• Given the same error injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages.


Exponential Models

• Exponential Model (Weibull, m=1)

1ce

tc

-

f(t) = ExponentialWeibull, m = 1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0.0 0.5 1.0 1.5 2.0

• Error detection rate• Instantaneous failure rate• Hazard rate

• In general, this is a reliability growth model


Fitting A Model to Data• EasyFit from www.mathwave.com


Model Summary

• Rayleigh is Weibull distribution with m=2

• Other values of m produce other probability distributions

• There are many models; try several that fit

• “All models are wrong. Some models are useful.”

– George E. Box


Summary

• A defect management program includes

– A balanced solution

– Training & explaining

– Metrics that are used

Questions?

Chris [email protected]

Next Class: Tools for Metrics

Date post:	26-Nov-2014
Category:	Documents
Upload:	niteen-kumar
View:	194 times
Download:	2 times

Defect Metrics

Documents