Predicting reliability of software systems

under development

A multiple case study of large industrial embedded software projects

Support organizations in making decisions with respect to:

–Optimal allocation of test resources

–Asses the release readiness of software under

development

SRGMs: Software Reliability Growth Models

Objectives

Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

• RQ1: Which SRGMs are best to assist decisions for

optimal allocation of testing resources?

• RQ2: Which SRGMs are best for assessing the release

readiness of a software system?

• RQ3: Does using information from earlier projects

improve release readiness assessment?

• RQ4: How to make the choice of SRGM more

effective?

Research Questions

CASE STUDY DESIGN

Company

(unit of analysis)

Application

domain

Software development process for

studied projects

Volvo Cars

CorporationAutomotive

V-shaped software development mostly

using sub-suppliers for implementation

Ericsson Telecom Agile development, mostly in-house

SAAB EDS Defense EquipmentWaterfall development (old projects)

with development concentrated in-house

Software Development Process

SRGMs: Software Reliability Growth Models

No Model Name Shape Structure Mean Value Function Ref.

1 Musa-Okumoto (MO) Concave NHPP 𝑚 𝑡 = 𝑎 ln(1 + 𝑏𝑡) [28]

2 Goel-Okumoto (GO) Concave NHPP 𝑚 𝑡 = 𝑎 (1 − 𝑒−𝑏𝑡) [29]

3 Inflection-S model S-shaped NHPP 𝑚 𝑡 =𝑎 (1 − 𝑒−𝑏𝑡)

(1 + 𝛽𝑒−𝑏𝑡)

[30]

4 Delayed-S model S-shaped NHPP 𝑚 𝑡 = 𝑎 (1 − 1 + 𝑏𝑡 𝑒−𝑏𝑡 ) [31]

5 Rayleigh model S-shaped NHPP 𝑚 𝑡 = 𝑎 (1 − 𝑒−

𝑡𝑏

2

)[32]

6 Logistic model S-shaped Trend 𝑚 𝑡 =𝑎

(1 + 𝑒−𝑏(𝑡−𝑐))[33]

7 Gompertz model S-shaped Trend 𝑚 𝑡 = 𝑎 𝑒−𝑏𝑒−𝑐𝑡 [34]

8 Linear model Linear Trend 𝑚 𝑡 = 𝑔 ∗ 𝑡 + 𝑐 [27]

Metrics used for evaluation

𝑀𝑆𝐸 =1

𝑛

1

𝑛

𝑌𝑖 − 𝑌𝑖2

𝐵𝑃𝑅𝐸 =𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 − 𝐴𝑐𝑡𝑢𝑎𝑙

𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 + 2𝜂 ∗ (𝐴𝑐𝑡𝑢𝑎𝑙 − 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑);

Mean Square Error (MSE)

Balanced Predicted Relative Error (BPRE)

𝑊ℎ𝑒𝑟𝑒, 𝜂 = 0 𝑖𝑓 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 > 𝐴𝑐𝑡𝑢𝑎𝑙1 𝑖𝑓 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 < 𝐴𝑐𝑡𝑢𝑎𝑙

Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

a) Which SRGMs are best to assist decisions

for optimal allocation of testing resources?

Data

Metrics used for evaluation

Mean Square Error (MSE) Balanced Predicted Relative Error (BPRE)

Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

RQ1: Which SRGMs are best to assist decisions for optimal allocation of testing resources?

RQ2: Which SRGMs are best for assessing the release readiness of a software system?

RQ3: Does using information from earlier projects improve release readiness assessment?

a) Which SRGMs are best to assist decisions

for optimal allocation of testing resources?

b) Which SRGMs are best for assessing the

release readiness of a software system?

c) Does using information from earlier projects

improve release readiness assessment?

Summary of results

Case unit (domain)

Software

development

process

Observed shape

of defect inflow

profile

Recommended SRGMs

For testing

resource(s)

allocation

For release readiness assessment

Only using current

project data

Using historic

information

1. Automotive V-model S-shape, Concave Logistic Logistic Logistic

2. Telecom Lean + Agile Concave, Convex Gompertz Logistic Musa-Okumoto

3. Defense Equip Waterfall S-shape, Concave Logistic Gompertz Logistic

RQ4: How to make the choice of SRGM more effective?

How to make the choice of SRGM more effective?

How to make the choice of SRGM more effective?

How to make the choice of SRGM more effective?

Projects/

Releases

Defect inflow intensity trend until half-way through the project Predicted

shape of

defect inflow

profile

Overall

trend

Trend after

reaching

maximum

Defect inflow intensity trend characteristics

A1, A3, A4

& C1Increasing Decreasing

Defect inflow intensity first increases, maximizes near

to half-way and then decreasesS-shape

B1, B3 &

B4Decreasing Decreasing

Early defects, defect inflow intensity maximum early

then decreases smoothlyConvex

A2, B2, B5

& C2Increasing Increasing

Late defects, defect inflow intensity trend is positive

throughout half-way of project timelineConcave

How to make the choice of SRGM more effective?

Predicted

shape of

defect inflow

profile

Recommended SRGMs

For testing resource(s)

allocation

For release readiness

assessment using

current project data

S-shape Logistic Logistic

Convex Gompertz Gompertz

Concave Delayed-S Logistic

Thank You