1

Aditya P. Mathur

Head and Professor

Department of Computer Science,

Purdue University

ABB, SwedenMonday April 7, 2008

Towards a Radically New Theory of Software Reliability

2

Reliability

Probability of failure free operation in a given environment over a

given time.

Mean Time To Failure (MTTF)

Mean Time To Disruption (MTTD)

Mean Time To Restore (MTTR)

3

Claim

Existing theories of software reliability simplify the

problem to the extent that they (almost) maximize the

uncertainty associated with the estimated software

reliability.

4

Operational profile

Probability distribution of usage of features and/or scenarios.

Captures the usage pattern with respect to a class of customers.

5

Reliability estimation

Operationalprofile

Random or semi-random Test generation

Test execution Failure/Defect data collection

Reliability estimation [Uncertainty evaluation?]

Decision process

6

Issues: Operational profile

Variable. Becomes known only after customers have access to the product. Is a stochastic process…a moving target!

Random test generation requires an oracle. Hence is generally limited to specific outcomes, e.g. crash, hang.

What about an operational profile with impulse? This creates a non-differentiable probability function of the time-to-failure.

7

Issues: Failure data

Should we analyze the failures?

If yes then after the cause is removed then the reliability estimate is invalid.

If the cause is not removed because the failure is a “minor incident” then the reliability estimate corresponds to irrelevant incidents.

8

Issues: Failure rate

“That is, the failure rate, when unambiguously defined, does not have a physical reality; rather, it is a technical device, whose sole purpose is to convey the engineer’s personal opinion about the life characteristic of software.”

Nozer Singpurwalla, “The failure rate of software: does it exists?”, IEEE Transactions on Reliability, vol. 44, no.

3,1995.

9

Issues: Model selection

Rarely does a model fit the failure data.

Model selection becomes a problem. 200 models to choose from? New ones keep arriving!

Markov chain models suffer from a lack of estimate of transition probabilities.

To compute these probabilities, you need to execute the application.

During execution you obtain failure data. Then why proceed further with the model?

10

Issues: Markovian models

Markov chain models suffer from a lack of estimate of transition probabilities.

To compute these probabilities, you need to execute the application.

During execution you obtain failure data. Then why proceed further with the model?

C1

C3

C212

13 32

21

12 + 13=1

11

Issues: Assumptions

Software does not degrade over time; memory leak is not degradation and is not a random process; a new version is a different piece of software.

Reliability estimate varies with operational profile. Different customers see different reliability.

Can we not have a reliability estimate that is independent of operational profile?

Can we not advertise quality based on metric that are a true representation of reliability..not with respect to a subset of features but over the entire set of features?

12

Estimating Uncertainty

Estimates of software reliability must the associated with uncertainty. But how to quantify uncertainty?

Entropy based approach [Katerina et al. 2002] Moments based approach [Katerina et al. 2003] Monte Carlo approach [Katerina et al. 2003] Bayesian approach [Dai et al. 2007]

13

Estimating Uncertainty

Basic idea:

Model the parameters as random variables. Use statistical (e.g. moments) or Simulation approaches to

estimate variance.

Problem: Does not correlate with likely faulty components in the program

under test.

14

Sensitivity of Reliability to test adequacy

Coverage

low

low

high

high

Desirable

Suspect modelUndesirable

Risky

Rel

iabi

lity

Problem with existing approaches to reliability estimation.

15

Basis for an alternate approach

Why not develop a theory based on coverage of testable items and test adequacy?

Testable items: Variables, statements,conditions, loops, data flows, methods, classes, etc.

Pros: Errors hide in testable items.

Cons: Coverage of testable items is inadequate. Is it a good predictor of reliability?

Yes, but only when used carefully. Let us see what happens when coverage is not used or not used carefully.

16

Saturation Effect

FUNCTIONAL, DECISION, DATAFLOWAND MUTATION TESTING PROVIDETEST ADEQUACY CRITERIA.

Reliability

Testing EffortTrue reliability (R)Estimated reliability (R’)Saturation region

Mutation

Dataflow

Decision

Functional

RmRdfRdRf

R’f R’d R’df R’m

tfs tfe tds tde tdfs tdfe tms tfeu:uncertainty

u1

u2

u3

u4

17

An experiment [TeX]

Tests generated randomly exercise less code than those generated using a mix of black box and white box techniques. Application: TeX. Creator: Donald Knuth. [Leath ‘92]

18

An experiment [sort utility]

UNIX sort utility [DelFrate et al. 1995]

19

An experiment [coverage-reliability correlations]

Unix utilities and space application [Garg 1995. MS Thesis]

20

Modeling an application

OSComponent

Component

Component

Interactions

Component

Component

Component

Interactions

Component

Component

Component

Interactions

……….

21

Reliability of a component

R(f)= (covered/total), 0<<1.

Reliability, probability of correct operation, of function f based on a given finite set of testable items.

Issue: How to compute ?

Approach: High correlation between coverage metrics and failures has been established via empirical studies. Such studies could provide estimate of and its variance for different sets of testable items.

22

Reliability of a subsystem

R(C)= g(R(f1), R(f2), ..R(fn), R(I))

C={f1, f2,..fn} is a collection of components that collaborate with each other to provide services.

Issue 1: How to compute R(I), reliability of component interactions?

Issue 2: What is g ?

Issue 3: Theory of systems reliability creates problems when (a) components are in a loop and (b) are dependent on each other.

23

Scalability

Is the component based approach scalable?

Powerful coverage measures lead to better reliability estimates whereas measurement of coverage becomes increasingly difficult as more powerful criteria are used.

Solution: Use component based, incremental, approach. Estimate reliability bottom-up. No need to measure coverage of components whose reliability is known.

24

Next steps

Develop component based theory of reliability.

Do experimentation with large systems to investigate the applicability of the their and its effectiveness in predicting and estimating various reliability metrics.

Base the new theory on existing work in software testing and reliability.

25

The Future

Apple Confidence: 0.999

Level 0: 1.0

Level 1: 0.9999

Level 2: 0.98

Boxed and embedded software with independently variableLevels of Confidence.

Mackie Confidence: 0.99

Level 0: 1.0

Level 1: 0.9999

26

Select References

F. Del Frate, P. Garg, A. P. Mathur, and A. Pasquini. On the Correlation Between Code Coverage and Software Reliability, Proceedings of the Sixth International Symposium on Software Reliability Engineering, IEEE Press,Toulouse, France, pp 124-132, October 24-27, 1995

S. Krishnamurthy and A. P. Mathur. On the Estimation of Reliability of a Software System Using Reliabilities of its Components, Proceedings of the 8th International Symposium on Software Reliability Estimation, Albuquerque, New Mexico, November 1997.

M. H. Chen. A. P. Mathur, and V. J. Rego. A Case Study To Investigate Sensitivity Of Reliability Estimates To Errors In The Operational Profile, Proceedings of the Fifth International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Monterey, California, November 6-9, 1994, pp 276-281.

Katerina Goseva–Popstojanova and Sunil Kamavaram. Assessing Uncertainty in Reliability of Component–Based Software. Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03), 2003.

Yuan-Shun Dai and Min Xie and Quan Long and Szu-Hui Ng. Uncertainty Analysis in Software Reliability Modeling by Bayesian Analysis with Maximum-Entropy Principle, IEEE Trans. Softw. Eng.,V 33, No. 11, 2007, pp 781--795.

P. Garg. On code coverage and software reliability. MS Thesis. Department of Computer

Science, Purdue University. May 1995.