Apr 22, 2023 2

Outline

• Basic Concepts• Software reliability models• Software reliability measure simulation• A project application• Conclusion and future work• Q&A

Apr 22, 2023 Section 1-1 3

What is Software Reliability

• The Probability of failure-free software operations for a specified period of time in a specified environment .

Some relative concepts: Failure, Fault, time Failure functions...

Apr 22, 2023 Section 1-2 4

Failure: It is the departure of the external results of program operation from requirements.Fault: It is the defect in the program that, when executed under particular conditions, causes a failure. It is the cause of failure.

“Failure” is something dynamic. The program has to be executing for a failure to occur.

“Fault” is static. It’s a property of program not a property of its execution behavior. It is created when a programmer makes an error.

Apr 22, 2023 Section 1-3 5

Time: Reliability quantities are defined with respect to time.

We are concerned three kinds of time:execution time: is the CPU time that is actually spent by the computer in executing the software; calendar time: is the time people normally experience in terms of years, months, weeks,etc.; clock time: is the elapsed time from start to end of computer execution in running the software.

Apr 22, 2023 Section 1-4 6

There are four general ways of characterizing failure occurrences in time:

1). Time of failure,

2). Time interval between failures,

3). Cumulative failures experienced up to a given time,

4). Failures experienced in a time interval.

Apr 22, 2023 Section 1-5 7

Failure functions: when a time basis is determined, failures can be expressed in several ways

• The cumulative function (mean value function): denotes theaverage cumulative failures associated with each point of time,• The failure intensity function: represents the rate of changeof the cumulative failure function,• The failure rate function (rate of occurrence of failures): defined as the probability that a failure per unit time occurs in the interval [t,t+t], given that a failure has not occurred before t. We will use failure rate functions for our simulation

Apr 22, 2023 Section 2-1 8

Software reliability modeling• One particular aspect of SRE (Software Reliability

engineering) that has received the most attention.• There are many models have been proposed since 1970s.

The basic idea: A software reliability model describe failures as a random process, which is characterized in either times of failures or the number of failures at fixed times.

Apr 22, 2023 Section 2-2 9

Let N(t) be a random process representing the numberof failures experienced by time t. Then (t), the meanvalue function, is defined as

(t)=E[N(t)], which represents the expected numberof failures at time t. The failure intensity function of the (t) process is theinstantaneous rate of change of the expected failuresnumber with respect to time, it is defined by

dttdt )()(

Apr 22, 2023 Section 2-3 10

Nonhomogeneous Poisson Process (NHPP) Models

• The counting process{N(t),t0} is modeled by NHPP N(t) follows a Poisson distri-bution. The probability thatN(t) is a given integer n is:

• m(t) is called mean value function, it describes the expected cumulative number of failures in [0,t)

,...2,1,0,!)]([})({ )( ne

ntmntNP tm

n

Apr 22, 2023 Section 2-4 11

The general assumptions of NHPP models• N(0) = 0,• {N(t), t0} has independent increments,• P{[N(t+h)-N(t)]=1}=(t)+o(h),• P{[N(t+h)-N(t) 2}=o(h).o(h) denotes a quantity which tends to zero for small h. (t) isthe instantaneous failure intensity is defined as:

.)}()({)(0 t

tNttNPLimtt

Apr 22, 2023 Section 2-5 12

The Goel-Okumoto (GO) model Assumptions• The cumulative number of faults detected at time t follows a Poisson distribution,• All faults are independent and have the same chance of being

detected,• All detected faults are removed immediately and no new faults are introduced. The failure process is modeled by an NHPP model with mean value function (t) given by

;0,0),1()( baeat bt

Apr 22, 2023 Section 2-6 13

• a and b are parameters to be determined by using collected failure data. Note that for this model we have ()= a and (0)=0 a is the final number of faults that can be detected by the test process.• b is a constant of proportionality, can be interpreted as the failure occurrence rate per fault.• The intensity function (t) is the derivative of (t) is then

btabedt

tdt )()(

• The expected number ofremaining faults at time t is

.)1()()()(_

btbt aeeaattNE

Apr 22, 2023 Section 2-7 14

The shape of the intensity function and the meanvalue of the GO model

(t)

(t)t

Apr 22, 2023 Section 2-8 15

S-shaped NHPP model• At beginning of testing, some faults are “covered” by other faults. Removing a detected fault at beginning does not decrease the failure intensity very much.• Software reliability testing usually involves a learning process. Skills and effectiveness improve gradually.

t

(t) (t)=a[1-(1+bt)e-bt]; b>0.

.)1()()( 2 btbtbt teababeebtabdt

tdt

Apr 22, 2023 Section 2-9 16

Markov Models

• A stochastic process {X(t),t0} is said to be a Markov process if its future Development depends only on the present state of the process, that is

P[X(t) x(t)|X(t1) x1,…,X(tn) xn]=P[X(t) x(t)|X(tn) xn], for all t1<t2…<t.

Markov property which has the following simple explanation, Given the present state of the process, its future behavior is independent of the past history of the process. This is the most important feature

Apr 22, 2023 17

The process {N(t), t0} where N(t) is the number of events in a Markov process, such as the number of detected faults in a software context, is called a Markov counting process is the birth-death process for which a so-called birth increases the size of the process by one and a death decreases the size by one.

t

N(t)

0 t1 t2 t3 t4 t5 t6 t7

7

654321

A realization of a Markov counting process

Apr 22, 2023 Section 2-11 18

The Jelinski_Moranda (JM) model

• Assumptions1. The number of initial faults is an unknown but fixed

constant;2. A detected fault is removed immediately and no new fault is introduced;3. Times between failures are independent, exponential distributed random quantities;4. All remaining faults contribute the same amount of the software failure intensity.

Apr 22, 2023 Section 2-12 19

• The main property of the JM model is constant FI between the detection of two consecutive failures.

This is quite reasonable if the software is unchanged and the testing is random and homogeneous.

(i)

i

N

01 N+1N

The failure intensity versus the number

of removed faults

Apr 22, 2023 Section 2-15 20

General model characteristics and limitations• Random Process (both error introduce and run selection process are random)• The failures are independent each other (failure times are independent each other)• with and without repair: two situations in testing phase

Limitations: 1) model’s assumptions 2) future prediction, must noting the environment, using recent data.

Apr 22, 2023 Section 3-1 21

Software reliability simulation

• Present a particular attractive computational alternative for investigating software reliability;

• it can model a wider range of reliability phenomena than mathematical analyses;

• It can provide a “virtual” environment to predict or study software reliability for some software projects;

• averts the need for overly restrictive assumption

Apr 22, 2023 Section 3-2 22

u s in g th e fa ilu re ra te fu n c tion scon s id e rin g th e n u m b er o f fa ilu res

a s toc h as tic series x(t)w h ose b eh avio r d ep en d s on ly on

a ra te fu n c t ion

ra te b as ed s im u la tion

con s id e r m an y asp ec ts o f p rog ram m an y in p u ts

art ifac t b ased s im u la tion

S im u la tion w ays

Apr 22, 2023 Section 3-3 23

Failure rate functions:

(t)=n0e-t , GO model, n0 is the initial failure rate,

is the failure rate decay factor. Inputs parameters; n(t)=0(1-n/n0), JM model, 0n0 is the estimated number

of initial faults, 0 is initial failure rate; (t)=ab2te-bt, S-shaped model, a: expected failure number b: failure detect rate.. In our simulation, including 7 failure rate functions

Apr 22, 2023 Section 3-4 24

Simulation approaches

• Black-box simulation: treat software as a whole, only itsinteractions with the outside world are modeled, not con-cerning the internal structure and component combination.

• White-box simulation: assume the software comprisingof m components, its architecture is specified by the inter-component transition probabilities, denoted by wij (Comp.i

-->Comp.j)

Apr 22, 2023 Section 3-5 25

The basic algorithm of black-box simulationInitialization(set max_time, dt,..)

(t)*dt<a

While (time<max_time)

Produce random number 0<a<1

Num_failur+1,time=time+dt,

N

(t) : value of rate function at that time

Num_failure:

cumulative

number

of failures

Apr 22, 2023 Section 3-6 26

• The input of black-box simulation is a failures indicator for the software, such as: go 130.6 0.0048 “go” indicate using GO model, the latter two real numbers are the first and second parameters. We use CASRE to getthe parameters of failure rate functions.

• The output of the simulation are cumulative failures nu-mbers versus time, and failure intensity.

Apr 22, 2023 Section 3-7 27

In white-box simulation, event (failure) producing algorithm issimilar to black-box. In additional, at each time_step we mustcalculate which component will execute. This calculation isbased on the input file (control-flow matrix). Another inputof white-box simulation failure behavior file

3

0.00 0.80 0.20

070 0.00 0.30

0.60 0.40 0.00

go 130.6 0.0048

jm 63.78 0.3288

ys 88.50 0.0098

3-component control-flow

matrix

3-component failure file

Apr 22, 2023 Section 3-8 28

3-component internal architecture

component1

component2

component3

0.8

0.2

0.7

0.3

0.60.4

Apr 22, 2023 Section 4-1 29

Application for a project

• We have applied the simulation techniques into a project and got some interesting results.

• This is a three successive generations of switching system (TROPICO-R) software. The three products are: PRA, PRB,PRC. Each product has four functions (components) :1)Telephone(TEL): local call processing, charge-metering, etc2) Defense(DEF): on-line testing, traffic measurement, error…3)Interface(INT): communication with local devices…4)Management(MAN): communication with external devices..We have simulated every product and each function using GO,JM, S-shaped models

Apr 22, 2023 Section 4-2 30

PRA_TEL simulation results

0

2 0

4 0

6 0

8 0

10 0

12 0

14 0

0 2 0 4 0 6 0

m on th

cum

ula

tive

fai

lure

s

PRAT da ta

PRAT G O mod elda taPRAT JM mod elda taPRAT S -shap edmo del da ta

Apr 22, 2023 Section 4-3 31

PRB_TEL simulation results

0

10

20

30

40

50

60

70

0 10 20 30 40 50 60

month

cum

ulat

ive

failu

res

PRBT data

PRBT GO modeldataPRBT JM modeldataPRBT S-shapedmodel data

Apr 22, 2023 Section 4-4 32

PRC_TEL simulation results

02468

1012141618

0 10 20 30 40 50 60

month

cum

ulat

ive

failu

res PRCT real data

Simulate data (GOmodel)

Simulate data (JMmodel)

Simulate data (S-shaped model)

Total number of failures is small, JM has better prediction

Apr 22, 2023 Section 4-5 33

PRA and PRB whole system simulation results

PRA system simulation results

0

50

100

150

200

250

300

350

400

450

0 10 20 30 40 50

month

cum

ulat

ive

failu

res

PRAsystemdataGO modeldata

JM modealdata

S-shapedmodel data

PRB system simulation results

0

20

40

60

80

100

120

140

160

0 20 40 60month

cum

ulat

ive

failu

res

PRB systemdata

GO model data

JM model data

S-shaped modeldata

Apr 22, 2023 Section 4-6 34

PRC system simulation results

0

10

20

30

40

50

60

0 20 40 60

month

cum

ulat

ive

failu

res

PRC_SYSTEM data

PRC_SYSTEM GOmodel data

PRC SYSTEM JMmodel data

PRC SYSTEM S-shaped model data

Apr 22, 2023 Section 4-7 35

Some analyses of simulation

• In the case of large number total failure (PRA, PRB), In general, the three models are better fitting and predictions;• For the successive two generations products (PRB,PRC) the S-shaped model has better fitting at early phase of testing;• In the case of small number of total failure (PRC), JM model has better prediction than S-shaped model and GO model.

Apr 22, 2023 Section 4-8 36

PRA system simulation deviation: (simulation value)-(real data value)

PRA simulation deviations(3 models)

-60

-40

-20

0

20

40

60

0 10 20 30

month

devi

atio

n

GO model

JM model

S-shapedmodel

Apr 22, 2023 Section 4-9 37

PRB system simulation deviation

-55

-45

-35

-25

-15

-5

5

15

25

35

0 10 20 30 40month

devi

atio

ns

GO model

JM model

S-shaped model

Apr 22, 2023 Section 4-10 38

PRC system simulation deviation

-15

-10

-5

0

5

10

15

0 10 20 30 40 50month

devi

atio

n

GO model

JM model

S-shapedmodel

Apr 22, 2023 Section 4-11 39

From the Figures of deviation we can see

• There are exist some large deviations at some time point, this indicates that modeling and simulation can catch the general trend or prediction of software reliability measures, and it is difficult to get accurate measures with exact time point,• Around 10th month during the software testing, there is a sensitive transition for fault exposure.

Apr 22, 2023 Section 5-1 40

Conclusion• Combined analytical models with simulation techniques to give effective and practical method for software relia- bility measures• Implemented a rate-based simulator for software reliabi- lity measurements (it is not computation complexity; it enables models combination approach; it takes into

account the internal structure or dependency of software components)• The application for project demonstrate it can be used for analyses, prediction and evaluation in software reliability literature.

Apr 22, 2023 Section 5-2 41

Future work

• To introduce more software environment factors into the simulation process for software reliability measures,• To develop more effective methods or algorithms to simulate or analyze the dependency between components in complex software systems,• To extend the methods and approaches investigated in our work for network (e.g., the Internet) software reliability analyses.

Apr 22, 2023 42

Question and Answer

The End

Thank you very much!