Advanced Software Engineering: Software TestingCOMP 3705(Lecture 2)
Sada Narayanappa
Anneliese Andrews (Chair DU)Thomas ThelinCarina Andersson
Adv. Software Engineering: Sada Narayanappa 2
News & Project
♦News♦ Updated course program♦ Reading instructions♦ Questions?
♦Project Option 1♦ IMPORTANT to read the project description thoroughly♦ Schedule, Deadlines, Activities♦ Requirements (7-10 papers), project areas♦ Report, template, presentation
♦Project Option 2: date!
Adv. Software Engineering: Sada Narayanappa 3
Lecture
♦Some more testing fundamentals♦Chapter 4 (Lab 1)
♦Black-box testing techniques♦Chapter 12 (Lab 2)
♦Statistical testing♦Usage modelling♦Reliability
Adv. Software Engineering: Sada Narayanappa 4
Terminology
♦ Unit testing: testing a procedure, function, or class.♦ Integration testing: testing connection between units and
components.♦ System testing: test entire system.♦ Acceptance testing: testing to decide whether to purchase
the software.
Adv. Software Engineering: Sada Narayanappa 5
Terminology (2)
♦Alpha testing: system testing by a user group within the developing organization.♦Beta testing: system testing by select customers.♦Regression testing: retesting after a software modification.
Adv. Software Engineering: Sada Narayanappa 6
Test Scaffolding
Allows us to test incomplete systems.♦Test drivers: test components.♦Stubs: test a system when some components it
uses are not yet implemented.Often a short, dummy program --- a method with an
empty body.
Adv. Software Engineering: Sada Narayanappa 7
Test Oracles
♦ Determine whether a test run completed with or without errors.
♦ Often a person, who monitors output.♦ Not a reliable method.
♦ Automatic oracles check output using another program.♦ Requires some kind of executable specification.
Adv. Software Engineering: Sada Narayanappa 8
Testing Strategies:Black Box Testing
♦Test data derived solely from specifications.Also called “functional testing”.
♦Statistical testing.Used for reliability measurement and prediction.
Adv. Software Engineering: Sada Narayanappa 9
Testing Theory:Why Is Testing So Difficult?
♦Theory often tells us what we can’t do.♦Testing theory main result: perfect testing is impossible.
Adv. Software Engineering: Sada Narayanappa 10
An Abstract View of Testing
♦Let program P be a function with an input domain D (i.e., set of all integers).
♦We seek test data T, which will include selected inputs of type D.♦T is a subset of D.♦T must be of finite size.Why?
Adv. Software Engineering: Sada Narayanappa 11
We Need a Test Oracle
♦Assume the best possible oracle --- the specification S, which is function with input domain D.
♦On a single test input i, our program passes the test whenP(i) = S(i)Or if we think of a spec as a Boolean function that
compares the input to the output: S(i, P(i))
Adv. Software Engineering: Sada Narayanappa 12
Requirement For Perfect Testing [Howden 76]
1. If all of our tests pass, then the program is correct. ∀x[x ∈ T ⇒ P(x) = S(x)]
⇒∀y[y ∈ D ⇒ P(y) = S(y)] • If for all tests t in test set T, P(t) = S(t), then we
are sure that the program will work correctly for all elements in D.
• If any tests fail we look for a bug.
Adv. Software Engineering: Sada Narayanappa 13
Requirement For Perfect Testing
2. We can tell whether the program will eventually halt and give a result for any t in our test set T.∀x[x∈ T ⇒ “∃ a computable procedure for
determining if P halts on input x”]
Adv. Software Engineering: Sada Narayanappa 14
But, Both Requirements Are Impossible to Satisfy.
♦1st requirement can be satisfied only if T= D.We test all elements of the input domain.
♦2nd requirement depends on a solution to the halting problem, which has no solution.We can demonstrate the problem with Requirement 1 [Howden 78].
Adv. Software Engineering: Sada Narayanappa 15
Other Undecidable Testing Problems
♦Is a control path feasible? Can I find data to execute a program control path?
♦Is some specified code reachable by any input data?These questions cannot, in general, be answered.
Adv. Software Engineering: Sada Narayanappa 16
Software Testing Limitations
♦There is no perfect software testing.♦Testing can show defects, but can never show correctness.We may never find all of the program errors during testing.
Adv. Software Engineering: Sada Narayanappa 17
Why test techniques?♦Exhaustive testing (use of all possible inputs and conditions) is impractical
♦ must use a subset of all possible test cases♦ want must have high probability of detecting faults
♦Need processes that help us selecting test cases♦ Different people – equal probability to detect faults
♦Effective testing – detect more faults♦ Focus attention on specific types of fault♦ Know you’re testing the right thing
♦Efficient testing – detect faults with less effort♦ Avoid duplication♦ Systematic techniques are measurable
Adv. Software Engineering: Sada Narayanappa 18
Dimensions of testing
♦Testing combines techniques that focus on♦Testers – who does the testing♦Coverage – what gets tested♦Potential problems – why you're testing (risks / quality)♦Activities – how you test♦Evaluation – how to tell whether the test passed or
failed♦All testing should involve all five dimensions
♦Testing standards (e.g. IEEE)
Adv. Software Engineering: Sada Narayanappa 19
Black-box testing
Ie
Input test data
OeOutput test results
System
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
Adv. Software Engineering: Sada Narayanappa 20
userqueries
numericaldata
output format requests
responsesto prompts
command key input
mouse picks on menuPartitioning is basedon input conditions
Equivalence partitioning
Adv. Software Engineering: Sada Narayanappa 21
Equivalence partitioning
Sy stem
Ou tputs
Invalid in pu ts Valid in pu ts
Input condition:♦is a range
♦ one valid and two invalid classes are defined
♦requires a specific value ♦ one valid and two invalid
classes are defined♦is a boolean
♦ one valid and one invalid class are defined
Adv. Software Engineering: Sada Narayanappa 22
Test Cases
♦Which test cases have the best chance of successfully uncovering faults?
♦ as near to the mid-point of the partition as possible♦ the boundaries of the partition and
♦Mid-point of a partition typically represents the “typical values”♦Boundary values represent the atypical or unusual values♦Usually identify equivalence partitions based on specs and experience
Adv. Software Engineering: Sada Narayanappa 23
♦Consider a system specification which states that a program will accept between 4 and 10 input values (inclusive), where the input values must be 5 digit integers greater than or equal to 10000♦What are the equivalence partitions?
Equivalence Partitioning Example
Adv. Software Engineering: Sada Narayanappa 24
Example Equivalence Partitions
Between 10000 and 99999Less than 10000 More than 99999
999910000 50000
10000099999
Input values
Between 4 and 10Less than 4 More than 10
34 7
1110
Number of input values
Adv. Software Engineering: Sada Narayanappa 25
userqueries
numericaldata
output format requests
responsesto prompts
command key input
mouse picks on menu
outputdomain
Boundary value analysis
Adv. Software Engineering: Sada Narayanappa 26
Boundary value analysis
♦Range a..b ⇒ a, b, just above a, just below b♦Number of values: max, min, just below min, just above max♦Output bounds should be checked♦Boundaries of externally visible data structures shall be checked (e.g. arrays)
Component–1 0…99 100 -10 –9…499 500–1 0…99 100 -10 –9…499 500
Adv. Software Engineering: Sada Narayanappa 27
Some other black-box techniques
♦Risk-based testing, random testing♦Stress testing, performance testing♦Cause-and-effect graphing♦State-transition testing
Adv. Software Engineering: Sada Narayanappa 28
Black Box Testing:Random Testing
♦Generate tests randomly.♦“Poorest methodology of all” [Meyers].♦Promoted by others.♦Statistical testing:
♦Test inputs from an operational profile.♦Based on reliability theory.♦Adds discipline to random testing.
Adv. Software Engineering: Sada Narayanappa 29
Black Box Testing:Cause-Effect Analysis
♦Rely on pre-conditions and post-conditions and dream up cases.♦Identify impossible combinations.♦Construct decision table between input and output conditions.
Each column corresponds to a test case.
Adv. Software Engineering: Sada Narayanappa 30
Error guessing
♦Exploratory testing, happy testing, ...♦Always worth including♦Can detect some failures that systematic techniques miss♦Consider
♦ Past failures (fault models)♦ Intuition♦ Experience♦ Brain storming♦ ”What is the craziest thing we can do?”♦ Lists in literature
Adv. Software Engineering: Sada Narayanappa 31
Black Box Testing:Error Guessing
♦“Some people have a knack for ‘smelling out’errors” [Meyers].♦Enumerate a list of possible errors or error prone situations.♦Write test cases based on the list.Depends upon having fault models, theories on the causes and effects of program faults.
Adv. Software Engineering: Sada Narayanappa 32
Usability testing
Characteristics
♦ Accessibility
♦ Responsiveness
♦ Efficiency
♦ Comprehensibility
Environments
♦ Free form tasks
♦ Procedure scripts
♦ Paper screens
♦ Mock-ups
♦ Field trial
Adv. Software Engineering: Sada Narayanappa 33
Specification-based testing
♦Formal method♦Test cases derived from a (formal) specification (requirements or design)
Specification Test case generation
Test execution
Model (state chart)
Adv. Software Engineering: Sada Narayanappa 34
Model-based Testing
Specification
Top-levelDesign
DetailedDesign
Coding
Unit Test
Integration
VALIDATION
Test phase
Requirements
Model
Usage
Adv. Software Engineering: Sada Narayanappa 35
Need For Test Models
♦ Testing is a search problem.♦ Search for specific input & state combinations that cause
failures.♦ These combinations are rare.
♦ Brute force cannot be effective.Brute force can actually lead to overconfidence.
♦ Must target & test specific combinations.♦ Targets based on fault models.♦ Testing is automated to insure repeatable coverage of targets:
“target coverage”.
Adv. Software Engineering: Sada Narayanappa 36
Model-Based Coverage
♦ We cannot enumerate all state/input combinations for the implementation under test (IUT).
♦ We can enumerate these combinations for a model.♦ Models allow automated target testing.
“Automated testing replaces the tester’s slingshot with a machine gun.”
“The model paints the target & casts the bullets.”
Adv. Software Engineering: Sada Narayanappa 37
Test Model Elements
♦Subject: the IUT.♦Perspective: focus on aspects likely to be buggy
based on a fault model.♦Representation: often graphs (one format is UML)
or checklists.♦Technique: method for developing a model and
generate tests to cover model targets.
Adv. Software Engineering: Sada Narayanappa 38
The Role of Models in Testing
♦Model validation: does it model the right thing?♦Verification: implementation correct?
Informal: checklistFormal: proof
♦Consistency checking: is representation instance of meta model?
♦Meta-model: UML, graphs, etc + technique♦Instance model: representation constructed
Adv. Software Engineering: Sada Narayanappa 39
Models Roles in Testing
♦Responsibility-based testing.Does behavior conform to the model representation?
♦Implementation-based testing.Does behavior conform to a model of the
implementation.♦Product validation.
Does behavior conform to a requirements model, for example, Use Case models?
Adv. Software Engineering: Sada Narayanappa 40
Models That Support Testability
♦Represent all features to be tested.♦Limit detail to reduce testing costs, while preserving essential detail.♦Represents all state events so that we can generate them.♦Represents all state & state actions so that we can observe them.
Adv. Software Engineering: Sada Narayanappa 41
Model Types
♦Combinational models:♦Model combinations of input conditions.♦Based on decision tables.
♦State Machines:♦Output depends upon current & past inputs.♦Based on finite state machines.
♦UML models: model OO structures.
Adv. Software Engineering: Sada Narayanappa 42
Combinational Model - Spin Lock [Binder Fig. 6.1]
8
7
6
5
4
3
2
0
9
Adv. Software Engineering: Sada Narayanappa 43
Combinational Model: Use a Decision Table
♦1 of several responses selected based on distinct input variables.♦Cases can be modeled as mutually exclusive Boolean expressions of input variables.♦Response does not depend upon prior input or output.
Adv. Software Engineering: Sada Narayanappa 44
Decision Table Construction
1. Identify decision variables & conditions.2. Identify resultant actions to be selected.3. Identify the actions to be produced in response to
condition combinations.4. Derive logic function.
Combinational Models
Adv. Software Engineering: Sada Narayanappa 45
Auto Insurance Model
♦Renewal decision table: Table 6.1.♦Column-wise format: Table 6.2.♦Decision tree: Fig. 6.2.♦Truth table format: Fig. 6.3.♦Tractability note:
♦Decision table with n conditions has a maximum of 2 variants.
♦Some variants are implausible.
Combinational Models
Adv. Software Engineering: Sada Narayanappa 46
Insurance Renewal Decision Table[Binder Table 6.1]
yesyes0Any5 or >7noyes200>252 to 46
noyes400<262 to 45
nono50>2514
noyes100<2613
nono25>2502
nono50<2601
CancelSend WarningPrem. Incr.Age# ClaimsVariant
Action SectionCondition Sec.
Adv. Software Engineering: Sada Narayanappa 47
Yes
YesNoNoNoNoNoNoCancel
NoYesYesNoYesNoNoSendwarning
0200400501002550Premium increase($)
ActionSection
Any26 or older
25 or younger
26 or older
25 or younger
26 or older
25 or younger
Insuredage
5 ormore2 to 42 to 41100
Number of Claims
ConditionSection
7654321
Variant
Insurance Renewal Decision Table Column-wise Format
Adv. Software Engineering: Sada Narayanappa 48
Implicit Variants
Variants that can be inferred, but not given.♦ In the insurance example, we don’t care about age if
there are five or more claims. The action is to cancel for any age.
♦Other implicit variants are those that cannot happen ---one cannot be both under 25 yrs old and over 25.
Adv. Software Engineering: Sada Narayanappa 49
Test Strategies
All explicit variants at least once– When decision boundaries are systematically exercised– weak, if can’t happen conditions or undefined
boundaries result in implicit variants
Adv. Software Engineering: Sada Narayanappa 50
Non binary Variable Domain Analysis
♦Exactly 0 claims, age 16-25♦Exactly 0 claims, age 26-85♦Exactly 1 claim, age 16-25♦Exactly 1 claim, age 26-85♦Two, 3, or 4 claims, age 16-25♦Two, 3, or 4 claims, age 26-85♦Five to 10 claims, age 16-85
Adv. Software Engineering: Sada Narayanappa 51
NoNoNoNoNoCancel
NoNoNoYesNoSend warning
25505010050Premium increase
V2AcceptRejectAcceptRejectV3AcceptExpected result
202020InTypical
26Off
25On<= 25
15OffInsuredAge
16On>= 16
0000InTypical
-1Off(below)
1Off(above)
NumberOf Claims
0On==0
54321Variant 1 Boundaries
Test Cases
Adv. Software Engineering: Sada Narayanappa 52
Additional Heuristics
♦Vary order of input variables♦Scramble test order♦Purposely corrupt inputs
Adv. Software Engineering: Sada Narayanappa 53
Statistical testing /Usage based testing
Reliability estimation
Failure logging
Test execution
Test case generation
Usage specification
Test Case
1.1.3Setup1.1.4Call
Failure Report
#13
Output failure
Item 1
Item 2
Item 3
Up
Dow
n SelectShown
Hidden
Up
Up
Dow
nD
own
Select
SelectInvoke
Down
Adv. Software Engineering: Sada Narayanappa 54
Usage specification models
<test_case> ::= <no_commands> @ <command> <select>;
<no_commands> ::= ( <unif_int>(0,2) [prob(0.9)] | <unif_int>(3,5) [prob(0.1)]);
<command> ::=(<up> [prob(0.5)] | <down> [prob(0.5)]);
State hierarchy model
Grammar modelAlgorithmic models
Usage
Usertype
Usertype
User User User
Servic e Service Se rvice Service
Subtype
SubtypeSub
type
User
Service
Adv. Software Engineering: Sada Narayanappa 55
Usage specification models
Markov modelOperational profile
Domain based models
UserSelect menu
Function x
Function y
Normal use
System mode ZItem 1
Item 2
Item 3
10%
90%
25%
50%
25%
40%
30%
30%
Item 1
Item 2
Item 3
Up
Dow
n SelectShown
Hidden
Up
Up
Dow
nD
own
Select
SelectInvoke
Down
Adv. Software Engineering: Sada Narayanappa 58
Statistical testing / Usage-based testing
Usage model
Code
Random sample
Adv. Software Engineering: Sada Narayanappa 59
Usage Modelling
♦Each transition corresponds to an external event♦Probabilities are set according to the future use of the system♦Reliability prediction
Main Window
DialogBox
Right-click
Invoke
Terminate
Move
ResizeCANCEL or OKwith Valid Hour
Click on OK withnon-valid hour
Close Window
Adv. Software Engineering: Sada Narayanappa 60
Markov model
♦ System states, seen as nodes
♦ Probabilities of transitions
Conditions for a Markov model:
♦ Probabilities are constants♦ No memory of past states From
Node
Transition matrixTo Node
P4400P41N4
P34P330P31N3
P240P22P21N2
P14P13P12P11N1
N4N3N2N1
N1 N2
N3N4
P12
P13
P14
P34
P24P31
P41P21
Adv. Software Engineering: Sada Narayanappa 61
Model of a program
♦ The program is seen as a graph♦ One entry node (invoke) and one exit node (terminate)♦ Every transition from node Ni to node Nj has a
probability of Pij♦ If no connection between Ni and Nj, then Pij= 0
N1 N2
N3N4
P12
P13
P14
P34
P24P31
P21
Output
Input F
Adv. Software Engineering: Sada Narayanappa 62
Clock SoftwareClock
Options
24 Aug 1997
AnalogDigitalClock OnlySecondsDateChange Time/Date…Info…Exit
Change Time/Date
current time: 11:10:27a new time: ________
current date: Sat 24 Aug 1996 new date: ___________
OK
Info
Vaporware Clock, version 1.0
OK
Adv. Software Engineering: Sada Narayanappa 63
Input Domain – Subpopulations
♦Human users – keystrokes, mouse clicks♦System clock – time/date input♦Combination usage - time/date changes from the OS while the clock is executing
♦Create one Markov chain to model the input from the user
Adv. Software Engineering: Sada Narayanappa 64
Operation modes of the clock
♦Window = {main window, change window, info window}♦Setting = {analog, digital} ♦Display = {all, clock only}♦Cursor = {time, date, none}
Adv. Software Engineering: Sada Narayanappa 65
State of the system
♦A state of the system under test is an element of the set S, where S is the cross product of the operational modes.♦States of the clock
{main window, analog, all, none}{main window, analog, clock-only, none}{main window, digital, all, none}{main window, digital, clock-only, none}{change window, analog, all, time}{change window, analog, all, date}{change window, digital, all, time}{change window, digital, all, date}{info window, analog, all, none}{info window, digital, all, none}
Adv. Software Engineering: Sada Narayanappa 66
Top Level Markov Chain
Window operational mode is chosen as the primary modelingmode
te rm ina ted no tinvoked
m a inw indow
in fow indow
changew indow
m a inw indow
changew indow
invoke(prob=1)
op tions .in fo
op tions.change
op tions.ex it(p rob=1 /3 )
(p rob=1 /3 )
(p rob=1 /3 )ok (p rob=1)
end (p rob=1)
Rules for Markov chainsEach arc is assigned a probability between 0 and 1 inclusive,The sum of the exit arc probabilities from each state is exactly 1.
Adv. Software Engineering: Sada Narayanappa 67
Top Level Model – Data Dictionary
The info window will disappear and the main window will be given the focus
Press the ok button on the info windowok
The change window will disappear and the main window will be given the focus
Choose any action and return to the main window
end
The software will terminate, end of test caseSelect the “Exit” option from the “Options” menu
options.exit
The title bar must be on to apply this inputThe info window should appear and be given the focusTester should verify window appearance and modality and ensure that it accepts no illegal input
Select the “Info...” item from the “Options” menu
options.info
All window features must be displayed in order to execute this commandThe change window should appear and be given the focusTester should verify window appearance and modality and ensure that it accepts no illegal input
Select the “Change Time/Date...” item from the “Options” menu
options.change
Main window displayed in fullTester should verify window appearance, setting, and that it accepts no illegal input
Invoke the clock software invoke
Comments/Notes for TesterInput to be AppliedArc Label
Adv. Software Engineering: Sada Narayanappa 68
Level 2 Markov Chain
options.info
options.change
invoke
ok
end
options.exit
digital, all digital,clock only
analog, all analog,clock only
double-click
double-click
options.secondsoptions.date
options.clock-only
options.digital
options.secondsoptions.date
options.clock-only
options.analog
Submodel for the Main Window
Adv. Software Engineering: Sada Narayanappa 69
•The change window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated byissuing either an options.analog or an options.digital input
Choose any action (cancel or change the time/date) and return to the main window
end
•The software will terminate, end of test caseSelect the “Exit” option from the “Options” menu
options.exit
•The title bar must be on to apply this input•The info window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input
Select the “Info...” item from the “Options” menu
options.info
•All window features must be displayed in order to execute this command•The change window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input
Select the “Change Time/Date...” item from the “Options” menu
options.change
•Main window displayed in full•Invocation may require that the software be calibrated by issuing either an options.analog or an options.digital input•Tester should verify window appearance, setting, and ensure thatit accepts no illegal input
Invoke the clock software invoke
Comments/NotesInput to be AppliedArc Label
Data Dictionary – Level 2
Adv. Software Engineering: Sada Narayanappa 70
•The clock face should be replaced by the entire clock window
Double click, using the left mouse button, on the face of the clock
double-click
•The date should be toggled either on or off depending on its current status
Select the “Date” item from the “Options” menu
options.date
•The second hand/counter should be toggled either on or off depending on its current status
Select the “Seconds” item from the “Options” menu
options.seconds
•The clock window should be replace by a display containing only the face of the clock, without a title, menu or border
Select the “Clock Only” item from the “Options” menu
options.clock-only
•The analog display could be replaced by a digital display
Select the “Digital” item from the “Options” menu
options.digital
•The digital display should be replaced by an analog display
Select the “Analog” item from the “Options” menu
options.analog
•The info window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated by issuing either an options.analog or an options.digital input
Press the ok button on the info window
ok
Comments/NotesInput to be AppliedArc Label
Data Dictionary – Level 2
Adv. Software Engineering: Sada Narayanappa 71
Level 2 Markov Chain
op tions .change
end
tim e da tem ove
ed it tim e
m ove
ed it da te
Submodel for the Change Window
Adv. Software Engineering: Sada Narayanappa 72
Data Dictionary
•The valid input format is shown on the screenChange the date in the “new date” field or enter an invalid date
edit date
•The valid input format is shown on the screenChange the time in the “new time” field or enter an invalid time
edit time
•Tester should verify cursor movement and also verify both options for moving the cursor
Hit the tab key to move the cursor to the other input field or use the mouse to select the other field
move
•The change window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated by issuing either an options.analog or an options.digital input
Choose either the “Ok” button or hit the cancel icon and return to the main window
end
•All window features must be displayed in order to execute this command•The change window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input
Select the “Change Time/Date...” item from the “Options” menu
options.change
Comments/Notes for TesterInput to be AppliedArc Label
Adv. Software Engineering: Sada Narayanappa 73
Failu
re R
ate
Time
♦Techniques♦ Markov models♦ Reliability growth models
Software Reliability
Adv. Software Engineering: Sada Narayanappa 74
Dimensions of dependability
Dependability
Availability Reliability Security
The ability of thesystem to deliver
services whenrequested
The ability of thesystem to deliver
services as specified?
The ability of thesystem to operate
without catastrophicfailure
The ability of thesystem to protect itelfagainst accidental ordeliverate in trusion
Safety
Adv. Software Engineering: Sada Narayanappa 75
Costs of increasing dependability
Cost
Low Medium High Veryhigh
Ultra-high
Dependability
Adv. Software Engineering: Sada Narayanappa 76
Availability and reliability
♦Reliability♦The probability of failure-free system operation over a
specified time in a given environment for a given purpose
♦Availability♦The probability that a system, at a point in time, will be
operational and able to deliver the requested servicesBoth of these attributes can be expressed
quantitatively
Adv. Software Engineering: Sada Narayanappa 77
Term DescriptionSystem failure An event that occurs at some point in time when
the system does not deliver a service as expectedby its users
System error Erroneous system behaviour where the behaviourof the system does not conform to itsspecification.
System fault An incorrect system state i.e. a system state thatis unexpected by the designers of the system.
Human error ormistake
Human behaviour that results in the introductionof faults into a system.
Reliability terminology
Adv. Software Engineering: Sada Narayanappa 78
Usage profiles / Reliability
Removing X% of the faults in a system will not necessarily improve the reliability by X%!
Possibleinputs
User 1
User 3User 2
Erroneousinputs
Adv. Software Engineering: Sada Narayanappa 79
Reliability achievement
♦Fault avoidance♦Minimise the possibility of mistakes♦Trap mistakes
♦Fault detection and removal♦ Increase the probability of detecting and correcting
faults♦Fault tolerance
♦Run-time techniques
Adv. Software Engineering: Sada Narayanappa 80
Reliability quantities
♦Execution time ♦ is the CPU time that is actually spent by the computer in
executing the software
♦Calendar time♦ is the time people normally experience in terms of years, months,
weeks, etc
♦Clock time♦ is the elapsed time from start to end of computer execution in
running the software
Adv. Software Engineering: Sada Narayanappa 82
Nonhomogeneous Poisson Process (NHPP) Models
♦N(t) follows a Poisson distribution. The probability that N(t) is a given integer n is:
,...2,1,0,!)]([})({ )( === − ne
ntmntNP tm
n
♦m(t) = μ(t) is called mean value function, it describes the expected cumulative number of failures in [0,t)
Adv. Software Engineering: Sada Narayanappa 83
The Goel-Okumoto (GO) model
Assumptions♦The cumulative number of failures detected at time t follows a Poisson distribution♦All failures are independent and have the same chance of being detected♦All detected faults are removed immediately and no new faults are introduced♦ The failure process is modelled by an NHPP model with mean value function μ(t) given by:
;0,0),1()( >>−= − baeat btμ
Adv. Software Engineering: Sada Narayanappa 84
Goel-Okumoto
The shape of the mean value function (μ(t)) and the intensity function (λ(t)) of the GO model
μ(t)
λ(t)t
Adv. Software Engineering: Sada Narayanappa 85
S-shaped NHPP model
♦ μ(t)=a[1-(1+bt)e-bt], b>0
t
μ(t)
.)1()()( 2 btbtbt teababeebtabdt
tdt −−− =−+==μλ
Adv. Software Engineering: Sada Narayanappa 86
The Jelinski-Moranda (JM) model
Assumptions1. Times between failures are independent, exponential
distributed random quantities2. The number of initial failures is an unknown but fixed
constant3. A detected fault is removed immediately and no new fault is
introduced4. All remaining faults contribute the same amount of the
software failure intensity