Software Reliability

Risks of faulty software

Example: – Therak 25, – AT&T network failure– Airport traffic control

Costs of software errors can be huge– Y2K problem detection, fix, litigation

Testing

Testing is the process of searching for errors, with the goal of finding as many as possible so that they can be turned over to those responsible for fixing them

Types of errors

Syntax errors - typos, easy to fix Validity errors - wrong result Verification errors - undetected bad input Run-time errors - General Protection Fault Maintenance errors

– one fix produces another problem

Cost to repair errors – tied to ease of identification

Two approaches to testing

Black-box testing– Only inputs and outputs are tested– Code is ignored (black box)

Glass-box testing– Testing based on enumerated special cases

and anticipated possible inputs– Must examine code to see what conditions it

expects to see

Unit vs. System Testing

Design->implementation-> unit testing-> system testing

Unit testing– Testing individual components of the system– Easy to isolate bugs this way– Based on the program design

System testing– After units check out do a ‘system build’– Based on the program specifications

Unit testing

Unit testing handles the individual components one at a time.

In OOD the units to be tested are the various classes

It is important to catch bugs at this level, rather than later on after the system has been integrated.

System integration

After unit testing is finished you are ready for a ‘system build’

You are looking for errors in the overall operation of the system.

Follow the specifications that preceded the design.

Acceptance testing

This form of system testing is overseen by the client to insure that the program meets their specs.

Functional testing - testing to demonstrate that the function of each specification is met.

System testing components

Look at specifications for the end product Some errors may require redesign Functional testing

– Specifying required functions which must work

Beta testing– Software release to ‘guinea pigs’

Test plans

Used by trained testing teamsScientific method

– (hypothesis verification)Test cases

– 1. Typical ones– 2. Extreme cases– 3. Invalid input

Example

Specifications:– Write a program called Max that will

read in a sequence of integers and print out the largest one entered.

A test plan for Max should handle all types of errors.

Test plan for Max program

Test Input Output Type

1 2 38 2 938 20 47 2 -109 0 38 938 1

2 100 29 28 10 4 -3 84 17 -9 100 2

3 938 2847 17273 2867 238 19288 19288 2

4 -1 -2 -3 -4 -5 -6 -7 -1 2

5 1 1 1 1 1 1 1 1 1 1 1 2

6 2.5 0.9 1 2 3 error 3

7 error 3

Unit testing techniques

Drivers– Small programs designed to test a function– A driver program calls the function with a

set of test data, and checks to see if the function processes it correctly.

– A suitable value is returned by the function, but the driver program really does not need it

Test driver for max function(Part 1 of 2)

#include <iostream.h> int max(int a[], int n); // defined in cx3-3.cpp; link with the driver

int main() { int a[100], i; cout << "Max driver” << endl; cout << "Enter numbers terminated by -9999”; cout << endl; cout << "Length of input must be <= 100”; cout << endl;

Test driver for max function(Part 2 of 2)

for (i = 0; i < 100; i++) { int val; cin >> val; if (val == -9999) // sentinel break; else a[i] = val; } cout << "\nMax is " << max(a, i) << endl; cout << endl << endl; return 0;}

Stubs

A stub is– a function that does nothing more than give

calling routines what they ask for.

– Used to test a function that depends on other functions that are not yet available.

Sample stub for function median

#include <iostream.h> int median(int a[], int n) { cout << "function median called with n = " << n <<; cout << “a[] = "; int i; for (i = 0; i < n; i++) cout << a[i] << '\t'; cout << "\nType in a value to return:"; int return_value; cin >> return_value; return return_value; }

Alternatives to testing

The problem with testing is that you cannot test all possible situations.

So you can never ‘prove’ that the program works.

A better way of testing would be using a mathematical proof of the programs’ correctness

Problems with proofs

However, proving that a program will perform correctly is not the same as a guarantee that it will do what you want it to.

We can show that the program specifications are being met, but we cannot show that a specification itself is correct.

Example

We could show that a vending machine program meets the spec. of dispensing the fewest number of coins possible as change.

We cannot show that this specification is the correct strategy (I.e. postal machines that give you Susan B. Anthony dollars)

Program Correctness Techniques

Proving program correctness is a primary goal of computer science, but a very broad topic

There is one aspect we can use.Assertions

– Precise statements of the behavior of the program at a particular spot.

Assertions

An assertion is a statement of the specifications of a contract.

The program can be evaluated to see if it lives up to the terms of the contract at any given point.

There are many key types of assertions

Assertions

Preconditions– Statements about what we expect to be

true before the function begins Loop invariants

– Statements about conditions that are true for each iteration of the loop

Postconditions– What we expect the result of the function to

be

Proof of correctness

A proof of correctness uses the loop invariant to show that– given a certain set of preconditions– the invariant guarantees that the

postcondition will always be met.

Pre and post conditions

These are special assertions that describe how the program performs.

“If when function f is called A is true, then, when function f returns, B will be true.”

Example

If when the max function is called, it is true that ‘a’ is an array ranging from 0 to n-1, then, when max returns, the value returned will be equal to the largest value in ‘a’.

Contracts

Preconditions and postconditions are contracts between client code and the function.

If the preconditions are met by the client, then the function guarantees the specified result upon its return.

Broken contracts

The program has a bug in it if it cannot fulfill its contracts

One common way of breaking a contract is for client code to fail to meet the precondition

This happens whenever there is failure to correctly exchange data through the function interface.

Loop invariants

Loop invariants are statements that are true at the beginning and ending of each iteration of the loop.

Loops are another major source of broken contracts– they do not terminate at the right time– they do not terminate at all– they do not execute

Code Example Function max

int max (int a[], int n) { // assertion 1: THE PRECONDITION

// a is an array with subscripts from 0 to n-1 int max_val(a[0]), i; for (i = 1; i < n; i++) // assertion 2: THE LOOP INVARIANT

// (max_val >= a[k] for 0 <= k < i) and // (max_val = a[j] for some j, 0 <= j < i) if (max_val < a[i]) max_val = a[i]; // assertion 3: THE POST CONDITION

// (max_val >= a[k] for 0 <= k < n) and // (max_val == a[j] for some j,0 <= j< n) // i.e., max_val is the value of largest int in a return max_val; }

Assertions are like contracts

Guarantees between program segments

Function interfaces are prime spots for bugs– General Protection Faults (Turbo)– Segmentation faults (unix)

Assert.h

#include <iostream.h> #include <assert.h> int max (int a[], int); // See next slide int main() { int a[] = { 45,21,76,34,62,58,92,34,10,98 }; int n = 10; cout << "The largest is: " << max(a,n) << endl; return 0; }

Example: assertions

int max (int a[], int n) {// assertion 1: a is an array with subscripts ranging from 0 to n-1 assert (0 < n); // n is positive int max_val(a[0]), i; for (i = 1; i < n; i++) { if (max_val < a[i]) max_val = a[i]; // assertion 2: (max_val >= a[k] for 0 <= k < i) and // (max_val = a[j] for some j, 0 <= j < i) assert (max_val >= a[i]); } // assertion 3: (max_val >= a[k] for 0 <= k < n) and // (max_val == a[j] for some j, 0 <= j < n) // i.e., max_val is equal the value of largest int in array a return max_val; }

Loop invariants

Anything that is true for every iteration is a loop invariant

There are many possible invariantsi == i is true but uninterestingi < n is also true but uninteresting,

even though it is the termination condition

Important invariants

Operant condition– (max_val >= a[k] for 0 <= k < i) and– (max_val == a[j] for some j, 0 <= j < i)

After the loop is over we can substitute n for the value of i– (max_val >= a[k] for 0 <= k < n) and– (max_val == a[j] for some j, 0 <= j < n)

Now we have the post condition!

Establishing a loop invariant

We must show that two things are true– The invariant is true the first time the

loop is entered.– If it is true for one interation it is true

for the next as well.

The invariant is true the first time

When we enter the loop i is 1, therefore we have:– (max_val >= a[k] for 0 <= k < 1) and– (max_val == a[j] for some j, 0 <= j < 1)

This is true since k and j are 0

The invariant is true for each successive time

For all i, a[i] is either greater that max_val or not

If it is, max_val becomes a[i]If it is not, max_val stays unaltered

Illustrated loop invariant for function max

max_val >= everything in here this part is unknown

i

Proving termination

How do we know the loop will end?Because i starts out less than n and is

incremented in each iteration - it will eventually get to n no matter what n is.

Steps for analyzing a loop

Establish the preconditionFind the invariantEstablish the postconditionProve termination

sum function

int sum(int a[], int n) { // Precondition: a is an array with // subscripts from 0 to n-1

int i; total(0); for (i = 0; i < n; i++) // Loop invariant:

// total = a[0] + a[1] + … + a[i] total += a[i]; // Postcondition: // total = a[0] + a[1] + … + a[n-1]

return total; }

Example

The insertion sort

Basic idea of Insertion Sort

7 11 17 26 33

7 11 17 26 3323

23

Unprocessed Items

Insertion Sort

void insertNextItem(int a[], int i); void insertionSort(int a[], int n) {// Precondition:

// a is an array with subscripts from 0 to n-1 for (int i = 1; i < n; i++) // Loop invariant: // items in range from 0 to i-1 are sorted;

// items from i to n-1 haven’t been examined. insertNextItem(a, i); // Postcondition: array a is sorted }

InsertNextItem

void insertNextItem(int a[], int i) {// Precondition: array a is sorted from 0 to i-1 int newItem(a[i]), insertPos(i); for (;insertPos && newItem < a[insertPos-

1];insertPos--) // Loop Invariant: newItem <= a[insertPos+1] .. a[i] && // a[insertPos+1] .. a[i] are sorted a[insertPos] = a[insertPos-1]; // slide item right a[insertPos] = newItem; // Postcondition: array a is sorted from 0 to i }

Proof of termination

Two ways to exit the loop– insertPos <= 0– newItem < a[insertPos - 1]

We must show that one of these eventually happens

If insertPos starts out greater than 0 then the first will always be met since we subtract 1 each iteration.