Slide 6.1

© The McGraw-Hill Companies, 2002

Object-Oriented and Classical Software

Engineering

Fifth Edition, WCB/McGraw-Hill, 2002

Stephen R. Schachsrs@vuse.vanderbilt.edu

Slide 6.2

© The McGraw-Hill Companies, 2002

CHAPTER 6

TESTING

Slide 6.3

© The McGraw-Hill Companies, 2002

Overview

Quality issues Nonexecution-based testing Execution-based testing What should be tested? Testing versus correctness proofs Who should perform execution-based testing? When testing stops

Slide 6.4

© The McGraw-Hill Companies, 2002

Testing

Two types of testing– Execution-based testing– Nonexecution-based testing

Slide 6.5

© The McGraw-Hill Companies, 2002

Testing (contd)

“V & V”– Verification

» Determine if the phase was completed correctly

– Validation» Determine if the product as a whole satisfies its requirements

Slide 6.6

© The McGraw-Hill Companies, 2002

Testing (contd)

Warning – “Verify” also used for all nonexecution-based testing

Slide 6.7

© The McGraw-Hill Companies, 2002

Software Quality

Not “excellence” The extent to which software satisfies its

specifications Software Quality Assurance (SQA)

– Goes far beyond V & V– Managerial independence

» Development group» SQA group

Slide 6.8

© The McGraw-Hill Companies, 2002

Nonexecution-Based Testing

Underlying principles– We should not review our own work– Group synergy

Slide 6.9

© The McGraw-Hill Companies, 2002

Walkthroughs

4–6 members, chaired by SQA Preparation—lists of items Inspection

– Up to 2 hours– Detect, don’t correct– Document-driven, not participant-driven– Verbalization leads to fault finding– Performance appraisal

Slide 6.10

© The McGraw-Hill Companies, 2002

Inspections

Five-stage process– Overview– Preparation, aided by statistics of fault types– Inspection– Rework– Follow-up

Slide 6.11

© The McGraw-Hill Companies, 2002

Fault Statistics

Recorded by severity and fault type Compare with previous products What if there are a disproportionate number

of faults in a module? Carry forward fault statistics to the next phase

Slide 6.12

© The McGraw-Hill Companies, 2002

Statistics on Inspections

82% of all detected faults (IBM, 1976) 70% of all detected faults (IBM, 1978) 93% of all detected faults (IBM, 1986) 90% decrease in cost of detecting fault

(Switching system, 1986) 4 major faults, 14 minor faults per 2 hours

(JPL, 1990). Savings of $25,000 per inspection

Number of faults decreased exponentially by phase (JPL, 1992)

Warning– Fault statistics and performance appraisal

Slide 6.13

© The McGraw-Hill Companies, 2002

Metrics for Inspections

Fault density (e.g., faults per KLOC) Fault detection rate (e.g., faults detected per hour) By severity (major/minor), by phase What does a 50% increase in the fault detection

rate mean?

Slide 6.14

© The McGraw-Hill Companies, 2002

Execution-Based Testing

Definitions– Failure (incorrect behavior)– Fault (NOT “bug”)– Error (mistake made by programmer)

Nonsensical statement – “Testing is a demonstration that faults are not present”

Slide 6.15

© The McGraw-Hill Companies, 2002

What Is Execution-Based Testing?

“The process of inferring certain behavioral properties of the product based, in part, on the results of executing the product in a known environment with selected inputs.”– Inference– Known environment– Selected inputs

But what should be tested?– Correctness (of course!)– What else?

Slide 6.16

© The McGraw-Hill Companies, 2002

Utility

Does it meet the user’s needs?– Ease of use– Useful functions– Cost effectiveness

Slide 6.17

© The McGraw-Hill Companies, 2002

Reliability

Frequency and criticality of failure– Mean time between failures– Mean time to repair– Mean time, cost to repair the results of a failure

Slide 6.18

© The McGraw-Hill Companies, 2002

Robustness

Range of operating conditions Possibility of unacceptable results with valid input Effect of invalid input

Slide 6.19

© The McGraw-Hill Companies, 2002

Performance

Extent to which space and time constraints are met Real-time software

Slide 6.20

© The McGraw-Hill Companies, 2002

Correctness

A product is correct if it satisfies its specifications

Slide 6.21

© The McGraw-Hill Companies, 2002

Correctness of specifications

Incorrect specification for a sort:

Function trickSort which satisfies this specification:

Slide 6.22

© The McGraw-Hill Companies, 2002

Correctness of specifications (coptd)

Incorrect specification for a sort:

Corrected specification for the sort:

Slide 6.23

© The McGraw-Hill Companies, 2002

Correctness (contd)

Technically, correctness is– NOT necessary– NOT sufficient

Slide 6.24

© The McGraw-Hill Companies, 2002

Correctness Proofs

An alternative to execution-based testing

Slide 6.25

© The McGraw-Hill Companies, 2002

Example of Correctness Proof

Code segment to be proven correct

Slide 6.26

© The McGraw-Hill Companies, 2002

Example of Correctness Proof (contd)

Flowchart of code segment

Slide 6.27

© The McGraw-Hill Companies, 2002

Example of Correctness Proof

Slide 6.28

© The McGraw-Hill Companies, 2002

Correctness Proof Case Study

Never prove a program correct without testing it as well

Slide 6.29

© The McGraw-Hill Companies, 2002

Episode 1

1969 — Naur Paper “Naur text-processing problem”

Given a text consisting of words separated by a blank or by newline characters, convert it to line-by-line form in accordance with the following rules:

(1) line breaks must be made only where the given text contains a blank or newline;

(2) each line is filled as far as possible, as long as

(3) no line will contain more than maxpos characters Naur constructed a procedure (25 lines of Algol

60), and informally proved its correctness

Slide 6.30

© The McGraw-Hill Companies, 2002

Episode 2

1970 — Reviewer in Computing Reviews– The first word of the first line is preceded by a blank

unless the first word is exactly maxpos characters long

Slide 6.31

© The McGraw-Hill Companies, 2002

Episode 3

1971 — London finds 3 more faults Including:

– The procedure does not terminate unless a word longer than maxpos characters is encountered

Slide 6.32

© The McGraw-Hill Companies, 2002

Episode 4

1975 — Goodenough and Gerhart find three further faults

Including: – The last word will not be output unless it is followed by a

blank or newline

Slide 6.33

© The McGraw-Hill Companies, 2002

Proofs and Software Engineering

Lesson: Even if product is proved correct,

it must STILL be tested.

Slide 6.34

© The McGraw-Hill Companies, 2002

Three Myths

Software engineers do not have enough mathematics for proofs

Proving is too expensive to be practical Proving is too hard

Slide 6.35

© The McGraw-Hill Companies, 2002

Can we trust a theorem prover ?

How to find input–output specifications, loop invariants

What if the specifications are wrong? Can never be sure that specifications

or a verification system are correct

Proofs and Software Engineering (contd)

void theoremProver(){ print This product is correct ;}

Slide 6.36

© The McGraw-Hill Companies, 2002

Proofs and Software Engineering (contd)

Correctness proofs are a vital software engineering tool, WHERE APPROPRIATE.

If– Human lives are at stake– Indicated by cost/benefit analysis– The risk of not proving is too great

Also, informal proofs can improve the quality of the product– Assert statement

Slide 6.37

© The McGraw-Hill Companies, 2002

Who Performs Execution-Based Testing?

Programming is constructive Testing is destructive A successful test finds a fault Solution

– 1. The programmer does informal testing– 2. SQA does systematic testing– 3. The programmer debugs the module

All test cases must be– Planned beforehand, including the expected output– Retained afterwards

Slide 6.38

© The McGraw-Hill Companies, 2002

When Testing Can Stop

Only when the product has been irrevocably retired