v. 1.1, 31/Jul/2004, (slide 1 - 2). Copyright Les Hatton 2004-
The author’ s cynicism, sarcasm and general dismay at many aspects of computing do not in any way derive from the views of his colleagues.
He has arrived at this position all on his own aided only by 30 years of working in industry.
Although it may not be immediately apparent, he remains an optimist.
Disclaimer
v. 1.1, 31/Jul/2004, (slide 1 - 3). Copyright Les Hatton 2004-
❖ How good is good ?– Here we go again– An interesting survey– Problems with interfaces– Problems with defect longevity
❖ The effect of imprecise specifications❖ Isn’ t open source wonderful !❖ Should we stop testing after delivery ?
Overview
v. 1.1, 31/Jul/2004, (slide 1 - 4). Copyright Les Hatton 2004-
❖ Here we go again❖ An interesting survey❖ Problems with interfaces❖ Problems with defect longevity
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 5). Copyright Les Hatton 2004-
How good is good ?
Avionics …
• 28/Jul/2003. “ As recently as February, test pilots of the new F/A-22 (Raptor) fighter were spending an average of 14 minutes per flight rebooting critical systems. This is now down to only 36 seconds per flight.
W a s h in g t o n P o s t .
v. 1.1, 31/Jul/2004, (slide 1 - 6). Copyright Les Hatton 2004-
Cars:
• 14/Apr/2004. Ford is recalling 363,440 of its 2001-2003 Ford Escape vehicles due to software problems in power-train causing engine stalling.
D e t r o i t N e w s
• 17/Mar/04. 2003 US vehicle recalls hit 19.5 million in spite of ‘ engineering never being better’ . Experts cite problem-prone computers as significant factor.
• 09/Mar/04, Toyota faces US safety investigation and potential recall of 1 million of its best-selling Camry and Lexus ES300 sedans because of reports of unexpected acceleration causing 30 crashes.
D e t r o i t F r e e P r e s s
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 7). Copyright Les Hatton 2004-
And yet more cars:
• 13/Mar/2004. The new BMW M3 has an undocumented ‘ cheat’ code built into the shifter which disables a road-safety feature and allows ‘ jack-rabbit’ starts, (5000rpm and drops clutch). It is reduced to 2500 rpm in the US but dealers will readily install the European version.
More than 15 uses invalidates the warranty.h t t p :/ /w w w .b o in g b o in g .c o m /
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 8). Copyright Les Hatton 2004-
NIST (US National Institute of Standards and Technology)
• 2002 report estimating costs of software failure in US alone at $60 billion per year
• 80% of software development costs are finding and fixing defects
v. 1.1, 31/Jul/2004, (slide 1 - 9). Copyright Les Hatton 2004-
Royal Academy of Engineering (UK) reported
• Only 16% of projects in the UK were considered successful
• This suggests that around GBP 17 billion will be wasted in 2003/2004 alone.
• “ The challenges of complex IT projects” , 22/Apr/2004
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 10). Copyright Les Hatton 2004-
A useful criterion
• Define a defect as a fault that has failed
• Define an executable line of code as any line of code which generates an executable statement
• Define asymptotic defect density as the upper bound of the total number of defects ever found in the product’ s entire life-cycle divided by the lines of code
If your asymptotic defect density is < 1 defect per KXLOC (thousand executable lines of code), you are doing about as well as has ever been achieved.
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 11). Copyright Les Hatton 2004-
❖ Here we go again❖ An interesting survey❖ Problems with interfaces❖ Problems with defect longevity
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 12). Copyright Les Hatton 2004-
❖ Debugging is a close neighbour of testing. Andrews (2002) found the following in a recent survey of debugger use:-
– By far the most common debugging tool in both academic and commercial users is the print statement
– About 20% of all respondents had never heard of static checking
– About 50% of all respondents had never heard of slicing to show dependencies between statements
– Of 12 well-known debugging techniques, half of the respondents had never heard of half of them.
❖ Testing smarter means educating your testers as well as possible. It also means educating the tool developers.
A n d r e w s , M . (2 0 0 2 ) P h .D t h e s is , U n iv e r s i t y o f K e n t , U K
An interesting survey
v. 1.1, 31/Jul/2004, (slide 1 - 13). Copyright Les Hatton 2004-
❖ Here we go again❖ An interesting survey❖ Problems with interfaces❖ Problems with defect longevity
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 14). Copyright Les Hatton 2004-
❖ Alarm clock 1– Purchased 5 years ago and faultless ever since.
– Staggeringly simple and intuitively obvious interface which has never required the instructions to be consulted.
A tale of two alarm clocks
v. 1.1, 31/Jul/2004, (slide 1 - 15). Copyright Les Hatton 2004-
A tale of two alarm clocks – the sublime
Hour
Minute
Time set
Alarm set
Alarm
On/off
Drowse
v. 1.1, 31/Jul/2004, (slide 1 - 16). Copyright Les Hatton 2004-
❖ Alarm clock 2– Purchased 1 year ago and frequently resets itself
– Staggeringly complex and intuitively non-obvious interface.
A tale of two alarm clocks
v. 1.1, 31/Jul/2004, (slide 1 - 17). Copyright Les Hatton 2004-
A tale of two alarm clocks – the ridiculous
Mode
- ARGH !
More
buttons
Desk
Random
numbers
v. 1.1, 31/Jul/2004, (slide 1 - 18). Copyright Les Hatton 2004-
❖ Alarm clock 2 – some examples
– Time setting◆ Press M ODE but ton T HREE t imes. T hen press SELECT and
SET repeatedly .
– Alarm setting◆ Press M ODE but ton T WO t imes. T hen follow t ime set t ing.
– Hourly chime on/off◆ Press SELECT and M ODE but ton simultaneously
– Alarm on◆ Press DA T E and SELECT button simultaneously
– Stop watch / lap counter◆ Press M ODE but ton ONCE and then beg for mercy as clock
g ibbers w ith entertain ing series of random beeps.
– Turning alarm off◆ Hurl out of w indow after standing on clock screaming.
A tale of two alarm clocks
v. 1.1, 31/Jul/2004, (slide 1 - 19). Copyright Les Hatton 2004-
Testing smarter means involving testers from the design stage onwards.
A tale of two alarm clocks
v. 1.1, 31/Jul/2004, (slide 1 - 20). Copyright Les Hatton 2004-
❖ Here we go again❖ An interesting survey❖ Problems with interfaces❖ Problems with defect longevity and subtlety
How good is good ?
v. 1.1, 31/Jul/2004, (slide 1 - 21). Copyright Les Hatton 2004-
A famous study by Adams (1984) showed that around one third of all defects took longer than 5000 execution years to surface.If you find this hard to believe consider the following 1000-2000 year
defects found in seismic data processing software as used in oil and gas exploration.
A d a m s , E . (1 9 8 4 ) “ O p t im is in g p r e v e n t iv e s e r v ic e o f s o f t w a r e p r o d u c t s ” ,
I B M J . R e s , 2 8 (1 ) p 2 -1 4
Problems with defect longevity
v. 1.1, 31/Jul/2004, (slide 1 - 22). Copyright Les Hatton 2004-
Details of study:-
– 9 sets of software with same published mathematical specifications but developed independently in fiercely competitive environment
– Same data, (actually the same input tapes)
– Same disposable parameter values
– Same programming language
– Each package contained around 1,000,000 lines of code of which around 10% was ‘ exercised’ .
– Data is subjected to 14 processes in a pipeline, (the input of process N+1 Is the output of process N, 1 <= N <= 13). Process 1 reads the input tape.
Defect longevity in large systems
v. 1.1, 31/Jul/2004, (slide 1 - 23). Copyright Les Hatton 2004-
How to collect seismic data
Borrow around 20 million
dollars and buy one of
these
If it doesn’ t work out you
can always run booze
cruises.
v. 1.1, 31/Jul/2004, (slide 1 - 24). Copyright Les Hatton 2004-
Similarity v. coordinate: No feedback
v. 1.1, 31/Jul/2004, (slide 1 - 25). Copyright Les Hatton 2004-
Defect example 1: feedback detail
v. 1.1, 31/Jul/2004, (slide 1 - 26). Copyright Les Hatton 2004-
Similarity v. coordinate: Feedback to company 8
v. 1.1, 31/Jul/2004, (slide 1 - 27). Copyright Les Hatton 2004-
Defect example 2: feedback detail
v. 1.1, 31/Jul/2004, (slide 1 - 28). Copyright Les Hatton 2004-
Similarity v. coordinate: Feedback to company 3
v. 1.1, 31/Jul/2004, (slide 1 - 29). Copyright Les Hatton 2004-
The end product: 9 subtly different views of the geology
v. 1.1, 31/Jul/2004, (slide 1 - 30). Copyright Les Hatton 2004-
❖ Useful lessons– The differences are due to subtle defects
Testing smarter means being careful to design tests with sufficient precision for the desired accuracy
– These defects had exceptionally long lives and can cost a fortune. Software which will accumulate thousands of execution years should depend more on static testing methods than dynamic testing methods.
Testing smarter means carefully balancing test resources between static and dynamic methods to match the expected life-time exposure of the software.
The outcome …
v. 1.1, 31/Jul/2004, (slide 1 - 31). Copyright Les Hatton 2004-
❖ How good is good ?– Here we go again– An interesting survey– Problems with interfaces– Problems with defect longevity
❖ The effect of imprecise specifications❖ Isn’ t open source wonderful !❖ Should we stop testing after delivery ?
Overview
v. 1.1, 31/Jul/2004, (slide 1 - 32). Copyright Les Hatton 2004-
The MISRA C standard
❖ In April 1998, the Motor Industry
Software Research Association
(M ISRA) published a set of C
guidelines for use in vehicle-based
software.
– 93 rules + 34 guidelines
– Consistent with development to SIL3
– Sensible deviation policy
– Some uncertain areas
– The standard is cross-referenced
against the ISO C 9899:1990
standard for traceability
v. 1.1, 31/Jul/2004, (slide 1 - 33). Copyright Les Hatton 2004-
The MISRA C standard
❖ The test suite was constructed after
the standard was published
24% of the rules were sufficiently imprecise that test cases could not be constructed at all or could not be constructed without breaking other rules.
(h t t p :/ /w w w .le s h a t t o n .o r g / f o r m o r e
d e t a i ls )
Testing smarter means being careful to design tests concurrently with specifications to make sure they are feasible.
v. 1.1, 31/Jul/2004, (slide 1 - 34). Copyright Les Hatton 2004-
❖ How good is good ?– Here we go again– An interesting survey– Problems with interfaces– Problems with defect longevity
❖ The effect of imprecise specifications❖ Isn’ t open source wonderful !❖ Should we stop testing after delivery ?
Overview
v. 1.1, 31/Jul/2004, (slide 1 - 35). Copyright Les Hatton 2004-
❖ Scripting languages– Portable (reducing test setup costs substantially)
– Expressive (reducing test implementation costs)
– Free (you can probably work this bit out)
– Examples◆ Tcl/Tk
◆ Perl
◆ Python
Open source tooling
v. 1.1, 31/Jul/2004, (slide 1 - 36). Copyright Les Hatton 2004-
❖ Scripting languages– An example follows of a portable test harness written in Tcl/Tk,
(about 700 lines of harness including all graphics). This script was developed on Linux and simply moved to Windows along with the regression oracle, (which are simple text files).
Open source tooling
v. 1.1, 31/Jul/2004, (slide 1 - 37). Copyright Les Hatton 2004-
❖ Scripting languagesTesting smarter means using the power of scripting languages to design
and implement efficient regression and other test suites and harnesses.
Open source tooling
v. 1.1, 31/Jul/2004, (slide 1 - 38). Copyright Les Hatton 2004-
❖ How good is good ?– Here we go again– An interesting survey– Problems with interfaces– Problems with defect longevity
❖ The effect of imprecise specifications❖ Isn’ t open source wonderful !❖ Should we stop testing after delivery ?
Overview
v. 1.1, 31/Jul/2004, (slide 1 - 39). Copyright Les Hatton 2004-
❖ Safer C toolset– Uses direct and indirect techniques to find defects in systems written
in C.
– Released 3 years ago, several hundred users on both Windows and Linux machines
– 14 update releases in 3 years
– Server written in C (127,000 lines of code)
– GUI client written in Tcl/Tk (26,000 lines of code), (a C like scripting language pioneered by John Ousterhout and collaborators).
Case history 1
v. 1.1, 31/Jul/2004, (slide 1 - 40). Copyright Les Hatton 2004-
❖ Principle test methodologies– Regression testing against 275,000 test cases built up over 5 years.
Done twice a day on Windows, Linux and Solaris using portable test harness. (Server only)
– Code inspection of changes but not whole system. (GUI Client and Server)
– Static testing using tools
Case history 1
v. 1.1, 31/Jul/2004, (slide 1 - 41). Copyright Les Hatton 2004-
Defect densities in Client and Server, case history 1
Asymptotic
defect density
(defects per
KLOC)
GUIServer
Internally found
Externally found
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
v. 1.1, 31/Jul/2004, (slide 1 - 42). Copyright Les Hatton 2004-
❖ Gundalf– Models underwater releases of compressed air for geophysical
exploration
– Released 2 years ago, twenty users on both Windows and Linux machines
– 21 update releases in 2 years (internet download)
– Server written in C (83,511 lines of code)
– GUI client written in Tcl/Tk (24,239 lines of code).
Case history 2
v. 1.1, 31/Jul/2004, (slide 1 - 43). Copyright Les Hatton 2004-
❖ Principle test methodologies– Regression testing against 55 calibrated test cases. Done twice a day
on Windows and Linux using portable test harness. (Server only)
– Code inspection of changes but not whole system. (GUI Client and Server)
– Static testing using tools on Server
Case history 2
v. 1.1, 31/Jul/2004, (slide 1 - 44). Copyright Les Hatton 2004-
Comparative defect densities in Client and Server
GUIServer
Internally found
Externally found0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Asymptotic
defect density
(defects per
KLOC)
v. 1.1, 31/Jul/2004, (slide 1 - 45). Copyright Les Hatton 2004-
❖ Useful lessons– More than half of the defects found after release were found
by internal testingTesting smarter means continuing to test after release.
(Modern web delivery can greatly reduce upgrade costs).
– GUI testing still presents a problemTesting smarter means expecting unusual difficulties with
GUIs and making allowance for it.
Case histories
v. 1.1, 31/Jul/2004, (slide 1 - 46). Copyright Les Hatton 2004-
❖ Smarter testing requires:-– Better education for testers
– Early involvement with design and specification
– Awareness of the properties of failure and the intended use and ubiquity of the product
– Awareness of how good is good
– Continuing to test after a product is released, (assuming product updates are economic to do)
– Test and defect measurements and the patterns concealed in them.
General conclusions
v. 1.1, 31/Jul/2004, (slide 1 - 47). Copyright Les Hatton 2004-
This paper and many other downloadable papers and pieces of software can be found at:-
