Post on 16-Apr-2017
transcript
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
How to Learn The History of Software Testing
Keizo Tatsumi
2010-12-19 Translated into English on July 2014
If it wasn't useful, it wouldn't be learned.
If it couldn't ever be interesting, it wouldn't
deserve to be learned.
WACATE 2010 Winter
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 2
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Prehistory
The First Computer
Charles Babbage’s Analytical Engine
• Conceived in 1837 (unfinished)
• The input via punched cards
• The output via a printer, a curve plotter and a bell
The First Programmer
Ada Byron, Lady Lovelace
• Worked on the Analytical Engine (1843)
– Ada’s notes on the Analytical Engine were recognized as a description of a computer and software
(1/2)
(C) K. Tatsumi 2014 3
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Prehistory
Who was The First Tester ?
(2/2)
The first programmer, Lady Lovelace, who coded for
Charles Babbage's wonderful but unfinished
computer in the nineteenth century, I'm sure often
said, "Just one more week, Mr. Babbage, and it'll be
done." Lucky for her the hardware was never finished,
so she never did have to go beyond desk checking.
(B. Beizer, Software system testing and quality assurance, 1984, p.277)
Boris Beizer wrote: • “The first discussion of testing and debugging predates
computers by almost a century, in the memoirs of lady Ada Lovelace who wrote software for Babbage’s (never completed) mechanical computer.”
(posted to comp.software.testing on Sep. 18 2007)
(C) K. Tatsumi 2014 4
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 5
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The History of Computers and Software Engineering
Timeline http://a-lifelong-tester.cocolog-nifty.com/Chronology/History_of_Software_TestingChronology_20120508_English.pdf
(C) K. Tatsumi 2014 6
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Victor Basili and John Musa, The Future Engineering of Software: A Management Perspective, 1991
1960s: The Functional Era
• IT penetrated institutions
• How to exploit IT to meet institutional needs
1970s: The Schedule Era • Software crisis NATO Software Engineering Conference, 1968
• How to develop software in a timely, planned, and controlled fashion
• Life-cycle models and schedule tracking ex. Royce waterfall model, Quality models
1950 1980 1990 1960 1970 2000 2010
Schedule Era Cost Era Quality Era Functional Era
Historical Division by Management Perspective (1/2)
(C) K. Tatsumi 2014 7
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
1980s: The Cost Era
• Hardware costs decreased, PC created a mass market
• The importance of productivity in software development increased
• Various cost models ex. Function point, Putnam model, COCOMO
1990s: The Quality Era
• The increased dependence of institutions on information processing
• The consumer mass market increases the demands on quality
1950 1980 1990 1960 1970 2000 2010
Schedule Era Cost Era Quality Era Functional Era
Historical Division by Management Perspective (2/2)
(C) K. Tatsumi 2014 8
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Software Engineering Evolution
Barry Boehm,
A Hegelian View of Software Engineering Evolution,
2006
Autonomy;
Bio-
Computing
1990’s 2010’s 2000’s 1970’s 1980’s 1960’s 1950’s
COTS
Software
as Craft
Theses
Syntheses
Antitheses
Formality,
Waterfall
Productivity;
Reuse;
Objects;
Peopleware
Plan-
Driven
Software
Maturity
Models
Agile
Methods
Engineer
Software
like
Hardware
Risk-Based
Agile/Plan-
Driven Hybrids;
Model-Driven
Development
Integrated
Sw-systems
Engineering
Value-Based Methods;
Collaboration; Global
Development; Enterprise
Architectures Software
Differences,
Engineer
Shortages
Many defects
Scalability,
Risk Mgmt.
Prototyping
Time to Market,
Rapid Change
Scalability
Domain Engr.
Risk Mgmt.
Compliance
Process Overhead
Software
Value-Add
Soft
SysE
Global
Systems
of
Systems
B. Boehm, A View of 20th and 21st Century Software Engineering, 2006
(C) K. Tatsumi 2014 9
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 10
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Growth of Software Testing
D. Gelperin and W. Hetzel,
The Growth of Software Testing, 1988
Five evolutionary periods divided by landmark literatures
1. The Debugging-Oriented Period ( -1956)
2. The Demonstration-Oriented Period (1957-1978)
3. The Destruction-Oriented Period (1979-1982)
4. The Evaluation-Oriented Period (1983-1987)
5. The Prevention-Oriented Period (1988- )
(C) K. Tatsumi 2014 11
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Debugging-Oriented Period
Testing was not differentiated from debugging
Evaluation
Demonstration Destruction
Debugging Prevention
1950 1980 1990 1960 1970 2000 2010
S. Gill, The diagnosis of mistakes in programmes on the EDSAC, p.539, 1951
" Software problems were submerged in the concern for hardware reliability."
"The difficulty lies not in detecting the presence of a mistake, but in diagnosing it."
You "wrote" a program and then you "checked it out."
• Program checkout, debugging, and testing were not clearly differentiated.
McCracken, Digital Computer Programming, 1957 (The earliest programming text)
• Techniques referred to as today‘s debugging and testing are described in "Chapter 13 Program Checkout".
(C) K. Tatsumi 2014 12
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Demonstration-Oriented Period
Testing to make sure that the software satisfies its requirements
C. Baker, Review of "Digital Computer Programming", 1957
• "McCracken fails to distinguish the two phases of program checkout." – Debugging: The process of making sure that the program does what the
coder meant it to do. – Testing: The process of making sure that the program solves the problem
it is intended to solve.
Big Projects (late 1950s-1960s)
• SAGE (US air defense control system), SABRE (Computer reservation system), NASA's Mercury, Gemini, and Apollo projects
• IBM OS/360 ref. F. Brooks, "Mythical Man-Month"
"Software Engineering"
• NATO Software Engineering Conferences (1968, 1969) – "Testing shows the presence, not the absence of bugs." (Dijkstra,1969)
(1/2)
Demonstration Debugging Evalua
tion Destruction
Prevention
1950 1980 1990 1960 1970 2000 2010
(C) K. Tatsumi 2014 13
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Demonstration-Oriented Period
Growth of Professionalism
• Debugging techniques
– Debugging Techniques in Large Systems Symposium, 1970
• The first formal conference on software testing
– The Computer Program Test Methods Symposium, 1972
• The first book on software testing
– Hetzel (Ed.), "Program Test Methods," 1973
Growth of Software Testing Research
• The fundamental theorem of software testing
– Goodenough and Gerhart, "Toward a Theory of Test Data Selection," 1975
• The first workshop on software testing
– Software Testing and Test Documentation Workshop, 1978
The origin of ISSTA (International Symposium on Software Testing and Analysis)
cf. W. Wulf, Concise definitions of software quality attributes, 1973
B. Boehm, et al., Software quality characteristics, 1973
Demonstration Debugging Evalua
tion Destruction
Prevention
1950 1980 1990 1960 1970 2000 2010
(2/2)
(C) K. Tatsumi 2014 14
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Destruction-Oriented Period
Testing to detect implementation faults
G. Myers, The Art of Software Testing, 1979 • "Testing is the process of executing a program with the
intent of finding errors"
• "Since exhaustive testing is out of the question, the objective should be to maximize the yield of the testing investment by maximizing the number of errors found by a finite number of test cases."
• "Test-case design is so important because complete testing is impossible; a test of any program must be necessarily incomplete. The obvious strategy, then, is to try to make tests as complete as possible."
1950 1980 1990 1960 1970 2000 2010
Demonstration Debugging Evalua
tion Destruction
Prevention
(C) K. Tatsumi 2014 15
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Evaluation-Oriented Period
Testing to evaluate products during the software lifecycle FIPS 101, Guideline for Lifecycle Validation, Verification, and
Testing of Computer Software, 1983
1950 1980 1990 1960 1970 2000 2010
Demonstration Debugging Evalua
tion Destruction
Prevention
• "A VV&T methodology is a procedure of review, analysis, and testing employed throughout the software lifecycle from software planning through the end of software use to ensure the production and maintenance of quality software."
• "No single VV&T technique can guarantee correct, error-free software. However, a carefully chosen set of techniques for a specific project can help to ensure the development and maintenance of quality software for that project."
(C) K. Tatsumi 2014 16
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Prevention-Oriented Period
Testing to prevent faults in requirements, design, and implementation
W. Hetzel, The Complete Guide to Software Testing (2nd Ed.), 1988
1950 1980 1990 1960 1970 2000 2010
Demonstration Debugging Evalua
tion Destruction
Prevention
• STEP (Systematic Test and Evaluation Process) A life cycle prevention model that sees testing parallel to
development with an activity sequence containing planning, analysis (setting test requirements or objectives), design (specifying an architecture for the set of tests and details
of individual cases and procedures), implementation (acquiring or developing test data, procedures,
and test support software), execution (running and rerunning tests and determining the
results), and maintenance (saving and updating the tests as the software
changes).
cf. W-Model (C) K. Tatsumi 2014 17
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Phases in a Tester‘s Mental Life
Phase 0 - There‘s no difference between testing and debugging. Other than in support of debugging, testing has no purpose.
Phase 1 - The purpose of testing is to show that the software works.
Phase 2 - The purpose of testing is to show that the software doesn't work.
Phase 3 - The purpose of testing is not to prove anything, but to reduce the perceived risk of not working to an acceptable value.
Phase 4 - Testing is not an act. It is a mental discipline that results in low-risk software without much testing effort.
B. Beizer, Software Testing Techniques, 2nd Ed., 1990
(C) K. Tatsumi 2014 18
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 19
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Testing Techniques : Beginnings
The state of the art in software testing before 1970s
< The early 1960s >
Published papers (cumulative)
• -1969: 57 papers (*1) cf. -1973: 200 papers (*2), -1977: 400 papers+ (*3)
• There are few papers on software testing techniques.
• Many papers only address testing in a peripheral manner such as test procedures or automation.
< The late 1960s >
IBM OS/360 Project (F. Brooks, "Mythical Man-Month")
• Released in 1966. MVT, multi tasking OS, was released in 1967.
• A lot of software testing work must have been carried out.
• Much know-how about test process, test techniques, and test management must have been accumulated.
(1/2)
(*1) W. Elmendorf, "Program Testing – A Bibliography of Published Literature, 1962–1968", 1969
(*2) W. Hetzel, Number of references in "Program Test Methods", 1973
(*3) E. Miller, Number of references in "Tutorial: Program Testing Techniques", COMPSAC '77, 1977
(C) K. Tatsumi 2014 20
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Testing Techniques : Beginnings
The test control process, IBM W. Elmendorf, "Controlling the functional testing of
an operating system," 1969
(2/2)
• Elmendorf started to work on software testing for OS/360 in 1965.
• A disciplined test control process
- From the "laissez-faire approach" to a disciplined approach
- From the "testing-is-an-art" approach to a scientific approach
A pioneering work of a preventive test
approach like W-Model
(C) K. Tatsumi 2014 21
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Equivalence Partitioning / Boundary-Value Analysis
<History>
1979: The names and concepts of EP and BVA were first introduced by G. Myers of IBM in his "The Art of Software Testing."
Similar concepts had already appeared in Elmendorf‘s paper in 1967.
• External Interaction Variations
"... If the actions taken by the control program are predictable, acceptable ranges of field values, field sizes, field repetitions, etc., will be identified as variations at, and just beyond, the extremes of each range. There are external limits which, if exceeded, cause unpredictable actions."
W. Elmendorf, Evaluation of the Functional Testing of Control Programs, 1967
(C) K. Tatsumi 2014 22
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Cause-Effect Graphing
<History>
1970: W. Elmendorf developed Cause-Effect Graphing by adopting a hardware logic testing method.
• Extending the concepts to cover software specific issues such as constraints on the inputs
• Development of a test case generation tool
– TELDAP (TEst Library Design Automation Program) was developed for generating test cases from a cause-effect graph, based on the techniques for generating test patterns for the logic circuits in hardware.
1979: Cause-Effect Graphing became widely known after publication of Myers‘ book which introduced this technique.
W. Elmendorf, Automated Design of Program Test Libraries, IBM-TR-00.2089, 1970
(C) K. Tatsumi 2014 23
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Decision Table Testing
<History>
Circa 1958: Decision tables were developed by General Electric and Sutherland corp. individually
• To express the logic of product design, operation planning, management decision rules, etc. .
1960: GE developed TABSOL (Tabular systems oriented language) which generated program codes from decision tables.
1965: The early paper to apply DT to software testing
• Test categories and test parameters are entered to condition fields. Consequences and test actions are entered to action fields.
1975: Condition table method by Goodenough & Gerhart
• A technique derived from decision table techniques for developing and describing test predicates
B. Grad, Tabular form in decision logic, DATAMATION, July 1961
B. Scheff, An application of decision tables as the source language for automatic testing, 1965
J. Goodenough and S. Gerhart, Toward a theory of test data selection, 1975
T. Kavanagh, TABSOL A fundamental concept for system-oriented languages, 1960
(C) K. Tatsumi 2014 24
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Orthogonal Array / Pairwise Testing
<History>
1920s-30s: Design of Experiments (DoE), R. Fisher
1940s: Concepts of orthogonal arrays as DoE, C. Rao
1950s: Quality engineering (Taguchi Methods)
1980s: Applying DoE to software testing
• 1984: Operating system testing, Fujitsu Ltd., Japan
• 1985: Ada compiler testing, R. Mandl, US
1990s: Spreading in the US from the mid 1990s
• 1992: OATS, AT&T Bell Lab.
• 1994: CATS, AT&T
• 1994: AETG, commercial tool, Belcore
S. Sato and H. Shimokawa, Methods for setting software test parameters using the DoE, 1984
R. Mandl, Orthogonal Latin squares: an application of experiment design to compiler testing, 1985
R. Brownlie, J. Prowse, M. Phadke, Robust testing of AT&T PMX/Starmail using OATS, 1992
G. Sherwood, Effective testing of factor combinations, 1994
D. Cohen et al., The automatic efficient test generator (AETG) system, 1994
(C) K. Tatsumi 2014 25
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Timeline of Combinatorial Testing
(1957: Decision table)
~1967: Equivalence partitioning, Boundary value analysis
1970: Cause Effect Graph [Elmendorf]
(1983~)1984: Orthogonal array / Combination Table
[Satoh, Shimokawa]
1987: Test Case Design Support System [Tatsumi]
(198x~)1992: OATS [Brownlie, Prowse, Phadke]
(1990~)1994: CATS [Sherwood]
(1992~)1994: AETG [Cohen, et. al]
1998: IPO [Lei, Tai]
2000: Covering arrays [Williams]
2000: CTE XL [Daimler Chrystler]
(2000~)2004: PICT [Microsoft]
2007: FireEye 2009:ACTS (IPOG) [Lei, Kuhn]
AT&T, Bellcore
1988: Category-partition method [Ostrand, Balcer]
1993: Classification-tree method [Grochtmann, Grimm]
(1976: Test Factor Analysis Method) [Fujitsu]
(199x~)2004: Orthogonal Array (HAYST method)
[Akiyama(Fuji Xerox)]
1980 1990 2000 2010 1950 1960 1970 ‘85 ‘95 ‘05
Combinatorial techniques
Input condition analysis techniques
(1983~)1985: Orthogonal Latin Squares [R. Mandl]
Fujitsu
(C) K. Tatsumi 2014 26
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Control Flow Testing (1/2)
<History>
1960s: Applying graph theory to software development
• 1960: Compiler development, Karp
• 1963: Applying to software test design, Miller & Maloney
• 1976: Cyclomatic complexity and Basis path testing, McCabe
1960s: Measuring test coverage
• Code coverage tool development in IBM Poughkeepsie
– C. Warner Jr., Evaluation of program testing, 1964
Earliest known use of a hardware instruction coverage monitor: COBOL and FORTRAN source. (from Beizer)
– I. Hirsh, MEMMAP/360, 1967
Earliest known description of a software statement and branch coverage analyzer. (from Beizer)
R. Karp, A note on the application of graph theory to digital computer programming, 1960
J. Miller and C. Maloney, Systematic mistake analysis of digital computer programs, 1963
T. McCabe, A Complexity Measure, 1976
(C) K. Tatsumi 2014 27
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Control Flow Testing (2/2)
Early 1970s: Dynamic analysis systems (Coverage measurement tools)
• 1972: PACE (Product Assurance Confidence Evaluator), TRW
• 1972: PET (Program Evaluator and Tester), McDonnell Douglas
• 1974: RXVP (Requirement Evaluation & Verification Package), General Research
J. Brown, et al., Automated Software Quality Assurance: A Case Study of Three Systems, 1972
L. Stucki, Automatic Generation of Self Metric Software, 1972
E. Miller, et al., Structurally based automatic program testing, 1974
Early 1970s: Coverage criteria
• 1972: TER (Test Effectiveness Ratio) [Brown]
• 1975: C0, C1, C2, … [Miller]
(1975-1977) C0 : Programmer‘s intuition, C1 : Every statement in a program exercised at least once, C2 : Every program predicate outcome exercised at least once, ...
(1977-) C0 : Every statement executed at least once, C1 : Every segment executed at least once, C1
p : Every predicate term executed for each
outcome, C2 : C1 + interior and boundary tests for each iteration, ...
J. Brown, Practical applications of automated software tools, 1972
E. Miller, The Art and the Theory of Program Testing, 1975
E. Miller, Coverage levels, in "Infotech State of the Art Report: Software Testing," Vol. 1, 1979
(C) K. Tatsumi 2014 28
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Data Flow Testing
<History>
Late 1960s: Data flow analysis techniques were developed in the field of optimizing compilers. F. Allen( IBM)
1974: Applying data flow analysis techniques to software testing, Osterweil & Fosdick (Univ. of Colorado)
1982-: Data flow criteria, Rapps & Weyuker Coverage criteria subsumption including control flow criteria and data flow criteria
F. Allen and J. Cocke, A program data flow analysis procedure, 1976
L. Osterweil and L. Fosdick, Data Flow Analysis as an Aid in Documentation, Assertion Generation, Validation and Error Detection, 1974
S. Rapps and E. Weyuker, Data Flow Analysis Techniques for Test Data Selection, 1982
(C) K. Tatsumi 2014 29
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
State Transition Testing
<History>
Originally, state transition diagrams and state transition tables were developed to represent finite state machines (FSM).
1956: Moore‘s "Gedanken-experiments on sequential machines"
• The origin of FSM model based testing
1978: Coverage criteria "n-switch cover" proposed by Chow
E. Moore, Gedanken-experiments on sequential machines, 1956
T. Chow, Testing software designs modeled by finite-state machines, 1978
(C) K. Tatsumi 2014 30
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 31
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The History of Testing in Japan (1/3)
1958, The Earliest paper on software testing
Fujiwara, A report on program testing on the IBM 704 large scale electronic computer (in Japanese), 1958
• Preliminary program testing for a numerical weather prediction system of the Japan Meteorological Agency
1964, Discussion on the program inspection
Information Processing Society of Japan, Special features : Software ,IPSJ Magazine, 1964
• Necessity of independent section for software inspection • Role of software quality assurance section • Process improvement through the feedback from QA section
1969, Hitachi’s Software Works (Factory)
Establishing the function of software quality assurance in the software factory system
1971: Also, Fujitsu created a formal product inspection procedures for release to the customer.
(C) K. Tatsumi 2014 32
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The History of Testing in Japan (2/3)
1972, An Approach for Software Inspection A. Kanno, An Approach for Software Inspection, 1972
• Concepts, activities and techniques in software quality assurance department in Hitachi software factory
1974, Development of QA techniques K. Sakata(Hitachi), Formulation for predictive methods in
software production control, 1974 • Static prediction and failure rate transition model
Quality prediction by reliability growth curve or Gompertz curve
• Dynamic prediction: quality probe
Quality prediction by "quality probe" that comprises a small percentage of the complete regular test
(C) K. Tatsumi 2014 33
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The History of Testing in Japan (3/3)
1980-, R&D of testing techniques and tools
1980: AGENT(Automated GENeration system for Test cases), Hitachi • Test case generation tool based on cause-effect graphing
1984: AGENT Function Diagram, Hitachi • Representation of functional specification (dynamic part -> state transition
diagram, static part -> decision table or cause-effect graph)
1984: Application of DoE to software testing, Fujitsu • Test case generation technique and tool for application of DoE
1988: CFD(Case Flow Diagram or Cause Flow Diagram), NEC • External and internal specifications are organized into Cause Flow Diagram,
then decision table is generated from the diagram.
1991, Japan's Software Factories
M. Cusumano, Japan's Software Factories: A Challenge to U.S. Management, 1991
• A management research work about Japanese software development and a survey of the making of Japan‘s computer industry, Hitachi, Toshiba, NEC, and Fujitsu etc.
(C) K. Tatsumi 2014 34
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 2014 35
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Forefront of Software Testing
A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," 2007
• 29th ICSE, Future of Software Engineering track
• Bertolino is the KA co-leader for Software Testing in the IEEE Guide to the SWEBOK
The software testing research roadmap
• Achievements The most notable achievements from past research
• Dreams The desired destination
• Challenges The challenges faced by current and future testing research
(C) K. Tatsumi 2014 36
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
The Many Faces of Software Testing
Software testing:
Observing a sample of executions, and giving a verdict over them
6 aspects of software testing
WHY: test objective looking for faults?, the product can be released?, evaluate the usability?
HOW: test selection ad hoc, at random, or systematic way
HOW MUCH: test adequacy coverage analysis, reliability measures
WHAT: levels of testing unit test, component/subsystem test, integration test
WHERE in house, simulated environment, the target final context
WHEN when is it in the product lifecycle that we perform the observations?
(C) K. Tatsumi 2014 37
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Software Testing Research Roadmap
A. Bertolino, “Software Testing Research: Achievements, Challenges, Dreams,” FOSE’07, p.85-103, 2007, Figure 1
Testing
process
Reliability
testing
WHY How How much What Where When
Protocol
testing
Test
criteria
Comparison among
test criteria
Co
mp
on
en
t-bas
ed
testin
g
Ob
ject-o
rien
ted
testin
g
Education of software testers
Testing patterns
Controlling evolution
Leveraging user population
and resources
Understanding the
costs of testing
Test input
generation
On-line testing
Test
oracles Model-based testing
Anti-model-based testing
Explicit test
hypotheses
Test effectiveness
Empirical body of evidence
Compositional
Testing
Domain-specific
test approaches
Achievements Challenges Dreams
Efficacy-
maximized
test engineering
100% automatic
testing
Test-based
modeling
Universal
test theory
(C) K. Tatsumi 2014 38
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Achievements
Testing process
Test criteria
Comparison among test criteria
Object-oriented testing
Component-based testing
Protocol testing
Reliability testing
(C) K. Tatsumi 2014 39
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Dreams
(1) Universal test theory A comprehensive theory which is useful to backup and nourish
test technology
(2) Test-based modeling Test-based modeling is closely related to the old idea of
“Design-for-testability.”
(3) 100% automatic testing Advanced techniques for generating the test inputs, innovative
support procedures to automate the testing process
(4) Efficacy-maximized test engineering Practical testing methods, tools and processes for development
of high quality software
(C) K. Tatsumi 2014 40
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Dream: Universal test theory
<Challenges>
Explicit test hypotheses • Making explicit for each technique which are its underlying
assumptions
Test effectiveness • Assessing the effectiveness of existing and novel test criteria
Compositional testing • Reusing the test results observed in the separate testing of the
individual levels
• A foundational theory for component-based software reliability
Empirical body of evidence • An empirical body of knowledge which is at the basis for
building and evolving the theory for testing
(C) K. Tatsumi 2014 41
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Dreams: Test-based Modeling
<Challenges>
Model-based testing • Combining different styles of modeling (transition-based,
pre/post condition-based and scenario-based)
• Integrating model-based testing practice into current software processes
Anti-model-based testing • A model is derived a posteriori via testing for the cases in which
the models do not exist or are not accessible, such as for COTS or legacy components.
Test oracles
(C) K. Tatsumi 2014 42
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Dream: 100% automatic testing
<Challenges>
Test input generation • Model-based test generation, random test generation, search-
based test generation
Domain-specific test approaches
On-line testing • Monitoring a system‘s behavior in real life operation using
dynamic analysis and self-test techniques
(C) K. Tatsumi 2014 43
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Dream: Efficacy-maximized test engineering
<Challenges>
Controlling evolution
Leveraging user population and resources
Testing patterns
Understanding the costs of testing
Education of software testers
(C) K. Tatsumi 2014 44
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Transversal challenges
<Challenges>
Testing within the emerging development paradigm • Testing of service-oriented applications in the Service-oriented
Computing
• A special importance for testing of services (monitoring the real-world execution is the only way to observe the application behavior)
Coherent testing of functional and extra-functional properties • Conventional functionality testing does not provide for any
notion of time nor tackle resource usage and workloads.
• For the model-based approach, we need effective ways to enhance models with desired extra-functional constraints.
(C) K. Tatsumi 2014 45
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Concluding Remarks
The Analects - Chinese Text Project http://ctext.org/analects
• The Master said, "If a man keeps cherishing his old knowledge, so as continually to be acquiring new, he may be a teacher of others."
• The Master said, "Is it not pleasant to learn with a constant perseverance
and application? Is it not delightful to have friends coming from distant quarters? Is he not a man of complete virtue, who feels no discomposure though men may take no note of him?"
• The Master said, "Learning without thought is labor lost; thought without
learning is perilous."
• The Master said, "You, shall I teach you what knowledge is? When you
know a thing, to hold that you know it; and when you do not know a thing, to allow that you do not know it - this is knowledge."
(C) K. Tatsumi 2014 46
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
HIS
TO
RY
OF
SO
FT
WA
RE
TE
ST
ING
Now is Your Turn to Make History!
Thank you !
(C) K. Tatsumi 2014 47