PRACTICALSOFTWARETESTING

SpringerNew YorkBerlinHeidelbergHong KongLondonMilanParisTokyo

PRACTICALSOFTWARETESTING

APROCESS-ORIENTED

APPROACH

I LENE BURNSTEIN

Ilene BurnsteinDepartment of Computer ScienceIllinois Institute of Technology10 West 31 StreetChicago, IL 60616USAburnstei@babbage2.cs.iit.edu

Library of Congress Cataloging-in-Publication DataBurnstein, Ilene.

Practical software testing : a process-oriented approach / Ilene Burnstein.p. cm.

Includes bibliographical references and index.ISBN 0-387-95131-8 (hc : alk. paper)

1. Computer software—Testing. I. Title.QA76.76.T48 B87 2002 2002024164005.1�4–dc21

ISBN 0-387-95131-8 Printed on acid-free paper.

� 2003 Springer-Verlag New York, Inc.All rights reserved. This work may not be translated or copied in whole or in part without thewritten permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, NewYork, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly anal-ysis. Use in connection with any form of information storage and retrieval, electronic adapta-tion, computer software, or by similar or dissimilar methodology now known or hereafterdeveloped is forbidden.The use in this publication of trade names, trademarks, service marks, and similar terms, evenif they are not identified as such, is not to be taken as an expression of opinion as to whetheror not they are subject to proprietary rights.

Capability Maturity Model and CMM are registered trademarks of the Software EngineeringInstitute and Carnegie Mellon University. Testing Maturity Model and TMM are service marksof Illinois Institute of Technology.

Printed in the United States of America.

9 8 7 6 5 4 3 2 1 SPIN 10779083

www.springer-ny.com

Springer-Verlag New York Berlin HeidelbergA member of BertelsmannSpringer Science�Business Media GmbH

C O N T E N T S

Preface xv

1I N T RODUCT I ON TO T E S T I N GA S AN ENG I N E E R I N G ACT I V I T Y1.0 The Evolving Profession of Software Engineering 1

1.1 The Role of Process in Software Quality 4

1.2 Testing as a Process 6

1.3 Overview of the Testing Maturity Model (TMM) 8

1.3.1 TMM Levels 10

List of Key Terms 16

Exercises 16

References 17

2 T E S T I N G FUNDAMENTA L S2.0 Introduction 19

2.1 Basic Definitions 19

vi | Contents

2.2 Software Testing Principles 26

2.3 The Tester’s Role in a Software Development Organization 34

List of Key Terms 36

Exercises 36

References 37

3 D E F ECTS , H Y POTH E S E S , A ND T E S T S

3.0 Origins of Defects 39

3.1 Defect Classes, the Defect Repository, and Test Design 43

3.1.1 Requirements and Specification Defects 44

3.1.2 Design Defects 46

3.1.3 Coding Defects 48

3.1.4 Testing Defects 51

3.2 Defect Examples: The Coin Problem 51

3.3 Developer/Tester Support for Developing a Defect Repository 57

List of Key Terms 58

Exercises 58

References 59

4 S TRAT EG I E S AND METHODS FOR T E S T CAS E D E S I GN I

4.0 Introduction to Testing Design Strategies 61

4.1 The Smart Tester 62

4.2 Test Case Design Strategies 63

4.3 Using the Black Box Approach to Test Case Design 66

4.4 Random Testing 66

4.5 Equivalence Class Partitioning 67

4.6 Boundary Value Analysis 72

4.7 An Example of the Application of Equivalence Class Partitioning and

Boundary Value Analysis 73

4.8 Other Black Box Test Design Approaches 76

4.8.1 Cause-and-Effect Graphing 78

4.8.2 State Transition Testing 82

4.8.3 Error Guessing 85

vi iContents |

4.9 Black Box Testing and Commercial Off-the-ShelfComponents (COTS) 86

4.10 Black Box Methods and TMM Level 2 Maturity Goals 88

List of Key Terms 91Exercises 92References 95

5 S TRAT EG I E S AND METHODS FOR T E S T CAS E D E S I GN I I

5.0 Using the White Box Approach to Test Design 975.1 Test Adequacy Criteria 985.2 Coverage and Control Flow Graphs 1015.3 Covering Code Logic 1035.4 Paths: Their Role in White Box–Based Test Design 1085.5 Additional White Box Test Design Approaches 111

5.5.1 Data Flow and White Box Test Design 1115.5.2 Loop Testing 1155.5.3 Mutation Testing 116

5.6 Evaluating Test Adequacy Criteria 1185.7 White Box Testing Methods and the TMM 124

List of Key Terms 127Exercises 127References 130

6 L E V E L S O F T E S T I N G

6.0 The Need for Levels of Testing 133

6.0.1 Levels of Testing and Software Development Paradigms 135

6.1 Unit Test: Functions, Procedures, Classes, and Methods as Units 1376.2 Unit Test: The Need for Preparation 1386.3 Unit Test Planning 1396.4 Designing the Unit Tests 1416.5 The Class as a Testable Unit: Special Considerations 1426.6 The Test Harness 1486.7 Running the Unit Tests and Recording Results 150

vi i i | Contents

6.8 Integration Test: Goals 152

6.9 Integration Strategies for Procedures and Functions 153

6.10 Integration Strategies for Classes 158

6.11 Designing Integration Tests 159

6.12 Integration Test Planning 162

6.13 System Test: The Different Types 163

6.13.1 Functional Testing 166

6.13.2 Performance Testing 167

6.13.3 Stress Testing 169

6.13.4 Configuration Testing 171

6.13.5 Security Testing 172

6.13.6 Recovery Testing 175

6.14 Regression Testing 176

6.15 Alpha, Beta, and Acceptance Tests 176

6.16 Summary Statement on Testing Levels 178

6.17 The Special Role of Use Cases 179

6.18 Levels of Testing and the TMM 181

List of Key Terms 184

Exercises 184

References 186

7 T E S T GOA L S , P O L I C I E S , P L ANS , A ND DOCUMENTAT I ON

7.0 Introductory Concepts 189

7.1 Testing and Debugging Goals and Policies 191

7.2 Test Planning 197

7.3 Test Plan Components 200

7.4 Test Plan Attachments 216

7.4.1 Test Design Specifications 217

7.4.2 Test Case Specifications 218

7.4.3 Test Procedure Specifications 220

7.5 Locating Test Items: The Test Transmittal Report 221

7.6 Reporting Test Results 221

7.7 The Role of the Three Critical Groups in Test Planning and

Policy Development 226

ixContents |

7.8 Process and the Engineering Disciplines: The Role of the

Individual as a Process Facilitator 230

List of Key Terms 231

Exercises 231

References 232

8 TH E T E S T ORGAN I Z A T I ON

8.0 Introducing the Test Specialist 235

8.1 Skills Needed by a Test Specialist 237

8.2 Building a Testing Group 240

8.3 The Structure of the Testing Group 242

8.4 The Technical Training Program 247

8.5 Career Paths for Testers: An Industry Example 250

8.6 Tester Certification 252

8.7 Integrating Testing Activities in the Software Life Cycle 253

8.8 The Test Organization, Technical Training Program, and Test Integration:

Support from the Three Critical Views 257

Exercises 261

References 262

9CONTRO L L I N G AND MON I TOR I NGTH E T E S T I N G PROC E S S9.0 Defining Terms 263

9.1 Measurements and Milestones for Controlling and Monitoring 266

9.1.1 Measurements for Monitoring Testing Status 271

9.1.2 Measurements for Monitoring Tester Productivity 275

9.1.3 Measurements for Monitoring Testing Costs 276

9.1.4 Measurements for Monitoring Errors, Faults and Failures 277

9.1.5 Monitoring Test Effectiveness 279

9.2 Status Meetings, Reports, and Control Issues 283

9.3 Criteria for Test Completion 289

9.4 Software Configuration Management 292

x | Contents

9.5 Controlling and Monitoring: Three Critical Views 296

List of Key Terms 300

Exercises 300

References 302

10 R E V I EWS AS A T E S T I N G ACT I V I T Y

10.0 Expanding the Testing Activity Umbrella 303

10.1 Types of Reviews 307

10.1.1 Inspections as a Type of Technical Review 308

10.1.2 Walkthroughs as a Type of Technical Review 310

10.2 Developing a Review Program 311

10.3 The Need for Review Policies 313

10.4 Components of Review Plans 314

10.4.1 Review Goals 315

10.4.2 Preconditions and Items to be Reviewed 315

10.4.3 Roles, Participants, Team Size, and Time Requirements 317

10.4.4 Review Procedures 320

10.4.5 Review Training 320

10.4.6 Review Checklists 324

10.5 Reporting Review Results 333

10.6 Review, Rework, and Follow-Up 337

10.7 Review Metrics 337

10.8 Support from the Extended/Modified V-Model 340

10.9 The Self-Check or Personal Review 340

10.10 Reviews and the TMM Critical Views 343

List of Key Terms 345

Exercises 345

References 347

11A MEASUR EMENT PROGRAM TO SUPPORTPRODUCT AND PROC E S S QUA L I T Y11.0 The Need for a Formal Test Measurement Program 349

11.1 Some Measurement-Related Definitions 353

xiContents |

11.2 Initiating a Measurement Program 354

11.3 Software Quality Evaluation 364

11.4 Measurement and TMM Levels 372

11.4.1 Measurements for TMM Level 1 373

11.4.2 Measurements for TMM Level 2 375

11.4.3 Measurements for TMM Level 3 377

11.4.4 Measurements for TMM Level 4 381

11.4.5 Measurements for TMM Level 5 383

11.5 A Test Measurement Program, Software Quality Valuationsand the Three Critical Views 386

List of Key Terms 389

Exercises 389

References 391

12E V A LUAT I NG SO F TWAR E QUA L I T Y :A QUANT I T A T I V E A P PROACH

12.0 Review of Quality Concepts 393

12.1 Quality Costs 395

12.2 What Is Quality Control? 397

12.3 The Role of Operational Profiles and Usage Models inQuality Control 399

12.4 Support for Quality Control: Statistical Testing 407

12.5 Software Reliability 410

12.5.1 Measurements for Software Reliability 413

12.6 Reliability, Quality Control, and Stop-Test Decisions 414

12.6.1 Applying Reliability Models 417

12.7 Confidence Levels and Quality Control 422

12.8 Usability Testing and Quality Control 424

12.9 An Approach to Usability Testing 425

12.9.1 Exploratory Usability Testing 426

12.9.2 Assessment Usability Testing 427

12.9.3 Validation Usability Testing 427

12.9.4 Comparison Test 429

12.9.5 Usability Testing: Resource Requirements 429

12.9.6 Usability Tests and Measurements 430

xi i | Contents

12.10 Software Quality Control and the Three Critical Views 433

List of Key Terms 436Exercises 436References 437

13 D E F ECT ANA L Y S I S AND PR E V ENT I ON

13.0 Processes and Defects 43913.1 History of Defect Analysis and Prevention 44113.2 Necessary Support for a Defect Prevention Program 44413.3 Techniques for Defect Analysis 44713.4 Defect Causal Analysis 45013.5 The Action Team: Making Process Changes 45413.6 Monitoring Actions and Process Changes 45713.7 Benefits of a Defect Prevention Program 45913.8 Defect Prevention and the Three Critical Views 460

Exercises 462References 463

14 TH E T E S T E R S ’ WORKB ENCH

14.0 Goals for the Testers’ Workbench 46514.1 Evaluating Testing Tools for the Workbench 46714.2 Tool Categories 470

14.2.1 Maturity Goals for TMM Level 1-Initial 47214.2.2 Tools for TMM Level 1 47214.2.3 TMM Level 2: Maturity Goals for Phase Definition 47414.2.4 Tools for Phase Definition 47514.2.5 TMM Level 3: Maturity Goals for Integration 47814.2.6 Tools for Integration 48014.2.7 TMM Level 4: Maturity Goals for Management and

Measurement 48714.2.8 Tools for Management and Measurement 48914.2.9 TMM Level 5: Maturity Goals for Optimization/Defect

Prevention/Quality Control 49214.2.10 Tools for Optimization/Defect Prevention/Quality

Control 494

xi i iContents |

14.3 The Testers’ Workbench and the Three Critical Views 498

Exercises 500

References 501

15 PROC E S S CONTRO L AND OPT IM I Z A T I ON

15.0 TMM Maturity Goals: Support for a Quality Testing Process 503

15.1 Process Engineering and Quality Control 504

15.2 Fundamentals of Quantitative Process Control 509

15.3 Activities for Quantitative Test Process Control 512

15.4 Examples of the Application of Statistical Process Control 516

15.5 Test Process Optimization: The Role of a Process ImprovementGroup 518

15.6 Technology Transfer 523

15.7 Process Reuse 526

15.7.1 Templates for Reusable Processes 529

15.7.2 Procedures for Process Reuse 531

15.8 Activities, Tasks and Responsibilities for Test ProcessControl and Optimization 533

Exercises 535

References 536

16TH E T E S T I N G MATUR I T Y MOD E L ANDT E S T PROC E S S A S S E S SMENT16.0 The Need for a Testing Maturity Model 537

16.1 Approach to Model Development 538

16.2 Process Improvement Model Representation 543

16.3 The TMM Structure: The Testing Maturity Levels 545

16.4 The TMM Assessment Model: Design Approach 548

16.5 The TMM Assessment Model Components 549

16.5.1 Assessment Team Selection and Training 549

16.5.2 The Assessment Procedure 551

16.5.3 The TMM Assessment Questionnaire 556

16.6 The TMM Ranking Procedure 558

16.7 Forms and Tools for Assessment Support 562

xiv | Contents

16.8 Relationship of the TMM to Other Process Improvement Models 563

16.9 Industrial Applications of the TMM 569

16.9.1 TMM Application I: Evaluating the Usability of the TMMQuestionnaire 569

16.9.2 TMM Application II: Identifying Test Problem Areas andRisks 572

16.9.3 TMM Application III: Software Test Consulting 573

16.9.4 TMM Application IV: Role of Human Factors in ProcessAssessment 576

16.9.5 Lessons Learned from the TMM Studies 581

References 583

A P P END I X I : T E S T - R E L A T E D R E F E R ENC E S 5 8 7

A P P END I X I I : S AMP L E T E S T P L AN 6 1 1

A P P END I X I I I : T E S T I N G MATUR I T Y MOD E L 6 3 3

Part 1: The TMM Questionnaire 633

Section 1. Instructions for the Respondent 634

Section 2. Respondent Identification and Background 635

Section 3. Organizational Background 637

Section 4. The TMM Questions 639

Section 5. Testing Tool Questions 659

Section 6. Testing Trends Questions 662

Section 7. Comments from Respondents 663

Section 8. Glossary of TMM-Related Terms 663

Part 2: TMM Activities, Tasks and Responsibilities 670

Index 701

P R E F A C E

oftware development is evolving into an engineering discipline. Indica-S tions of this new direction can be found, for example, in the ‘‘SoftwareEngineering Body of Knowledge (SWEBOK)’’ and the code of ethics thathave been developed recently through the efforts of joint IEEE/ACM taskforces [1,2]. Licensing procedures for software engineers are also underdevelopment. Software testing is a subdiscipline in this emerging field.The software industry is actively seeking and promoting professionalswho are educated and trained in the areas of testing and quality assurance,and who will promote the development of high-quality software.

Graduate schools have slowly been responding to this industry need,and a growing number are offering courses focused on software testingand quality assurance as part of advanced degree programs in softwareengineering. To support these programs, as well as the educational needsof practicing professionals in the industry, a new type of book on softwaretesting is needed. The book should have an engineering/process orienta-tion, and promote the growth and value of software testing as a profes-sion. This text was developed to meet these needs. It has been designedto serve as (i) a text for students enrolled in a graduate-level testing/qualityassurance class, and (ii) a knowledge source and learning tool for profes-sionals currently working in the field.

xvi | Preface

The text is unique in its approach to presenting the field of softwaretesting. It introduces testing concepts that are managerial, technical, andprocess-oriented in nature. Process is emphasized because of its essentialrole in all engineering disciplines. The widespread application of the Ca-pability Maturity Model (CMM)� and other process improvement mod-els attests to the importance of process in today’s software developmentindustry. Unfortunately, discussions of this topic are lacking in the ma-jority of books on software testing.

The author makes use of the Testing Maturity Model (TMM)SM,whichwas developed to support organizations in assessing and improving theirtesting processes, as a guiding framework for presenting testing concepts,and as a context for introducing the reader to test process issues. The textuses TMM levels and goals to support a structured presentation of fun-damental and advanced test-related concepts to the reader. The TMMstructure highlights the important relationshipsbetween the testingprocessand key players such as managers, testers, and client groups. The readershould note that adaptation of the Testing Maturity Model is not a nec-essary condition for using this text to learn about software testing. Usingthis text, you can learn about good testing practices and test process issuesand apply them in the context of your individual and organizational needs.

Finally, the author believes that educational material developed forsoftware engineers should be guided by the contents of the Software En-gineering Body of Knowledge (SWEBOK). In this context this text en-deavors to cover many of the topics outlined in the ‘‘Software Testing’’chapter of the SWEBOK. It also covers material from the chapters on‘‘Software Quality’’ and ‘‘Software Engineering Process’’

G o a l s

In view of the growth of the software engineering profession, the educa-tional requirements of a software testing specialist, and the need for em-phasis on process issues, the author’s goals for this text are to:

• introduce testing concepts, techniques, and best practices in a system-atic way that reflects an orderly evolution of testing process growthon both an individual and organizational level;

xvi iPreface |

• introduce a view of testing as a process that goes through a set ofevolutionary stages to an optimal state of continuous improvement;

• introduce software quality concepts, standards, measurements, andpractices that support the production of quality software;

• enable a software professional to build an individual testing processof the highest caliber that is integratable with an organizational test-ing process;

• enable a software professional to serve as an agent for changewhen an organization decides that its overall testing process needsimprovement;

• introduce the concepts of test process evaluation and improvementand their importance to the software industry;

• support the growth of the profession of software test specialist byproviding the educational background necessary for a professional inthat field.

O r g a n i z a t i o n a n d F e a t u r e s

Each chapter in this text covers a managerial, technical and/or process-related topic related to testing. The topics are designed to support thereader’s growth as a test specialist. Within each chapter, the relationshipof chapter contents to one or more TMM maturity goals is described.The first nine chapters contains basic material that allows the reader tomaster fundamental testing concepts on the technical level, and to learnabout basic managerial concepts that promote a repeatable and definedtesting process. These chapters also highlight the importance of an inde-pendent test group, and promote monitoring and controlling of the testingprocess. Maturity goals at levels 2 and 3 of the TMM are integrated intothe chapter material .

Chapters 10–15 cover more advanced topics related to levels 4 and5 of the TMM. These chapters support reviews as a testing activity, andthe automation of testing activities with tools. They also promote quali-tative and quantitative evaluation of the test process and its continuousevolution. Qualitative and quantitative evaluation of the software prod-uct under test is also addressed. Chapter 16 provides a discussion of test

xvi i i | Preface

process assessment using the TMM Assessment Model, and describessome applications of the TMM in industry.

The last sections of the text are its appendices. Appendix I, called‘‘Supplementary References,’’ contains a collection of test-related refer-ences which the reader will find useful to supplement the material in thetext. In this appendix a complete bibliography, organized alphabeticallyby author is presented that includes all references in the book chapters.It also contains a listing of additional textbooks, papers and Internet sitesthat are rich sources of material for the test specialist. They supportcontinual professional growth in a rapidly evolving field. Appendix IIcontains a sample test plan to illustrate the typical contents of such adocument. Appendix III contains the TMM Questionnaire, ranking al-gorithms, and the full set of TMM Activities, Tasks, and Responsibilities(ATRs) for those readers interested in test process assessment.

Other features to note in this text include definitions of key terms ineach chapter which are shown in italics. At the end of most of the chaptersthe reader will find exercises that will help him/her to learn the conceptsthat are discussed. Some exercises provide hands-on experience in apply-ing the concepts. A set of references is included at the end of each chapterfor the reader who would like a more in-depth discussion of the topics.

This text is one of the tools you can use to develop as a professionalsoftware tester. To use the text effectively you should have a backgroundin basic software engineering concepts, and some experience in softwaredevelopment. The best approach to learning the material is to read thechapters carefully and work out the exercises in the back of each chapter.Feedback from an instructor with respect to homework exercises andexaminations is also very valuable. Discussions with instructors, class-mates, and/or colleagues will also help you to integrate and clarify con-cepts. It is the author’s objective to assist you in accumulating the knowl-edge and expertise you need to develop as a professional software tester.

I n t e n d e d A u d i e n c e

Readers who would benefit from this text are senior undergraduates andgraduate students in computer science and software engineering pro-grams, and software professionals who are interested in improving theirtesting skills and learning more about testing as a process. For students,

xixPreface |

the text is a tool that can be used to develop the testing skills necessaryto become a professional software tester. For those in the software in-dustry it can help to enhance testing skills, and provide guidelines forevaluating and improving organizational testing processes. To use the texteffectively, readers should have a background in basic software engineer-ing concepts and some experience in developing software.

N o t e s t o E d u c a t o r s

This text can be used for several types of graduate courses including thosein software testing, software quality assurance, software verification andvalidation, and systems engineering. It can also be used as a text for anundergraduate two-semester software engineering course.

For educators using this book as a text for a one-semester course insoftware testing, covering the first ten chapters and Chapter 14, will giveyour students a solid foundation in testing fundamentals so that they candevelop into professional software testers. Chapters covering more ad-vanced topics, including the TMM, can be discussed if time permits. Stu-dents should be assigned homework problems from the chapters and re-ceive feedback on their results. A suggested team project for the course isthe development of a system test plan with attachments for a simple soft-ware system. Students will need a requirements and/or design descriptiondepending on the nature of the requested test plan.

For software professionals using this text, there is much material thatcan help to enhance your knowledge of the testing field. The materialrelating to the TMM can be applied to evaluate and make changes in yourtesting process in a manner consistent with organizational goals.

P e r m i s s i o n s

IEEE term definitions, test plan components, and steps in a software qual-ity metrics methodology reprinted with permission from:

IEEE Standard Glossary of Software Engineering Terminology (IEEEStd 610.12-1990), copyright 1990 by IEEE

IEEE Standard for Software Test Documentation (ANSI/IEEE Std829–1983), copyright 1983 by IEEE.

xx | Preface

IEEE Standard for a Software Quality Metrics Methodology (IEEE Std1061–1992), copyright 1993, by IEEE.

The IEEE disclaims any responsibility or liability resulting from the place-ment and use in the described manner.

Pearson Education has granted permission for use of material from‘‘Software Metrics: Establishing a Company-Wide Program’’ by Gradyand Caswell.[1] A. Abran, J. Moore, P. Bourque, R. Dupuis, editors, ‘‘Guide to the Software EngineeringBody of Knowledge, Trial Version,’’ IEEE Computer Society Press, Los Alamitos, CA, 2001.

[2] D. Gotterbarn, K. Miller, S. Rogerson, ‘‘Computer Society and ACM Approve SoftwareEngineering Code of Ethics,’’ IEEE Computer, Vol. 32, No. 10, 1999, pp. 84–88.

A c k n o w l e d g m e n t s

In preparing this text I have had support from many people includingfamily, colleagues, students, and publishers. The support has been inmany different forms. I would first like to thank my university, IllinoisInstitute of Technology, for granting me a sabbatical leave that enabledme to complete a good portion of this text. Colleagues who have beensupportive of my work include Professor Anneliese A. Andrews,(Colorado State University), Professor Robert Carlson (Illinois Instituteof Technology), and Professor Martha Evens (Illinois Institute ofTechnology).

I have used drafts of this text in my ‘‘Software Testing and QualityAssurance’’ class over the last two years and I would like to thank thestudents in these classes (CS 589) for their comments about the text. Ms.Yachai Limpiyakorn, who was the teaching assistant for the course, hasalso provided useful comments.

I would like to acknowledge the major contributions of Drs. TaratipSuwannasart, and Ariya Homyen (Wichitnuntakorn) to the developmentof the Testing Maturity Model during the course of their doctoral studies.The model provided the framework for the development of this text.My editors at Springer-Verlag, in particular, Wayne Wheeler and WayneYuhasz, have been very patient, and have provided suggestions and usefulcomments that I have incorporated into the text. Anonymous reviewershave also been very helpful in suggesting changes that improved the textquality.

xxiPreface |

Finally, I would like to thank my husband, Ray Burnstein for hisencouragement, and advice in the writing of this text, and for always‘‘being there’’ for me. I would like to thank my sons Kenneth and Jona-than who have expressed enthusiasm for this authoring project. Thankyou one and all!

Ilene Burnstein