Dr. Thomas Hicks Computer Science Department Trinity University 1 Software Engineering CSCI 3321.

Dr. Thomas HicksComputer Science Department

Trinity University 1

Software EngineeringCSCI 3321

Software Engineering : A Practitioner’s Approach

Software Testing Strategies

Dr. Thomas E. HicksComputer Science Department

Trinity University

Thanks To Ian Sommerville & Roger Pressman For Much Of The Slide Content

Chapter 13

AboutSoftware Testing

Software Testing

Software Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user.

Testing is the exposure of a system to trial input to see whether it produces the correct output

What Testing Shows 4 Things!

1. errors1. errors

2. requirements conformance2. requirements conformance

3. performance3. performance

4. an indication4. an indicationof qualityof quality

What is a “Good” Test?

1. A Good Test has a high probability of finding an error

2. A good test is not redundant.

3. A good test should be “best of breed”

4. A good test should be neither too simple nor too complex

Test Case Design

""BugsBugs lurklurk inin cornerscorners andand congregatecongregate atat boundariesboundaries ..." ..."

BorisBoris BeizerBeizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

withwith aa minimumminimum ofof efforteffort andand timetime

Software Testing: The Basics of the Trade (Kindle Edition) by Gaia Asher

The Art of Software Testing, Second Edition by Glenford J. Myers, Corey Sandler, Tom Badgett, and Todd M. Thomas

Software Quality Approaches: Testing, Verification, and Validation: Software Best Practice 1 (Software Best Practice, 1) by Michael Haug, Eric W. Olsen, and Luisa Consolini

Professional Software Testing with Visual Studio 2005 Team System: Tools for Software Developers and Test Engineers (Programmer to Programmer) by Tom Arnold, Dominic Hopton, Andy Leonard, and Mike Frost (Paperback - Sep 11, 2007)

Software Testing (2nd Edition) by Ron Patton

Lessons Learned in Software Testing by Cem Kaner, James Bach, and Bret Pettichord

Foundations of Software Testing: Certification by Dorothy Graham, Erik van Veenendaal, Isabel Evans, and Rex Black

Testing Computer Software, 2nd Edition by Cem Kaner, Jack Falk, and Hung Q. Nguyen

Software Testing Foundations: A Study Guide for the Certified Tester Exam, 2nd Edition by Andreas Spillner, Tilo Linz, and Hans Schaefer

http://www.amazon.com/Software-Testing-2nd-Ron-Patton/dp/0672327988/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204043572&sr=8-1

http://www.amazon.com/Software-Testing-2nd-Ron-Patton/dp/0672327988/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204043572&sr=8-1

http://www.amazon.com/Lessons-Learned-Software-Testing-Kaner/dp/0471081124/ref=pd_bbs_sr_2?ie=UTF8&s=books&qid=1204043572&sr=8-2

http://www.amazon.com/Foundations-Software-Testing-ISTQB-Certification/dp/1844803554/ref=pd_bbs_sr_3?ie=UTF8&s=books&qid=1204043572&sr=8-3

http://www.amazon.com/Foundations-Software-Testing-ISTQB-Certification/dp/1844803554/ref=pd_bbs_sr_3?ie=UTF8&s=books&qid=1204043572&sr=8-3

http://www.amazon.com/Testing-Computer-Software-2nd-Kaner/dp/0471358460/ref=pd_bbs_sr_4?ie=UTF8&s=books&qid=1204043572&sr=8-4


http://www.amazon.com/Software-Testing-Foundations-Certified-Tester/dp/1933952083/ref=pd_bbs_6?ie=UTF8&s=books&qid=1204043572&sr=8-6





Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional by Rex Black

Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing by Rex Black

Software Testing Fundamentals: Methods and Metrics by Marnie L. Hutcheson

Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel

A Practitioner's Guide to Software Test Design by Lee Copeland

http://www.amazon.com/Pragmatic-Software-Testing-Effective-Professional/dp/0470127902/ref=pd_bbs_7?ie=UTF8&s=books&qid=1204043572&sr=8-7



http://www.amazon.com/Managing-Testing-Process-Practical-Techniques/dp/0471223980/ref=pd_bbs_8?ie=UTF8&s=books&qid=1204043572&sr=8-8





41.

Software Testing: A Craftsman's Approach, Third Edition by Paul C. Jorgensen (Hardcover - Feb 15, 2008)

Component-Based Software Testing with UML by Hans-Gerhard

Software Testing and Analysis: Process, Principles and Techniques by Mauro Pezze and Michal Young

Software Testing In The Real World: Improving The

Process (ACM Press) by Edward Kit

Software Testing: Testing Across the Entire Software Development Life Cycle by Gerald D. Everett and Raymond, Jr. McLeod

Who TestsSoftware?

http://images.google.com/imgres?imgurl=http://www.suu.edu/ss/success/images/testing.jpg&imgrefurl=http://www.suu.edu/ss/success/testing/&h=239&w=450&sz=18&hl=en&start=17&tbnid=SlslEio_ROIClM:&tbnh=67&tbnw=127&prev=/images%3Fq%3Dtesting%26svnum%3D10%26hl%3Den%26sa%3DN

Who Tests the Software?

DeveloperDeveloper Independent TesterIndependent Tester

Understands the system Understands the system but, will test "gently" and, but, will test "gently" and, is is driven by "delivery"driven by "delivery"

MustMust learn about the system,learn about the system,but, will attempt to break itbut, will attempt to break itand, is and, is driven by qualitydriven by quality

1. Misplaced Faith: Those Who Write The Software generally don’t believe they will find any faults and generally won’t find as many

2. Must Fix Faults: Testers Who have to fix any faults they find do not tend to find as many

3. Mixed Goals: Those Who Write The Software want the code to be fault free but effective testers must want to find faults

3 Reasons The Software Developer Should Not (be

the only one to )Test His/Her Own Software

DeveloperDeveloper

4 Major TestingPhases

Unit/ComponentUnit/ComponentTestingTesting

IntegrationIntegrationTestingTesting

Final/Beta/ValidationFinal/Beta/ValidationTestingTesting

System/AlphaSystem/AlphaTestingTesting

4 Major Testing Phases

Unit/ComponentUnit/ComponentTestingTesting

IntegrationIntegrationTestingTesting

Final/Beta/ValidationFinal/Beta/ValidationTestingTesting

System/AlphaSystem/AlphaTestingTesting

1. Unit/Component Testing: Testing of Code generally written by a single programmer


2. Integration Testing: Testing of Several Units, Combined To Form A Unit or SubSystem By Testing Team


3. System/Alpha Testing: Testing of The Finished Program, By Program Testers


4. Final/Beta Testing: Testing of The Finished Program, By End Users or Their Representatives


1. Unit/Component Testing: Testing of Code generally written by a single programmer

2. Integration Testing: Testing of Several Units, Combined To Form A Unit or SubSystem By Testing Team

3. System/Alpha Testing: Testing of The Finished Program, By Program Testers

4. Final/Beta Testing: Testing of The Finished Program, By End Users or Their Representatives

4 Major Testing Phases (Summary)

Testing Strategy

Testing Strategy

We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’

For conventional software

1. The module (component) is our initial focus

2. Integration of modules follows

For OO software

The testing focus is an OO class that encompasses attributes and operations and implies communication and collaboration

http://images.google.com/imgres?imgurl=http://www.ktf-split.hr/glossary/image/atom.gif&imgrefurl=http://www.ktf-split.hr/glossary/hr_o.php%3Fdef%3Datom&h=198&w=198&sz=6&hl=en&start=3&tbnid=tixeTqznw-SEYM:&tbnh=104&tbnw=104&prev=/images%3Fq%3Datom%26svnum%3D10%26hl%3Den

Successful Software Testers Do 7 Things: 1- 4

1. State testing objectives explicitly.

2. Understand the users of the software and develop a profile for each user category.

3. Develop a testing plan that emphasizes “rapid cycle testing.”

4. Build “robust” software that is designed to test itself

http://www.rpls.ws/Bev/Super/hero.JPG

Successful Software Testers Do 7 Things: 5 - 7

5. Use effective formal technical reviews as a filter

prior to testing

6. Conduct formal technical reviews to assess the

test strategy and test cases themselves.

7. Develop a continuous improvement approach for

the testing process.

Unit & Component

Testing

Unit Testing

modulemoduletoto bebe

testedtested

test casestest cases

resultsresults

softwaresoftwareengineerengineer

Unit/ComponentTesting

IntegrationTesting

Final/Beta/ValidationTesting

System/AlphaTesting

About Unit Testing’s Test Cases

interface interface locallocal datadata structuresstructuresboundaryboundary conditionsconditionsindependentindependent pathspathserrorerror handlinghandling pathspaths

modulemoduleto beto be

testedtested

test casestest casesUnit/Component

TestingIntegration

Testing


System/AlphaTesting

Unit Test Environment

ModuleModule

stubstub stubstub

Driver/HarnessDriver/Harness

RESULTSRESULTS

interfaceinterfacelocallocal datadata structuresstructuresboundaryboundary conditionsconditionsindependentindependent pathspaths

errorerror handlinghandling pathspaths

testtest casescases

Test Harness, also called a Test Driver,

is the supporting code and data used to

provide an environment for testing part

of a system in isolation.

Test Harness

A Stub is a dummy procedure, module,

or unit, that stands in for an unfinished

portion of the program.

Stub


IntegrationTesting


System/AlphaTesting

The Component Testing ProcessAlso Called Unit Testing

Component Testing

Component/Unit Testing is the testing of individual

program components

Component Testing is usually the responsibility of

the component developer (except sometimes for

critical systems)

Component Tests are derived from the developer’s

experience

IntegrationTesting

The Integration Testing Process

Integration Testing

Integration Testing is the testing of groups of components integrated to create a system or sub-system

Integration Testing is the responsibility of an independent testing team

Integration Testing is based on the system specifications Unit/Component

TestingIntegration

Testing


System/AlphaTesting

Integration Testing Strategies

Options:Options:•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy


IntegrationTesting


System/AlphaTesting

Top-Down Design Testing &

Implementation

http://images.google.com/imgres?imgurl=http://www.nasm.si.edu/research/dsh/artifacts/images/SS-Astro/A19830128000d.jpg&imgrefurl=http://www.nasm.si.edu/research/dsh/artifacts/SS-IUE.htm&h=421&w=550&sz=108&hl=en&start=28&tbnid=apDSnuLiIi57wM:&tbnh=102&tbnw=133&prev=/images%3Fq%3Dintegration%2Btest%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26sa%3DN

http://www.edwards.af.mil/ti/images/instlab2.jpg

Top-Down Integration & Testing

Create the top levelCreate the top levelmodule module AA


Create stubs for eachCreate stubs for eachAnd every module it callsAnd every module it calls

AA

Stub BStub B Stub CStub C Stub DStub D

Designing & Testing Stubs

http://images.google.com/imgres?imgurl=http://www.horsecart.com/images/horseharnesslg.jpg&imgrefurl=http://www.horsecart.com/harness.html&h=617&w=533&sz=39&hl=en&start=25&tbnid=8ittmpc6vKG3KM:&tbnh=136&tbnw=117&prev=/images%3Fq%3Dharness%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26sa%3DN

Upper Level – Calling Function/Procedure

void Goo (int X, int Y){int Temp = 20;

X = Foo (3,Temp);}

Create the Stub – Pass Something Known To It!

int Foo (int A, int & B){ }

void Goo (void){int Temp = 20 X = 5, Y = 6;. . . Temp = Foo (X,Y);. . .}

Create the Stub – Verify All In-Coming Communication

int Foo (int A, int & B){ printf (“Stub - Foo A = %ld B = %ld\n”, A, B);

}


Create the Stub – Alter All Reference Variables

int Foo (int A, int & B){ printf (“Stub -> Foo A = %ld B = %ld\n”, A, B);

A++; B++;

printf (“Stub <- Foo A = %ld B = %ld\n”, A, B);}


Create the Stub – Assign A Known Return


A++; B++; printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (1);}


Stubs are all about testing communication.

We do not yet know that this works effectively

Create the Stub – Verify Return Info


A++; B++; printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (1);}

void Goo (void){int Temp = 20 X = 5, Y = 6; printf (“X =% ld Y = %ld Temp = %ld\n”, X, Y, Temp); Temp = Foo (X,Y); printf (“X = %ld Y = %ld Tem p= %ld\n”, X, Y, Temp);}

It is much better yet to use one of your test cases for the stub.

Create the Stub – Verify Return Info

int Add (int Addend1, Addend2){ printf (“Stub -> Foo Addend1 = %ld ”, Addend1 printf (“Addend2 = %ld\n”, Addend2);

Addend1 = 1;Addend1 = 2;

// Let us assume that the module is complex – will fill in pieces later - communication printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (11);}

void Goo (void){int Sum = 20 X = 5, Y = 6; printf (“X =% ld Y = %ld Sum = %ld\n”, X, Y, Sum ); Sum = Add (X,Y); printf (“X = %ld Y = %ld Sum %ld\n”, X, Y, Sum );}

Remember, with the exception of arrays, modules should be able to change only

that data the module is designed to change. As Needed Basis!

Top-Down Design Testing &

Implementationreturn


Create the top levelCreate the top levelmodule module AA


Create stubs for eachCreate stubs for eachAnd every module it callsAnd every module it calls

AA

Stub BStub B Stub CStub C Stub DStub D


AA

BB Stub CStub C Stub DStub D

stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"


AA



Stub EStub E Stub FStub F Stub GStub G


AA



EE Stub FStub F Stub GStub G


AA




Stub HStub H


AA




HH


AA




HH

Stub IStub I Stub JStub J


AA




HH

II Stub JStub J


AA




HH

II JJ


AA



EE FF Stub GStub G

HH

II JJ


AA



EE FF GG

HH

II JJ


AA



EE FF GG

HH

II JJ

Stub KStub K Stub LStub L Stub MStub M


AA



EE FF GG

HH

II JJ

KK Stub LStub L Stub MStub M


AA



EE FF GG

HH

II JJ

KK LL Stub MStub M


AA



EE FF GG

HH

II JJ

KK LL Stub MStub M

Stub NStub N


AA



EE FF GG

HH

II JJ

KK LL Stub MStub M

NN


AA



EE FF GG

HH

II JJ

KK LL MM

NN


AA

BB CC Stub DStub D


EE FF GG

HH

II JJ

KK LL MM

NN


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN

Stub NStub N Stub OStub O


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN

NN Stub OStub O


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN

NN OO


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN

NN OO

Stub PStub P


AA

BB CC DD


EE FF GG

HH

II JJ

KK LL MM

NN

NN OO

PP

Top-Down can be done within units and

sub-systems

Top-Down Integration & TestingAlternate Implementation Strategy

Bottom-Up Design Testing &

Implementation

Bottom-Up Integration & Testing

Lowest level – bottom modules Lowest level – bottom modules tested first – Maybe start withtested first – Maybe start withModule AModule A

AA


Lowest level – bottom modules Lowest level – bottom modules tested firsttested firstAA

Harness - BHarness - B

Create just enough of a Create just enough of a harness to test Bharness to test B


AA


CCLowest level – bottom modules Lowest level – bottom modules tested first – Maybe continue tested first – Maybe continue with Module Cwith Module C


AA


Modify the harness to test EModify the harness to test E

CCLowest level – bottom modules Lowest level – bottom modules tested first – Maybe continue tested first – Maybe continue with Module Ewith Module E


AA


CC DD


AA


CC DD

Harness - EHarness - E


AA


CC DD


FF


AA


CC DD


FF GG


AA


CC FF


GG HH


AA


CC DD


FF GG HH


AA


CC DD


FF GG HH

Harness - IHarness - I


AA


CC DD


FF GG HH


JJ


AA

BB

CC DD


FF GG HH


JJ


AA

BB

CC DD


FF GG HH


JJ

Harness - KHarness - K


AA

BB

CC DD

EE

FF GG HH


JJ



AA

BB

CC DD

EE

FF GG HH


JJ



AA

BB

CC DD

EE

FF GG JHJH

II

JJ



AA

BB

CC DD

EE

FF GG HH

II

JJ

Harness - KHarness - K Harness - LHarness - L


DD

CC

EE FF

EE

HH II JJ

KK

LL

MM Harness - NHarness - N


DD

CC

EE FF

EE

HH II JJ

KK

LL

MM Harness - NHarness - N

Harness - OHarness - O


DD

CC

EE FF

EE

HH II JJ

KK

LL

MM NN



DD

CC

EE FF

EE

HH II JJ

KK

LL

MM NN



DD

CC

EE FF

EE

HH II JJ

KK

LL

MM NN

OO

Bottom-Up can be done within units and

sub-systems

Sandwich Testing &

Implementation

Sandwich Integration & Testing

MM NN

OO

Might Work On Upper Portion Some


JJ

KK

LL

MM NN

OO

Might Work On Lower Portions Some


DD

CC

EE JJ

KK

LL

MM NN

OO


More


DD

CC

EE FF

EE

HH II JJ

KK

LL

MM NN

OO


More


DD

CC

EE FF

EE

HH II JJ

KK

LL

MM NN

OO

Gradually Fit It All Together

No Specific Order!

OOP

Testing

http://www.bible.ca/tracks/alley-oop-living-speciman.jpg

Object-Oriented TestingOO Testing begins by evaluating the correctness

and consistency of the OO models

Testing Strategy Changes

In OO, the concept of the ‘unit’ testing broadens due to encapsulation

In OO, integration testing focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario

In OO, validation uses conventional black box methods

Test case design draws on conventional methods, but also encompasses special features

http://images.google.com/imgres?imgurl=http://i1.trekearth.com/photos/7039/oop.jpg&imgrefurl=http://www.trekearth.com/gallery/Europe/Netherlands/photo277333.htm&h=444&w=800&sz=135&hl=en&start=68&tbnid=Yo2TK1tMnTs00M:&tbnh=79&tbnw=143&prev=/images%3Fq%3Doop%26start%3D60%26ndsp%3D20%26svnum%3D10%26hl%3Den%26sa%3DN

SmokeTesting

3 Steps Of Smoke Testing

Smoke Testing is a common approach for creating “daily

builds” for product software

Smoke Testing Steps:

1. Software components, that have been translated into

code, are integrated into a “build.”

A build includes all data files, libraries, reusable

modules, and engineered components that are

required to implement one or more product

functions.

3 Steps Of Smoke Testing Smoke Testing Steps (cont):

2. A series of tests is designed to expose errors that will

keep the build from properly performing its function.

The intent should be to uncover “show stopper”

errors that have the highest likelihood of throwing

the software project behind schedule.

3. The build is integrated with other builds and the entire

product (in its current form) is smoke tested daily.

The integration approach may be top down or

bottom up.

“show stopper”Smoke TestingFind The Fires!

TestingHierarchy


IntegrationTesting


System/AlphaTesting

7 Testing Objectives

1. Validation Testing

Focus is on software requirements

2. System Testing

Focus is on system integration

3. Alpha/Beta Testing

Focus is on customer usage

4. Recovery Testing

forces the software to fail in a variety of ways and verifies that recovery is properly performed

7 Testing Objectives

5. Security Testing

verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration

6. Stress Testing

executes a system in a manner that demands resources in abnormal quantity, frequency, or volume

7. Performance Testing

test the run-time performance of software within the context of an integrated system


IntegrationTesting


System/AlphaTesting

Debugging

The Debugging Process

test casestest cases

resultsresults

DebuggingDebugging

suspectedsuspectedcausescauses

identifiedidentifiedcausescauses

correctionscorrections

regressionregressionteststests

new testnew testcasescases

“Art of Debugging”

A Diagnostic Process

Debugging Effort

time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause

time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests

Symptoms & Causes Of Program Bugs

symptomsymptomcausecause

symptom and cause may be symptom and cause may be geographically separatedgeographically separated

symptomsymptom maymay disappeardisappear whenwhen anotheranother problemproblem isis fixedfixed

causecause maymay bebe duedue toto a a combinationcombination ofof non-errorsnon-errors

causecause maymay bebe duedue toto a systema system oror compilercompiler errorerror

causecause maymay bebe duedue toto assumptionsassumptions thatthat everyoneeveryone believesbelieves

symptomsymptom maymay bebe intermittentintermittent

Consequences of Bugs

damagedamage

mildmild annoyingannoying

disturbingdisturbingseriousserious

extremeextremecatastrophiccatastrophic

infectiousinfectious

BugBug TypeType

Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.

http://www.learnasp.com/team/artwork/donkitchen/debug-computer.gif

3 Major Debugging Strategies

1.1. Brute Force – Brute Force – Most commonly used and least effective – use memory dumps, run traces, and/or many print statements.

2.2. Backtracking Backtracking – can be used In small programs – begin where error is found and manually backtrack examining logic and output statements until problem found.

3.3. Cause Elimination – Cause Elimination – use induction and deduction to produce lists of potential causes is identified. Test cases are constructed to systematically prove or disprove each potential hypothesis.

Debugging: Final Thoughts

1.1. Don't run off half-cocked, Don't run off half-cocked, think think about the about the symptom you're seeing.symptom you're seeing.

2.2. Use tools (e.g., dynamic debugger) to gain Use tools (e.g., dynamic debugger) to gain

more insight.more insight.

3.3. If at an impasse, get help from someone else.If at an impasse, get help from someone else.

4.4. Be absolutely sure to conduct regression Be absolutely sure to conduct regression

tests when you do “fix” the bug.tests when you do “fix” the bug.

http://www.procolix.nl/uploads/ZA/5E/ZA5EQNbp4K-shyPESBnLDw/debug.gif

Exhaustive

Testing

Exhaustive Testing is executing a program with all possible combinations of inputs or values for program variables.

Only Exhaustive Testing can show a program is free from defects. However, exhaustive testing is impossible

Testing Priorities

Exhaustive Testing

loop < 20 Xloop < 20 X

1414There are 10 possible paths! If we execute one test There are 10 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this per millisecond, it would take 3,170 years to test this

program!!program!!

SelectiveTesting

Selective Testing

loop < 20 Xloop < 20 X

Selected pathSelected path

There are 10 possible paths!There are 10 possible paths!

http://images.google.com/imgres?imgurl=http://www.through-the-maze.org.uk/symbols_x4/choose.gif&imgrefurl=http://www.through-the-maze.org.uk/symbol.php%3Fname%3Dchoose&h=480&w=480&sz=7&hl=en&start=2&tbnid=-pa7CnU8NhMBsM:&tbnh=129&tbnw=129&prev=/images%3Fq%3Dchoose%26gbv%3D2%26svnum%3D10%26hl%3Den

DefectTesting

Defect Testing

The Goal of Defect Testing is to discover defects in programs

A Successful Defect Test is a test which causes a program to behave in an anomalous way

Tests Show the Presence not the Absence of Defects

Only Exhaustive Testing can show a program is free from defects. However, exhaustive testing is impossible

Tests Should Exercise a system's capabilities rather than its components

Testing Old Capabilities is more important than testing new capabilities

Testing Typical Situations is more important than boundary value cases

Testing Priorities

Test Data are Inputs which have been devised to test the system

Test Cases are Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification

Test Cases Have 3 Parts Goal

Input & System State

Expected behavior

Test Data & Test Cases

http://images.google.com/imgres?imgurl=http://volcanoes.usgs.gov/Imgs/Jpg/Redoubt/dds39_063_large.jpg&imgrefurl=http://volcanoes.usgs.gov/About/What/Monitor/Hydrologic/AFMRedoubt.html&h=500&w=800&sz=24&hl=en&start=22&tbnid=FbhIZ4s4Iu_3CM:&tbnh=89&tbnw=143&prev=/images%3Fq%3Dtest%2Bcase%26start%3D20%26gbv%3D2%26ndsp%3D20%26svnum%3D10%26hl%3Den%26sa%3DN

Black-Box

Testing

Software Testing

Methods

Strategies

white-boxmethods

black-box methods

Black-Box Testing

requirementsrequirements

eventseventsinputinput

outputoutput

Black-Box Testing Black-Box Testing is an approach to testing where the

program is considered as a ‘black-box’; it is also known as function testing.

In Black-Box Testing the program test cases are based on the system specification

In Black-Box Testing is a software testing technique whereby the internal workings of the item being tested are not known by the tester

In Black-Box Testing on a software design, the tester only knows the inputs and what the expected outcomes should be and not how the program arrives at those outputs.

In Black-Box Testing the tester does not ever examine the programming code and does not need any further knowledge of the program other than its specifications.

Black-Box Testing – Concerns?

How is functional validity tested?

How is system behavior and performance tested?

What classes of input will make good test cases?

Is the system particularly sensitive to certain input values?

How are the boundaries of a data class isolated?

What data rates and data volume can the system tolerate?

What effect will specific combinations of data have on system operation?

Pros & Cons Of Black-Box Testing

The advantages of Black-Box testing include: 1. The test is unbiased because the designer and the tester are

independent of each other.

2. The tester does not need knowledge of any specific programming languages.

3. The test is done from the point of view of the user, not the designer.

4. Test cases can be designed as soon as the specifications are complete.

The disadvantages of Black-Box testing include: 1. The test can be redundant if the software designer has already

run a test case.

2. The test cases are difficult to design.

3. Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.

Black-Box Testing

Ie

Input test data

OeOutput test results

System

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects

How To Construct Test Cases

EquivalencePartitioning

Equivalence Partitioning

Input Data & Output Results often fall into different classes where all members of a class are related

Each of these Classes is an Equivalence Partition where the program behaves in an equivalent way for each class member

Test Cases should be chosen from each partition


useruserqueriesqueries mousemouse

pickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata

Sample Equivalence Classes

user supplied commandsuser supplied commands

responses to system promptsresponses to system prompts

file namesfile namescomputational datacomputational data physical parameters physical parameters bounding valuesbounding values initiation valuesinitiation valuesoutput data formattingoutput data formattingresponses to error messagesresponses to error messagesgraphical data (e.g., mouse picks)graphical data (e.g., mouse picks)

data outside bounds of the program data outside bounds of the program physically impossible dataphysically impossible dataproper value supplied in wrong placeproper value supplied in wrong place

Valid dataValid data

Invalid dataInvalid data

Boundary Value Analysis

useruserqueriesqueries

mousemousepickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata

outputoutputdomaindomaininput domaininput domain

Partition System Inputs and Outputs into ‘Equivalence Sets’

Suppose the input is a 5-digit integer between 10,000 and 99,999

Equivalence Partitions are

> 10,000 [Too Big]

10,000 - 99,999 [Expected]

< 10, 000 [Too Small]


Write Good Test Data For The Equivalence Partitions

Most Authorities Agree That This Equivalence Partitioning Should Have at least 7

Equivalence Partitions 5-digit integer between 10,000 and 99,999

> 10,000 [Too Big]

10,000 - 99,999 [Expected]

< 10, 000 [Too Small] Choose One Expected General Case

50,000 Choose One Clearly Too Big

150,000

Choose One Clearly Too Small

5,000

Test The Boundary Conditions

9,999 & 10,000 & 99,999 & 100,000

Testing Equivalence Partitioning


System

Outputs

Invalid inputs Valid inputs

There are a number of ways of writing a specification. Sometimes, the easiest way to communicate with

developers is with conditional logic. This would not the best way to communicate with stakeholders.

procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition-- the array has at least one elementT’FIRST <= T’LAST

Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)

or -- the element is not in the array( not Found and

not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

Binary Search Routine Specification






What Are The Search Routine Equivalence Partitions?

How Many Search Routine Equivalence Partitions?

1. Inputs which Conform to the Pre-Conditions

2. Inputs where a Pre-Condition Does Not Hold

3. Inputs where the Key Element is a Member of the Array

4. Inputs where the Key Element is Not a Member of the Array






4 Search Routine - Input Partitions

Testing Binary Search Routine SpecificationNot My Choice Of Data Representation –

Designer’s Choice!

1. Inputs where a Pre-Condition Does Hold

This Is Often The Case For The Software Testing Team

Max

Last

First0

1

2

3

4

5

6

7

6

2

12

13

34

45

66

7

T

Testing Binary Search Routine Specification

2. Inputs where a Pre-Condition Does Not Hold

Any Other Condition 2Scenario To Test?

Say Yes!

Max

Last

First0

1

2

3

4

5

6

7

0

4

8

9

4

5

6

7

T


Empty!Empty!

2.2. Inputs where a Pre-Condition Does Not HoldInputs where a Pre-Condition Does Not Hold

Max

Last

First0

1

2

3

4

5

6

7

-1

-1

7

T


3.3. Inputs where the Key Element is a Member of the ArrayInputs where the Key Element is a Member of the Array

Choose Key In Middle 34

Choose Key At Boundary Points 66 & 11

Check Return Found = True Check T[L] = Key

Max

Last

First0

1

2

3

4

5

6

7

6

2

11

13

34

45

66

7

Key

34

Found

T

L

4

T

Testing Binary Search Routine Specification4. Inputs where the Key Element is Not a Member of the Array

Choose Non-Existent Key In Middle 15

Choose Non-Existent Key Inside Boundary Points 12 & 65

Check Return Found = False

Key

15

Found

F

L

--

Max

Last

First0

1

2

3

4

5

6

7

6

2

11

13

34

45

66

7

T






A Good Software Testing Team Often Identifies & Corrects Weak Specifications

This Specification Has A Real Obvious Omission. Create A Test Case To This Specification Has A Real Obvious Omission. Create A Test Case To Illustrate The Weakness Of This Specification.Illustrate The Weakness Of This Specification.

Our Existing Examples Also Need Further Refinement!Our Existing Examples Also Need Further Refinement!


4. Inputs where the Key Element is Not a Member of the Array

Choose Non-Existent Key In MiddleChoose Non-Existent Key In Middle 1515

Choose Non-Existent Key Inside Choose Non-Existent Key Inside Boundary PointsBoundary Points 12 & 65 12 & 65

Choose Non-Existent Key Outside Choose Non-Existent Key Outside Boundary PointsBoundary Points67 & 1067 & 10

Check Return Found = FalseCheck Return Found = False

Key

67

Found

F

L

--

Max

Last

First0

1

2

3

4

5

6

7

6

2

11

13

34

45

66

7

T

Testing Binary Search Routine SpecificationParting Thoughts About Arrays!

Successful Test Case of 1 Element!

Unsuccessful Test Case of 1 Element!

Test With A Full Array To Verify That There Are No Subscript Errors In Which T[-1] or T[8] are erroneously examined!

Nothing in the specification addressed duplicates; does the algorithm break with a duplicate?

Nothing in the specification addressed duplicates; which duplicate should be returned?

Experience Helps!

Key

12

Found

T

L

?

Max

Last

First0

1

2

3

4

5

6

7

6

2

12

13

34

45

66

7

T

71

12

12

StructuralWhite-Box

Testing

Software Testing

Methods

Strategies

white-boxmethods

black-box methods

White-Box Testing

... our goal is to ensure that all statements and ... our goal is to ensure that all statements and conditions have been executed at least once ... conditions have been executed at least once ...

White-box Testing, sometimes called “structural testing”

or “clear box testing” or “open box testing”, is a software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data.

The Objective of Structural/White-Box Testing is to exercise all program statements (not all path combinations)

White-Box Testing derives test cases according to program structure. Knowledge of the program is used to identify these additional structural test cases

Structural Testing – White-Box Testing

White-Box testing uses specific knowledge of programming code to examine outputs. The test is accurate only if the tester knows what the program is supposed to do.

The White-Box tester can then see if the program diverges from its intended goal.

White box testing does not account for errors caused by omission, and all visible code must also be readable.

Structural Testing – White-Box Testing

Why Do White-Box Testing

LogicLogic errorserrors andand incorrectincorrect assumptions assumptions are inversely proportional to a path'sare inversely proportional to a path's execution probabilityexecution probability

TypographicalTypographical errorserrors areare random; it's random; it's likely that untested paths will contain likely that untested paths will contain some some

White-Box Testing

Componentcode

Testoutputs

Test data

DerivesTests

class BinSearch {

// This is an encapsulation of a binary search function that takes an array of// ordered objects and a key and returns an object with 2 attributes namely// index - the value of the array index// found - a boolean indicating whether or not the key is in the array// An object is returned because it is not possible in Java to pass basic types by// reference to a function and so return two values// the key is -1 if the element is not found

public static void search ( int key, int [] elemArray, Result r ){

int bottom = 0 ;int top = elemArray.length - 1 ;int mid ;r.found = false ; r.index = -1 ;while ( bottom <= top ){

mid = (top + bottom) / 2 ;if (elemArray [mid] == key){

r.index = mid ;r.found = true ;return ;

} // if partelse{

if (elemArray [mid] < key)bottom = mid + 1 ;

elsetop = mid - 1 ;

}} //while loop

} // search} //BinSearch

Binary search (Java)Consider White-Box

Paths

Need To Exercise Each

Line At Least Once!

Pre-conditions satisfied, key element in arrayPre-conditions satisfied, key element in array

Pre-conditions satisfied, key element not in Pre-conditions satisfied, key element not in arrayarray

Pre-conditions unsatisfied, key element in Pre-conditions unsatisfied, key element in arrayarray

Pre-conditions unsatisfied, key element not in Pre-conditions unsatisfied, key element not in arrayarray

Input array has a single valueInput array has a single value

Input array has an even number of valuesInput array has an even number of values

Input array has an odd number of valuesInput array has an odd number of values

Binary Search - Equivalent PartitionsBased On White-Box Examination Of The Code

Binary Search Equivalent Partitions

Mid-point

Elements < Mid Elements > Mid

Equivalence class boundaries

Search Routine White-Box Input PartitionsAs Explained In Text






Using the specification above as a starting point, create a good specification for

inserting a new item into the sorted array

Let us assume that our lowest element will always be in T[0] – No Duplicates Allowed

bool UniqueArrayInplace (in out long T[ ], long NewInfo, in out long Max, in out long

ActNo)

Write The Specification

Successful

T

NewInfo

50

Max

ActNo

Sorted0

1

2

3

4

5

6

7

7

T

12

13

34

45

66

7

T

2

1

UniqueT

Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in

out long ActNo)

Pre-Condition-- Not Full-Room For 1 More ActNo <= Max

Or

-- if Unique & Sorted – if Binary Search Successfulor -- if not sorted & Not Sorted – if Sequential Search Successful

Post Condition

-- if sorted - Add NewInfo To Correctly Ordered Position

Or

-- if not sorted - Append NewInfo To Next Available Position

Successful

T

NewInfo

50

Max

ActNo

Sorted0

1

2

3

4

5

6

7

7

T

12

13

34

45

66

7

T

2

1

UniqueT

Suppose This Is The Specification

Partition System Inputs and Outputs into Black-Box Inspection ‘Equivalence Sets’!

Write Them!Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max,

in out long ActNo)in out long ActNo)

Pre-ConditionPre-Condition-- Not Full-Room For 1 More ActNo <= Max-- Not Full-Room For 1 More ActNo <= Max

OrOr

-- if Unique & Sorted – if Binary Search Successful-- if Unique & Sorted – if Binary Search Successfuloror -- if not sorted & Not Sorted – if Sequential Search Successful -- if not sorted & Not Sorted – if Sequential Search Successful

Post ConditionPost Condition

-- if sorted - Add NewInfo To Correctly Ordered Position-- if sorted - Add NewInfo To Correctly Ordered Position

OrOr

-- if not sorted - Append NewInfo To Next Available Position-- if not sorted - Append NewInfo To Next Available Position

Suppose This Is The Specification

We could have you write the ArrayInplace function in C Bool function BinSearch (T: ELEM_ARRAY, SoughtInfo ELEM, long ActNo, long Position

Bool function SequentialSearch (T: ELEM_ARRAY, SoughtInfo ELEM, long ActNo, long Position)

Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in out long ActNo)long Max, in out long ActNo)

Pre-ConditionPre-Condition-- Not Full-Room For 1 More ActNo <= Max-- Not Full-Room For 1 More ActNo <= Max

OrOr

-- if Unique & Sorted – if Binary Search Successful-- if Unique & Sorted – if Binary Search Successfuloror -- if not sorted & Not Sorted – if Sequential Search Successful -- if not sorted & Not Sorted – if Sequential Search Successful

Post ConditionPost Condition

-- if sorted - Add NewInfo To Correctly Ordered Position-- if sorted - Add NewInfo To Correctly Ordered Position

OrOr

-- if not sorted - Append NewInfo To Next Available Position-- if not sorted - Append NewInfo To Next Available Position

Examine your C solution!

Write the White-Box Inspection ‘Equivalence Sets’!

Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in out long ActNo)


Or


Post Condition


Or


Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in

out long ActNo)


Or


Post Condition


Or


Successful

T

NewInfo

50

Max

ActNo

Sorted0

1

2

3

4

5

6

7

7

T

12

13

34

45

66

7

T

2

1

UniqueT

PathTesting

Path TestingBinary-Search Example

The Objective of Path Testing is to ensure that the set of test cases is such that each path/branch through the program is executed at least once

1

2

3

4

65

7

while bottom <= top

if (elemArray [mid] == key

(if (elemArray [mid]< key8

9

bottom > top

The Starting Point for Path Testing is a Program Flow Graph that shows Nodes representing Program Decisions and Arcs Representing the Flow of Control

Statements With Conditions are therefore Nodes in the flow graph

Binary Search Cyclomatic Complexity = _______________

Program Flow Graphs

Program Flow Graphs describes the program control flow. Each branch is shown as a separate path and loops are shown by arrows looping back to the loop condition node

1

2

3

4

65

7

while bottom <= top



9

bottom > top

The Program Flow Graph is used as a basis for computing

the Cyclomatic Complexity Cyclomatic Complexity = # Edges - # Nodes + 2

11 - 9 + 2 = 4

Program Flow Graphs

Cyclomatic Complexity =

7 – 5 + 2 = 4

# Edges - # Nodes + 2

Sketch The Program Flow Graph

If (. . .) Then{ X = . . . . . .}Else{ Y + . . . . . .}

Start 1

End 2

3 – 3 + 2 = 2

Cyclomatic Complexity


If - 3

a

TF

bc

List The Paths For The Program Flow Graph

If (. . .) Then{ X = . . . . . .}Else{ Y + . . . . . .}

3 – 3 + 2 = 2


Testing Paths


ab ac

Start 1

End 2

If - 3

a

TF

bc

Sketch the Program Flow GraphsIf (. . .) Then{ Y = . . . Z = . . . While (. . .) { If (. . .) Then . . . Else . . . }}Else . . .

Start 1

End 2

While - 4

If - 5



If - 3

a

TF

c

b

T

F

d

eT F

f g

7 - 5 + 2 = 4

If (. . .) Then{ Y = . . . Z = . . . While (. . .) { If (. . .) Then . . . Else . . . }}Else . . .

Start 1

End 2

While - 4

If - 5



If - 3

a

TF

c

b

T

F

d

eT F

f g


7 - 5 + 2 = 4

Testing Paths

ac abe

abdfe abdge

Sketch The Program Flow Graph

Number the nodes 1,2,3,…Label the edges a,b,c,d,…

Compute Cyclomatic ComplexityList all of the paths

Sketch The Program Flow GraphIf (. . .) Then{ X = . . . If (. . .) Then { While (. . .) { If (. . .) Then . . . } } Else . . .}Else{ For (. . . ) { If (. . .) Then . . . Else . . . }

}

Number the nodes 1,2,3,…Label the edges a,b,c,d,…

Compute Cyclomatic ComplexityList all of the paths

Graph Matrices

A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph

Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes.

By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing

Cyclomatic Complexity - 1

Cyclomatic Complexity is the most widely used member of a class of static software metrics.

Cyclomatic Complexity may be considered a broad measure of soundness and confidence for a program.

Cyclomatic Complexity was introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.

Cyclomatic Complexity provides a single ordinal number that can be compared to the complexity of other programs

http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html

Cyclomatic Complexity – 2“McCabe’s Complexity” - “Program Complexity”

Cyclomatic Complexity is often referred to simply as “Program Complexity”, or as “McCabe's Complexity”.

Cyclomatic Complexity is often used in concert with other software metrics. As one of the more widely-accepted software metrics, it is intended to be independent of language and language format

Cyclomatic Complexity has also been extended to encompass the design and structural complexity of a system

Cyclomatic Complexity – 3Assists In Risk Evaluation!

Cyclomatic Complexity Risk Evaluation

1-10 a simple program, without much risk

11-20 more complex, moderate risk

21-50 complex, high risk program

greater than 50 untestable program (very high risk)

$$

Someone Has To Estimate The Cost For Each & Every Project

http://www.mccabe.com/iq_research_metrics.htm



Other Complexity Instruments

Complexity Measurement Primary Measure of

Halstead Complexity Measures

Algorithmic complexity, measured by counting operators and operands

Henry and Kafura metrics

Coupling between modules (parameters, global variables, calls)

Bowles Metrics Module and system complexity; coupling via parameters and global variables

Troy and Zweben Metrics

Modularity or coupling; complexity of structure (maximum depth of structure chart); calls-to and called-by

Ligier Metrics Modularity of the structure chart

List All Of The Paths For The Binary Search!

Through 1-2-8-9

Through1-2-3-8-9

Through1-2-3-4-6-7

Through1-2-3-4-5-7

What Was TheCyclomatic ComplexityFor The Binary Search?

4

How Many PathsFor The Binary Search?

4?


Cyclomatic Complexity =

7 – 5 + 2 = 4


A D

A B C

A B F G

A B F E B C

The Number of Tests to test all control

statements equals the Cyclomatic Complexity

Does The Cyclomatic Complexity equals number of Conditions in a Program? – [You Do!]

Testing Each Path is certainly useful if used with care, but it does not imply adequacy of testing.

Although all paths are executed, all combinations of paths are not executed

Path TestingCyclomatic Complexity

In SummaryIndependent Paths

1, 2, 3, 8, 9

1, 2, 3, 4, 6, 7, 2

1, 2, 3, 4, 5, 7, 2

1, 2, 3, 4, 6, 7, 2, 8, 9

Test cases should be derived so that all of these paths are executed

A Dynamic Program Analyser may be used to check that paths have been executed

1

2

3

4

65

7

while bottom <= top



9

bottom > top

Available In Some Case Tool Sets!

LoopTesting

Loop Testing

Nested Nested LoopsLoops

ConcatenatedConcatenated Loops Loops Unstructured Unstructured

LoopsLoops

Simple Simple looploop

Loop Testing: Simple Loops

MinimumMinimum conditions—Simpleconditions—Simple LoopsLoops

1.1. skipskip thethe looploop entirelyentirely2. only one pass through the loop2. only one pass through the loop3. two passes through the3. two passes through the looploop4. m passes through the loop m < n4. m passes through the loop m < n5. (n-1), n, and (n+1) passes through5. (n-1), n, and (n+1) passes through thethe looploop

wherewhere n is the maximum number n is the maximum number of allowableof allowable passespasses

Loop Testing: Nested Loops

Start at the innermost loop. Set all outer loops to their minimum iteration Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been typical values. Continue this step until the outermost loop has been tested.tested.

If the loops are independent of one anotherIf the loops are independent of one another thenthen treattreat eacheach asas a simple loopa simple loop else* treat as nested loopselse* treat as nested loops

endifendif**

forfor example, the final loop counter value of loop 1 is example, the final loop counter value of loop 1 is

used to initialize loop 2.used to initialize loop 2.

Nested LoopsNested Loops

Concatenated LoopsConcatenated Loops

Software EngineeringCSCI 3342

Dr. Thomas E. HicksComputer Science Department

Trinity University

Textbook: Software EngineeringBy Roger Pressman

Textbook: Software EngineeringBy Ian Sommerville

Special Thanks To WCB/McGraw-Hill & Addison Wesley For Providing Graphics Of Some Of Text Book Figures For Use In This

Presentation.

Date post:	31-Dec-2015
Category:	Documents
Upload:	mary-george
View:	231 times
Download:	0 times

Dr. Thomas Hicks Computer Science Department Trinity University 1 Software Engineering CSCI 3321.

Documents