Post on 18-Nov-2014
description
transcript
1
COSC 4406COSC 4406
Software Software EngineeringEngineering
Haibin Zhu, Ph.D.Haibin Zhu, Ph.D.Dept. of Computer Science and mathematics, Nipissing University, Dept. of Computer Science and mathematics, Nipissing University,
100 College Dr., North Bay, ON P1B 8L7, Canada, 100 College Dr., North Bay, ON P1B 8L7, Canada, haibinz@nipissingu.ca, http://www.nipissingu.ca/faculty/haibinz, http://www.nipissingu.ca/faculty/haibinz
2
Lecture 10Lecture 10 Software Testing Strategies Software Testing Strategies
3
Software Software TestingTesting
Testing is the process of exercising aTesting is the process of exercising aprogram with the specific intent of findingprogram with the specific intent of findingerrors prior to delivery to the end user.errors prior to delivery to the end user.
Verification: refers to the set of activities that ensure that software correctly implements a specific function.
Validation: refers to a different set of activities that ensure that software that has been built is traceable to customer requirements.
4
What Testing ShowsWhat Testing Shows
errorserrors
requirements conformancerequirements conformance
performanceperformance
an indicationan indicationof qualityof quality
5
Who Tests the Who Tests the Software?Software?
developerdeveloper independent testerindependent tester
Understands the system Understands the system
but, will test "gently"but, will test "gently"
and, is driven by "delivery"and, is driven by "delivery"
Must learn about the system,Must learn about the system,but, will attempt to break itbut, will attempt to break itand, is driven by qualityand, is driven by quality
6
Testing StrategyTesting Strategy
unit testunit test integrationintegrationtesttest
validationvalidationtesttest
systemsystemtesttest
7
Testing and SLDCTesting and SLDC
8
Testing StrategyTesting Strategy
We begin by ‘We begin by ‘testing-in-the-small’testing-in-the-small’ and move and move toward ‘toward ‘testing-in-the-large’testing-in-the-large’
For conventional softwareFor conventional software The module (component) is our initial focusThe module (component) is our initial focus Integration of modules followsIntegration of modules follows
For OO softwareFor OO software our focus when “testing in the small” changes from an our focus when “testing in the small” changes from an
individual module (the conventional view) to an OO individual module (the conventional view) to an OO class that encompasses attributes and operations and class that encompasses attributes and operations and implies communication and collaborationimplies communication and collaboration
9
Strategic IssuesStrategic Issues
Specify product requirements in a quantifiable manner long Specify product requirements in a quantifiable manner long before testing commences.before testing commences.
State testing objectives explicitly. State testing objectives explicitly. Understand the users of the software and develop a profile Understand the users of the software and develop a profile
for each user category.for each user category. Develop a testing plan that emphasizes “rapid cycle testing.”Develop a testing plan that emphasizes “rapid cycle testing.” Build “robust” software that is designed to test itselfBuild “robust” software that is designed to test itself Use effective formal technical reviews as a filter prior to Use effective formal technical reviews as a filter prior to
testingtesting Conduct formal technical reviews to assess the test strategy Conduct formal technical reviews to assess the test strategy
and test cases themselves. and test cases themselves. Develop a continuous improvement approach for the testing Develop a continuous improvement approach for the testing
process. process.
10
Test StepsTest Steps
11
Unit TestingUnit Testing
modulemoduleto beto betestedtested
test casestest cases
resultsresults
softwaresoftwareengineerengineer
12
Unit TestingUnit Testing
interface interface local data structureslocal data structures
boundary conditionsboundary conditionsindependent pathsindependent pathserror handling pathserror handling paths
modulemoduleto beto betestedtested
test casestest cases
13
Selective TestingSelective Testing Common errors:Common errors:
Misunderstood or incorrect arithmetic precedenceMisunderstood or incorrect arithmetic precedence Mixed mode operationsMixed mode operations Incorrect initializationsIncorrect initializations Precision inaccuracyPrecision inaccuracy Incorrect symbolic representation of an expressionIncorrect symbolic representation of an expression
The errors that should be uncoveredThe errors that should be uncovered Comparison of different data typesComparison of different data types Incorrect operators or precedenceIncorrect operators or precedence Expectation of equality when precision error makes equality Expectation of equality when precision error makes equality
unlikelyunlikely Incorrect comparison of variablesIncorrect comparison of variables Proper or nonexistent loop terminationProper or nonexistent loop termination Failure to exit when divergent iteration is encounteredFailure to exit when divergent iteration is encountered Improperly modified loop variables Improperly modified loop variables
14
Unit Test Unit Test EnvironmentEnvironment
ModuleModule
stubstub stubstub
driverdriver
RESULTSRESULTS
interface interface
local data structureslocal data structures
boundary conditionsboundary conditions
independent pathsindependent paths
error handling pathserror handling paths
test casestest cases
15
Integration Testing StrategiesIntegration Testing StrategiesIntegration testingIntegration testing is a systematic technique for is a systematic technique for constructing the software architecture while at the same constructing the software architecture while at the same time conducting tests to uncover errors associated with time conducting tests to uncover errors associated with interfacing.interfacing.
Options:Options:•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy
16
Top Down IntegrationTop Down Integration
top module is tested with top module is tested with stubsstubs
stubs are replaced one at stubs are replaced one at a time, "depth first"a time, "depth first"
as new modules are integrated, as new modules are integrated, some subset of tests is re-runsome subset of tests is re-run
AA
BB
CC
DD EE
FF GG
17
Bottom-Up IntegrationBottom-Up Integration
drivers are replaced one at a drivers are replaced one at a time, "depth first"time, "depth first"
worker modules are grouped into worker modules are grouped into builds and integratedbuilds and integrated
AA
BB
CC
DD EE
FF GG
clustercluster
18
Sandwich TestingSandwich Testing
Top modules areTop modules aretested with stubstested with stubs
Worker modules are grouped into Worker modules are grouped into builds and integratedbuilds and integrated
AA
BB
CC
DD EE
FF GG
clustercluster
19
Regression TestingRegression Testing
It is the re-execution of some subset of tests that It is the re-execution of some subset of tests that have already been conducted to ensure that have already been conducted to ensure that changes have not propagated unintended side changes have not propagated unintended side effects.effects.
It is the activity that helps to ensure that It is the activity that helps to ensure that changes do not introduce unintended behaviors changes do not introduce unintended behaviors or additional errors.or additional errors.
20
Smoke TestingSmoke Testing A common approach for creating “daily builds” A common approach for creating “daily builds”
for product softwarefor product software Smoke testing steps:Smoke testing steps:
Software components that have been translated into Software components that have been translated into code are integrated into a “build.” code are integrated into a “build.”
A build includes all data files, libraries, reusable A build includes all data files, libraries, reusable modules, and engineered components that are required modules, and engineered components that are required to implement one or more product functions.to implement one or more product functions.
A series of tests is designed to expose errors that will A series of tests is designed to expose errors that will keep the build from properly performing its function. keep the build from properly performing its function.
The intent should be to uncover “show stopper” errors The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software that have the highest likelihood of throwing the software project behind schedule.project behind schedule.
The build is integrated with other builds and the entire The build is integrated with other builds and the entire product (in its current form) is smoke tested daily. product (in its current form) is smoke tested daily.
The integration approach may be top down or bottom up.The integration approach may be top down or bottom up.
21
Object-Oriented Object-Oriented TestingTesting
begins by evaluating the correctness and begins by evaluating the correctness and consistency of the OOA and OOD modelsconsistency of the OOA and OOD models
testing strategy changestesting strategy changes the concept of the ‘unit’ broadens due to encapsulationthe concept of the ‘unit’ broadens due to encapsulation integration focuses on classes and their execution integration focuses on classes and their execution
across a ‘thread’ or in the context of a usage scenarioacross a ‘thread’ or in the context of a usage scenario validation uses conventional black box methodsvalidation uses conventional black box methods
test case design draws on conventional methods, test case design draws on conventional methods, but also encompasses special featuresbut also encompasses special features
22
Broadening the View of Broadening the View of “Testing”“Testing”
It can be argued that the review of OO analysis and It can be argued that the review of OO analysis and design models is especially useful because the design models is especially useful because the same semantic constructs (e.g., classes, attributes, same semantic constructs (e.g., classes, attributes, operations, messages) appear at the analysis, operations, messages) appear at the analysis, design, and code level. Therefore, a problem in the design, and code level. Therefore, a problem in the definition of class attributes that is uncovered definition of class attributes that is uncovered during analysis will circumvent side effects that during analysis will circumvent side effects that might occur if the problem were not discovered might occur if the problem were not discovered until design or code (or even the next iteration of until design or code (or even the next iteration of analysis). analysis).
23
OOT OOT StrategyStrategy
class testing is the equivalent of unit testingclass testing is the equivalent of unit testing operations within the class are testedoperations within the class are tested the state behavior of the class is examinedthe state behavior of the class is examined
integration applied three different strategiesintegration applied three different strategies thread-based testing—integrates the set of classes thread-based testing—integrates the set of classes
required to respond to one input or eventrequired to respond to one input or event use-based testing—integrates the set of classes use-based testing—integrates the set of classes
required to respond to one use caserequired to respond to one use case cluster testing—integrates the set of classes cluster testing—integrates the set of classes
required to demonstrate one collaborationrequired to demonstrate one collaboration
24
Validation testingValidation testing Focus is on software requirementsFocus is on software requirements Test if the product is the right oneTest if the product is the right one
The function or performance characteristics conforms to The function or performance characteristics conforms to specification.specification.
A deviation from specification is uncovered and a A deviation from specification is uncovered and a deficiency list is created.deficiency list is created.
Configuration reviewConfiguration review To ensure that all elements of the software configuration To ensure that all elements of the software configuration
have been properly developed have been properly developed Alpha/Beta testingAlpha/Beta testing
Focus is on customer usageFocus is on customer usage Alpha test is conducted at the developer’s site.Alpha test is conducted at the developer’s site. Beta test is conducted at the end-user’s site.Beta test is conducted at the end-user’s site.
25
System testingSystem testing System testing System testing focuses on system integrationfocuses on system integration
It is a series of different test whose primary purpose is to fully It is a series of different test whose primary purpose is to fully exercise the computer-based system. exercise the computer-based system.
Recovery testingRecovery testing forces the software to fail in a variety of ways and verifies that forces the software to fail in a variety of ways and verifies that
recovery is properly performed, MTTR is evaluated.recovery is properly performed, MTTR is evaluated. Security testingSecurity testing
verifies that protection mechanisms built into a system will, in verifies that protection mechanisms built into a system will, in fact, protect it from improper penetrationfact, protect it from improper penetration
Stress testingStress testing executes a system in a manner that demands resources in executes a system in a manner that demands resources in
abnormal quantity, frequency, or volumeabnormal quantity, frequency, or volume Performance TestingPerformance Testing
test the run-time performance of software within the context of test the run-time performance of software within the context of an integrated systeman integrated system
26
Debugging: Debugging: A Diagnostic ProcessA Diagnostic Process
27
Debugging processesDebugging processes
28
Debugging EffortDebugging Effort
time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause
time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests
29
Symptoms & CausesSymptoms & Causes
symptomsymptomcausecause
symptom and cause may be symptom and cause may be geographically separated geographically separated
symptom may disappear when symptom may disappear when another problem is fixedanother problem is fixed
cause may be due to a cause may be due to a combination of non-errors combination of non-errors
cause may be due to a system cause may be due to a system or compiler erroror compiler error
cause may be due to cause may be due to assumptions that everyone assumptions that everyone believesbelieves
symptom may be intermittentsymptom may be intermittent
30
Consequences of BugsConsequences of Bugs
damagedamage
mildmild annoyingannoying
disturbingdisturbingseriousserious
extremeextremecatastrophiccatastrophic
infectiousinfectious
Bug TypeBug Type
Bug Categories:Bug Categories: function-related bugs, function-related bugs, system-related bugs, data bugs, coding bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards design bugs, documentation bugs, standards violations, etc.violations, etc.
31
Debugging TechniquesDebugging Techniques
brute force / testingbrute force / testing
backtrackingbacktracking
inductioninduction
deductiondeduction
32
Debugging: Final Debugging: Final ThoughtsThoughts
Don't run off half-cocked, Don't run off half-cocked, thinkthink about the about the symptom you're seeing.symptom you're seeing.
Use toolsUse tools (e.g., dynamic debugger) to gain (e.g., dynamic debugger) to gain more insight.more insight.
If at an impasse, If at an impasse, get helpget help from someone else.from someone else.
Be absolutely sure to Be absolutely sure to conduct regression testsconduct regression tests when you do "fix" the bug.when you do "fix" the bug.
1.1.
2.2.
3.3.
4.4.
33
SummarySummary
Software TestSoftware Test To uncover errors that were made inadvertently.To uncover errors that were made inadvertently.
Strategic IssuesStrategic Issues Test Strategies for conventional softwareTest Strategies for conventional software Test Strategies for OO softwareTest Strategies for OO software Validation testingValidation testing System testingSystem testing Debugging Debugging