7/24/2019 CTFL Chapter 1 Fundamentals
1/80
MSTB-GTB 2011Version 1.0
Chapter
Software Testing FoundationsCertified Tester
Fundamentals of Testing
1
7/24/2019 CTFL Chapter 1 Fundamentals
2/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 2
After this lecture you should.
Be able to use examples to describe the way in which a software defect can cause harm topeople, to the environment, or to a company;
Know what is meant by the terms deficiency, defect, failure, defect condition / defect and errorand explain this with examples and compare;
Be able to distinguish between the cause of a defect and its effects;
Derive by examples why testing is necessary;
Be able to describe why testing is part of quality assurance and give examples of how testingcontributes to a higher quality;
Know the quality characteristic according to ISO 9126;
Recall the general objectives and principles of testing;
Differentiate between testing and debugging;
Know how the fundamental test process looks like;
Describe how expected values are calculated during testing; Know and characterize the psychological problems during the testing;
Differentiate between the different mindset of a tester and a developer;
Know the code of ethics of a software tester.
7/24/2019 CTFL Chapter 1 Fundamentals
3/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 3
Learning Objectives for Fundamentals of Testing(according to the Certified Tester Foundation Level Syllabus, Version 2011)
1.1 Why is Testing Necessary? (K2) LO-1.1.1 Describe, with examples, the way in which a defect in software can cause
harm to a person, to the environment or to a company (K2)
LO-1.1.2 Distinguish between the root cause of a defect and its effects (K2)
LO-1.1.3 Give reasons why testing is necessary by giving examples (K2)
LO-1.1.4 Describe why testing is part of quality assurance and give examples of howtesting contributes to higher quality (K2)
LO-1.1.5 Explain and compare the terms error, defect, fault, failure and thecorresponding terms mistake and bug, using examples (K2)
1.2 What i s Testing? (K2)
LO-1.2.1 Recall the common objectives of testing (K1) LO-1.2.2 Provide examples for the objectives of testing in different phases of the
software life cycle (K2)
LO-1.2.3 Differentiate testing from debugging (K2)
7/24/2019 CTFL Chapter 1 Fundamentals
4/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 4
Learning Objectives for Fundamentals of Testing(according to the Certified Tester Foundation Level Syllabus, Version 2011)
1.3 Seven Testing Principles (K2) LO-1.3.1 Explain the seven principles in testing (K2)
1.4 Fundamental Test Process (K1)
LO-1.4.1 Recall the five fundamental test activities and respective tasks from planningto closure (K1)
1.5 The Psychology of Testing (K2)
LO-1.5.1 Recall the psychological factors that influence the success of testing (K1)
LO-1.5.2 Contrast the mindset of a tester and of a developer (K2)
7/24/2019 CTFL Chapter 1 Fundamentals
5/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 5
Terms you should be familiar with
Incident
Error Guessing
Test Plan
M a s
t e r T e s
t P l a n
R i s k
T e s
t B a s
i s T e s
t C o n
d i t i o n
I n d e p e n
d e n c e
Fundamental Test Process
S o
f t w a r e
T e s
t
Error cause
Confirmation Test
Test Policy
Quality AssuranceDebugging
T e s
t D e s
i g n
T e s
t R u n
L o g
Regression Test
F a
i l u r e
ErrorTest Case
Defect
R e v i e w
RequirementError
T e s
t S u m m
a r y
R e p o r t
E x
i t C r i
t e r i a
T e s
t D a
t a
Testware
Test CoverageExhaustive Test
T e s t O
b j e c
t i v e
Q u a
l i t y
Testing Test Suite
Test Execution
7/24/2019 CTFL Chapter 1 Fundamentals
6/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 6
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing da
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
7/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 7
In General: What is a Defect or a Deficiency?
A situation can only be classified as defective if it is pre-defined, howthe expected, correct and therefore the non-defective situation shouldlook like.Defect
Is the inability to fulfill a specific requirement. It is a discrepancy
between the actual behaviour (during the execution of the tests oridentified operation) and the expected behaviour (in the testspecification or the stated requirements).Deficiency
Non-fulfilment of a requirement related to an intended or specifieduse. A deficiency is for example the impairment of usability, whilemeeting performance or failure to perform any reasonableexpectation.
Based on: DIN EN ISO 9000:2005Quality Management Systems Fundamentals and Terms (German)Beuth Verlag, Berlin, 2005
7/24/2019 CTFL Chapter 1 Fundamentals
8/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 8
Connection:Error - Fault/Defect - Failure
Error
ISTQB Glossary:
Error: Human action that produces an incorrect result [after IEEE 610].
People make mistakes; they commit errors.
Example: A programmer commitsan error by re-using a
a piece of software which isnot intended in the contextof the current project(the Ariane 5 satellitelaunching rocket)
7/24/2019 CTFL Chapter 1 Fundamentals
9/80Chapter 1 Software Testing Foundations Certified TesterPage 9 Copyright 2011 to MSTB/GTBV 1.0
Error Other Definitions
Other defintions:1. Human action (of the developer) that
produces an error condition in the software2. Human action (of the user) that produces
an unwanted result (failure) as aconsequence (misoperation)
3. Unknowingly, inadvertently or intentionallyperformed act, or omission which leads under givencircumstances (task, environment) to an impairment of therequired function of a product
7/24/2019 CTFL Chapter 1 Fundamentals
10/80Chapter 1 Software Testing Foundations Certified TesterPage 10 Copyright 2011 to MSTB/GTBV 1.0
ISTQB Glossary:
Defect: A flaw in a component or system that can cause the componentor system to fail to perform its required function, e.g. an incorrect statement ordata definition A defect, if encountered during execution, may cause afailure of the component or system.
Defect/Fault
in a program
Error
Connection:Error - Fault/Defect - Failure
An error made by a person(e.g. a developer) can resultin a defect in the sofware.(Defect and Fault aresynonymous terms)
Example: After theprogrammer hasre-used the piece ofcode in the wrongcontext, the softwarecontains a defect/fault.
(the full definition follows on page 12 )
7/24/2019 CTFL Chapter 1 Fundamentals
11/80Chapter 1 Software Testing Foundations Certified TesterPage 11 Copyright 2011 to MSTB/GTBV 1.0
Connection:Error - Fault/Defect - Failure
Failure
which appears
Defect/Fault
in a program
Error
ISTQB Glossary:Deviation of the component orsystem from its expected delivery,service or result [after Fenton].
A failure is the effect ofa defect when executinga program (test object)which appears to theoutside.
Example:The Ariane 5rocket crasheson its firstmission.Cost ~ $7 billion
Boom
7/24/2019 CTFL Chapter 1 Fundamentals
12/80
Chapter 1 Software Testing Foundations Certified TesterPage 12 Copyright 2011 to MSTB/GTBV 1.0
Fault/Defect Failure Additional comments
Defect (full definition from ISTQB glossary): A flaw in acomponent or system that can cause the component or system tofail to perform its required function, e.g. an incorrect statement ordata definition. A defect, if encountered during execution, maycause a failure of the component or system.
Defect masking : An occurrence in which one defect prevents thedetection of another [after IEEE 610]
A failure can also be called a malfunction or an external defect can also (but rarely) be caused by cosmic radiation, electro-
magnetic fields or hardware failure. Such failures can causeslow execution, incorrect output or the termination of execution.
Debugging is the development activity that finds, analyzes andremoves the cause of the failure.
7/24/2019 CTFL Chapter 1 Fundamentals
13/80
Chapter 1 Software Testing Foundations Certified TesterPage 13 Copyright 2011 to MSTB/GTBV 1.0
Testing Definition and Goals
The process consisting of all life cycle activities, both static anddynamic, concerned with planning, preparation and evaluation ofsoftware products and related work products to
determine that they satisfy specified requirements demonstrate that they are fit for purpose detect defects
Source: [ISTQB Glossary]
7/24/2019 CTFL Chapter 1 Fundamentals
14/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 14
Validation and Verification - Definitions
Validation
Confirmation by examination and through provision ofobjective evidence that the requirements for a specificintended use or application have been fulfilled .
[ISO 9000] and [ISTQB Glossary]Did we implement the right system ?
Verification
Confirmation by examination and through the provisionof objective evidence that specified requirements havebeen fulfilled [ISO 9000] and [ISTQB Glossary]Did we implement the system right ?
7/24/2019 CTFL Chapter 1 Fundamentals
15/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 15
What is Software Quality?
A feature or characteristic that affects an item's quality [IEEE 610]. A set of attributes of a software product by which its quality is
described and evaluated. A software quality characteristic may berefined into multiple levels of sub-characteristics [ISO 9126]*.
Quality attributes relate to requirements
Functional requirements (capabilities, interfaces,professionalism, ...) Non-functional requirements (quality and implementation
requirements, project-specific requirements, ...)*Remark:
The description of product quality attributes provided in ISO9126 isused as a guide to describing software quality attributes. Otherstandards, such as the ISO/IEC 25000 series, may also be of use.
7/24/2019 CTFL Chapter 1 Fundamentals
16/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 16
Software Quality according to ISO/IEC 9126 (1)
Software Quality
ISO/IEC 9126: Rate of software products, quality characteristicsand guidance on usage
Quality in Use External andInternal Quality
Fulfillment of taskswithin theexpenditure limits
for users (time,costs, ...)
Fulfillment of taskswithin the limits ofrisk (life and
health,business...)
Subjective usersatisfaction withinthe context of use
Productivity Security Satisfaction
Fulfillment of taskswithin the
accuracy andcompleteness
limits
Effectiveness
7/24/2019 CTFL Chapter 1 Fundamentals
17/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 17
Software Quality according to ISO/IEC 9126 (2)
Suitability
Accuracy
Interoperability
(Data) Security
Compliance
MaturityFault tolerance
Recoverability
Compliance
Understandability
Learnability
Operability
Attractiveness
Compliance
Time behavior /Performance
Resourceutilization
Compliance
Analyzability
Changeability
Stability
Testability
Compliance
Adaptivity
Installability
Co-Existence
Replaceability
Compliance
Software Quality
Functionality Reliability Usability Efficency Maintainability Portability
ISO/IEC 9126: Evaluation of software products, quality characteristics and guidance on usage
Quality in Use External andInternal Quality
7/24/2019 CTFL Chapter 1 Fundamentals
18/80
Chapter 1 Software Testing Foundations Certified TesterPage 18 Copyright 2011 to MSTB/GTBV 1.0
External Quality Attribute: Functionality (1)
Existence of functions with specified attributes. These functions meet the specified or implied requirements. sub-attributes of the quality attribute functionality are described
below and on the next page.
Suitability
The capability of the software product to provide an appropriate setof functions for specified tasks and user objectives [ISO 9126].
Accuracy
The capability of the software product to provide the right or agreed
results or effects with the needed degree of precision [ISO 9126].
(Further sub-attributes of the quality attribute functionality follow)
7/24/2019 CTFL Chapter 1 Fundamentals
19/80
Chapter 1 Software Testing Foundations Certified TesterPage 19 Copyright 2011 to MSTB/GTBV 1.0
External Quality Attribute: Functionality (2)
(Further sub-attributes of the quality attribute functionality)Interoperability
The capability of the software to interact with one or more specifiedcomponents or systems [after ISO 9126].
(Data) Security
Attributes of software that bear on its ability to preventunauthorised access, whether accidental or deliberate, to programsand data [after ISO 9126].
Compliance
Attributes of the software which demonstrate that the softwarefulfills application-specific standards or agreements or statutorydirectives or similar regulations. [after ISO 9126].
Note: this also applies to all other quality attributes (e.g. reliability)
7/24/2019 CTFL Chapter 1 Fundamentals
20/80
Chapter 1 Software Testing Foundations Certified TesterPage 20 Copyright 2011 to MSTB/GTBV 1.0
External Quality Attribute: Reliability
Attributes that relate to the ability of the software to maintain a specifiedlevel of performance under given conditions and for a specified time period.
Maturity
The capability of the software product to avoid failure as a result ofdefects in the software [ISO 9126].
Fault tolerance
The capability of the software product to maintain a specified level ofperformance in cases of software faults (defects) or of infringementof its specified interface [ISO 9126].
Recoverability The capability of the software product to re-establish a specified level
of performance and recover the data directly affected in case of failure[ISO 9126].
7/24/2019 CTFL Chapter 1 Fundamentals
21/80
Chapter 1 Software Testing Foundations Certified TesterPage 21 Copyright 2011 to MSTB/GTBV 1.0
External Quality Attribute Usability
Attributes that relate to the effort required to use, and on the individualassessment of such use by a specific or implied group of users.
Understandability
The capability of the software product to enable the user tounderstand whether the software is suitable, and how it can be usedfor particular tasks and conditions of use [ISO 9126].
Learnability
The capability of the software product to enable the user to learn itsapplication [ISO 9126].
Operability The capability of the software product to enable the user to operate
and control it [ISO 9126].
7/24/2019 CTFL Chapter 1 Fundamentals
22/80
Chapter 1 Software Testing Foundations Certified TesterPage 22 Copyright 2011 to MSTB/GTBV 1.0
External Quality Attribute: Efficiency
Attributes that relate to the relationship between the performancelevels of the software and the amount of equipment used underspecified conditions.
Time behaviour / Performance
The degree to which a system or component accomplishes itsdesignated functions within given constraints regarding processingtime and throughput rate [after IEEE 610].
Resource utilization
The capability of the software product to use appropriate amounts
and types of resources, for example the amounts of main andsecondary memory used by the program and the sizes of requiredtemporary or overflow files, when the software performs its functionunder stated conditions [after ISO 9126].
7/24/2019 CTFL Chapter 1 Fundamentals
23/80
Chapter 1 Software Testing Foundations Certified TesterPage 23 Copyright 2011 to MSTB/GTBV 1.0
Internal Quality Attribute: Maintainability (1)
Attributes that relate to the effort that is necessary to carry out softwarechanges.
Analyzability
The capability of the software product to be diagnosed for deficienciesor causes of failures in the software, or for the parts to be modified tobe identified [ISO 9126].
Changeability
The capability of the software product to enable specified modificationsto be implemented [ISO 9126].
(Further sub-attributes of the quality attribute maintainability follow)
7/24/2019 CTFL Chapter 1 Fundamentals
24/80
Chapter 1 Software Testing Foundations Certified TesterPage 24 Copyright 2011 to MSTB/GTBV 1.0
Internal Quality Attribute: Maintainability (2)
(Further sub-attributes of the quality attribute maintainability)
Stability
The capability of the software product to avoid unexpected effects
from modifications in the software [ISO 9126].Testability
The capability of the software product to enable modified software tobe tested [ISO 9126].
7/24/2019 CTFL Chapter 1 Fundamentals
25/80
Chapter 1 Software Testing Foundations Certified TesterPage 25 Copyright 2011 to MSTB/GTBV 1.0
Internal Quality Attribute Portability (1)
Attributes that relate to the ability of software to be transferred fromone environment to another.
Adaptability
The capability of the software product to be adapted for differentspecified environments without applying actions or means otherthan those provided for this purpose for the software considered[ISO 9126].
Installability
The capability of the software product to be installed in a specified
environment [ISO 9126].
(Further sub-attributes of the quality attribute portability follow)
7/24/2019 CTFL Chapter 1 Fundamentals
26/80
Chapter 1 Software Testing Foundations Certified TesterPage 26 Copyright 2011 to MSTB/GTBV 1.0
Internal Quality Attribute: Portability(2)
(Further sub-attributes of the quality attribute portability)
Co-Existence The capability of the software product to co-exist with other
independent software in a common environment sharing commonresources [ISO 9126].
Replaceability
The capability of the software product to be used in place of
another specified software product for the same purpose in thesame environment [ISO 9126].
7/24/2019 CTFL Chapter 1 Fundamentals
27/80
Chapter 1 Software Testing Foundations Certified TesterPage 27 Copyright 2011 to MSTB/GTBV 1.0
Quality Requirements
Quality requirements state what quality attributes the productshould have at what quality level.
The sum of all quality attributes and the specific variants
Not all quality attributes can be achieved equally well.
E.g. efficiency may come at the expense of maintainability Set priorities
In close consultation with clients and users. Quality requirements (with the exception of functionality) are
part of the non-functional requirements in the specifications.
7/24/2019 CTFL Chapter 1 Fundamentals
28/80
Chapter 1 Software Testing Foundations Certified TesterPage 28 Copyright 2011 to MSTB/GTBV 1.0
Testing and Quality
Testing measures the quality e.g. based on the number of locatedfailures and defects.
Testing increases the quality indirectly, as defects are detectedwhich can be corrected prior to delivery.
Testing increases the process quality indirectly, as defects aredocumented which can then be analysed and help prevent similarerrors from taking place in the future.
Testing increases the confidence in the quality of the system whenfew or no failures and defects are found.
7/24/2019 CTFL Chapter 1 Fundamentals
29/80
7/24/2019 CTFL Chapter 1 Fundamentals
30/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 30
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
31/80
Chapter 1 Software Testing Foundations Certified TesterPage 31 Copyright 2011 to MSTB/GTBV 1.0
Seven Principles of Testing (1)
In the previous 40 years, the following seven testing principles haveemerged and can serve as guidelines:
Principle 1: Testing shows presence of defects
Testing can show that defects are present, but cannot prove thatthere are no defects.Program testing can be used to show the presence of bugs,but never to show their absence! Edger W. Dijkstra, 1970
Principle 2: Exhaustive testing is impossible
Complete / exhaustive testing is not feasible except for trivialprograms (see slides in chapter 1 on exhaustive testing)
(continued)
7/24/2019 CTFL Chapter 1 Fundamentals
32/80
Chapter 1 Software Testing Foundations Certified TesterPage 32 Copyright 2011 to MSTB/GTBV 1.0
Seven Principles of Testing (2)
Principle 3: Early testingTesting is not a phase in the software development that happens atthe end; it shall be started as early as possible. Through early testing(e.g. reviews) which take place parallel to the development activities,defects are detected earlier and thus cost less to fix.
Principle 4: Defect clustering
Testing effort shall be focused proportionally to the expected and theobserved defect density of modules. A small number of modules oftencontains most of the defects discovered during pre-release testing, oris responsible for most of the operational failures.(continued)
7/24/2019 CTFL Chapter 1 Fundamentals
33/80
Chapter 1 Software Testing Foundations Certified TesterPage 33 Copyright 2011 to MSTB/GTBV 1.0
Seven Principles of Testing (3)
Principle 5: Repetitions have no effects - Pesticide ParadoxJust repeating tests brings no new insights. Test cases need to beregularly reviewed, revised and modified.
Principle 6: Testing is context dependent
Testing is depending on context. For example, safety-critical softwaresystems are tested differently (more intensely and using othertechniques) from commercial applications.
Principle 7: Absence-of-errors fallacy
A system without failures does not imply that the system will meet theusers expectations.
7/24/2019 CTFL Chapter 1 Fundamentals
34/80
Chapter 1 Software Testing Foundations Certified TesterPage 34 Copyright 2011 to MSTB/GTBV 1.0
Testing Effort?
Testing is economically useful, as long as the costs of finding andfixing a defect in testing are lower than the costs that areassociated with the occurrence of a defect when used.
A good test is like a liability insurance: it costs big money, butallows the project manager and the customer to sleep peacefully. Agood sleep includes a good insurance policy that covers allpossible risks. The trust in a software product includes a good testthat fully covers the reality of production.
Successful testing (detection of failures) reduces costs
M. Pol, T. Koomen, A. Spillner:Management and Optimization of the Test Process(German). dpunkt-Verlag, 2. Edition, 2002
Siedersleben, J. (Hrsg):Software methodology: Practical Knowledge forSoftware Engineers, (German). 2. Edition, Hanser,2003
7/24/2019 CTFL Chapter 1 Fundamentals
35/80
Chapter 1 Software Testing Foundations Certified TesterPage 35 Copyright 2011 to MSTB/GTBV 1.0
More regarding Testing (1)
Testing effort in practice: 25% to 50% of the development effort
Test intensity and scope depends on risks and the criticality of theproject, the system and the domain
Defects can result in huge costs: Estimated losses due to software defects in medium and large
companies in Germany are approximately 84.4 billion Euros perannum
Productivity losses due to computer failures caused by softwaredefects are approximately 2.6% of sales - 70 billion Euros perannum
Study by LOT Consulting, KarlsruheIT-Services 3/2001, P. 31 (German)
7/24/2019 CTFL Chapter 1 Fundamentals
36/80
Chapter 1 Software Testing Foundations Certified TesterPage 36 Copyright 2011 to MSTB/GTBV 1.0
More regarding Testing (2)
Testing is always subject to limited resources(e.g. time, test personal)
Particularly important:
Select appropriate test procedures which are compatible withthe test objectives and quality objectives
Avoid unnecessary tests which provide no new information
Take positive and negative test conditions into account
It can also be useful to test for functionality that has not beenrequested
7/24/2019 CTFL Chapter 1 Fundamentals
37/80
Chapter 1 Software Testing Foundations Certified TesterPage 37 Copyright 2011 to MSTB/GTBV 1.0
Prioritization of Tests
Criteria for priorit ization: Expected impact (the severity of the damage a defect would cause) Probability of occurrence of a failure
Combined consideration of impact and likelihood of occurrence( Risk = Likelihood * Impact) Perception of the failure Prioritization of requirements by the customer Importance of quality characteristics by the customer
Priority of test cases from the perspective of development (seriousconsequences and / or complex cases first) High project risk Where many defects are found, more defects are likely to be
discovered! A change in the priority must be possible!
Limited resources (time and personnel) require that the most importanttest cases are carried out first!
7/24/2019 CTFL Chapter 1 Fundamentals
38/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 38
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing
Fundamental Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
39/80
Chapter 1 Software Testing Foundations Certified TesterPage 39 Copyright 2011 to MSTB/GTBV 1.0
SW-Development Models: Waterfall-Model - 1970
Royce, W. W.:Managing the Development of Large SoftwareSystems: Concepts and TechniquesProc. WESCON, IEEE Computer Society Press, Los
Alamitos, CA, 1970.Reprinted at the ICSE'87, Monterey, California, USA.March 30 - April 2, 1987
7/24/2019 CTFL Chapter 1 Fundamentals
40/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 40
SW-Development Models: Waterfall-Model - 1981
Boehm, B.:Software Engineering EconomicsPrentice-Hall Inc., London, 1981
7/24/2019 CTFL Chapter 1 Fundamentals
41/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 41
V-Model (B. Boehm, 1979)
Barry W. Boehm:Guidelines for Verifying and Validating SoftwareRequirements and Design Specification.EURO IFIP 79, P.A. Samet (eds.) North-Holland, IFIP1979, 711-719
7/24/2019 CTFL Chapter 1 Fundamentals
42/80
Chapter 1 Software Testing Foundations Certified TesterPage 42 Copyright 2011 to MSTB/GTBV 1.0
General V-Model
FunctionalSystem Design
TechnicalSystem Design
Programming
Defini tion of Requirements
ComponentSpecification
System Testing
Integration Testing
Acceptance Testing
Component Testing
KeyTest cases based oncorresponding documents
7/24/2019 CTFL Chapter 1 Fundamentals
43/80
Chapter 1 Software Testing Foundations Certified TesterPage 43 Copyright 2011 to MSTB/GTBV 1.0
Test Process
Is closely linked with software development Is, however, a separate, independent process
A test plan is necessary for every test level (level test plan)
Testing cannot be considerd as a single activity (e.g. test execution)
Many individual tasks are performed within testing
The test process represents these individual testing activities in a coherentprocess
7/24/2019 CTFL Chapter 1 Fundamentals
44/80
Chapter 1 Software Testing Foundations Certified TesterPage 44 Copyright 2011 to MSTB/GTBV 1.0
Activities of the Fundamental Test Process
Test Planning and Control Test Analysis and Design
Test Implementation and Execution
Evaluating Exit Criteria and
Reporting Test Closure Activities
These activities may overlap or take
place concurrently. The Fundamental Test Process is to
be tailored to each test level (e.g.module test, system test)
Planning and
Analysis and Design
Implementation andExecution
Closure
Start
Finish
Control
Evaluation andReport
7/24/2019 CTFL Chapter 1 Fundamentals
45/80
Chapter 1 Software Testing Foundations Certified TesterPage 45 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Planning and Control (1)
Specification of Test Plan Determine test objectives, amount and
risks of testing. Specify test approach (techniques*,
test objects, coverage, teams whoparticipate in testing, testware).
Details of Test plan Test resources (e.g. staff, test environment, PCs). How intensive should particular parts of the system be tested? Which test techniques should be used? Determine exit criteria. What level coverage should be reached?
Prioritizing of test (taking defect risks into consideration). Planning for tool support (use, acquisition, training). Test schedule and test strategy are to be recorded in the test plan
Planning and
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Control
Evaluation andReport
*refer to chapters 4 and 5
7/24/2019 CTFL Chapter 1 Fundamentals
46/80
Chapter 1 Software Testing Foundations Certified TesterPage 46 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Planning and Control (2)
Drawing up a rough schedule Schedule tasks for test analysis and test
specification. Schedule test implementation, test
execution and test evaluation.
Test Control Measuring and analysing results. Monitoring and recording of test progress,
achieved test coverage and exit criteria. Initiate corrective actions.
Adapting time and resource planning. Making decisions.
Refer to chapter 6 for more details
Planning and
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Control
Evaluation andReport
7/24/2019 CTFL Chapter 1 Fundamentals
47/80
Chapter 1 Software Testing Foundations Certified TesterPage 47 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Analysis and Design (1)
General testing objectives are detailled to specifictest requirements and criteria.
Basis are all documents showing therequirements for the software (test basis).
Reviewing the test basis (requirements,architecture, design, interface, ...).
Evaluating testability of requirements and the system. Identifying test conditions / test requirements. Designing the test environment (infrastructure, tools, ...). Designing test cases using the chosen test design techniques. Distinction to be made between high level test cases and detailed
test cases. Creating bi-directional traceability between test basis and test
cases
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
S
7/24/2019 CTFL Chapter 1 Fundamentals
48/80
Chapter 1 Software Testing Foundations Certified TesterPage 48 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Analysis and Design (2)
Test cases contain more that just test data! Criteria and constraints that have to be met
before execution (e.g. condition of testobject, data, network status)
Before test execution determine which resultor behavior is expected and the post-conditions,that have to be fulfilled after the test, e.g. conditionof the test object and the database.
Refer to the next section in this chapter for further details Test cases are partly linked together to create test suites (test
procedure specification) for test execution Each test case should only contain one or a few elementary
process steps or system calls Pay attention in the test design to the post-conditions of the
test cases. These are the pre-conditions of next test cases.
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
St t
7/24/2019 CTFL Chapter 1 Fundamentals
49/80
Chapter 1 Software Testing Foundations Certified TesterPage 49 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Implementation
Specifying and prioritizing test cases. Creating test data and test scenarios.
Creating test suites.
Preparing test harnesses and (possibly) writingautomated test scripts.
Verifying that the test system and test environment has been setup correctly.
Before test execution: Verifying completeness of parts to be tested.
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
50/80
Chapter 1 Software Testing Foundations Certified TesterPage 50 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Execution (1)
First check main functions. If any incidentsare found here, continuing with deeper testsmay not make much sense!
Execute test cases either manually orby using test execution tools according to thetest procedure specification
Record the test execution accurately and completely (versions oftest object, test tools and testware ) in the test log.
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
51/80
Chapter 1 Software Testing Foundations Certified TesterPage 51 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Execution (2)
Compare actual results with expected results. Report discrepancies as incidents and analyze
them. If a discrepancy of actual and expectedresult is detected, it may not always meana software defect has been detected.
Check if any of the following aspects is true: incorrect expected result test environment not set up properly the test case specification is incorrect
Determine defect severity and priority for fixing Repeat test activities as a result of action taken to resolve each
incident/defect ( Re-Testing ).
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
52/80
Chapter 1 Software Testing Foundations Certified TesterPage 52 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Execution (3)
Create a test log The test log must contain details on which
parts were tested, when, by whom,how intensive and with what result.
On the one hand the description of testexecution in the test log has to becomprehensive for those not directly involved(e.g. clients)
On the other hand it has to be traceable if the planned test strategyhas really been applied
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
53/80
Chapter 1 Software Testing Foundations Certified TesterPage 53 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Evaluating Exit Criteria
Compare results of test execution with thedefined objectives of the test.
Checking test logs against the exit criteriaspecified in test planning.
Reached test end? Achieved coverage?
(Problem: unreachable program code) Deciding if more tests are needed or if the specified exit criteria
should be changed (only after consulting stakeholders!) Further effort justified?
Addition practical factors for ending the test: No more time No more money
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
54/80
Chapter 1 Software Testing Foundations Certified TesterPage 54 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Reporting (1) Analysis and
Design
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Write a test summary report forstakeholders.
The test summary report may have twopurposes:
Communicate test status
Report on test completion Refer to chapter 6 for more details on test reporting
Start
7/24/2019 CTFL Chapter 1 Fundamentals
55/80
Chapter 1 Software Testing Foundations Certified TesterPage 55 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Reporting (2)
Example for exit criteria:Found defects in order of failure severity
0
0,5
1
1,5
2
2,5
3
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
New / Testing Hour
low
medium
high
critical
Failure Severity
Wx: Calendar Week
Analysis andDesign
Implementationand Execution
Test Closure
Start
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
56/80
Chapter 1 Software Testing Foundations Certified TesterPage 56 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Reporting (3)
Example for exit criteria:Comparison of found failures and corrected defects
0
10
20
30
40
50
60
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
New failures
Corrected defects
In Process
Wx: Calendar Week
Analysis andDesign
Implementationand Execution
Test Closure
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
57/80
Chapter 1 Software Testing Foundations Certified TesterPage 57 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Closure Activities (1)
Collect data from completed test phases andconsolidate experience, testware, facts, metricsetc.
Check which planned deliverables have beendelivered
Close incident reports or raise change requestsfor any that remain open
Document the acceptance of the system
Analysis andDesign
Implementationand Execution
Test Closure
Finish
Evaluation andReport
Planning and
Control
Start
7/24/2019 CTFL Chapter 1 Fundamentals
58/80
Chapter 1 Software Testing Foundations Certified TesterPage 58 Copyright 2011 to MSTB/GTBV 1.0
Testing Tasks: Test Closure Activities (2)
Finalize and archive testware, thetest environment and the test infrastructure
Preserve and hand over the testware to themaintenance organization everything shouldbe reusable during maintenance, so it has to be
portable and updatable Analyse and document any lessons learned
for later projects and to improve test maturity Evaluation of test process Assessment of test process
Recognizing potential for improvement Improve the test process by applying the
findings
Analysis andDesign
Implementationand Execution
Test Closure
Finish
Evaluation andReport
Planning and
Control
7/24/2019 CTFL Chapter 1 Fundamentals
59/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 59
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing
Fundamental Test Process
Test Cases, Expected Results, Test Oracles ch\
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
60/80
Chapter 1 Software Testing Foundations Certified TesterPage 60 Copyright 2011 to MSTB/GTBV 1.0
Criteria for Test Cases
Test cases for verification of specified results and of test objectdelivered results and reactions Positive test; expected input / operation ( normal )).
Test cases that verify the specified handling of exceptionalsituations and defects also need to be considered Negative test; expected false input / operation ( exceptional ). Remark: It is often difficult to create the conditions necessary
for execution of negative test cases (e.g. the overload of anetwork connection).
Test cases for verification of reactions of the test object for invalidand unexpected inputs or constraints, for which there was noexception handling specified Negative test / robustness test; unexpected erroneous inputs /
operations ( catastrophic ).
Test Specification
7/24/2019 CTFL Chapter 1 Fundamentals
61/80
Chapter 1 Software Testing Foundations Certified TesterPage 61 Copyright 2011 to MSTB/GTBV 1.0
Test Specification High Level and Specific Test Cases (1)
Example: A company has ordered a program that should calculate theChristmas bonus of the staff in relation to the time they have beenworking there. In the description of the requirements you find thefollowing text:
Staff who have been with the company for more that three years willreceive 50 % of the monthly salary as Christmas bonus. Staff whohave been with the company for more than five years will receive 75%. Staff who have been with the company for more than eight years
will receive 100 % of their monthly salary.
How do the test cases look like?
Test Specification
7/24/2019 CTFL Chapter 1 Fundamentals
62/80
Chapter 1 Software Testing Foundations Certified TesterPage 62 Copyright 2011 to MSTB/GTBV 1.0
Test Specification High Level and Specific Test Cases (2)
From the text you can set up the following relationship between theallowance of the bonus and the time working for the company:
years with the company
7/24/2019 CTFL Chapter 1 Fundamentals
63/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 63
Test Specification High Level and Specific Test Cases (3)
High Level (logical)Test Case
1 2 3 4
Input value x(years with the company)
x
7/24/2019 CTFL Chapter 1 Fundamentals
64/80
Chapter 1 Software Testing Foundations Certified TesterPage 64 Copyright 2011 to MSTB/GTBV 1.0
Expected Results and Test Oracle
After each excecuted test case it must be decided whether there isa failure or not.
For this decision the monitored result (actual result/behavior) hasto be compared with the expected result (expectedresult/behavior).
The expected behavior of the test object has to be determined inadvance for each test case.
The tester must obtain this information from appropriate sourceswhen specifying a test case.
In this connection we also speak of an oracle or test oracle thatcan be consulted and that predicts the expected results.
7/24/2019 CTFL Chapter 1 Fundamentals
65/80
Chapter 1 Software Testing Foundations Certified TesterPage 65 Copyright 2011 to MSTB/GTBV 1.0
The expected results are to be deducted of the specification.
Following three possibilities: The tester derives the expected date from the input date on the basis
of the specification of the test object. This is the common practise. If a formal specification is available, an executable prototype can be
created with tool support. The results of the prototype can be usedas a test oracle for testing the actual program.
Parallel designs of the software program may be created by differentdevelopment groups. Each version of the program will then be testedagainst another version using the same test data. If there aredifferent results in the two versions there must be a failure in one of
the versions. Only those failures that have the same effect in all theversions remain undetected. This procedure is called Back-to-backtest.
Expected Results and Test Oracle
7/24/2019 CTFL Chapter 1 Fundamentals
66/80
Chapter 1 Software Testing Foundations Certified TesterPage 66 Copyright 2011 to MSTB/GTBV 1.0
Other Test Oracles
Making use of user manuals The system itself (create operational profile)
This is the worst possibility
(Tested!) previous versions
Programs with similar functionality Exact values cannot always be predicted or calculated. Determine
range of tolerance, verify plausibility
Experience is important
7/24/2019 CTFL Chapter 1 Fundamentals
67/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 67
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing
Fundamental Test Process
Test cases, Expected Results, Test Oracle
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
68/80
Chapter 1 Software Testing Foundations Certified TesterPage 68 Copyright 2011 to MSTB/GTBV 1.0
Psychology of Testing
Making mistakes is human- but who likes admitting that?
Development = constructiveTest = destructive?
Testing is an extremely creativeand intellectually challenging task
Is it possible for developer to test
his own program?
What to you think?
Myers, Glenford J.:Systematic Testing of ProgramsOldenbourg, 2001 (7. Edition)
7/24/2019 CTFL Chapter 1 Fundamentals
69/80
Chapter 1 Software Testing Foundations Certified TesterPage 69 Copyright 2011 to MSTB/GTBV 1.0
Psychology of Testing Developer Test
Being blind to see own mistakes If the developer has implemented a basic design error, e.g.
because he has misunderdstood the task, he is not able todetect this through his own tests.
He will not be able to specify appropriate test cases.
No familiarisation The developer knows his test object
7/24/2019 CTFL Chapter 1 Fundamentals
70/80
Chapter 1 Software Testing Foundations Certified TesterPage 70 Copyright 2011 to MSTB/GTBV 1.0
Psycholoy of Testing - Independent Test Team
An independent test team is unbiased It his not his/their product The possible assumptions and misunderstandings of the developer
are not necessarily the assumptions and misunderstandings of thetester.
Need for familiarization To create test cases the tester needs to gain knowledge about the test
object. This requires time.
Test Know-how The tester should have test know-how A developer probably does not have this or first has to gain it (often
with not enough time available)
Even better, the know-how has already been acquired at university
7/24/2019 CTFL Chapter 1 Fundamentals
71/80
Chapter 1 Software Testing Foundations Certified TesterPage 71 Copyright 2011 to MSTB/GTBV 1.0
Psychology of Testing Possible Levels
Tests are carried out with different levels of independence By the developer himself By colleagues of the developer in the same project By persons of other departments
By persons of other organizations. Tools can be used to support all possibilities.
The selection depends on the product as well as the project.
It is important to get the right mix and balance between
independent testing and tests performed by the developer.
7/24/2019 CTFL Chapter 1 Fundamentals
72/80
Chapter 1 Software Testing Foundations Certified TesterPage 72 Copyright 2011 to MSTB/GTBV 1.0
Psychology of Testing Communication of Defects
Communication of identified defects or incidents to the developerand/or the management requires
neutral, fact-focused and constructive way of communication Undisturbed, open communication.
Proving another persons error is not an easy task. It requires goodinterpersonal skills.
Reproduceability of defects is important! Recording the test environment Differences to the development environment
Clear requirements, precise specification Its not a bug, its a feature!
Mutual understanding between testers and developers Collaboration rather than battles!
7/24/2019 CTFL Chapter 1 Fundamentals
73/80
Chapter
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 73
1
Fundamentalsof Testing
Terms and Motivation
Principles of Testing
Fundamental Test Process
Test cases, Expected Results, Test Oracle
Psychology of Testing
Ethics of Testing }
7/24/2019 CTFL Chapter 1 Fundamentals
74/80
Chapter 1 Software Testing Foundations Certified TesterPage 74 Copyright 2011 to MSTB/GTBV 1.0
Ethics of Testing (1)
Software testers often have access to confidential and legally privilegedinformation. This must not be used inappropriately.
Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB states the following code of ethics :
1. PUBLIC
Certified software testers shall act consistently with the public interest.
2. CLIENT AND EMPLOYERCertified software testers shall act in a manner that is in the best interestsof their client and employer, consistent with the public interest.
7/24/2019 CTFL Chapter 1 Fundamentals
75/80
Chapter 1 Software Testing Foundations Certified TesterPage 75 Copyright 2011 to MSTB/GTBV 1.0
Ethics of Testing (2)
3. PRODUCTCertified software testers shall ensure that the deliverables they provide(on the products and systems they test) meet the highest professionalstandards possible.
4. JUDGMENTCertified software testers shall maintain integrity and independence intheir professional judgment.
5. MANAGEMENTCertified software test managers and leaders shall subscribe to andpromote an ethical approach to the management of software testing.
7/24/2019 CTFL Chapter 1 Fundamentals
76/80
Chapter 1 Software Testing Foundations Certified TesterPage 76 Copyright 2011 to MSTB/GTBV 1.0
Ethics of Testing (3)
6. PROFESSIONCertified software testers shall advance the integrity and reputation ofthe profession consistent with the public interest.
7. COLLEAGUESCertified software testers shall be fair to and supportive of theircolleagues, and promote cooperation with software developers.
8. SELFCertified software testers shall participate in lifelong learning regardingthe practise of their profession and shall promote an ethical approach tothe practice of the profession.
7/24/2019 CTFL Chapter 1 Fundamentals
77/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 77
Summary
Failure Not fulfilling the requirements! Describing this situation as failure cause: fault/defect in the software that has been caused by an error made by a person
Tests are important measures for securing the quality criteria
Functionality, reliability, usability, efficiency, maintainability undportability [according to ISO 9126]
Exhaustive testing is not possible. Tests are always randomchecks, therefore they can detect failures only with a certainprobability
The intensity and amount of testing depends on the expected riskof faulty behavior of the program
Tests should be prioritized
7/24/2019 CTFL Chapter 1 Fundamentals
78/80
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 78
Summary
The fundamental test process consists of the main activities
Test Planning and Control Test Analysis and Design
Test Implementation and Execution
Evaluating Exit Criteria and Reporting
Test Closure Activities
A test case consists of
an input value and
an expected result as well as
preconditions and constraints for the execution of the test case, and
postconditions that have to be fulfilled after the test case.
While executing the test case, the test object shows its actual behavior. If there is a discrepancybetween expected and actual behavior there is a failure.
A test oracle determines the expected values for each test case before test execution
Humans make mistakes but they do not like admitting them!
By now, you should be able to answer the following
7/24/2019 CTFL Chapter 1 Fundamentals
79/80
? ? ?
Copyright 2011 to MSTB/GTBV 1.0Chapter 1 Software Testing Foundations Certified TesterPage 79
questions
Define the terms software failure, defect, error.
What is defect masking?
Explain the difference between testing and debugging.
Define the terms verification and validation.
Explain why tests cannot be exhaustive. Name the main software quality attributes according to ISO 9126.
Define the term reliability of a system.
Explain the main activities of the fundamental test process.
What is a test oracle?
Why should developers not test their own programs?
7/24/2019 CTFL Chapter 1 Fundamentals
80/80
Finally
The process cannot be stopp ed. The process cannot be stopped.
Copy file:
Installation
Sort results
1 entry was found.Would you like to sort th e results?
Yes No
Cancel