7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 1/58
Software testing
Software Engineering 1
1
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 2/58
Objectives
Software Engineering 1
2
To discuss the distinctions between validationtesting and defect testing
To describe the principles of system and component
testing To describe strategies for generating system test
cases
To understand the essential characteristics of tool
used for test automation
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 3/58
Topics covered
Software Engineering 1
3
Introduction
Black box and white box testing
System testing
Component testing Test case design
Test automation
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 4/58
Why Software Testing?
• The rocket self-destructed 37 seconds after launch• Reason: A control software bug that went undetected
– Conversion from 64-bit floating point to 16-bit signed integer valuehad caused an exception
•The floating point number was larger than 32767 (max 16-bit signedinteger)
– Efficiency considerations had led to the disabling of the exceptionhandler.
– Program crashed rocket crashed
• Total Cost: over $1 billion
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 5/58
Building Quality Software?
Software Engineering 1
5
What Impacts the Software Quality? External Correctness Does it do what it suppose to do? Reliability Does it do it accurately all the time? Efficiency Does it do with minimum use of resources?
Integrity Is it secure? Internal Portability Can I use it under different conditions? Maintainability Can I fix it? Flexibility Can I change it or extend it or reuse it?
Quality Assurance The process of uncovering problems and improving the quality of
software. Testing is a major part of QA.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 6/58
The testing process
Software Engineering 1
6
Component testing
Testing of individual program components;
Usually the responsibility of the component developer (exceptsometimes for critical systems);
Tests are derived from the developer’s experience. System testing
Testing of groups of components integrated to create a system orsub-system;
The responsibility of an independent testing team;
Tests are based on a system specification.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 7/58
Testing phases
Software Engineering 1
7
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 8/58
Defect testing
Software Engineering 1
8
The goal of defect testing is to discover defects inprograms
A successful defect test is a test which causes a
program to behave in an anomalous way Tests show the presence not the absence of defects
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 9/58
Testing process goals
Software Engineering 1
9
Validation testing
To demonstrate to the developer and the system customer that thesoftware meets its requirements;
A successful test shows that the system operates as intended.
Defect testing To discover faults or defects in the software where its behaviour is
incorrect or not in conformance with its specification;
A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 10/58
The software testing process
Software Engineering 1
10
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 11/58
Testing policies
Software Engineering 1
11
Only exhaustive testing can show a program is free fromdefects. However, exhaustive testing is impossible,
Testing policies define the approach to be used in selectingsystem tests:
All functions accessed through menus should be tested;
Combinations of functions accessed through the same menu should be tested;
Where user input is required, all functions must be tested withcorrect and incorrect input.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 12/58
System testing
Software Engineering 1
12
Involves integrating components to create a systemor sub-system.
May involve testing an increment to be delivered tothe customer.
Two phases: Integration testing - the test team have access to the system
source code. The system is tested as components areintegrated.
Mainly concerned with finding defects. Release testing – test a version that could be delivered to users.
The test team test the complete system to be delivered.
It’s usually a “black - box” testing; to demonstrate that the systemdoes or does not work properly.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 13/58
Integration testing
Software Engineering 1
13
Involves building a system from its components andtesting it for problems that arise from componentinteractions.
Top-down integration Develop the skeleton of the system and populate it with
components.
Bottom-up integration Integrate infrastructure components then add functional
components. To simplify error localisation, systems should be
incrementally integrated.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 14/58
Incremental integration testing
Software Engineering 1
14
T3
T2
T1
T4
T5
A
B
C
D
T2
T1
T3
T4
A
B
C
T1
T2
T3
A
B
Test sequence 1 Test sequence 2 Test sequence 3
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 15/58
Testing approaches
Software Engineering 1
15
Architectural validation Top-down integration testing is better at discovering errors in the
system architecture.
System demonstration
Top-down integration testing allows a limited demonstration at anearly stage in the development.
Test implementation Often easier with bottom-up integration testing.
Test observation
Problems with both approaches. Extra code may be required toobserve tests.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 16/58
Release testing
Software Engineering 1
16
The process of testing a release of a system that will be distributed to customers.
Primary goal is to increase the supplier’s confidencethat the system meets its requirements.
Release testing is usually black-box or functionaltesting The system treated as “black - box” whose behaviour can be
seen by the outputs generated from the input supplied.
Tests are derived/based on the system specification only; Testers do not have knowledge of the system implementation.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 17/58
Black-box testing
Software Engineering 1
17
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 18/58
Testing guidelines
Software Engineering 1
18
Testing guidelines are hints for the testing team tohelp them choose tests that will reveal defects in thesystem Choose inputs that force the system to generate all error
messages; Design inputs that cause buffers to overflow;
Repeat the same input or input series several times;
Force invalid outputs to be generated;
Force computation results to be too large or too small.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 19/58
Testing scenario
Software Engineering 1
19
A student in Scotland is studying A merican History and has been asked to write a paper on ÔFrontier mentality in the American West from 1840 to 1880Õ.To do this, she needs to
find sources from a range o f libraries. She logs on to the LIBSYS system and uses thesearch facility to discover if she can acce ss original documents from that time. She
discovers sources in various US university libraries and down loads copies of some of these. However, for one document, she needs to have confirmation from her university
that she is a genu ine student and that use is for non-commercial purposes. The studentthen uses the facility in LIBSYS that can request such permission and registers her
request. If granted, the document will be downloaded to the registered libraryÕs server
and printed for her. She receives a message f rom LIBSYS telling her that she will receivean e-mail message when th e printed document is available for collection.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 20/58
System tests
Software Engineering 1
20
1. Test the login mechanism using correct and incorrect logins to check
that valid users are accepted and invalid users are rejected.
2. Test the search facility using diff erent queries against known sources to
check that the search mechanism is actually finding documents.
3. Test the system presentation facility to check that information about
documents is displayed properly.
4. Test the mechanism to request permission for downloading.
5.
Test the e-mail response indicating that the downloaded document isavailable.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 21/58
Use cases
Software Engineering 1
21
Use cases can be a basis for deriving the tests for asystem. They help identify operations to be testedand help design the required test cases.
From an associated sequence diagram, the inputsand outputs to be created for the tests can beidentified.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 22/58
Collect weather data sequence chart
Software Engineering 1
22
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 23/58
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 24/58
Stress testing
Software Engineering 1
24
A variant of performance testing
Exercises the system beyond its maximum design load.Stressing the system often causes defects to come to light.
Stressing the system test failure behaviour.. Systems should
not fail catastrophically. Stress testing checks forunacceptable loss of service or data.
Stress testing is particularly relevant to distributed systemsthat can exhibit severe degradation as a
network becomes overloaded.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 25/58
Component testing/Unit testing
Software Engineering 1
25
Component or unit testing is the process of testingindividual components/program units in isolation.
Does each module do what it is supposed to do?
It is a defect testing process. Components may be:
Individual functions or methods within an object;
Object classes with several attributes and methods;
Composite components with defined interfaces used to accesstheir functionality.
A unit test is at the level of a method/procedure/ or aclass …
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 26/58
Object class testing
Software Engineering 1
26
Complete test coverage of a class involves Testing all operations associated with an object;
Setting and interrogating all object attributes;
Exercising the object in all possible states.
Inheritance makes it more difficult to design objectclass tests as the information to be tested is notlocalised.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 27/58
Weather station object interface
Software Engineering 1
27
identifier repor tWeather ()calibrate (instruments)test ()
star tup (instruments)shutdown (instruments)
WeatherStation
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 28/58
Weather station testing
Software Engineering 1
28
Need to define test cases for reportWeather,calibrate, test, startup and shutdown.
Using a state model, identify sequences of state
transitions to be tested and the event sequences tocause these transitions
For example: Waiting -> Calibrating -> Testing -> Transmitting -> Waiting
Shutdown -> Waiting-> Shutdown
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 29/58
Interface testing
Software Engineering 1
29
Objectives are to detect faults due to interface errorsor invalid assumptions about interfaces.
Particularly important for object-oriented
development as objects are defined by theirinterfaces.
Errors in the composite component may arise because of interactions between its parts.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 30/58
Interface testing
Software Engineering 1
30
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 31/58
Interface types
Software Engineering 1
31
Parameter interfaces Data passed from one procedure to another.
Shared memory interfaces
Block of memory is shared between procedures or functions.
Procedural interfaces Sub-system encapsulates a set of procedures to be called by other
sub-systems.
Message passing interfaces
Sub-systems request services from other sub-system.s
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 32/58
Interface errors
Software Engineering 1
32
Interface misuse A calling component calls another component and makes an error in
its use of its interface e.g. parameters in the wrong order.
Interface misunderstanding
A calling component embeds assumptions about the behaviour of thecalled component which are incorrect.
Timing errors
The called and the calling component operate at different speeds andout-of-date information is accessed.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 33/58
Interface testing guidelines
Software Engineering 1
33
Design tests so that parameters to a called procedure are atthe extreme ends of their ranges.
Always test pointer parameters with null pointers.
Design tests which cause the component to fail.
Use stress testing in message passing systems.
In shared memory systems, vary the order in whichcomponents are activated.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 34/58
Test case design
Software Engineering 1
34
A test case is the specification of the inputs to thetest and the expected output from the system plus astatement of what is being tested.
Involves designing the test cases (inputs and
outputs) used to test the system. The goal of test case design is to create a set of tests
that are effective in validation and defect testing.
Design approaches: Requirements-based testing;
Partition testing;
Structural testing.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 35/58
Requirements based testing
Software Engineering 1
35
A general principle of requirements engineering isthat requirements should be testable.
Requirements-based testing is a validation testing
technique where you consider each requirement andderive a set of tests for that requirement.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 36/58
LIBSYS requirements
Software Engineering 1
36
The user shall be able to search either all of the initial set of databases or select a
subset from it.
The system shall provide appropriate viewers for the user to read documents in thedocument store.
Every order shall be allocated a unique identifier (ORDER_ID) that the user shall
be able to copy to the account’s permanent storage area.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 37/58
LIBSYS tests
Software Engineering 1
37
Initiate user search for searches for items that are known to be present and known not to be present, where the set of databases includes 1 database.
Initiate user searches for items that are known to be presentand known not to be present, where the set of databases
includes 2 databases Initiate user searches for items that are known to be present
and known not to be present where the set of databases
includes more than 2 databases.
Select one database from the set of databases and initiate
user searches for items that are known to be present and
known not to be present. Select more than one database from the set of databases
and initiate searches for items that are known to be presentand known not to be present.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 38/58
Partition testing
Software Engineering 1
38
Input data and output results often fall into differentclasses where all members of a class are related.
Each of these classes is an equivalence partition or
domain where the program behaves in an equivalent way for each class member.
Test cases should be chosen from each partition.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 39/58
Equivalence partitioning
Software Engineering 1
39
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 40/58
Equivalence partitions
Software Engineering 1
40
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 41/58
Search routine specification
Software Engineering 1
41
procedure Search (Key : ELEM ; T: SEQ of ELEM;Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition -- the sequence has at least one elementT’FIRST <= T’LAST
Post-condition -- the element is found and is referenced by L( Found and T (L) = Key)
or
-- the element is not in the array( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 42/58
Search routine - input partitions
Software Engineering 1
42
Inputs which conform to the pre-conditions.
Inputs where a pre-condition does not hold.
Inputs where the key element is a member of
the array. Inputs where the key element is not a member of the
array.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 43/58
Testing guidelines (sequences)
Software Engineering 1
43
Test software with sequences which have only asingle value.
Use sequences of different sizes in different tests.
Derive tests so that the first, middle and lastelements of the sequence are accessed.
Test with sequences of zero length.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 44/58
Search routine - input partitions
Software Engineering 1
44
Sequence Element
Single value In sequence
Single value Not in sequence
More than 1 value First element in sequence
More than 1 value Last element in sequence
More than 1 value Middle element in sequence
More than 1 value Not in sequence
Input sequence (T) Key (Key) Output (Found, L)
17 17 true, 1
17 0 false, ??
17, 29, 21, 23 17 true, 1
41, 18, 9, 31, 30, 16, 45 45 true, 7
17, 18, 21, 23, 29, 41, 38 23 true, 4
21, 23, 29, 33, 38 25 false, ??
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 45/58
Structural testing
Software Engineering 1
45
Sometime called “white- box” / “glass- box” testing.
Essentially, we want to execute each statement in theprogram at least once.
Derivation of test cases according to programstructure. Knowledge of the program is used toidentify additional test cases.
Objective is to exercise all program statements (not
all path combinations).
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 46/58
Structural testing
Software Engineering 1
46
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 47/58
Binary search - equiv. partitions
Software Engineering 1
47
Pre-conditions satisfied, key element in array.
Pre-conditions satisfied, key element not inarray.
Pre-conditions unsatisfied, key element in array. Pre-conditions unsatisfied, key element not in array.
Input array has a single value.
Input array has an even number of values.
Input array has an odd number of values.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 48/58
Binary search equiv. partitions
Software Engineering 1
48
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 49/58
Binary search - test cases
Software Engineering 1
49
Input array (T) Key (Key) Output (Found, L)
17 17 true, 1
17 0 false, ??
17, 21, 23, 29 17 true, 19, 16, 18, 30, 31, 41, 45 45 true, 7
17, 18, 21, 23, 29, 38, 41 23 true, 4
17, 18, 21, 23, 29, 33, 38 21 true, 3
12, 18, 21, 23, 32 23 true, 4
21, 23, 29, 33, 38 25 false, ??
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 50/58
Path testing
Software Engineering 1
50
It’s a structural testing strategy The objective of path testing is to ensure that the set
of test cases is such that each path through theprogram is executed at least once.
The starting point for path testing is a program flow graph that shows nodes representing programdecisions and arcs representing the flow of control.
Statements with conditions are therefore nodes in
the flow graph.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 51/58
Binary search flow graph
Software Engineering 1
51
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 52/58
Independent paths
Software Engineering 1
52
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
1, 2, 3, 4, 5, 14
1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …
Test cases should be derived so that all of these pathsare executed
A dynamic program analyser may be used to check that paths have been executed
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 53/58
Test automation
Software Engineering 1
53
Testing is an expensive process phase. Testing workbenchesprovide a range of tools to reduce the time required andtotal testing costs.
Systems such as Junit support the automatic execution of
tests. Most testing workbenches are open systems because testing
needs are organisation-specific.
They are sometimes difficult to integrate with closed design
and analysis workbenches.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 54/58
A testing workbench
Software Engineering 1
54
Dynamic
analyser
Pr og r am
being testedTest r esults
Test
pr edictions
File
comparator
Execution
r epor t Simulator
Sour ce
code
Test
manager Test data Or acle
Test data
gener ator Specification
Repor t
gener ator
Test r esults
r epor t
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 55/58
Testing workbench adaptation
Software Engineering 1
55
Scripts may be developed for user interfacesimulators and patterns for test data generators.
Test outputs may have to be prepared manually forcomparison.
Special-purpose file comparators may be developed.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 56/58
Key points
Software Engineering 1
56
Testing can show the presence of faults in a system; itcannot prove there are no remaining faults.
Component developers are responsible for componenttesting; system testing is the responsibility of a separate
team. Integration testing is testing increments of the system;
release testing involves testing a system to be released to acustomer.
Use experience and guidelines to design test cases in defecttesting.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 57/58
Key points
Software Engineering 1
57
Interface testing is designed to discover defects in theinterfaces of composite components.
Equivalence partitioning is a way of discovering testcases - all cases in a partition should behave in the
same way. Structural analysis relies on analysing a program and
deriving tests from this analysis.
Test automation reduces testing costs by supporting
the test process with a range of software tools.
7/30/2019 6 - Software Testing
http://slidepdf.com/reader/full/6-software-testing 58/58
END
QUESTIONS and CONCERNS.
Revision for Main Exam
58