INF5530/ 31.01.2013 / © Raluca Popescu 1
Testing throughout the software
life cycle
Lecturer: Raluca Popescu
INF5530 : Foundations of Software Testing
31. January 2013
Chapter 2
INF5530/ 31.01.2013 / © Raluca Popescu 2
Overview
1. Software development models
2. Test levels
3. Test types
4. Maintenance testing
INF5530/ 31.01.2013 / © Raluca Popescu 3
1. Software development models
• 1.1 Sequential dev. model (V-model)
• 1.2 Iterative-incremental dev. model
• 1.3 Testing within a life cycle model
INF5530/ 31.01.2013 / © Raluca Popescu 4
1.1 Sequential development model (V-model)
• Testing needs to begin as early as possible in the life cycle.
• Testing can be integrated into each phase of the product life cycle.
• Within the V-model, validation testing takes place especially during
• the early stages (i.e. reviewing the user requirements),
• and late in the life cycle (i.e. during user acceptance testing.
1. Software development models
INF5530/ 31.01.2013 / © Raluca Popescu 5
1.2 Iterative-incremental development
model
1. Software development models
• Iterative-incremental development is the process of :
• establishing requirements,
• designing,
• building
• and testing a system,
done as a series of shorter development cycles.
• An increment, added to others developed previously, forms a
growing partial system, which should also be tested.
• Regression testing is increasingly important on all iterations after the
first one.
INF5530/ 31.01.2013 / © Raluca Popescu 6
1.3 Testing within a life cycle model
• In any life cycle model, there are several characteristics of good
testing:
• Test levels can be combined or reorganized depending on the nature
of the project or the system architecture
1. Software development models
1 Every development activity has a corresponding testing activity.
2 Each test level has test objectives specific to that level.
3 The analysis & design of tests for a given test level should begin during the
corresponding development activity.
4 Testers should be involved in reviewing documents as soon as drafts are available in
the development life cycle.
INF5530/ 31.01.2013 / © Raluca Popescu 7
2. Test levels
• 2.1 Component testing
• 2.2 Integration testing
• 2.3 System testing
• 2.4 Acceptance testing
INF5530/ 31.01.2013 / © Raluca Popescu 8
• Integration
Interface between components;
interactions with other
systems (OS, HW, etc)• Unit
Any module, program,
object separately testable
• SystemThe behavior of the
whole product(system) as defined by the scope of the project
• AcceptanceIs the responsibility of the
customer – in general. The goal is to gain confidence in the system; especially in its non-functional characteristics
2. Test levels
INF5530/ 31.01.2013 / © Raluca Popescu 9
• For each test level, please note:
– The generic objectives
– The test basis (docs/products used to derive test cases)
– The test objects (what is being tested)
– Typical defects and failures to be found
– Specific approaches and responsibilities
2. Test levels
INF5530/ 31.01.2013 / © Raluca Popescu 10
• Objectives: Component testing includes testing of functionality and specific
non-functional characteristics, such as:
– resource-behavior (e.g. memory leaks)
– robustness testing
– structural testing (e.g. branch coverage).
• Test basis:
– specification of the component
– software design
– the data model
• Tools: Stubs, drivers and simulators may be used.
• Approaches:
– test-driven development (prepare and automate test cases before coding)
– Defects are fixed as soon as they are found, without formal recording of incidents
• Responsible: (QA or programmer in theory) Programmer in practice
2. Test levels
2.1 Component testing
INF5530/ 31.01.2013 / © Raluca Popescu 11
• Objectives: It tests:
– interfaces between components,
– interactions with different parts of a system, such as :
• the operating system,
• file system,
• hardware,
• interfaces between systems.
Types of integration tests:
1. Component integration : tests the interactions between software components – is done after component testing;
2. System integration: tests the interactions between different systems – is done after system testing.
• Approaches: to reduce the risk of late defect discovery, integration
should normally be incremental rather than “big bang”.
• Both functional and structural approaches may be used.
• Responsible: Ideally, testers should understand the architecture and
influence integration planning.
2. Test levels
2.2 Integration testing
INF5530/ 31.01.2013 / © Raluca Popescu 12
2. Test levels
2.3 System testing
• Objectives: testing the behavior of the whole system as defined by the scope
of the project.
• Test basis:
– System requirements specification
– Business processes
– Use cases
– Other high level description of the: system behavior, interactions with OS/system
resources
Requirements may exist as text and/or models.
Testers also need to deal with incomplete or undocumented requirements.
• Approaches:
• - test environment should correspond to the production environment as much
as possible.
– First, the most suited black-box technique
– Then, white-box technique to assess the thoroughness of testing
• Responsible: independent test team
INF5530/ 31.01.2013 / © Raluca Popescu 13
2. Test levels
2.4 Acceptance testing• Objectives: to establish confidence in the system / part of system / non-
functional characteristics of the system
• Types of acceptance testing:
• User acceptance testing - verifies the fitness for use of the system by users.
• Operational testing (done by the system administrators, and include):
– testing of backup/restore;
– disaster recovery;
– user management;
– maintenance tasks;
– periodic checks of security vulnerabilities.
• Contract and regulation acceptance testing - performed against a contract’s acceptance
criteria (i.e. governmental, legal or safety regulations)
• Alpha and beta testing
- Alpha testing is performed at the developing organization’s site.
- Beta testing (field testing), is performed by people at their own locations.
Both are performed by potential customers, not the developers of the product.
• Responsible: customers or users of a system; other stakeholders may be
involved as well
INF5530/ 31.01.2013 / © Raluca Popescu 14
3. Test types
• 3.1 Functional testing
• 3.2 Non functional testing
• 3.3 Structural testing
• 3.4 Testing related to changes
INF5530/ 31.01.2013 / © Raluca Popescu 15
3. Test types
Non functional
Related to changes
Structural
Functional
• Suitability
• Interoperability
• Security
• Accuracy
• Compliance
• Performance, Load, Stress
• Reliability (robust, fault tolerant, recoverable)
• Usability (understand, learn, operate, like)
• Efficiency (time behavior, resource
utilization)
• Maintainability (analyze, change, stabilize,
test)
• Portability (adapt, install, co-exist, replace)
• Confirmation testing
• Regression testing
“What” the system does “How” the system works
• Code coverage
INF5530/ 31.01.2013 / © Raluca Popescu 16
• Objectives: Test what a system should do and considers the external
behavior of the software.
• Test levels: may be performed at all test levels (e.g. tests for components
may be based on a component specification).
• Test basis: The expected behavior description can be found in work
products such as:
- requirements specification
- use cases
- functional specification
- may be undocumented.
• Examples:
- security testing - investigates the functions (e.g. a firewall)
relating to detection of threats, such as viruses, from malicious outsiders.
- interoperability testing - evaluates the capability of the software
product to interact with one or more specified components or systems.
3. Test types
3.1 Functional testing (Black box testing)
INF5530/ 31.01.2013 / © Raluca Popescu 17
• Objectives: measuring characteristics of software that can be
quantified on a varying scale:
– ex: response times for performance testing.
• Test levels: may be performed at all test levels.
• You can find more about them in ‘Software Engineering –
Software Product Quality’ (ISO 9126).
3. Test types
3.2 Non functional testing
INF5530/ 31.01.2013 / © Raluca Popescu 18
• Objectives: help measure the thoroughness of testing through
assessment of coverage of a type of structure.
• Test levels: may be performed at all test levels, but especially in
component testing and component integration testing
• Approach:
– Structural techniques are best used after specification-based techniques, in order to
help measure the thoroughness of testing
– Structural testing is based on the architecture of the system(i.e. a calling hierarchy)
• Tools can be used to measure the code coverage of elements, such as
statements or decisions.
3. Test types
3.3 Structural testing (white box testing)
INF5530/ 31.01.2013 / © Raluca Popescu 19
• Confirmation testing:
After a defect is detected and fixed, the software should be retested
to confirm that the original defect has been successfully removed.
• Regression testing:
the repeated testing of an already tested program, after modification,
to discover any defects introduced or uncovered as a result of the
change(s).
Test levels:
– may be performed at all test levels,
– applies to functional, non-functional and structural testing.
• Approach:
- The extent of regression testing is based on the risk of finding
defects in software that was working previously.
- Regression test suites are run many times and generally evolve
slowly, so regression testing is a strong candidate for automation.
3. Test types
3.4 Testing related to changes
INF5530/ 31.01.2013 / © Raluca Popescu 20
4. Maintenance testing
INF5530/ 31.01.2013 / © Raluca Popescu 21
• Objectives:
Maintenance testing is done on an existing operational system, and is triggered by
modifications, migration, or retirement of the software or system.
• Types of maintenance testing:
• Modifications include:– planned enhancement changes (e.g. release-based),
– corrective and emergency changes (patches), and changes of environment (operating system or database
upgrades)
• Migration (e.g. from one platform to another) should include:– operational tests of the new environment
– tests on the changed software.
• Retirement of a system includes: – the testing of data migration or archiving if long data-retention periods are required.
Maintenance testing also includes extensive regression testing to parts of the system
that have not been changed.
4. Maintenance testing
INF5530/ 31.01.2013 / © Raluca Popescu 22
• The scope of maintenance testing is related to:
– the risk of the change,
– the size of the existing system
– the size of the change.
• Test levels:
Depending on the changes, maintenance testing may be done at any or all test levels and
for any or all test types.
• Approach:
– Determining how the existing system may be affected by changes is called impact analysis, and
is used to help decide how much regression testing to do.
• Note:
• Maintenance testing can be difficult if specifications are out of date or missing.
4. Maintenance testing