Politehnica University of TimisoaraMobile Computing, Sensors Network and Embedded Systems

Laboratory

Embedded systems testing

Testing throughout the software life cycle

instructor: Razvan BOGDAN

Outlines 1.Software Development Models

2.Test Levels

3.Testing Types

4.Maintenance Testing

SOFTWARE DEVELOPMENT MODELS • The development process adopted for a project will depend on the project aims and goals.

• There are numerous development life cycles that have been developed in order to achieve different required objectives.

• These life cycles range from lightweight and fast methodologies, where time to market is of the essence, through to fully controlled and documented methodologies where quality and reliability are key drivers.

SOFTWARE DEVELOPMENT MODELS • The life cycle model that is adopted for a project will have a big impact on the testing that is carried out

• Test activities are highly related to software development activities

• The life cycle model will define the what, where, and when of our planned testing, influence regression testing, and largely determine which test techniques to use.

SOFTWARE DEVELOPMENT MODELS • In every development life cycle, a part of testing is focused on verification testing and a part is focused on validation testing.

• Verification focuses on the question 'Is the deliverable built according to the specification?'.

• Validation focuses on the question 'Is the deliverable fit for purpose, e.g. does it provide a solution to the problem?'.

SOFTWARE DEVELOPMENT MODELS • Waterfall• It has a natural timeline where tasks are executed in a sequential fashion.

• We start at the top of the waterfall with a feasibility study and flow down through the various project tasks finishing with implementation into the live environment

• Testing tends to happen towards the end of the project life cycle so defects are detected close to the live implementation date.

SOFTWARE DEVELOPMENT MODELS

SOFTWARE DEVELOPMENT MODELS • Waterfall • With this model it has been difficult to get feedback passed backwards up the waterfall

• There are difficulties if we need to carry out numerous iterations for a particular phase

SOFTWARE DEVELOPMENT MODELS • V-model •The V-model was developed to address some of the problems experienced using the traditional waterfall approach

• The V-model provides guidance that testing needs to begin as early as possible in the life cycle

• There are a variety of activities that need to be performed before the end of the coding phase. These activities should be carried out in parallel with development activities

V-model

SOFTWARE DEVELOPMENT MODELS • Testing along the V-model• Development and test are two equal branches

• Each development level has a corresponding test level• Tests (right hand side) are designed in parallel with software development (left hand side)• Testing activities take place throughout the complete software life cycle

SOFTWARE DEVELOPMENT MODELS • Testing along the V-model• Software development branch

• Requirements Definition• specification documents

• Functional System Design• design functional program flow

• Technical System Design• define architecture/interfaces, their interactions

• Component specification• structure of component

• Programming• create executable code

SOFTWARE DEVELOPMENT MODELS • Testing along the V-model• Software test branch

• 4. Acceptance Testing• formal test of customer requirements

• 3. System Testing• integrated system, specifications

• 2. Integration Testing• component interfaces• software does what should be doing from a functional point of view

• 1. Component Testing• component’s functionality

SOFTWARE DEVELOPMENT MODELS • Verification vs. Validation• Verification

• Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. [ISO 9000]• Main issue: Did we proceed correctly when building the system? Did we use correctly this component?

• Validation• Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]• Main issue: Did we build the right software system? Was it the matter to use that particular component or maybe we should have used another one?

SOFTWARE DEVELOPMENT MODELS • Verification within the general V-Model• Each development level is verified against the contents of the level above it

• to verify: to give proof of evidence, to substantiate

• to verify means to check whether the requirements and definitions of the previous level were implemented correctly

Verifi

catio

n

SOFTWARE DEVELOPMENT MODELS • Validation within the general V-Model• Validation refers to the correctness of each development level

• to validate: to give proof of having value

• to validate means to check the appropriateness of the results of one development level

• e.g., is the component offering the expected behavior? Is the system offering that particular behavior which is mentioned/agreed in the System Design Document?

Verifi

catio

n Validation

V-model • Although variants of the V-model exist, a common

type of V-model uses four test levels.

1. component testing: searches for defects in and verifies the functioning of software components (e.g. modules, programs, objects, classes etc.) that are separately testable;

2. integration testing: tests interfaces between components, interactions to different parts of a system such as an operating system, file system and hard ware or interfaces between systems;

V-model • Although variants of the V-model exist, a common

type of V-model uses four test levels.

3. system testing: concerned with the behavior of the whole system/product as defined by the scope of a development project or product. The main focus of system testing is verification against specified requirements;

4. acceptance testing: validation testing with respect to user needs, require ments, and business processes conducted to determine whether or not to accept the system.

SOFTWARE DEVELOPMENT MODELS • Iterative life cycles• Not all life cycles are sequential. V-Cycle takes a large amount of time, being

applied in those cases where the requirements are not changing (ok, that often )

• There are also iterative or incremental life cycles where, instead of one large development time line from beginning to end, we cycle through a number of smaller self-contained life cycle phases for the same project.

SOFTWARE DEVELOPMENT MODELS • Iterative life cycles• A common feature of iterative approaches is that the delivery is divided

into increments or builds with each increment adding new functionality

• The increment produced by an iteration may be tested at several levels as part of its development.

• Subsequent increments will need testing for the new functionality, regression testing of the existing functionality, and integration testing of both new and existing parts

SOFTWARE DEVELOPMENT MODELS • Iterative life cycles

• Regression testing is increasingly important on all iterations after the first one. This means that more testing will be required at each subsequent delivery phase which must be allowed for in the project plans

• This life cycle can give early market presence with critical functionality, can be simpler to manage because the workload is divided into smaller pieces, and can reduce initial investment although it may cost more in the long run.

SOFTWARE DEVELOPMENT MODELS • Examples of iterative or incremental development models are:

• Prototyping• Building quickly a usable representation of the system, followed by successive

modification until the system is ready

• Rapid Application Development (RAD)• The user interface is implemented using out-of-the box functionality taking the

functionality which will be later developed

• Rational Unified Process (RUP) • Object oriented model and a product of the company Rational/IBM. It mainly

provides the modelling language UML and support for the Unified Process

• Agile development • Development and testing take place without formalized requirements

specification

SOFTWARE DEVELOPMENT MODELS • Rapid Application Development (RAD) is formally a parallel development of functions and subsequent integration.

• Components/functions are developed in parallel as if they were mini projects, the developments are time-boxed, delivered, and then assembled into a working prototype

• This can very quickly give the customer something to see and use and to provide feedback regarding the delivery and their requirements

• This methodology allows early validation of technology risks and a rapid response to changing customer requirements.

• Dynamic System Development Methodology [DSDM] is a refined RAD process that allows controls to be put in place in order to stop the process from getting out of control

Rapid Application Development

SOFTWARE DEVELOPMENT MODELS • The RAD development process encourages active customer feedback.

• An early business-focused solution in the market place gives an early return on investment (ROI) and can provide valuable marketing information for the business

SOFTWARE DEVELOPMENT MODELS • Extreme Programming (XP) is currently one of the most well-known Agile development life cycle models.

• Some characteristics of XP are: • It promotes the generation of business stories to define the functionality.

• It demands an on-site customer for continual feedback and to define and carry out functional acceptance testing.

• It promotes pair programming and shared code ownership amongst the developers.

SOFTWARE DEVELOPMENT MODELS • Agile development • Some characteristics of XP are:

• It states that component test scripts shall be written before the code is written and that those tests should be automated.

• It states that integration and testing of the code shall happen several times a day.

• It states that we always implement the simplest solution to meet today's problems.

SOFTWARE DEVELOPMENT MODELS • XP developers write every test case they can think of and automate them.

• Every time a change is made in the code it is component tested and then integrated with the existing code, which is then fully integration-tested using the full set of test cases.

• It gives continuous integration, which mean that changes are incorporated continuously into the software build.

• At the same time, all test cases must be running at 100% meaning that all the test cases that have been identified and automated are executed and pass.

SOFTWARE DEVELOPMENT MODELS • Iteration models: Test Driven Development (TDD)• Based on: test case suites

• Prepare test cycles• Automated testing using test tools

• Development according to test cases• Prepare early versions of the component for testing• Automatic execution of tests• Correct defects on further versions• Repeat test suites until no errors are found

• First the tests are designed, then the software is programmed; • test first, code after

SOFTWARE DEVELOPMENT MODELS • Iteration models: Test Driven Development (TDD)• Based on: test case suites

• Prepare test cycles• Automated testing using test tools

• Development according to test cases• Prepare early versions of the component for testing• Automatic execution of tests• Correct defects on further versions• Repeat test suites until no errors are found

• First the tests are designed, then the software is programmed; • test first, code after

SOFTWARE DEVELOPMENT MODELS • Principles of all models• each development activity must be tested

• no piece of software may be left untested, whether it was developed “in one procedure” or iteratively

• each test level should be tested specifically• each test level has its own test objectives• the test performed at each level must reflect these objectives

• testing begins long before test execution• as soon as development begins, the preparation of the corresponding tests can start• this is also the case for document reviews starting with concepts specification and overall design

SOFTWARE DEVELOPMENT MODELS - Summary

• whichever life cycle model is being used, there are several characteristics of good testing:

• for every development activity there is a corresponding testing activity;

• each test level has test objectives specific to that level;

• the analysis and design of tests for a given test level should begin during the corresponding development activity;

• testers should be involved in reviewing documents as soon as drafts are available in the development cycle.

Outlines 1.Software Development Models

2.Test Levels

3.Testing Types

4.Maintenance Testing

TEST LEVELS• Testing levels:

• Component testing • Integration testing • System testing • Acceptance testing

TEST LEVELS• 1. Component (unit) testing

• also known as unit, module or program• test of each software component after its realization• searches for defects in, and verifies the functioning of software (e.g. modules, programs, objects, classes, etc.) that are separately testable.

• Due to the naming of components in different programming languages, the component test may be referred to as:

• module test (e.g. in C) • class test (e.g. in Java or C++)• unit test (e.g. in Pascal)

• The components are referred to as modules, classes or units. Because of the possible involvement of developers in the test execution, they are also called developer’s test

TEST LEVELS• 1. Component testing

• Test cases may be derived from (= Test basis ):• Component requirements• (Detailed) software design• Code• Data models

• Typical test objects:• Components/classes/units/modules• Programs• Data conversion/migration programs• Database modules

TEST LEVELS• 1. Component testing: Scope

• Only single components are tested• components may consist of several smaller units• test object often cannot be tested stand alone

• Every component is tested on its own• finding failures caused by internal defects• cross effects between components are not within the scope this test

TEST LEVELS• 1. Component testing: Functional/non functional testing• Testing functionality

• Every function must be tested with at least one test case• are the functions working correctly, are all specifications met?

• Defects found commonly are:• defects in processing data, often near boundary values• missing functions

• Testing robustness (resistance to invalid input data)• How reliable is the software?

• Test cases representing invalid inputs are called negative tests• A robust system provides an appropriate handling of wrong inputs

• Wrong inputs accepted in the system may produce failure in further processing (wrong output, system crash)

• Other non-functional attributes may be tested• e.g., performance and stress testing, reliability

TEST LEVELS• 1. Component testing: Test harness /1•Test execution of components often requires drivers and stubs

• Drivers handle the interface to the component

• calls a component to be tested • Simulate inputs, records outputs and provide a test harness• they use programming tools

• Stubs replace or simulate components not yet available or not part of the test object

• called from the software component to be tested

• To program drivers and/or stubs one• must have programming skills• need to have the source code available• may need some special tools

TEST LEVELS• 1. Component testing: Test harness /2• Example 1: Stubs..• Q: How to test module L?

Module L

Module K Module F(not developed) (not developed)

TEST LEVELS• 1. Component testing: Test harness /3• Example 2: Stubs..• A: Stubs: called from the software component to be tested

Module L

Module K Stub Module F StubDummy code that SIMULATES the functionality of the

undeveloped modules

TEST LEVELS• 1. Component testing: Test harness /4• Example 1: •A: Stubs: called from the software component to be tested

void functionToBeTested(params..) {………...int p = price(param1);………..

}void price(int param) { //this is the stub

return 10; // We don’t care what the price is. We just need a value so we can test the other function}

TEST LEVELS• 1. Component testing: Test harness /5• Example 2: What about drivers?..• Q: How to test module L?

Module K Module F

Module L(not developed)

TEST LEVELS• 1. Component testing: Test harness /6• Example 2: What about drivers?..• Q: How to test module L?• A: A driver calls a component to be tested

Module K Module F

Module L Driver

Dummy code that returns values from Module L in… Module K

TEST LEVELS• 1. Component testing: Test harness /7• Example 2: What about drivers?..• Q: How to test module L?• A: A driver calls a component to be tested

void functionThatCallsPrice (params..) { //this is the driver

int p = price(param1);

printf(“Price is: %d”, p);

}

void price(int param) {

//complex ecuations and DB interogations that determin the real price

}

TEST LEVELS• 1. Component testing: Test harness /8• Example 3: Driver - UI in which the tester can introduce some data, simulate some input for the Component, this application being connected to the Component

Unit Under Test (F)

Driver

TEST LEVELS• 1. Component testing: Methods• The program code is available to the tester

• In case “tester = developer”:testing takes place with a strong development focus•Knowledge about functionality, component structure and variables may be applied to design test cases•Often functional testing will apply•Additionally, the use of debuggers and other development tools (e.g. unit test frameworks) will allow direct access to program variables

•Source code knowledge allows to use white box methods for component test

TEST LEVELS• 1. Component testing - Summary • A component is the smallest system unit specified

• Module, unit, class and developer’s test are used as synonyms • Drivers will execute the component functions and adjacent functions that are replaced by stubs

• Component tests may check functional and non-functional system properties

TEST LEVELS• 2. Integration testing • Test basis

• Software and system design• Architectural design• Workflows• Use cases• Interface specifications• Data models

• Typical test objects• Subsystems database implementation• Infrastructure• Interfaces• System configuration

• Configuration data

TEST LEVELS• 2. Integration (interface) testing• tests interfaces between components, interactions to different parts of a system such as an operating system, file system and hardware or interfaces between systems.

• each component has already been tested for its internal functionality (component test ) => integration tests examine the external functions after component testing• examines the interaction of software elements (yes! components) between different systems or hardware and software • Integration is the activity of combining individual software components into a larger subsystem or in a series of systems• Further integration of a subsystems is also part of the system integration process

• Integration testing is often carried out by the integrator, but preferably by a specific integration tester or test team

TEST LEVELS• 2. Integration testing: Scope /1• Integration tests assume that the components have already been tested • Integration tests examine the interaction of software components (subsystems) with each other:

• interfaces with other components• interfaces among GUIs/MMIs

• Integration tests examine the interfaces with the system environment• In most cases, the interaction tested is that of the component and simulated environment behavior• Under real conditions, additional environmental factors may influence the components behavior

TEST LEVELS• 2. Integration testing: Scope /2• A subsystem, composed of individual components, will be tested

• each component has an interface either external and/or interacting with another component with the subsystem

• Test drivers (which provide the process environment of the system or subsystem) are required

• to allow for or to produce input and output of the subsystem• to log data

• Test drivers of the component tests may be reused here

TEST LEVELS• 2. Integration testing: Scope /3• Monitoring tools logging data and controlling tests can support testing activities• Stubs replace missing components

• data or functionality of a component that have not yet been integrated will be replaced by programming stubs • stubs take over the elementary tasks of the missing components

TEST LEVELS• 2. Integration testing: Approach• Integration tests aim at finding defects in the interfaces. They check the correct interaction of components

• among other reasons, in order to check performance and security aspects, requiring additional non-functional tests

• Replacing test drivers and stubs with real components may produce new defects , such as

• loosing data, wrong handling of data or wrong inputs• the components involved interpret the input data in a different manner• the point in time where data is handed over is not correct: too early, too late, at a wrong frequency

TEST LEVELS• 2. Integration testing: Strategies /1• There are different strategies for integration testing

• ‘Big-bang' integration testing - one extreme - all components or systems are integrated simultaneously, after which everything is tested as a whole.

• Big-bang testing has the advantage that everything is finished before integration testing starts. There is no need to simulate (as yet unfinished) parts

• The major disadvantage of the big-bang is that in general it is time-consuming and difficult to trace the cause of failures with this late integration.

TEST LEVELS• 2. Integration testing: Strategies /1• There are different strategies for integration testing

• Incremental testing - another extreme - all programs are integrated one by one, and a test is carried out after each step.

• The incremental approach has the advantage that the defects are found early in a smaller assembly when it is relatively easy to detect the cause.

• A disadvantage is that it can be time-consuming since stubs and drivers have to be developed and used in the test

TEST LEVELS• 2. Integration testing: Strategies /2• There are different strategies for integration testing

• Within incremental integration testing a range of possibilities exist, partly depending on the system architecture:

• Top-down: testing takes place from top to bottom, following the control flow or architectural structure (e.g. starting from the GUI or main menu). Components or systems are substituted by stubs.

• Bottom-up: testing takes place from the bottom of the control flow upwards. Components or systems are substituted by drivers.

• Functional incremental: integration and testing takes place on the basis of the functions or functionality, as documented in the functional specification.

TEST LEVELS• 2. Integration testing: Strategies /3• There are different strategies for integration testing

• The preferred integration sequence and the number of integration steps required depend on the location in the architecture of the high-risk interfaces.

•The best choice is to start integration with those interfaces that are expected to cause most problems

• Ad-hoc integration• components will be tested, if possible, directly after programming and component tests have been completed• implies early start of testing activities, possibly allowing for a shorter software development process as a whole

TEST LEVELS• 2. Integration testing: Summary• Integration means building up groups of components

• Integration tests examine component interactions against the specification of interfaces

• Integrating sub-systems (they consist of integrated components) is also a form of integration

TEST LEVELS• 3. System testing• Test basis (may include tests based on)

• System and software requirement specification• Use cases, business process• Functional specification• Risk analysis reports

• Typical test objects• System, user and operation manuals• System configuration and configuration data

TEST LEVELS• 3. System testing• The process of testing an integrated system to verify that it meets specified requirements. [Hetzel]

• The system testing means the behavior of the whole system• The scope is defined in the Master Test Plan (or Level Test Plan)• Software quality is looked at from the user’s point of view

• we want to use a user’s point of view

• System tests refer to (as per ISO 9126):• functional and non-functional requirements (functionality, reliability, usability, efficiency, maintainability, portability)• The characteristics of the data quality are the base for testing

TEST LEVELS• 3. System testing: Scope• Test of the integrated system from the user’s point of view

• Complete and correct implementation of requirements• Deployment in the real system environment with real life data

• The test environment should match the true environment• No test drivers or stubs are needed!• All external interfaces are tested under true conditions• close representation of the later true environment

• No tests in the real life environment!• Induced defects could damage the real life environment• Software under deployment is constantly changing. Most tests will not be reproducible

TEST LEVELS• 3. System testing: functional requirements /1• Goal: to prove that the implemented functionality exposes the required characteristics

• According to ISO9126, characteristics to be tested include:• Suitability

• are the implemented functions suitable for their expected use• the software should not be more complex than necessary

• Accuracy• Do the functions produce correct (agreed upon) results?

• Interoperability• Does interaction with the system environment show any problems/

• Compliance• Does the system comply with applicable norms and regulations?

• Security• Are data/programs protected against unwanted access or loss?• e.g. unauthorized access to programs & data not available

TEST LEVELS• 3. System testing: functional requirements /2• Three approaches for testing functional requirements:

• Business process based test• each business process serves as basis for deriving tests• the ranking order of the business process can be applied for prioritizing test cases

• Use case based test• test cases are derived from sequences of expected or reasonable use• sequences used more frequently receive a higher priority

• Requirements based test (building blocks)• test cases are derived from the requirement specification• the number of test cases will vary according to the type/depth of specification Requirements based test

TEST LEVELS• 3. System testing: Non-functional requirements /2• Compliance with non-functional requirements is difficult to achieve:

• Their definition is often very vague (e.g. easy to operate, well structured user interface, etc.)• They are not stated explicitly. They are an implicit part of system description, still they are expected to be fulfilled • Quantifying them is difficult, often non-objective metrics must be used, .g. looks pretty, quite safe, easy to learn.

• Examples: Testing/inspecting documentation• Is documentation of programs in line with the actual system, is it concise, complete and easy to understand?

• Example: Testing maintainability• All programmers comply to the respective Coding-Standards?• Is the system designed in a structured, modular fashion?

TEST LEVELS• 3. System testing: Non-functional requirements /2• Compliance with non-functional requirements is difficult to achieve:

• Their definition is often very vague (e.g. easy to operate, well structured user interface, etc.)• They are not stated explicitly. They are an implicit part of system description, still they are expected to be fulfilled • Quantifying them is difficult, often non-objective metrics must be used, .g. looks pretty, quite safe, easy to learn.

• Examples: Testing/inspecting documentation• Is documentation of programs in line with the actual system, is it concise, complete and easy to understand?

• Example: Testing maintainability• All programmers comply to the respective Coding-Standards?• Is the system designed in a structured, modular fashion?

TEST LEVELS• 3. System testing: Summary• System testing is performed using functional and non-functional test cases

• Functional system testing confirms that the requirements for a specific intended use have been fulfilled (validation)• Non-functional system testing verifies non-functional quality attributes

• e.g. usability, efficiency, portability etc.

• Non-functional quality attributes are often an implicit part of the requirements, this makes it difficult to validate them

• An independent test team often carries out system testing

TEST LEVELS - Class work•Which of these is a functional test?

• a. Measuring response time on an on-line booking system. • b. Checking the effect of high volumes of traffic in a call-center system. • c. Checking the on-line bookings screen information and the database contents

against the information on the letter to the customers. • d. Checking how easy the system is to use.

TEST LEVELS - Class work•Which of these is a functional test?

• a. Measuring response time on an on-line booking system. • b. Checking the effect of high volumes of traffic in a call-center system. • c. Checking the on-line bookings screen information and the database contents

against the information on the letter to the customers. • d. Checking how easy the system is to use.

TEST LEVELS• 3. Acceptance testing

• Test basis (from where can the test cases be generated)• User requirements• System requirements • Use cases• Business processes • Risk analysis reports

• Typical test objects• Business processes on fully integrated system • Operational and maintenance processes• User procedures• Forms• Reports• Configuration data

TEST LEVELS• 3. Acceptance testing: contract and regulation acceptance• Does the software fulfill all the contractual requirements?

• Within the acceptance test for a business-supporting system, two main test types can be distinguished; they are usually prepared and executed separately • A. Acceptance testing: user acceptance testing

• Often customer selects test cases for acceptance testing• Focuses mainly on the functionality thereby typically verifies the fitness for the use of the system by business users

• “The customer knows best”

• Testing is done using the customer environment• Customer environment may cause new failures

TEST LEVELS• 3. B. Acceptance testing: operational acceptance testing• Requires that software is fit for use in a productive environment

• Integration of software into the customer IT-infrastructure (Backup/Restore-System, restart, install and de-install ability, disaster recovery etc.)• User management, interfacing to file and directory structures in use• Compatibility with other systems (other computers, database servers etc.)• Maintenance tasks• Data load and migration tasks• Periodic checks of security vulnerabilities

• Operational acceptance testing is often done by the customer’s system administrator

TEST LEVELS• 3. Acceptance testing: alpha testing and beta testing• A stable preliminary version of the software is needed (preliminary acceptance testing)

• Mostly done for market software (also called Commercial off the shelf COTS software)

•Alpha testing: This test takes place at the developer's site. •A cross-section of potential users and members of the developer's organization are invited to use the system. •Developers observe the users and note problems. •Alpha testing may also be carried out by an independent test team.

• Beta testing, or field testing, sends the system to a cross-section of users who install it and use it under real-world working conditions.

•The users send records of incidents with the system to the development organization where the defects are repaired.

•Advantages of alpha and beta tests:•Reduce the cost of acceptance testing•Use different user environment•Involve a high number of users

TEST LEVELS• 3. Acceptance testing: Summary

• Acceptance testing is the customer’s systems test

• Acceptance testing is a contractual activity, the software will then be verified to comply with customers requirements

• Alpha- and beta tests are tests performed by potential or existing customers wither at the developer’s site (alpha) or at the customers site (beta)

Outlines 1.Software Development Models

2.Test Levels

3.Testing Types

4.Maintenance Testing

TESTING TYPES• Test types and test levels• Test levels

• The previous section explained the various testing levels, i.e. component test, integration test etc.• At every test level the test objectives have a different focus!• Therefore different test types are applied during different test levels

• Test types• Functional testing ( Goal: Testing of function )• Non-functional testing ( Goal: Testing product characteristics )• Structural Testing ( Goal: Testing of SW structure / architecture )• Change related testing ( Goal: Testing after changes )

Test types: the targets of testing • A test type is focused on a particular test objective, which could be:

• the testing of a function to be performed by the component or system; • The testing of a non-functional quality characteristic, such as reliability or

usability; • The testing of the structure or architecture of the component or system; • Testing related to changes, i.e. confirming that defects have been fixed

(confirmation testing, or re-testing) and looking for unintended changes (regression testing).

TESTING TYPES• Testing of Function ( Functional testing )• Goal: the function of the test object

• Functionally can be linked to input and output data of the test object

• Black box methods are applied to design the relevant test cases• Testing is to verify functional requirements ( As stated in specifications,

concepts, case studies, business rules or related documents )• Area of use• Functional testing may be performed at all test levels• Execution• The test object is executed using text data derived from test cases• The results of the test execution are compared to the expected results• Security testing• Type of functional testing dealing with external threads• Malicious attacks could damage programs or data

TESTING TYPES• Testing of Non-functional Software Characteristics• Goal: software product characteristics

• How well does the software perform its functions?• The non-functional quality characteristics ( ISO 9126: reliability, usability, efficiency,

maintainability, portability ) are often vague, incomplete or missing all together, making testing difficult

• Area of use• Non-functional testing may be performed at all test levels• Typical non-functional testing:

• Load testing / performance testing / volume testing / stress testing • Testing of safety features• Reliability and robustness testing / compatibility testing• Usability testing / configuration testing

• Execution• Compliance with the non-functional requirements is measured using selected

functional requirements

TESTING TYPES• Non-functional Testing (system test )

• Load test• System under load ( minimum load, more users / transactions )• Performance test• How fast does the system perform a certain function• Volume test• Processing huge volumes of data / files• Stress test• Reaction to overload / recovery after return to normal• Reliability test• Performance while in “continuous operation mode”• Test of robustness• Reaction to input of wrong or unspecified data• Reaction to hardware failures / disaster recovery

TESTING TYPES• Non-functional Testing (system test )

• Compliance testing• Meeting rules and regulations (internal/ external )

• Test for usability• Structured, understandable, easy to learn for user

• Other non-functional quality aspects:• portability: replaceability/, install-ability,

conformance/compliance, adaptability

• maintainability: verifiability, stability, analyzability, changeability• reliability: maturity, robustness, recoverability

TESTING TYPES• Testing of Software Structure/Architecture (Structural

Testing)• Goal: Coverage

• Analyses the structure of the test object ( white box approach )• Testing aims at measuring how well the structure of the test object is

covered by the test cases• Area of use

• Structural testing possible on all test levels, code coverage testing using tools mainly done during component and integration testing

• Structural test design is finalized after functional tests have been designed, aiming at producing a high degree of coverage

• Execution• Will test the internal structure of a test object (e.g. control flaw within

components, flow through a menu structure)• Goal: all identified structural elements should be covered by test cases

TESTING TYPES• Testing related to Changes / 1 • Goal: test object after changes

• After a test object or its system environment has been changed, results related to the change have become invalid: tests have to be repeated• Two main reasons for changing software

• Error correction• Functional extension

• Because of undesired side effects of extended or new functionality, it is necessary to also retest adjacent areas!

Error correction

Functional extension

New test case

Confirmation testing

Regression testing

Release software

TESTING TYPES• Testing related to Changes /2 ( Re-testing and Regression Testing )• Areas of use

• Repeating a test of functionality that has already been verified is called a regression test.

• The scope of the regression test depends on the risk, that the newly implemented functionality (extension or error fix) imposes to the system.

• Analyzing this risk can be done with an impact analysis• Change related testing may be performed at all test levels.• Typical tests after changes are:

• Re-testing ( = Testing after correction of errors )• Regression testing ( = Testing to uncover newly introduced defects)

TESTING TYPES• Testing related to Changes / 3 • Execution

• Basically, execution takes place as in previously executed test iterations

• In most cases, a complete regression test is not feasible, because it is too expensive and takes too much time

• A high degree of modularity in the software allows for more appropriate reduced regression tests

• Criteria for the selection of regression test cases:• Test cases with high priority• Only test standard functionality, skip special cases and

variations• Only test configuration that is used most often• Only test subsystems / selected areas at the test object

• If during early project phases, it becomes obvious that certain tests are suitable for regression testing, test automation should be considered

TESTING TYPES• Summary

• On different test levels different types of tests are used• Test types are: functional, non-functional, structural and change

related testing• Functional testing examines the input / output behavior of a test

object• Non-functional testing checks product characteristics• Non-functional testing includes, but is not limited to, load testing,

stress testing, performance testing, robustness testing• Common structural tests are tests that check data and control flow

within the test object, measuring the degree of coverage• Important tests after changes are: re-tests and regression tests

Outlines 1.Software Development Models

2.Test Levels

3.Testing Types

4.Maintenance Testing

MAINTENANCE TESTING• Testing after Product Acceptance / 1

• Customer has approved the product and sets it into production• The initial development cycle, including its related tests, has been

completed• The software itself is at the beginning of its life cycle:

• It will be used for many years to come, if will be extended• It will most likely still have defects, hence if will be further modified

and corrected• It needs to adapt to new conditions and to be integrated into new

environments• It needs to change or extended the configuration data• It will one day be retired, put out of operation

• Any new version of the product, any new update and any other change in the software requires additional testing!

MAINTENANCE TESTING• Testing after Product Acceptance / 2• Configuration

• The composition of a component or system as defined by the number, nature, and interconnections of its constituent parts.

• Impact analysis• The assessment of change to the layers of deployment

documentation, test documentation and components, to implement a given change to specified requirements.

• Maintenance testing• Testing the changes to an operational system or the impact of a

changed environment to an operational system.

MAINTENANCE TESTING• Testing after Product Acceptance / 3

• Software maintenance covers two different fields:

• Maintenance as such: correction of defects or implementation of hot-fixes, that already were part of the initial version of the software• Planned software releases: adaptations as a result of a changed environment or new customer requirements

• Test scope of maintenance testing • Hot fixes and defect correction require retests• Extended functionality requires new test cases• Migration to another platform requires operational tests• In addition, intensive regression testing is needed

Error correction

Functional extension

New test case

Confirmation testing

Regresion testing

Release software

MAINTENANCE TESTING• Testing after Product Acceptance / 4• Scope of testing is affected by the impact of the change

• Impact analysis is used to determine the affected areas to decide the amount of regression testing

• Problems might occur, if documentation of the old software is missing or incomplete

• Software retirement• Test after software retirement may include

• Data migration tests• Verifying archiving data and programs• Parallel testing of old and new systems

MAINTENANCE TESTING

• Summary• Ready developed software needs to be adapted to new conditions,

errors have to be corrected• An impact analysis can help to judge the change related risks• Maintenance tests make sure, that• New functions are implemented correctly ( new test cases )• Errors have been fixed successfully ( old test cases )• Functionality, that has already been verified, is not affected

( regression test )• If software gets retired, migration tests or parallel tests may be

necessary