Ivica Crnkovic Mälardalen University Department of Computer Engineering [email protected]

Page 1, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©

Ivica Crnkovic

Mälardalen University

Department of Computer Engineering

[email protected]


Software faults and failuresSoftware faults and failures

What does it mean that the software has failed?

The software does not do what the requirements describe

Reasons to the failures

Requirements incomplete, inconsistent, impossible to implement

Wrong interpretation of the requirements

The system design may contain a fault (misunderstanding requirements, wrong architecture, incomplete design)

Program design may contain a fault - wrong algorithm

The program code may be wrong - improper, incomplete, inconsistent implementation

Documentation can be wrong - can describe incorrectly the behavior of the system


Some other reasons to the failuresSome other reasons to the failures

Integration problems

The components integrated into a systems are not compatible (wrong components, wrong component versions)

The performance of the entire system does not correspond to the requirements (expectations)

The system requires too much resources

Not failures but still unsuccessful

Bad Graphical User Interface

Complicated process

Missed points with requirements (users actually wanted something else)


Defect classificationsDefect classifications

What type of defect?

When it occurred (when failure, when fault?)

Severity level?

What is the foot cause of the defect?

….


Specification/requirements

Environment/support

Documentation OtherDesign Code

ORIGIN: WHERE?

MODE:WHY?

TYPE

: W

HA

T?

Missing Unclear Wrong Changed Better way

HW interface

SW interface

User interface

Functionaldescription

Test HW

Test SW

Integration SW

Developmenttools

Logic

Computation

Data handling

Moduleinterface/

implementation

Standards

(Inter-)Processcommunications

Data definition

Module designLogic

description

Error checking

Standards

Fault classification (HP)Fault classification (HP)

Requirementsor

specifications

Functionality


Data handling6%

Documentation19%

Requirements5%

Hardware4%

Process/interprocess

5%Logic32%

Computation18%

Other code11%

Example:Faults statistics (HP)

What are the most frequent faultsWhat are the most frequent faults


Costs to repair the failuresCosts to repair the failures

Specification/requirements

IntegrationDesign Code

$$$ to fix the problems

Origin of problems


How to avoid a late detection of the faults?How to avoid a late detection of the faults?

Use the development process that suits the best

Do proper planning

Ensure the quality in each process step

perform reviews

perform tests


REQUIREMENTSANALYSIS

SYSTEMDESIGN

PROGRAM DESIGN

CODING

UNIT & INTE-GRATION TESTING

SYSTEMTESTING

OPERATION& MAINTENANCE

Verify design

Validate requirementsACCEPTANCE

TESTING

V Model - document producedV Model - document produced

RS

Acceptance testdescription

SD, FSSystem design test description

DS, CodeModule test description & code

Test analysis report.


The testing processThe testing process

Sub-systemtesting

Moduletesting

Unittesting

Systemtesting

Acceptancetesting

Componenttesting

Integration testing Usertesting


Unittest

Unittest

Unittest

Integrationtest

Functiontest

Performancetest

Acceptancetest

Installationtest

Com

pone

nt c

ode

Com

pone

nt c

ode

.

.

.

Tested component

Test

ed c

ompo

nent

Integratedmodules

Functioningsystem

Verified,validatedsoftware

Acceptedsystem

SYSTEMIN USE!

Designspecifications

Systemfunctional

requirements

Othersoftware

requirements

Customerrequirementsspecification

Userenvironment

The testing process - detailsThe testing process - details


Program authors (module tests)

The developer group (module tests, component tests)

Test teams

Integration tests, subsystem tests

System tests

“Big configuration test”

Special customers (running alpha, beta releases)

Customers with support people (acceptance tests)

Who performs the tests?Who performs the tests?

New functions

Development Alpha releases Beta releases

Fixing defects

Officialrelease


Unit TestingUnit Testing

Different methods to test the code

Code review - go trough the code

Code author, or a review group in the developer team (peer review)

Code inspection

Inspect the code against a prepared list of concerns (data-types, style guide…)

Proving code correct

Formal verification

Choosing test cases

Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification


Black-box testingBlack-box testing

Approach to testing where the program is considered as a ‘black-box’

The program test cases are based on the system specification

Test planning can begin early in the software process

Ie

Input test data

OeOutput test results

System

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects


Interface testingInterface testing

Takes place when modules or sub-systems are integrated to create larger systems

Objectives are to detect faults due to interface errors or invalid assumptions about interfaces

Particularly important for object-oriented development as objects are defined by their interfaces

Testcases

BA

C

The test is applied to sub-systems, not to individual modules


Interfaces typesInterfaces types

Parameter interfaces

Data passed from one procedure to another

Shared memory interfaces

Block of memory is shared between procedures

Procedural interfaces

Sub-system encapsulates a set of procedures to be called by other sub-systems

Message passing interfaces

Sub-systems request services from other sub-systems


White-box testing (structural testing)White-box testing (structural testing)

Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases

Objective is to exercise all program statements (not all path combinations)

Componentcode

Testoutputs

Test data

DerivesTests


Path testing Path testing

A white-box testing

Testing different combinations of program paths

Measuring the coverage percentile of the test

if-then-else loop-while case-of


A

DCB

FE G

Integration testingIntegration testing

Test on the sub-systems levels

Testing how the components work together


TestE

TestF

TestG

TestB,E,F

TestC

TestD,G

TestA,B,C,D,

E,F,G

Integration test - bottom-up principleIntegration test - bottom-up principle

A

DCB

FE G


TestA

TestA,B,C,D

TestA,B,C,D,

E,F,G

Top-down testingTop-down testing

Test from the top level

all components are not necessary completed (and tested)

writing STUBS (which simulates the component)

A

DCB

FE G


TestA

TestA,B,C,D

TestA,B,C,D,

E,F,G

TestB

TestC

TestD

TestE

TestC

TestD

TestF

TestG

Combination of Top-down and Down-top modelCombination of Top-down and Down-top model

A

DCB

FE G


TestA

TestB,E,F Test

A,B,C,D,E,F,G

TestE

TestD,G

TestF

TestG

A

DCB

FE G

Sandwich IntegrationSandwich Integration


Milestone 1: Most critical featuresand shared components Design, code, prototype

Usability testing Daily builds

Feature integration Eliminate severe faults

Milestone 2: Desirable features Design, code, prototype


Feature integration

Milestone 3: Least critical features Design, code, prototype


Feature integration and completionRelease to manufacturing

Microsoft synch-and stabilize approach


Object-oriented systems testingObject-oriented systems testing

Less closely coupled systems. Objects are not necessarily integrated into sub-systems

Cluster testing. Test a group of cooperating objects

Thread testing. Test a processing thread as it weaves from object to object. Discussed later in real-time system testing

Complication with inheritance


Object-oriented systems testingObject-oriented systems testing

Modularity

Small methods

Reuse

Interfaces identified earlier

Encapsulation

Inheritance

Polymorphism

Dynamic binding

Complex Interfaces

More integration

Easier parts of testing Harder parts of testing


Automated testing toolsAutomated testing tools

Static Analyzers

Code analysisShow some potential program errors and low quality code (too many

branches, complicated structure, bad naming policy

Generate different graphs showing the program structureData analyzer (initialization, usage, etc.)

Dynamic Analyzers

Checking a running program

Memory leakage

non proper use of variables

Measuring timing

Measuring execution coverage


Probabilityof existenceof additional

faults

Number of faults found to date

When to stop with testing?When to stop with testing?


Requirementsanalysis

Systemdesign

Programdesign

Programimplementation

Unit/integration testing

System testing

Maintenance

Incorrect, missing or unclear requirementsIncorrect or unclear translation

Incorrect or unclear design specification

Incorrect or unclear design specificationMisinterpretation of system design

Misinterpretation of program designIncorrect documentationIncorrect syntax or semantics

Incomplete test procedures New faults introduced when old ones corrected

Incomplete test procedures

Incorrect user documentationPoor human factorsNew faults introduced when old one correctedChanges in requirements


Integratedmodules

SYSTEMIN USE!

Functiontest

Performancetest

Acceptancetest

Installationtest

Functioningsystem

Verified,validatedsoftware

Acceptedsystem

Systemfunctional

requirements

Othersoftware

requirements

Customerrequirementsspecification

Userenvironment


TEST PLAN

SYSTEM TESTFUNCTION

Function 1 1Function 2 3,4 . .

TEST SPECIFICATION TEST 1Requirements tested

Functions tested

Methods

Conditions

TEST SPECIFICATION TEST 2Requirements tested

Functions tested

Methods

Conditions

TEST DESCRIPTION TEST 1Test data

Test procedures1.

2.

TEST DESCRIPTION TEST 2Test data

Test procedures1.

2.

TEST ANALYSISREPORT

TEST 1ResultsPerform

test 1

Performtest 2

TEST ANALYSISREPORT

TEST 2Results


S.P0204.6.10.3016FAULT REPORT

ORIGINATOR: Joe Bloggs

BRIEF TITLE: Exception 1 in dps_c.c line 620 raised by NAS

FULL DESCRIPTION Started NAS endurance and allowed it to run for a few minutes. Disabled the active NASlink (emulator switched to standby link), then re-enabled the disabled link and CDIS exceptioned as above. (I

think the re-enabling is a red herring.) (during database load)

ASSIGNED FOR EVALUATION TO: DATE:

CATEGORISATION: 0 1 2 3 Design Spec DocnSEND COPIES FOR INFORMATION TO:

EVALUATOR: DATE:

CONFIGURATION ID ASSIGNED TO PART

COMMENTS: dpo_s.c appears to try to use an invalid CID, instead of rejecting the message. AWJ

dpo_s.c

8/7/92

ITEMS CHANGED

CONFIGURATION ID IMPLEMENTOR/DATE REVIEWER/DATE BUILD/ISSUE NUM INTEGRATOR/DATE

COMMENTS:

CLOSED

FAULT CONTROLLER: DATE: 9/7/92

dpo_s.c v.10 AWJ 8/7/92 MAR 8/7/92 6.120 RA 8-7-92


DISCREPANCY REPORT FORM

DRF Number:__________________________________________ Tester name:____________________________Date: ___________________________________ Time: ________________________________Test Number: ______________________________Script step executed when failure occurred: __________________________________________________________Descripton of failure: _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Activities before occurrence of failure:______________________________________________________________________________________________________________________________________________________________________________Expected results:______________________________________________________________________________________________________________________________________________________________________________Requirements affected:______________________________________________________________________________________________________________________________________________________________________________Effect of failure on test:______________________________________________________________________________________________________________________________________________________________________________Effect of failure on system:______________________________________________________________________________________________________________________________________________________________________________Severity level:(LOW) 1 2 3 4 5 (HIGH)

Date post:	31-Dec-2015
Category:	Documents
Upload:	brynn-farrell
View:	27 times
Download:	0 times

Ivica Crnkovic Mälardalen University Department of Computer Engineering [email protected]

Documents