Date post: | 18-Nov-2014 |
Category: |
Documents |
Upload: | softwarecentral |
View: | 3,072 times |
Download: | 1 times |
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 1
Verification and Validation
l Assuring that a software systemmeets a user's needs
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 2
Objectives
l To introduce software verification and validationand to discuss the distinction between them
l To describe the program inspection process andits role in V & V
l To explain static analysis as a verificationtechnique
l To describe the Cleanroom software developmentprocess
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 3
Topics covered
l Verification and validation planning
l Software inspections
l Automated static analysis
l Cleanroom software development
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 4
l Verification:"Are we building the product right"
l The software should conform to its specification
l Validation: "Are we building the right product"
l The software should do what the user reallyrequires
Verification vs validation
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 5
l Is a whole life-cycle process - V & V must beapplied at each stage in the software process.
l Has two principal objectives• The discovery of defects in a system
• The assessment of whether or not the system is usable inan operational situation.
The V & V process
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 6
l Software inspections Concerned with analysis ofthe static system representation to discoverproblems (static verification)• May be supplement by tool-based document and code analysis
l Software testing Concerned with exercising andobserving product behaviour (dynamicverification)• The system is executed with test data and its operational
behaviour is observed
Static and dynamic verification
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 7
Static and dynamic V&V
Formalspecification
High-leveldesign
Requirementsspecification
Detaileddesign
Program
Prototype Dynamicvalidation
Staticverification
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 8
l Can reveal the presence of errors NOT theirabsence
l A successful test is a test which discovers oneor more errors
l The only validation technique for non-functionalrequirements
l Should be used in conjunction with staticverification to provide full V&V coverage
Program testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 9
l Defect testing• Tests designed to discover system defects.
• A successful defect test is one which reveals the presenceof defects in a system.
• Covered in Chapter 20
l Statistical testing• tests designed to reflect the frequence of user inputs. Used
for reliability estimation.
• Covered in Chapter 21
Types of testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 10
V& V goals
l Verification and validation should establishconfidence that the software is fit for purpose
l This does NOT mean completely free of defects
l Rather, it must be good enough for its intendeduse and the type of use will determine the degreeof confidence that is needed
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 11
V & V confidence
l Depends on system’s purpose, user expectationsand marketing environment• Software function
» The level of confidence depends on how critical the software is toan organisation
• User expectations» Users may have low expectations of certain kinds of software
• Marketing environment» Getting a product to market early may be more important than
finding defects in the program
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 12
l Defect testing and debugging are distinctprocesses
l Verification and validation is concerned withestablishing the existence of defects in a program
l Debugging is concerned with locating andrepairing these errors
l Debugging involves formulating a hypothesisabout program behaviour then testing thesehypotheses to find the system error
Testing and debugging
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 13
The debugging process
Locateerror
Designerror repair
Repairerror
Re-testprogram
Testresults Specification Test
cases
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 14
l Careful planning is required to get the most out oftesting and inspection processes
l Planning should start early in the developmentprocess
l The plan should identify the balance betweenstatic verification and testing
l Test planning is about defining standards for thetesting process rather than describing producttests
V & V planning
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 15
The V-model of development
Requirementsspecification
Systemspecification
Systemdesign
Detaileddesign
Module andunit codeand tess
Sub-systemintegrationtest plan
Systemintegrationtest plan
Acceptancetest plan
ServiceAcceptance
testSystem
integration testSub-system
integration test
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 16
The structure of a software test plan
l The testing process
l Requirements traceability
l Tested items
l Testing schedule
l Test recording procedures
l Hardware and software requirements
l Constraints
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 17
Software inspections
l Involve people examining the sourcerepresentation with the aim of discoveringanomalies and defects
l Do not require execution of a system so may beused before implementation
l May be applied to any representation of thesystem (requirements, design, test data, etc.)
l Very effective technique for discovering errors
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 18
Inspection success
l Many diffreent defects may be discovered in asingle inspection. In testing, one defect ,maymask another so several executions are required
l The reuse domain and programming knowledgeso reviewers are likely to have seen the types oferror that commonly arise
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 19
Inspections and testing
l Inspections and testing are complementary andnot opposing verification techniques
l Both should be used during the V & V process
l Inspections can check conformance with aspecification but not conformance with thecustomer’s real requirements
l Inspections cannot check non-functionalcharacteristics such as performance, usability, etc.
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 20
Program inspections
l Formalised approach to document reviews
l Intended explicitly for defect DETECTION (notcorrection)
l Defects may be logical errors, anomalies in thecode that might indicate an erroneous condition(e.g. an uninitialised variable) or non-compliancewith standards
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 21
Inspection pre-conditions
l A precise specification must be available
l Team members must be familiar with theorganisation standards
l Syntactically correct code must be available
l An error checklist should be prepared
l Management must accept that inspection willincrease costs early in the software process
l Management must not use inspections for staffappraisal
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 22
The inspection process
Inspectionmeeting
Individualpreparation
Overview
Planning
Rework
Follow-up
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 23
Inspection procedure
l System overview presented to inspection team
l Code and associated documents aredistributed to inspection team in advance
l Inspection takes place and discovered errorsare noted
l Modifications are made to repair discoverederrors
l Re-inspection may or may not be required
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 24
Inspection teams
l Made up of at least 4 members
l Author of the code being inspected
l Inspector who finds errors, omissions andinconsistencies
l Reader who reads the code to the team
l Moderator who chairs the meeting and notesdiscovered errors
l Other roles are Scribe and Chief moderator
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 25
Inspection checklists
l Checklist of common errors should be used todrive the inspection
l Error checklist is programming languagedependent
l The 'weaker' the type checking, the larger thechecklist
l Examples: Initialisation, Constant naming, looptermination, array bounds, etc.
Inspection checks
Fault class Inspection checkData faults Are all program variables initialised before their values
are used?Have all constants been named?Should the lower bound of arrays be 0, 1, or somethingelse? Should the upper bound of arrays be equal to the size ofthe array or Size -1?If character strings are used, is a delimiter explicitlyassigned?
Control faults For each conditional statement, is the condition correct?Is each loop certain to terminate?Are compound statements correctly bracketed?In case statements, are all possible cases accounted for?
Input/output faults Are all input variables used?Are all output variables assigned a value before they areoutput?
Interface faults Do all function and procedure calls have the correctnumber of parameters?Do formal and actual parameter types match? Are the parameters in the right order? If components access shared memory, do they have thesame model of the shared memory structure?
Storage managementfaults
If a linked structure is modified, have all links beencorrectly reassigned?If dynamic storage is used, has space been allocatedcorrectly?Is space explicitly de-allocated after it is no longerrequired?
Exceptionmanagement faults
Have all possible error conditions been taken intoaccount?
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 27
Inspection rate
l 500 statements/hour during overview
l 125 source statement/hour during individualpreparation
l 90-125 statements/hour can be inspected
l Inspection is therefore an expensive process
l Inspecting 500 lines costs about 40 man/hourseffort = £2800
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 28
Automated static analysis
l Static analysers are software tools for source textprocessing
l They parse the program text and try to discoverpotentially erroneous conditions and bring theseto the attention of the V & V team
l Very effective as an aid to inspections. Asupplement to but not a replacement forinspections
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 29
Static analysis checksFault class Static analysis check
Data faults Variables used before initialisationVariables declared but never usedVariables assigned twice but never usedbetween assignmentsPossible array bound violations Undeclared variables
Control faults Unreachable codeUnconditional branches into loops
Input/output faults Variables output twice with no interveningassignment
Interface faults Parameter type mismatchesParameter number mismatchesNon-usage of the results of functionsUncalled functions and procedures
Storage managementfaults
Unassigned pointersPointer arithmetic
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 30
Stages of static analysis
l Control flow analysis. Checks for loops withmultiple exit or entry points, finds unreachablecode, etc.
l Data use analysis. Detects uninitialisedvariables, variables written twice without anintervening assignment, variables which aredeclared but never used, etc.
l Interface analysis. Checks the consistency ofroutine and procedure declarations and theiruse
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 31
Stages of static analysis
l Information flow analysis. Identifies thedependencies of output variables. Does notdetect anomalies itself but highlightsinformation for code inspection or review
l Path analysis. Identifies paths through theprogram and sets out the statements executed inthat path. Again, potentially useful in the reviewprocess
l Both these stages generate vast amounts ofinformation. Must be used with care.
LINT static analysis
138% more lint_ex.c
#include <stdio.h>printarray (Anarray) int Anarray;{ printf(“%d”,Anarray);}main (){ int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ;}
139% cc lint_ex.c140% lint lint_ex.c
lint_ex.c(10): warning: c may be used before setlint_ex.c(10): warning: i may be used before setprintarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) ::lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) ::lint_ex.c(11)printf returns value which is always ignored
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 33
Use of static analysis
l Particularly valuable when a language such as Cis used which has weak typing and hence manyerrors are undetected by the compiler
l Less cost-effective for languages like Java thathave strong type checking and can thereforedetect many errors during compilation
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 34
l The name is derived from the 'Cleanroom'process in semiconductor fabrication. Thephilosophy is defect avoidance rather thandefect removal
l Software development process based on:• Incremental development
• Formal specification.
• Static verification using correctness arguments
• Statistical testing to determine program reliability.
Cleanroom software development
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 35
The Cleanroom process
Constructstructuredprogram
Definesoftware
increments
Formallyverifycode
Integrateincrement
Formallyspecifysystem
Developoperational
profileDesign
statisticaltests
Testintegrated
system
Error rework
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 36
Cleanroom process characteristics
l Formal specification using a state transitionmodel
l Incremental development
l Structured programming - limited control andabstraction constructs are used
l Static verification using rigorous inspections
l Statistical testing of the system (covered in Ch.21).
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 37
Incremental development
Formalspecification
Develop s/wincrement
Establishrerquirements
Deliversoftware
Frozenspecification
Requirements change request
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 38
Formal specification and inspections
l The state based model is a system specificationand the inspection process checks the programagainst this model
l Programming approach is defined so that thecorrespondence between the model and thesystem is clear
l Mathematical arguments (not proofs) are used toincrease confidence in the inspection process
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 39
l Specification team. Responsible for developingand maintaining the system specification
l Development team. Responsible fordeveloping and verifying the software. Thesoftware is NOT executed or even compiledduring this process
l Certification team. Responsible for developinga set of statistical tests to exercise the softwareafter development. Reliability growth modelsused to determine when reliability is acceptable
Cleanroom process teams
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 40
l Results in IBM have been very impressive withfew discovered faults in delivered systems
l Independent assessment shows that theprocess is no more expensive than otherapproaches
l Fewer errors than in a 'traditional' developmentprocess
l Not clear how this approach can be transferredto an environment with less skilled or lesshighly motivated engineers
Cleanroom process evaluation
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 41
Key points
l Verification and validation are not the same thing.Verification shows conformance withspecification; validation shows that the programmeets the customer’s needs
l Test plans should be drawn up to guide the testingprocess.
l Static verification techniques involve examinationand analysis of the program for error detection
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 42
Key points
l Program inspections are very effective indiscovering errors
l Program code in inspections is checked by asmall team to locate software faults
l Static analysis tools can discover programanomalies which may be an indication of faults inthe code
l The Cleanroom development process depends onincremental development, static verification andstatistical testing