Reviews and Audits

byJohn J. Marciniak

Presenter: Dennis Uspensky

Overview

Reviews are a valuable tool Used in many industries and disciplines Provide assessment of:

Progress Used procedures Products of the project

Concentrate on the types applicable to SE

Origins

IBM Corporation Early ’70s Michael Fagan – Fagan Inspection Economic value: early defect detection

IBM reported 83% detection rate AT&T 92% detection rate

Reviews

Different forms and names: Formal reviews Walk-throughs Audits Inspections

Distinguishing characteristics: Purpose Scope Method

Reviews (contd.)

Type Scope Purpose MethodReview Usually

broadProject progress assessment of milestone completion

Ad hoc

Walk-through

Fairly narrow

Assess specific development products

Static analysis of products

Audit Narrow-to-broad

Check processes and products of development

Formal, mechanical and procedural

Inspection Narrow Assess specific development products

Non-interactive, fairly procedural, rigid

Scope

Wide range Entire project Design Singe document Etc.

No effect on the review procedure

Purpose

Auditing (e.g. procedure or product) Inspecting (e.g. document, design specs) Assessment (e.g. milestone completion) Etc.

Method

Also widely ranges Informal (free form) Semi-formal Formal (specific methodology)

Review Definition

IEEE Standard Glossary of SE Terminology: A process or meeting during which a work

product, or a set of work products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include code review, requirements review, and test readiness review.

This is the most general form

Review Formality Classifications

Formal Contractual requirement Rigid responsibilities of review participants

Scribe, moderator, etc.

Informal No contractual requirements Less rigorous approach

Internal Management Senior management within developing organization Progress assessment Specific issues

Review Formality Classifications (contd.)

Peer Conducted by peers Confined to a single product, component or code unit

Life Cycle Applications

Types and methods of reviews will normally be specified in the Software Development Plan or Program Management Plan. Some are dictated by a contract.

Reviews consist of three parts: Planning Review Conduct Post-Review

All three are very important for a successful review

Planning Phase

Stating purpose of the review Selecting and arranging participants Distribution of review materials

Provide ahead of time

Setting physical location Preparing an agenda

Conduct Phase

Keeping to the agenda Remember, the purpose of the review is to identify

the problems and assign action for their resolution, not fixing the problems themselves

Review leader/moderator must maintain control Scribe/recorder puts proceedings into written form

Post-Review Phase

Depends on the actions required Progress on AIs may be reported at the next

review Unsatisfactory results of a review may

require another one

Management Reviews

Participants are management team members Objective is to communicate progress,

provide recommendations and coordinate decisions Evaluation of progress Changing project direction Allocation of resources

Management Review - Roles

Leader Administration Review conduct

Reporter Project status Documentation of findings and actions to be taken

Team Member Participation

Management Review - Output

Management Review Report Participants Agenda and inputs to the review Review objectives Action items and their ownership Project status and list of issues Recommendations for further reviews

Walk-Throughs

Overall objective is to evaluate a specific element and provide evidence of conformance to specifications

Unique established objectives Detect, identify and describe element defects Examine alternatives and stylistic issues Provide a mechanism for collection of valuable

feedback, yet maintaining authority for making decisions

Walk-Through - Roles

Leader Administration Review conduct Usually the author

Scribe Writing down all comments pertaining to errors found,

omissions, suggestions, alternative approaches, or questions of style

Team Member Review all input materials prior to walk-through Participation

Walk-Through - Output

Walk-through report Participants Purpose of walk-through Statement of objectives List of noted deficiencies, omissions,

contradictions, and suggestions for improvement Recommendations made Follow-up walk-through suggestions

Formal Inspections

A variant of a structured walk-through Participants are peers/project staff Objective is to detect defects in the product Use a checklist that contains the types of defects

common to the nature of product being inspected Syntax errors Logical errors

Formal Inspection - Roles

Moderator Technical expert From outside of the project Planning and administration Inspection conduct (step-by-step walkthrough)

Preparer/developer/author Participant/inspector

Peers of preparer Familiar with project from technical point of view

Recorder/scribe

Formal Inspection – Output

Formal inspection report List of defects Categorized by severity Defects corrected after inspection; checked by moderator May call for re-inspection if product deemed

unsatisfactory Report data is used to compile a base for statistical

measure and to build an expertise base used in other similar projects

Audits

Objective is to provide an independent confirmation that product development and process execution adhere to standards, guidelines, specifications and procedures Software elements Processes for producing them Projects Quality programs

Objective audit criteria like contracts, standards, requirements or specifications are used for evaluation of element(s) being audited

Audits (contd.)

IEEE specific forms of audits Functional Configuration Audit (FCA) verifies:

Development of a configuration item has been completed successfully

Item achieved performance and functional attributesComplete documentation available

Physical Configuration Audit (PCA) verifies:Conformity to the technical documentation

Audit - Roles

Leader Organizes and directs efforts Coordinate preparation of audit report Audit conduct

Team member Understanding of organization being audited Product and process understanding Audit report preparation

Audit - Output

Audit report Audit identification Scope Conclusions Synopsis Follow-up Recommendations

Final audit report

Inspections

Limited form of an audit Similar procedures Narrow focus

Assess a degree of compliance by means of set guidelines and checklists

Very rigid structure, no “freewheeling”

Future Directions

Use of reviews becoming more blurred Mechanisms will not change, though Electronic techniques may enhance the

interactive nature of reviews (e.g. on-line) Possible real time changes

Summary

Several flavors of review types Number and types may be specified in the

management or software development plans Reasons for reviews

Assess progress or integrity of a product or process Gathering of data

Reviews provide a basic performance and assessment technique

Sources

Michael Fagan, "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal, 15, 3, 1976, pp. 182-211

John J. Marciniak, “Reviews and Audits”, Kaman Sciences Corporation, pp. 107-116

Don O’Neill, “Peer Reviews”, Encyclopedia of Software Engineering