Software Quality Assurance

Introduction

What is software quality? How can it be measured?

How can it be measured before the software is delivered?

Some key quality factors Some measurable indicators of software

quality

Introduction Think of an everyday object

e.g. a chair How would you measure it’s “quality”?

construction quality? (e.g. strength of the joints,…) aesthetic value? (e.g. elegance,…) fit for purpose? (e.g. comfortable,…)

All quality measures are relative there is no absolute scale we can say A is better than B but it is usually hard to say how much better

For software: construction quality( 建造的品質 )?

software is not manufactured aesthetic value ( 美學上的價值 )?

but most of the software is invisible aesthetic value matters for the user interface, but is only a marginal concern

fit for purpose? Need to understand the purpose

McCall’s Quality Factors

operation

revi

sion transition

Correctness reliability usability integrity efficiency

MaintainabilityFlexibilityTestability

PortabilityReusabilityInteroperability

軟體品質因素

Measuring Quality

The Quality Concepts(abstract notions ofquality properties)

Measurable Quantities(define some metrics)

Counts taken fromDesign Representations(realization of the metrics)

usabilityusability

minutes taken for

some user task???

minutes taken for

some user task???

time taken to learn how to use?

time taken to learn how to use?

complexitycomplexity

count procedure

calls???

count procedure

calls???

information flow

between modules?

information flow

between modules?

reliabilityreliability

run it and count

crashes per hour???

run it and count

crashes per hour???

mean time to failure?

mean time to failure?

examples...

SQA: What is SQA?

SQA is the process of assuring people that every effort has been made to ensure that software products have the desired level of reliability, maintainability, usability, and salability.

SQA 是一種執行軟體評估與衡量的活動 (Baker and Fisher) Set of systematic activities providing evidence of the ability of the

software process to produce a software product that is fit to use (G. Schulmeyer and J. McManus, Software Quality Handbook, Prentice Hall, 1998.)

SQA and CMMI

Software (Process and Product ) quality assurance is the key process in Level 2, Managed.

The CMMI lists four goals that must be achieved to satisfy this key process area: Goal 1: Objectively Evaluate Processes and Work Products

Adherence of the performed process and associated work products and services to applicable process descriptions, standards, and procedures is objectively evaluated.

Goal 2: Provide Objective Insight Noncompliance issues are objectively tracked and communicated,

and resolution is ensured. Goal 3: Institutionalize a Managed Process Goal 4: Institutionalize a Defined Process

Software Quality Assurance

How do you assure the quality of your software? Good processes Good documentation and artifacts Accountability ( 可說明性 ) Learn and improve

The Objectives of SQA

Monitoring processes and products throughout the software development lifecycle to ensure the quality of the delivered product(s) Monitoring the processes

Provides management with objective feedback regarding process compliance to approved plans, procedures, standards, and analyses

Monitoring the products Focus on the quality of product within each phase of the SD

LC e.g., requirements, test plan, architecture, etc.

Objective: identify and remove defects throughout the lifecycle, as early as possible

SQA: An SQA Program

Minimizing number of defects in delivered s/w

Creating mechanisms for controlling software develo

pment and maintenance so that costs and schedule

s can be met

Making certain that the delivered product can be use

d in its intended marketplace

Improving the quality of future products

Software defects

Mistakes made at any point in the software process Requirements, design, coding, maintenance

Consequences Inconvenience, loss of service, financial loss, equi

pment damage, injury, death

The percentage of defects found by various methods

Personal checking of design documents

Informal group design reviews

Formal design specifications

Formal code inspections

Modeling or prototyping

Personal desk-checking of code

Unit testing (single routines)

Function testing (related routines)

Integration testing

Field testing

35%

40%

55%

60%

65%

40%

25%

35%

45%

50%

The truth of defects

1. The later in the life cycle that an error is detected the more expensive it is to repair.

2. Errors remain latent and are not detected until well after the stage at which they are made. 54% of errors detected after coding and unit testing. 45% of these errors were requirements and design errors.

3. There are numerous requirements errors. Estimates indicate that 56% of all errors are errors during

the requirements stage.

Relative Cost to Repair based on when it was found

Requirements - 1 time Design - 3-6 times Code - 10 times Unit test - 15-40 times System test - 30-70 times Field operation - 40-1000 times

When should quality assurance be done?

At every stage in the software process

Testing Process of exercising or evaluating a system b

y manual or automatic means to verify that it satisfies specified requirements or to identity differences between expected and actual results.

Testing

Begins as soon as requirements and specifications are written

>= 40% of total project effort Constructive activity Includes formal reviews and walkthroughs

Testing Objectives

Testing is a process of executing a program with the intent of finding an error.

A good test case is one that has a high probability of finding an as yet undiscovered error.

A successful test is one that uncovers an as yet undiscovered error.

Testing Limitations

Cannot determine whether software meets user’s needs; only whether it appears to conform to user’s requirements

Cannot show that a system is free of defects; only that it contains defects

Does not correct defects; only finds defects

The Process of SQA

定義品質需求 制定 SQA 計畫 需求評估

設計評估

測試評估

需求分析

設計

測試

評估客戶滿意需求程

度

measurement

feedback

feedback

功能與完整性

系統完整性與一致性

執行效率與正確性

在需求分析階段完成

SQA Planning

IEEE Std 730-2002 Standard for Software Quality Assurance Plans 12 pages

IEEE Guide for Software Quality Assurance Planning draft P730.2 87 pages

SQA Planning IEEE SQAP

Purpose (Section 1 of the SQAP) Reference documents (Section 2 of the SQAP) Management (Section 3 of the SQAP) Documentation (Section 4 of the SQAP) Standards, practices, conventions, and metrics (Section 5 of the SQAP) Reviews and audits (Section 6 of the SQAP) Test (Section 7 of the SQAP) Problem reporting and corrective action (Section 8 of the SQAP) Tools, techniques, and methodologies (Section 9 of the SQAP) Code control (Section 10 of the SQAP) Media control (Section 11 of the SQAP) Supplier control (Section 12 of the SQAP) Records collection, maintenance, and retention (Section 13 of the SQAP) Training (Section 14 of the SQAP) Risk management (Section 15 of the SQAP)

Contents of SQA Plan (sect 1&2)

Purpose list software covered state portion of software life cycle covered

Reference Documents complete list of documents referenced elsewhere

Sect 3 - Management

organization - depict structure of org. responsibilities

tasks tasks to be performed relationship between tasks and checkpoints sequence of tasks

responsibilities of each organizational unit

Sect 4 - Documentation

identify required documents state how documents will be evaluated minimum documents

SRS - Software Requirements Specification SDD - Software Design Description SVVP – S. Verification and Validation Plan SVVR - S. Verification and Validation Report User documentation - manual, guide SCMP – S. Configuration Management Plan

Verification and Validation

驗證 (Verification): " 我們是否正確的開發了產品？ "

軟體應該與規格相符 確認 (Validation):

"我們是否開發了對的產品？ " 軟體應該執行使用者真實的需求

驗證與確認著重於讓程式的缺失得以浮現 除錯著重於找出缺失並加以改正

S. Verification and Validation Plan

•測試規劃與建立測試程序的標準有關，而非著重於描述產品如何測試

Sect 5- Standards, Practices, Conventions and Metrics

Identify S,P,C,and M to be applied How compliance is to be monitored and assur

ed Minimum

documentation standards, logic structure standards, coding standards, testing standards

selected SQA product and process metrics

Sect 6 - Reviews and Audits

purpose define what reviews/audits will be done how they will be accomplished what further actions are required

Minimum Software Requirements Reviews Preliminary Design Review

evaluate technical adequacy of top-level design

Min Set of Reviews/Audits (cont)

Detailed Design Review acceptability of detailed designs

Software Verification and Validation Plan Review adequacy of planned verification and validation

Functional Audit all requirements in SRS have been met

Physical Audit software and documents are consistent and ready

In-Process Audit Managerial Reviews

Sect 7 - Test

All tests that are not included in SVVP

Sect 8 - Problem Reporting

Practices and Procedures for reporting, tracking, and resolving problems

Organizational responsibilities

Sect 9 - Tools, Techniques and Methodologies

identify the special software tools, techniques and methodologies purpose describe use

SQA: 6 SIGMA QUALITY

Sigma = “Standard Deviation”

Typical software has 3 to 4 defects per KLOC

6 Sigma = 3 to 4 defects per million lines of code

Average companies accept 99.98% quality = 4S

6 Sigma = 99.9999998% level of quality

SQA: 6 SIGMA QUALITY

Quality Improvements: 3Sigma to 4Sigma = 10 fold

4Sigma to 5Ssigma = 30 fold

5Sigma to 6Sigma = 70 fold

Best-in-Class companies in some industries operate

at 6Sigma (Airline = 6.4Sigma; 2.5M:1)

Software organizations need to assess this

SQA: Quality Software

People Process

ManagementDiscipline

SQA

SQA: Pursuing SQA

What organizations are doing Nothing (42%)

Slogans( 口號 ) - “Quality is Job One!” (4%)

Improved testing (24%)

Focus on defect prevention (20%)

Process Improvements (9%)

Other... (1%)

SQA: Software Reliability (MTBF)

Putnam’s Software Reliability Model

ERRORS

0

NUMBER

OF

T I M E

OperationalCapability

0

SQA: Pursuing SQA the Deming Way

Purpose includes improvement

Quality philosophy Eliminate mass inspection

s Award business based on

more than price Continuous improvement Institute OJT(On the job tr

aining) Institute Leadership

Drive out fear Break down barriers Eliminate slogans Eliminate numerical

quotas & goals Remove barriers to pride

of workmanship Institute education and

self-improvement Get everyone involved

SQA: Strategy by Yourdon

If management or customer says... Speed up testing...just say NO!

Don’t worry about a few bugs...just say NO!

We’ll pin down the specs later...just say NO!

Don’t worry, its just a beta version...just say NO!

I don’t care if there are bugs, get it out the door...just say

NO!

Summary Software quality generally means fitness for purpose

need to know what that purpose is… …what functions must it perform …what other properties must it have (e.g. modifiability, reliability,

usability…) Not all quality attributes can be measured during design

because quality is not an attribute of software in isolation but we can look for predictors

Reliability, efficiency, maintainability, usability are usually the four most important quality factors …although different authors give different lists

Modularity is often a good predictor of quality measure it by looking at cohesion and coupling