Date post: | 03-Oct-2014 |
Category: |
Documents |
Upload: | phuoc-thinh-tran |
View: | 61 times |
Download: | 2 times |
QuảnQuản lýlý chấtchất lượnglượng phầnphần mềmmềmSoftware Quality ManagementSoftware Quality Management
GV: Nguyễn Ngọc TúGV: Nguyễn Ngọc Tú
Email: [email protected]: [email protected]
Are we building the right software for the need ?
Are we building the software right ?
DEFECT CLASSIFICATION AND DEFECT CLASSIFICATION AND ANALYSIS ANALYSIS SQSSQS –– SQMSQM –– SQASQA –– SRESRE
Chapter 09
Hoa Sen University 2NNTu - SQM W2009 HoaSen University
“The software is done.We are just trying to get it to work”
Unknown
2
AIMSAIMS
� (actual/potential) defect� or quality� in
current and future products.
HoaSen University 3NNTu - SQM W2009
ContentsContents
� General Types of Defect Analyses
– Defect distribution analysis
� What: Distribution over defect types
� Where: Distribution over defect locations
� General observations about defect distribution
– Defect trend analysis and defect dynamics model
– Defect causal analysis
� DEFECT CLASSIFICATION AND ODC
– ODC concepts
– Defect classification using ODC: A comprehensive example
– Adapting ODC to analyze web errors
� DEFECT ANALYSIS FOR CLASSIFIED DATA
– One-way analysis: Analyzing a single defect attribute
– Two-way and multi-way analysis: Examining cross-interactions
HoaSen University 4NNTu - SQM W2009
IntroductionIntroduction
� Defect CA
– help both developers and testers to detect and remove potential defects
– help other project personnel to improve the development process, to prevent injection of similar defects and to manage risk better
� defect data
– from the main QA activities
HoaSen University 5NNTu - SQM W2009
General Types of Defect General Types of Defect AnalysesAnalyses
� A defect is discovered,
– various individual analyses can be performed.
� defect data are accumulated over time
– collective analyses can be performed
� General defect analyses:
– Questions: what/where/when/how/why?
– Distribution/trend/causal analyses.
HoaSen University 6NNTu - SQM W2009
General Types of Defect General Types of Defect Analyses Analyses [2][2]
� Analyses of classified defect data:
– Prior: defect classification.
– Use of historical baselines.
– Attribute focusing in 1-way and 2-way analyses.
– Tree-based defect analysis
HoaSen University 7NNTu - SQM W2009
General Types of Defect General Types of Defect Analyses Analyses [3][3]
� What– The identification and classification of the discovered defects can be
performed to identify what they are and classify them by some
consistent scheme
� Where– Where was the defect found or discovered? This information can be
used to provide valuable feedback to the development process through
defect distribution analysis.
� When– The identification of the exact time or associated development phase or
subphase when a defect is injected and when it is discovered is
important, because it provides information to analyze the overall defect
trend and serves as the basis for quality prediction into the future.
HoaSen University 8NNTu - SQM W2009
General Types of Defect General Types of Defect Analyses Analyses [4][4]
� Pre- or Post release– An important extension to the “when” question is whether a defect is a
pre-release defect or a post-release defect
� How and Why– How was the defect injected into the software, and why? These two
questions are closely related, both pertaining to the cause of the
discovered defects
HoaSen University 9NNTu - SQM W2009
Defect distribution analysisDefect distribution analysis
� can help us answer the what and where
questions– distribution of defects over different defect types
– can find out the distribution of defects over different areas or product
components
� typically deal with faults or defect fixes
instead of failures or errors
� Defect fixes– is typically used if actual fixing of discovered problem took place before
defect analyses were performed
HoaSen University 10NNTu - SQM W2009
Defect distribution analysisDefect distribution analysis
Type Description #failures
A permission denied 2079
B no such file or directory 14
C stale NFS file handle 4
D client denied by server configuration 2
E file does not exist 28,631
F invalid method in request 0
G invalid URL in request connection 1
H mod_mime_magic 1
I request failed 1
J script not found or unable to start 27
K connection reset by peer 0
All types 30,760
HoaSen University 11NNTu - SQM W2009
http://lyle.smu.edu/~tian/class/8317.09f/webM.pdf
What: Distribution over defect What: Distribution over defect typestypes
Type Errors %
.gif 12489 43.62
.class 4913 17.16
directory 4425 15.46
.html 3656 12.77
.jpg 1323 4.62
other 394 1.38
All 28631 100
HoaSen University 12NNTu - SQM W2009
Table 20.2 Characterizing web errors by file types
Where: Distribution over Where: Distribution over defect locationsdefect locations
DF= 0 1 2 3 4 5 6 7 8 9 10-19 20-
37
all
module
#
771 174 102 63 31 29 23 25 16 7 50 14 1295
% 58.8 13.4 7.9 4.9 2.4 2.2 1.8 1.9 1.2 0.5 3.9 1.1 100
DFsum 0 174 204 189 124 145 138 175 128 63 673 417 2367
% 0 7.4 8.6 8 5.2 6.1 5.8 7.4 5 2.7 28.4 17.
6
100
HoaSen University 13NNTu - SQM W2009
Table 20.3 Distribution of DF for a commercial product LS
General observations about General observations about defect distributiondefect distribution
DF= 0 1 2 3 4 5 6 7 8 9 10-19 20-49 >50 all
modul
e#
23 13
1
11
2
12
0
99 94 68 50 38 32 147 68 13 995
% 2.3 13.
2
11 12 9.9 9.4 6.8 5 3.8 3.2 14.8 6.8 1.3 100
DFsum 0 13
1
22
4
36
0
39
6
47
0
40
8
35
0
30
4
28
8
2109 2040 910 7824
% 1.6
7
2.8
6
4.6 5.1 6 5.2 4.5 3.9 3.7 3.1 26.96 26.07 11.6
3
100
HoaSen University 14NNTu - SQM W2009
Defect trend analysis and Defect trend analysis and defect dynamics modeldefect dynamics model
� defect data contains some timing
information
– Minimum: pre-release or post-release
HoaSen University 15NNTu - SQM W2009
Injection
phase
Removal Phase
req. spec. design coding testing post-rel all
phases
requirement 10 22 8 0 5 2 47
specification 10 20 2 0 1 33
design 52 120 32 5 209
coding 198 320 46 564
testing 58 7 65
post-release 2 2
all phases 10 32 80 320 415 63 920
Defect causal analysisDefect causal analysis
� two forms
– logical analysis
� is a deterministic analysis that examines the logical
link between the effects and the corresponding
causes, and establishes general causal relations
– statistical analysis
� is a probabilistic analysis that examines the
statistical link between causes and effects and
deduces the probable causal relations between the
two
HoaSen University 16NNTu - SQM W2009
Defect causal Defect causal analysis analysis [2][2]
� The effects
– the observed failures or discovered (or fixed) faults
� the corresponding causes
– are the faults that caused the failures or the errors
that caused the injection of the faults, respectively.
� causal relations
– are determined by the developers or code owners
– faults are typically determined through dedicated
defect causal analysis
HoaSen University 17NNTu - SQM W2009
Defect causal Defect causal analysis analysis [3][3]
� Root cause analysis
– is human intensive, and should be performed by experts with thorough knowledge about the product, the development process, the application domain, and the general environment
� Gilb inspection
� for all the critical defects
HoaSen University 18NNTu - SQM W2009
Defect causal Defect causal analysis analysis [4][4]
� Statistical analysis
– is based on empirical evidence collected either locally or from other similar projects
– can be fed to various models to establish the predictive relations between causes and effects
� employ various statistical models
– correlation analysis
– Eg. the number of defects per module may be closely
correlated to module control flow complexity
HoaSen University 19NNTu - SQM W2009
DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC
� various detailed information can be collected
and recorded regarding the problems or the
defects
� part of this information
– is usually derived from explicit or implicit root cause analysis
� Such information
– can be organized in a systematic way for further analyses
� more valuable and specific feedback: use statistical models
HoaSen University 20NNTu - SQM W2009
DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC [2][2]
� The systematic classification and analysis
of defect data bridge the gap between
causal analysis and statistical quality
control, and provide valuable in-process
feedback to the development or
maintenance process and help assure and
improve product quality
HoaSen University 21NNTu - SQM W2009
DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC [3][3]
� Orthogonal defect classification, or ODC,
developed initially at IBM (Chillarege et al.,
1992)
– is the most influential among general frameworks for software defect classification and analysis.
� to identify problematic areas, and to
improve overall software product quality
HoaSen University 22NNTu - SQM W2009
ODCODC
� Key elements of ODC– Aim: tracking/analysis/improve
– Approach: classification and analysis
– Key attributes of defects
– Views: both failure and fault
– Applicability: inspection and testing
– Analysis: attribute focusing
– Need for historical data
HoaSen University 23NNTu - SQM W2009
ODCODC conceptsconcepts
� ODC has a rich and extensive category of
defect attributes, stemming from both the
failure view and the fault view
� The attributes (8) – related to the former are typically completed by
software testers or inspectors who initially observed
problems and opened defect reports; while those
related to the latter are typically completed by the
software developers or system maintenance
personnel who fixed the reported problems and
updated the corresponding defect reports
HoaSen University 24NNTu - SQM W2009
ODCODC
� 8 attributes– Activity
� refers to the actual process step (code inspection, function
test, etc.) when defects are discovered.
– Trigger
� describes the environment or condition that had to exist to
expose the defects.
– Impact
� refers to either perceived or real impact on users.
– Target
� represents the high-level identity (i.e., design, code, ID, etc.)
of the entity that was fixed.
HoaSen University 25NNTu - SQM W2009
http://www.ibm.com/developerworks/rational/library/aug06/gu/index.htmlODC v5.11, IBM Center for Software Engineering
ODCODC
� 8 attributes– Type
� represents the nature of the actual correction that was made.
– Qualifier
� specifies whether the fix that was made was due to missing,
incorrect, or extraneous code or information.
– Source
� indicates whether the defect was found in code written in-
house, reused from a library, ported from one platform to
another, or outsourced to a vendor.
– Age
� identifies the history of the target (i.e., design, code, ID, etc.)
that had the defect.
HoaSen University 26NNTu - SQM W2009
http://www.ibm.com/developerworks/rational/library/aug06/gu/index.html
ODCODC conceptsconcepts
� failure view and information collected at
defect discovery – Defect impact,
� with attribute values covering functionality, reliability, etc.
– Defect trigger,
� with attribute values corresponding to the specific types of
testing
– Defect severity,
� with commonly used attribute values: critical, major, minor, or
or inspection activities or scenarios that triggered the defect
detection. some numerical scale.
HoaSen University 27NNTu - SQM W2009
ODCODC conceptsconcepts
� fault view collected at defect fixing– Defect type,
� with attribute values: function, interface, algorithm, timing,
etc.
– Number of lines changed for the fixing.
� Some additional causal analyses – Defect source,
� with attribute values: vendor code, new code, base code, etc.
– Where the defect was injected,
� located to subsystems, modules, or components.
– When the defect was injected,
� typically identified with the development phase.
HoaSen University 28NNTu - SQM W2009
Defect classification using Defect classification using ODCODC: : A comprehensive exampleA comprehensive example
� various defect attribute data according to
ODC were collected in the system testing
stage
� a defect is detected,
– a formal report (called Problem Tracking Report or PTR in IBM) is recorded and tracked until its final resolution
HoaSen University 29NNTu - SQM W2009
Defect classification using Defect classification using ODCODC: : A comprehensive A comprehensive example example [2][2]
Label Name Possible Values or Categories & Labels
imp impact
c=capability, im=implementation, in=installation, ma=maintenance,
mi=migration, p=performance, r=reliability, sec=security, ser=service,
std=standard, u=usability
t r i g trigger
i=installation, m=migration, s=stress, b=backup, c=communications, f=file
ilo, co=coexistence, e=exception, hc=hlw config., sc=slw config., a=ad-hoc,
ss=startup/shutdown, o=normal operation
sev severity range from 1 (highest) to 4 (lowest) in severity
wk week week detected, counted from the start of the project
ftypefix
type
o=other product, s=specification, hld=high-level design,
lld=low-level design, c=code, b=build process
act action a=add, d=delete, c=change
srccode
source
b=base, v=vendor, n=new, c=changed, i=incremental (added to old),
s=scaffolded, p=previous defect fix
injphase
injected
p=previous release, s=specification, hld=high-level design, Ild=low-level
design, c=coding, ut=unit test, ft=function test, st=system test, d=customer
usage
HoaSen University 30NNTu - SQM W2009
Defect classification using Defect classification using ODCODC: : A comprehensive A comprehensive example example [3][3]
� Defect impact – “If this defect is not fixed, how will it impact the customer?”
– Pre-defined impact categories (possible answers) include
performance, reliability, etc.
� Defect trigger categories – closely resemble test scenario classes used for managing the
testing process for this product.
� Defect severity – can be 1 (critical problem), 2 (major problem), 3 (minor problem),
and 4 (minor inconvenience).
� The week – when the defect is detected, counted from the start of the project.
HoaSen University 31NNTu - SQM W2009
Defect classification using Defect classification using ODCODC: : A comprehensive A comprehensive example example [4][4]
� The information collected at defect fixing – pertains to the actions taken by the developers to
locate, identify and correct the faults that caused
detected failures:
� Fix type: fix to design, code, etc.
� Number of lines changed for the fixing.
� Fix action: adding, deleting, or changing to design or code.
� Some simple causal analyses– Defect source: vendor code, new code, base code, etc.
– The development phase when the defect was injected: previous
release or waterfall like development phases in the current
release.
HoaSen University 32NNTu - SQM W2009
Adapting Adapting ODCODC to analyze web to analyze web errorserrors
� For web-based applications,
– ODC-like defect classification can be defined and relevant defect data can be extracted from existing web server logs for analysis (Ma and Tian, 2003)
� data collection is always a big hurdle that
requires developers and testers to devote
substantial time to analyze the defects and
report the findings
HoaSen University 33NNTu - SQM W2009
Adapting Adapting ODCODC to analyze web to analyze web errors errors [2][2]
� Attributes
– Defect impact� corresponds to web error type, which indicates what problem was
experienced by web users. It can be analyzed directly based on
information extracted from the error logs or from response code used in
web access logs (Kallepalli and Tian, 2001).
– Defect trigger� corresponds to specific usage sequences or referrals that lead to
problems recorded in the error logs. It can be analyzed by examining the
referral pair information that can be extracted from the access logs (Ma
and Tian, 2003).
– Defect source� corresponds to specific files or file types that need to be changed,
added, or removed to fix problems recorded in the error logs. It can be
analyzed by examining both the specific errors and referral pairs.
HoaSen University 34NNTu - SQM W2009
Adapting Adapting ODCODC to analyze web to analyze web errors errors [3][3]
� Various other attributes
– can also be adopted or adapted from the original
ODC attributes through a close examination of the
web environment and data availability. Such
adaptation to different environments can help
people analyze problems or issues of concern to
them and fulfill different purposes.
HoaSen University 35NNTu - SQM W2009
DEFECT ANALYSIS FOR DEFECT ANALYSIS FOR CLASSIFIED DATACLASSIFIED DATA
� Various techniques
� most obvious and most straightforward analyses– are to apply defect distribution and trend analyses
� one-way analysis– examines one attribute at a time, either its overall distribution or its trend
over time
� Two-way analysis – can be used to examine the crossinteraction of two attributes (Bhandari
et al., 1993).
� Higher-order analysis is also possible, – such as using tree-based modeling on all the ODC attributes (Tian and
Henshaw, 1994).
HoaSen University 36NNTu - SQM W2009
expected defect profile
OneOne--way analysis: Analyzing a way analysis: Analyzing a single defect attributesingle defect attribute
� For each defect attribute,
– the overall distribution of its values can beexamined.
– distribution of defects among the differentdefect impact areas is very non-homogeneous
� When similar distribution data are available
over time or different development phases,
– can trace them to perform defect trend analysis
HoaSen University 37NNTu - SQM W2009
OneOne--way analysis: Analyzing a way analysis: Analyzing a single defect attributesingle defect attribute
HoaSen University 38NNTu - SQM W2009
OneOne--way analysis: Analyzing a way analysis: Analyzing a single defect attributesingle defect attribute
HoaSen University 39NNTu - SQM W2009
Figure 20.2 gives us the trend of type E errors for the SMU/SEAS web site over 26 days
TwoTwo--way and multiway and multi--way analysis: way analysis: Examining crossExamining cross--interactionsinteractions
� analysis examines the interaction between
two attributes, and can be applied to all
the attributes in pair-wise fashion
� simplest form
– is the conditional analysis of an individual
attribute under the condition of another attribute
taking a specific value.
HoaSen University 40NNTu - SQM W2009
TwoTwo--way and multiway and multi--way analysis: way analysis: Examining Examining crosscross--interactions interactions [2][2]
Impact Severity
1 2 3 4
Capability 2 12 13 1
Documentation 0 1 14 10
Installability 0 6 6 4
Maintainability 0 6 19 7
Migration 0 0 0 1
Performance 1 1 3 0
Reliability 27 96 66 7
Security 1 3 3 0
Service 0 0 4 4
Standards 0 1 2 1
Usability 0 10 44 19
HoaSen University 41NNTu - SQM W2009
ReferenceReference
� [1]Chapter 20
� [1]Chapter 13
� [3]Chapter 5
� [4]Chapter 04
� ODC - http://www.research.ibm.com/softeng/ODC/DETODC.HTM IBM
HoaSen University 42NNTu - SQM W2009
Q/AQ/A
HoaSen University 43NNTu - SQM W2009