ODCODCOrthogonal Defect ClassificationOrthogonal Defect Classification
Ram ChillaregeRam Chillarege
Tutorial 2000Tutorial 2000
[email protected]@chillarege.com(914) 739 2826(914) 739 2826
Table of ContentsTable of Contents
BackgroundBackgroundConceptConceptODC SpecificationODC SpecificationApplication & Case StudiesApplication & Case StudiesThe ODC PracticeThe ODC Practice
Chillarege 1-2
BackgroundBackground
What is ODC ?What is ODC ?
ODC is a measurement technology for Software Engineering - ODC is a measurement technology for Software Engineering - and the operative word is and the operative word is measurement.measurement.
It uses the defect stream, found in any software development It uses the defect stream, found in any software development activity as a source of information about two thingsactivity as a source of information about two things
The software product under developmentThe software product under developmentThe software process that is being used for developmentThe software process that is being used for development
This knowledge can then be used for a variety of software This knowledge can then be used for a variety of software engineering work -engineering work -
project management, prediction, analysis, assessmentproject management, prediction, analysis, assessmentquality control, close feedback loops, etc.quality control, close feedback loops, etc.research on better software engineering method, etc.research on better software engineering method, etc.
Chillarege 3-4
ODC SummaryODC Summary
ODC was invented circa 1990 and evolved over 10 yearsODC was invented circa 1990 and evolved over 10 yearsReaches around 3000 IBM engineers worldwide ('99)Reaches around 3000 IBM engineers worldwide ('99)Motorola, Telcordia, Nortel, Lucent, .. are also usersMotorola, Telcordia, Nortel, Lucent, .. are also users
Low implementation cost, very significant returnsLow implementation cost, very significant returnsReduces root cause analysis cost by a factor of 10xReduces root cause analysis cost by a factor of 10xCost savings of $100 M / year in one legacy productCost savings of $100 M / year in one legacy product
ODC/Butterfly models provide accuracy, and tradeoffsODC/Butterfly models provide accuracy, and tradeoffsPredict number, nature, and timing of field failuresPredict number, nature, and timing of field failuresProcess signatures help in-process managementProcess signatures help in-process management
ODC is a new foundation for software engineering methodsODC is a new foundation for software engineering methodsIs a fastpath to Is a fastpath to SEI level 3 and aboveSEI level 3 and aboveNew methods for test effectiveness, risk management, .. New methods for test effectiveness, risk management, ..
Defect analysis methods vary vastly in their Defect analysis methods vary vastly in their capability and costs tradeoffscapability and costs tradeoffs
CapabilityCapability
CostCost
Detailed Detailed RCARCA
Defect Rate CurvesDefect Rate Curves
ODCODC
Change in paradigm: Change in paradigm: semantics -->> semantics -->> measurementmeasurement
Chillarege 5-6
ODC in IBMODC in IBMThe evolution of a Software Engineering PracticeThe evolution of a Software Engineering Practice
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
Year
0
1
2
3
4
Th
ou
san
ds
En
gin
eers
Rea
ched Concept
Proof
Pilots
Scalable
Deployable
At concept and proof of concept - it is still and ideaAt concept and proof of concept - it is still and ideaPilot is when the main stream is engaged - say 10-25 peoplePilot is when the main stream is engaged - say 10-25 peopleRepeatability of the pilot experience is critical and signifies the Repeatability of the pilot experience is critical and signifies the beginnings of a practice - which may scalebeginnings of a practice - which may scaleDeployment implies professional implementationDeployment implies professional implementationThis evolution cycle can easily take between 5 and 15 yearsThis evolution cycle can easily take between 5 and 15 years
PoughkeepsiePoughkeepsie
San JoseSan Jose
Santa TeresaSanta Teresa
EndicottEndicott
TorontoToronto
AustinAustin
YamatoYamato
RaleighRaleigh
KinsgtonKinsgton
BethesdaBethesda
BoulderBoulder
BoeblingenBoeblingen
Boca RatonBoca Raton
IGS MadridIGS Madrid
IGS GaithersburgIGS Gaithersburg
IGS TampaIGS Tampa
IGS PortsmouthIGS Portsmouth
IGS DenverIGS Denver
ConceptsConcepts
Chillarege 7-8
ODC: How does it work ?ODC: How does it work ?
Type: how fixed ?
Trigger: catalyst ?
Source: what code ?
Impact: so what ?
Defect semantics captured by four attribute-value setsDefect semantics captured by four attribute-value setsIndependent of product, process, language, applicationIndependent of product, process, language, applicationRigorous models exploit multi-dimensional dataRigorous models exploit multi-dimensional dataInsights for developers, testers, and managersInsights for developers, testers, and managersDevelopment experience is communicable through ODC Development experience is communicable through ODC
Innovation : Turn the rich defect steam's knowledge into Innovation : Turn the rich defect steam's knowledge into a a quantifiablequantifiable information source information source
{{ Defect Stream Defect Stream
In software development - there are always defects -In software development - there are always defects - (changes (changes required in code and design)required in code and design)Defects are rich in information Defects are rich in information - "who done it to whom ?"- "who done it to whom ?"The defects appear to a statistician like a sampling of the The defects appear to a statistician like a sampling of the product over the process space product over the process space ODC converts the semantics into a quantifiable information ODC converts the semantics into a quantifiable information sourcesource
Chillarege 9-10
How does ODC turn defect semantics into a measurement ? How does ODC turn defect semantics into a measurement ?
0
1
2
3
4
5
FunctionAssignmentInterfaceTiming
StartStart ReleaseRelease
The The defect type defect type dd istribution tells us "where we are"istribution tells us "where we are"And the data is actionable to recommend corrective decisionsAnd the data is actionable to recommend corrective decisions
01A
pr87
01Ju
l87
01O
ct87
01Ja
n88
01A
pr88
01Ju
l88
01O
ct88
01Ja
n89
01A
pr89
01Ju
l89
01O
ct89
01Ja
n90
01A
pr90
01Ju
l90
01O
ct90
Date
0100200300400500600700800900
PTMSPeriod 0 Period 1 Period 2 Period 3
Product Example 1:Product Example 1:Hindsight analysis of the development defects in a component of a high-end Hindsight analysis of the development defects in a component of a high-end
operating system - approximately 70 KLOCoperating system - approximately 70 KLOC
Chillarege 11-12
Period 0 Period 1 Period 2 Period 30
10
20
30
40
50
60
Per
cent
age
Funct ion Defect Type
Period 0 Period 1 Period 2 Period 30
5
10
15
20
25
30
35
Per
cent
age
Assignment Defect Type
Period 0 Period 1 Period 2 Period 30
5
10
15
20
25
Per
cent
age
In ter face Defect Type
Period 0 Period 1 Period 2 Period 30
5
10
15
20
25
Per
cent
age
Checking Defect Type
Product Example 1:Product Example 1:Signature revealed by ODC analysisSignature revealed by ODC analysis
Activity/Trigger
Interaction
Classifying by AttributesTracking
Open
FixClosePriority
Etc.
How defect detected Customer ViewImpact
QualifierMissing, Incorrect, Extraneous?
AgeHistory
Feedback on Development Process
Feedback on Verification Process
Target/TypeWhat was fixed
Multi-dimensional semantic data relating cause and effectUniform across developmentConsistent across products, labs, etc.
SourceWhere defect located
Orthogonal Defect ClassificationOrthogonal Defect Classification
Chillarege 13-14
2Q 4Q 6Q 8Q
0
20
40
60
80
HW ConfigStress
SW ConfigRecovery
Startup
Trigger by Quarter after Release
Example Triggers and their distribution Example Triggers and their distribution Subgroup of field found defects (ST Triggers) Subgroup of field found defects (ST Triggers)
Points to notice:Points to notice:
Changing DistributionChanging DistributionSignature of Customer Signature of Customer UsageUsageMeasures maturityMeasures maturity
Quiz:Quiz:Does it meet the Does it meet the criterion for ODC ?criterion for ODC ?
What makes a defect classification What makes a defect classification ODC - Orthogonal Defect Classification ?ODC - Orthogonal Defect Classification ?
Few fundamental requirements: on the Attribute - Value setFew fundamental requirements: on the Attribute - Value set
IndependentIndependentEach of the value classifications need to be as independent Each of the value classifications need to be as independent of the other as possible - since, they are semantic of the other as possible - since, they are semantic classifications there can be fuzz - but it should be clear.classifications there can be fuzz - but it should be clear.
Distribution ChangesDistribution ChangesA collection of defects should demonstrate a changing A collection of defects should demonstrate a changing pattern in their distribution as a function of processpattern in their distribution as a function of processAnd the value sets be collectively exhaustive - so that it can And the value sets be collectively exhaustive - so that it can explain the position of the productexplain the position of the product
Process InvarianceProcess InvarianceThe first two properties should be invariant across most of The first two properties should be invariant across most of the software development processes and practicesthe software development processes and practices
Chillarege 15-16
LLD
Code
Missing Interface/MessagesMissing Timing/SerializationMissing Relationship
Req
Missing CheckingIncorrect Algorithm/MethodMissing Assignment/Initiliazation
Incorrect Assignment/Init Incorrect Checking
HLD
Missing Function/Class
Incorrect Function/Class
Missing Algorithm/Method
Incorrect Interface/MessagesIncorrect Timing SerializationIncorrect Relationship
ODC Defect Types mapped to generic activity ODC Defect Types mapped to generic activity injecting defectsinjecting defects
CodeInspection
Unit Test
Function Test
SystemTest
DesignReview
Design Conf.
Logic/Data Flow
Backward Comp.
Lateral Comp.
Concurr. Language Dependency
Internal Doc.
SideEffect
Rare Sit.
Simple Path Complex Path
Coverage Variation Sequencing Interaction
Normal Mode
StartUp/Restart
Hardware Config
Software Config
Recovery/Exception
Workload/Stress
Sample Trigger-Trigger relationships to map escapesSample Trigger-Trigger relationships to map escapes
Chillarege 17-18
ODC
Captures the semantics of each defect and links the
distribution to process and product maturity - Thus, it levers the low cost of mathematical
methods with the power of qualitative
analysis
The spectrum of defect measurement and analysisThe spectrum of defect measurement and analysis
Statistical Defect ModelsStatistical Defect Models
Counting techniquesCounting techniquesComparison to Comparison to historical datahistorical dataGrowth Curve Growth Curve modelingmodeling
Not easily Not easily translatable into translatable into corrective actionscorrective actions
Causal Analysis Causal Analysis
Qualitative analysisQualitative analysis
Investigates the details Investigates the details of a few defectsof a few defects
Time consuming and Time consuming and expensiveexpensive
Has fewer restrictions on Has fewer restrictions on what can be analyzedwhat can be analyzed
Easy to translate to Easy to translate to individual actions, but individual actions, but hard to aggregatehard to aggregate
ODC SpecificationODC Specification
Chillarege 19-20
Definition: Defect TypeDefinition: Defect Type
The defect type is the meaning of the fixThe defect type is the meaning of the fix
It is not the cause of the bugIt is not the cause of the bugIt is what was needed to fix the problem - It is what was needed to fix the problem - From a programming perspective From a programming perspective Collectively the defect types describe the nature of workCollectively the defect types describe the nature of work
Definition: Defect TriggerDefinition: Defect Trigger
Trigger is the catalyst that helps a defect surfaceTrigger is the catalyst that helps a defect surface
Defects are latent by definition - until detectedDefects are latent by definition - until detectedTriggers - are not "cause", they are facilitatorsTriggers - are not "cause", they are facilitatorsThe The triggertrigger is the catalyst which surfaces the defect is the catalyst which surfaces the defect
trigger 1
trigger 2
trigger 3
trigger k
Fault Failure
Chillarege 21-22
System Test
RecoveryStart and RestartWorkload, StressHW ConfigurationSW ConfigurationBlocked Test (old
Normal mode)
Review Inspection
Backward compatibilityLateral compatibilityDesign conformanceConcurrencyLogic flowSide EffectsRare Situation
Function Test
CoverageVariationSequencingInteractionSimple PathCombination Path
Triggers - associated by typical activitiesTriggers - associated by typical activities
2Q 4Q 6Q 8Q
0
20
40
60
80
HW ConfigStress
SW ConfigRecovery
Startup
Trigger by Quarter after ReleaseThe MVS Field example:The MVS Field example:
Distribution specific to Distribution specific to product and captures product and captures customer usage customer usage Helps validate test plans Helps validate test plans against field usage against field usage The refinement yields The refinement yields Butterfly prediction Butterfly prediction > 90% accuracy> 90% accuracyFacilitates dynamic Facilitates dynamic staffing with appropriate staffing with appropriate skills to manage skills to manage workload.workload.
Examples of ways to exploiting the TriggerExamples of ways to exploiting the Trigger
Chillarege 23-24
ODC Version 5.1 Submitter InputODC Version 5.1 Submitter Input
InstallabilitySecurityPerformanceMaintenanceServiceabilityMigrationStandardsDocumentationUsabilityReliabilityCustomer ExpectationsCapability
Activity Trigger
Design Review/Code Inspection
Unit Test/Function Test
Design ConformanceUnderstanding FlowConcurrenceBackward CompatibilityLateral CompatibilityInternal DocumentLanguage DependencySide EffectsRare Situations
Simple Path Complex Path Coverage Variation Sequencing Interaction
Information Development
AccuracyAestheticsClarityConsistencyCompletenessOrganizationRetrievabilityStyleTask
System Test
Workload/StressRecoveryStartUp/RestartHW ConfigSW ConfigBlocked Test
{ {Black Box
White Box
Activity Trigger
Design ConformanceIcon/Widget AppearanceScreen Text/CharactersInput DevicesNavigationWidget/GUI Behavior
Graphical UserInterface
(code implications)
Graphical UserInterface
(online text content)
use Information Development Triggers
ODC Version 5.1 Submitter InputODC Version 5.1 Submitter Input
Chillarege 25-26
Target Defect Type
DesignCode
InformationDevelopment
Assignment-InitializationCheckingAlgorithm/MethodFunctionTiming/SerializationInterfaceRelationship
Editorial TechnicalNavigational
ReferenceTasksMessagesExamplesPresentationConcepts
Source Age
BaseNewRewrittenRefixed
Developed in HouseReused from LibraryPortedOutsourced
Content Type
ODC Version 5.1 Responder InputODC Version 5.1 Responder Input
MissingIncorrectExtraneous
Defect Qualifier
Target Defect Type Source Age
National LanguageSupport
TranslationCharacter HandlingUser InterfaceLanguage ExpectationEnablement
Process ConformanceCode IntegrationInstall DependenciesShipped filesPackaging ScriptsFeature Install DependenciesMedia
Build / Package
BaseNewRewrittenRefixed
Developed in HouseReused from LibraryPortedOutsourced
ODC Version 5.1 Responder InputODC Version 5.1 Responder Input
Chillarege 27-28
Inspector's ExperienceTriggers
New/Trained
WithinProject
WithinProduct
CrossProduct
Solution(CrossPlatform)
Design Conformance x xUnderstanding Flow x xInternal Document consistency/completeness
x
Backward Compatibility xLateral Compatibility x x
Concurrency x xLanguage Dependency xSide Effects x x xRare Situations x x x
Inspection triggers by (minimum) experienceInspection triggers by (minimum) experience
Tester's ExperienceTriggers
New/Trained
WithinProject
WithinProduct
CrossProduct
Solution(CrossPlatform)
White Box
Simple Path Coverage x x Complex Path Coverage x x Side Effects x x xBlack Box
Single-function coverage x x Single-function variation x x x Multi-function sequencing x x x Multi-function interaction x x x
Function test triggers by (minimum) Function test triggers by (minimum) experienceexperience
Chillarege 29-30
Tester's ExperienceTriggers
New/Trained
WithinProject
WithinProduct
CrossProduct
Solution(CrossPlatform)
Software Configuration x x x
Hardware Configuration x x x
Startup/Restart x x x x
Recovery x x x x
Workload/Stress x x x
Normal Mode x x x x x
System test triggers by (minimum) experienceSystem test triggers by (minimum) experience
ODC ReferencesODC References
ODC Standard definitions (e.g. ODC 5.1)ODC Standard definitions (e.g. ODC 5.1)
http://www.research.ibm.com/softenghttp://www.research.ibm.com/softeng
ODC Articles, presentations, reports, etcODC Articles, presentations, reports, etc
http://chillarege.com/odchttp://chillarege.com/odc
Chillarege 31-32
Target Source
Impact
Capability
Usability
Performance
Reliability
Maintenance
Integrity/Security
DocumentationStandards
TriggerInstallabilityDefect ID
Phase Found
Date Open
Date Closed
SeverityComponent
Type
Qualifier
MissingIncorrectExtraneous
RequirementsDesignCodeInformationUIBuildNLS
Target Specific
ActivitySpecific
Activity
Review/Inspection
Unit Test /Func TestSystem Test
Product Use Information
Target Specific
Su
bm
itter
Res
po
nd
erODC Defect AttributesODC Defect Attributes
Application & Application & Case Studies Case Studies
Chillarege 33-34
Process SignatureProcess SignatureA representation of the process, characterized by the A representation of the process, characterized by the normalized percent distribution of defect type and triggers normalized percent distribution of defect type and triggers by activity. Regardless of product, these normalized by activity. Regardless of product, these normalized distributions remain consistent as long as the process does distributions remain consistent as long as the process does not changenot change
Product ProfileProduct ProfileA description of characteristics associated with a specific A description of characteristics associated with a specific product, including (but not limited to) process (signature), product, including (but not limited to) process (signature), environment, function, Product Difficulty Index, and environment, function, Product Difficulty Index, and customer usagecustomer usageThe distribution of defects by ODC attributes, reflecting The distribution of defects by ODC attributes, reflecting expected or real volumes, across the process activities expected or real volumes, across the process activities and field discoveries and field discoveries
Signature and Profile distinctions Signature and Profile distinctions
Assign Check Alg Func Time Interf RelDesign FlowConcurrencyBack CompLat CompSide EffectsRare SitSimple PathComplex PathCoverageVariationSequencingInteractionSoftware ConfigHardware ConfigWorkloadStartupRecoveryBlocked /Normal
Principle Assocation - The Trigger powerPrinciple Assocation - The Trigger power
Chillarege 35-36
670
390
926
576
550
567
Inprocess
Field
0 100 200 300 400 500 600 700 800 900 1000
Number of Defects
Inspection Unit/FunctionTest System Test
Example 1 - Improvement Example 1 - Improvement Release 2Release 2
122
53
198
54
257
51
Inprocess
Field
0 50 100 150 200 250 300
Number of Defects
Inspection Unit/FunctionTest System Test
Example 1- Improvement Example 1- Improvement Release 6Release 6
Chillarege 37-38
Code size 1.4 MlocBase Inspection is the lowest investmentDistribution very different from customer usageHigh volume of test coverage and normal mode indicate incomplete or ineffective inspections
Example 1Example 1Release 2: In-process defects Release 2: In-process defects Activity by ODC TriggersActivity by ODC Triggers
Inspection
Function Test
System Test
Act
ivit
y
0100
200300
400500
600700
800900
1000
Number of Defects
Design ConformanceLogic FlowBackward CompatibilityLateral CompatibilityConcurrencySide EffectsRare SituationDocument ContentSimple PathComplex PathTest CoverageTest VariationTest SequenceTest InteractionHardware ConfigSoftware ConfigRecoveryStartup/RestartWorkloadNormal Mode
Field volumes are 75% of the in-process volume and they are both high.Many field escapes related to inspectionCustomer usage exposures should be mapped to skills (e.g. backward compatibility)Was there opportunity to test adequately, given removal of basic problemsCustomer usage indicates where test plans should be focused
Example 1Example 1Release 2: Field defects Release 2: Field defects ODC TriggersODC Triggers
Inspection
Function Test
System Test
Mo
st li
kely
act
ivit
y
0 200 400 600 800 1000
Number of Defects
Design ConformanceLogic FlowBackward CompatibilityLateral CompatibilityConcurrencySide EffectsRare SituationDocument ContentSimple PathComplex PathTest CoverageTest VariationTest SequenceTest InteractionHardware ConfigSoftware ConfigRecoveryStartup/RestartWorkloadNormal Mode
Chillarege 39-40
FinancialFinancialWarranty Cost reduced 30% each year (1990 - 1994)Warranty Cost reduced 30% each year (1990 - 1994)Development Cost reduced by 85%Development Cost reduced by 85%Number of licenses on new release doubled previous releaseNumber of licenses on new release doubled previous release
Defect and Problem ReductionsDefect and Problem ReductionsValid Unique Apars reduced by 97% from 1990 - 1994; continuous, Valid Unique Apars reduced by 97% from 1990 - 1994; continuous, significant improvement every releasesignificant improvement every releaseCustomer Reported Problems (DOPs and NDOPs) reduced by 85% from Customer Reported Problems (DOPs and NDOPs) reduced by 85% from 1990 - 1994; continuous, significant improvement every release1990 - 1994; continuous, significant improvement every releaseRate of defects discovered in-process decreased from 18/MLOC to Rate of defects discovered in-process decreased from 18/MLOC to 2/MLOC (improved prevention) 2/MLOC (improved prevention)
Customer SatisfactionCustomer Satisfaction Overall product satisfaction attained 100% satisfaction Overall product satisfaction attained 100% satisfaction Service satisfaction from 4Q92 - 4Q95 Service satisfaction from 4Q92 - 4Q95 VERY SAT from 15.1% to 32.8% VERY SAT from 15.1% to 32.8% Non-sat (includes neutral) from 27.4% to 16.1% Non-sat (includes neutral) from 27.4% to 16.1%
Example 1Example 1ResultsResults
RecoveryStart/Restart
Stresshw/sw Config
0
20
40
60
80
No ActionAction (P)Actual
Development Defects
RecoveryStart/Restart
Stresshw/sw Config
0
20
40
60
80
No ActionAction (P)Projected
Field Defects
Product Example 2:Product Example 2:Reducing warranty cost: ODC Triggers help identify Reducing warranty cost: ODC Triggers help identify issues that demand focus issues that demand focus
Standard practice would focus on system test with high expense and poor return. Precise development actions target specific triggers at negligible incremental cost and high return.
Chillarege 41-42
Product Example 2Product Example 2Feedback from ODC Trigger analysis yielded specific Feedback from ODC Trigger analysis yielded specific actions which the team owned actions which the team owned
Observations:Observations:incorrect serialization defects consistent in Ver 1/ Ver 2incorrect serialization defects consistent in Ver 1/ Ver 2Checking defects increased in Ver 1Checking defects increased in Ver 1GRS interfaces are defect freeGRS interfaces are defect free
Documentation defects in Ver 1Documentation defects in Ver 1Recommendations:Recommendations:
checkout timing & recovery in design and reviewscheckout timing & recovery in design and reviewsaccess Ver 2 System test plans for timing & recoveryaccess Ver 2 System test plans for timing & recovery
Committed Actions:Committed Actions:hands on FCTesting in parallel to ST in Feb/Marhands on FCTesting in parallel to ST in Feb/MarX starts FCT/ hands on using G Star for exposureX starts FCT/ hands on using G Star for exposureX will be using G Start in its ART testing X will be using G Start in its ART testing G will involve its self in a Joint Cust Ship during 2nd qtr G will involve its self in a Joint Cust Ship during 2nd qtr
Product Example 2: Product Example 2: Feedback from ODC Trigger analysis yielded specific Feedback from ODC Trigger analysis yielded specific actions which the team ownedactions which the team owned
Observations:Observations:incorrect serialization defects consistent in Ver 1/Ver 2incorrect serialization defects consistent in Ver 1/Ver 2Checking defects increased in Ver 2Checking defects increased in Ver 2GRS interfaces are defect freeGRS interfaces are defect free
Documentation defects in Ver 3Documentation defects in Ver 3Recommendations:Recommendations:
checkout timing & recovery in design and reviewscheckout timing & recovery in design and reviewsaccess Ver 3 System test plans for timing & recoveryaccess Ver 3 System test plans for timing & recovery
Committed Actions:Committed Actions:hands on FCTesting in parallel to ST in Feb/Marhands on FCTesting in parallel to ST in Feb/MarComp 1 starts FCT/ hands on using Comp 2 for exposureComp 1 starts FCT/ hands on using Comp 2 for exposureComp 1 will be using Comp 2 in its ART testing Comp 1 will be using Comp 2 in its ART testing Comp 3 will involve itself in a Cust test during 2nd qtr Comp 3 will involve itself in a Cust test during 2nd qtr
Chillarege 43-44
A B
Component
0
100
200
300
400
500
Def
ect
Co
un
t StandardsSecurity/IntegrityServiceabilityDocumentationInstallabilityReliabilityPerformanceUsabilityCapability
This chart uses the ODC attribute This chart uses the ODC attribute ImpactImpactIBM uses this attribute to capture IBM uses this attribute to capture customer satisfaction issues through customer satisfaction issues through customer surveys customer surveys Two components A and B haveTwo components A and B havevery different impact profilesvery different impact profilesCustomers found major deficiency in Customers found major deficiency in the delivered functions in A.the delivered functions in A.Customers found B not very reliable Customers found B not very reliable with considerable usability problems, with considerable usability problems, in addition to some functional in addition to some functional deficienciesdeficienciesIn conjunction with other ODC In conjunction with other ODC attributes, this analysis can point to attributes, this analysis can point to specific areas for targeted action to specific areas for targeted action to improve customerimprove customer satisfaction satisfaction
Product Example 3:Product Example 3:Customer reported defects in two components of two Customer reported defects in two components of two releases of a workstation operating system softwarereleases of a workstation operating system software
Ass
ign
men
tC
hec
kin
gA
lgo
rith
m/M
eth
oF
un
ctio
nIn
terf
ace
Tim
ing
/Ser
iali
zat
Rel
atio
nsh
ip
Defect Type
0
50
100
150
200
250
300
Def
ect
Co
un
t
IncorrectMissing
Defect Type vs Qualifier for Defect Type vs Qualifier for Component AComponent A
Over 50% of the Over 50% of the Function defects are Function defects are Missing pointing to Missing pointing to incomplete Design incomplete Design and/or Requirements and/or Requirements processprocessThe large fraction of The large fraction of Algorithm defects point Algorithm defects point to nonexistent or to nonexistent or inadequate low level inadequate low level designdesign
Product Example 3:Product Example 3:
Chillarege 45-46
Example 4Example 4Field defect escapes - analysis Field defect escapes - analysis
Analyzing field defect escapes is probably one of the most Analyzing field defect escapes is probably one of the most valuable exercises for entire development and service team.valuable exercises for entire development and service team.
There are different slices of data providing insights on There are different slices of data providing insights on different elements of the process, product and the customer different elements of the process, product and the customer setsetThe following examples illustrate some of themThe following examples illustrate some of them
This was an experienced software development team, but This was an experienced software development team, but new to networkingnew to networkingInitially the volume reduction progression seemed very Initially the volume reduction progression seemed very good. Inspection was heavy, then function and system. But good. Inspection was heavy, then function and system. But from an ODC standpoint it is deficient in the areas that are from an ODC standpoint it is deficient in the areas that are related to network products - Lateral Compatibility, related to network products - Lateral Compatibility, Interface, and SW Config - given the environment.Interface, and SW Config - given the environment.However looking at the field escapes it is apparent that the However looking at the field escapes it is apparent that the escapes are dominated by Interaction, Variation and escapes are dominated by Interaction, Variation and Software Config.Software Config.These triggers are indicative of The interfaces and These triggers are indicative of The interfaces and algorithms that need to be checked out better. Infact, the algorithms that need to be checked out better. Infact, the team knew that they lacked that skill in this domain. Also team knew that they lacked that skill in this domain. Also confirmed in the type chart confirmed in the type chart
Example 4Example 4Comparing Triggers between development and FieldComparing Triggers between development and Field
Chillarege 47-48
Example 4Example 4Evaluating Test EffectivenessEvaluating Test Effectiveness"Activity" versus "Trigger (In-Process) "Activity Most "Activity" versus "Trigger (In-Process) "Activity Most Likely to Have Found...." versus "Trigger" (Field)Likely to Have Found...." versus "Trigger" (Field)
0 50 100 150 200 250 300
Inspection
Function Test
System Test
0 50 100 150 200 250 300
Design ConformLogic FlowConcurrencyBack CompatLat CompatSide Effects
Rare SitCoverageVariationSequenceInteraction
Hardware ConfigSoftware ConfigWorkload StressRecoveryStartup/Restart
In-Process Field
Example 4Example 4Evaluating Product StabilityEvaluating Product Stability"Activity" versus "Type" (In-Process) "Activity Most Likely "Activity" versus "Type" (In-Process) "Activity Most Likely ....." versus "Type" (Field) ....." versus "Type" (Field)
0 50 100 150 200 250 300
Inspection
Function Test
System Test
0 50 100 150 200 250 300
AssignmentChecking
AlgorithmFunction
TimingInterface
Relationship
In-Process Field
Chillarege 49-50
Example 5Example 5Customer Usage:Customer Usage:Where is the impact, and what are the types ?Where is the impact, and what are the types ?
Capability
Usability
Performance
Reliability
Installability
Maintenance
Serviceability
Customer Require
0 10 20 30 40 50 60
AssignmentChecking
AlgorithmFunction
TimingInterface
Relationship
Question: Who do you think the audience(s) for this chart might be?
Algorithm seems to dominate. Which is indicative that the Algorithm seems to dominate. Which is indicative that the sort-of works is being enhanced or tuned - even for sort-of works is being enhanced or tuned - even for capability and satisfies new requirements.capability and satisfies new requirements.The type of triggers that drive algorithm in the field are: The type of triggers that drive algorithm in the field are: variation, workload and complex path. Ideally, this could variation, workload and complex path. Ideally, this could have been addressed through design and flow. If we looked have been addressed through design and flow. If we looked at the triggers, that will probably be the case.at the triggers, that will probably be the case.Note that Impact includes Customer Requirements - we are Note that Impact includes Customer Requirements - we are accepting requirements through the service streamaccepting requirements through the service stream
Example 5Example 5Customer Usage:Customer Usage:Where is the impact, and what are the types ?Where is the impact, and what are the types ?
Chillarege 51-52
Example 6Example 6Customer Usage ProfileCustomer Usage ProfileSeparating the age of the codeSeparating the age of the code
1Q 2Q 3Q 4Q 5Q 6Q 7Q 8Q0
5
10
15
20
25
30
35
New Base
Question: Who do you think the audience(s) for this chart might be?
Tells you that as the release matures, the failures are Tells you that as the release matures, the failures are dominated more by the old code than the new code.dominated more by the old code than the new code.
Implications:Implications:Service team should knowService team should know
A plan for warranty cost reduction A plan for warranty cost reduction
Example 6Example 6Customer Usage ProfileCustomer Usage ProfileSeparating the age of the codeSeparating the age of the code
Chillarege 53-54
1Q 2Q 3Q 4Q 5Q 6Q 7Q 8Q0
50
100
150
In-House Ported
Example 7Example 7Process EffectivenessProcess EffectivenessSimple, Over time Views "Source" over timeSimple, Over time Views "Source" over time
1Q 2Q 3Q 4Q0
50
100
150
200
250
300
In-Process Field
ported code was 25% of product - and volume of defects ported code was 25% of product - and volume of defects was a third) was a third) Error rate per failing module was double. Error rate per failing module was double. Ported code is an issue - particularly over time - and skillPorted code is an issue - particularly over time - and skillLittle investment in process - which is typical of a ported Little investment in process - which is typical of a ported code environmentcode environmentTherefore use the field triggers to best prioritize the next Therefore use the field triggers to best prioritize the next cycle of development. And since, inspection is usually out cycle of development. And since, inspection is usually out of the question - the module analyis .... etc. comes to play.of the question - the module analyis .... etc. comes to play.A few dozen modules that can be focused on it. A few dozen modules that can be focused on it.
Example 7Example 7Source vs Time: Inhouse and fieldSource vs Time: Inhouse and field
Chillarege 55-56
1Q 2Q 3Q 4Q 5Q 6Q 7Q 8Q0
20
40
60
80
100
Example 8Example 8Customer Usage Profile - Process EffectivenessCustomer Usage Profile - Process EffectivenessSimple, Over time Views "Type" over timeSimple, Over time Views "Type" over time
1Q 2Q 3Q 4Q0
10
20
30
40
50
60
70
80
90
100
AssignmentCheckingAlgorithm
FunctionTiming
InterfaceRelationship
In-ProcessField
Example 9Example 9Customer Usage Profile - Evaluating Product StabilityCustomer Usage Profile - Evaluating Product Stability"Trigger" versus "Qualifier (In-Process and Field)"Trigger" versus "Qualifier (In-Process and Field)
Missing Incorrect0
50
100
150
200
250
300
350
Missing Incorrect0
50
100
150
200
250
300
350
Startup/RestartRecoveryWorkload StressSoftware ConfigHardware ConfigInteraction
SequenceVariationCoverageRare SitSide Effects
Lat CompatBack CompatConcurrencyLogic FlowDesign Conform
In-Process Field
Chillarege 57-58
Example 9Example 9Customer Usage Profile - Evaluating Product StabilityCustomer Usage Profile - Evaluating Product StabilityStatic, Aggregate ViewsStatic, Aggregate Views"Impact" versus "Qualifier (In-Process and Field)"Impact" versus "Qualifier (In-Process and Field)
Missing Incorrect0
50
100
150
200
250
300
350
ServiceabilityDocumentationMaintainability
InstallabilityReliabilityPerformance
UsabilityCapability
Missing Incorrect0
50
100
150
200
250
300
350
ServiceabilityDocumentationMaintainabaility
InstallabilityReliabilityPerformance
UsabilityCapability
In-Process Field
350
3
23
143
1611
CapabilityUsabilityPerformanceReliabilityInstallabilityDocumentationServiceabilitySecurity/IntegrityStandards
Field defects in a Field defects in a component of Workstation component of Workstation OS show: OS show:
Impact inference: Impact inference: functional deficiencies are the major issue
Defect type inference: Defect type inference: Over 50% Missing Function indicates incomplete Design or Requirements Dominant algorithm defects indicate inadequate low level design
Example 10Example 10How to improve customer satisfaction ? How to improve customer satisfaction ? - use ODC Impact distribution to guide development focus- use ODC Impact distribution to guide development focus
Ass
ign
men
t
Ch
ecki
ng
Alg
ori
thm
/Met
ho
d
Fu
nct
ion
Inte
rfac
e
Tim
ing
/Ser
ializ
ati
Rel
atio
nsh
ip
0
50
100
150
200
250
300
Def
ect C
ount
IncorrectMissing
Chillarege 59-60
Product Success: > $100M annual savings Product Success: > $100M annual savings
A host product A host product 6 releases in 7 years 6 releases in 7 years 1.4 Millions lines of code to 4 M1.4 Millions lines of code to 4 MLarge reduction in resources Large reduction in resources
Standard methods plateaued with diminishing ROIStandard methods plateaued with diminishing ROI
Major effort, large investment, little informationMajor effort, large investment, little information~100 PY gave 4x , and hit a stone wall~100 PY gave 4x , and hit a stone wall
ODC and Butterfly changed the game to "smart methods"ODC and Butterfly changed the game to "smart methods"~5-15 PY gave additional 20x ~5-15 PY gave additional 20x
OverallOverall1729 defects/mloc down to 21 defects/mloc1729 defects/mloc down to 21 defects/mlocComparable code size (250 kloc/rel) with less peopleComparable code size (250 kloc/rel) with less people
The ODC PracticeThe ODC Practice
Chillarege 61-62
Understanding the priorities and objectives of a Practice Understanding the priorities and objectives of a Practice User versus the Owner helps distinguish the roles and User versus the Owner helps distinguish the roles and responsibilities that go with them.responsibilities that go with them.
A Practice User
A Practice Owner
Primary concern is the Primary concern is the product.product.A practice is only a means A practice is only a means to an end.to an end.Wants minimum ownership, Wants minimum ownership, maximum value.maximum value.Can manage Can manage implementation, but only if implementation, but only if needed to.needed to.
Primary concern is the Primary concern is the practice.practice.The application at a product The application at a product is only an engagement.is only an engagement.Develops and grows the Develops and grows the practice as a long term practice as a long term investment.investment.May manage May manage implementation to assure implementation to assure success. success.
A Practice Deployment ModelA Practice Deployment Model
Practice Owner Practice Owner Practice User Practice User
Practice Owner Site Champions
Local Test Group
Local Test Group
Local Test Group
Local Test Group
Site finally owns the implementationPractice owner should jump start a practiceBuild local competency - leading to a site practice champion
Step 2
Step 1
Chillarege 63-64
Evolution of a Software Engineering PracticeEvolution of a Software Engineering Practice. . . from an art form to a professional practice. . . from an art form to a professional practice
At concept and proof of concept - it is still and ideaAt concept and proof of concept - it is still and ideaPilot is when the main stream is engaged - say 10 peoplePilot is when the main stream is engaged - say 10 peopleRepeatability of the pilot experience is critical and signifies Repeatability of the pilot experience is critical and signifies the beginnings of a practicethe beginnings of a practiceDeployment implies professionally implementationDeployment implies professionally implementationThe cycle can easily take between 2 to 5 yearsThe cycle can easily take between 2 to 5 years
Concept Proof Pilots Scalable Deployable
For example, some numbers to reflect on:For example, some numbers to reflect on:
After 20 years: After 20 years:
Software Inspection (circa 1976)Software Inspection (circa 1976)Recognized as a significant impact on qualityRecognized as a significant impact on qualityMay be 10% of the code written is inspectedMay be 10% of the code written is inspected
Object Oriented Design & ProgrammingObject Oriented Design & ProgrammingWidely accepted as a good methodologyWidely accepted as a good methodologyLess than 20% of code written follows the methodology Less than 20% of code written follows the methodology truly. truly.
Regardless, these numbers do not diminish the significance of Regardless, these numbers do not diminish the significance of these practices.these practices.
Chillarege 65-66
ODC is a very efficient way of managing defects (in-process or field) and ODC is a very efficient way of managing defects (in-process or field) and actions at a minimal cost actions at a minimal cost
Low cost: Takes only two minutes to classify a defectLow cost: Takes only two minutes to classify a defectIn contrast Causal analysis takes approximately 1 hour per defect In contrast Causal analysis takes approximately 1 hour per defect
Provides the bedrock to build models for defect projection algorithmsProvides the bedrock to build models for defect projection algorithmsUsing Trigger and Defect Type/Qualifier, it is possible to predict the Using Trigger and Defect Type/Qualifier, it is possible to predict the number of field defects in commercial software with a very high accuracy number of field defects in commercial software with a very high accuracy (> 90%)(> 90%)
Analysis of defects becomes a natural part of software development Analysis of defects becomes a natural part of software development developers and testers obtain technical feedback on projectsdevelopers and testers obtain technical feedback on projectsrelease managers can assess risk better to make decisionsrelease managers can assess risk better to make decisionsclosing feedback loops become a natural part of project managementclosing feedback loops become a natural part of project managementODC is a fast path to reach SEI level 3, or higherODC is a fast path to reach SEI level 3, or higher
Currently, ODC used by several IBM labs and software companies Currently, ODC used by several IBM labs and software companies
SummarySummary
Chillarege 67-68
