22
Quality Assurance Involvement Compared to Program Results Jill A. Brooks Network Centric Systems
Quality Assurance Involvement Compared to Program Results Jill A. BrooksNetwork Centric Systems
10/25/2005 Page 2
Agenda• Introduction• Software Engineering Institute Insight• Raytheon North Texas Data
– Cost Performance– Schedule Performance– Quality
• Lessons Learned• Other Considerations• Next Steps
Quality Assurance Involvement Compared to Program Results
10/25/2005 Page 3
Introduction - Raytheon• Raytheon is an industry leader in defense and government electronics,
space, information technology, technical services, and business aviation and special mission aircraft
• Network Centric Systems (NCS) develops and produces mission solutions for networking, command and control, battlespace awareness, and air traffic management
• Space and Airborne Systems (SAS) provides electro-optic/infrared sensors, airborne radars, solid state high energy lasers, precision guidance systems, electronic warfare systems, and space-qualified systems for civil and military applications
• Raytheon-specific data examined for this presentation draws on both NCS and SAS programs executed in North Texas. Data is from software programs
10/25/2005 Page 4
Introduction – Raytheon continued• For NCS North Texas:
– 8 QE engineers– 145 Software Engineers– 30 Programs (including maintenance efforts)
10/25/2005 Page 5
Introduction – Raytheon continued
IPI CMM -based Internal Process Improvement AssessmentRTIS Raytheon/TI SystemsCMMI CMM Integrated
IPI CMM -based Internal Process Improvement AssessmentRTIS Raytheon/TI SystemsCMMI CMM Integrated
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
Software Sigma& Cycle Time
Software Sigma& Cycle TimeRTIS Policies &
ProceduresRTIS Policies &
Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
Level 4 IPI
Level 4 IPI
10x Fault DensityImprovementLevel 3
Level 2
BaldridgeAward
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
Level 3BaselineValidation
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 4
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
Software Sigma& Cycle Time
Software Sigma& Cycle TimeSoftware Sigma
& Cycle TimeSoftware Sigma
& Cycle TimeRTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
RTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
Level 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPI
10x Fault DensityImprovement
10x Fault DensityImprovementLevel 3Level 3
Level 2Level 2
BaldridgeAward
BaldridgeAward
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
Level 3BaselineValidation
Level 3BaselineValidation
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 2Repeatable
Level 2Repeatable
Level 3DefinedLevel 3Defined
198919911990
1992 1993 1994
198919891991199119901990
19921992 19931993 19941994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 4/5Managed/
Optimizing
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 200320002000
19951995 19961996 1997 1999-1997 1999-20012001 20022002 20032003
Level 4Level 4
IPI CMM -based Internal Process Improvement AssessmentRTIS Raytheon/TI SystemsCMMI CMM Integrated
IPI CMM -based Internal Process Improvement AssessmentRTIS Raytheon/TI SystemsCMMI CMM Integrated
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
Software Sigma& Cycle Time
Software Sigma& Cycle TimeSoftware Sigma
& Cycle TimeSoftware Sigma
& Cycle TimeRTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
RTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
Level 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPI
10x Fault DensityImprovement
10x Fault DensityImprovementLevel 3Level 3
Level 2Level 2
BaldridgeAward
BaldridgeAward
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
Level 3BaselineValidation
Level 3BaselineValidation
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 2Repeatable
Level 2Repeatable
Level 3DefinedLevel 3Defined
198919911990
1992 1993 1994
198919891991199119901990
19921992 19931993 19941994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 4/5Managed/
Optimizing
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 200320002000
19951995 19961996 1997 1999-1997 1999-20012001 20022002 20032003
Level 4Level 4
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
RTIS Integrated Product
Development Process
Software Sigma& Cycle Time
Software Sigma& Cycle TimeSoftware Sigma
& Cycle TimeSoftware Sigma
& Cycle TimeRTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
RTIS Policies & Procedures
RTIS Policies & Procedures
RTIS Software Operating
Instructions
RTIS Software Operating
Instructions
Level 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPI
10x Fault DensityImprovement
10x Fault DensityImprovementLevel 3Level 3
Level 2Level 2
BaldridgeAward
BaldridgeAward
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
• Self-assessment• Software Improvement
Team Formed
Level 3BaselineValidation
Level 3BaselineValidation
Level 2Repeatable
Level 3Defined
198919911990
1992 1993 1994
Level 2Repeatable
Level 2Repeatable
Level 3DefinedLevel 3Defined
198919911990
1992 1993 1994
198919891991199119901990
19921992 19931993 19941994
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 2003
Level 4/5Managed/
Optimizing
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 200320002000
19951995 19961996 1997 1999-1997 1999-20012001 20022002 20032003
Level 2Repeatable
Level 2Repeatable
Level 3DefinedLevel 3Defined
198919911990
1992 1993 1994
198919891991199119901990
19921992 19931993 19941994
Level 2Repeatable
Level 2Repeatable
Level 3DefinedLevel 3Defined
198919891991199119901990
19921992 19931993 19941994
198919891991199119901990
19921992 19931993 19941994
Level 4/5Managed/
Optimizing
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
RaytheonAcquisition & Transition
20001995 1996 1997 1999-
2001 2002 200320002000
19951995 19961996 1997 1999-1997 1999-20012001 20022002 20032003
Level 4/5Managed/
Optimizing
Level 4/5Managed/
Optimizing
RaytheonAcquisition & Transition
RaytheonAcquisition & Transition
2000200019951995 19961996 1997 1999-1997 1999-
20012001 20022002 2003200320002000
19951995 19961996 1997 1999-1997 1999-20012001 20022002 20032003
Level 4Level 4
Level 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPILevel 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPILevel 4 IPI
Level 4 IPILevel 4
IPILevel 4
IPILevel 4 IPI
Level 5CMMI
Level 4
Level 5
10/25/2005 Page 6
Introduction – The Burning Platform• Although the CMMI introduces Quality Assurance (QA) at
Level 2, and QA is further expanded at higher levels of maturity, QA functions still have to “prove” their worth as QA is often viewed as an “overhead” function
10/25/2005 Page 7
Software Engineering Institute (SEI) Insight• Quality Assurance is introduced at Level 2 of the Capability Maturity Model
Integrated (CMMI)• Quality activities are in all process areas• As organizations move up the maturity ladder, there are improvements in program
performance• SEI has both qualitative and quantitative data to support the previous statement
The SEI has collected data which illustrates the correlation between organizational maturity and
improved performance
Product Inspector
Reactive QC
Process Participant
Proactive QE
Process Facilitator
Process Analyst
Quantitative Analysis
Best Practices Consultant
Process deployment
Continuous improvement
Process Change Agent
Predictive assessment
Product Inspector
Reactive QC
Process ParticipantProcess Participant
Proactive QE
Process Facilitator
Process Analyst
Quantitative Analysis
Best Practices Consultant
Process deployment
Continuous improvement
Process Change Agent
Predictive assessment
10/25/2005 Page 8
SEI Insight• Performance results summarized by the Software
Engineering Institute, March 4, 2005
Reference: http://www.sei.cmu.edu/cmmi/results.html
13 : 1
55%
94%376%90%87%
High
3 : 1
14%
50%50%50%38%
Median
82 : 1Return on Investment
510%Customer Satisfaction
1629%Quality1311%Productivity1420%Schedule144.5%Cost
Number of Data Points
LowPerformance Improvement Category
10/25/2005 Page 9
Raytheon North Texas Data –Cost Performance
• QE Involvement is measured as a percentage of the total effort on the program
• Cost Performance Index (CPI) is measured as the ratio of the Budgeted Cost of Work Performed (BCWP) to the Actual Cost of Work Performed (ACWP)
BCWPACWP
CPI =
10/25/2005 Page 10
There is a positive correlation between QE Involvement and Program Cost (measured via CPI)
Less SQE Involvement More QE Involvement0.6
0.7
0.8
0.9
1
1.1
1.2
0 0.02 0.04 0.06 0.08 0.1 0.12 0.1Ove
r Bud
get
Und
er B
udge
tRaytheon North Texas Data –Cost Performance
10/25/2005 Page 11
Raytheon North Texas Data –Schedule Performance
• QE Involvement is measured as a percentage of the total effort on the program
• Schedule Performance Index (SPI) is measured as the ratio of the Budgeted Cost of Work Scheduled (BCWS) to the Actual Cost of Work Performed (ACWP)
BCWSACWP
SPI =
10/25/2005 Page 12
Raytheon North Texas Data –Schedule Performance
No apparent correlation between QE Involvement and Program Schedule (via SPI)
Less SQE Involvement More QE InvolvementBeh
ind
Sche
dule
A
head
of S
ched
ule
10/25/2005 Page 13
Raytheon North Texas Data – QualityDefect Containment
• QE Involvement is measured as a percentage of the total effort on the program
• Defect Containment (DC) is measured as the ratio of the number of defects which were detected “in phase” versus the total number of defects
In-phase DefectsTotal Number of Defects
DC =
10/25/2005 Page 14
Raytheon North Texas Data – QualityDefect Containment
There is a positive correlation between QE involvement and Defect Containment
Less SQE Involvement More QE Involvement
Low
er C
onta
inm
ent
H
ighe
r Con
tain
men
t
10/25/2005 Page 15
Raytheon North Texas Data – QualityDefect Density
• QE Involvement is measured as a percentage of the total effort on the program
• Defect Density (DD) is measured as the ratio of the number of defects which were detected post delivery versus the size of the product. Note the Equivalent Lines of Code was used to adjust for programs with significant amounts of legacy code
Post Delivery DefectsEquivalent Lines of Code
DD =
10/25/2005 Page 16
Raytheon North Texas Data – QualityDefect Density (DD)
There is a negative correlation between QE involvement and Defect Density, which is a good thing!
0
Less SQE Involvement More QE Involvement
Low
er D
D
Hig
her D
D
10/25/2005 Page 17
Lessons Learned
• Data, data, data
– Multiple data repositories
– The color of money
Level of granularity: QE sometimes counted as part of management, planning and control
QE may perform expanded role activities (non-traditional QE activities) which are sometimes counted in the QE “bucket”
10/25/2005 Page 18
Other Considerations• Execution of QE improved (QE productivity/efficiency)
– Don’t currently have a formal metric for this– Process has matured– QE staff has had very little attrition– Getting more “bang” for the QE buck?
QE productivity / efficiency is an opportunity for future analysis
10/25/2005 Page 19
Other Considerations• Customer value of QE involvement
– Don’t currently have a formal metric for this (have customer satisfaction scorecards, but it is not clear if these have the level of granularity required to examine customer perceived value of QE activities)
– QE involvement required by some contracts– QE often has established long-standing relationships with customers– Customers request QE participation in various activities
Customer Value of QE Involvement is another relationship to examine
10/25/2005 Page 20
Next Steps• Continuous Improvement continues…
– Data repository consolidation – NCS is moving towards standardized cost
collection system with increased granularity– Metrics are being standardized across
disciplines: Systems Engineering, Software Engineering, and Hardware Engineering
Although there is evidence that increased QE involvement has a positive impact on program success, there are opportunities for improvement of the data and
more analysis in the future!
10/25/2005 Page 22
Questions
