Date post: | 15-Dec-2015 |
Category: |
Documents |
Upload: | alondra-tippen |
View: | 217 times |
Download: | 1 times |
The Statistical Analysis of the Impact of Adequate Testing on the Defense
Systems Development
Zaw P. TunGWU, EMSE PhD Candidate
Dr. Shahram Sarkani, Ph.D., P.E.Dr. Thomas Mazzuchi, D.Sc.
April 18, 2023
Overview Purpose Background Focus Area Data Extraction Source of Data Proposed Methodology Conclusion Future Work
Introduction Purpose
• Analyze the cost growth trend to determine if adequate testing plays an essential role in successful development of new systems in the DOD acquisition environment
Background• DOD is managing $1.68 trillion worth of acquisition programs• DOD’s acquisition costs has increased by $135 billion over the past 2 years for
98 major defense acquisition programs• GAO found that most of DOD’s portfolio total cost growth in past 2 years
occurred in the DOD’s largest programs, which are all in production– This means programs experience significant problems and changes well after the
programs and their costs should have stabilized
Focus Areas
GAO has continuously stated that in order to mitigate risks of uncovering problems in the later phases of the development cycle, knowledge based testing should be conducted at key junctures, specifically around the Engineering and Manufacturing Development (EMD) phase
Source: GAO (2011), Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP
Data Extraction
Mission Need
Statement
Between MS-B and MS-C• Appropriate TRL level (GAO knowledge point 1)• PDR (GAO knowledge point 1)• CDR (GAO knowledge point 2)
• Minimum design changes after this point• Manufacturing process should be stabilized before entering Production phase (GAO knowledge point 3)
• Should not have significant cost growth at this point
Beyond MS-C• RDT&E cost growth should be minimum
Data form SAR, GAO, and DOT&E Annual Report, etc. Data form SAR, GAO, and DOT&E Annual Report, etc.
Significant RDT&E cost growth in the Production and Development Phase is the indicator of insufficient T&E activities in earlier phases of the acquisition
Primary Source of Data Selected Acquisition Reports (SARs)
• RAND conducted several research studies based on SAR data and identified numerous “disadvantages" that an analyst must be aware of in order to maintain the validity of the data . Most prevalent problems are:o Failure of some programs to use a consistent baseline cost estimateo Exclusion of some significant elements of costo Exclusion of certain classes of major programs (e.g., special access programs)o Constantly changing preparation guidelineso Inconsistent interpretation of preparation guidelines across programso Unknown and variable funding levels for program risko Cost sharing in joint programso Reporting of effects of cost changes rather than their root causes
• Using SARs provides key “advantages”o SARs conform to a strict reporting format, providing consistency to the datao Those who create SAR reports receive annual SAR training, which adds to the consistency of
the data o SARs are presented to Congress, so the level of scrutiny that SARs receive in the review process
bolsters both the consistency and accuracy of the documents. Databases in general contain inaccuracies, but a database built from SAR data arguably withstands scrutiny better than most
Supplementary Data Sources DOT&E Annual Reports
• Current Activities• Cost data• Issues • Milestone dates• Recommendations
GAO Reports• Technology, Design, and Production Maturity• Cost Data• Milestone Data• Issues
Proposed Methodology Data Extraction
• Base year RDT&E dollars from SAR chosen for analysis, since these dollars exclude estimated inflationary effects
• This format facilitates conversion of the various base years of individual estimates into a single base year, making possible easy comparison across programs
Data Analysis • Relationship between early adequate testing and program success
– RDT&E cost growth Prior/Post MS-C Schedule Slip and Unit Cost Growth– Performance indicators Schedule Slip and Unit Cost Growth
Using SEM for Data Analysis Powerful multivariate techniques – Specialized version of
other analysis methods Advantages over Multiple Regression
• SEM programs provide overall tests of model fit and individual parameter estimate tests simultaneously
• Regression coefficients, means, and variances may be compared simultaneously, even across multiple between subjects groups
• Measurement and confirmatory factor analysis models can be used to purge errors, making estimated relationships among latent variables less contaminated by measurement error
• Ability to fit non-standard models, including flexible handling of longitudinal data, databases with auto correlated error structures (time series analysis), and databases with non-normally distributed variables and incomplete data
An Example of Cost Growth Due to Inadequate Early Testing
• Compare Baseline and Current Status(Cost and Schedule)• RDT&E Cost increase in Production Phase indicate
that more problems discovers during that phase • It is an indicator that program did not perform
adequate early testing
GAO Early Knowledge Based Testing Activities
Recently DOT&E conducted a systematic review of recent 67major acquisition programs that experienced delays• 56 programs (or 84 percent) had performance problems in
testing (either DT, OT, or both) 56 programs were categorized and cost growth were
analyzed and compared with other programs in the same category
Preliminary results shows that programs with inadequate testing in the earlier phase of the development has higher cost growth
Preliminary Results Average % Growth nAircraft 171.65 16Helicopter 139.89 8Ship 116.94 8Missile 95.35 11Satellite 90.92 9Land Vehicle 84.82 6C3I 42.12 13
Inadequate Testing Cost Growth Other Cost Growth
Aircraft 244.94% 114.64%
Missile 116.26% 77.92%
Preliminary data analysis shows that aircraft acquisition programs has highest average percentage of total acquisition cost growth compared to others programs.
Conclusion Knowledge based testing is essential in
detecting problems early in the acquisition phase
Program success depends on the knowledge obtained during earlier tests in the development of the system
Structural Equation Modeling (SEM) is a powerful statistical technique to analyze program success factors from the T&E perspective.
Future Work
Complete Data collection and data base construction
Refine Performance Indicators and Conceptual Model based on further research
Perform data analysis using Structural Equation Modeling (SEM) to test model and find the correlation between program success and “Performance Indicators”
Contact InformationZaw P. Tun
(407) [email protected]
Shahram Sarkani, Ph.D., P.E(888) 694-9627
Thomas A. Mazzuchi, D.Sc.(202) 994-7541
References• Arena, M., Leonard, R., Murray, S., & Younossi, O. (2006). Historical Cost Growth of Completed
Weapon System Programs. Santa Monica CA: RAND• GAO (2011). Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP• GAO (2011). WEAPONS ACQUISITION REFORM :Actions Needed to Address Systems Engineering and
Developmental Testing Challenges, GAO-11-806 • Hough, Paul G. (1992). Pitfalls in Calculating Cost Growth from Selected Acquisition Reports. Santa
Monica CA: RAND• Jarvaise, J., Drenzer, J., & Norton, D. (1996). The Defense System Cost Performance Database: Cost
Growth Analysis Using Selected Acquisition Reports. Santa Monica CA: RAND• Sipple , V. (2002). ESTIMATING ENGINEERING COST RISK USING LOGISTIC AND MULTIPLE
REGRESSION, Master Thesis, AFIT.• The office of the Director, Operational Test and Evaluation (DOT&E) (2011). FY 2011 Annual Report