Acquisition Modeling: The Key toManaging Acquisition Complexity?
January 26, 2004
Linda A. AbelsonRichard J. AdamsSuellen Eslinger
© 2004 The Aerospace Corporation.
3rd OSD Conference on the Acquisition of Software-Intensive Systems
2
Acknowledgements
This work would not have been possible without assistance from thefollowing:
• Reviewers• Douglas Buettner, Software Architecture and Engineering Department• Peter Hantos, Software Acquisition and Process Office• Karen Owens, Software Acquisition and Process Office• Mary Rich, Software Engineering Subdivision
• Funding Source• Aerospace Sponsored Independent Research and Development
3
Outline
• Introduction
• Selected Modeling Technique
• Current Model Structure
• Status and Conclusion
4
Personal Observation & Experience
• Cost and schedule overruns, and performance and qualitydeficiencies are the norm for software-intensive space systemacquisitions
• Problems persist despite• The best efforts of “Blue Ribbon” commissions and a long list of acquisition
initiatives• Significant advances in systems and software engineering technical
processes
• These problems are observed in every large program and programresponses result in similar unsatisfactory outcomes• This happens even though the Government and contractor personnel are
different from program to program
Why do these problems continue to happen?
5
Proposition
• DoD Acquisitions have
• A unique set of Government and contractor organizations and personnel
• A large diverse set of policies, strategies and processes
• A large unique set of political, management and technical activities
• A dynamic political, budgetary and technical environment
• One chance to succeed
• Acquisitions are complex, dynamic, interconnected socioeconomicsystems
• Consequences of System Program Office (SPO) and contractoractions are not evident for months or even years
Modeling is the most cost effective way to describethe complexity of these phenomena
6
Aerospace Research
• Aerospace has a multiyear research project to study theusefulness of acquisition modeling techniques to improveacquisition success• Enhance acquisition management capabilities• Analyze the long-term impact of policy alternatives• Provide a project management simulation training environment
• This briefing provides an overview of the research to date
7
Outline
• Introduction
• Selected Modeling Technique
• Current Model Structure
• Status and Conclusion
8
System Dynamics Modeling
• “System Dynamics is the application of feedback control systemprinciples and techniques to managerial, organizational andsocioeconomic problems” (Dr. E. B. Roberts)
• “System Dynamics is a method of studying the behavior ofsystems to show how decisions, policies, structures and delaysare interrelated to influence growth and stability.” (Dr. J. Forrester)
*
* Adapted from “Integrating System Dynamics and Agent-Based Modeling”
Characteristic System Dynamics Agent-Based Building block Feedback loop connecting
behavioral variables Individual agents connected
by feedback loop Object of interest Structure of the System Agents' Rules
Research approach Deductive: Infer from structure to
behavior
Inductive: Infer from individual agent's behavior to system behavior
Handling of Time Continuous simulation Discrete or continuous Character of object of
interest over time Structure is fixed Agents' rules can be fixed or
adaptive
9
How Could Modeling Help?
System Technical Performance Analysis
• How is coverage affected by a change in constellation altitude?
• What is the impact on throughput of this system architecture?
Policy, Strategy and Process Performance Analysis
• Is this program executable using the proposed policies, strategies,processes, budget and schedule?
• Where should the SPO focus its effort to make the greatest impact?
• What is the impact of performing these unplanned tasks?
• What is the impact of delays in these decisions?
• What is the impact of product quality on acquisition success?
• What is the impact of rework caused by these requirementschanges?
• How does SPO size and experience impact acquisition success?
• Etc.
10
Outline
• Introduction
• Selected Modeling Technique
• Current Model Structure
• Status and Conclusion
11
SPO External Influences
Contractor & Contractor External Influences
SPO
Acquisition Management Organizations
External Technology
Organizations
Staff Organizations
Government Maintenance
Organizations
Certification Organizations
Interfacing System
Organizations
Line Management Organizations
User Organizations
Line Management
Organizations
Quality Assurance
Organizations
Process Improvement Organizations
Policy Making
Organizations
Shared Resources
Organizations
Product DevelopmentOrganization
The SPO’s Essential Role in Acquisition Success
The SPO is theCenter of theAcquisitionHour Glass
12
Real Progress Rate
UndiscoveredRework Rework
GenerationRate
7
Simplified SPO Rework Sector*
* Adapted fromReferences 3, 4,
5 & 6
Work ToBe Done
1
Apparent Progress Rate2
SPO Workforce Sector
SPO Workforce Size
Civilian Military
3
SPO Productivity Sector
SPO Productivity
AverageCivilian
Experience
AverageMilitary
Experience
4
Work DoneCorrectly
5
SPO Quality Sector
SPO Quality 6
SPO Schedule Sector
SPO Schedule Pressure8
Discovering Rework Rate
9
13
SPO Rework Sector
SPO Quality Sector
SPO Productivity Sector
SPO Schedule Sector
SPO Workforce Sector
SPO Average ExperienceSPO Workforce SizeSPO FatigueSPO Schedule PressureSPO Work Done CorrectlySPO Undiscovered Rework Contractor Average Work Quality
SPO Effort Perceived RemainingSPO Fraction Perceived CompleteSPO Workforce SizeScheduled Completion Date
Time RemainingEffort Perceived RemainingMilitary Transfer In RateMilitary Transfer Out RateCivilian Hire RateCivilian Attrition Rate
SPO Average ExperienceSPO Workforce SizeSPO FatigueSPO Schedule PressureContractor Productivity
SPO Workforce Size
SPO Quality SPO Schedule Pressure
SPO Productivity
SPO Rework Influences*
* Adapted fromReference 6
14
SPO Rework Sector
SPO Quality Sector
SPO Productivity
Sector
SPO Schedule
Sector
SPO Workforce
Sector
Contractor Rework Sector
Contractor Quality Sector
Contractor Productivity
Sector
Contractor Schedule
Sector
Contractor Workforce
Sector
2 SPO Qualityaffects
Contractor Quality
1 Contractor Qualityaffects
SPO Quality
5 Contractor ReworkaffectsSPO
Productivity
6 SPO Reworkaffects
Contractor Productivity
4 SPO Rework affectsContractor Rework
3 Contractor Rework affects SPO Rework
3
4
5
6
1
2
SPO-Contractor Model Sector Relationships*
SPO Half
ContractorHalf
* Adapted fromReference 6
15
Outline
• Introduction
• Selected Modeling Technique
• Current Model Structure
• Status and Conclusion
16
Model Status
• Completed the first iteration
• Currently gathering data needed to begin model validation• Initial focus is on a retrospective analysis of an existing program
• Potential model enhancements• Decompose the model into multiple space system acquisition phases• Decompose communication/decision delays by type• Explicitly incorporate the impact of concurrent engineering task
parallelism• Explicitly incorporate SPO personnel task overload
17
Conclusion
• System dynamics modeling is a way to describe SPO andcontractor alternative courses of action and answerquestions about their long term consequences• Has already been used in many industries for the contractor half
• Acquisition modeling success depends on model validationand the willingness of decision makers to act on the results• Has already been done in many industries for the contractor half
• Successful use of SPO-contractor system dynamicsmodels in the software-intensive space system acquisitiondomain depends on• Demonstrating than the models provide useful insight into
questions important to acquisition decision makers and• Acquisition decision makers trusting the models sufficiently to act
based on the models’ results
Backup Material
19
Contact Information
• Linda A. Abelson• Email: [email protected]• Telephone: 310 336-7350
• Richard J. Adams• Email: [email protected]• Telephone: 310 336-2907
• Suellen Eslinger• Email: [email protected]• Telephone: 310 336-2906
20
Acquisition Modeling Research Goals: Support IncreasedAcquisition and Program Execution Success
Training
PolicyAnalysis
ProgramOffice Support
• Program ManagementSimulators
• Discipline Specific Simulators• Systems & SoftwareEngineering
• Facilitated Problem SolvingWorkshops
• Acquisition Strategy• Cost and ScheduleAnalysis
• Strategic ProgramManagement
• Tactical ProgramManagement
• Program Post-MortemAnalysis
• Facilitated ProblemAnalysis
• Budget Changes• Acquisition PolicyChanges
• Requirements Changes• Schedule Changes• Resource Allocation• Systems of SystemsIntegration
Implementation & Education
Long Term Objectives
Start
Feasibility
FY 03
FY 04
FY 05+
21
SPO Work to be Done
SPO Work Done
Correctly
SPO ApparentProgress
Rate
SPO UndiscoveredRework
SPO Total Effort
SPO InitialUnrecog
Proj Work
SPO Active
SPO Real Progress
Rate
SPO Rework Generation
Rate
SPO CumulativePerceived Progress
SPODiscovering
Rework
SPO Change inProj Size
SPO Time to Add New Work
SPO UndiscoveredWork
SPO Discovering Work
SPO InitialProj Size Est
SPO InitialUnrecog
Proj Work
~
SPO Fract of UndiscoveredWork Recog
SPO Effective Pdy
SPO Time to Detect Rework
SPO SchedulePressure
SPO Fract PerceivedComplete
SPO CurrentProj Definition
~SPO Fract Percieved Complete
Rework Detection Effect
KTR Progress Effect onSPO Rework Discovery
~SPO Sched Pressure
Rework Detection Effect
SPO Normal Timeto Detect Rework
SPO Min Time toDetect Rework
SPOQuality
MIL Workforce
Civ Workforce
SPO Rework Sector – Stella® Version*
*High Performance Systems
22
General Project Model Sectors*
* Adapted fromReference 6
Rework Sector
Quality Sector
Productivity Sector
ScheduleSector
WorkforceSector
Models the flow of work and rework through the project
The sectors below influence how quickly and how well the work is performed
Models how much of the work
performed will need to be reworked later
in the project
Models how much work can be performed
per person
Models how many people with what experience are
needed to perform the work and when
they can be available
Models how quickly work
product defects are identified for
rework
23
References - 1
Introductory• “System Dynamics Modeling for Project Management”
• Available on the author’s website, http://web.mit.edu/jsterman/www/• John D. Sterman
• “A Skeptics Guide to Computer Models”• Available on the author’s website, http://web.mit.edu/jsterman/www/• John D. Sterman
• References 1 & 2 are useful, short introductions to system dynamics and computer models
• “The Rework Cycle: Benchmarks for the Project Manager”• Project Management Journal, Volume XXIV, March 1993
• “Swords and Plowshares: The Rework Cycles of Defense and Commercial Software Development Projects”• American Programmer, May 1993
• The $2000 Hour: How Managers Influence Project Performance Through the Rework Cycle• Project Management Journal, Volume XXV, March 1994• Kenneth G. Cooper
• References 3, 4 & 5 are important early papers
Program Related Analysis• “The Cost and Cycle Time Implications of Selected Contractor and Air Force System Program Office
Management Policies During the Development Phase of Major Aircraft Acquisition Programs”• MIT Masters Thesis, 1999• Sean Morgan
• The paper that instigated the current research and the inspiration for the current model
24
References –2
• “Understanding Firefighting in New Product Development”• The Journal of Product Innovation Management 18, (2001) 258-300• Nelson P. Repenning
• Analyzes resource limited program behavior
• “Iteration Management for Reduced Cycle Time in Concurrent Engineering”• Concurrent Engineering Research and Applications Journal Special Issue: A Complex Systems
Perspective on Concurrent Engineering, 2003• David N. Ford and John D. Sterman
• “Dynamic Modeling of Product Development Processes”• System Dynamics Review, Vol. 14, No. 1, (Spring 1998): 31-68• David N. Ford and John D. Sterman
• References 8 & 9 are related papers on program concurrent engineering characteristics
• “Systemic Project Risk Management – the way ahead”• International Journal of Risk Assessment and Management, Vol. 1, Nos. 1 and 2, 149-159, 2000• Terry M. Williams
• Use of system dynamics in project risk management
Recent Project Management Use• “Strategic Management of Complex Projects: A Case Study Using System Dynamics”
• System Dynamics Review 17, 237-260, (2002)• Lyneis, James M., Cooper, Kenneth G., Els, Sharon A.
• Contractor focused system dynamics application on the Air Force Peace Shield Program
• “Learning to Learn, From Past to Future”• Fourth European Project Management Conference, PMI Europe 2001• Kenneth G. Cooper
• References 11 & 12 are related papers for the Peace Shield Program
