Acquisition Models and Simulations

i

DEFENSE SYSTEMS MANAGEMENT COLLEGE

Systems Acquisition Manager’s Guidefor the use of

MODELS AND SIMULATIONS

Report of theDSMC 1993-1994

Military Research Fellows

Colonel Lalit K. Piplani, USA

Lieutenant Colonel Joseph G. Mercer, USAF

Lieutenant Colonel Richard O. Roop, USAF

September 1994

PUBLISHED BY THEDEFENSE SYSTEMS MANAGEMENT COLLEGE PRESS

FORT BELVOIR, VIRGINIA 22060-5565

ii

DISCLAIMER

This book was produced in the Department of Defense (DoD) schoolenvironment in the interest of academic freedom and the advance-ment of national defense-related concepts. The views expressed inthis book are those of the authors and do not reflect the official po-sition or policy of the DoD or those of the United States Govern-ment.

For sale by the U.S. Government Printing OfficeSuperintendent of Documents, Mail Stop: SSOP, Washington, DC 20402-9328

ISBN 0 - 16 - 045161 - 2

iii

DEDICATION

In this, the fiftieth anniversary year of the Normandy invasion, theDSMC Military Research Fellows dedicate this guidebook to allwho have faithfully served the Armed Forces of the United States ofAmerica, and to their families for their support.

iv

ACKNOWLEDGEMENTS

This guidebook is the result of an 11-month Military Research Fel-lowship program sponsored by the Defense Systems ManagementCollege (DSMC) and Under Secretary of the Air Force, Acquisition(SAF/AQ) and funded by the Defense Modeling and SimulationOffice (DMSO).

v

“The advantage we had in DESERT STORM hadthree major components. We had an advantage inpeople, an advantage in readiness, and anadvantage in technology.”

Dr. William J. Perry’s (as Deputy Secretaryof Defense) remarks to the NationalContract Management Association,Washington, DC, November 18, 1993

“We need to preserve that part of the industrial basewhich will give us a technological advantage, butwe have to do it at a reduced cost and increasedefficiency in procurement.”

Dr. William J. Perry’s (as Deputy Secretaryof Defense) remarks to the NationalContract Management Association,Washington, DC, November 18, 1993

“Simulation and modeling technology can beapplied to every major DoD weapon developmentprogram to reduce design and production cost,improve performance, improve diagnostics andmaintenance, assist in better and faster training ofpersonnel, and improved command and control onthe battlefield.”

Mrs. Colleen Preston’s (Deputy UnderSecretary of Defense Acquisition Reform)Testimony to Congress, (March 1994)

vi

NOTICE

Copies of this report may be obtainedby writing or faxing the:

DEFENSE SYS MGMT COLGATTN OS PR9820 BELVOIR RD STE G38FT BELVOIR VA 22060-5565

Telephone: (703) 805-2743DSN: 655-2743Fax: (703) 805-3857

vii

TABLE OF CONTENTS

Page

Preface ......................................................................................................................... xiii

Chapter 1 — Introduction1.1 Purpose ......................................................................................................... 1-21.2 Methodology ................................................................................................ 1-21.3 Spectrum of Military Modeling and Simulation ......................................... 1-31.4 Assumptions ................................................................................................. 1-31.5 Acquisition Environment ............................................................................. 1-41.6 Models and Simulations in Acquisition ....................................................... 1-51.7 Objective....................................................................................................... 1-6

Chapter 2 — Background2.1 Today’s Applications .................................................................................... 2-22.2 Systems Acquisition Process ....................................................................... 2-22.3 Requirements Generation System................................................................ 2-4

Chapter 3 — Organization and Policy3.1 Office of the Secretary of Defense Organization and Management

of Modeling and Simulation ................................................................... 3-13.2 Joint Staff Organization and Management

of Modeling and Simulation ................................................................... 3-43.3 Department of the Army Organization and Management

of Modeling and Simulation ................................................................... 3-63.4 Department of the Navy Organization and Management

of Modeling and Simulation ................................................................... 3-103.5 Marine Corps Organization and Management

of Modeling and Simulation ................................................................... 3-133.6 Department of the Air Force Organization and Management

of Modeling and Simulation ................................................................... 3-15

Chapter 4 — Classification of Models and Simulations4.1 Constructive Models and Simulations ......................................................... 4-24.2 Virtual Simulation ........................................................................................ 4-34.3 Live Simulations .......................................................................................... 4-44.4 hierarchy of Models and Simulations .......................................................... 4-54.5 Hybrid Models and Simulations .................................................................. 4-84.6 Summary ...................................................................................................... 4-14

viii

Chapter 5 — Modeling and Simulaiton in Support of Acquisition5.1 Modeling and Simulation Across the Acquisition Life Cycle ..................... 5-15.2 Modeling and Simulation in Support of Acquisition Activities .................. 5-115.3 Summary ...................................................................................................... 5-27

Chapter 6 — Modeling and Simulation Issues6.1 Introduction .................................................................................................. 6-16.2 Credibility of Models and Simulations (or “How can I

believe what I’m seeing?”)...................................................................... 6-16.3 Verification, Validation and Accreditation (VV&A) ................................... 6-26.4 Integration .................................................................................................... 6-66.5 Advanced Distributed Simulation (ADS) .................................................... 6-86.6 Standards for the Modeling and Simulation Environments ........................ 6-96.7 Maintaining Credibility of M&S ................................................................. 6-10

Chapter 7 — Management Considerations i Modeling and Simulation7.1 Introduction .................................................................................................. 7-17.2 Survey of ACAT I & II PMOs ..................................................................... 7-17.3 Planning the Modeling and Simulation Effort ............................................. 7-47.4 Models and Simulations as Deliverables ..................................................... 7-11

Chapter 8 — Movement Toward a Future State of Modeling and Simulation8.1 Introduction .................................................................................................. 8-18,2 The Evolution of Modeling and Simulation (M&S) in Acquisition ........... 8-18.3 Virtual Prototyping at the U.S. Army Tank Automotive and

Armament Research, Development, and Engineering Center (TARDEC) ................................................................................... 8-2

8.4 Getting to the Future State of M&S............................................................. 8-68.5 Management Actions.................................................................................... 8-88.6 Some Applicaiton Best Practices ................................................................. 8-88.7 Some Management Best Practices ............................................................... 8-128.8 Summary ...................................................................................................... 8-16

Epilogue

APPENDICES

Appendix A DoD Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .................................. A-1

Appendix B Department of the Army Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .......... B-1

ix

Appendix C Department of the Navy Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .......... C-1

Appendix D Marine Corps Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .......... D-1

Appendix E Department of the Air Force Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .......... E-1

Appendix F Additional Sources of Information for Modeling and Simulation in Weapon Systems Acquisition .................................. F-1

Appendix G Modeling and Simulation Templates for Use in Weapon Systems Acquisition Activities ........................................ G-1

Appendix H Abbreviations and Acronyms Used Throughout this Modeling and Simulation Guidebook ............................................ H-1

Appendix I Glossary ............................................................................................... I-1

LIST OF FIGURES

Figure 1-1. Systems Acquisition Process Cycle .................................................... 1-5

Figure 2-1. Three Major Decision Making Support Systems ............................... 2-3

Figure 2-2. Requirements Generation System....................................................... 2-4

Figure 3-1. Office of the Secretary of Defense Organization for M&S Management ......................................................................... 3-2

Figure 3-2. Joint Staff Organization and Management of Modeling and Simulation ............................................................................... 3-5

Figure 3-3. Army Organization for M&S Policy Making and Dissemination...... 3-8

Figure 3-4. Department of the Navy Modeling and Simulation Management .....3-10

Figure 3-5. Marine Corps Modeling and Simulation Management ...................... 3-14

Figure 3-6. Air Force Organization for Modeling and Simulation Management ................................................................ 3-16

x

Figure 3-7. Air Force Requirements Generation System ...................................... 3-17

Figure 4-1. Classes of Models and Simulations .................................................... 4-2

Figure 4-2. Hierarchy of models and Simulations................................................. 4-5

Figure 4-3. Attributes and Uses of Models and Simulations within the Hierarchy....................................................................... 4-9

Figure 4-4. Hardware/Software-In-The-Loop Simulation (HW/SWIL) .... ........... 4-11

Figure 4-5. Relationship of Program Documents, Needs and Measures .............. 5-4

Figure 5-2. Models and Simulations in Requirements Definition ........................ 5-13

Figure 5-3. Simulation Based Design .................................................................... 5-17

Figure 5-4. Missile Data Requirements and Test Assets ....................................... 5-20

Figure 5-5. Sidewinder Firing History ................................................................... 5-21

Figure 5-6. Model-Test-Model Approach to Development ................................... 5-24

Figure 6-1. Verification, Validation and Accreditation .......................................... 6-2

Figure 6-2. M&S Confidence Assessment Scope-of-Evidence ............................ 6-5

Figure 6-3. Increasing Cross-Functional Integration of M&S in Systems Acquisition ................................................................... 6-7

Figure 7-1. How Program Management Offices are Currently Using Modeling & Simulation ................................................................. 7-2

Figure 7-2. Summary of Modeling and Simulation Needs Perceived byPMOs ... 7-3

Figure 7-3. Outline for a Simulation Support Plan (SSP) ..................................... 7-6

Figure 8-1. Virtual Prototyping Process ................................................................ 8-3

Figure 8-2. The Tracked Vehicle Workstation: Expanding Vehicle Simulation’s Role In The Vehicle Development Process .............. 8-4

Figure 8-3. Enabling Technologies ........................................................................ 8-6

xi

Figure 8-4. Notional Hierarchy of Technologies ................................................... 8-7

Figure 8-5. Combat System Engineering Analysis Laboratory Applicaiton to System Development ............................................. 8-9

Figure 8-6. BAT System Validation: Process......................................................... 8-14

Figure 8-7. BAT Test Methodology ....................................................................... 8-15

Figure 8-8. TOMAHAWK Simulation Management Board Certificate ...............8-16

Figure B-1. Army Organizations Active in M&S for Acquisition ......................... B-1

Figure F-1. IACs Associated with Military Missions/Functions ........................... F-3

Figure F-2. IACs Associated with DoD Key Technologies ................................... F-4

Figure G-1. Modeling and Simulation Application in Requirements Definition... G-2

Figure G-2. Modeling and Simulation Application in Program Management ...... G-3

Figure G-3. Modeling and Simulation Application in Design and Engineering ... G-4

Figure G-4. Modeling and Simulation Application in Manufacturing................... G-5

Figure G-5. Modeling and Simulation Application in Test and Evaluation ........... G-6

Figure G-6. Modeling and Simulation Application in Logistics Support .............. G-7

Figure G-7. Modeling and Simulation Application in Training ............................. G-8

xii

xiii

PREFACEThis guidebook represents the combined efforts of three military Research Fellows par-ticipating in an 11-month Senior Service College Research Fellowship program spon-sored by the Under Secretary of Defense for Acquisition and Technology (USD(A&T)).The fellowship program has two purposes: first, to provide professional and military edu-cation for selected officers from the Army, Navy and Air Force; and second, to conductresearch in a subject of vital interest to the U.S. defense acquisition community. In keep-ing with its role as the center for systems management education in the DoD, the DefenseSystems Management College (DSMC), cooperating with the Harvard University Gradu-ate School of Business, provided the means for conducting this study. The fellowshipprogram included a 12-week resident Program for Management Development (PMD) courseat Harvard University.

Our research effort was funded by the Defense Modeling and Simulation Office (DMSO).The focus of the research was to develop and produce a guidebook on modeling and simu-lation (M&S) as it relates to systems acquisition management. The book will be broadlydistributed to policy makers, military departments, government offices, research centers,libraries and academic institutions.

The purpose of this research guidebook is to provide the reader with current DoD initia-tives, forthcoming policy and guidance, and identification of newly formed organizationsin M&S. This book will provide a better understanding of how models are developed,applied in simulations and their value in analysis; and more importantly, it provides aquick reference for the application of these tools in the development of weapon systems.Also, this guidebook offers lessons learned from current acquisition managers, advice onmanagement practices, points of contact (POCs) and how to access M&S data bases.

During the first month of the research effort time was spent initiating background re-search, developing a research plan and consulting with the DSMC faculty. The next 12-weeks were spent attending the Harvard PMD course. Upon our return to DSMC, weimmersed ourselves in this research effort.

As we began our research, we were quickly overwhelmed by the amount of information onM&S with two exceptions; M&S documented history and the role of M&S in acquisition.Since our topic was preselected and funded by DMSO, we focused our efforts to provide aconsolidated guidebook for M&S in acquisition with little attention to history.

To start with, even a basic understanding of terms is needed. We will begin here by provid-ing definitions of model and simulation. A model is a physical, mathematical or otherwiselogical representation of a system, entity, phenomenon or process. The definition of simu-lation is twofold: a method for implementing a model over time; and a technique for test-

xiv

ing, analyzing or training in which real-world systems are used; or where real-world andconceptual systems are reproduced by a model.

The scope of this effort was another difficult aspect. What resulted, with the assistance ofmembers from the Acquisition Task Forces on Modeling and Simulation (ATFM&S), wasto provide a quick reference type guide.

Our task was to produce this guidebook with a focus on Program Management. Difficul-ties arose in how to approach the DoD’s Systems Acquisition Process while weaving M&Sinto the process. The complexity was compounded because the Systems Acquisition Pro-cess is comprised of three separate but interfacing systems: the requirements generationsystem; the acquisition management system; and the planning, programming and budget-ing system (PPBS).

The program management world includes all three systems. However, the only system thatpresently uses the entire realm of M&S is the acquisition management system. Therefore,this guidebook focuses on the acquisition system with some discussion of the require-ments generation system and the PPBS. For a very good, quick tutorial on the DoD sys-tems acquisition process, refer to the book Introduction to Defense Acquisition Manage-ment, by Joseph H. Schmoll.1

Readers pressed for time may wish to proceed directly to Chapter Seven for Modeling andSimulation Management Considerations and Chapter Eight for The Future State of Mod-eling and Simulation and Best Practices. For others, Chapter One provides an introduc-tion to the acquisition environment and applicability of M&S. Chapter Two provides gen-eral information about M&S. Chapter Three provides a quick reference for policy andguidance. Chapter Four contains a more detailed discussion regarding The Classes of Modelsand Simulation. Chapter Five provides a view of M&S Applications from two differentaspects—across the acquisition life cycle and in support of specific acquisition activities.Chapter Six provides managers with information on some of the issues in the use andmanagement of M&S. The Appendices are provided for expanded information about eachindividual Service, as well as additional M&S information sources.

We owe our gratitude to many people. During this research effort, we have been genuinelythankful for their help. The faculty and staff at Harvard University and DSMC were ex-tremely helpful with their encouragement, insight and support. A number of people havebeen particularly helpful: Dr. Adelia Ritchie, Dean of the Research, Consulting and Infor-mation Division at DSMC, served as our mentor providing helpful advice and guidancethroughout this research effort. Special thanks to Mr. Dan Chapla, CDR John Farlin, LtColWayd Weber, LTC James Huskins, Mr. Chuck Cochrane, Mr. Bill Motley, CDR JamesGrayson and Mr. Joel Manary of the DSMC faculty for their valuable insights and forreviewing our document. We owe our gratitude to the DSMC librarians for their outstand-ing support throughout our effort; as well as to the DSMC Press staff for their many hoursof work to make this a quality product.

xv

This document would not have been possible without a few key players outside DSMC. Aspecial thanks to: Major Joe Bond, DMSO, for all his support from funding to consolida-tion of survey information to coordination of ATFM&S activities; Mr. Ed Muendel andMr. Gil Brauch, Logistics Management Institute (LMI), for their consolidation of surveyinformation, advice for site visits to Navy and Army installations, and candid commentsduring our reviews; Mr. Chip Ferguson and Mr. Jim O’Looney, Science Applications In-ternational Corporation (SAIC), for their consolidation of survey information, advice forsite visits to Air Force locations, and for their candid comments during our reviews; andDr. Stuart Starr, the MITRE Corporation, for his insight into technology. Additionally, theefforts of Mr. Ron Cross, HQ TRADOC, Ms. Trell Hudson, MARCORSYSCOM, Dr.Robert Smith, NAWCWPNS, Mr. David Berry, SPAWAR 312-3, Ms. Sylvia Diaz, HQDAOASA(RDA), Mr. Michael Rybacki, USALEA, LtCol Mike Baum, Joint Staff/SPED (J-8), LtCols Cheryl Balombini and Mike Hathaway, HQ USAF/XOM, LtCol Harry Mandros,ACC/DRM, Maj Dean Fish, MCMSMO and Maj Steve Chimelski, HQ AFMC/XRX wereessential in pulling this document together.

The Research Fellows extend a special note of appreciation to Ms. Joan Sable, DSMCMilitary Research Fellowship Coordinator. Ms. Sable’s efforts were invaluable in this ef-fort; managing the budget, ensuring adequate administration support, and coordinatingand facilitating the document reviews. She knew where the “show-stoppers” were andkept our project on track, on budget and on schedule. Ms. Sable was tireless in her effortsto ensure that we were free to concentrate our efforts toward providing a document thatwould be useful and meaningful to the reader.

There are many others that deserve recognition, but in fairness to all, there are too many tomention. The three research fellows would like to thank all of those interviewed. We hopethis guidebook is as helpful to you as you were to us—thank you.

ENDNOTES

1DSMC Press, March 1993, GPO no. (ISBN 0-16-041725-2)

1-1

11INTRODUCTION

In June 1991, the Deputy Secretary of De-fense approved a plan to strengthen the useof modeling and simulation (M&S). He alsodesignated the Under Secretary of Defensefor Acquisition and Technology(USD(A&T)), formerly the USD(A), as re-sponsible for “strengthening the use of mod-eling and simulation in joint education andtraining, research and development, test andevaluation and operation and cost analysis.”1

In June 1992, the Institute for DefenseAnalyses (IDA) published a report titled “AReview of Study Panel Recommendationsfor Defense Modeling and Simulation.”2

IDA reviewed 179 recommendations madeby 25 separate study panels, over a 16-yearperiod, concerning defense M&S. The De-fense Modeling and Simulation Office(DMSO), using this document as a concep-tual foundation, reviewed and classified therecommendations and set off to plan for andimplement those that provided for new andextended applications for M&S. The DMSOespecially focused on systems acquisitionand test and evaluation (T&E). The initia-tive instituted by the Deputy Secretary ofDefense is now referred to as the DefenseModeling and Simulation Initiative.

Some key conclusions drawn from the IDA

review were:

• Specific areas, such as the architecturalissues of interoperability and specificationof standards, and the life-cycle support ofdefense models and simulations, deservemore attention and support.

• There are many areas to which defenseM&S either should be applied anew or ex-tended, especially those associated with sys-tems acquisition.

• There are substantial needs and oppor-tunities for improving management and co-ordination of defense M&S activities.

In a DoD Inspector General (IG) audit onM&S, 1 Mar 1993,3 the following findingshighlighted many shortcomings in DoD’sability to effectively and efficiently utilizemodels and simulations:

1. DoD lacks adequate M&S policy;

2. Most M&S applications lack verifica-tion, validation and accreditation (VV&A);

3. Significant effort is devoted to devel-oping “stand alone” models with no intentto reuse—costly duplication;

1-2

4. Absence of a central library resourcecontributes to redundant investment; and

5. FY93 DoD expenditures were esti-mated to be from $1.3 to $1.6 billion—con-solidation of effort could have saved an es-timated $800 million.

The philosophy of M&S evolves aroundthree overlapping areas: operational plan-ning, acquisition and training. Operationalplanning helps utilize our equipment andforces to best achieve our national objectivesand identify new requirements. Acquisitionprovides the items, systems and technologythe commander can use to support opera-tional planning. Finally, training teaches ourpeople how to employ forces, use systemsand apply technology provided through ac-quisition to support operational planning.The use of M&S provides a comparativelyinexpensive way to plan, acquire and train.

Throughout the DoD there is an increasinginterest in addressing the problems identi-fied by the DoD IG. The formation of theAcquisition Task Force on Modeling andSimulation (ATFM&S), sponsored by theDMSO, represents a commitment of theDoD to find ways to optimize and fully uti-lize hardware and software tools and databases available to all DoD agencies, particu-larly program managers (PMs).

1.1 PURPOSE

The purpose of this guidebook is to assistthe acquisition community, by providing in-formation on DoD policy regarding M&S,identifying the existing M&S capability anddescribing how M&S can be appliedthroughout the acquisition cycle. Under-standing these topics gives PMs the oppor-tunity to pursue dual-use technologies, in-cluding commercial-military integration; al-

lows for faster and lower cost manufactur-ing and complements operational test andevaluation (OT&E).4

1.2 METHODOLOGY

This project was approached by focusing onour target customers, the program manag-ers, and what would help make their jobeasier.

This guidebook was developed using amodified approach to designing a technicaldocument. The in-process reviews (IPRs)(sometimes called murder boards) wereconducted, first by the fellows, then by thefaculty at DSMC. Additional IPRs werescheduled with selected PMs and represen-tatives from the ATFM&S and Service M&Srepresentatives, at DSMC prior to publish-ing.

This guidebook was based on the currentsystems acquisition process and discussesthe capabilities of using M&S to enhancethe efficiency of that process. Since the useof models and simulations is functionallyoriented, this document retains its applica-bility to any future modification of the ac-quisition process.

At the front of this effort with DMSO fund-ing, the Logistics Management Institute(LMI)—with support from the Science Ap-plications International Corporation (SAIC)and the MITRE Corporation—sent surveysto selected ACAT I and ACAT II programoffices. These surveys were used to deter-mine what models and simulations exist;how they are used, managed and analyzed;and what are the “opportunities” and “pit-falls” surrounding their use. The contractorsconsolidated the information and providedit to the Research Fellows and the ATFM&S.Using the surveys as a starting point, data

1-3

were gathered from product centers, re-search centers, contractors, using com-mands, test centers, and training and simu-lation conferences by site visits and telecom-munications.

1.3 SPECTRUM OF MILITARYMODELING AND SIMULATION

The using community of models and simu-lations is broad; extending from operatorsof weapon systems all the way to analystsin laboratories. Full comprehension of M&Sterminology is difficult—this document willnot make the reader an expert, but it will aidin the reader’s understanding of this verycomplex topic.

There have been several discussions aboutappropriate definitions and use of termsthroughout the DoD; this book is no excep-tion. However, after a review of this guide-book readers will have a solid foundationfor discussing and improving their knowl-edge of M&S.

There are many ways to characterize M&S.The spectrum of M&S includes broad types,classes, hierarchy and applications (func-tional areas). The three general types of mod-els are: wargaming; training; and acquisition.

Wargaming models range from single en-gagement (one-on-one) to joint theater levelcampaign operations. Training models rangefrom single template instructional systemsto complex virtual reality simulations. Ac-quisition models range from physical levelphenomenon models through engineeringcomponent design tools to models of sys-tems-in-the-end-use-environment.

Classes of models and simulations includevirtual, live and constructive. These are de-scribed in Chapter 4.

The hierarchy of models (usually depictedas a pyramid) represents the standard build-ing block approach for interfaces within themodeling world, indicating an ever increas-ing complexity from the bottom up. This hi-erarchy is also discussed further in Chapter 4.

Applications of models and simulations aregenerally viewed from the functional per-spective, which are categorized as education,training and operations; research and devel-opment; test and evaluation; analysis; andproduction and logistics. Applications ofmodels and simulations to specific activi-ties is further discussed in Chapter 5.

This brief insight is intended to whet thereaders appetite and show how M&S is avital part of the DoD systems acquisitionprocess.

The advances being made in computer hard-ware and software technologies are provid-ing increasing opportunities to leverageM&S applications across traditional func-tional lines. For instance, some programs arebeginning to use engineering level systemmodels in virtual combat environments nor-mally used by the training community. Prop-erly done, this cross-functional applicationallows for the early evaluation of designtrade-offs on overall combat performance.Using existing combat simulations has twopayoffs:

1. Consistency of evaluation perspectivebetween the acquisition and user communi-ties; and

2. Reduction of duplication and overlap-ping simulation development efforts.

1.4 Assumptions

The following assumptions provide a com-

1-4

mon starting point to present M&S applica-tions to the systems acquisition process.

• The acquisition community is operat-ing under the acquisition process establishedby DoDD 5000.1, Defense Acquisition andDoDI 5000.2, Defense Acquisition Manage-ment Policies and Procedures, both datedFebruary 1991.

• DoDD 5000.59, DoD Modeling andSimulation (M&S) Management definitionsare accepted throughout the DoD as the cor-rect and standard definitions.

• The reader already has a basic under-standing of the systems acquisition processand how its functional elements work.

• The reader knows very little aboutM&S.

• The reader is interested in knowing thefollowing:

— What people are talking about whenreferring to M&S being used in acqui-sition.

— How to explain the use of M&S toolsin support of a particular program.

— How to make sure the right modelsand simulations are being used.

— How to interpret what comes out ofM&S.

— How to assure the use of M&S iscontrolled with respect to a particularprogram.

— Where to go to get help.

• This guidebook talks to acquisition pro-grams under the DoD 5000 series versus the

DoD 8000 series, but the concepts presentedherein apply equally as well to software ac-quisition.

1.5 Acquisition Environment

The DoD is faced with a new world-wideorder of political, economic and militaryaffairs. National security has many new chal-lenges. The Government is committed toproviding a strong force capable of effec-tively deterring threats to the United Statesand its allies.

The downsizing of the military, the reduc-tion of available resources and a process thattakes more than a decade to exploit advanc-ing technologies and meet evolving require-ments, indicate the need for improved effi-ciencies and process improvement.

1.5.1 Existing Process

The DoD systems acquisition processevolved over the years to ensure fair treat-ment to contractors; prevent fraud, waste andabuse; keep a government check on its au-thority over and demand on suppliers; andenhance socioeconomic objectives. The fi-nal acquisition process ended up being cum-bersome; taking excessively long to meetwarfighter requirements. In addition, theadministra- tive process drove DoD costshigher and higher.

1.5.2 Future Acquisition

The challenge is to procure state-of-the-arttechnology and products, rapidly, from reli-able suppliers who utilize the latest manu-facturing and management techniques; as-sist United States companies now predomi-nantly dependent on DoD business to tran-sition to dual-use production; aid in thetransfer of military technology to the com-

1-5

mercial sector; and preserve defense-uniquecore capabilities.

There are many pros and cons in the powerof M&S that should be tempered with ahealthy dose of reality. All acquisition prob-lems will not be solved by M&S. However,M&S usage is the key ingredient to successin today’s environment.

The risk associated with the acquisition pro-cess of today can be minimized with properplanning, use and understanding of modelsand simulations. For example, in a statementto the U.S. Senate Subcommittee on Fed-

eral Services, Post Offices and Civil Service,March 22, 1994, the DUSD(AR) stated,“simulation offers the potential of a cheaperand quicker way to find failure modes thandoes field testing.”5

1.6 Models and Simulations in Acquisition

Models and simulations are viewed as a po-tential answer to many of DoD’s systemsacquisition process problems.

Models are generally used to prove concepts.This can be anything from a mathematicalcalculation to building full-scale replicas and

Figure 1-1. Systems Acquisition Process Cycle

Systems Acquisition Process Cycle

NEED

WARFIGHTINGCAPABILITY

DEPLOYMENT BUILD

FEEDBACK

DESIGN

DEFENSEGUIDANCE

PROGRAMMATIC RISKOPERATIONAL RISK

OPERATINGCOMMANDS

STUDIES &ANALYSIS

PMO- Engineering- Productionn- Logistics- New Equip. Trng.

ANALYSIS MODELS

THEATER/CAMPAIGNMISSION/BATTLE

ENGAGEMENT MODELS

ENGINEERING, RAM, TEST BEDS, HW/SWIL,HUMAN-IN-LOOP, VIRTUAL PROTOTYPES,

ETC...

MISSIONREHEARSAL

VIRTUALSYSTEMS

TRAINING

OPERATING &SUPPORT UNITS

MATERIEL ALTERNATIVES

OT&E DT&E

1-6

submitting them to controlled environments.An underlying reason for using models andsimulations is to reduce risk.

Risk reduction is the unifying conceptthroughout the entire systems acquisitionprocess. It is very logical to view modelsand simulations as tools to minimize risk tocost, schedule, performance and support-ability for the PM. From this viewpoint thevalue added of models and simulations canbe communicated.

An explanation of what risk means in theacquisition system is needed at this point.When a system is fielded (operational), it isput there to meet a particular requirementbased on a need. The system’s ability to meetthe mission requirements is continuallyevaluated. As the system becomes outdated,the risk associated with the system’s abilityto accomplish the mission increases. A riskassessment is conducted to determine if themission can be accomplished by changingthe use(s) of the system (i.e. tactics); modi-fying the system; or acquiring a new sys-tem. Once the level of risk is at the pointwhere a major modification or a new sys-tem is needed, the operational risk, throughrequirements documentation is translatedinto programmatic risk and becomes sharedwith the acquisition community.

The acquisition community receives direc-tion to provide a system that satisfies re-quirements evolved from the mission need.The cost, schedule, performance and sup-

portability risks associated with the acqui-sition process are inherent. The acquisitioncommunity helps find the best contractor,manages the development and reportsprogress up through the chain to aggres-sively provide the operational community asystem to meet its need.

Since time and resources are limited, thisusually creates a situation where trade-offsmust be made frequently using informationgenerated by models and simulations to getthe system operational at an acceptable per-formance level. This completes the systemsacquisition process cycle. Figure 1-1 illus-trates this cycle.

Why make this trip around the acquisitionrisk cycle? We do it to help clarify the needto use M&S to minimize risk through properuse in the systems acquisition process.

1.7 Objective

The objective of this guide is to provide areference for acquisition managers whichdescribes M&S policies, types of modelsand simulations, applications, and key tech-nical and management issues. This guide-book is intended for use by program man-agement offices (PMOs), acquisition sup-port agencies, policy makers, military de-partments, government offices, research cen-ters, libraries, industry and academic insti-tutions. It should enable the manager tomake better use of models and to better un-derstand the results.

1-7

1. Deputy Secretary of Defense Memorandum, June21, 1991.

2. IDA Document D-1161, June 1992.

3. DoD IG Audit Report No. 93-060, “Duplication/Proliferation of Weapons Systems’ Modeling andSimulation Efforts Within DoD”, March 1, 1993.

ENDNOTES

4. Extract from John M. Deutch, USD(A&T), in hisTestimony on Defense Reinvestment and Conver-sion Programs, May 4, 1993.

5. Colleen Preston, Statement to U.S. Senate Subcom-mittee, March 22, 1994.

2-1

22BACKGROUND

puter-generated or synthetic environment)for example is significantly changing ourlives; entertainment, work, learning, traveland communications are all incorporatingvirtual reality. Information is being movedversus people.

Benefits are also being gained by utilizingvirtual prototypes, computer based simula-tion of systems with a degree of functionalrealism. For example, virtual prototypeswith properly modeled fluid dynamics canbe used in designing aircraft, ships and mis-siles to replace wind tunnel testing: a costlyand time consuming process.1

With this technological revolution, thepresent acquisition process is impractical.This chapter will provide background infor-mation on why the push for M&S; and in-formation about some uses of models andsimulations.

In the beginning it is absolutely essential todefine modeling and simulation (M&S) (de-fined previously in Preface) as it is usedthroughout this guidebook (from DoDD5000.59, DoD Modeling and SimulationManagement).

A model is a physical, mathematical or oth-

The word model brings to mind several pre-conceived mental descriptions. Some re-member the plastic pieces of an airplane, caror ship; all connected in an orderly fashionto be pulled apart, assembled, trimmed andsometimes painted. Others may rememberwatching an old war movie when the ma-neuver plans were drawn in the dirt; or aselaborate as a wood and metal scale modelof a Nazi Germany’s officer recreation andrecovery facility used for mission rehearsal.There are many different descriptions anddefinitions of a model throughout the DoD.

Since World War II, technology has ad-vanced at an ever increasing rate. This ex-plosion of technology is moving faster thanproducts can be acquired by the acquisitioncommunity. With technology, we can ac-complish what was considered five years agoto be impossible. With the utilization ofmicroproces- sors, several new spin-off tech-nologies are popping up. These technolo-gies range from an internetted nervous sys-tem type communication link of satellites,under sea and land fiber optic networks tocustomized biological organisms used to eatocean oil spills.

This explosion is a dream come true formany. Virtual reality (an interactive, com-

2-2

erwise logical representation of a system,entity, phenomenon or process.

Simulation is twofold: a method for imple-menting a model over time; and a techniquefor testing, analyzing, or training in whichreal-world systems are used, or where real-world and conceptual systems are repro-duced by a model.

2.1 Today’s Applications

The user community is very broad, span-ning not only those involved in the employ-ment of weapon systems, but also those in-volved in all phases of systems acquisition.Primary developers of today’s models arewar colleges, industry, DoD laboratories anduniversities.

There is a difference of opinion over mod-eling techniques; the amount of detail re-quired; and the value of analytical models,simulations, games and field exercises. Inexamining these differences of opinion, thereare a variety of models and each user has adifferent application in mind for the samemodel. This guide will not cover all the differ-ent opinions, nor provide an opinion on theuse of one model over another. The deci-sions regarding the specific use of modelsand simulations within a given program be-long to the readers. Consideration should begiven to the particular activities within theirprograms, policies, and the guidelines and in-formation contained within this guidebook.

The user community is divided into the fol-lowing functional areas: education, trainingand operations; research and development;test and evaluation; analysis; and productionand logistics.

Specific applications for each of the func-tional areas are broken out below.

• Education, training and operations —Re-creation of historical battles, doctrine andtactics development, command and unittraining, operational planning and rehearsal,and wartime situation assessment.

• Research and development — Require-ments definition, engineering design supportand systems performance assessment.

• Test and evaluation — Early operationalassessment, development and operationaltest design; and operational excursions andpost-test analysis.

• Analysis — Campaign analysis, forcestructure assessment, system configurationdetermination, sensitivity analysis and costanalysis.

• Production and logistics — Systemproducibility assessment, industrial baseappraisal and logistics requirements deter-mination.

This list is not exhaustive and not mutuallyexclusive, but it certainly is representativeof the many applications of M&S through-out the user community.

An important aspect of the utilization ofmodels and simulations is the application formore than one purpose. An example is themanned weapon system simulation network(SIMNET) initially developed for training,however now is being used in the develop-ment of doctrine and tactics.

2.2 Systems Acquisition Process

The goal of the systems acquisition processis to deploy, in a timely manner, and sustainan effective system that satisfies a specificuser’s need at an affordable cost. As statedpreviously, an assumption was made the

2-3

reader has some understanding of the sys-tems acquisition process as it exists today.However, the research discovered some con-fusion within different portions of the pro-cess.

In order to ensure the reader has a clear un-derstanding of the systems acquisition pro-cess and information on the requirementsgeneration system, this guidebook willbriefly review those areas.

Readers that feel comfortable with the sys-tems acquisition process may wish to skipto section 2-3 for a review of the require-ments generation system—an often over-looked, but vital part of the process.

The DoDD 5000.1 establishes broad poli-

cies governing defense systems acquisitionprograms. It states that the three decision-making support systems must interact andinterface with each other in order for theprocess to work effectively. The three sys-tems illustrated in Figure 2-12 are: 1) require-ments generation, 2) acquisition manage-ment and 3) planning, programming andbudgeting system (PPBS).

The first formal interface between the re-quirements generation system and the ac-quisition management system occurs atmilestone 0, supported by the Joint Require-ments Oversight Council (JROC). MilestoneI marks the first formal interface betweenthe acquisition management system and thePPBS. This milestone also marks programinitiation.

Figure 2-1. Three Major Decision Making Support Systems

Three Major Decision Making Support Systems

Systems Acquisition Process

Planning,Programming,& Budgeting

System

AcquisitionManagement

System

RequirementsGeneration

System

2-4

The acquisition management system, therequirements generation system and thePPBS all interface to meet decision pointsat major milestone; and during each Pro-gram Objective Memorandum (POM) cycle.

2.3 Requirements Generation System

Requirements generation is based on a con-tinuing process of assessing the capabilitiesof the current force structure to meet theprojected threat; while taking into accountopportunities for technological advance-ment, cost savings and changes in nationalpolicy or doctrine. Figure 2-23 depicts thisprocess.

The requirements generation system con-sists of four distinct phases: definition, docu-mentation, validation and approval.

2.3.1 Definition

The definition phase is an identification ofa deficiency, or mismatch between currentcapabilities and the future (projected) threat.This process is known as a mission area as-sessment (MAA). Once identified, thesedeficiencies need to be resolved.

The first alternative is to change the organi-zation; doctrine or tactics; or requirementsfor additional training. These alternatives are

Requirements Generation System

Figure 2-2. Requirements Generation System

THREATFUTURE

CURRENTFORCE

LOOK FORNON-MATERIEL

SOLUTIONS

LOOK FORMATERIEL

SOLUTIONS

Changes inPolicy/Doctrine

MISSION AREA ANALYSIS

IDENTIFY DEFICIENCIES

RECOMMENDATIONS IMPLEMENTATION

TechnologicalAdvancement

Analysis

CostReductionAnalysis

2-5

called non-materiel alternatives. They are in-vestigated first because of their relativelylow cost and ease of implementation. Shouldnon-materiel alternatives prove incapable ofresolving the deficiency, the next alternativeis a materiel solution.

If a materiel solution is pursued, definitiontakes the form of translating the deficiency,or technological opportunity, into a MissionNeed Statement (MNS). The MNS definesthe need in broad operational capabilityterms, and the format is described in DoD5000.2-M, “Defense Acquisition Manage-ment Documentation and Report,” Febru-ary 1991, part 2.

2.3.2 Documentation

This documentation phase is the formalpreparation of documents that must be co-ordinated with the affected authority. TheMNS originator will determine whether theprogram is a potential major defense acqui-sition program (MDAP), which requiresJROC action.

Materiel solutions are considered in the fol-lowing order of precedence:

1) Use or modification of an existing U.S.military system.

2) Use or modification of an existingcommercially developed or allied system(Non-Developmental Item (NDI) approach).

3) Cooperative research and developmentprogram with one or more Allied nations.

4) New Joint-Service program.

5) New Service-unique development pro-gram.

The MNS must be coordinated with affectedServices, Commanders in Chief (CINCs),and agencies, as well as any necessaryhigher headquarters, before forwarding tothe validation authority.

2.3.3 Validation

The validation phase is a formal review pro-cess of the documentation by an operationalauthority (other than the user) to confirm theidentified need and operational requirement.As a minimum, the operational validationauthority reviews the MNS, confirms that anon-materiel solution is not feasible, as-sesses the Joint Service potential, and for-wards the MNS with a recommendation tothe milestone decision authority (MDA) formilestone 0 action.

The Defense Intelligence Agency (DIA) willvalidate the potential threat to be counteredand certify intelligence requirements forpotential ACAT I programs.

For any C4 capability, the Director, J-6, JointStaff, must certify the need and operationalrequirements for conformance to joint C4policy and doctrine, interoperability, archi-tectural integrity, and joint potential beforeapproval. Validation is a necessary, but notsufficient, step for approval.

2.3.4 Approval

The approval phase is the activity to formallyor officially sanction the identified need and/or operational capabilities described in thedocumentation. Approval also certifies thatthe requirements documentation has beensubject to the uniform process of the DoD5000 series and the Chairman of the JointChiefs of Staff (CJCS) Memorandum ofPolicy No. 77 (MOP 77).

2-6

Should the MNS be approved by the JROC,it will be forwarded to the Defense Acquisi-tion Board (DAB) with the recommendationthat concept direction studies be initiated.Based on a review by the DAB Committeeand the DAB, the Under Secretary of De-fense, Acquisition and Technology(USD(A&T)) makes the final decision as towhether or not the warfighting deficiencywarrants the initiation of concept directionstudies. The resulting milestone 0 decisionis documented in an Acquisition DecisionMemorandum (ADM), signed by theUSD(A&T), the Defense Acquisition Ex-ecutive (DAE) .

The MNSs for potential ACAT I level pro-grams which are disapproved are returnedto the originating service/agency.

The validation and approval authority forACAT II, III, and IV MNSs are the service(or defense agency) chiefs or CINC of therespective Unified or Specified Command.Approved MNSs for less than ACAT I levelprograms are forwarded to the componentacquisition executive for action.

Each Service has their own methodologyfor meeting the guidance provided by theCJCS MOP 77 and the DoD 5000 seriesdocuments.

1. The Road to 2012. (1993). Washington, DC: U. S.Department of Transportation.

2. DODD 5000.1, Part 2, Par. A.

ENDNOTES

3. Schmoll, J. H. (March 1993). Introduction to De-fense Acquisition Management. (2nd Ed.).Ft.Belvoir, VA: DSMC Press.

3-1

33ORGANIZATION AND POLICY3.1 OFFICE OF THE SECRETARY OF DEFENSE ORGANIZATION

AND MANAGEMENT OF MODELING AND SIMULATION

increased emphasis on M&S. During histestimony before the House Armed ServicesCommittee in connection with thePresident�s budget, March 30, 1993, theSecretary of Defense stated that DoD is plan-ning to undertake acquisition reforms thatare even bolder than the Packard Commis-sion proposed. Goals included streamliningand improving acquisition, simplifying ac-quisition guidance and establishing joint ci-vilian-military requirements. The use ofmodels and simulations is viewed as a vitalaspect of acquisition reform.

Models and simulations are powerful toolsto improve the acquisition process�such asimproved, up-front analysis and definitionof requirements; early simulation of the de-velopment process (design, test, manufac-ture, support, etc.); common shared databases; and the potential for conducting mul-tivariate analysis in the complex �what if�world of the program manager (PM). Thebottom line is M&S saves resources.

Working toward that end, the Director of

3.1.1 Background

Two offices under the Secretary of Defense(SecDef) joined efforts to improve theDepartment�s management and technologyin the areas of modeling and simulation(M&S). This led to the Deputy Secretary ofDefense (DepSecDef) approving a plan tostrengthen M&S applications and assigningthe Under Secretary of Defense for Acqui-sition and Technology (USD(A&T)), the re-sponsibility. This plan also established theExecutive Council on Modeling and Simu-lation (EXCIMS) and the Defense Model-ing and Simulation Office (DMSO).

The EXCIMS is a flag officer level advi-sory group to USD(A&T) on M&S policy, ini-tiatives, standards and investments. The DMSOprovides a full-time focal point for M&S ac-tivities, and promulgates USD(A&T) directedM&S policy, initiatives and guidance: promot-ing cooperation among DoD components.

Downward trends in the DoD acquisitionbudget have provided a forcing function for

3-2

Defense Research and Engineering (DDR&E),established the Acquisition Task Force onModeling and Simulation (ATFM&S), June30, 1993, for a one year effort. The ATFM&Scharter was to recommend actions thatwould lead to the more effective, integrateduse of M&S throughout the acquisition pro-cess. Figure 3-1 illustrates the relationshipamong these organizations.

3.1.2 Vision for Modeling and Simulation

The stated vision of the EXCIMS reads:

Defense modeling and simulation will providereadily available, operationally valid environ-ments for use by DoD components

� to train jointly, develop doctrine and

tactics, formulate operational plans and as-sess war fighting situations

� as well as to support technology assess-ment, system upgrade, prototype and fullscale development and force structuring.

Furthermore, common use of these environ-ments will promote a closer interaction be-tween operations and acquisition commu-nities in carrying out their respective respon-sibilities. To allow maximum utility and flex-ibility, these modeling and simulation envi-ronments will be constructed from afford-able, reusable components interoperatingthrough an open systems architecture.1

More specifically, simulated warfightingenvironments can be constructed; allowing

Figure 3-1. Office of the Secretary of Defense Organization for M&S Management

OfOfOfOfOffice of the Secrfice of the Secrfice of the Secrfice of the Secrfice of the Secretaretaretaretaretary of Defensey of Defensey of Defensey of Defensey of DefenseOrOrOrOrOrganization for M&S Managementganization for M&S Managementganization for M&S Managementganization for M&S Managementganization for M&S Management

OSDOSDOSDOSDOSD

USD (A&T)USD (A&T)USD (A&T)USD (A&T)USD (A&T) DSBDSBDSBDSBDSB

EXCIMSEXCIMSEXCIMSEXCIMSEXCIMS

DDR&EDDR&EDDR&EDDR&EDDR&E AAAAATFM&STFM&STFM&STFM&STFM&S

ARPARPARPARPARPAAAAA DMSODMSODMSODMSODMSO

3-3

the Services to train the forces, plan opera-tions and assess the status of actual opera-tions. The training would be joint, span sev-eral echelons, involve large simulated forces,bridge large geographic regions, and involvesenior commanders, as well as, units and theindividual soldiers. Status monitoring wouldbe based on electronic sand tables wheredisposition of friendly and enemy forces canbe realistically portrayed and the conse-quences of those courses of action simu-lated.

Similarly, these environments could supportthe acquisition process. Simulation test bedswould allow new concepts to be exploredand system requirements to be refined be-fore bending metal and committing to ex-pensive alternate developments. Operationaltesting (OT) can be augmented by embed-ding live tests in a broader simulated envi-ronment allowing a more comprehensivesystems test. Together these simulations canbe used to test new systems or technologies;allowing the doctrinal and tactical implicationsof the new capabilities to be explored prior toany procurement or prototype development.

3.1.3 Office of the Secretary of Defense(OSD) Policy Organizations

3.1.3.1 Under Secretary of Defense,Acquisition and Technology (USD(A&T))

The USD(A&T) is the principal staff assis-tant and advisor to the SecDef for all mattersrelating to the DoD acquisition systems, re-search and development, production, logis-tics, military construction and procurement.The USD(A&T) issues plans, programs,policies and procedures for DoD M&S, incoordination with the DoD components.This office is responsible for establishingDoD-wide M&S goals and objectives andan investment strategy to achieve them.

3.1.3.2 Director, Defense Research andEngineering (DDR&E)

The DDR&E is responsible to theUSD(A&T) for matters pertaining to re-search and engineering planning, invest-ment, implementation and development. TheDDR&E is the chair for the EXCIMS, andprovides EXCIMS-developed recommenda-tions and advice to the USD(A&T).

3.1.3.3 Assistant Secretary of Defense,Program Analysis and Evaluation ASD(PA&E):

The ASD(PA&E) is the principal staff assis-tant to the SecDef for DoD PA&E. TheASD(PA&E) is responsible for the critical re-view of requirements, performance and life-cycle costs of current and proposed weaponsystems. Review of the Cost and OperationalEffectiveness Analysis (COEA) falls directlyunder the purview of the ASD(PA&E). TheASD(PA&E) also provides leadership to theCost Analysis Improvement Group (CAIG).

3.1.3.4 Director, Operational Test andEvaluation (DOT&E)

This office prescribes policies and proce-dures governing the conduct of OT&E. Pro-vides independent assessments and reportsas required by current statutes. This officealso establishes policy for the application ofM&S in support of OT&E.

3.1.4 Related Organizations

3.1.4.1 Executive Council on Modeling andSimulation (EXCIMS)

The EXCIMS is the advisory forum for applica-tion and control of M&S, providing recommen-dations to the USD(A&T) on DoD M&S goals,objectives and investment strategy. The EXCIMSoversees development of DoD M&S plans, pro-

3-4

grams, policies and procedures.

3.1.4.2 Defense Modeling and SimulationOffice (DMSO)

The DMSO serves as the DoD focal pointfor M&S. The DMSO also serves as theExecutive Secretariat for the EXCIMS andfacilitates ATFM&S meetings; disseminatespolicy and guidance to the Services; andmaintains the Defense Modeling and Simu-lation Information System.

3.1.4.3 Acquisition Task Force on Modelingand Simulation (ATFM&S)

A short term (one year) acquisition taskforce, the ATFM&S was chartered to rec-ommend action that would lead to the moreeffective, integrated use of M&S through-out the acquisition process.

3.1.5 OSD Policy Documents

3.1.5.1 DoDD 5000.59, DoD Modeling andSimulation (M&S) Management (January 4,1994)

The DoDD 5000.59 establishes DoD policy,assigns responsibilities and prescribes pro-cedures for the management of M&S. It es-tablishes the DoD EXCIMS and the DMSO.

3.1.5.2 Director, Operational Test and EvaluationPolicy for the Application of Modeling andSimulation in Support of Operational Test andEvaluation (January 24, 1989)

This document provides policy guidance for theuse of M&S in support of OT&E. It describesappropriate application of models and simula-tions, and guidelines to ensure credible results.

3.1.6 Additional Information

Points of contact (POCs) for M&S within OSDand DoD agencies are found in Appendix A.

3.2 JOINT STAFF ORGANIZATION

AND MANAGEMENT OFMODELING AND SIMULATION

3.2.1 Background

Joint models and simulations (JM&S) arethose models and simulations that representJoint and Service forces, capabilities, mate-rials and services used in the joint environ-ment; or by two or more of the military Ser-vices. The JM&S support assessments, in-puts to the Planning, Programming, andBudgeting System (PPBS), joint profes-sional military education and training, real-time operational support, wargaming, andreconstruction of operations and exercises.

In the past, DoD�s wargaming and assess-ment infrastructure were a heterogeneousmixture of systems and applications. These

were independently developed, supported,and operated by the combatant commands:the Services and the Joint Staff.

The objective for the future envisions a co-ordinated development of JM&S to lever-age both the considerable investment in ex-isting capabilities and the competence thatexists in widespread centers and users - thatallow the exchange of M&S capabilitiesacross the JM&S community. The core con-cept is centralized management with decen-tralized execution, called Distributed Mod-els and Simulations (DMS).

Tailoring of specific JM&S capabilities fromdistributed M&S centers (e.g. USEUCOM

3-5

Warrior Preparation Center, CFC-KoreaBattle Simulation Center, Joint WarfightingCenter, USPACOM Joint Task Force Simu-lation Center, Army National SimulationCenter, Air Force Blue Flag, etc.), throughfully operational distribution networks to sup-port specific requirements for a specific useris a concept called DMS pinpoint support.

This netted pooling of expertise will (1) al-low M&S to evolve as the Service/combat-ant command needs and capabilities evolve,(2) maximize effectiveness by allowing the

Service and combatant command areas ofexpertise to directly affect the tools avail-able for joint tasking and (3) maximize effi-ciency by enabling multiple, non-redundant,parallel operations that will help reduce theoverhead of traditional stand-alone centers.This will also reduce competition for re-sources, assets and applications assistance.

3.2.2 Organization

Figure 3-22 depicts how JM&S will receivedistributed support from the Services and

Figure 3-2. Joint Staff Organization and Management of Modeling and Simulation

Joint StafJoint StafJoint StafJoint StafJoint Staff Orf Orf Orf Orf Organization and Managementganization and Managementganization and Managementganization and Managementganization and Managementof Modeling and Simulationof Modeling and Simulationof Modeling and Simulationof Modeling and Simulationof Modeling and Simulation

APPLICATIONS - TRAINING - ASSESSMENTS - OBSERVATIONS/PLANS

UNCCFC KOREA

SERVICES

PACOM

SOUTHCOM

CENTCOM

TRANSCOMFORSCOM

STRATCOM

SPACECOM

ACOM

SOCOM

EUCOM

AGENCIES

CJCS

ACQUISITION OPERATIONS

THE JOINT STAFF

STRATEGY &POLCY

3-6

ENDNOTES

participate in its policy development. A con-figuration management (CM) proponencywill be established for each Service andjoint-community developed JM&S. Central-ized sourcing and maintenance of commondata bases required for joint training and as-sessment will be developed and distributedvia distributed networks.

The Joint Modeling and Simulation Execu-tive Panel (JMSEP) chaired by the DeputyDirector for Technical Operations, J-8, pro-vides a forum for JM&S community inter-action and cooperation.

3.2.3 Key Organizations

The Vice Director, Joint Staff, is the senior in-formation resource management official andsupervises the JM&S master planning pro-cess and approves the investment plan. AllStaff Directorates play a role in developingand using JM&S within their areas of re-sponsibility, but the key JM&S organizationson the Joint Staff are J-7 and J-8, both of whichprovide representatives to the DoD EXCIMS.

3.2.3.1 J-7, Operational Plans andInteroperability Directorate

The J-7 coordinates development of policyapplications for the use of JM&S in supportof joint doctrine development, joint training,and joint exercises. Through the Joint WarfightingCenter, J-7 establishes liaison with combatantcommands and Services for JM&S require-

ments supporting the above activities.

3.2.3.2 J-8, Force Structure, Resources andAssessment Directorate

The J-8 coordinates the development of policyapplications for the use of JM&S in all areas in-volving analysis and analytical methodologies.The J-8 develops and maintains the JM&S mas-ter plan and JM&S investment plan, Chairs theJMSEP and formulates policy for the develop-ment, acquisition and life-cycle management ofJM&S in support of joint applications.

3.2.4 Key Documents

3.2.4.1 Joint Modeling and SimulationSummary, (October 1992)

This summary presents the existing joint userresults of a survey of the community as a firststep in the initiative to establish and maintainthe JM&S Master Planning Process.

3.2.4.2 Joint Modeling and SimulationEvolutionary Overview, (February 1994)

This document provides the JM&S commu-nity with a vision and associated policy ini-tiatives and technical activities to support thecombatant commands and Joint Staff.


Joint Staff POCs for M&S may be found inAppendix A.

3.3 DEPARTMENT OF THE ARMY ORGANIZATION ANDMANAGEMENT OF MODELING AND SIMULATION

3.3.1 Background

The Army has used explicit representationsof combat systems, combat and other pro-

cesses for a number of years. Long beforethe advent of computers, the Army reliedheavily upon information derived from theconduct of simulations. History contains ex-

3-7

amples of planning and rehearsing missionsusing sand-tables, and developing forcestructure and tactics using substitutes forweapons and weapon systems (such as jeepsfor tanks and broom handles for rifles) dur-ing the Louisiana Maneuvers of 1940.

The sophistication of tools used to model andsimulate combat systems and combat processesevolved over the years. Many major field andcommand post exercises were conducted us-ing probability tables and the rolling of die tosimulate the occurrence of events. Rapid ad-vances in computer technology sped the evo-lution of M&S into the synthetic environment.

During the evolution from predominantlyphysical representations, the Army�s use ofM&S continued to support a variety of appli-cations for five major purposes: education,training, and military operations; analysis; re-search and development (R&D); test and evalu-ation (T&E); and production and logistics.

The Army introduced management of itsmodels and simulations in the early 1980swith TRADOC Regulation 5-4. Today, theArmy executes management of its M&Sthrough the Army Model and SimulationManagement Office (AMSMO) with policyprescribed in Army Regulation (AR) 5-11.

Until as recently as the mid-1980s, theArmy�s development and use of M&S wereaccomplished on an as-needed and as-af-forded basis. In the late 1980s, the adventof distributed simulation technology, led bythe Advanced Research Projects Agency(ARPA), introduced the Army to SimulationNetwork (SIMNET). The SIMNET, coupledwith a downward trend in defense budgets,led the Army to seek M&S applications si-multaneously addressing more than one ofthe purposes mentioned above.

In distributed simulation technology, theArmy recognized the potential for linkingM&S of various types, fidelities and reso-lutions, and of establishing these linkagesfrom geographically separated sites both inCONUS and overseas. In addition, the Armyhas been assigned the role of executive agentfor DoD in developing the technology andinfrastructure to support military applica-tions of distributed interactive simulation(DIS).

The Army�s model and simulation hierar-chy is defined at the following levels -

1. Theater and global models and simu-lations, typically aggregated at brigade leveland above.

2. Division/corps level models and simu-lations, typically aggregated at maneuverbattalion level.

3. Combined arms task force, brigadelevel and below, high resolution models andsimulations, typically representing indi-vidual weapon systems.

4. Individual item level models, which in-clude those down to a weapon�s subsystemand component level.

Army policy requires the use of Touchstonemodels and simulations (in effect, modelsof choice) to the maximum extent possible.Commanders are required to ensure adequateresearch into the ability of existing models andsimulations, preferably Touchstones, to meetemerging requirements prior to initiation ofan M&S development activity.

The Army has brought on line and populatedthe MOdels and Simulations: Army IntegratedCatalog (MOSAIC) as a hypertext tool avail-able to all users and developers to browse

3-8

through an array of existing models andsimulations. The MOSAIC offers a meansto begin a search of existing M&S resources.Readers interested in more detail on MO-SAIC are referred to Appendix B.

Beginning in 1994, the Army implementeda policy that requires all Army acquisitionstrategies for ACAT I and II programs (sub-sequently expanded to Advanced Technol-ogy Demonstrations (ATD) and Top LevelDemonstrations (TLD)) to include a Simu-lation Support Plan (SSP). In this plan, thePM must lay out the functional requirementsfor M&S to support engineering and com-bat developments, test and evaluation, train-ing and military exercises to support the pro-gram. The PM must also develop an M&Sacquisition strategy identifying resourcesrequired to bring the M&S to fruition.

3.3.2 Organization

Figure 3-3 depicts the relationship betweenselected activities involved in making policyfor the Army�s management, developmentand use of models and simulations.


This section describes the functions of thoseactivities specifically involved in recom-mending, establishing or promulgatingM&S policy for the Army.

3.3.3.1 Deputy Under Secretary of the Armyfor Operations Research (DUSA(OR))

The DUSA (OR) serves as Headquarters, De-partment of the Army (HQDA) proponent forArmy policy on M&S, and establishes proce-dures and policy to support DoD M&S efforts.

Figure 3-3. Army Organization For M&S Policy Making and Dissemination

ArArArArArmy Ormy Ormy Ormy Ormy Organization for M&Sganization for M&Sganization for M&Sganization for M&Sganization for M&SPolicy Making and DisseminationPolicy Making and DisseminationPolicy Making and DisseminationPolicy Making and DisseminationPolicy Making and Dissemination

SEC ARMY

AMSEC

ASA (RDA) CSADUSA (OR)

AMSMO MACOMS

3-9

3.3.3.2 Army Model and SimulationExecutive Council (AMSEC)

The AMSEC recommends M&S policyguidance to the DUSA(OR); defines thescope of, and approves activities to be in-cluded in, the Army Model ImprovementProgram (AMIP) and Simulation Technol-ogy Program (SIMTECH); and nominatesprojects for incorporation by the DMSO.

3.3.3.3 Army Model and SimulationManagement Office

The AMSMO serves as executive secretariatfor the AMSEC, and executive agent for ex-ecution of the AMIP and SIMTECH pro-grams; promulgates M&S managementpolicy and implementing procedures, suchas DA Pamphlet 5-11, Verification, Valida-tion and Accreditation (VV&A) of ArmyModels and Simulations; develops and pub-lishes the Army Model and Simulation Mas-ter Plan; maintains MOSAIC; and acts as theArmy�s focal point for dealing with DMSO.

3.3.4 Army Policy RelatedDocuments

3.3.4.1 Army Regulation (AR) 5-11, ArmyModel and Simulation Management Program(AMSMP)

Policy for Army M&S management is prescribedin AR 5-11. The Army�s program for managementof models and simulations is formalized in AR 5-11. This regulation, and the Army Model and Simu-lation Master Plan take into account emergingDoD initiatives concerning management of thesetools, prescribes policies and responsibilities fortheir management within the Army and describesorganizational responsibilities.

3.3.4.2 Department of The Army Pamphlet(DA PAM) 5-11, Verification, Validation, and

Accreditation of Army Models and Simulations

The DA PAM 5-11 provides procedures toassist model developers, proponents andsponsors to conform to the policies in AR5-11. Specifically, it provides guidance forthe development, execution and reporting ofall VV&A activities. It also addresses datacertification in reference to proper M&S use.

3.3.4.3 Army Model and SimulationMaster Plan

This master plan provides a blueprint forinvestment of Army resources in an effec-tive, efficient fashion in collaboration withother members of the DoD community. Italso addresses the environment which willallow M&S technologies to advance the ca-pabilities of a smaller, power projectionArmy; capable of land force dominance.

3.3.4.4 OASA(RDA) Policy memorandum,�Simulation Support to Army Acquisition,�May 24, 1993

This memorandum instructs the PM to pre-pare a Simulation Support Plan for each ACATI and II program going for milestone review.

3.3.4.5 DA PAM 70-XX, Army AcquisitionProcedures, Part 5, Section H, �SimulationSupport to Army Acquisition.� (Draft)

Section H of DA PAM 70-XX providesguidance to assist in the preparation and sub-mission of the SSP.


Readers needing more information on theM&S functions performed by the manyArmy commands and agencies are referredto Appendix B and the Army Model andSimulation Master Plan. Copies of that docu-

3-10

ment can be obtained from the AMSMO.

(N812) of the Assessments Division withinthe Office of the Deputy Chief of NavalOperations (Resources, Warfare Requirementsand Assessments). The Naval Modeling andSimulation Program then encompassedM&S activities associated with R&D and

3.4 DEPARTMENT OF THE NAVY ORGANIZATION ANDMANAGEMENT OF MODELING AND SIMULATION

3.4.1 Background

Management of M&S in the Navy originallywas focused on wargaming and tactical readi-ness. In 1992, management of M&S movedto the Assessment and Affordability Branch

Figure 3-4. Department of the Navy Modeling and Simulation Management

DeparDeparDeparDeparDepartment of the Navytment of the Navytment of the Navytment of the Navytment of the NavyModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation Management

SECRETSECRETSECRETSECRETSECRETARARARARARYYYYYof theof theof theof theof theNANANANANAVYVYVYVYVY

UNDER SECRETUNDER SECRETUNDER SECRETUNDER SECRETUNDER SECRETARARARARARYYYYYof theof theof theof theof theNANANANANAVYVYVYVYVY

Chief ofChief ofChief ofChief ofChief ofNaval OperaitonsNaval OperaitonsNaval OperaitonsNaval OperaitonsNaval Operaitons

(CNO)(CNO)(CNO)(CNO)(CNO)

CommandantCommandantCommandantCommandantCommandantMarine CorpsMarine CorpsMarine CorpsMarine CorpsMarine Corps

Assistant SecrAssistant SecrAssistant SecrAssistant SecrAssistant SecretaretaretaretaretaryyyyyNavy (RDA)Navy (RDA)Navy (RDA)Navy (RDA)Navy (RDA)

OfOfOfOfOffice of Navalfice of Navalfice of Navalfice of Navalfice of NavalResearResearResearResearResearchchchchch

PrPrPrPrProgram Executiveogram Executiveogram Executiveogram Executiveogram ExecutiveOfOfOfOfOfficersficersficersficersficers

EXEC. AGENTEXEC. AGENTEXEC. AGENTEXEC. AGENTEXEC. AGENTMCCDCMCCDCMCCDCMCCDCMCCDC

EXEC. AGENTEXEC. AGENTEXEC. AGENTEXEC. AGENTEXEC. AGENTN6N6N6N6N6

M&S ADVISORM&S ADVISORM&S ADVISORM&S ADVISORM&S ADVISORYYYYYCOUNCILCOUNCILCOUNCILCOUNCILCOUNCIL

(USMC/USN)(USMC/USN)(USMC/USN)(USMC/USN)(USMC/USN) DON M&S Management OfDON M&S Management OfDON M&S Management OfDON M&S Management OfDON M&S Management OfficeficeficeficeficeN6-x/DirN6-x/DirN6-x/DirN6-x/DirN6-x/Directorectorectorectorector, MCMSMO, MCMSMO, MCMSMO, MCMSMO, MCMSMO

Space & ElectrSpace & ElectrSpace & ElectrSpace & ElectrSpace & ElectroniconiconiconiconicWWWWWarararararfarfarfarfarfare Dire Dire Dire Dire Directorateectorateectorateectorateectorate

(N6)(N6)(N6)(N6)(N6)

USMC PolicyUSMC PolicyUSMC PolicyUSMC PolicyUSMC Policy& Coor& Coor& Coor& Coor& Coord Ofd Ofd Ofd Ofd Officeficeficeficefice(Dir(Dir(Dir(Dir(Dir., MCMSMO)., MCMSMO)., MCMSMO)., MCMSMO)., MCMSMO)

TTTTTechnicalechnicalechnicalechnicalechnicalSupporSupporSupporSupporSupport Grt Grt Grt Grt Groupoupoupoupoup(SP(SP(SP(SP(SPAAAAAWWWWWAR 31-x)AR 31-x)AR 31-x)AR 31-x)AR 31-x)

USN Policy &USN Policy &USN Policy &USN Policy &USN Policy &CoorCoorCoorCoorCoord Ofd Ofd Ofd Ofd Officeficeficeficefice

(N6-x)(N6-x)(N6-x)(N6-x)(N6-x)

Functional AreaFunctional AreaFunctional AreaFunctional AreaFunctional AreaManagersManagersManagersManagersManagers

Naval AirNaval AirNaval AirNaval AirNaval AirSystems CommandSystems CommandSystems CommandSystems CommandSystems Command

Naval SeaNaval SeaNaval SeaNaval SeaNaval SeaSystems CommandSystems CommandSystems CommandSystems CommandSystems Command

Space & NavalSpace & NavalSpace & NavalSpace & NavalSpace & NavalWWWWWarararararfarfarfarfarfareeeee

Systems CommandSystems CommandSystems CommandSystems CommandSystems Command

COMOPTEVFORCOMOPTEVFORCOMOPTEVFORCOMOPTEVFORCOMOPTEVFOR

NANANANANAWC (WWC (WWC (WWC (WWC (Weapons Div)eapons Div)eapons Div)eapons Div)eapons Div)

NANANANANAVC (AirVC (AirVC (AirVC (AirVC (Aircraft Div)craft Div)craft Div)craft Div)craft Div)

NANANANANAWC (TWC (TWC (TWC (TWC (Training Sys Div)raining Sys Div)raining Sys Div)raining Sys Div)raining Sys Div)

NUWCNUWCNUWCNUWCNUWC

NUWC (CarNUWC (CarNUWC (CarNUWC (CarNUWC (Carderderderderderock Div)ock Div)ock Div)ock Div)ock Div)

NSWC (Dahlgren Div)NSWC (Dahlgren Div)NSWC (Dahlgren Div)NSWC (Dahlgren Div)NSWC (Dahlgren Div)

NSWC (Coastal Systems Station)NSWC (Coastal Systems Station)NSWC (Coastal Systems Station)NSWC (Coastal Systems Station)NSWC (Coastal Systems Station)

NSWC (Indian Head Div)NSWC (Indian Head Div)NSWC (Indian Head Div)NSWC (Indian Head Div)NSWC (Indian Head Div)

NSWC (PorNSWC (PorNSWC (PorNSWC (PorNSWC (Port Hueneme Div)t Hueneme Div)t Hueneme Div)t Hueneme Div)t Hueneme Div)

NSWC (Crane Div)NSWC (Crane Div)NSWC (Crane Div)NSWC (Crane Div)NSWC (Crane Div)

NCCOSCNCCOSCNCCOSCNCCOSCNCCOSC

3-11

T&E, education and training, production andlogistics, and analysis.

Space and Naval Warfare Systems Com-mand, (SPAWAR) provided support includ-ing maintaining a catalog for Naval Model-ing and Simulation.

A Naval Warfare Analytical/Modeling andSimulation Oversight council called �TeamMike� was formed to provide guidance andcoordination for the Navy�s diverse modelingand simulation efforts. This organizationincludes representatives from the functionaloffices within the Office of the Chief ofNaval Operations, as well as throughout theNavy support organizations, i.e., Naval Sys-tems Commands and Naval Warfare Centers.

3.4.2 Organization

Because of the wide-ranging spectrum of M&Sand breadth of the functional disciplines whichM&S support; in 1994 the Navy decided to es-tablish a management structure, shown in Fig-ure 3-4, which addresses oversight; policy andtechnical support; and the users of M&S.


The key organizations and activities in-cluded in the M&S management structureare described below.

3.4.3.1 Modeling and Simulation AdvisoryCouncil

Membership in this Council is from both theNavy and Marine Corps. This Council will:

� Advise on the formulation of the De-partment of the Navy (DON) M&S vision;

� Guide the development of policy, co-ordination and technical support; and

� Promote the use of DON-wide commonsupport services.

3.4.3.2 Department of the Navy Modelingand Simulation Management Office

This office consists of Navy and MarineCorps Policy and Coordination offices anda Technical Support Group.

3.4.3.3 Navy and Marine Policy andCoordination Offices

Each office is responsible for:

� Writing and maintaining their respec-tive Service Modeling and Simulation Mas-ter Plans and Investment Strategies;

� Coordinating plans, programs, policiesand procedures across functional areas; and

� Maintaining instructions and standardsnecessary to manage M&S.

The Head of the Navy Policy and Coordi-nation Office resides within the Space andElectronic Warfare Directorate (N6). Thehead of the Marine Corps Policy and Coor-dination Office is the Director, Marine CorpsModeling and Simulation Management Of-fice (MCMSMO).

3.4.3.4 Technical Support Group

The Technical Support Group provides tech-nical advice and assistance in the executionof M&S activities throughout the DON. Thisgroup�s responsibilities include:

� Maintaining an automated Navy M&SMaster Catalog and acting as Naval POC forinput into other DoD M&S catalogs;

� Providing management of the DON

3-12

VV&A process;

� Advising in standards and protocols tobe used;

� Assisting in selection and developmentof M&S applications;

� Building common services, tools anddata bases for future development;

� Supporting development of the multi-service common simulation framework andnecessary infrastructure and tools, such ascommon interface units, catalogs, etc.; and

� Designing distributed simulation exer-cises.

The Technical Support Group is led by the Spaceand Naval Warfare Systems Command Model-ing, Simulation, and Analysis Group (SPAWAR31) and is augmented as necessary by personnelfrom various Naval organizations, such asSYSCOMs, warfare centers, laboratories and fed-erally funded research and development centers.

3.4.3.5 Executive Agents

The Director, Space and Electronic Warfare(N6) and the Commanding general, MarineCorps Combat Development Command act asExecutive Agents; which participate in theModeling and Simulation Advisory Counciland jointly provide oversight to the DON Mod-eling and Simulation Management Office.

3.4.3.6 Functional Area Managers

Functional Area Managers are designatedfor the following areas: acquisition, researchand development; doctrine and training sys-tems; test and evaluation; logistics; opera-tions; training and education; and assess-

ment. These managers generally residewithin the Chief of Naval Operations orAssistant Secretary of the Navy organiza-tions (the exception is the Doctrine andTraining Systems Manager, who is the Com-mander, Naval Doctrine Command). Thefunctional area managers deal with issuesacross a spectrum of organizations.

� Participate as members of the M&S Ad-visory Council.

� Provide vision for employment of M&S tocommands, facilities and organizations workingwithin their specified functional areas.

� Promote and support participation injoint and cooperative research, development,acquisition and operation of M&S systems;technologies; and capabilities within theirfunctional areas.

� Participate in the development of ser-vice M&S master plans, investment plansand other service planning documents.

3.4.3.7 M&S Developers and Users

M&S developers and users, such as systemcommands, warfare centers and laboratories,are responsible for ensuring compliancewith applicable policies and procedures.

3.4.4 DON Policy Related Documents

The Navy has prepared instructions whichimplement the Navy M&S policies and man-agement structure.

3.4.4.1 SECNAVINST 5200.XX: Depart-ment of the Navy Modeling and SimulationProgram (Draft)

This document describes the managementstructure, organizational responsibilities and

3-13

prescribes policy and guidance for the DONM&S program.

3.4.4.2 SECNAVINST 5200.XX; Verifica-tion, Validation & Accreditation of Modelsand Simulations (Draft)

This instruction establishes policy and pro-cedures, and assigns responsibilities for

M&S VV&A activities within the DON.


A list of other DON organizations, docu-ments and a description of the Navy Mod-eling and Simulation Catalog are containedin Appendix C.

3.5 MARINE CORPS ORGANIZATION AND MANAGEMENTOF MODELING AND SIMULATION

3.5.1 Background

The need to carry out diversified missions, andtrain and equip its forces within a constrainedDoD resource environment has led the Ma-rine Corps to examine more efficient meth-ods to define requirements; evaluate solutions;and refine system and equipment designs. Standalone models and simulators, and advanced dis-tributed simulation are recognized as provid-ing the basis for improving training, acquisi-tion decisions, test and evaluation, force struc-ture decisions and requirements definition.

The Marine Corps has taken a series of stepsto accelerate employment of M&S technolo-gies. These include: development of a battlestaff training tool, the Marine Air-GroundTask Force (MAGTF) Tactical WarfareSimulation (MTWS); development of anM&S Master Plan to provide a coherentstrategy for implementing the Marine Corpssimulation environment, and establishmentof the Marine Corps Modeling and Simula-tion Office as a central focal point for M&S.

3.5.2 Organization

The management of M&S in the MarineCorps is centered in the Marine Corps Mod-eling and Simulation Management Office

(MCMSMO) located within the Training &Education Division of the Marine CorpsCombat Development Command(MCCDC). To facilitate communication,integration and decision making, the MarineCorps M&S management structure as shownin Figure 3-5 parallels the DoD M&S man-agement structure outlined in DoD Direc-tive 5000.59.

The Marine Corps M&S management struc-ture consists of an Executive SteeringGroup, the Marine Corps Modeling andSimulation Management Office , and theMarine Corps Modeling and SimulationWorking Group (MCMSWG).



3.5.3.1 Executive Steering Group

A General Officer steering group designatedby the Assistant Commandant of the Ma-rine Corps (ACMC). The functions of thisgroup include:

� Oversees the Marine Corps M&S pro-

3-14

gram;

� Approves Modeling and SimulationMaster Plan and Investment Strategy;

� Approves and provides resources forM&S initiatives as part of the POM; and

� Approves VV&A policies and proce-dures.

3.5.3.2 Marine Corps Modeling & Simula-tionManagement Office (MCMSMO)

The MCMSMO is the Marine Corps� singlefocal point for M&S; providing managerialoversight of the Marine Corps M&S pro-gram, managing the Marine Corps M&SMaster Plan and Investment Strategy, andsupporting development of VV&A policiesand procedures

The Director, MCMSMO is also the Direc-tor of the USMC Policy and Coordination

Figure 3-5. Marine Corps Modeling and Simulation Management

Marine Corps Modeling and Simulation ManagementMarine Corps Modeling and Simulation ManagementMarine Corps Modeling and Simulation ManagementMarine Corps Modeling and Simulation ManagementMarine Corps Modeling and Simulation Management

CommandantCommandantCommandantCommandantCommandantU.S. Marine CorpsU.S. Marine CorpsU.S. Marine CorpsU.S. Marine CorpsU.S. Marine Corps

Marine Corps CombatMarine Corps CombatMarine Corps CombatMarine Corps CombatMarine Corps CombatDevelopment CommandDevelopment CommandDevelopment CommandDevelopment CommandDevelopment Command

(MCCDC)(MCCDC)(MCCDC)(MCCDC)(MCCDC)

Marine Corps Computer &Marine Corps Computer &Marine Corps Computer &Marine Corps Computer &Marine Corps Computer &TTTTTelecommunications Activityelecommunications Activityelecommunications Activityelecommunications Activityelecommunications Activity

(MCCT(MCCT(MCCT(MCCT(MCCTA)A)A)A)A)

Marine CorpsMarine CorpsMarine CorpsMarine CorpsMarine CorpsSystems CommandSystems CommandSystems CommandSystems CommandSystems Command(MARCORSYSCOM)(MARCORSYSCOM)(MARCORSYSCOM)(MARCORSYSCOM)(MARCORSYSCOM)

Marine CorpsMarine CorpsMarine CorpsMarine CorpsMarine CorpsOperational TOperational TOperational TOperational TOperational Test & Evaluationest & Evaluationest & Evaluationest & Evaluationest & Evaluation

Activity (MCOTEA)Activity (MCOTEA)Activity (MCOTEA)Activity (MCOTEA)Activity (MCOTEA)

Marine Corps TMarine Corps TMarine Corps TMarine Corps TMarine Corps TacticalacticalacticalacticalacticalSystems SupporSystems SupporSystems SupporSystems SupporSystems Support Activityt Activityt Activityt Activityt Activity

(MCTSSA)(MCTSSA)(MCTSSA)(MCTSSA)(MCTSSA)

ExecutiveExecutiveExecutiveExecutiveExecutiveDirectorDirectorDirectorDirectorDirector

Amphibious WAmphibious WAmphibious WAmphibious WAmphibious WarararararfarfarfarfarfareeeeeTTTTTechnology Direchnology Direchnology Direchnology Direchnology Directorateectorateectorateectorateectorate

(A(A(A(A(AWT)WT)WT)WT)WT)

DirectorDirectorDirectorDirectorDirectorPrPrPrPrProgram Supporogram Supporogram Supporogram Supporogram Supporttttt

MARCORSYSCOMMARCORSYSCOMMARCORSYSCOMMARCORSYSCOMMARCORSYSCOMM&S Focal PointM&S Focal PointM&S Focal PointM&S Focal PointM&S Focal Point

Executive Steering GrExecutive Steering GrExecutive Steering GrExecutive Steering GrExecutive Steering Groupoupoupoupoup

ACMCACMCACMCACMCACMCCG, MCCDCCG, MCCDCCG, MCCDCCG, MCCDCCG, MCCDC

CG,MARCORSYSCOMCG,MARCORSYSCOMCG,MARCORSYSCOMCG,MARCORSYSCOMCG,MARCORSYSCOM

MCMSMOMCMSMOMCMSMOMCMSMOMCMSMO

Marine CorpsMarine CorpsMarine CorpsMarine CorpsMarine CorpsM&S WM&S WM&S WM&S WM&S Working Grorking Grorking Grorking Grorking Groupsoupsoupsoupsoups

TTTTTraining &raining &raining &raining &raining &Education DivEducation DivEducation DivEducation DivEducation Div.....

Marine CorpsMarine CorpsMarine CorpsMarine CorpsMarine CorpsModeling & SimulaitonModeling & SimulaitonModeling & SimulaitonModeling & SimulaitonModeling & Simulaiton

OfficeOfficeOfficeOfficeOffice(MCMSMO)(MCMSMO)(MCMSMO)(MCMSMO)(MCMSMO)

WWWWWararararargaming &gaming &gaming &gaming &gaming &Combat Simulation DivCombat Simulation DivCombat Simulation DivCombat Simulation DivCombat Simulation Div.....

MAGTF/ExpeditionarMAGTF/ExpeditionarMAGTF/ExpeditionarMAGTF/ExpeditionarMAGTF/ExpeditionaryyyyyTTTTTraining Centerraining Centerraining Centerraining Centerraining Center

3-15

Office supporting the DON Modeling andSimulation Management Office as previ-ously shown in Figure 3-5.

3.5.3.3 The Marine Corps Modeling &Simulation Working Group (MCMSWG).

The MCMSWG is chaired by the head ofMCMSMO and performs such functions as:supports information exchange across func-tional areas; participates in the developmentof M&S policies and procedures; providesinput to the M&S Master plan and Invest-ment Strategy; recommends M&S projectsfor inclusion in the POM; and recommendsVV&A policies and procedures.

The MCMSWG consists of representativesfrom functional organizations throughout theMarine Corps, and has five standing commit-tees: VV&A, CM, functional integration,M&S Information, and data bases/scenarios.

3.5.4 Marine CorpsPolicy Related Documents

The Marine Corps has prepared instructionswhich implement the Marine Corps M&Spolicies and management structure.

3.5.4.1 Marine Corps Modeling and Simu-lation Master Plan, Marine Corps Modeling& Simulation Management Office, (Draft as

of March 16, 1994)

This plan articulates the vision for the Ma-rine Corps simulation environment; de-scribes the technical objectives for MarineCorps constructive, virtual, and live simu-lations; and describes an implementationstrategy including policy and managementframework.

Referred to within the Marine Corps Mod-eling and Simulation Master Plan and its ap-pendices, but intended to be published asseparate documents are the following:

� Marine Corps Modeling and SimulationInvestment Plan, (projected date May 1995)

� Procedures and Guidelines for Verifi-cation, Validation, and Accreditation

� Procedures and Guidelines for Configu-ration Management of Marine Corps Mod-els and Simulators

� Marine Corps Modeling and Simula-tion Catalog


Other Marine Corps Organizations involvedwith the development, management, and/oruse of M&S are listed in Appendix D.

3.6 DEPARTMENT OF THE AIR FORCE ORGANIZATIONAND MANAGEMENT OF MODELING AND SIMULATION

3.6.1 Background

The Air Force has a long history of M&Sapplications. From the Blue Box flightsimulator called Pilot maker to the computeraided design (CAD) and computer aidedmanufacturing (CAM) development of the

B-2: but the road has been bumpy.

The Air Force Scientific Advisory Board(SAB) reported that the Air Force has manyexcellent examples of M&S; a growing needfor M&S to aid Air Force decision making;demonstrated low confidence in M&S by

3-16

decision makers; and a lack of coherentpolicy and structure.3

As the Services stand up to adopting moreM&S applications, significant organizationalchanges and new initiatives are occurring.

3.6.2 Organization

The Air Force has designated the Director-ate of Modeling, Simulation, and Analysis(MS&A)(HQ USAF/XOM) as the singlePOC for M&S issues and activities within

the Air Force. They also represent the AirForce in joint, multi-service and multi-agency M&S efforts. The HQ USAF/XOM,in conjunction with all Air Force user com-munities, is developing formal M&S policyfor the Assistant Vice Chief of Staff�s (HQUSAF/CVA) approval. Figure 3-64 showsthe Air Force�s organization for M&S.

The Air Force uses MS&A at five differentlevels:

1. Strategic/National Military Strategy level

Figure 3-6. Air Force Organization for Modeling and Simulation Management

Air ForAir ForAir ForAir ForAir Force Orce Orce Orce Orce Organization forganization forganization forganization forganization forModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation ManagementModeling and Simulation Management

ARMY &ARMY &ARMY &ARMY &ARMY &NANANANANAVY LABSVY LABSVY LABSVY LABSVY LABS

HQ USAF/TEHQ USAF/TEHQ USAF/TEHQ USAF/TEHQ USAF/TE

TEPTEPTEPTEPTEP TERTERTERTERTER

HQ USAF/LGHQ USAF/LGHQ USAF/LGHQ USAF/LGHQ USAF/LG

HQ AFMC/DOHQ AFMC/DOHQ AFMC/DOHQ AFMC/DOHQ AFMC/DO

DOPDOPDOPDOPDOP

HQ AFMC/LGHQ AFMC/LGHQ AFMC/LGHQ AFMC/LGHQ AFMC/LG

LGPLGPLGPLGPLGP

HQ USAF/XOHQ USAF/XOHQ USAF/XOHQ USAF/XOHQ USAF/XO

XOMXOMXOMXOMXOM

HQ AFMC/XRHQ AFMC/XRHQ AFMC/XRHQ AFMC/XRHQ AFMC/XR

XRXXRXXRXXRXXRXOASOASOASOASOAS

SAF/AQSAF/AQSAF/AQSAF/AQSAF/AQ

AQXAQXAQXAQXAQX AQTAQTAQTAQTAQT

HQ AFMC/ENHQ AFMC/ENHQ AFMC/ENHQ AFMC/ENHQ AFMC/EN

ENMENMENMENMENM

HQ AFMC/STHQ AFMC/STHQ AFMC/STHQ AFMC/STHQ AFMC/ST

STCSTCSTCSTCSTC

JDLJDLJDLJDLJDL

PrPrPrPrProduct Centersoduct Centersoduct Centersoduct Centersoduct Centers

ASCASCASCASCASC ESCESCESCESCESC HSCHSCHSCHSCHSC SMCSMCSMCSMCSMC

WLWLWLWLWL RLRLRLRLRL ALALALALAL PLPLPLPLPL

XRsXRsXRsXRsXRs SPOsSPOsSPOsSPOsSPOs

Logistics CentersLogistics CentersLogistics CentersLogistics CentersLogistics CentersTTTTTest Centersest Centersest Centersest Centersest Centers

OO-ALCOO-ALCOO-ALCOO-ALCOO-ALCOC-ALCOC-ALCOC-ALCOC-ALCOC-ALC

SM-ALCSM-ALCSM-ALCSM-ALCSM-ALCSA-ALCSA-ALCSA-ALCSA-ALCSA-ALC

WR-ALCWR-ALCWR-ALCWR-ALCWR-ALC

AEDCAEDCAEDCAEDCAEDC

AFDTCAFDTCAFDTCAFDTCAFDTC

AFFTCAFFTCAFFTCAFFTCAFFTC

HQ USAF/CVHQ USAF/CVHQ USAF/CVHQ USAF/CVHQ USAF/CVAAAAA

UserUserUserUserUserMAJCOMsMAJCOMsMAJCOMsMAJCOMsMAJCOMs

TIsTIsTIsTIsTIs

MODELING & SIMULAMODELING & SIMULAMODELING & SIMULAMODELING & SIMULAMODELING & SIMULATION TPIPTTION TPIPTTION TPIPTTION TPIPTTION TPIPT

FOURMOSTFOURMOSTFOURMOSTFOURMOSTFOURMOSTLaboratoriesLaboratoriesLaboratoriesLaboratoriesLaboratories

3-17

2. Theater/Campaign level

3. Mission level

4. Engagement/Sub-mission level

5. System/subsystem component (engi-neering) level

3.6.2.1 Air Force Requirements GenerationSystem

As stated earlier, each Service has their ownmethod for meeting the Chairman of theJoint Chiefs of Staff (CJCS) Memorandumof Policy No. 77 (MOP 77) and DoD 5000series guidance for requirements generation.The Air Force follows the same basic pro-cess as the guidance presents and adds a few

management tools to assist the internal co-ordination for all mission area stakeholders.Refer to Figure 3-75.

The Mission Area Assessment (MAA) iden-tifies the tasks to support mission objectivesof a strategy-to-task analysis.

The Mission Needs Analysis (MNA) docu-ments deficiencies in our ability to accomplishthose tasks. This document never gets to theDAB. It provides the basis for the OperationalRequirements Document (ORD). This docu-ment takes the task-to-need view.

The Mission Area Plan (MAP) outlines,through a series of roadmaps, required cor-rective actions. The MAPs provide the need-to-capability plans.

Figure 3-7. Air Force Requirements Generation System

Air ForAir ForAir ForAir ForAir Force Requirce Requirce Requirce Requirce RequirementsementsementsementsementsGeneration SystemGeneration SystemGeneration SystemGeneration SystemGeneration System

MissionArea

Assess-ment

MissionNeeds

Analysis

MissionAreaPlan

Acquisi-tion

Process

CombatCapability

MNS/ORDProcess

AF POMProcess

TechnologyMaster Process

IR&D Process

Technology

Force Structure

3-18



3.6.3.1 HQ USAF/XOM: Directorate forModeling and Simulation

This Directorate is the single POC in the AirForce for policy on modeling, simulationand analysis activity. Specifically, providessupport to the Major Commands and HQUSAF in modeling, simulation and analy-sis that involve Air Force plans, operationsand operational requirements.

3.6.3.1.1 XOME: Evaluation Support Divi-sion provides evaluation support for forcestructure analysis. Focus is on Studies andAnalysis (S&A) policy guidance on majorevaluation activities, including the COEAprocess and the test and evaluation process.

3.6.3.1.2 XOMT: Technical Support Divi-sion provides technology support; includingpolicy for architecture, standards and VV&A.

3.6.3.1.3 XOMW: Warfighting Support Divi-sion provides S&A policy guidance for edu-cation and training of Commanders andBattle Staffs; real-time and interactive.

3.6.3.1.4 AFSAA: Air Force Studies andAnalysis. The recent reorganization of HQUSAF/XO incorporated Air Force S&Aunder XOM as a Field Operating Agency.They provide force application support withcampaign analysis, theater air defense in-formation, weapons and tactics information,and force employment analysis; force en-hancement information and analysis for glo-bal deterrence, global mobility, space andC3I, and force support; and support forwargaming exercises.

3.6.3.2 HQ AFMC/XRX: Director forOpera-tional Requirements

Provides M&S direction/policy for AFMCacquisition M&S. This office is the singleAFMC POC for acquisition M&S corre-spondence with the HQ USAF and SAF of-fices and AFMC centers/field units.

3.6.3.3 Technical Planning Integrated Prod-uct Team (TPIPT)

The ESC/XRP chairs this team that is com-prised of members from all AFMC productcenters. The M&S TPIPT integrates andcoordinates AFMC M&S activities, providesresponses to higher headquarters and plansfor M&S tools that will support acquisition;while being accessible and beneficial towargaming and training.

3.6.4 Air Force Policy RelatedDocuments

3.6.4.1 AFPD 16-10, Modeling and Simula-tion (M&S) Management, (Draft, May 1994)

This directive provides general policy forM&S throughout the Air Force and assignsresponsibilities. It also implements DoDD5000.59, DoD Modeling and SimulationManagement.

3.6.4.2 AFI 16-1001, Verification, Validationand Accreditation, (Draft)

Provides specific responsibilities, proce-dures, formats and guidelines for VV&A.

3.6.4.3 AFI 16-1002, M&S Management,(Draft)

Provides specific responsibilities, proce-dures, formats and guidelines on the man-agement of M&S.

3-19

3.6.4.4 AFMCP 800-66, Air Force MaterielCommand Models and Simulations Guide,(July 1993)

This document provides guidance for M&Sapplications through the life cycle of AirForce weapon systems. The guide focuseson the systems engineering approach and isdesigned to support the Integrated WeaponSystem Management (IWSM) concept. Thescope is a general overview of M&S appli-cations, not detailed descriptions.

3.6.4.5 AFPD 10-6, Mission Needs andOperational Requirements, (January 1993)

This policy directive establishes generalpolicy, assigns oversight responsibility,implements DoD 5000 series documentsinto the Air Force and identifies AFI 10-601as a companion to this policy directive.

3.6.4.6 AFI 10-601, Mission Needs andOperational Requirements Guidance andProcedures, (1994)

The AF 10-601 provides guidance in pre-

paring, validating and approving MissionNeed Statements (MNS), ORDs (includingthe Requirements Correlation Matrix(RCM)), and COEA.

3.6.4.7 AFPD 10-14, Air Force Policy Direc-tive on Modernization Planning, (Draft)

This policy directive establishes the mod-ernization planing process to identify andcorrect deficiencies in mission and func-tional areas.

3.6.4.8 AFI 10-1401, Modernization Plan-ning Documentation, (Draft)

This instruction establishes responsibilitiesand defines major processes for develop-ment of MAPs used in the modernizationplanning process.


Additional information on relative docu-mentation, key organizations and points ofcontact are contained in Appendix E.

1. Executive Council on Modeling and Simulation. (May1992). Vision as stated in the Defense Modeling andSimulation Initiative. Washington, DC: Department ofDefense.

2. Joint Modeling and Simulation Evolutionary Overview,The Joint Staff (J8/RPPD). February 1994.

3. Air Force Sscientific Advisory Board ad-hoc com-mittee on Modeling and Simulation, December1991; and 1993 Summer Study for TMD.

4. HQ Air Force Materiel Command, Development Plan-ning Division, Deputy Chief of Staff for Requirements(HQAFMC/XRX), Wright-Patterson AFB, OH.

5. HQ Air Combat Command, Requirements Man-agement Division (ACC/DRM), Langley AFB,VA.

ENDNOTES

4-1

44CLASSIFICATION OF MODELS

AND SIMULATIONS

among the various classes of models andsimulations are becoming blurred �tech-nology allows linkage and interoperabilityamong the various classes of models andsimulations, and human interactions canspan across all the classes. Therefore, oneoften is not simply talking about a singlemodel or simulation, but rather hybridsformed from among two or more classes.

The Defense Science Board in 1992 definedan appropriate classification of models andsimulations similar to that depicted in Fig-ure 4-1.3 These classifications find usefulapplication to the systems acquisition pro-cess and have been elaborated upon in morerecent publications as follows:

�Constructive. Wargames, models and ana-lytic tools ...

Virtual. Systems simulated both physicallyand by computer. Real people fight [andtrain] on synthetic battlefields, interactingwith each other and with artifacts in thesimulation. Examples include individual air-craft [weapon system] simulators and vir-tual prototypes.

There were over 500 Models and Simula-tions (predominantly computer models)listed in the Twelfth Edition of the JointChiefs of Staff (J-8) Models and SimulationsCatalog1 alone. Many more models andsimulations are listed in other catalogsthroughout DoD or used in support of spe-cific programs without being included in anycatalogs or formal listings.

The intent of this guidebook is to describethe forms that models and simulations takeand their uses in acquisition; rather thanserve as an additional catalog.2

The discussion of the types of models andsimulations will begin by laying out a frame-work of model and simulation classes. Thisis to show the breadth that models and simu-lations encompass and to provide the acqui-sition community with an understanding ofterminology.

Before proceeding further, the reader is cau-tioned not to become too enamored with theterminology, nor should one try to fit everymodel or simulation neatly into one of theclasses. In the authors� opinion, the lines

4-2

Live. Operations with live forces and realequipment in the air, on the ground, on andbelow the sea. Also included are hardwareprototypes on instrumented ranges.�4

This Chapter will concentrate on furtherdescribing these three classes of models andsimulations; introducing the reader to whatis termed hierarchies of models and simu-lations, and providing a discussion of hybridapplications. Chapter 5 will then describe theapplication of modeling and simulation(M&S) across the acquisition life cycle, as wellas to various acquisition activities.

4.1 Constructive Models and Simulations

The models and simulations containedwithin this class currently represent the pre-

dominant form of M&S tools used withinor in support of a program office.

Constructive models and simulations con-sist of computer models, wargames and ana-lytical tools which are used across a rangeof activities. At the lowest levels, they maybe used for detailed engineering design andcosting, or subsystem and system perfor-mance calculations to support developmentof technical specifications. Higher levelmodels and simulations provide informationon the outcomes of battles or major cam-paigns involving joint or combined forces,identify mission needs and support opera-tional effectiveness analyses.

A variety of constructive models may be usedto represent a system and its employment at

Figure 4-1. Classes of Models and Simulations

Classes of Models and SimulationsClasses of Models and SimulationsClasses of Models and SimulationsClasses of Models and SimulationsClasses of Models and Simulations

DESCRIPTIONDESCRIPTIONDESCRIPTIONDESCRIPTIONDESCRIPTION CLASSESCLASSESCLASSESCLASSESCLASSES EXAMPLESEXAMPLESEXAMPLESEXAMPLESEXAMPLES

Wargames, models, • CEM (Army)analytical tools • Brawler (Air Force)

• ENWGS (Navy)• MTWS (Marine Corps)• Engineering, Cost, Support models

Systems simulated both physically • Aircraft/Vehicle/Ship simulatiorsand by computer. Troops in • Virtual Proptotypes (e.g. NLOS)simulators fight on synthetic • Appended Trainersbattlesfields.

Operations with live forces • REFORGERand real equipment in the field • Red Flag

• National Training Center• Strike University• Instrumented prototypes

CONSTRUCTIVECONSTRUCTIVECONSTRUCTIVECONSTRUCTIVECONSTRUCTIVE

VIRVIRVIRVIRVIRTUALTUALTUALTUALTUAL

LIVELIVELIVELIVELIVE

4-3

different levels of detail, from engineeringphysics of piece parts to aggregated combatforces in a campaign analysis.

Many constructive simulations may be per-formed either with or without human inter-action. Without human interaction, theymight be run in multiple iterations to pro-vide statistical confidence in the outcomesof the simulation. With human interaction,they are often referred to as wargamingsimulations and are used for battle staff train-ing or tactics development. The tactics de-veloped in such interactive simulations maythen be used for establishing tactics withinthe non-interactive simulations.

Within acquisition, the uses of constructivemodels and simulations include design andengineering trade-offs, cost, supportability,operational and technical requirements defi-nition and operational effectiveness assess-ments.

4.2 Virtual Simulation

4.2.1 Human-in-the-Loop

Virtual simulation brings the system (or sub-system) and its operator together in a syn-thetic, or simulated environment. Althoughthis document uses the term human-in-the-loop to represent these simulations, othernames include man-in-the-loop, warfighter-in-the-loop, or person-in-the-loop.

In a virtual simulation, the system may in-clude actual hardware which is driven(stimulated) by the outputs of computer simu-lations. As an example, a weapon system simu-lator may employ a near-real crew compart-ment with the correct equipment, controls anddisplay panels. A computer generated syn-thetic environment is then displayed on ascreen in front of the crew and reflected in

the crew compartment instrumentation anddisplays. Motion of the platform may bedriven by the computer simulation to repre-sent the system dynamics. Sounds of thesystem and equipment can also be dupli-cated. The operators are thereby immersedin an environment driven by the simulatorthat to them looks, feels, and behaves likethe real thing. During simulated missions,the crew must operate the equipment, receivecommands and control weapons just as in areal system.

Human-in-the-loop simulations provide abetter understanding of human reactions anddecision processes and man-machine inter-faces. They can provide a platform for crewtraining prior to live exercises and tests, orrealistic mission rehearsal in preparation foractual combat operations.

Linked to other simulators, the interactionof multiple weapon systems can be exam-ined, leading to changes in tactics or engage-ment rules. These simulations also providepowerful tools for evaluation of actual sys-tem hardware and software within realisticenvironments for developmental programs.

Human-in-the-loop simulations run in realtime, and hence fewer iterations may be per-formed than with non-interactive construc-tive simulations.

4.2.2 Virtual Prototypes

A more advanced concept for virtual simu-lation is on our doorstep�virtual proto-typ-ing. In this realm, a three-dimensional elec-tronic, virtual mockup, of a system or sub-system allows an individual to interface witha realistic computer simulation within a syn-thetic environment.

The representation is solely a computer

4-4

simulation rather than actual hardwareand may be applied in early prototypingwork to evaluate concepts; human-ma-chine-interfaces; or to allow designers,logistics engineers and manufacturingengineers to interface with the same de-sign. Such an approach supports Inte-grated Product and Process Development(IPPD) or concurrent engineering, by pro-viding a common platform from whichall functional disciplines can work.

This concept of the designer, operator, main-tainer and manufacturer all interacting withthe same realistic three-dimensional repre-sentation of the system will become moreprevalent in future acquisition. For a detaileddescription of the capabilities of virtualprototyping, the reader should examine ref-erence 4 at the end of this chapter.

4.3 Live Simulations

�Everything is simulation except combat.�5

Live exercises where troops use equipmentunder actual environmental conditions ap-proaches real life in combat. The live simu-lation provides a testing ground to providelive data on actual hardware and softwareperformance in an operational environment.

These data also can be used to validate themodels and simulations used in an acquisi-tion program. This form of simulation pro-vides the stress and decision-making that isassociated with human-in-the-loop simula-tion. The introduction of multiple types ofplatforms allows for evaluation of actual in-teraction and interoperability. However, as-sembling the personnel and equipment andconducting a live simulation is a resourceintensive enterprise requiring time, fundsand people.

Constructive and virtual simulations may

already have been conducted prior to livesimulations to plan the tests or exercises,identify critical issues, rehearse the missionor train the participants. They may also beused to analyze results after the test, or aug-ment tests to address scenarios that may notbe feasible due to safety or environmentalreasons. With the high cost of live simula-tions (tests), the use of other, less resourceintensive forms of M&S is a smart prepara-tion tool. For example, an air-to-air missilein development might be valued at $1M, anda training torpedo firing could cost up to$50K. As an integral part of test planningand support, M&S will allow a programmanager (PM) to use such valuable assetsmore efficiently. For even greater benefitsto their programs, managers must insure thatlive simulations include adequate instrumen-tation. The data thereby collected will servetwo important purposes: further validationof models and simulations; and providing�ground truth� data to support post-exercisedebriefs.

As the reader has seen within the previousdiscussion, human interaction may be a partof any of the classes of M&S. The acquisi-tion program manager may choose to em-ploy human interaction in M&S to satisfytwo functions:

� Determination of human decision mak-ing or logic patterns and their impact on sys-tem performance and effectiveness. Simu-lations of any class requiring human inputmay serve this function.

� Identification and refinement of human-machine interfaces. This results from simu-lations which allow for the human to act aspart of the system, such as in manned simu-lators or live exercises.

These three classes of models and simula-

4-5

tions (constructive, virtual and live) may befound in varying levels of detail to supportactivities ranging from detailed engineeringdesign to the military utility of a new sys-tem or technology on the battlefield. To de-scribe the different levels of models andsimulations used to support these activities,a hierarchy of models and simulations is in-troduced.

4.4 Hierarchy of Models and Simulations

Models and simulations support acquisitionprogram activities ranging from design tooperational effectiveness assessments. This

assortment of tasks requires a suite ofmodels and simulations with differinglevels of detail suited to their particularapplication. These models and simula-tions form what may be called a hierar-chy of models and simulations.

Hierarchies of models and simulations aredescribed in documented form6 and alsofound in undocumented form throughout theDoD.

The authors have found that these hierarchiesare similar in concept and vary only in detail.Some extend to higher levels, including na-

HierarHierarHierarHierarHierarchy of Models and Simulationschy of Models and Simulationschy of Models and Simulationschy of Models and Simulationschy of Models and Simulations

Figure 4-2. Hierarchy of Models and Simulations

Air WingEW

AircraftSystem

Air Vehicle

Fire Control

Radar

Air Wings

Corps

BattleGroups

Joint/CombinedForces

A/C

DesignManufacturingCostSupport DevelopmentTech Req'ts Dev.

Operational Req'ts Development

Effectiveness Analysis

Tactics Development

Mission Planning & Rehearsal

Test & Evaluation

ActualPerformance

IncreasingResolution

IncreasingAggregation

ComparativeResults

(System/Subsystem/Component)

(One-on-One)

(Many-on-Many)

(Groups of Systems - Force Package)

Theater/Campaign

Mission/Battle

Engagement

Engineering

RESOLUTION FUNCTIONS SUPPORTED MODELS & SIMULATIONS FORCE OR SYSTEM LEVEL

4-6

tional policy and force structure planning,while others extend down to include actualtesting. This document describes a hierar-chy that is representative of those that thereader may come across or use. This hierar-chy is depicted in Figure 4-2, alongside aforce level and system work breakdownstructure (WBS) to indicate the system levelthat corresponds with the level of analysisto be performed.

The levels within this hierarchy include:

� Engineering: for design, cost, manu-facturing and supportability. Provides mea-sures of performance (MOP).

� Engagement: for evaluating system ef-fectiveness against enemy systems. Providesmeasures of effectiveness (MOE) at the sys-tem-on-system level.

� Mission/Battle: effectiveness of a forcepackage, or multiple platforms performinga specific mission. Provides MOE at theforce-on-force level.

� Theater/Campaign: outcomes of joint/combined forces in a theater/campaign levelconflict. Provides measures of value addedat the highest levels of conflict, sometimescalled measures of outcome (MOO).

4.4.1 Engineering Level Models and Simu-lations

Engineering level models and simulationsare concerned with the performance;

producibility; supportability; cost of com-ponents, subsystems and systems; and thetrade-offs associated therewith. At the engi-neering level there are literally thousands ofmodels and simulations including:

� Basic phenomenology such as aerody-namics, fluid flow, hydrodynamics, heattransfer, acoustics, fatigue, etc.

� Physics based models of components; sub-systems; and systems for design, performance,costing, manufacturing and supportability.

For acquisition, engineering level modelsand simulations provide the basis for designtrade-offs at the component, subsystem andsystem levels; support development of tech-nical design specifications; and support testand evaluation. Cost models provide devel-opment, production, and operations and sup-port costs. Support models can include reli-ability, availability and maintainability; levelof repair; and provisioning analyses. Manu-facturing models and simulations can pro-vide information on producibility of a par-ticular design, as well as, simulation of workflow on the factory floor and identifyfacilitization requirements.

These engineering level models indicateperformance capabilities, often termedMOP. Examples of these measures includeradar acquisition range, miss distance, range,payload or speed. Such performance param-eters might be used in the system and de-velopment specifications.

The representations of the system in higherlevel models and simulations should havetheir basis in these engineering level mod-els. It is in those higher level models andsimulations that the actual impacts ofweapon system performance on combat ef-fectiveness is evaluated.

○ ○

○

○

○

○

○

○

○

ENGINEERING

4-7

4.4.2 Engagement Level Models and Simu-lations

Engagement models and simulations repre-sent the system in a limited scenario, suchas one-on-one, few-on-few or sometimesmany-on-many. This level of simulationevaluates the effectiveness of an individualplatform and its weapons systems against aspecific target or enemy threat system. Thesemodels rely on system performance, kine-matics and sensor performance from theengineering level models and simulations.They provide, survivability, vulnerabilityand lethality results for measures of systemeffectiveness or for use in higher level mod-els. Detailed performance of the subsystemssuch as propulsion, combat systems, sen-sors, and guidance and control may be in-cluded and evaluated.

The outputs of engagement level models andsimulations indicate the effectiveness of sys-tems and subsystems in an engagement sce-nario and are termed MOE. Examples in-clude probability of kill, losses or missionaborts.

Acquisition uses of engagement level mod-els and simulations include identifying sys-tem effectiveness and performance to sup-port requirements documents (mission needstatement (MNS) and operational require-ments document (ORD)) and Cost and Op-erational Effectiveness Analyses (COEA),system level performance trade-offs, test andevaluation support, and evaluation of tacticschanges and new weapon concepts.

4.4.3 Mission/Battle Level Models andSimulations

Mission/battle level models and simulationsreflect the ability of a multi-platform forcepackage to accomplish a specific missionobjective, such as air superiority, interdic-tion or strike which might span a period ofhours. It might consist of an attacking forceof fighter and electronic warfare aircraft; acombined arms group attack or defense; orcarrier battle group operations consisting ofaircraft, ships and combat systems againstan integrated air defense (e.g., Surface-to-Air Missiles, enemy air assets).

In conjunction with human participation,mission/battle level simulations may be usedfor wargaming, training and tactics devel-opment.

The outputs of mission/battle level modelsand simulations are MOE. Typically at aforce package level rather than at the levelof the individual platform and its weaponsystem. Examples of these MOEs might in-clude loss exchange ratios, probabilities ofengagement or success in achieving a spe-cific mission objective.

The acquisition applications of such M&Sinclude analysis in support of requirementsfor the MNS or ORD; operational effective-ness analyses for alternatives evaluation inCOEAs; examination of interoperability andcompatibility issues; and in support of testand evaluation.

○

○

○

○

○

○

○

ENGAGEMENT

○

○

○

○

○

○

○ ○ ○ ○ ○ ○ ○

MISSION/BATTLE○

○

○

○

4-8

4.4.4 Theater/Campaign Models and Simu-lations

Theater/campaign models and simulationsrepresent combined force combat operationsand are used to determine the long term out-come of a major theater or campaign levelconflict. Forces are often represented as ag-gregations of lower level forces and systems.These models and simulations can identifymajor deficiencies in capabilities of forcestructures and employment alternatives.

Since these simulations usually encompasslonger periods of warfare they are morelikely to include sustainment representationswithin the model. These models usually re-quire the results of lower level (engineer-ing, engagement or mission/battle) modelsand simulations as inputs to generate theaggregated-force level capabilities. Somemay even have the capability to directly in-corporate more detailed models of specificsystems within their input architectures.

As with models and simulations within otherlevels of the hierarchy, theater/campaignlevel simulations might be run with humaninteraction. In this interactive mode, theymay by used as a wargaming tool for battlestaff training or tactics development.

Whereas the engineering level models areused to determine actual performance val-ues for the components, subsystems, or sys-tems being modeled; the higher level mod-els in the hierarchy are used to establishtrends, identify driving factors and obtain

relative comparisons of military utilityamong systems or groups of systems beinganalyzed.

The measures which result from theater/campaign level models and simulations aresometimes termed outcomes. Examples mayinclude force drawdowns or battle grouplosses, air superiority and ground forcemovements.

Acquisition applications of theater/cam-paign level models and simulations includeevaluation of force level combat outcomesin conducting Mission Area Assessments(MAA) leading to development of MNS; sup-port of COEAs; and evaluation of the impactsof new systems or operational concepts.

The hierarchy discussed above represents anintegrated framework for analysis of perfor-mance, effectiveness, tactics and doctrine,and conflict outcomes. Each level in thisintegrated framework is aimed at address-ing specific issues and relies on informationobtained in analyses conducted at other lev-els. Figure 4-3 summarizes the primary at-tributes of the various models and simula-tions within each level of the hierarchy alongwith representative examples.

4.5 Hybrid Models and Simulations

Up to this point, the classes (constructive,virtual and live) and a hierarchy of modelsand simulations have been described. Inmany applications, linkage among two ormore classes is actually used resulting inhybrid models and simulations. Such a hy-brid might employ constructive analyticalmodels to represent a threat or the kinemat-ics of a weapon in conjunction with actual(live) system hardware and software. Ex-amples of such hybrid applications includephysical simulations, stimulators, hardware/

○

○

○

○

○

○

○

○

○

THEATER/CAMPAIGN

○

○

○

○

○

○

○

○

4-9

Figure 4-3. Attributes and Uses of Models and Simulations within the Hierarchy

The

ater

/Cam

paig

n

Join

t/Com

bine

d

Hig

hly

Agg

rega

ted

Dow

n to

Indi

vidu

alE

ntiti

es (

Tan

k, S

hip,

A/C

)

Wee

ks-D

ays

• C

ampa

ign

Out

com

e (e

.g A

ir S

uper

iorit

y, F

orce

Dra

wdo

wns

, G

roun

d F

orce

Mov

emen

ts)

• A

ltern

ativ

e E

val (

CO

EA

)•

Req

uire

men

ts (

MN

S, O

RD

)•

Tac

tcs/

Em

ploy

men

t•

War

gam

ing

• B

attle

Sta

ff T

rain

ing

• S

usta

inm

ent I

ssue

s

Cam

paig

n E

xerc

ise

Mod

el

EN

WG

S, C

AA

M

AW

SIM

, Thu

nder

MT

WS

Eng

agem

ent

One

to a

Few

Frie

ndly

Ent

ities

Vs.

one

to a

few

Ene

my

Ent

ities

Eng

agem

ent

Indi

vidu

al e

ntiti

esD

etai

led

subs

yste

ms

Min

utes

-Sec

onds

• S

yste

m

Effe

ctiv

enes

s

(e.g

. Pro

babi

lity

of K

ill,

Los

ses,

Abo

rts,

Sur

viva

bilit

y, V

ulne

rabi

lity)

• A

ltern

ativ

e E

val (

CO

EA

)•

Req

uire

men

ts (

MN

S, O

RD

)•

Sys

tem

Effe

ctiv

enes

s•

Sys

tem

Tra

deof

fs•

Tac

tics,

rul

es o

f

Eng

agem

ent

• T

est S

uppo

rt

Eag

le, J

anus

SS

TO

RM

Bra

wle

r,E

SA

MS

Mis

sion

/Bat

tle

Mul

ti-P

latfo

rm, M

ulti

Tas

king

For

ce P

acka

ge

Som

e A

ggre

gatio

n or

Indi

vidu

al E

ntiti

es

Hou

rs-M

inut

es

• M

issi

on

Effe

ctiv

enes

s

(e.

g. L

oss

exch

ange

rat

ios,

pro

babi

litie

s o

f Eng

agem

ent)

• A

ltern

ativ

e E

val (

CO

EA

)•

Req

uire

men

ts (

MN

S, O

RD

)•

Dep

loym

ent

• W

eapo

ns In

tegr

atio

n•

Inte

rope

rabi

lity

• T

actic

s &

Ops

Con

cept

s•

Tra

inin

g &

War

gam

ing

Janu

s, E

agle

WE

PT

AC

, SIM

II

Sup

pres

sor,

EA

DS

IM

Eng

inee

ring

Sin

gle

Wea

pon

Sys

tem

s,S

ubsy

stem

s, C

ompo

nent

s

Hig

hly

deta

iled

- do

wn

to

indi

vidu

al p

iece

par

ts, t

heir

inte

ract

ion

& p

heno

men

olog

y

Mon

ths

- S

ub-S

econ

ds

• M

easu

res

of

Per

form

ance

of S

yste

m, S

ubsy

stem

s,

& C

ompo

nent

s (

e.g.

Mis

s D

ista

nce,

targ

et a

cqui

sitio

n ra

nge)

• C

ost,

Sup

port

abili

ty,

Pro

duci

bilit

y

• D

esig

n•

Sub

syst

em &

Com

pone

nt P

erfo

rman

ce &

Tra

deof

fs•

Spe

cific

atio

n R

eqtu

irem

ents

& C

ompl

ianc

e•

Cos

t, S

uppo

rt, P

rodu

cibi

lity

• T

est S

uppo

rt•

Fac

ilita

te IP

PD

Man

y, th

roug

hout

R&

D C

ente

rs,

Labs

, Con

trac

tors

, suc

h as

:

LEV

EL

OF

MO

DE

L

FO

RC

E

LEV

EL

OF

DE

TA

IL

TIM

E S

PA

N

OU

TP

UT

S

US

E

EX

AM

PLE

S(T

ypic

al U

ses)

CA

D/C

AM

Cos

t6-

DO

FH

W/S

WIL

Fac

tory

sim

ulat

ion

Sup

port

(LO

RA

, RA

M)

Att

rib

ute

s an

d U

ses

of M

od

els

amd

Sim

ula

tio

ns

wit

hin

th

e H

iera

rA

ttri

bu

tes

and

Use

s o

f Mo

del

s am

d S

imu

lati

on

s w

ith

in t

he

Hie

rar

Att

rib

ute

s an

d U

ses

of M

od

els

amd

Sim

ula

tio

ns

wit

hin

th

e H

iera

rA

ttri

bu

tes

and

Use

s o

f Mo

del

s am

d S

imu

lati

on

s w

ith

in t

he

Hie

rar

Att

rib

ute

s an

d U

ses

of M

od

els

amd

Sim

ula

tio

ns

wit

hin

th

e H

iera

r ch

yc

hy

ch

yc

hy

ch

y

4-10

software-in-the-loop (HW/SWIL) simula-tions and advanced distributed simulations(ADS).

4.5.1 Physical Simulation

Much of the discussion within this Chapterhas focused on electronic representations ofsystems and subsystems. Physical simula-tions may refer to a physical representationof the actual operating environment (e.g.temperature, humidity, shock, vibration,etc.) or physical models used in simulatingthe operation of a system or subsystem.Examples of physical simulation include:

� Munitions shock and vibration testingusing a simulated environment;

� Survivability/ vulnerability evaluationsusing prototype structures and environmen-tal conditions, such as high speed airflowover the structure, simulating flight condi-tions;

� A firing impulse simulator (a hydrauli-cally operated ram) used to provide an im-pulse to the gun barrel to physically repli-cate the shock of a round being fired fordurability and shock testing of the artillerypiece; and

� A test facility which simulates dynamicloads and motion for evaluation of trackedand wheeled vehicle suspension systems.Such a simulation might be driven using livesimulation data obtained from vehicles tra-versing over actual test courses.

These are just a representative sample ofphysical simulations. They can be usedthroughout the acquisition process at com-ponent, subsystem and system level; forevaluating new technologies and early pro-totypes; replicating field failures and veri-

fying fixes. The data collected from thesesimulations can also be useful for validat-ing models and simulations at all levels ofthe hierarchy.

4.5.2 Stimulation

In many instances, the actual signals repre-senting the outside environment to a test ar-ticle are not available. These signals mightrepresent a radar return from a target, a sig-nal from another weapon system such asbetween a platform and its weapons, orbackground noise in the midst of which thesystem must operate.

Simulations are therefore used to stimulatethe test article just as if the outside signalwas present. These stimulations may comefrom computer models, virtual simulationsor from live instrumented tests. They can be�hard-wired� into the system, or applied inthe same manner as in the real environment�e.g. through a sensor system. Stimulatorsmay be used in acquisition to simulatethreats or other phenomena either in a HW/SWIL simulation or a live simulation (test)of a weapon system.

4.5.3 Hardware/Software-In-The-Loop

The HW/SWIL simulations are often de-scribed as engineering level simulations.They typically consist of multiple classes ofsimulations. The HW/SWIL includes actualhardware and software, mathematical mod-els, and external stimuli used together todemonstrate the capability of a system orsubsystem to operate within an environmentsimulating actual conditions. A HW/SWILsimulation has proven to be an importanttool in system development, test and opera-tional support.

Figure 4-4 shows a typical HW/SWIL simu-

4-11

lation for a missile guidance system. In thisexample, both the guidance and control sec-tion are included in the simulation.7 Thestimulation to the guidance system is a simu-lated target radar return that the missilewould see in operation. Computer modelsrepresent the threat environment and pro-vide missile aerodynamics and kinematicswhich determine the target-to-missile posi-tioning. This HW/SWIL can include the ac-tual hardware, (sensors and processors); soft-ware (operational flight program); stimulator(threat simulation); and mathematical modelsof the missile dynamics�hence, a linkage ofmultiple types of simulation.

An extension of the missile simulation notedabove is the Guided Weapons Evaluation

Facility (GWEF) and the Preflight Integra-tion of Munitions and Electronic Systems(PRIMES) facility at Eglin AFB. The GWEFis a HW/SWIL facility used to evaluateweapon performance from launch to targetintercept. The PRIMES is a fighter aircraftsized anechoec chamber and associatedlaboratories supporting one-on-one ormany-on-one testing in a flight simulationcondition. Linkage of these two simulationfacilities provides the capability to simulateweapons on the aircraft; allowing an inte-grated simulation of target identification,tracking, and missile launch and flightthrough target intercept.

Another example of the use of HW/SWILsimulators is the Combat System Engineer-

Figure 4-4. Hardware/Software-In-The-Loop Simulation (HW/SWIL)

DATAACQUISITION

ANDDISPLAY

TELEMETRYCOMPUTER SYSTEM

SIMULATIONCONTROL

AERODYNAMICS

INITIALCONDITIONS

GEOMETRY

THREATENVIRONMENT

MODELS

KINEMATICS

RATES, ACCELERATIONS

MULTI-SPECTRALSTIMULATION

ENVIRONMENTGENERATOR

AUTOPILOTHARDWARE &

SOFTWARE

acGUIDANCEHARDWARE & SOFTWARE

HarHarHarHarHardwardwardwardwardware/Software/Software/Software/Software/Software-In-The-Loop Simulatione-In-The-Loop Simulatione-In-The-Loop Simulatione-In-The-Loop Simulatione-In-The-Loop Simulation(HW/SWIL)(HW/SWIL)(HW/SWIL)(HW/SWIL)(HW/SWIL)

4-12

ing and Analysis Laboratory (CSEAL) at theNaval Undersea Warfare Center. This simu-lation facility provides a human interactiveprototyping environment to support devel-opment, integration, evaluation of combatsystem technology products and develop-ment models for submarine combat systems.It uses the actual hardware and softwarebeing evaluated along with realistic simula-tions of the ocean environment, submarine,weapon systems and threat performance.The CSEAL allows for rapid prototyping,preparation of the prototype for live at-seatrials and timely analysis of the at-sea testdata.

The HW/SWIL is important in test andevaluation support, which is discussed inmore detail in Chapter 5. In development,HW/SWIL simulation can be used to dem-onstrate new technology; evaluate designs,concepts, and prototypes; and show the in-tegration of hardware and software. In sup-port of test programs, the HW/SWIL simu-lations allow for pre-test simulation to iden-tify test conditions. As a risk reduction mea-sure, they are used for checkout of actualhardware and software. These simulationsare also used to conduct post test analysisand to fill in a test matrix for conditionswhich are either not testable or for whichno test assets are available. The HW/SWILsimulations allow early identification andcorrection of developmental problems andallow one to identify and focus live tests(simulations) toward critical issues.

In production and operations support, theHW/SWIL simulation can be used for pro-duction lot sampling, P3I studies and evalu-ation of changes in operational software pro-grams. With the increasing reliance on soft-ware within weapon systems, and regularsoftware changes to enhance or modify per-formance, HW/SWIL has become a primary

tool in verifying the effects of softwarechanges on performance.

Typically, these simulations are run in real-time, with sensors, processors, guidance andcontrol systems operating. Many runs willbe performed to obtain statistically signifi-cant results for each condition - as an ex-ample, approximately 3200 HW/SWILsimulation runs were used to examine a testmatrix of 160 conditions for the Sparrowmissile8 (160 cases x 20 simulation runs percase = 3200 simulation runs). Over a periodof six years (1976-1982), a total of 39,300simulation runs were conducted to supportongoing missile development and conductparametric evaluations.

One challenge for the program office ortechnical support activity comes about whenlonger range systems are evaluated. Sincethese simulations run in real time, longerrange systems require more run time: po-tentially limiting the number of simulationruns that can be performed. Methods toevaluate only critical segments of a missionmight need to be pursued requiring in-depthexamination of initial conditions for eachsegment.

Other management considerations whichmust be integrated into the planning for useof HW/SWIL simulations include:

� Requiring appropriate interfaces (e.g.for signal inputs and outputs) designed intothe weapon system component so that it iscompatible with the simulation facility andthe desired data can be accessed;

� Facility and/or weapon system provi-sions to allow repeated operation in simula-tions resulting in usage far exceeding itsplanned mission time. This may, for example,include provisions for simulation facility

4-13

supplied cooling of weapon electronics;and

� Simulation facility development effortswhich may be necessary to properly supportnew weapon system technologies or newthreats.

4.5.4 Advanced Distributed Simulation(ADS)

The ADS is an emerging form of simula-tion that has demonstrated the ability to linkdifferent types of simulators at dispersedlocations; permitting the simulators and theircrews to conduct operations on the samesimulated battlefield environment.

The term distributed refers to geographicallyseparated simulations, each hosted on itsown computer without a central computer.These simulations are interactive, indicat-ing that simulations or simulators are linkedso that they can act upon one another in acommon environment (e.g. terrain, ocean,weather, etc.). The linked simulations maybe any combination of constructive, virtualand live; and likely to include human-in-the-loop simulations. The infrastructure withinwhich such distributed simulation takesplace is termed Distributed Interactive Simu-lation (DIS), which is discussed further inChapter 6.

Acquisition-related uses of distributed simu-lation include advanced concept and tech-nology evaluations within a simulated battle-field environment leading to requirementsdefinition. Performance and requirementstrade-offs may also be conducted to definesystem performance objectives, thresholds,manpower constraints, critical system char-acteristics and man-machine interfaces. Ad-vanced technology demonstrators, used inconjunction with simulations and live exer-

cises, may allow evaluation of technology,manufacturing and interoperability withother systems prior to major commitmentof resources. Distributed simulation mightalso support test and evaluation planning,test operator training, scenario development,execution (e.g. with additional simulatedforces) and post-test evaluation of results.9

To date, use of distributed simulation in ac-quisition has been limited. Two examplescited in 1993 by the Defense Science Boardinclude an operational test of the Non Line-Of-Sight (NLOS) missile and prototypingof M-1 main battle tank upgrades.10

The NLOS is a vehicle mounted missilewhich underwent early operational testingto assess the concept, requirements andhardware; along with its adaptation by forcesin engagements against helicopters. Thistesting required thirteen months and $15.5million. A parallel evaluation with the useof distributed simulation using an NLOSsimulator and helicopter simulators at twodifferent locations and operating on a com-mon terrain database was subsequently con-ducted. This evaluation was conducted inonly 3 months at a cost of $2 million.

In 1984, a test bed (a live simulation) of anupgraded M-1 tank was undertaken. After twoyears� time and expenditure of $40 million,the simulation was not yet functional. At thattime, the simulation was shifted to a modi-fied aircraft dome and successfully com-pleted in six months at a cost of $1 million.

In the future, distributed simulation will cer-tainly increase in capability to support theacquisition process. There has been an in-creasing emphasis in the M&S communityon linkages, within and among, classes ofmodels and simulations. There is a trendtoward local networks within given research

4-14

and development centers along with the aimof making simulations and such networkscompatible with the DIS environment to fa-cilitate distributed simulation. To take ad-vantage of this future ADS environment,managers in their M&S planning shouldconsider:

� The potential use of the ADS capabil-ity in support of their program activities;

� The likely requirement to make repre-sentations of their system available for usein others� scenarios; and

� The incorporation of appropriate data

communication standards to address in-ter-operability requirements of modelsand simulations they develop.

4.6 Summary

This chapter provided an overview of theclasses of models and simulations whichmay be used during the acquisition life cycle.As shown in Figure 4-5, a program willlikely employ a suite of models and simula-tions. The engineering level models will pro-vide measures of performance along withdesign, cost, producibility and supportabil-ity information for components, subsystemsor system. The military utility of the system

Figure 4-5. Relationships of Models and Simulations

Theater/Campaign

Mission/Battle & Engagement

(Outcomes)

(Effectiveness)

Engineering(Performance)

Data Bases

• Human in Loop (Wargaming)• Virtual Simulators• Virtual Prototypes

• Force Against Force• Sortie Generation (Sustainance)

• Fly Out Models• Intercept Models• Countermeasures

• Penetration• Force Pkg. vs. Threat• Platform vs. Platform• Threats (missiles, torpedos, guns)

• Range• Weight• Weaponry• Weight• Thrust/Drag

• Strength• Fatigue• Aerodynamics• Kinematics• Signature• Physics

• Vulnerability• Cost• Support• RAM• Manufacturing• Human-Machine Interface

• System• Subsystem• Weapons• Sensors

• Tactics • Mission Profiles• Scenarios• Ops Concepts

• Weather• Environment• Terrain• Targets

• Threats• Bases of Operation• Support Concept

Relationship of Models and SimulationsRelationship of Models and SimulationsRelationship of Models and SimulationsRelationship of Models and SimulationsRelationship of Models and Simulations

4-15

is evaluated within engagement and mis-sion/battle level models which indicateMOE. At the highest level, the outcomesof major conflicts involving combinedforces are evaluated within theater/cam-paign level models. Human-in-the-loop,virtual simulators and virtual prototypesmay provide information at all levels ofthe hierarchy. As in any analysis, the in-put data and assumptions are major driv-

ers in the results of all simulations. Justsome of the system, environment, threatand tactics data requirements are shownin Figure 4-5. The PM should rememberthat there is no single model or simula-tion that will suit all of a program�s needs.Each model or simulation has a specificpurpose for which it is intended and willprovide information at the requisite levelof detail to support specific activities dur-ing the program life cycle.

ENDNOTES

1. Force Structure, Resource, and Assessment Director-ate (J-8), The Joint Staff. (1992). Catalog of Wargamingand Simulation Models (12th ed.). Washington DC.

2. Note: For those readers who need further informa-tion on specific models and simulations, the ap-pendices of this guidebook contain points of con-tact within each Service, and sources of models andsimulations catalogs.

3. Office of the Under Secretary of Defense for Ac-quisition. (1993). Impact of Advanced DistributedSimulation on Readiness, Training andPrototyping. Washington, DC: Report of the De-fense Science Board Task Force on Simulation,Readiness, and Prototyping.

4. Garcia, A., Gocke, R. Jr., Johnson, N. (1994) Vir-tual Prototyping, Report of the DSMC 1992-1993Military Research Fellows, Ft. Belvoir, VA: DSMCp 34.

5. Office of the Under Secretary of Defense for Ac-quisition. (1993). Impact of Advanced DistributedSimulation on Readiness, Training andPrototyping. Washington, DC: Report of the De-fense Science Board Task Force on Simulation,

Readiness, and Prototyping.

6. AFMCP 800-66: AFMC Models and Simulations (M&S)Guide (July 1993). Wright-Patterson AFB, OH: HQAir Force Materiel Command; Military OperationsResearch Society. (1989). Military Modeling (2d ed.)W. P. Hughes, Jr. (Ed).

7. Adpated from notes provided by Emil J.Eichblatt,Naval Air Warfare Center (Weapons Division),Point Mugu, CA.

8. Eichblatt E. J. Jr. (1983). Performance Simulationin support of Test and Evaluation. Point Mugu, CA:Pacific Missile Test Center (now is Naval Air Weap-ons Center).

9. Distributed Interactive Simulation Master Plan.(Draft, 1994). Washington, DC: HQDA, Office ofthe Assistant Deputy Chief of Staff for Operationsand Plans.

10.Office of the Under Secretary of Defense for Ac-quisition. (1993). Impact of Advanced DistributedSimulation on Readiness, Training andPrototyping. Washington, DC: Report of the De-fense Science Board Task Force on Simulation,Readiness, and Prototyping.

5-1

55MODELING AND SIMULATION IN

SUPPORT OF ACQUISITION� Evolving broad mission needs into sys-

tem and subsystem requirements;

� Assessing alternative concepts whicheventually develop into a stable, producibledesign configuration; and

� Establishing initial affordability objec-tives which evolve into firm unit costs.

The DoD Instruction 5000.2 describes themanagement policies and procedures whichare to be applied throughout the acquisitionprocess and across the functional disciplines.An overview of the process, key activities andM&S application follows.

It is recommended that readers not simplyproceed directly to the discussion regardingthe particular phase their program currentlymay be in. Having skipped or already passeda phase in development does not negate theneed for conducting those past developmentand planning activities which can influencethe remainder of the program. Therefore, theuses of models and simulations, and theplanning activities discussed in conjunctionwith earlier phases of development, (particu-larly those described within phase 0), should

The use of modeling and simulation (M&S)within acquisition is a multi-dimensionalactivity which:

� supports the milestone decision pro-cess;

� supports multiple communities (opera-tor, developer, designer, manufacturer, sup-porter, tester and trainer); and

� consists of various classes and types ofM&S each with a specific purpose.

This chapter provides an overview of howM&S may be used across the phases of ac-quisition and a discussion of its applicationto specific acquisition related activities.

5.1 Modeling and Simulation Across theAcquisition Life Cycle

The DoD Directive 5000.1 establishes a dis-ciplined defense acquisition managementapproach to conducting stable, affordableacquisition programs that meet the user�sneeds. This management framework in-cludes:

5-2

be reviewed regardless of the current phaseof the program.

5.1.1 PRE MS 0

(Ref DoDI 5000.2, pg 3-2 through 3-6)

5.1.1.1 Focus: To determine mission need.

5.1.1.2 Activities: Mission area assessments(MAA) and mission need analyses (MNA) areconducted to examine the ability of existingsystems to satisfy mission objectives and ana-lyze capability improvements that are neededto meet deficiencies. Assuming that the defi-ciency can not be met by a change in doctrine,tactics, operational concepts, training or or-ganization (called �non materiel� solutions)a mission need statement (MNS) will be writ-ten describing the validated threat, deficien-cies, constraints, and potential alternatives toovercome the deficiency.

5.1.1.3 M&S: As with many other analysesperformed in support of a system, there isno single, stand-alone model or simulationto conduct the MAA. A suite of models andsimulations, along with supporting data in-cluding threat, environment, tactics, etc. arerequired.

� Engineering level models of new de-signs provide system and subsystem perfor-mance to support higher level models.

� Engagement and mission/battle levelsimulations evaluate the effectiveness ofdesigns in an operational environment andevaluate the consequences of different en-gagement tactics.

� Campaign/theater level models exam-ine the outcomes of new system capabili-ties, technologies, and tactics in extended,combined force conflicts.

� Human interaction in simulations maybe used to either identify the tactics for usein other models and simulations, or to ex-amine the operational impacts of alterna-tive tactical schemes or concepts of opera-tion. The reader is reminded of the discus-sion in Chapter 4 regarding the two uses ofhuman-interactive simulations: determina-tion of decision-making and its effects; anddefinition of human-machine interfaces.The purpose served here is within the firstcategory.

� Virtual prototypes also demonstratemilitary utility of new tactics, technologiesand systems.

This suite of models and simulations allowsfor analytical evaluation of tactics or con-cepts of operation changes with existingbaseline systems prior to evaluation of newsystems in accordance with DoDI 5000.2.1

The campaign/theater level models andsimulations, used in conjunction with the re-sults of the lower level models, will developthe data used to identify warfighting needsto be documented in the MNS. The engage-ment and mission level models will identifythe features and characteristics that providethe required capabilities with potential tosatisfy those needs.

Chapter 1 introduced a �risk cycle� and dis-cussed the concept of �operational risk�. TheM&S tools used at this time provide insightinto that risk and furthermore, are used toidentify either non materiel or materiel ap-proaches to mitigate those risks.

5.1.2 Phase 0 - Concept Exploration andDefinition (CED)


5.1.2.1 Focus: To define and evaluate the

5-3

feasibility of alternative concepts and assessthe relative merits of each concept.

5.1.2.2 Activities: During the CED phase,materiel alternatives are examined and a costand operational effectiveness analysis(COEA) is performed to determine the rela-tive cost effectiveness of those concepts.Operational requirements are defined anddocumented in an operational requirementsdocument (ORD) and concepts are definedat the system level resulting in a draft systemspecification. The critical system character-istics and operational constraints are defined,and the initial cost, schedule and performanceobjectives are developed. High risk areas andrisk management approaches are identified.Initial manufacturing and logistics supportplanning is begun. Training devices are alsoidentified in the ORD produced during thisphase. By the end of this phase the systemthreat is validated and an affordability assess-ment of the proposed new system is conductedto determine if the proposed new system fitswithin the Defense Planning Guidance andlong range investment plans.2

5.1.2.3 M&S: In the CED phase, many ofthe same classes and types of models andsimulations used to define requirements areagain employed to examine the capabilitiesof specific materiel alternatives at the engi-neering through campaign analysis levels.

� Engineering level models and simulationsof proposed technologies and new designs willbe used to project performance and examineperformance trade-offs. Logistics supportmodels will be used in defining the overall sup-port concept and operations and support costs.Cost models, will be used to determine lifecycle costs for use in the program cost esti-mate, for evaluation of alternatives in theCOEA, and to develop a preliminary designto unit production cost objective.

� Engagement and mission/battle modelsand simulations will be used to determinemission effectiveness measures for the pro-posed alternatives, again in support of theCOEA and ORD development.

� The theater/campaign level models andsimulations will be used to evaluate the pro-posed systems and determine their impacton the outcome of conflicts. These resultswill support the COEA and ORD, and beused to evaluate how well the proposedsystem(s) meet the previously identifiedneed.

� Human interactive simulations willcontinue to be used to develop and examinetactics and decisions within the above frame-work of constructive models.

� Virtual simulations might be used toevaluate new technologies, system conceptsand tactics in a realistic battle environment.This may range from a single simulator tomultiple simulators of a new system linkedto other friendly and enemy system simula-tors on a synthetic battlefield.

It is in this phase, that initial program plan-ning takes place and the key program docu-ments which set the stage for the entire pro-gram are developed. Consistency should bemaintained among all of the acquisitionmanagement documentation. This includesthe measures of effectiveness, measures ofperformance, and criteria in the OperationalRequirements Document, the cost and op-erational effectiveness analysis, the Test andEvaluation Master Plan (TEMP) and theacquisition program baseline. Paraphrasingthe words of one acquisition manager, �Ifyou cannot show how you intend to relatethe measures of effectiveness (MOEs) andmeasures of performance (MOPs) usedacross the COEA, ORD, acquisition pro-

5-4

gram baseline (APB), TEMP, as well as tothe MNS, then don�t bother showing up tothe DAB planning meeting.� This linkage isdepicted in Figure 5-1.

The TEMP, first prepared during this phase,must identify M&S resources which will beused to support development and opera-tional testing.3 The program office shouldconsider, at this time, how models and simu-lations will be used across all of the func-tional disciplines as the appropriate plans(e.g., integrated logistics support plan, sys-

tems engineering management plan andmanufacturing plan) are developed.

Modeling and simulation is a powerful toolto assist the acquisition manager in estab-lishing and maintaining a consistent relation-ship among MOEs, MOPs and programdocumentation. The activities initiated inthis phase will continue or be repeated withincreasing detail and specificity as the sys-tem design matures. The program officemust lay the groundwork for continuing ap-plication of models and simulations through-

Figure 5-1. Relationship of Program Documents, Needs and Measures

Relationship of PrRelationship of PrRelationship of PrRelationship of PrRelationship of Program Documents, Needs and Measurogram Documents, Needs and Measurogram Documents, Needs and Measurogram Documents, Needs and Measurogram Documents, Needs and Measureseseseses

M&S Establishes and Maintains ConsistencyM&S Establishes and Maintains ConsistencyM&S Establishes and Maintains ConsistencyM&S Establishes and Maintains ConsistencyM&S Establishes and Maintains Consistency

MAAMAAMAAMAAMAA MNSMNSMNSMNSMNS ORDORDORDORDORD SPECSSPECSSPECSSPECSSPECS

TEMPTEMPTEMPTEMPTEMPCOEACOEACOEACOEACOEA

• DEFICIENCY

• OPPORTUNITY

• OPERATIONALCAPABILITY NEED

• MOE

• MOP

• CRITICAL SYS.CHARACTERISTICS

• THRESHOLDS& OBJECTIVES

• MOP

• THRESHOLDS & OBJECTIVES

• MOP(TECHNICALPERFORMANCEREQUIREMENTS)

• MOE

• MOP (TECHNICAL REQUIREMETNS)

• CRITICAL SYSTEM CHARACTERISTICS

• THRESHOLDS & OBJECTIVES

• MOE(RELATE DIRECTLYTO SYSTEMPERFORMANCE)

APBAPBAPBAPBAPB

LEGENDLEGENDLEGENDLEGENDLEGEND

MAA: MISSION AREA ASSESSMENTMNS: MISSION NEED STATEMENT

COEA: COST AND OPERATIONAL EFFECTIVENESS ANALYSISORD: OPERATIONAL REQUIREMENTS DOCUMENTAPB: ACQUISITION PROGRAM BASELINE

TEMP: TEST AND EVALUATION MASTER PLANSPECS: TECHNICAL SPECIFICATIONS

MOE: MEASURES OF EFFECTIVENESSMOP: MEASURES OF PERFORMANCE

5-5

out the life cycle. This should include con-siderations of factors such as use and reuseof models and simulations, integration andinteroperability, and common data bases.Planning for model and simulation develop-ment should also address future compat-ibility with the synthetic battlefield throughDIS communication standards and the even-tual transition or application of developedmodels and simulations to the training envi-ronment. The main objective is to allow theprogram to later build upon models alreadydeveloped, thus reducing duplication andproviding for consistency through the phasesand among the documents and activitieswithin a given phase.

The use of models and simulations can sup-port early risk management activities in theacquisition program. The M&S tools willidentify system performance and effective-ness levels required to meet the specifiedthreat. Acquisition managers may use theseresults to aid in establishing risk levels as-sociated with each of the functional areas(threat, technology, design and engineering,manufacturing, support, cost, and schedule)to be reported within the risk assessment ofthe integrated program summary.4 TheM&S tools can also support evaluation ofsystem or technology alternatives which mayoffer reduced risk. From a program plan-ning point of view, models and simulationscan serve as a tool to assist in relating theMOP, MOE and integrating program plans,documentation, and functional disciplines,fostering Integrated Product and ProcessDevelopment (IPPD) at the start of the pro-gram.

5.1.3 Phase I - Demonstration and Validation(DemVal)


5.1.3.1 Focus: To examine multiple designapproaches and technologies for system can-didates; address supportability, manufactur-ing, and affordability; and establish perfor-mance objectives.

5.1.3.2 Activities: As the system begins tobe defined at greater levels of detail, draftdevelopment specifications are produced;and the system specification is approvedinitiating configuration management of thesystem. Critical design characteristics andperformance requirements at the systemlevel are refined and preliminary require-ments for subsystems are developed. A newCOEA is conducted and program plans anddocumentation such as the ORD, TEMP,SEMP, and ILSP and acquisition strategy areupdated. Critical technologies are demon-strated; and prototyping, testing, and earlyoperational assessment of critical systems,subsystems and components are conducted.Risk areas and management actions will beassessed. Producibility engineering planningis conducted and a preliminary manufactur-ing plan is produced. Logistics support ac-tivities examine factors such as level of re-pair. These analyses provide a basis forcommunication between logistics and designengineers as initial design trade-offs aremade.

5.1.3.3 M&S: Modeling and simulation in theDemonstration and Validation (DemVal)phase continues to support and extend activi-ties that were initially conducted in CED.During DemVal, as the focus of develop-ment starts shifting toward design of sub-systems and components, the models andsimulations take on better definition. Usesof models and simulations in this phase in-clude:

� Engineering level models and simula-tions of proposed system and subsystem con-

5-6

cepts at increasing levels of detail will beused to provide a better estimate of perfor-mance for development of design specifi-cations and for use in other models. Hard-ware/software-in-the-loop (HW/SWIL)simulations will be used to evaluatebrassboard designs, plan prototype tests andto identify and correct problems as a riskreduction measure. Logistics support mod-els such as repair level analyses allow thelogistics and design community to investi-gate the sensitivities of reliability and main-tainability, and repair level in concert withdesign trade-offs.

Computer-aided design (CAD) and com-puter-aided manufacturing (CAM) modelssupport both design and producibility plan-ning. By the end of this phase, the programoffice should require factory simulations tosupport EMD and subsequent proposals forrate production and facilitization.

Cost models will begin to incorporate engi-neering cost estimates and these will be usedto determine life cycle costs in the programcost estimate, to evaluate alternatives in theCOEA, and to refine the design to unit pro-duction cost estimate.

� Engagement and mission/battle modelsand simulations will again be used to evalu-ate mission effectiveness measures for theproposed systems in support of the COEAand ORD. They may also be used to definethe interoperability requirements amongsystems and the impacts of the proposed sys-tem on existing weapon systems.

� The theater/campaign level models andsimulations will be used to evaluate the pro-posed system�s impact on the outcome ofconflicts. These results will support theCOEA, and evaluate how well the systemmeets the requirements stated in the ORD.

� Human interactive simulations willcontinue to examine tactics within the aboveframework of constructive models and willbegin to examine the human-machine inter-face in virtual simulations.

� Virtual simulations may be used toevaluate the systems and subsystems and ex-amine tactics in a realistic battle environ-ment. In the DemVal phase, such simula-tions can employ more detailed performancemodels and actual prototype HW/SWIL.This may range from a single simulator tomultiple simulators of a new system linkedto other friendly forces and engaging theenemy on a synthetic battlefield. These link-ages may include any combination of vir-tual simulations, live simulations (exercises),or constructive models of systems andforces. The virtual prototypes will be par-ticularly useful in examining human-ma-chine interfaces and conducting trade-offswithout building actual hardware.

During DemVal, the program office shouldbe taking advantage of and implementingthe plans for M&S developed during CED.Simulations at various levels may be usedin support of the early operational assess-ments, particularly when little hardware isavailable for live testing. Simulations canalso be used in this phase to assist in sourceselection. Contractors might be required tobring their hardware into a governmentsimulation facility for evaluation. In someinstances, contractors might be provided acopy of the simulations that the governmentintends to use in system comparisons dur-ing source selection.

During this phase, the use of M&S will con-tinue to help the acquisition manager tomanage risk. The early use of simulationssuch as HW/SWIL with brassboard equip-ment can allow early identification and so-

5-7

lution of technical problems. The virtual pro-totypes with a single common data base whichare accessible to all the functional disciplineswill facilitate IPPD; and allow better designdecisions with less potential for having over-looked impacts to other disciplines.

5.1.4 Phase II - Engineering and Manu-facturing Development (EMD)


5.1.4.1 Focus: To translate the most prom-ising design approach into a stable, produc-ible and affordable design.

5.1.4.2 Activities: The EMD phase includesdetailed system, subsystem and componentdesign; including the associated manufactur-ing and support processes. Testing is con-ducted to verify that the system meets speci-fication requirements and satisfies missionneed and minimum operational perfor-mance requirements. Production planningis refined and logistics support activities in-clude provisioning. Low Rate Initial Produc-tion (LRIP) late in this phase is conductedto verify producibility and production costs.By the end of this Phase, the system is de-fined to its lowest levels (individual parts).Initial operational test and evaluation(IOT&E) and live fire testing is completedprior to a decision for full rate production.

5.1.4.3 M&S: A major focus of M&S inEMD is at the engineering level modelswhich are used for design, engineering trade-offs, test planning and support, subsystemand system performance, and verification ofcompliance with specifications. Models andsimulations also support COEA and ORDupdates, DT and OT&E, and prepare forproduction and deployment of the system.

Several uses of models and simulations in

EMD are defined below.

� Engineering level models and simula-tions of proposed systems and subsystemswill be used for detailed design and assem-bly of subsystems, components and pieceparts. Performance requirements will beverified using a combination of testing andsimulation.

The HW/SWIL simulations will be used ina model-test-model process for pre-testplanning, test execution, and post-test analy-sis. Such simulations are able to identifyproblems in actual test hardware before con-ducting live tests (i.e. live simulations) onthe range. They also provide for parametervariation studies, and augment the matrixof test conditions. The performance esti-mates from simulations during this phasealong with live simulation (test) data pro-vide input for models and simulations atother levels or of other classes.

Logistics support models will examine suchfactors as reliability, availability, maintain-ability, transportability, and provisioning(spares, support equipment, manpower).The CAD/CAM models will produce de-signs that can be electronically transmittedto the shop floor resulting in fewer manu-facturing errors. Factory simulations areused to plan facilities and equipment anddefine production flows to meet plannedproduction rates in support of both designand producibility planning. If not alreadyaccomplished, the program office shouldrequire factory simulations to support pro-posals to substantiate the ability to achieverate production and identify requiredfacilitization.

Cost models will be able to incorporate costdata from engineering models and actualLRIP hardware for the program cost esti-

5-8

mates and COEA updates.

� Engagement and mission/battle modelsand simulations will again be used to evalu-ate how well the designs allow the proposedsystem to achieve the necessary MOE.

� The theater/campaign level models andsimulations will be used to assess the pro-posed system and determine its impact onthe outcome of conflicts.

� Human interactive simulations willcontinue to examine tactics within the aboveframework of constructive models, but willmore likely focus on continued refinementof human-machine interfaces.

� Virtual simulations can be used toevaluate systems performance and effective-ness. A virtual prototype can be used to sup-port development efforts including design,support (e.g. maintenance walk throughs),manufacturing and training. Members fromevery functional discipline share the sameelectronic representation of the system fa-cilitating integrated product and processdevelopment. Weapon system trainers be-ing developed should take maximum advan-tage of the models and simulations used indeveloping the system itself. As these train-ers are developed and made available, theymay be used for training test crews, andmission rehearsal for live simulations (e.g.OT&E planning).

� Live simulations may take the form oflive exercises, or instrumented prototypetests, including IOT&E. Managers shouldinsist that data obtained in these tests areused to further validate the models andsimulations.

A combination of engineering, engagement,mission and campaign simulations, as de-

scribed in the program TEMP, will be re-quired to augment the developmental andoperational test program during EMD. Ear-lier program efforts to define the appropri-ate models and simulations; VV&A them;and determine the relationships amongMOE and MOP are critical to the success-ful application of M&S to support or aug-ment the test program.

At the end of EMD, detailed design of thesystem including definition of productionand support processes is complete. In ac-cordance with M&S planning conducted be-ginning in the CED phase, the program of-fice should be prepared to maintain thosemodels and simulations which will beneeded for continued support of the weaponsystem during its life cycle. The programmanager (PM) also needs to consider howto make representations (models) of the sys-tem available to others outside the programoffice that may have a need to use them.

Models and simulations will support de-tailed design during this phase. They willcontinue to be key tools for IPPD, and willreduce design risk by allowing all of thefunctional disciplines to work from the samedesign data base. A reduced number of en-gineering change proposals (ECP) will bean important result of this activity. The HW/SWIL simulations will result in significantrisk reduction in test and evaluation throughplanning, hardware checkout and missionrehearsal. Finally, the transition to produc-tion will take place with reduced risk by theelectronic transfer of digital design data di-rectly to the manufacturing floor.

5.1.5 Phase III - Production andDeployment

(Ref DoDI 5000.2, pg 3-26 & 3-27)

5-9

5.1.5.1 Focus: To produce and deploy thesystem.

5.1.5.2 Activities: The objectives of this phaseare to establish a stable, efficient productionand support base; conduct follow-on opera-tional test and evaluation (FOT&E); achieveoperational capability that satisfies missionneed; identify, verify, and incorporate engi-neering changes; identify operational and/or support problems; and identify the need formajor upgrades or modifications requiringan MS IV review.

5.1.5.3 M&S: In the production and de-ployment phase, models and simulations canbe used to support continued testing, verifi-cation of design changes, training of crewmembers and development of operationaltactics. Below are some applications for thisphase of the program.

� Engineering level models and simula-tions may be used for evaluation and verifi-cation of engineering design changes. TheHW/SWIL simulations will continue to beused in a model-test-model process for pre-test planning, test execution, and post-testanalysis in support of further developmenttesting and FOT&E. They continue to beused to check out actual test hardware andsoftware before conducting live tests (simu-lations) on the range. Simulations may alsobe used in production lot sampling or pro-duction acceptance as a less expensive al-ternative to live testing. Logistics supportmodels will incorporate actual field data todetermine system readiness.

Factory simulations can evaluate the effectsof design and manufacturing processchanges on production. They can also pro-vide information on whether current facilitiesand resources can accommodate changes inproduction rates or the manufacturer�s busi-

ness base. The support community may alsochoose to use such simulations in depot plan-ning for weapon system maintenance.

Cost models will have the ability to use ac-tual production cost data and preliminaryO&S costs in maintaining oversight of pro-gram costs.

� Engagement, mission/battle and the-ater/campaign level models and simulationsmay continue to be used to evaluate opera-tional consequences of system performancechanges or changes in threat. In the interac-tive mode, these simulations will be usedas wargaming simulations to train battlestaffs on new tactics and concepts of opera-tion.

� Virtual simulations, such as weaponsystem trainers, can be used for training op-erational crews and mission rehearsal forFOT&E. They can also be used along withother simulations of friendly forces to en-gage enemy forces on a synthetic battlefield.

� Live simulations will include live exer-cises, or instrumented prototype tests, suchas FOT&E.

With the system being produced and de-ployed, real data are becoming available forcontinued validation of the models andsimulations used within the program. Thisis an opportunity to improve the models forfuture use either in the current program oranother program�it is useless to place thedata on the shelf and forget about it!

At this point in the program, the programoffice is implementing the plans for lifecycle maintenance of the models and simu-lations. This is particularly important to keepin mind for weapon system post-deploymentsupport activity.

5-10

Within this phase, the use of modeling andsimulation will continue to reduce programrisk. On the acquisition side, the electronictransmittal of design data to the shop floorwill result in fewer manufacturing errors andless rework. The factory should be able tomeet production requirements based uponplanning using the factory simulations.

From the operational perspective, theweapon system operators will have availablea training system which accurately simulatesthe weapon system and its weapons. It willbe able to use models and simulations re-flecting the characteristics of its weaponsand enemy threats, and even employ theappropriate data protocols so it may inter-act with other live or simulated entitieswithin a synthetic battlefield environment.

5.1.6 Phase IV - Operations and Support(O&S)


5.1.6.1 Focus: Insure system continues tomeet mission needs and identify shortcom-ings and deficiencies.

5.1.6.2 Activities: In-service engineeringsupport, implementation of design changesand service life extension programs are anongoing activity. A change in threat, a defi-ciency in capability or the opportunity toreduce the cost of ownership (such as vianew technology) may result in a decision toseek a major modification approval, mile-stone IV, for which a COEA will be pre-pared.

5.1.6.3 M&S: In the O&S phase, models andsimulations can be used to support contin-ued testing, verification of design changes,training of crew members, mission rehearsaland development of operational tactics.

More importantly, M&S will be used in sup-port of decisions to initiate major modifica-tions of the system and to identify deficien-cies.

� Engineering level models and simula-tions will continue to be used for evaluationand verification of engineering changes. TheHW/SWIL simulations will be used to�check out� actual test hardware and soft-ware before conducting live tests (simula-tions) on the range.

For some systems there is a potential forperformance to degrade as a result of longterm storage. Shelf-life evaluations of sys-tems or components can be conducted viasimulation, in lieu of live tests. In this case,engineering level HW/SWIL simulations canevaluate performance of articles after ex-tended storage. The operational impacts, orpotential changes in tactics to accommodateperformance degradations, can then beevaluated in the higher level models.

Logistics support models will continue touse actual field data to determine system readi-ness. The support community may chooseto use factory simulations to support depotplanning for weapon system maintenance,particularly useful when major changes inworkload are anticipated.

Cost models will have the ability to use ac-tual production and O&S cost data to main-tain oversight of program costs and supporta COEA for major modification.

� Engagement, mission/battle and the-ater/campaign level models and simulationsmay continue to be used to evaluate opera-tional consequences of system performancechanges or changes in threat. In the inter-active mode, these simulations will be usedas wargaming simulations to train battle

5-11

staffs on tactics and concepts of operation.More importantly, these models and simu-lations may be used in identifying deficien-cies because of threat changes, or may dem-onstrate the military utility of new technolo-gies which can lead to a milestone IV deci-sion for major modification approval.

� Virtual simulations will continue to beused for training operational crews and mis-sion rehearsal. They may also be used to ex-amine the potential of new technology ap-plications for system improvement or exam-ine the impact of threat changes within asynthetic battlefield environment with othersystems.

� Live simulations will continue to in-clude live exercises, or instrumented testsof the system. With the system in use withinits operating environment, either in trainingor live exercises, real data are available forcontinued validation of the models andsimulations used within the program.

Nearly every weapon system today employscomputer software. Software updates toimprove capability or to counter a new threatare an ongoing activity for the life of thesystem. Simulations are the primary methodused to test and verify any changes made inthe system software. This fact emphasizesthe importance of a simulation plan thataddresses the use and maintenance of mod-els and simulations throughout the systemlife cycle.

The �risk cycle� introduced in Chapter 1 hascome full circle. The models and simulationswill be used to reduce risk in several areas.They will be employed to replicate problemsor failures encountered in operations andverify solutions to maintain mission capa-bility. In those instances where performancehas degraded over time, for whatever rea-

son, models and simulations will quantifyperformance impacts and evaluate changesin tactics or employment to accommodatethem. Lastly, the suite of models and simu-lations will identify capability deficienciesbecause of threat changes and evaluate nonmateriel and materiel approaches to miti-gate the operational risk caused by thosedeficiencies.

5.1.7 Summary

As the above discussion shows, M&S willbe used by a program from the earliest stagesthrough operations and support. Many of thesame models, or types of models will beused repeatedly. A significant amount ofprogram planning takes place during phase0, CED. It is in this early planning stage thatM&S should be identified and woven intothe appropriate program and functional dis-cipline plans.

5.2 Modeling and Simulation in Support ofAcquisition Activities

The remainder of the chapter describes howM&S supports key acquisition functions asthey span the phases of the process. Thesefunctions are: requirements definition, pro-gram management, design and engineering,manufacturing, test and evaluation, logisticssupport and training. For these functionalareas, the reader will find a template in Ap-pendix G showing key activities conductedand the types of models and simulations thatmight be employed. Those templates shouldserve as examples to stimulate PMs as theyperform the detailed planning for their ownprograms. In applying M&S to activities de-scribed in the remainder of this chapter totheir own programs, PMs should look foropportunities in two areas:

5-12

� How the program can use the M&Stools across phases of the acquisition pro-cess; and

� How the program might make use ofM&S to integrate activities across functionalboundaries.

5.2.1 Requirements Definition

Chapter 2 provided a detailed discussion ofthe requirements development process. Inthis section, we will describe the types ofmodels and simulations that can be used inthe process of developing requirementsdocuments (MNS, ORD, specifications).The reader is referred to Figure 5-2 for thefollowing discussion.

As with most other analyses conducted dur-ing the acquisition process, a complimen-tary suite of models and simulations is likelyto be used, ranging from engineering per-formance to theater/campaign levels.

The input data to the analysis process in-cludes ground rules such as:

� Defense Intelligence Agency (DIA)threat estimates along with scenarios andmissions derived from the Defense PlanningGuidance (DPG);

� Environmental data including weather,terrain, ocean environment, countermea-sures, etc.;

� A selection of operational concepts andtactics, which allow for evaluation of poten-tial non- material solutions as required byDoDI 5000.2;

� System options to include existing, up-grades or new systems; and

� New technologies that may be availablethrough DoD�s science and technology pro-grams, advanced technology demonstrationsor industry.

These data address a variety of scenarios,systems and tactics and will be used in analy-ses conducted at each level in the M&S hi-erarchy described in Chapter 4. Using theengineering level of models, analyses pro-vide performance estimates for existing andimproved capability systems taking into ac-count the emerging technology opportuni-ties. The performance and design trade-offsof system and subsystem design conceptsand technologies are evaluated at this level.These system/subsystem performance capa-bilities are evaluated within the engagementand mission/battle level models and simu-lations to determine system effectiveness(e.g. probability of kill, losses, survivability,vulnerability) and mission effectiveness (e.g.loss exchange ratios, probability of engage-ment) in a limited engagement or mission.These capabilities support campaign levelmodels to examine effects of force mix, tac-tics or new capabilities on outcomes, typi-cally in terms of force exchange ratios, drawdowns or troop movements.

The analyses are repeated for a variety ofoperational concepts and each of the sys-tem options under consideration. The en-gagement, mission and campaign modelsmay be run iteratively to provide statisticalsignificance to the outcomes. Material ca-pability needs are identified and documentedin a MNS.

The engineering models in conjunction withthe engagement and mission/battle levelmodels also provide the basis for the descrip-tion of broad capabilities and technology de-velopments which should be studied in CED.

5-13

Figure 5-2. Models and Simulations in Requirements Definition

Mo

del

s an

d S

imu

lati

on

s in

Req

uir

Mo

del

s an

d S

imu

lati

on

s in

Req

uir

Mo

del

s an

d S

imu

lati

on

s in

Req

uir

Mo

del

s an

d S

imu

lati

on

s in

Req

uir

Mo

del

s an

d S

imu

lati

on

s in

Req

uir

emen

ts D

efin

itio

nem

ents

Def

init

ion

emen

ts D

efin

itio

nem

ents

Def

init

ion

emen

ts D

efin

itio

n

TH

EA

TE

R/

CA

MP

AIG

N(O

UT

CO

ME

S)

MIS

SIO

N/B

AT

TLE

(MIS

SIO

N E

FF

EC

TIV

EN

ES

S)

EN

GA

GE

ME

NT

(SY

ST

EM

EF

FE

CT

IVE

NE

SS

)

EN

GIN

EE

RIN

G

(SY

ST

EM

/SU

BS

YS

TE

M/C

OM

PO

NE

NT

PE

RF

OR

MA

NC

E)

MIS

SIO

NN

EE

DS

MIS

SIO

NC

AP

AB

ILIT

Y

SY

ST

EM

CA

PA

BIL

ITY

SY

ST

EM

NE

ED

S

PE

RF

OR

MA

NC

ER

EQ

UIR

EM

EN

TS

PE

RF

OR

MA

NC

EC

AP

AB

ILIT

Y

MIS

SIO

NN

EE

DS

TA

TE

ME

NT

OP

ER

AT

ION

AL

RE

QU

IRE

ME

NT

SD

OC

UM

EN

T

SP

EC

IFIC

AT

ION

RE

QU

IRE

ME

NT

S

New

Upg

rade

s

Exi

stin

gS

yste

ms

OP

TIO

NS

EN

VIR

ON

ME

NT

S

OP

SC

ON

CE

PT

SD

PG

BA

SE

DT

HR

EA

TS &

SC

EN

AR

IOS

TE

CH

NO

LOG

YO

PP

OR

TU

NIT

IES

PH

YS

ICS

INP

UT

S

OU

TP

UT

S

MO

DE

LS &

SIM

ULA

TIO

NS

RE

QU

IRE

ME

NT

SF

LOW

DO

WN

CA

PA

BIL

ITY

CO

MP

LIA

NC

E

5-14

The ORD will be developed during the CEDphase. The ORD defines thresholds and ob-jectives in terms of operational effectivenessmeasures, system performance measures andcritical system characteristics. The ORD willbe updated during DemVal with refined andmore detailed capabilities and characteristics.It is likely that mission/battle and engagementmodels, in conjunction with engineering mod-els, will be used to develop the effectivenessand performance measures for the ORD.

Technical specifications similarly evolve. Adraft system level specification will be de-veloped during CED; development specifi-cations will be written during DemVal; andproduct, process and material specificationsduring EMD. Engineering level M&S (e.g.design, support, manufacturing and HW/SWIL) typically support the development ofthese requirements specifications.

There is not a simple one-to-one mappingbetween a particular level of M&S and aparticular requirements document. Rather,as was discussed in Chapter 4, a combina-tion of M&S (levels and classes) will likelybe needed to generate the various measuresand insure consistency of those measuresacross the program documents.

5.2.2 Program Management

The PM is faced with balancing cost, sched-ule and performance objectives throughoutthe program. Much of the current emphasisin M&S is on the performance or militaryutility arena, as has been the focus of muchof this guidebook. This next section willtouch upon some of the management toolsthat are in existence.

5.2.2.1 Program Management Tools

Many models used for program management

are actually data base references, or knowl-edge-based tools. Examples of two suchmanagement tools are the Navy�s ProgramManager�s Work Station (PMWS)5 and theAir Force Acquisition Model (AFAM)6.These will be discussed, followed by someguidelines on the use of cost models.

Program Manager�s Work Station (PMWS) is aseries of interrelated software tools designedto provide acquisition information (prima-rily engineering process oriented) to the user.Three of the modules may be of particularuse to the program office:

� Know-how: an automated handbooksystem that is reported to reduce search timein handbooks and manuals such as the DoD-5000 series, Best Practices (NAVSO-P-6071) and ISO 9000.

� Technical Risk Identification and Miti-gation System (TRIMS): based on the BestPractices (NAVSO-P-6071) it allows theuser to develop risk metrics and status.

� Best Manufacturing Practices database: Contains over 1800 abstracts fromcompanies documenting best practices.

Air Force Acquisition Model (AFAM) is anacquisition process model designed to fa-cilitate the process for major and non-ma-jor acquisition programs. This personal com-puter based model addresses activitiesacross the acquisition phases.

� Acquisition Guides and Insights: Pro-cedures which cover 2200 acquisition tasksbased on the DoD 5000 series and best man-agement practices and lessons learned.

� Acquisition Management Tools: Pro-vides guidance, task relationships, timelinesand expertise from actual program experi-

5-15

ence which one should tailor before usingin a specific program.

The PMWS and AFAM are but two ex-amples of program management tools whichprovide PMs with a disciplined thought pro-cess. They should serve to prompt PMs re-garding activities that should be performedand application of best practices.

5.2.2.2 Cost Models

Program managers develop cost estimatesduring the acquisition process for two pur-poses:

� Program life cycle cost estimates; and

� Cost estimates for alternatives evalua-tion in the COEA.

Two separate cost estimates are requiredfrom the DoD component in support ofmilestone I and subsequent reviews. One ofthese estimates will be prepared by the pro-gram office and the other by a separate or-ganization that does not report through theacquisition chain.7 Additionally, the OSDCost Analysis Improvement Group (CAIG)will develop an independent DoD estimateand prepare a report to the Under Secretaryof Defense for Acquisition and Technology(USD(A&T)) for ACAT ID programs, andto the DoD Component Acquisition Execu-tive for ACAT IC programs.

The second use of cost estimates is in thepreparation of the COEA to support mile-stone decisions beginning with milestone I.The COEA is prepared by an independentactivity within the component, and shouldaid decision-makers in judging which, if any,of the proposed alternatives to the currentprogram offer sufficient military benefit tobe worth the cost.8

We will not address specific cost modelsused throughout DoD in this guidebook,however, some general features of a costmodel might include:

� Cost estimating relationships� Statistics package� Ability to address various cost estimat-

ing methodologies� Learning curve calculations� Risk analysis� Sensitivity analyses� System Work Breakdown Structure

(WBS)� Multiple appropriations (R&D, Appro-

priations, O&S)� Time-phasing of costs� Overhead rates� Inflation indices.

The above features are contained in the Au-tomated Cost Estimating-Integrated Tools(ACE-IT),9 which is a framework withinwhich the analyst can develop a cost model.These features are shown only as an illus-tration of what might be addressed in a costmodel, and are not necessarily all-inclusivenor must any particular model contain allthose features.

The Office of the Secretary of Defense(OSD) Cost Analysis Guide provides someguidelines regarding the characteristics of agood model for O&S costing which, withtailoring, might be useful for any model ap-plication.10

� Consistency in cost element structure:The basic cost structure should not changeas a system passes through the acquisitionphases. However, the basic elements andtheir sub-elements should be expanded tocapture greater levels of detail.

� Consistency in data elements: Data el-

5-16

ements of the proposed system should beconsistent with those of operational systemsfor which actual data exists. This allows thecosts and cost driving parameters of thereference and proposed system to be com-pared.

� Flexibility in estimating techniques:The estimating techniques should be allowedto vary as a program progresses through thevarious acquisition phases.

� Simplicity: Complexity is not desirablein an O&S cost model. Models should bestructured in a way that allows them to ac-commodate more detailed information as aprogram progresses through the life cycle.

� Usefulness to the Design process:While estimating costs for a CAIG reviewis an important function, a model�s applica-bility to day-to-day program office and con-tractor decision making is equally important.

� Completeness: The model should cap-ture all significant costs that will be incurredby the weapon system over its useful life.

� Validity: The model should providesound, reproducible results for its intendedapplication.

The PM should recognize that in actual prac-tice, cost estimating is a melding of art andscience. There is no �one model fits all�, butrather typically a custom model for eachprogram, relying on various cost method-ologies or historical data bases to addressdifferent elements of the system. As with anyother M&S efforts, an experienced analystis key to obtaining credible results.

The Cost Analysis Requirements Document(CARD) describes the system and salient fea-tures of the program which will be used to

develop life cycle cost estimates.11 It providesa description of the system and its key charac-teristics (weight, size, payload, speed, power,etc.) for each WBS element. The CARD ad-dresses the operational concept, risk, quanti-ties, manpower, system usage rates, schedules,acquisition strategy, development plans andfacilities requirements. Since the CARD ad-dresses all the key cost elements of the sys-tem, it provides the basis for cost estimatingand the use of cost models.

A study being conducted by the Army Costand Economic Analysis Center will providethe results of a survey on cost models usedthroughout DoD, and may be of interest toPMs.12

5.2.3 Design and Engineering

The use of M&S is most prevalent in thisfunctional discipline. A visible example isthe Boeing 777 aircraft. This is the first air-plane designed solely by computer which wasaccomplished largely via the CATIA (Com-puter Aided Three Dimensional InteractiveApplication)13 system. Significant accom-plishments of this effort included:

� �Paperless� design - the blueprints re-sided in the computer;

� Design/build teams shared the same de-sign information contained in the computerdata base. 2200 networked work stations al-lowed all of the functional disciplines (e.g.Users, Design, Manufacturing, Mainte-nance) to communicate based on a commonframe of reference;

� The 3-D �virtual airplane� allowed en-gineers to make design changes and visual-ize the results, such as component interfer-ences; and

5-17

� Design data transferred electronicallyto the manufacturing floor.14

The results of this approach included morethan a 50 percent reduction in change errorand rework in manufacturing.15

Modeling and simulation pervades the vari-ous specialty disciplines involved with de-sign�ranging from finite element analysisfor structural design, to computational fluiddynamics for aerodynamics or hydrodynam-ics. For human factors, anthropometricmodels such as �Jack� can be used to exam-ine the ability of a crew member to operatecontrols, repair equipment or fit within crewcompartments. What these models and

simulations offer is the ability to modifydesigns, analyze the effects and refine thedesign repeatedly prior to building a singlehardware prototype.

In the future, with the integration of designand performance simulation models, one canachieve a �Simulation Based Design�16 inwhich 3-D virtual prototypes, properlyrepresenting both design and performance,function realistically in a virtual environ-ment and replace actual hardware mockups.Figure 5-3 depicts the process whereby allof the functional disciplines will use the samevirtual prototype to support activities acrossthe system life cycle�from operational re-quirements generation through engineering,

Figure 5-3. Simulation Based Design

VP’s - THE ENABLER OF INTEGRATED PRODUCT DEVELOPMENT TEAMS

WARFAREANALYSIS

REQUIREMENTS

PRODUCTDEFINITION

WEAPONSSYSTEM

LESSONSLEARNED

ENGINEERING

CONSTRUCTION

TRAINING

PRIMARYSIMULATION VISIBILITY TO DATE

THE VIRTUAL SYSTEM

TESTING

OPERATIONALSYSTEM

OPERATIONS& LOGISTICS

DEPLOYABLESYSTEM

Simulation Based DesignSimulation Based DesignSimulation Based DesignSimulation Based DesignSimulation Based DesignVVVVViririririrtual Prtual Prtual Prtual Prtual Prototyping in the System Life Cycleototyping in the System Life Cycleototyping in the System Life Cycleototyping in the System Life Cycleototyping in the System Life Cycle

5-18

construction, testing, training and operationsand logistics support.17 The reader will seeanother example of the use of virtualprototyping in vehicle design in Chapter 8.

5.2.4 Manufacturing

Producibility is intimately linked with prod-uct design�shape, features, materials, etc..The use of computer models to simulatemanufacturing processes such as metalforming, machining and casting allows oneto evaluate the ability to produce a designbefore actually �bending metal�. The use ofCAD/CAM models allows the design andmanufacturing communities to converge ona producible design that meets the require-ment. Using the same models and simula-tions for design and manufacturing, com-bined with the transfer of digital design databases directly to the manufacturing floor;reduces errors, rework and hence, produc-tion risk.

In addition to having a producible design,the program office must be assured that thenecessary capability/capacity is available tomeet planned production rates.

In the CED phase, production planning be-gins with an industrial base analysis. Con-siderations incldue the investments neces-sary for industrial capabilities to provide andsustain production; tooling; and facilities.18

During the DemVal phase, an initial manu-facturing plan is developed to portray thefacilities, tooling and personnel resourcesrequired for production.19 This plan is up-dated during the EMD phase based uponthe planned detailed manufacturing opera-tions. In production readiness reviews, con-ducted during EMD, the program manage-ment office (PMO) will evaluate the capac-ity of the production facility to meet the re-quired production rates. The PMO will also

evaluated the contractor�s production plan-ning, including manufacturing methods andprocesses, facilities, equipment and tooling,and plant layout.20

Factory simulations are used to aid in thiscycle of production planning which can sup-port the activities mentioned above. Thesesimulation tools can address production pro-cesses, factory process flow, statistical varia-tion in manufacturing operations, equip-ment, plant layout and manpower require-ments to meet production demands. Mili-tary and commercial programs are turningto such tools to improve efficiency or deter-mine facilitization requirements. These toolsmay be used for planning a new productionactivity, or to examine changes to an exist-ing program. An example follows showingthe use of simulation to plan changes in theperiodic maintenance of C-141 aircraft.21

In this case, the periodic depot maintenance(PDM) of the C-141 aircraft fleet was im-pacted when two structural problems werediscovered: wing and center wing boxcracks. Repair of the wing cracks and re-placement of the center wing box needed tobe incorporated into the ongoing PDM ofthe aircraft. Furthermore, replacement of thecenter wing box was a new process for thedepot - it had only been done once on a pro-totype aircraft at a contractor�s facility. TheSLAM II simulation language was used tosimulate the ongoing PDM along with theintroduction of the wing repair and centerbox replacement. A sample of the results ofthis simulation include:

� An achievable schedule for wing boxreplacement, but a shortfall for wing crackrepair;

� Bottleneck locations; and

5-19

� The preference to reallocate ratherthan purchase additional inspection equip-ment.

Commercially available and industry ownedfactory simulations are in use by manyweapon system contractors or maintenancedepots today. Factory simulations such as�Witness�22 are now being regularly used tosupport aircraft, missiles, and electronicsproduction, and depot activities. A listingof commercially available manufacturingrelated simulation programs can be foundin reference.23

Factory simulations can be used for the fol-lowing:

� Develop an assembly strategy;

� Graphically model the assembly se-quence;

� Develop and validate work sequences;

� Develop and validate manufacturingprocess plans;

� Model the factory floor, including fa-cilities and equipment;

� Identify what is achievable in terms ofcost and schedule;

� Identify bottlenecks;

� Compare different manufacturing strat-egies; and

� Identify impacts of engineeringchanges, new materials, machines or pro-cesses.24

All of these factors are important in deter-mining the robustness of production plan-

ning in proposal evaluation, or eventually,readiness for production. Their use by thecontractors, beginning no later than theDemVal phase can ensure the program of-fice that proper production planning hasbeen accomplished.

The Air Force has initiated a policy regard-ing the use of factory simulations in sup-port of depot upgrades. In November 1992,a policy letter was written establishing therequirement that all depot maintenancesponsored military construction projects orequipment projects greater than $0.5M shallbe modeled as a prerequisite to funding.25

The use of M&S in manufacturing is aimingtoward a future �Virtual Manufacturing�environment. In this approach, the opera-tional requirements identified in the syn-thetic battlefield environment are translatedinto design concepts using three-dimen-sional virtual simulations incorporating ge-ometry and performance. These designs arepassed along to a network of distributedmanufacturing simulations which may residethroughout a vendor base (i.e. prime con-tractor and its subcontractors) to identify themanufacturing processes, facilities and tool-ing requirements. This vendor base is clos-est to the manufacturing processes and is inthe best position to develop cost and sched-ule estimates. These estimates may then befed back up to provide better estimates ofcosts and schedules to support trade-offs andthe system level alternative evaluations inthe COEA.

The virtual manufacturing initiative is in-tended to provide the ties between new prod-uct design concepts and the processes nec-essary to manufacture them starting in theearliest phases of development. This will:

� Provide quick and improved cost and

5-20

delivery estimates;

� Smooth the transition of new processtechnologies into production facilities;

� Facilitate lean or agile manufacturing;and

� Facilitate IPPD.26

5.2.5 Test and Evaluation

The purpose of a test and evaluation pro-gram is to provide information for risk as-sessment and decision making, verify attain-ment of technical performance specifica-

tions and objectives, and verify that systemsare operationally effective and suitable fortheir intended use. Test planning begins inPhase 0, CED, resulting in the initialTEMP at milestone I. Models and simula-tions supporting the development (DT) oroperational test (OT) programs must be dis-cussed in the TEMP. For DT, the programmust �List all models and simulations to beused and explain the rationale for theiruse�27. For OT, the TEMP must �Identifyplanned sources of information (e.g., devel-opment testing, testing of related systems,modeling, simulation, etc.) that may be usedby the operational test agency to supplementthis phase of operational test and evaluation.

Figure 5-4. Missle Data Requirements and Test Assets

400K

300K

200K

100K

10k

1950 1960 1970 1980 1990 2000

T ES T AS S ET S

DAT A RE QUIRE MENT S

M&S “BRIDGES T HE GAP”

MISSILE

LAUNCHES

100

0

T&E

DATA

YEAR

Missile Data RequirMissile Data RequirMissile Data RequirMissile Data RequirMissile Data Requirements and Tements and Tements and Tements and Tements and Test Assetsest Assetsest Assetsest Assetsest Assets

5-21

Whenever models and simulations are to beused, explain the rationale for their credibleuse�.28

5.2.5.1 Developmental Test and Evaluation(DT&E)

Weapon systems being developed today areincreasingly more complex�technology isadvancing, the ability to process more in-formation is rapidly growing and the per-formance of systems is increasing. As an ex-ample, consider the illustration in Figure 5-429 of available test assets and data require-ments for missile development programsover the last 40 years. There has been a sig-nificant increase in missile complexity anddata requirements, but this increase in mis-sile complexity has not been accompaniedby a corresponding increase in missile launchassets�because of tighter program cost andschedule constraints.

Figure 5-5 illustrates this example further.30

This figure summarizes the number of testfirings for several Sidewinder models (AIM-9 series missile) developed over the last 30years. Most of the upgrades represent what

would now be called pre-planned productimprovements, involving changes only to themissile seeker. (The exception was the AIM-9L which included a new warhead and fusealong with seeker improvements.) The fir-ings shown represent the total number offirings from research and developmentthrough OT&E. The downward trend wasdriven by cost and schedule. Simulationswere used to maintain or increase the levelof understanding of system performanceeven though test need decreased.

Simulations, therefore, are used to �bridgethe gap� between the ever-increasing datarequirements and the relatively constant, oreven decreasing available test assets. Spe-cifically, simulations can be used for:

� Pre-test planning � Insuring that thetests to be conducted are, indeed, the mostcritical and verify instrumentation plans.Simulations can be used to identify the criti-cal test points on which to focus the live tests.Data from the simulation can be used priorto actual testing to check out and exercisethe data reduction processes.

Figure 5-5. Sidewinder Firing History

Sidewinder Firing History

Sidewinder Model Development Time Number of Firings

AIM-9D 1960-1964 129AIM-9L 1972-1975 69AIM-9M 1978-1981 35

AIM-9-8/9 1991-1993 21

5-22

� Mission rehearsal - �Walking through�the test from initial launch conditions to giveconfidence that tests will be successful. Onecan use actual hardware in captive carrybeing stimulated with threat simulators tocheck out the system and tactics prior to test.

� Post-test analysis - Taking the raw testdata and extracting the critical performanceparameters.

� Augment actual tests - Running largenumbers of simulations over many condi-tions for which test assets are unavailableor when environmental, political, resourceor safety constraints make testing infeasible.For example, over a six month period forone missile, a total of 4,280 HWIL simula-tion runs versus 7 actual launches were con-ducted.31

� Risk Reduction - Conducting simula-tions to reduce program �political� and tech-nical risks.

� Political risk reduction � Programsare increasingly under scrutiny fromall levels, and managers can ill afford therisk of a live test failure. Simulations toconduct mission rehearsals and checkoutof the actual test items can reduce thisrisk.

� Technical risk reduction � Simula-tions allow developers to evaluate farmore design alternatives over more con-ditions in shorter time periods than livetests. This allows identification and cor-rection of technical problems early in aprogram; resulting in a design that bettermeets technical and operational require-ments. An example of this latter case isthe use of HW/SWIL simulations.

Guidelines on the use of M&S in support of

DT&E may be found in the Modeling andSimulation Master Plan32 prepared by the USArmy Test and Evaluation Command(TECOM). This plan discusses the roles ofthe various participants in Army T&E ac-tivities, and provides a vision for the ad-vancement of M&S to increase the effi-ciency and cost effectiveness of T&E.

5.2.5.2 Operational Test and Evaluation(OT&E)

An OT&E is a comprehensive process whichuses analytical studies, analysis, componenttests and actual weapon system tests in acomplimentary manner. In accordance withTitle 10, U.S. Code, �The term operationaltest and evaluation� does not include anoperational assessment based exclusively on(a) computer modeling; (b) simulation; or(c) an analysis of system requirements,engineering proposals, design specifica-tions, or any other information contained inprogram documents.�33

However, this does not mean that models andsimulations do not have a role in OT&E.Constraints on testing such as cost, security,safety, ability to portray threats, treaty con-straints, limitations on test instrumentation,number/maturity of test articles, test spaceand lack of representative terrain or weathermay preclude a comprehensive evaluationbased on field testing alone. M&S tools canaugment or complement the actual field teststo provide decision-makers with needed in-formation which otherwise would not beavailable.

According to DOT&E Policy, dated 24 Jan1989, appropriate uses of M&S include testplanning; test data analysis and evaluationto augment, extend or enhance test results;tactics development; and early operationalassessments of expected capabilities.34

5-23

Specifically, the policy states: �Ideally, theuser, developer, and tester would agree onthe M&S needed for operationally-orientedassessments for a system under consider-ation not later than Milestone I.� This policyalso reiterates the importance of describingplans in the TEMP for the use of modelsand simulations in OT&E to augment, ex-tend or enhance field test results.

Credibility is a key part of successful use ofthe M&S in supporting OT&E. This includesan acceptable M&S approach; confidencein the models, users, methodology, and re-sults; and a robust VV&A process. Appen-dix B of the DTO&E policy provides a listof issues that should be addressed to pro-vide evidence of credible models and simu-lations in OT&E. The reader is also referredto further discussion of credibility containedin Chapter 6 of this guidebook.

The Service�s operational test agency is ac-countable for the OT results that they reportand, hence, results of any M&S it uses insupport of OT&E. As an example, theArmy�s guidance on M&S to support OTidentifies the Commander of Army Opera-tional Test and Evaluation Command(OPTEC) as the accrediting official for mod-els and simulations used within that organi-zation.35 Although no formal documentationwas found, discussions with the Air Forceand Navy OT&E agencies imply that thesame policy is followed within the otherServices.

In the future, the test community, using ad-vanced distributed simulation (ADS), willbe able to conduct live tests which are net-worked to geographically dispersed human-in-the-loop simulations within a syntheticenvironment. This provides for a realistictest/simulation in a war-like environmentwith a variety of friendly and hostile com-

batants. Currently, in the OT&E environ-ment, distributed simulation is more usefulin test planning than actual conduct of testsbecause of issues such as VV&A of entitieswithin a distributed environment.

A Joint Advanced Distributed SimulationJoint Feasibility Study (JADS/JFS) is cur-rently underway to address issues, prin-ciples, procedures and practices for the in-creased use of distributed simulation sup-port to both developmental and operationaltests and evaluations.36

5.2.5.3 Live Fire Testing (LFT)

Title 10 of the US Code37 requires realisticsurvivability testing of covered systems (orproduct improvement programs) and lethal-ity testing for major munitions programsprior to proceeding beyond low rate initialproduction. Examples of M&S supportingLFT include: aircraft and missile flight pathgeneration; detection, tracking, and shoot-ing performance of artillery; warhead-tar-get fragment interactions; penetration me-chanics and failure mode analysis.

Evaluations of materials, fuel system design,internal routing of lines and cables, etc. areaccomplished using models and simulationswhich can facilitate �design for survivabil-ity� early in development before hardwareis produced and tested. The Survivability/Vulnerability Information Analysis Center(SURVIAC)38 is a centralized informationresource for information on survivability andlethality. The SURVIAC has an inventoryof models and simulations and can provideprogram with technical advice.

The acquisition manager should recognizethat the use of M&S complements the T&Eactivities. It has been recommended that anintegrated model-test-model approach be

5-24

implemented in development programs withthree aims in mind:

� Ensure models and simulations stillmeet the developer�s needs;

� Use models and simulations to identifycritical tests, data requirements, analyze dataand reduce the amount of actual testing; and

� Ensure every test serves the dual pur-pose of evaluating system performance andvalidating the models and simulations.39

Such an approach has been common in elec-tronic combat system development pro-grams. These development programs em-ploy heavy use of models and simulationsprior to testing within integration laborato-ries, simulation facilities and finally, in theopen-air. Testing is then followed by furthermodeling to analyze test data and extract theMOP and MOE.40

This concept of model-test-model is appli-cable to all system development programsand an adaptation of the above two philoso-phies is illustrated in Figure 5-6.

Figure 5-6. Model-Test-Model Approach to Development

Model-TModel-TModel-TModel-TModel-Test-Model Apprest-Model Apprest-Model Apprest-Model Apprest-Model Approach to Developmentoach to Developmentoach to Developmentoach to Developmentoach to Development

MODELING & SIMULATION(PRE-TEST)

SYSTEMDEVELOPMENT TEST

(LIVE SIMULATION)

MODELING & SIMULATION(POST-TEST)

� ID CRITICAL TESTS

� DEVELOP TEST SCENARIO

� DEFINE INSTRUMENTATION

� PLAN DATA ANALYSIS

� PREDICT TEST RESULTS

� MISSION REHEARSALS

� HARDWARE/SOFTWARE

CHECKOUTS (HW/SWIL) � INSTRUMENT SYSTEM

� CONDUCT LIVE TESTS

� COLLECT & PROCESS DATA

(FOR BOTH TEST RESULTS

AND M&S VALIDATION)

� REQUIREMENTS

VERIFICATION/UPDATES

� DESIGN UPDATES

� ANALYZE DATA

-COMPARE WITH

PRE-TEST PREDICTIONS

� EXTRACT AND EXTRAPOLATE

MOPs & MOEs

� VALIDATE/UPDATE M&S

5-25

5.2.6 Logistics Support

Models and simulations support logisticsanalyses across the system life cycle�fromdefining system level supportability con-cepts to reliability, availability and maintain-ability design requirements�and eventuallymodeling actual operational capability dur-ing operations and support.

An overview of the way in which M&S canbe used to support logistics follows.41

Early activities in the logistics communityinclude building the baseline comparisonsystem which can be used along with M&Sto do a comparative analysis for the pro-posed new system; identify supportability,cost, and readiness drivers; and estimate theoperations and support portion of the lifecycle costs.

In DemVal, as the weapon system becomesmore defined at the subsystem level, levelof repair analysis (LORA) models are usedto identify candidate areas for interface be-tween logistics and design. These analyseshelp define trade-offs in manpower; reliabil-ity, availability, and maintainability; and al-ternate maintenance concepts and their ef-fects on supportability for specific subas-semblies. Using these models to quantify theimpacts on support, the logisticians can in-terface with the designers to produce designsthat lead to reduced overall support costs.The LORA models will then be used for theactual repair level decision-making and formthe basis for the system maintenance plan.

In EMD, models will be used to analyze re-pair tasks and identify the requirements inthe ILS elements for each component. Theresults of these analyses form a data reposi-tory, the Logistics Support Analysis Records(LSARs), which can be used in the detailed

identification of logistics resource require-ments for each element of logistics, as wellas projected readiness modeling. Among themodels used are provisioning models to de-termine initial spares requirements and theoptimum spare parts and quantities neces-sary to attain end item operational availabil-ity at the least cost.

Early in development, engineering estimatesof component failure rates are used in themodels. As the system matures and is even-tually fielded, test data and actual opera-tional data become available. This data re-places the initial estimates on failure andrepair in the LSARs. During O&S, this in-formation can be used in models and simu-lations to evaluate actual system readiness,adjust provisioning levels or support systemoperational planning. Models and simula-tions also find use in this phase to evaluatethe supportability impacts of proposed ECPsor modifications to the system.

The ILS elements and logistics supportanalysis (LSA) tasks are supported by anassortment of models or simulations. Onesource of information on these models andsimulations is the �Logistic Support Analy-sis Techniques Guide�.42 This guide is pre-pared by U.S. Army Materiel CommandLogistics Support Agency (LOGSA), whichis designated as the DoD lead agency forLSA.43 This guide contains descriptions of105 models or simulations cross referencedto LSA tasks and ILS elements, along withpoints of contact for each model.

Another source of information is the Sup-portability Investment Decision InformationAnalysis Center (SIDAC) which maintainsa small number of logistics models and canprovide assistance in preparing and runningthose models and using assorted logisticsrelated data bases.44

5-26

5.2.7 Training

Training is integral to achieving and main-taining force readiness. Despite reductionsin force structure and annual operatingfunds, the services are determined to main-tain their �warfighting edge� with superiortraining. Throughout DoD, simulation insupport of training spans all of the classesof simulation.

Wargaming is used to train battle staff inplanning and execution of tactics from indi-vidual system level through combined as-sets applications. This is often accomplishedusing constructive models representing sys-tems or groups of systems or may even belinked to live systems. Facilities which sup-port such simulations may allow multipleparticipants to interact and provide record-ing of events for subsequent data analysisand debriefing of participants.

Virtual simulators such as weapon systemsimulators (aircraft, tank, ship, etc.) are com-monly used for training. These simulatorsimmerse operators in a realistic environment(visual, aural, motion) allowing them to per-form a mission as if they were in the actualvehicle, thereby receiving combat realistictraining. Another example of immersing theoperator in a virtual environment might bean air defense simulator which allows op-erators at multiple consoles to track, iden-tify, allocate and control weapons usingcommand and control formats obtained fromother simulated platforms. Weapon charac-teristics might be provided via computergenerated weapon simulations.

Live simulations in support of training in-clude the Army National Training Center atFt. Irwin, the Navy �Strike University� atFallon Naval Air Station, the Air Force �RedFlag� at Nellis AFB, and the Marine Corps

Air-Ground Combat Center at TwentyninePalms. These simulations allow participantsto operate systems under environmentalconditions which approach real life in com-bat. Data gathered during instrumented ex-ercises can be used to debrief participants,and can provide the system acquisition com-munity valuable information on perfor-mance of weapon systems and human in-teraction during close to real combat condi-tions.

The future application of simulation to train-ing will involve a combination of live andvirtual participants within synthetic environ-ments and will allow for training with indi-vidual participants geographically distrib-uted. This will become a reality in the train-ing community when the Navy/Air ForceJoint Tactical Combat Training System(JTCTS), the Navy Battle Force TacticalTraining (BFTT) System, and the Army�sClose Combat Tactical Trainer (CCTT) arefielded.

The JTCTS is a training instrumentationsystem that will revolutionize the way theNavy and Air Force conduct air to air, air toground, ship air/surface/submarine warfareand joint training. It is a live training simu-lation that combines simulated and real tar-gets detected and displayed by platform sen-sors. A core computer system that performsscenario development, scenario transmis-sion, data logging and post-exercise debrief.Each participant (aircraft, ships, submarines)will have an instrumentation package anddata links to inject scenario events into theparticipant�s combat systems through simu-lation or stimulation.

The Navy BFTT is a virtual training systemthat uses the actual ship�s combat/sensorsystem as the training system. The BFTTcomputer on each ship will stimulate the

5-27

radar, sonar, electronic warfare equipmentand communications suite with scenarioevents controlled by the BFTT computer forsingle ship training; from another ship formulti-ship training; or from a shore basedBFTT computer for fleet or joint virtualtraining on the synthetic battlefield. Thissystem also provides a debrief for partici-pants to gain maximum benefit from eachtraining scenario.

The Army CCTT is a network of mannedsimulators providing combined arms andcollective training using force-on-force free-play simulation on an electronic battlefield.The manned simulators will include the M-1series tank, M-2/M-3 Bradley fighting ve-hicles, FIST-V vehicles and dismounted in-fantry. Fixed and mobile CCTT systems areplanned. The JTCTS, BFTT, and CCTT willrely on the distributed interactive simula-tion (DIS) communications standards fordata communications, which are discussedin more detail in Chapter 6.

The PM should aim to maximize the use ofsimulations between weapon system andtraining system development. One exampleof simulators being developed as weaponsystem trainers serving an additional func-tion for the acquisition program is in the B-2aircraft program. As part of the OperationalFlight Program (OFP) software develop-ment process, the B-2 aircraft program usedthe weapon system trainer as a systems in-tegration lab to compile and check run thesoftware in conjunction with other real andsynthetic data. After any debugging, theOFP was returned to the Flight Test Centerto be certified for flight.

The above discussion provides the acquisi-tion manager insight into both the present

and planned applications of models andsimulation to training. In many cases, themodels and simulations which support thedevelopment of the weapon system can beused to support the training systems, be theysystem simulators, or distributed trainingsystems combining live, virtual and con-structive simulations. Currently, models andsimulations for training purposes are oftendeveloped separately by another softwaredevelopment activity. The PM should nothave to pay for these simulations twice - anintegrated M&S plan during the CED phasecan help the transition of simulations be-tween the system and its training simulator.

5.3 Summary

This Chapter provided an overview of theuse of models and simulations across theacquisition life cycle and in specific acquisi-tion activities. The challenge for PMs in us-ing these models and simulations efficientlyis to:

� Integrate the use of M&S within pro-gram planning activities and documentation;

� Plan for life cycle application, supportand reuse of models and simulations; and

� Integrate M&S across the functionaldisciplines.

To aid PMs in their planning for the use ofmodels and simulations, Appendix G con-tains templates covering some of the acqui-sition activities discussed in this chapter.These templates should be considered asguidelines only; each program must tailorthe models and simulations it uses to thespecific activities to be accomplished withinthat program.

5-28

ENDNOTES

16. Simulation based Design is a program within Ad-vanced Research Projects Agency, Maritime Sys-tems Technology Office (ARPA/MSTO), in coor-dination with Naval Sea Systems Command(SEA-03).

17. Naval Sea Systems Command (SEA-03).

18. Ibid.

19. Acker, D., & Young, S. (April 1989). DefenseManufacturing Management Guide for ProgramManagers. (3rd ed.). (p. 3-9). Ft. Belvoir, VA:Defense Systems Management College.

20. DoDI 5000.2, Part 6 Section O, Attachment 1, Par6-O-2.a and Par 6-O-3.c.

21. Schuppe, T. F. Col, USAF, McElveen, D. V. Capt,USAF, Miyares, P. H. Capt, USAF, Harvey, R. G.1LT, USAF. (Winter-Spring 1993) C-141 DepotMaintenance: Using Simulation to Define Re-source Requirements. Air Force Journal of Lo-gistics.

22. "WITNESS�, AT&T Istel Visual Interactive Sys-tems, Inc.

23. Nwoke, B. U. & Nelson, D. R. (July 193). AnOverview of Computer Simulation in Manufac-turing. Industrial Engineering. (p. 43).

24. Correspondence from Mr. William Motley, FD-MM, Defense Systems Management College, FortBelvoir, VA, May 17, 1994.

25. Letter from HQ AFMC/LG, Subject: Simulationand Modeling Policy for the Justification of De-port Maintenance Facility and Equipment Projects,dated November 20, 1992.

26. Hitchcock, M. Virtual Manufacturing InitiativePresentation. Manufacturing Technology Direc-torate, Air Force Wright Laboratory, Wright-Patterson AFB, OH.

27. DoD 5000.2M, Part 7, Attachment 1, Par 3.c.(3).

1. DoDI 5000.2 Part 4, Section B, Par 3.b.

2. DoD 5000.2M Part 4, Section G.

3. Director, Operational Test & Evaluation PolicyLetter. (January 24, 1989). For the Application ofModeling and Simulation in support of Opera-tional Test and Evaluation; and DoD 5000.2M,Part 7, Par 3.c.(3) and Par 4.d.(3).

4. DoD 5000.2M, Part 4, Section E.

5. Program Manager�s Work Station. Computer Sci-ences Corporation, 6565 Arlington Blvd., FallsChurch, VA 22042.

6.Air Force Acquisition Model. Air Force Aeronauti-cal Systems Center/CYM, 2060 Monahan Way,Wright-Patterson AFB, OH 45433-6503.

7. DoDI 5000.2, Part 10, Section A.

8. DoDI 5000.2, Part 4, Section E.

9. Automated Cost Estimating-Integrated Tools(ACE-IT), Air Force Electronics Systems Center/FMC, Hanscom AFB, MA .

10. Office of the Secretary of Defense. (May 1992).Cost Analysis Guide Chapter 3.

11. DoD 5000.4-M (December 1992). Cost AnalysisGuidance and Procedures.

12. U. S. Army Cost and Economic Analysis Center,study called �Investigation of Cost Estimating andEconomic Products, Tools and Methods�, FallsChurch, VA. Scheduled for publication in 1994.

13. This computer-aided design system was writtenby Dassault and licensed in the U.S by IBM.

14. Reed, F. (April/May 1994). You can look but youcan�t touch. Air & Space, p 54.

15. Proctor, P. (April 11, 1994). Boeing Rolls out 777to tentative market. Aviation Week & Space Tech-nology, p 36.

5-29

28. DoD 5000.2M, Part 7, Attachment 1, Par 4.d.(3).

29. Adpoted from notes provided by Emil J. Eichblatt,Naval Air Warfare Center (Weapons Division),Point Mugu NAS, CA, February 10, 1994.

30. Naval Air Warfare Center (Weapons Division),China Lake, CA.

31. Notes provided by Emil J. Eichblatt, Naval AirWarfare Center, Point Mugu NAS, CA, February10, 1994.

32. TECOM Modeling and Simulation Master Plan(Final Draft). October 1993.

33. United States Code, Title 10, section 2399, �Op-erational test and evaluation of defense acquisi-tion programs�.

34. Director, Operational Test and Evaluation Policyfor the Application of Modeling and Simulationin support of Operational Test and Evaluation,January 24, 1989.

35. USAOPTEC Memorandum Number 73-21, De-cember 9, 1993.

36. JADS/JFS is sponsored by OSD DT&E. JADS/JFS, 8500 Gibson Blvd S.E., Kirtland AFB, NM87117-5558.

37. Title 10, United States Code, Section 2366, �Ma-jor systems and munitions programs: survivabil-ity testing and lethality testing required before full-scale production�.

38. Additional information on SURVAC is containedin Appendix F.

39. Johnson, L.H., Crocker, C. M. Jr.. Cost EffectiveWeapon System Development Through IntegratedModeling and Hardware Testing. U.S. Army Testand Evaluation Command, Redstone TechnicalTest Center, Redstone Arsenal, Alabama.

40. Deis, M. R. The Air Force Electronic Combat De-velopment Test Process. Air Force Developmen-tal Test center, (AFDTC/XRE), Eglin AFB, FL.

41. Based on Interview with Mr. J. Manary, DefenseSystems Management College, FD-LS, April 21,1994.

42. Logistic Support Analysis Techniques Guide,AMC Pamphlet AMC-P 700-4, Headquarters,U.S. Army Material Command, February 20,1991.

43. USAMC Logistics Support Agency, ATTN:AMXLG-AL, Redstone Arsenal 35898-7466Phone: (205) 955-9866.

44. Information on SIDAC and other InformationAnalysis Centers is found in Appendix F.

6-1

66MODELING AND SIMULATION

ISSUESperformance of the model. The perceivedrelevance of a given factor is also balancedagainst the cost and complexity of includ-ing the factor and modeling its effects. As aresult, modelers, analysts and users routinelymake assumptions about the relevance ofspecific factors to the performance of themodel. This assessment looks at the valuethat would be added by their inclusion, andthe complexity of effort involved in includ-ing them. For example, in most simulations,certain human sensory perceptions, such assmell, heat and cold are omitted, as beingeither analytically irrelevant or not cost-ef-fective to model.

Naturally, these assessments are effected byhow well the factor and its impact on theenvironment being modeled are understood.Early representations of a system are basedupon what is known, at that time, of the tech-nology and related phenomenologies. Mod-els evolve in a spiral fashion as knowledgeof these aspects is gained.

So, how do we know that we haven�t omit-ted, or overlooked, something important?More importantly, since results of modelsand simulations are analyzed to support de-

6.1 Introduction

Over the years, a migration has occurredfrom live toward constructive and virtualmodels and simulations. With this migration,software has become an ever-increasing partof models and simulations. This chapter dis-cusses certain technical aspects of softwaredevelopment, use and management whichplay key roles in planning for the use of thesemodels and simulations.

6.2 Credibility of Models and Simulations(or �How can I believe what I�m seeing?�)

The last thing a manager wants to do is totake a recommendation up to a decision-maker, be asked, �Why should I believe thisanalysis?�, and not be able to substantiatethe credibility of the underlying information.Not only would the manager have wastedvaluable time and money, but the credibil-ity of future program actions may have beenjeopardized.

By its very nature, a model is an abstractionof the real world. Factors are often not in-cluded in a model; based upon an assessmentof the relevance of specific factors to the

6-2

cision-making, how do we convince the de-cision-maker that the analyses are credible?

Both these concerns are addressed, in part,by ensuring that the modeling and simula-tion (M&S) supporting any analysis are bothcomplete and correct. To establish this fact,it is imperative that the M&S be subjectedto scrutiny.

A M&S must be verified, validated, and ac-credited (VV&A). This is a continuous pro-cess forming an integral part of an M&S�development, use and maintenance. Cred-ible documentation is also a critical ingre-dient in success of the VV&A process, as isthe control of the overall development pro-cess.

6.3 Verification, Validation andAccreditation (VV&A)

6.3.1 What is VV&A?

As Figure 6-1 depicts, VV&A of a model orsimulation collectively contribute to its cred-ibility. As the figure shows, VV&A involvesthe M&S developer, the functional expertand the user. Although the definitions areincluded in the glossary, this is probably agood place for a more focused definition ofeach term.

6.3.1.1 Verification � The process of deter-mining that a model implementation accu-rately represents the developer�s conceptualdescription and specifications.1

Proper conduct of verification answers thequestion: �Is it what I intended?�

Verification is applied at each stage of thelife cycle to ensure that the products of that

Figure 6-1. Verification, Validation and Accreditation

“It works as Ithought itwould.”

“It looks justlike the

real thing.”

“It suitsmy needs.”

VVVVVerificationerificationerificationerificationerification VVVVValidationalidationalidationalidationalidation AccreditationAccreditationAccreditationAccreditationAccreditation

DeveloperDeveloperDeveloperDeveloperDeveloper FunctionalFunctionalFunctionalFunctionalFunctionalExperExperExperExperExperttttt

Requester/Requester/Requester/Requester/Requester/UserUserUserUserUser

6-3

stage accurately implement the output fromthe previous stage and contribute to the over-all goal. It requires identification and exami-nation of explicit and implicit assumptionsand logic flows, and includes appropriatedata certification.

6.3.1.2 Validation - The process of determin-ing (a) the manner and degree to which amodel is an accurate representation of the realworld from the perspective of the intended useof the model, and (b) the confidence thatshould be placed on this assessment.2

Proper conduct of validation answers thequestion: �How well does the model repre-sent what it claims to represent?�

Validation, too, must encompass the entiresystem, the model, data and even the opera-tors and analysts who will use the model orits results. It is a rigorous process, involvingboth the structure of a model or simulation;as well as its output. Validation addressesboth the fidelity and resolution of a modelor simulation. It ensures that all aspects ofthe real world which should be representedare accounted for in sufficient detail to ad-equately establish cause-and-effect relation-ships. Validation provides the foundation forthe accreditation process to build upon.

6.3.1.3 Accreditation � The official certifi-cation that a model or simulation is accept-able for use for a specific purpose.3

Proper conduct of the accreditation processanswers the question: �Should I endorse thismodel?�

The agency using the results from a specificapplication of a M&S (the application spon-sor) must establish a set of standards, or ac-ceptability criteria, that a particular M&Smust meet to be accredited for a given use.

It is important to recognize that any use ofthe results of a M&S is considered de factoaccreditation by the M&S application spon-sor. Obviously, the preferred method of ac-creditation involves a determination, beforeuse, that the M&S is appropriate for that use.

Verification and validation (V&V) consistof varying levels and types of technicalevaluations for the model, its developmentprocess, as well as the data used to run it.

Accreditation is a management determina-tion, and is largely based upon the technicalevidence and audit trail resulting from theV&V process.

6.3.2 Why is VV&A necessary?

The purpose of VV&A is to establish thecredibility of a model or simulation as asource of data for analysis.

The output data from models are used tofeed analyses supporting decision-making inthe DoD acquisition process. These analy-ses define weapon systems requirements,influence procurement decisions, aid in en-gineering design, plan and conduct test andevaluation; and establish maintenance lev-els, sparing, production rates, etc. Further-more, since output data from one model areoften used as input for another, errors be-come compounded and their sources un-traceable.

A 1993 DoD Inspector General (IG) Auditfound that as many as 95 percent of modelsand simulations surveyed (and in use) hadnot fully incorporated VV&A.4 The IG�sreport recommended development of DoDpolicy, guidance, standards and criteria forVV&A of models and simulations.

At the time of this writing, official DoD

6-4

policy on VV&A was being developed, aswere official Navy, Marine and Air Forcepolicies. Department of The Army Pamphlet(DA PAM) 5-11, VV&A of Army Models andSimulations, provides methods, techniquesand procedures to conduct VV&A for M&S.However, overall Army policy with respectto development, use and management (in-cluding VV&A) of M&S is set forth in ArmyRegulation (AR) 5-11, Army Model andSimulation Management Program.

Some useful information on VV&A is avail-able from the Susceptibility Model Assess-ment and Range Test (SMART) projectsponsored by the Joint Technical Coordinat-ing Group on Aircraft Survivability (JTCG/AS). Although focused on survivability andvulnerability M&S, the SMART project hasdeveloped methodologies for VV&A andconducted research on VV&A proceduresthroughout DoD that may assist programoffices.5

6.3.3 Treatment of Legacy Models

Some models for years have been, and stillare, workhorses within the various functionalcommunities. These models have usuallyprovided information upon which solid de-cisions have been based - decisions that havelater been justified by actual weapon sys-tems performance in testing or in the field. Itwould be senseless to preclude the use ofthese models. Special provisions within mostpolicies must permit grandfathering of mod-els developed prior to the implementationdate of formal VV&A policies.

However, even for a legacy model, it is ab-solutely essential that the accreditation au-thority, as a minimum, step through the logi-cal thought process that establishes whetheror not it is appropriate for use in a certainapplication.

6.3.4 VV&A: When should it be done?

It is far better to plan and provide resourcesfor VV&A up-front, than to attempt recov-ery later.

The VV&A of models is neither an after-thought, nor a one-shot affair. An integralpart of the model�s development processmust be VV&A. One benefit of this ap-proach is that it becomes tougher to yield tothe temptation of cutting out VV&A, whilecontinuing apace with model development.Otherwise, VV&A is in danger of beingviewed as an unnecessary evil, and being cutwhen resources decline.

As a process, it is also much simpler to con-duct V&V as a model is being developed ormodified, than to try to re-construct the in-formation required later.

Managers who prefer not to live on the edge,should ensure the V&V status of the mod-els and simulations used to feed the analy-ses supporting their decisions. The M&Smust also be periodically re-accredited (AR5-11 specifies every five years), to ensurethat changes in the world around it have notimpacted its credibility. Re-accreditation isnot only when a model is changed or usedfor a new application. This also gives themodel manager an opportunity to reviewrecent usage of the model, and make a de-termination concerning its continued utilityto the users. The requirement for periodicre-accreditation also enforces the concept ofV&V as continuous processes through thelife cycle of an M&S.

Acquisition managers must understand thatthe emphasis on VV&A is increasing. Dataused in decision-making must be defensible.

6-5

It is a lot easier to challenge the credibilityof analyses on the basis of the VV&A sta-tus of underlying M&S; rather than to chal-lenge the analyses themselves.

6.3.5 VV&A: Getting it done.

The thrust of VV&A is to establish a degreeof confidence in the analyses resulting fromthe use of an M&S. The scope of technicalevidence that must be presented, before ex-pecting a management decision with respectto the credibility of an M&S as a tool, isdepicted in Figure 6-2.6

Acquisition managers who require de-velopment of M&S will be well-advised toensure that the following four steps are

part of the process.

1. Contracts call out the appropriate V&Vtasks and criteria.

2. Resources to accomplish the requiredV&V activities (in-house and contractually)are provided.

3. Data are collected at appropriate pointsfor validation, and that the resources re-quired for test activities are provided.

4. Program schedules allow for the per-formance of V&V tasks required, includingthe administrative processing requirementsto obtain formal sign-off by the accredita-tion authority.

Figure 6-2. M&S Confidence Assessment Scope-of-EvidenceFigure 6-2. M&S Confidence Assessment Scope-of-EvidenceFigure 6-2. M&S Confidence Assessment Scope-of-EvidenceFigure 6-2. M&S Confidence Assessment Scope-of-EvidenceFigure 6-2. M&S Confidence Assessment Scope-of-Evidence

M&S Confidence Assessment Scope-of-EvidenceM&S Confidence Assessment Scope-of-EvidenceM&S Confidence Assessment Scope-of-EvidenceM&S Confidence Assessment Scope-of-EvidenceM&S Confidence Assessment Scope-of-Evidence

DOCUMENTATIONCONFIGURATIONMANAGEMENT

VERIFICATIONVALIDATION

PRODUCTDEVELOPMENT

STUDIES &ANALYSIS

USER/ANALYSTDOCUMENTATIONDATA

QUALIFFORMAL

VALFORMAL

VERIF

DATAQUALIF

DATAQUALIF

DATAVAL

DATARELEASE

OUTPUTVALCODE

VERIF

LOGICALVERIF

STRUCTURALVAL

SIMULATIONCONFIDENCE

6-6

6.3.6 Who�s Responsible for VV&A?

Sponsor, developer, requester or user.Whether the developer or the sponsor, par-ticipation in the process is critical. Every-one has a vested interest in ensuring thatcredible and supportable decisions resultfrom the analyses fed by the use of a model.

Model developers are most familiar with themodel�s construction, and are normally bestequipped to verify that it has been built theway it was intended. This does not negatethe requirement for independent verificationof the design.

Likewise, functional experts are normallythe best equipped to determine whether themodel portrays the real world accuratelyenough. However, here too, an independentvalidation is required.

Accreditation is basically a matter of accep-tance. It answers the question, �Am I goingto accept the output of the model?� This isthe model user�s decision, since it is nor-mally this person who has to defend any rec-ommendation resulting from analyses of themodel�s output. It is for this reason that themodel sponsor/requester/user remains re-sponsible for ensuring that V&V has beenperformed on the model or simulation. Cer-tainly, one way to accomplish this is to in-clude a requirement for documentation ofV&V activities and stipulate acceptabilitycriteria in the Request For Proposal (RFP)and Statement of Work (SOW).

When designing the VV&A plan to gatherrelevant information to support the accredi-tation, the question of �How the model(s)will integrate� must be considered.

6.4 Integration

This guide�s focus, with regard to integration,principally refers to the extent to which mod-els and simulations are able to cross the barri-ers between the various functional communi-ties. To effectively support weapon systemsacquisition, models must be able to supportthe implementation of Integrated Product andProcess Development (IPPD). Their principalcontribution is in improving the cross-func-tional communications among members ofIntegrated Product Teams (IPT).

Existing models and simulations used insupport of acquisition are mostly of the typethat support analyses within the individualmajor functional areas; concepts and re-quirements development; design and engi-neering; logistics, manufacturing and pro-duction; test and evaluation; training; andprogram management.

There are distinct cases of limited cross-functionality between selected areas. How-ever, for the most part, the use of modelsand simulations appear to be functionallystovepiped. An acquisition environment sup-porting increased integration among func-tional areas is needed to break out of thestovepipes. This is an envisioned infrastruc-ture that would support the informationneeds of users, developers and decision-makers to perform integrated cross-func-tional analyses throughout the acquisitionprocess.

As an objective, it is not enough for IPT mem-bers to be able to just accept the validity ofoutput from models in other functional disci-plines�they must be able to use the output toassess the cross-functional implications.

The Director of Defense Research andEngineering�s (DDR&E) Acquisition Task

6-7

Force on Modeling and Simulation(ATFM&S) formed an IPT Focus Group to:

� Identify needs for models and simula-tions and related tools that support IPTs inan integrated acquisition environment; and

� Develop technology and non-technologyrecommendations that meet those needs.

The final report of the ATFM&S expressesthe need to develop links between varioustypes of design and engineering models andsimulations. Some of the other key acquisi-tion functional areas need to:

� Evaluate performance of alternative de-signs in combat environments;

� Assess logistics impact of alternativedesigns;

� Evaluate manufacturing and pro-ducibility implications of alternative designs;and

� Evaluate cost, schedule and perfor-mance impacts of alternative designs.

The creation of such linkages permits theevaluation of alternative weapon systemdesigns on the basis of the implicationsacross other critical functional areas - in ef-fect providing the means for moving towarda broader concurrent engineering frame-work. Figure 6-3 depicts the concept of in-creasing cross-functional integration of ac-quisition M&S.

Figure 6-3. Increasing Cross-Functional Integration of M&S in Systems Acquisition

Increasing Cross-Functional IntegrationIncreasing Cross-Functional IntegrationIncreasing Cross-Functional IntegrationIncreasing Cross-Functional IntegrationIncreasing Cross-Functional Integrationof M&S in Systems Acquisitionof M&S in Systems Acquisitionof M&S in Systems Acquisitionof M&S in Systems Acquisitionof M&S in Systems Acquisition

CostCostCostCostCostModelsModelsModelsModelsModels

ManufacturingManufacturingManufacturingManufacturingManufacturingModelsModelsModelsModelsModels

LogisticsLogisticsLogisticsLogisticsLogisticsModelsModelsModelsModelsModels

Design/Design/Design/Design/Design/Eng’gEng’gEng’gEng’gEng’g

ModelsModelsModelsModelsModels

SharedSharedSharedSharedSharedVisionVisionVisionVisionVision



IntegratedIntegratedIntegratedIntegratedIntegratedAcquisitionAcquisitionAcquisitionAcquisitionAcquisition

EnvironmentEnvironmentEnvironmentEnvironmentEnvironment

IncreasedIncreasedIncreasedIncreasedIncreasedCross-FunctionalityCross-FunctionalityCross-FunctionalityCross-FunctionalityCross-Functionality

Stand-aloneStand-aloneStand-aloneStand-aloneStand-aloneFunctionalFunctionalFunctionalFunctionalFunctional

“Stovepipes”“Stovepipes”“Stovepipes”“Stovepipes”“Stovepipes”

LimitedLimitedLimitedLimitedLimitedCross-FunctionalCross-FunctionalCross-FunctionalCross-FunctionalCross-Functional

EvaluationsEvaluationsEvaluationsEvaluationsEvaluationsNEAR-TERMNEAR-TERMNEAR-TERMNEAR-TERMNEAR-TERM


LONG-TERMLONG-TERMLONG-TERMLONG-TERMLONG-TERM

TODAYTODAYTODAYTODAYTODAY

6-8

With increasing cross-functionality, comesthe need for an increasing commitment to ashared vision. Besides the organizationalstructure, such a shared vision would alsoneed a commitment to the use of standards,when appropriate. Models supporting thevarious acquisition functions would need tobe able to interact with each other, sharingdata and information which will, in effect,allow them a common view of the same elec-tronic space.

6.5 Advanced Distributed Simulation (ADS)

A significant emerging capability of real-time simulation is the ability to syntheticallycreate large environments within which largenumbers of subjects can interact. These vir-tual worlds are made possible by electroni-cally linking individual simulations such thatthey cooperate in a shared representation ofspace. Termed ADS, the movement to createthese virtual worlds is being driven by DoD�sattempts to harness the potential opportuni-ties resulting from the explosive growth ininformation processing technology.

The capability demonstrated by ADS is thetype required to truly integrate the variousfunctional aspects of weapon system acqui-sition. The Defense Science Board has stud-ied the potential impact of ADS onprototyping, and determined that this tech-nology can provide the means to �transformthe acquisition system from within.�7

The concept of ADS carries with it two dis-tinct characteristics. The first is that of physi-cal separation; the second is that of electronicintegration, i.e. two or more physically sepa-rate, and separated, models and simulationsinteracting with each other because theyshare a common view of their electronicenvironment. The output resulting from theexecution of each model is seen, interpreted

and acted upon by the other(s) in real-time.

The Distributed Interactive Simulation (DIS)environment is an infrastructure whichimplements the concept of ADS. In the cur-rent DIS concept, the world is modeled as aset of entities that interact with each otherby events that they cause. Whenever the stateof an entity changes, such changes arebroadcast over the communications media,seen by all other entities, ignored by someand effect others: possibly causing them torespond, resulting in further interaction be-tween the entities within the models.

A key task of the DIS community in defin-ing and providing the infrastructure requiredto combine individual simulations into aseamless virtual world is the establishmentof a series of standards in the areas of:

� Interface definition� Communication� Representation of the environment� Management� Security� Field instrumentation� Performance measurement

The initial focus of applying ADS has beenon training. In-roads are being made in itsuse for evaluating new concepts in combinedarms doctrine, tactics, and to some extent,joint interoperability. To fulfill the vision ofbroader military applications will require itsextension into test and evaluation; missionrehearsal; and research, development andacquisition (RD&A).

The highest potential pay-off, at present,appears to be in several areas of RD&A.

� Assisting in defining requirements fornew battlefield systems, thereby reducing therisks of embarking on lengthy and expensive

6-9

development programs.

� Playing a key role in conducting certainaspects of early user testing and evaluation,as a weapon system design matures. Fre-quent involvement by the user in a syntheticenvironment will make the design-test-fixcycle faster. The use of simulation duringdesign, testing and prototyping a design fix(or model-test-model) can ensure an accept-able fix before major investment of funds inan actual build.

� Conducting portions of weapon sys-tems� developmental and operational testingand evaluation (DT&E and OT&E) in a syn-thetic environment. As noted in Chapter 5,several aspects of DT&E already involve theuse of simulation. Although models andsimulations have been used fairly exten-sively in support of OT&E, extending theuse of ADS to conduct OT&E is relativelyfertile ground. DoD seeks to apply a higherstandard of acceptability for models or simu-lations used in support of OT&E, requiringthat �special care is necessary to ensure theyare credible�.8 Understandably so, sinceOT&E is often the last line of defenseagainst carrying risks into the next phase ofacquisition. However, with VV&A for dis-tributed M&S being more complicated thanfor monolithic or stand-alone M&S, estab-lishing sufficient credibility for their use insupport of OT&E, will carry some specialchallenges.

Several technical issues still limit the full ex-tension of ADS into the acquisition domain.

� Program managers (PMs) intending toconstruct models for use in the DIS domainshould familiarize themselves with the is-sues9, and track their resolution over time.

� PMs should recognize the importance of

being in control of the manner in whichtheir system is represented and used on thesynthetic battlefield.

� In the longer term, PMs electing not tobuild DIS-compliant simulations, will runthe risk of one being developed and usedoutside their control.

� One of the critical technical issues in-volved in extending the use of ADS into ac-quisition remains that of establishing stan-dards; permitting the models and simula-tions to interact.

6.6 Standards for the Modeling and Simu-lation Environments

Charles D. Sullivan defines a standard as �acategory of documents whose function is tocontrol some aspect of human endeavor.�10

Standards are established to overlay a cer-tain discipline on a process or product. Un-less such discipline is established with spe-cific purposes in mind, control ends up be-ing established for no reason other than theexercise of power. The bad press DoD�s ap-plication of standards has had in recent yearsis not the result of the existence of these stan-dards; rather, it is the sometimes blind im-position or implementation of them.

The objective behind the creation of a stan-dard must be considered from two, oftenvery different, perspectives: �Why create astandard?� and �Why use one?� The clearerthe objectives behind the creation of a stan-dard, the more likely it is that the standardwill be used by those for whom it is intended.

If there were ever a discipline that could gainsignificantly from implementation and en-forcement of a reasoned approach to estab-lishment and adoption of standards, it is thatof the development of models.

6-10

Currently there is no universal standardwithin DoD for models and simulations.This appears to be a conscious decision toforce an evolution toward a set of broad stan-dards that could serve wider needs within theacquisition community. There are standardsdevelopment efforts underway within thevarious functional communities and appli-cation domains. Over time, the need to effi-ciently share information between functionalareas will provide a forcing function to evolvea set of common standards. These need tobe broad enough to encompass the needs ofeach community without requiring unaccept-able compromises, yet definitive enough topermit cross-functional interaction.

Even a model developed to comply with acertain set of standards is at risk of driftingover time, as its configuration changes. Toensure the continuing ability of a model toserve the purpose it was built for, its cred-ibility will have to be maintained over itslife cycle through a planned maintenanceand re-accreditation process.

6.7 Maintaining Credibility of M&S

Having established a model�s suitability foruse, whether for a class of applications orfor a specific one, a rigorous effort is re-quired to ensure that it remains credible.

Since a portion of most models or simula-tions is software, loss of configuration con-trol can occur insidiously, over time, and in-visibly. An Army Audit Agency AdvisoryReport mentions that, over a period of twoyears, �one model was modified 623 times(including 9 major changes) without revali-dation�.11

Managing and controlling a model�s con-figuration becomes crucial in maintainingit as a credible tool to support analyses.

6.7.1 Configuration Management (CM)

During the process of developing DA PAM5-11, VV&A of Army Models and Simula-tions, the writers concluded that a strong CMplan is one of the critical ingredients in en-suring the continued credibility of modelsand simulations. The CM plan for a modelmust ensure controls for the model itself,its development process and the input data.

Configuration management is defined12 as adiscipline applying technical and administra-tive direction over the life cycle of an item to:

� Identify and document the functionaland physical characteristics of the item andits major parts;

� Control changes to these parts and totheir related documentation;

� Establish a process for maintaining sta-tus of proposed changes, implementationstatus of approved changes, etc.; and

� Establish a process for conduct of au-dits to verify conformance of the item�s (andits major parts�) design and performancewith requirements documentation.

This general definition covers any configu-ration item within a system, be it hardware,software or firmware.

There are several reasons why a good CMis an especially important activity for modelsand simulations.

1. Facilitates repeatability: Maintaininga record copy of an M&S used in providinginformation to support an analysis, alongwith its associated input data, provides anM&S user the ability to reconstruct theanalysis at a later date.

6-11

2. Enables traceability: Maintaining aclear audit trail of changes to a model andinput data provides a mechanism for corre-lating each change to the circumstance(s)generating the requirement. This allowsproper analysis of the cause-and-effect re-lationships among a series of experiments.

3. Maintains credibility: This is probablythe strongest argument for a rigorous CMprogram, and is directly linked to the previ-ous two items. A well constructed and imple-mented CM plan for a model will signifi-cantly reduce the risk of its data output be-ing misused in an analysis.

4. Maintains interfaces: Integrated,interoperable and interactive models andsimulations are becoming more importantto efficient conduct of DoD�s business. It iscritical to identify and control any changeswhich may disrupt this interoperability.

Three elements are considered essential foradequate CM:

1. A CM policy establishing the adminis-trative processes for approving and docu-menting changes to the model or simulation;

2. A CM plan describing how changes tothe existing configuration will be accom-plished; and

3. A CM board or official with authorityto approve proposed changes to the currentlyapproved configuration.

Some managers have encountered problemsfollowing release of a model and recommendmore stringent measures to protect the con-figuration. The following guidelines haveproven effective for several model managers.

� Maintaining a baseline copy of each version

� Tight control of changes by users(implementing a policy of not releasingsource code)

� Execution of various forms of agree-ments with model users

� Formation of user�s groups that are par-ticularly useful in addressing CM issues per-taining to:

� Documenting changes made (or sug-gested) by users other than the modelmanager;

� Evaluation of proposed changes; and

� Suggesting and developing method-ologies for V&V of changes.

Across the board, one of the most usefulmechanisms for PMs appears to be mainte-nance of a record copy (baseline version pluschanges) of each model or simulation used,along with the validated data for that ver-sion. This is of tremendous benefit to themodel user since analyses supporting a de-cision must often be traceable to the spe-cific configuration of a model exercised: toprovide information for the analyses and thedata used in its execution.

Any CM can be resource intensive. Manag-ers are advised to assess any potential im-pacts. The PMs need to ask such questionsas:

� What resources are going to be requiredto implement an effective CM program?

� Do program management office per-sonnel have the required training (technicaland other) to oversee CM activities, facili-tate user group meetings, etc.?

6-12

� If the model manager maintains totalcontrol of source code, does the office havethe resources to be the only one makingchanges to the model? If not, will users bewilling to provide reimbursement for theeffort(s)?

If the objective is to get others to use a modelbecause of its ability to improve a processor product, it does not make much sense toimplement inordinately stringent controls.However, there may also be valid concernsabout proliferation of unauthorized versions,

loss of credibility in the model as a tool, etc.The structure of the CM process shouldimplement a level of control needed, with-out making it unnecessarily hard for poten-tial users to get the model and use it.

A good CM plan is critical to maintain thecredibility of a model, ensure its continuedcompliance with standards to which it wasdesigned and ensure its overall utility to theprogram. Investment in good M&S CM mayminimize future problems with credibility.

1. DoD Directive 5000.59.

2. Ibid.

3. Ibid.

4. DoD IG Audit Report No. 93-060, �Duplication/Proliferation of Weapons Systems� Modeling andSimulation Efforts Within DoD�, March 1, 1993.

5. For further information on SMART project, con-tact NAWCWPWS, 1 Administration Circle,ATTN: C21806, China Lake, CA 93555-6001.

6. Adapted from US Army SSDC �Extended Air De-fense Test Bed� Status Briefing, March 11, 1993.

7. "Impact of Advanced Distributed Simulation onReadiness, Training and prototyping�: Report ofthe Defense Science Board Task Force on Simu-lation, Readiness and Prototyping, January 1993.

8. Director, OT&E, 23 Jan 89 Memorandum promul-gating �Policy Guidance for the Application ofModeling and Simulation in Support of Opera-tional Test and Evaluation�.

9. "The DIS Vision: A Map to the Future of Distrib-uted Simulation�, (Comment Draft), October1993.

10. "Standards and Standardization: Basic Principlesand Applications�, Marcel Dekker, Inc., 1983.

11. Army Audit Agency Advisory Report, �Develop-ment of Computer-Based Models and Simula-tions�, July 12, 1991.

12. Mil-Std 973, �Configuration Management�, April17, 1992.

ENDNOTES

7-1

77MANAGEMENT CONSIDERATIONSIN MODELING AND SIMULATION

3. Identify improvements the DoD acqui-sition community felt were most needed toprovide better support to acquisition pro-grams in the future.

7.2.2 Survey Method

The survey was sent to selected programoffices across the Army, Navy, Marine Corpsand Air Force. Selected survey responseswere followed-up through site visits and in-terviews (one-on-one and group).

7.2.3 Summary of Survey Results

The results of the 56 responses provide somevaluable insights into M&S use. Responseinformation pertaining to how PMOs arecurrently using M&S is summarized in Fig-ure 7-1;1 and further explained below.

7.2.3.1 Modeling and Simulation Use

A majority of respondents indicated mod-erate-to-extensive use of models and simu-lations to support acquisition decision pro-cesses. While the largest number of useswere in the threat, requirements, cost andtesting areas; the largest single use by pro-

7.1 Introduction

This chapter will consolidate some of themanagement considerations involved in theuse of models and simulations in systemsacquisition. It will also provide a notionalprocess to plan the modeling and simula-tion (M&S) effort for a program.

7.2 Survey of ACAT I & II PMOs

In September 1993, the Defense Modelingand Simulation Office (DMSO) AcquisitionTask Force on Modeling and Simulation(ATFM&S) initiated a survey of DoD ACATI & II Program Management Offices(PMOs).

7.2.1 Purpose

The purpose of the survey was threefold:

1. Gain information on how the armedservices were currently using M&S;

2. Determine the degree of existing cross-functionality in current models and simula-tions; and

7-2

gram managers (PMs) was in the area ofsystems engineering and performance evalu-ation. Overall, the survey confirmed thatmodels and simulations were being usedacross a variety of functional areas.

7.2.3.2 Modeling & Simulation Needs

The survey results, with respect to expressedneeds of PMOs, are summarized in Figure7-2.2 Some of the key issues recommendedfor addressal were to provide adequate re-sources for M&S efforts, consistency in cost

models, user-friendly models and simula-tions, and more and better information(training and reference material) on the useof models and simulations.

With respect to what capabilities the PMswould like to see improved, the majority ofresponses were in the areas of: improvedscheduling models; improved cost and op-erational effectiveness analysis (COEA) andlife-cycle cost (LCC) tools; improvedinteroperability with other models; and im-proved program specific models.

Figure 7-1. How Program Management Offices are CurrentlyUsing Modeling & Simulation

How PMOs Are Using M&S Today(ATFMS Survey of Select ACAT I & II PMOs)

ACQUISITION TASK FORCE ONMODELING & SIMULATION (ATFMS)

MILESTONE DECISION SUPPORT

EXTENSIVE USE

CONSIDERABLE USE

MODERATE USE

LIMITED USE

NONE USED

50 10 15 20 25RESPONSES MARKED*

6

13

16

12

7

OVERALL M&S CONTRIBUTION

EXTENSIVE

CONSIDERABLE

MODERATE

LIMITED

NONE


6

24

17

6

4

M&S APPLICATIONS BY FUNC AREA

THREAT

REQUIREMENTS

COSTS

TESTING

PRODUCIBILITY

SUPPORTABILITY

PROGRAMMATICS

ENVIRONMENTAL

500 100 150 200 250NO OF M&S APPLICATIONS

6

SYS ENGR & PERF EVAL SUPPORT

EXTENSIVE USE

CONSIDERABLE USE

MODERATE USE

LIMITED USE

NONE USED


12

17

14

5

5

*CHARTS DO NO HAVE SAME TOTAL RESPONSES

7-3

7.2.3.3 Centralized vs DecentralizedManagement

The survey indicated a predominance ofdecentralized management, wherein themodel user(s) managed the development anduse of models they needed for their particu-lar functional area. However, the survey in-dicated a strong correlation between the typeof management of M&S efforts and per-ceived overall contribution from their use.

The PMOs reporting centralized manage-

ment perceived the overall contribution asconsiderable to extensive. This relationshipis strong enough to warrant considerationof centrally managing the M&S effort withinthe PMO. Each PM must make a consciouschoice in the type of M&S management toimplement. Factors to consider in determin-ing whether to centralize management un-der a simulation manager include: availabil-ity of an individual with the requisite breadthof knowledge and training; the PM’s man-agement style; the PM’s organizationalstructure; and the scope of the effort.

Figure 7-2. Summary of Modeling and Simulation Needs Perceived by PMOs

Summary of Needs Perceived by PMOs(ATFMS Survey of Select ACAT I & II PMOs)

ACQUISITION TASK FORCE ONMODELING & SIMULATION (ATFMS)

POLICY & ORGANIZATION

TECHNOLOGY

INFORMATION

DESIRED M&S CAPABILITY(TO SUPPORT MILESTONE DECISIONS)

l CONSISTENCY IN COSTING/COST MODELING

l EARLIER PROG OFFC INVOL IN REQMTS DEV

l PROG OFFC USING SAME M&S AS USERS

l MAINTAIN FLEX IN M&S CHOICES FOR BEST TOOLS

l FUNDED CTRS OF EXCELL TO MAINTAIN M&S

l CTR PROVIDES CONSISTENT TOOLS & DATABASES

l USER FRIENDLY M&S

l EFF H-LEV SIM LANG’S

l AUTOMATED VV&A AIDS

l CASE TOOLS

l FLEX RES MGMT TOOLS

l MULTI-LEVEL SEC CAPAB

l MORE INTEGR SIM CAPAB

l SYNCHRONICITY W/IN DIS

l DIS COMP W/T&E REQMTS

l M&S HANDBOOK ORIENTED FOR PROGRAM

l IMPROVED DOCUMENTATION OF M&S

l TOOLS & POLICY TO HELP TAILOR STDS, REGS & DIDS

l A MEANS TO SHARE INFO ABOUT:

?M&S TOOLS

?DATABASES & DATA AVAILABILITY

?BEST PRACTICES

ATFMS

IMPROVEDSCHEDULING MODELS

(6%)

OTHER COMMENTS(23%)

IMPROVEDCOEA/LCC MODELS

(41%)

IMPROVED PROGRAMSPECIFIC MODELS

(18%)

IMPROVED INTEROPERWITH OTHER MODELS

(21%)

7-4

The scope of the M&S effort within a PMOwill be determined by factors such as thenumber of models and simulations devel-oped, used and maintained; the purposes forwhich they are employed; and their com-plexity and sophistication. A universalmodel (one that models every aspect of theprogram) for a given program, is panacea.A given program will normally require theuse of several models and simulations, de-sirably using common data bases. Theirmanagement is not trivial.

7.3 Planning the Modeling and SimulationEffort

A manager, when initiating the M&S plan-ning effort, must take a bottoms-up ap-proach. The tool has to fit the need, requir-ing managers to start determining their M&Srequirements based upon what they are try-ing to accomplish.

The decision to employ models and simula-tions, as well as the subsequent decision(s)to use or modify an existing model, or cre-ate a new model, will depend on the circum-stances and specifics of a given program.The questions which follow are not offeredto trivialize the process with a cookbooksolution.

The templates in Appendix G provide abaseline from which PMs can develop ques-tions reflecting their programs. The support-ing functional managers can then stepthrough the questions below to fill in theblanks.

• What am I trying to achieve? What’sthe objective? What question(s) am I tryingto answer?

• What analyses will have to be con-ducted? When will the results be needed?

How long will the analyses take? Who willdo them?

• What information is required to supportthe analyses? How accurate does the infor-mation have to be?

• What’s the most efficient way to get theinformation? Are several excursions (or it-erations) going to be required? Is it a one-time requirement, or will this be an on-go-ing requirement? Do I need a model to pro-vide the information?

• Can any existing models or simulationsprovide the information I need? What is theverification, validation and accreditation(VV&A) status? Are they accredited for myclass of system? Will they need modifica-tion? What’s the extent of modification? Canthe model owner(s) do the modification in-house? Can I? Any proprietary issues thatmay lock me in to a sole source?

• What data are required by thesemodel(s)? Where and how can the data beobtained?

• What resources (funds, people, time,test articles, hardware, software, range fa-cilities, documentation) will I need to:

— build or modify the model(s)?

— conduct VV&A on the model(s) ormodification?

— implement configuration manage-ment (CM)?

— obtain and validate the data?

— run the model(s)?

— analyze the output?

7-5

— ensure that the model(s) are main-tained to accurately represent my system orprogram?

— transition the models and simula-tions to a supporting activity for mainte-nance upon dissolution of my PMO? Does“operation and support” funding provide formodel(s) maintenance after transition?

• Does the system design accommodateplans for hardware/software-in-the-loop(HW/SWIL) with regards to test and instru-mentation ports, etc. What do I need to popu-late my test bed(s) or simulationfacility(ies)?

Are my models and simulations consistentand integrated with the rest of my program?Are they reflected as tools contributing to re-quirements verification in a requirements cor-relation matrix (RCM)? Are their characteris-tics consistent with the COEA, test and evalua-tion master plan (TEMP), operational require-ments document (ORD), and acquisition pro-gram baseline (APB) with respect to measuresof outcome, effectiveness and performance(MOOs, MOEs, and MOPs)?

Once managers have completed this thoughtprocess, all the ingredients of a plan are inplace.

• The tasks, functions or decisions to besupported by M&S

• The specific M&S tools required

• When they are needed

• How the M&S are going to be acquired

• The resources required to acquire andmanage the M&S

7.3.1 The Simulation Support Plan

In an attempt to ensure early considerationand planning the use of M&S in major pro-grams, the Army has mandated developmentof a Simulation Support Plan (SSP) for allACAT I and ACAT II programs, as well asfor Advanced Technology Demonstrations(ATDs). The SSP forces PMs to view theentire program in the context of the deci-sions to be made, timing, and impact (rela-tive importance). It forces managers to con-sider information needs in light of the deci-sions to be supported, and assess the appli-cability of models and simulation to provid-ing the information. Figure 7-3 is a proposedoutline for an SSP.

PMs must consider the resources requiredto build the program, which includes the SSP.The SSP is considered an evolutionary docu-ment, and is supposed to be refined, throughperiodic review, as the program progresses.Like other components of an acquisitionprogram, the M&S requirements will coa-lesce and get more detailed over time.

It is not the SSP itself, but the “journey” throughthe process of identifying the program’s M&Sneeds that is more valuable. Creating a bureau-cracy that simply requires “another plan”,would be counter-productive.

One of the initial challenges a manager willface is in trying to identify existing resourcesthat could be used (either as is or with modi-fications) to address the M&S needs. Aplethora of models and simulations have al-ready been developed and are in variousstages of accreditation for different pur-

7-6

Outline for a Simulation Support Plan

I. Purpose• Brief statement as to why plan is required

- Focus on the use of M&S in the program

II. Executive Summary• Summary narrative of Section V.

III. System Description• Brief summary of weapon system

IV. Program Acquisition Strategy• Brief synopsis of system acquisition strategy• Overview of M&S acquisition strategy

- Include role of weapon system M&S in the distributed environment

V. Simulation Approach/Strategy and Rationale• What M&S is being done, and why

A. M&S used to date• Discuss all previous M&S used to support the program

- Name/Type of M&S (Live, Constructive or Virtual)- V&V performed on M&S - Accreditation status of M&S- To what phase/milestone was M&S applied- Issues addressed and results- Include M&S supporting:

— Mission area analyses— Operational analyses— Requirements trade-offs— Conceptual design studies— Systems engineering trade-offs— Cost and operation effectiveness analyses— Logistics analyses— Test and evaluation— Training

B. Future Simulation• Include on-going M&S• All planned M&S for future milestones• How planned M&S will support future milestones• How planned M&S supports the Service’s vision for M&S

(continued)

Figure 7-3. Outline for a Simulation Support Plan (SSP)

7-7

Outline for a Simulation Support Plan (continued)

VI. Related Simulation ActivitiesInclude• Other M&S activities the system relies upon• Other systems that rely upon this system’s M&S tools• All other related M&S that affect this system

VII. Management• Provide wiring diagram of PMO• Show simulation manager (if assigned) in diagram• Describe how simulation manager interacts with acquisition community

VIII. Facilities/Equipment RequirementsDescribe facility requirements for all M&S• All facilities, hardware, software, data, etc.

- Provided by PM, other Gov’t activities, contractor(s)- Identify who will provide- Identify schedule requirements and availability of items to support

schedule

Ensure government ownership of equipment (including simulators,hardware, software, data, etc.) critical for cost effective governmentmanageme nt of M&S

IX. Funding• Outline all expected expenditures to support M&S program• Include funded and unfunded• Designate type of funding (by Program Elements (PE), project, etc.)

X. Remarks/Supplemental Information• Any comments or related information

XI. Appendices• Program Schedules• M&S Schedules• Acronyms and abbreviations• Related standards• Related government documents

Figure 7-3(Cont’d). Outline for a Simulation Support Plan (SSP)

7-8

poses. However, the PM must know how toget the information needed to make a deci-sion as to whether one or more of these couldsatisfy an M&S need.

7.3.2 Models and Simulations Catalogs

The discipline of modeling has been likenedto a cottage industry, with a proliferation ofmodels among the various Services, andeven within individual services. A March1993 DoD Inspector General’s (IG’s) auditof selected models and simulations and theirdeveloping and using activities concludedthat “Model and simulation projects are be-ing procured and developed within the DoDwithout adequate coordination and control”,and that “This has resulted in redundantmodels/simulations and a proliferation ofsystem architectures and libraries”.3

The IG reports: results can be contributedto lack of communication. One Service notbeing aware of what models the others have.This can also be extended to the Serviceslack of awareness of the applicability, modi-fiability, portability, or V&V status of thosemodels within their respective Service. Thisissue is being addressed, to some extent, bythe creation of models and simulations cata-logs by each Service and the Joint Staff.These efforts, while valuable, have not re-sulted in complete listings. Many modelusers are unaware of the existence of thesecatalogs; many others, though aware of theirexistence, may not have submitted theirmodels for inclusion.

Appendices A through E include referenceto the service catalog(s) and information onhow to gain access to the information. Ap-pendix F includes a partial list of othersources of information. On-line access tomany catalogs is also provided by the Ser-vice sponsor, and by the DMSO, through

their Modeling and Simulation InformationSystem (MSIS).

Model developers and users should use thecatalogs as a starting point while preparingan SSP. However, they are not all-inclusive;so, even if they do not list a model that sat-isfies the need, points of contact (POCs) forsimilar models may be able to provide someadditional leads.

The PMOs often do not have resident ex-perts to answer the questions required to plantheir use of M&S. Often times DoD PMOsmust turn to contractors, consultants or otheragencies for assistance in determining theM&S needs for the program, as well as de-veloping the M&S tools.

7.3.3 Selecting a Modeling and SimulationsDeveloper

This section will assist the PM in identify-ing or selecting a contractor to assist in theM&S effort.

The PM must identify a M&S developer.This could be a government agency, an in-dependent contractor or the prime systemcontractor. Obviously, the developer mustunderstand M&S and the PM’s unique re-quirements. A list of questions that may helpthe PM select a developer is provided by VanB. Norman.4 This list is based upon twenty-years of building simulation models, andhiring and managing simulation consultants.

In DoD’s case, these questions can form thebasis for part(s) of a Request For Proposal(RFP), as well as provide ingredients forestablishing Source Selection Criteria.

The specific questions, and the level towhich they will have to be pursued, will de-pend upon certain factors within the program

7-9

acquisition strategy. Among these are fac-tors such as: whether the main effort and thesimulation effort are under a common con-tract; who is integrating the simulation ef-fort; whether the system and simulation de-velopment efforts are complementary, witheach leveraging off the other; or whetherthey are independent.

a. What is the contractor’s experience withthis type of system? What is the contractor’strack record?

A contractor’s experience with similar sys-tems is important, since it normally gener-ates efficiency, but it is not essential. If aspecific contractor has an unproven record,a software capability evaluation5 may pro-vide some insight into their ability to takeon a complex software modeling effort. Thisis particularly true if the M&S developmentis a parallel effort to the system develop-ment. The evaluation augments the acquisi-tion process by determining a contractor’sstrengths and weaknesses with respect to amaturity model. It establishes a method forcomparing a contractor’s “software capabil-ity” against a standard set of criteria as asource selection criterion.

b. How will the contractor approach theconstruction of the model? What simulationsoftware will be used?

Maybe there is a requirement for a specificsoftware tool. Ada may not be the most ap-propriate—if possible, choose a simulationtool that is widely used and will be aroundfor a few years.

c. Will the contractor produce a writtenspecification describing the system to be mod-eled, including all assumptions and questionsto be answered? Are you going to be provid-ing the contractor a specification?

In any case, the specification is necessaryto ensure that everyone is working towardthe same goal.

d. What questions about the system can-not be answered from use of the model?

Models need to be constructed with certainquestions in mind. An understanding of whatthe model will not answer is crucial to pre-vent misunderstanding between the projectoffice and the developing contractor. An-other reason why the model specification isso important.

e. What is the development schedule?

The model or simulation supports certaininformation needs of the project office. Un-less this information is timely, it could beworthless. The contractor’s prior record withrespect to ability to deliver on scheduleshould be a important criterion in selection.

f. How did the contractor arrive at thecost estimate for the projects?

Regardless of whether the contractor isworking on a cost-plus or fixed price basis,the contractor must understand the scope ofwork and schedule to develop a credible costestimate.

g. How do I determine value for cost?

Norman likens simulation consulting tobrain surgery - “if you want the lowest pricedsurgeon opening your head, then goodluck”,6 he says. He emphasizes the need toknow the experience, expertise, and recordof each candidate contractor, and to balancethese against the price being charged to de-termine value.

h. What data are required for the model?

7-10

Norman contends that most contractorswon’t know your business well enough tocollect the required data. The governmentwill often need to provide the data. Fortu-nately, sources of valid data exist within eachService and DoD. Each Service’s M&S of-fice can provide authoritative sources ofdata. Despite the unique needs of each Ser-vice, a common dictionary of data is requiredamong the Services.

i. Who will collect the data? When will itbe needed? What format will it be needed?Who will certify the data?

Crucial questions, if the government is go-ing to be on the hook to provide these to themodeler. In fact, any potential disconnectbetween the contractor’s requirements fordata and the government’s ability to provideit must be worked out early. The PMs mustalso understand the resource implications ofdata collection.

j. What parts of the system will be detailedand what parts will be simplified?

Again, an issue that the contractor must ad-dress in the proposed specification.

k. What types of model experiments willbe run?

As the model is built, experimentation pro-vides answers. If the modeler knows whattypes of experiments are contemplated, themodel can be built to make the experimentseasier.

l. How much time will be allowed for ex-perimentation?

The user’s understanding of the system maychange after reviewing experiment results,and the scope of work may be impacted. A

well structured CM process will make thisless painful.

m. How will you be assured the model is“correct”?

The government must be an active player inthe V&V process. The PMs must also bemindful of the accreditation authority’s re-quirements from the start - the requirementsmust be built into the contract.

n. What is the schedule for periodic modelreview meetings?

This is a crucial management mechanism forensuring that incremental model develop-ment is on track from cost, schedule andperformance viewpoints.

o. Can you use the model internally afterthe contractor is done?

The model has long term value. The systemwill probably change over time and themodel must be modified. This relates backto the need for CM and the requirement foradequate documentation.

Reusability of software and the increasingmove toward reconfigurable simulators andsimulations make this even more important.Also, the government’s aversion to lockingitself into a developer drives the need forthe government to identify in the contractall its M&S deliverable requirements, andtiming. Inclusion of contractor proprietarymaterial or data, without adequate rightsbeing released to (procured by) the govern-ment, could lead to problems.

p. What could go wrong with this part ofthe project?

Monitor the model (or simulation) develop-

7-11

ment effort as it proceeds. Given the criteriafor establishment of Work Breakdown Struc-ture (WBS) elements (MIL-STD 881B), andConfiguration Items (MIL-STD 973), PMsmust:

— Ensure visibility of the M&S effortsat the appropriate level for managementand

— Incorporate the development effortsinto their risk management program.

q. What kinds of analyses will the con-tractor perform, such as confidence inter-val calculation and design of experiments?

Whether it is the contractor or someone elsewho is going to be performing an analysisusing the model’s output, the analyst’s re-quirements have to be considered in design-ing the model.

Norman rounds out his suite of questionsensuring that the contractor will assist ingaining management and team support forthe model and for its use in support of theanalyses at hand. The contractor is, after all,part of the team.

r. “How will the contractor assist in ex-plaining the benefits and limitations of themodel?” “Will the contractor assist in pre-senting the model results to management(decision makers)?” and “Does the contrac-tor have the capability to provide a video ofthe model’s animation?”

All of these will go far in gaining and main-taining support for the effort, since they cre-ate an understanding of the need for themodel and subsequent analyses of its results.

7.4 Models and Simulations as Deliverables

The PM must be aware that some modelsand simulations will be developed by theprime contractor as a natural by-product ofthe system design and development process.However, the capabilities and limitations ofthese models and simulations, with respectto the acquisition process, must be under-stood. Decisions regarding whether or notto require specific M&S as deliverables,must be made on a case-by-case basis withthis understanding.

Contractors may also be reluctant to sharekey algorithms included in simulationsspecified for delivery. Based on theprogram’s acquisition strategy, the PM mustassess impacts of any restrictions the con-tractor may include, and determine whether(and how much) it would be worth payingfor their removal.

A PM must also recognize that when pro-duction is complete and a contract ends,M&S support will still be required for theremainder of the system’s life cycle. Mod-els and simulations that were constructedduring earlier phases of the acquisition pro-cess, and refined as the system evolved, willplay a major role in evaluating system modi-fication alternatives.

7-12

ENDNOTES

1. “Weapons Systems Acquisition Cycle Improve-ment (Through Integration of Modeling and Simu-lation),” Report of the US Army Materiel Com-mand Task Force, June 10, 1994.

2. Ibid.

3. DoD IG Audit Report No. 93-060, “Duplication/Proliferation of Weapons Systems’ Modeling andSimulation Efforts Within DoD”, March 1, 1993.

4. Used with permission of Van B. Norman, “TwentyQuestions for Your Simulation Consultant”, In-dustrial Engineering, May 1993.

5. Mission Critical Computer Resources Manage-ment. Section 8.6.3.6. Ft. Belvoir, VA: DefenseSystems Management College.

6. Van B.Norman, “Twenty Questions for Your Simu-lation Consultant”, Industrial Engineering, May1993.

8-1

88MOVEMENT TOWARDA FUTURE STATE OF

MODELING AND SIMULATIONprovements in the sophistication of infor-mation processing and display technologies.However, today’s technical and managerialuse of M&S in support of systems acquisi-tion is largely characterized by use of thesetools in stand-alone and system-specificmodes.

Scant information is provided on existingresources, institutional barriers and emer-gent policy. An interest in increasing com-munication among functional areas observedthroughout the acquisition community iscoupled with the continuing revolution ininformation processing technologies.

This strongly indicates that the future stateof M&S in acquisition will consist of envi-ronments which seamlessly integrate simu-lations. Integration will occur among simu-lations of similar and different classes (live,constructive and virtual) and across levelsof the M&S hierarchy (engineering, engage-ment, mission/battle and theater/campaign);while providing information that will supportplanning and decision making in all func-

8.1 Introduction

This chapter presents the reader with a vi-sion of what the combined technical andmanagement state-of-the-art appear to beconverging toward. The chapter also pro-vides some examples of DoD programs:where the use of models and simulations ispaying dividends today.

8.2 The Evolution of Modeling andSimulation (M&S) in Acquisition

Technology is rapidly evolving toward, withsome contending that it has already arrivedat, a state permitting the increasingly sophis-ticated implementation of integrated prod-uct and process teams. Acquisition manage-ment will need to evolve in directions thatwill allow managers to take advantage of thisintegration.

A migration is occurring from predominantuse of live and constructive simulations, toincreased interest on the use of virtual simu-lations. This shift is supported by rapid im-

8-2

tional areas and at the requisite level of reso-lution for specific decisions.

8.2.1 Vision for the future state of Model-ing and Simulation

John Hartley writes that technical develop-ments and environmental changes “will con-tinue to move the goal posts”.1 More pow-erful computer-aided design (CAD) work-stations, along with distributed systems shar-ing common object-oriented relational databases, will permit all departments workingon a project access to the same product data.Product data, when used by the manufac-turing department, will be suitable for gen-erating machine tool paths and basic dimen-sions for fixtures or die forms.

Using the same data, logistics engineerswould be able to conduct reliability andmaintainability analyses, and find the loca-tion of key maintenance points such as ac-cess plates, sockets, etc.. The same datawould be available to cost estimators in sucha way that costing could be done easily andfiscal impacts of engineering design changeswould become apparent.

Thus, each department would see the same(and latest) data at all times, from conceptthrough final design. Members of the prod-uct team, contacted about a proposedchange, would be able to run simulations toassess impacts from their department’s per-spectives and offer alternatives, which againother members of the team could evaluatefor impacts.

Hartley also writes about the likelihood thatcontractors will be brought more closely intothe net, relying on the same data base as thecustomer. He recognizes similar concerns tothose voiced by defense contractors; withrespect to proprietary nature of selected data

and the resultant issues that DoD and con-tractors wrestle with in the Continuous Ac-quisition and Life-Cycle Support (CALS)initiative.

A close representation of the future state isprovided by the recent development of toolssuch as the Computer-Aided Three Dimen-sional Interactive Applications (CATIA)software package. The CATIA is a series ofapplication programs that interact to form ahighly integrated design, analysis and manu-facturing system.2 Boeing used CATIA inthe highly publicized design of the 777 air-craft, and is currently using the tool in thedesign of the F-22 aircraft.

8.3 Virtual Prototyping at the US ArmyTank Automotive and Armament Research,Development, and Engineering Center(TARDEC)

Engineers at TARDEC are well on their wayto designing a limited implementation of thevision for the use of M&S in an integratedacquisition environment.

The TARDEC engineers have developedtheir virtual prototype process to lend itselfto continuous user participation.(Soldierswho will eventually use the system in thefield.) This process provides more rapidfeedback to the developer (the government-contractor team responsible for developingthe system). Using advanced computersimulation enables early evaluation of newvehicle concepts without actually buildinga physical vehicle. User/developer agree-ment is maintained throughout the processdepicted in Figure 8-1.

The steps in the process are explained below.

Step 1: Concepts – Solid models of alter-native materiel concepts, such as external

8-3

Figure 8-1. V

irtual Prototyping P

rocess

. . . ADVANCING TECHNOLOGIES . . .. . . STATE OF THE ART COMPONENTS . . .

INPUTS

PERFORMANCEMODELING

VIRTUALMOCK-

UPS

DEVELOPCONCEPTS

STEP 1STEP 2

VIRTUAL FACTORY

COMBATMODELING &WARGAMING

LOGISTICSCONCEPT

EVALUATIONS

STEP 3

REQUIREMENTS

DETAILEDDESIGN

STEP 5

DEVELOPCREW STATION

(SOLDIER IN THE LOOP)

FUNCTIONALMOTION

STEP 6

FORCE EVALUATION

OPERATIONAL

BUILD AND TEST

STEP 7

FIELD SUPPORT

STEP 9

MANUFACTURING

STEP 8

OUTPUTS

EARLY USER/DEVELOPERAGREEMENTS

ARMY OF THE21ST CENTURY

SEAMLESSBATTLEFIELDSIMULATIONGROUNDMANEUVERS

HOT BENCH

WARFIGHTINGCONCEPTS

. . . BATTLE LABS . . . EVOLVING USER NEEDS . . . LOUISIANA MANEUVERS . ..

STEP 4

LOGISTICS CONCEPT

SIGNATURE

WEIGHT/LOSS

LETHALITY

VULNERABILITY

MOBILITY

Virtual Prototyping Process

8-4

Figure 8-2. The Tracked Vehicle Workstation:Expanding Vehicle Simulation’s Role in the Vehicle Development Process

The Tracked Vehicle Workstation:Expanding Vehicle Simulation’s Rolein the Vehicle Development Process

8-5

versus turreted vehicles and wheeled versustracked are developed to meet requirementsfrom the user community. Two-dimensionaldrawings of the conceptual vehicles can beproduced from a CAD station.

Step 2: Performance Modeling – Analyti-cal models applied to the solid model per-mit evaluation of mobility, stealth, survivabil-ity, vulnerability, lethality and vehicle dy-namics. Results of the analyses are reflectedthrough changes to the solid model, optimiz-ing design through an iterative process.Competing conceptual designs can be evalu-ated and trade-offs to satisfy conflicting re-quirements can be worked with the user.

Step 3: Wargame Modeling – Resultingconceptual vehicles are evaluated using ini-tial, then more detailed M&S. Concept ef-fectiveness screening is done using construc-tive models, such as GROUNDWARS andCASTFOREM.

Step 4: Virtual Mockup/Detailed Design –Using concurrent engineering, the selectedconcept goes into detailed design; the solidmodel is refined to incorporate actualcomponents using the Tracked VehicleWorkstation as depicted in Figure 8-2. Anengineer selects components from a CADparts data base, and assembles the conceptvehicle. Simulated test scenario(s) are de-fined using a data base constructed fromprevious instrumented tests and extrapola-tions.

A test of the CAD model is run in the vir-tual environment, with real-time quick-lookanimation available for monitoring. Test re-sults are analyzed and, depending on per-formance, the CAD model of the conceptvehicle is modified. A 3-D virtual prototypeis developed in which the user can actuallyexplore the inside of the vehicle and provide

valuable human-machine interface feedback.

With virtual prototyping, an obvious pro-gression would be to allow the user to fightthe vehicle, by interfacing it with other con-structive and virtual simulations, before con-struction of a crew station envelope (step 6).

Step 5: Virtual Factory – Actual manufac-turing processes are engineered in parallel withthe detailed design (step 4). Machine toolpaths, production line set-up, material flows,assembly, etc. are laid out and simulatedprior to implementation on the factory floor.

Step 6: Crew Station Development – Thevirtual mockup results in the design of acrew station envelope (simulator) to assesshuman-machine interfaces under static anddynamic conditions. The simulator is con-nected via the distributed simulation internetto enable users to evaluate the effect of theconcept’s design on tactics and force effec-tiveness.

Step 7: Build and Test Hardware – Fabri-cation of a test-bed vehicle is done in part,by directly outputting from the CAD stationto numerically-controlled machines. TheTARDEC’s strategy for electronics integra-tion is to use a laboratory hot-bench to re-solve integration issues prior to building theactual electronics hardware.

Step 8: Manufacturing – The virtual fac-tory provides the detailed layout for the manu-facturing process. Any changes in design, re-sulting from test-bed evaluation, are fed backinto the system to ensure compatibility withproduction line set-up, material flow, etc.

Production is based on CAD and computer-aided manufacturing (CAM) machine out-put to machines, providing for more accu-rate and faster parts manufacture. The goal

8-6

is to electronically transfer parts’ design di-rectly to a flexible manufacturing facility forproduction, in the spirit of CALS implemen-tation.

Step 9: Field Support – The virtual proto-type process provides more responsive fieldsupport in terms of failure prediction, analy-sis and retention of the historical engineer-ing data base. Electronic data transfer andflexible manufacturing also provide for ex-panded potential of a smaller industrialbase; possibly reducing logistics spareswarehousing.

Not all aspects of TARDEC’s virtual proto-type process exist today. However, the orga-nization is making significant strides to clos-ing the gaps.

8.4 Getting to the future state of M&S

Many of the enabling technologies, see Fig-

ure 8-3,3 associated with emerging M&S andrelated tools that have the potential to con-tribute to the acquisition process, are com-mercially driven. While this allows DoD toleverage advances made in the commercialmarket for these technologies, others are ofspecific interest to the military. Developmentof these latter technologies will be deter-mined by DoD’s ability to marshal industryinnovation in the direction of its interests.

Each of the enabling technologies can beassigned to a level of the notional hierarchy4

of technologies shown in Figure 8-4.

At the base of the hierarchy are the stan-dards and protocols employed within theenvironment. These include many of thestandards associated with modern softwaresystems, the exchange of product model dataand simulation interoperability standards.

At the next level are the underlying collabo-

Enabling Technologies

Commercially Driven DoD Driven

Data base management systems Manufacturing Process Simulations

Man-machine interfaces Engineering design models

Software engineering tools Manned simulators

Local and wide area networks Stochastic wargaming simulations

High Performance computers Semiautomated forces

Computer image generators Instrumented ranges

Microcomputer systems Instrumentation

Microprocessors Simulation construction tools

Memory Multilevel security

Mass storage DoD protocols

Display devices DoD data bases

Figure 8-3. Enabling Technologies

8-7

rative technologies under development inacademia and industry. These include effortsto establish shared electronic workspaces,develop customized software wrappers thatfacilitate the reuse of legacy code and to cre-ate Groupware -facilitating the work ofgroups separated in space and time (e.g. theElectronic Meeting Room).

Utilities/infrastructure are the third level ofthe hierarchy. These include significant ex-isting capabilities such as moderately highcapacity communications, data managementtools and sophisticated human-machine in-terfaces.

The fourth level of the hierarchy consists of

the applications being developed by the acquisi-tion community. These applications are charac-terized by the type of tool (live, constructive orvirtual simulation) and the functional disciplinesupported by the tool’s employment (e.g. perfor-mance/effectiveness analyses, design/ engineer-ing). Efforts to integrate applications within andacross functional areas are logically supportedby technologies at lower levels of the notionalhierarchy.

The top level of the hierarchy is an integratedacquisition environment. This concept wasintroduced in Chapter 6, and is an infrastruc-ture that supports the information needs ofparticipants in the acquisition process byproviding an integrated set of M&S; related

Figure 8-4. Notional Hierarchy of Technologies

Notional Hierarchy of Technologies

IntegratedAcquisition

Environment

Applications

Utilities/Infrastructure

Underlying Collaborative Technologies

Standards and Protocols

8-8

tools—data base management, CAD/CAM,and computer-aided engineering (CAE);utilities—communications networks and datarepositories; and policies and procedures forimplementation.

8.5 Management Actions

Today’s managers must understand the ca-pabilities and limitations of M&S. Theymust also understand the challenges in-volved in managing and providing resourcesfor the effort. As a start, managers are en-couraged to

• Recognize that M&S are powerful toolsthat can offer significant opportunities to re-duce the acquisition cycle time on a program.

• Recognize that M&S are not a panaceafor all the issues confronting (and confound-ing) DoD’s weapons system acquisition pro-cess. Modeling and simulation must addvalue—not be incorporated just because itis “the in thing.”

• Managers must look down the life cycleand across the functional disciplines to iden-tify what specific functions M&S can per-form. Managers must also garner supportfrom the various functional proponents forthe conduct or support of specific tasks withM&S, using their expertise in the identifi-cation of the tasks.

• Gain user support for the conduct ofspecific user interface tasks with the sup-port of M&S. User advocacy is critical togain support for a viable M&S effort.

• Identify opportunities for achievingsynergy through the integration of modelsand simulations. Here too, integration for“integration’s sake” is counter-productive.Embrace standards and architectures facili-

tating interoperability and compatibilitywhen they add value to the program. Else, itmay inhibit management’s flexibility in thelong run.

• Plan and manage the M&S effort so thatit meshes completely with the rest of the pro-gram; becoming a part of it. Programs whoseacquisition strategies are developed to lever-age the use of models and simulations aremore likely to be the most efficient users. Itis difficult, although not impossible, to re-structure an on-going program to incorpo-rate M&S. The biggest challenge is the al-location of time, money and people. A man-ager must identify and acquire the resourcesrequired to find, develop or modify thesetools as early as possible.

• Designate a simulation manager withinthe program office, to advise you on allmatters pertaining to M&S. As complex andvisible as this area is getting, this individualmust either have or obtain special training.

• Form a Simulation Working Group(SIMWG), chaired by the simulation man-ager with representatives from every elementof the program team.

8.6 Some Application Best Practices

The purpose of this section is to providesome examples of using M&S in the acqui-sition process. The weapon systems dis-cussed in this section are in various phasesof acquisition. However, the perceived ben-efits are impressive enough to give cause toshare them with the reader.

8.6.1 Combat System Engineering andAnalysis Laboratory (CSEAL):

The CSEAL is a hardware/software-in-the-loop (HW/SWIL) simulation facility at New-

8-9

port, Rhode Island, which allows for evalu-ation and rapid prototyping of submarinecombat systems. It includes actual subma-rine computer systems and combat centerdisplays which allow for integration of pro-totype systems, along with human-in-the-loop response, decision making and evalua-tion. It has been used in support of the AN/BSY-2 submarine combat system since1988. This system performs many functionsincluding sensor management and sensordata processing, sonar displays, missionplanning, weapons systems targeting andother command functions such as weatherand ocean condition predictions.

The AN/BSY-2 is a major software devel-

opment project (3.6 million software linesof code) which has incorporated many soundsoftware development practices.5 The use ofthe CSEAL has been an ongoing part of thisprogram. At the start of the program, it wasused in conjunction with Navy operators toexamine what functions the system shouldperform and how the information should bedisplayed to the operator. In 1988 it was thenused to develop design specifications fromthe key system operating functions. As partsof the system are developed, the CSEAL hasbeen used for independent verification andvalidation.

The CSEAL also has allowed data from liveat-sea exercises to be used to stimulate the

Figure 8-5. Combat System Engineering Analysis LaboratoryApplication to System Development

New Ways of

Doing

Business

Future

Technology

Investment

Needed

Future

Platform

Design

Concepts

Industry(R&D)

ARPA

NAVY6.2/6.3

Technology Examples:

- Automation - Open Architecture - Advanced Hardware - Artificial Intelligence - Tactical Algorithms

Advanced Technology Products

Commercial Products (COTS)

Visionary Concepts(Unconstrained Options)

Warfare Requirements• All Sensor Info Mgmt• Platform

• Cost Reduction Goals• Manning Reduction Goals• Reliability/Maintainability Goals

Terms of Reference

Top LevelGuidance

Determine PerformanceNeeded

SystemOptions

Verification

System Level Prototyping Process

Existing Militarized Products/ (GOTS)

National SecurityStrategy

JCS Defense PlanningGuidance ScenariosOPNAV Joint MissionArea Assessments

MissionRequirements

• Integration• Availability• Affordability

Functional Analysis

Combat System Engineering AnalysisLaboratory Application to

System Development

8-10

AN/BSY-2 system during the developmentprocess. Eventually the CSEAL may sup-port design and evaluation of upgrades af-ter the system is fielded.

This application of the CSEAL demonstrateshow the simulation environment can supporta program across the system life cycle. Fig-ure 8-5 illustrates further how such a simu-lation laboratory can support the span of ac-quisition activities; including requirementsdefinition, technology evaluation, conceptdevelopment and system prototyping.6

8.6.2 Simulation support for Test andEvaluation - PM LONGBOW

The LONGBOW is a standoff, helicopterlaunched, anti-tank missile. A derivative ofthe HELLFIRE missile, it uses the same bus,but a millimeter wave seeker in place ofHELLFIRE’s laser-guided seeker. Re-usingdata and models from the HELLFIRE pro-gram and investing in a hardware-in-loop(HWIL) simulation facility, the LONGBOWengineering and manufacturing develop-ment (EMD) flight test program was re-duced (from 50) to 20 missiles.

All test flights were first simulated in theHWIL facility using actual flight hardware.In several instances, potential flight failureswere found and corrected before firing themissile down-range: resulting in time andmoney savings.

When the LONGBOW missile enteredproduction, a Simulation/Test Acceptance Fa-cility (STAF) was built at Redstone Arsenal,outside Huntsville, Alabama. The purpose ofthe STAF was to allow the LONGBOW PMOto effectively execute a production acceptanceprogram and test the seeker without firingmissiles. Components such as warhead, mo-tor and battery are evaluated using the

HELLFIRE fly-to-buy results, since thesecomponents are common to both missiles.

Estimated annual savings from the STAF is$7 million, with a payback period of undera year.

8.6.3 Modeling and Simulation in Specifi-cation Compliance and T&E - AIM-9X

The AIM-9X missile is a major improvementto the existing family of Sidewinder missiles.The AIM-9X will be used with selected air-craft to detect, intercept and destroy a widerange of high performance threat aircraft.This acquisition program is scheduled to be-gin Phase IV/I, Demonstration and Validation(DemVal), in January 1995. Extensive M&Swill be used throughout the life cycle of theAIM-9X, from source selection to post-pro-duction software support activities.

The M&S activities will be used to reducecost, reduce risks and ensure compliancewith top level system performance specifi-cations. These top level system requirementsinclude those spelled out in the ORD in ad-dition to other required/desired performancecapabilities, interface requirements and“ilities” requirements.

Program cost reduction will be achieved byusing M&S to reduce the number of missiletest firings. A philosophy has been adoptedthat assumes M&S will be used to quantifysystem performance, and that hardwaretests, of any kind, will be used to generatedata for simulation validation.

A matrix of requirements versus methodsto validate these requirements has been de-veloped. The requirements were taken directlyfrom the requirements correlation matrix(RCM) contained in the ORD, while meth-ods to validate the requirements were se-

8-11

lected from proposed simulations and hard-ware tests. A team of people familiar withsimulations, testing and AIM-9X require-ments generated this matrix; identifying thesimulations and hardware tests necessary tovalidate each requirement. This process re-duced the number of missile test firings bya factor of two from the original estimategenerated using a conventional philosophythat did not rely primarily on M&S.

Simulations will be used to ensure compli-ance with performance specification require-ments. For example, kinematic performancein the form of launch acceptability regions(LAR) is specified in the system specifica-tion, and will be evaluated using a validatedsix degrees-of-freedom (6-DOF) simulation.Program risk will be reduced by ensuringthat required missile firings will have a veryhigh probability of success (flight failuresput programs in political jeopardy) and bycontinuously monitoring projected systemperformance to quickly identify issues thatcould become cost or schedule problems.

Finally, simulations will be used to converthardware test results against available assetsinto equivalent performance against specifiedthreats (i.e. captive flight tests performanceagainst an F-18 used as a target aircraft mustbe translated into equivalent performanceagainst the specified threat targets).

8.6.4 Modeling and Simulation in SourceSelection, New Start - COMANCHE

Required by the government to use M&S todemonstrate their respective concepts, con-tractor teams for the RAH-66 COMANCHEmade extensive use of virtual prototyping inthe Army’s newest aviation program. Userinterface was established with the contrac-tor design team; with Army test pilots flyingthe virtual prototype. Insights gained from

these flights, along with other contractualinformation, permitted the PM to down-se-lect to a single source for the DemVal phaseof the COMANCHE program.

8.6.5 Modeling and Simulation in SourceSelection, Non-Developmental Item (NDI) -Non-Developmental Airlift Aircraft (NDAA)

The Air Force has been directed to evaluatethe possible procurement of NDAA tocomplement a reduced quantity buy of C-17strategic airlifters.

The data used to feed the Strategic AirliftForce Mix Analysis (SAFMA) comes fromthe Airlift Loading Model (ALM), and theMobility Analysis Support System (MASS).

The ALM, managed by Air Force Studiesand Analysis (AFSAA), is used to determinethe amount of cargo each aircraft can carry.The model, in turn, depends upon data onstandard pallet dimensions, vehicle sizes,weights and loads; typically required to meetmobilization requirements. This model isprovided to potential bidders to familiarizethem with tools to be used in source selection.

The mobility analysis support system(MASS), managed by US TransportationCommand, is a time-phased force and de-ployment data (TPFDD) driven simulationwhich considers the amount of cargo andtroops, as well as factors such as utilizationrates, crew availability and airfield handlingcapacities.

A SAFMA is being used to provide infor-mation to the C-17/NDAA MS IIIB DefenseAcquisition Board (DAB) pertaining totype(s) and number of NDAA to be pro-cured to complement the C-17 fleet. Thiscost and operational effectiveness analysis(COEA)-like analysis provides cost benefit

8-12

analysis between various possible mixes ofC-17, NDAA and the current airlift fleet. Thefollowing information is provided to deci-sion makers:

• Shortfalls in Mobility RequirementsStudy - Bottoms-Up Review Update (MRS-BURU);

• Evaluation of force mixes for additionalaircraft needed to overcome shortfalls;

• Identification of options among forcemixes that meet requirements;

• Life cycle cost analyses of those optionsfound suitable; and

• Recommendations on force structuresmost effectively meeting requirements.

Measure of effectiveness (MOE) are usedby the SAFMA to provide cost informationof each aircraft alternative mix and perfor-mance; such as throughput tons per day, clo-sure of combat units and total force; whichcould then be assessed against higher levelmeasures of outcome (MOO) such as theeffect(s) on a campaign.

The end result is a cost effective, time con-servative, systematic approach to matchingofferors’ solutions to the user’s require-ments. It uses proven methods and validatedmodels, de-emphasizing the empirical ap-proach, and produces the COEA as an inte-gral part of the overall study.

8.6.6 Modeling and Simulation in SourceSelection, Upgrade - Bradley-Stinger

The Bradley-Stinger PMO issued a requestfor proposal (RFP) to three contractors,under a limited competition strategy, to ob-tain design specifications for candidate sys-

tems to fulfill the line-of-sight forward,heavy (LOS-F-H) air defense mission. TheRFP included all requirements for the mis-sion and required delivery of system speci-fications as part of the technical approach;along with relevant cost and schedule pro-posals. The PM’s approach was to have theUS Army Missile Command (MICOM),using the expertise resident within their Re-search, Development and Engineering Cen-ter (RDEC), develop a simulation basedupon each of the three design specifications.The simulations were built to allow soldiersto operate the system in a synthetic envi-ronment, interacting with other virtual andconstructive models.

A crucial aspect of the PM’s managementof this effort was the early involvement ofall team members in a Simulation WorkingGroup (SIMWG). The SIMWG includedrepresentatives from the PMO, contractor(s),user, model developer, tester, V&V agencyand source selection authority.

Various aspects of the performance of eachcontractor’s design in a synthetic environ-ment were included as source selection cri-teria, along with cost and schedule.

8.7 Some Management Best Practices

8.7.1 Naval Undersea Warfare Center:Creating a Common Frame of Reference

The Naval Undersea Warfare Center hasimplemented a series of management practicesthat might be useful for the acquisition com-munity to consider. These include a closeworking relationship with the sponsoringProgram Executive Office (PEO) to developa common frame of reference, CM practicesand archival of simulation runs and data.

In this example, the Center and PEO staff

8-13

have worked together to employ a commonframe of reference providing for uniformapplication of models and simulations acrossall programs supporting that PEO. All systemsrely on the same set of data books which con-tain the latest approved performance char-acteristics of friendly and enemy systemsand environmental data. Consistent opera-tional tactics (physical environment alongwith models validated with live test data)assures senior management that a commonframe of reference is used to evaluate andassess systems. Management does not haveto repeatedly question the methodologies,analysis approach, model validity or datasource. This common framework also resultsin synergy, or better use of resources in sup-port of all programs within that warfare area.

The CM practices for computer simulationsinclude freezing the configuration of mod-ules currently in use; running test cases forproposed changes; and finally documentingchanges, developing manuals and notifyingusers of updates. For the HWIL simulators,a formal CM plan is in place that includes aconfiguration control board. These practicesinsure that current model and simulationconfigurations are properly validated anddocumented for use.

An audit trail for simulations is maintained.This includes archiving the actual model con-figuration and input data so any case can berecreated. This is vital when comparison analy-sis using a consistent baseline is requested at afuture date, or for evaluation of changes.

Accomplishing these practices requires amanagement commitment. In many cases,program offices are focused on near-termobjectives, but they also should consider theongoing maintenance which is necessary toprovide them, as well as senior leadership,with a capability at a future date.

8.7.2Modeling and Simulation Manage-ment - BAT

Originally “Brilliant Anti-Tank,” the BATsub-munition was being developed to killsecond and third echelon enemy movingtank formations. The BAT’s system effec-tiveness was defined using 672 differentconditions. A test program involving statis-tically valid information for each conditionwould have been unaffordable. Faced withthe task of designing a complex and prohibi-tively expensive test program, the PMO re-ceived DAB approval to evaluate the BATperformance and effectiveness using simu-lations subjected to rigorous verification,validation and accreditation (VV&A).

The BAT program uses a family of simula-tions that collectively contribute to systemeffectiveness prediction.

The BAT system validation process, shownin Figure 8-6, is iterative; using test data tovalidate the simulation. Furthermore, theBAT PMO developed its system simulationindependent from the prime contractor, butfrom the same specifications. Each modelis used as a cross-check on the other.

The BAT test methodology, shown in Fig-ure 8-7, uses the model-test-model conceptto ensure that the system model is kept insync with the tactical design. Predictionsfrom pre-test runs of the model are com-pared with post-test analyses to ensure thatany differences can be explained, or formthe basis for a modification.

The BAT PMO formed a SIMWG consist-ing of a wide constituency and a team ap-proach to validation and accreditation. Ac-ceptance of simulation results in support ofoperational testing is simplified throughshared responsibility of the operational test

8-14

accreditation process by the BAT PMO andthe sponsor (Army’s Operational Test andEvaluation Command).

The BAT SIMWG implemented a solid CMapproach involving configuration baselining,change control, evaluation and processing.Release of simulations is controlled throughexecution of a Memorandum of Understand-ing (MOU) with each user; as well as main-tenance of a data base of users, what theyhave (documentation, software, samplecases) and the simulation version.

The BAT program is recognized as a casestudy in the use of teaming to accomplisheffective use and control of performance andeffectiveness M&S.

8.7.3 Simulation Management - TOMAHAWK Cruise Missile

The TOMAHAWK cruise missile programhas had a simulation management board inplace for approximately ten years. Thissimulation management provides manyfunctions which include:

• Maintaining a simulation catalog con-taining a description of each simulation,technical status, limitations, functions andcurrent utilization. Entry into the catalog ispart of the simulation certification processand only those simulations in the catalog maybe used to define performance capability andsystem effectiveness estimates.

Figure 8-6. BAT System Validation: Process

BASELINE

MODELS & SIMULATION

SIMULATIONVALIDATION

TEST

TEST PLAN

BASELINEEVALUATION

SYSTEMREQUIREMENTS

PDR

CDR

FCA

REQUIREMENT COMPLIAMCESIMULATION ACCREDITATION

USERNEEDS

CHANGES

SYSTEM EVALUATIONCONFIDENCE METRIC

SHORTFALLS

BAT System Validation: Process

8-15

• Integrating individual simulation statusand plans within TOMAHAWK programplans.

• Maintaining a set of authorized refer-ence simulations which serve as perfor-mance baselines for the missile and forevaluation of proposed changes. Referencecheck cases are defined with inputs, initialconditions, assumptions and expected out-puts; typically validated by test data.

• A method of documenting/disclosing

the pedigree (certification) status of a simu-lation: a certificate which is approved by thechairman of the simulation managementboard. This certificate, shown in Figure 8-8, is used when publishing or presentingperformance analysis results.

The value of this management process iscoordination of simulation plans across theentire program, providing standard referencesimulations and check cases and centralmanagement of certified TOMAHAWKmodels and data bases for use; eliminating

Figure 8-7. BAT Test Methodology

UPDATE SIMULATIONTO TACTICAL DESIGN

IDENTIFY CRITICALPARAMETERSAND INPUTS

PRETESTPREDICTIONS

MODIFY IFNEEDED

COMPARE &ANALYZE

POST TESTANAYLSIS

TEST

MODEL - TEST- MODEL

BAT Test Methodology

8-16

redundancy of multiple simulation develop-ment efforts and potential errors.7

8.8 Summary

The instances discussed in the precedingsections are only representative of the manysuccessful uses of M&S in systems acqui-sition. The cases mentioned, however, rein-

force a significant aspect of the use of M&Sin acquisition. Careful early planning andinvestment for the use of M&S will pay divi-dends through cost avoidance. While costavoidance is not as easy to quantify orproject, as cost savings, the results for aninstitution facing a downward trend in bud-gets, are just as tangible.

Figure 8-8. TOMAHAWK Simulation Management Board Certificate

TOMAHAWK Simulation Mangement Board Certificate

To: (Responsible Organization)

From: TOMAHAWK Simulation Management Board

Subject: (Full/Provisional/Limited) Certification of TOMAHAWK {Simulation Name}

Certified to Represent for use in:

Responsible Organization:

This (Full/Provisional) Certification is for: The following material support certification:

Version Dated Certification Plan Users ManualCertification Report Benchmark

Provisional Pending: Configuration CheckcasesControl Plan Reference

Based on the following major activities: CheckcasesValidation against Flight Test Results Current Catalog SourceValidation against Entry Code/ListingOther

Previous Certification: Version Dated

Certification Recommended: Certification Approved:

SCP Chairman:

SCP Co-Chair:

(Define domain of certification applicability)

Chairman, Simulation Management Board, Date

8-17

ENDNOTES

1. Hartley, J. R. (1992) Concurrent Engineering:Shortening Lead Times, Raising Quality, and Low-ering Costs. Cambridge, MA.

2. “CATIA Data Management and CATIA DataManagement Access,” p.1.

3. Garcia, A. B., Gocke, R. P. Jr., Johnson, N. P. Jr.(1994) Virtual Prototyping: Concept to Produc-tion,” Report of the 1992-1993 Military ResearchFellows. (p. 36). Ft. Belvoir, VA: Defense Sys-tems Management College.

4. Final Report of the Acquisition Task Force onModeling and Simulation, June 1994.

5. “BSY-2 Ada Lessons Learned”, F.GregoryFarnham, 1993 Washington Ada Symposium,ACM 0-89791-609-3/93-0006-95.

6. Naval Undersea Warfare Center, Newport, RI.

7. “TOMAHAWK Simulation Management, aWorking Example”, Van D. Underwood, Chair-man, TOMAHAWK Simulation ManagementBoard, Johns Hopkins University Applied Phys-ics Laboratory, San Diego, California 92124.

EPILOGUE

The use of modeling and simulation (M&S) in defense acquisition is certain to grow dra-matically in coming years. Advances in simulated battlefield environments will supportmodels possessing the resolution and fidelity to address the complex issues at hand. Theextension of innovative implementations of advanced concepts, such as Distributed Inter-active Simulation (DIS), are creating exciting, cost effective means for the conduct ofwargaming, training, and requirements and concept development. The use of M&S toolsis expanding within each functional domain of weapon systems acquisition. Integration oftools used among the functional domains is beginning as a natural follow-on.

The primary intent of this document is stated in the title. However, the authors feel com-pelled to include a vision of how things may be in the not-too-distant future, as well as toreiterate some advice to Program Managers (PMs) and policy makers.

Economic, political, national, and international forces are responsible for much of therecent reorganization and process re-engineering within the Department of Defense (DoD).During the conduct of this research, we witnessed new ideas germinating and productstaking shape in response to these changes. These ideas are, largely, the result of a recogni-tion that DoD must use its resources with greater efficiency. An inevitable result of theseinnovations will be a better, more focused and timely allocation of resources.

A logical next step is an intense focus on mission areas and their requirements from a trulyjoint perspective. For example, if all services participate in the counter-air mission area,why not approach the Mission Area Assessment (MAA) and Mission Needs Analysis (MNA)from an all-service contribution standpoint? The potential for M&S to play a major role incontributing to such analyses, is significant.

Individual PMs must understand what M&S tools can do for their programs, as well asrecognize their limitations. They (M&S tools) offer significant benefits, but are not apanacea. The PMs must recognize that the management of M&S efforts require resourceinvestments, and that the resulting benefits are usually in cost avoidance rather than costsavings. Each PM must also recognize that any program will normally require the supportof several models and simulations, and that additional benefits may accrue from the inte-gration of some of these tools.

Policy makers, on the other hand, must recognize the PM’s lot. Efforts to institutionalizethe use of M&S in weapon systems acquisition must be based on an assessment of thevalue added. Guidance should, and must, be provided. However, it must be judicious inits mandate. PMs must be provided the latitude to do what is smart for their individualprograms.

Policy that adds baggage without creating value will stifle the very innovation so crucial toachieving the synergy that the application of M&S tools is capable of providing.

A-1

APPENDIX A

DoD SOURCES OF INFORMATION FOR MODELING

AND SIMULATION IN WEAPON SYSTEMS ACQUISITION

This appendix contains a listing of points of contact for modeling and simulation withinthe Office of the Secretary of Defense (OSD), Department of Defense (DoD) Agenciesand the Joint Staff.

Defense Modeling and Simulation Office (DMSO)Phone:(703)998-0660

Advanced Research Projects Agency (ARPA)Advanced Systems Technology Office (ASTO)

Phone:(703)696-2309DSN: 226-2309

Ballistic Missile Defense Office (BMDO)Modeling and Simulation Directorate (AQM)

Phone:(703)693-1594DSN: 223-1594

Defense Intelligence Agency (DIA)Acquisition/Plans Intelligence Support Division (PAN-2)

Phone:(202)373-3101DSN: 243-3101

Defense Information Systems Agency (DISA)Center for Standards

C3I Support Division (TBC)Phone:(703)487-3538

DSN: 364-3538

Information Processing Standards Division (TBE)Phone:(703)487-3552

DSN: 364-3552

Defense Logistics Agency (DLA)Warfighting and Integration Division (CAILW)

Phone:(703)274-6715DSN: 284-6715

A-2

Defense Mapping Agency (DMA)Plans and Requirements Directorate

Advanced Weapon & Systems Division (PRW)Phone: (703)285-9326

DSN: 356-9326

Defense Nuclear Agency (DNA)Operational Support(OPNA)

Phone: (703)325-7177DSN: 221-7177

National Defense University (NDU)War Gaming and Simulation Center (NDU-NSS-WGSC)

Phone: (202)475-1251DSN: 335-1251

THE JOINT STAFF

Director for Logistics (J4)Mobility Division (J4/MOB)

Phone: (703)697-6110DSN: 227-6110

Director for Command, Control, Communication, and Computer Systems (J6)Resource, Planning, and Evaluation Division (J6E)

Phone: (703)697-8590DSN: 227-8590

C4 Architecture and Integration Division (J6I)Phone: (703)614-8787

DSN: 224-8787

Director for Operational Plans and Interoperability (J7)Joint Simulation Division (J7/JSID)

PolicyPhone: (703)695-3047

DSN: 225-3047

Joint Test/EvaluationPhone: (703)693-3418

DSN: 223-3418

A-3

Joint Warfighting Center (JFWC-CC)Phone: (904)884-7720

DSN: 579-7720

Director for Force Structure, Resources, and Assessment (J8)Automated Systems Division (J8/ASD)

ApplicationsPhone: (703)697-7824

DSN: 227-7824

Data BasesPhone: (703)697-8899

DSN: 227-8899

H/W ConfigurationPhone: (703)693-4614

DSN: 223-4614

Resources and Joint Planning Division (J8/RJPD)

ContractsPhone: (703)614-7881

DSN: 224-7881

Contracted Advisory and Assistance Services (CAAS)Phone: (703)693-4608

DSN: 223-4608

Modern Aids to Program Planning (MAPP)Phone: (703)695-1763

DSN: 225-1763

System Programs Evaluation Division (J8/SPED)Phone: (703)697-6299

DSN: 227-6299

CATALOGS:

Catalog of Wargaming and Military Simulation Models. (1992). (12th ed.) Washington,DC: Force Structure, Resource, and Assessment Directorate, Technical Support andOperations Division (J-8/TSOD), The Joint Staff.

B-1

APPENDIX B

DEPARTMENT OF THE ARMY SOURCES OF INFORMATIONFOR MODELING AND SIMULATION

IN WEAPON SYSTEMS ACQUISITION

This appendix is organized primarily to assist Army acquisition managers in their searchfor information regarding modeling and simulation (M&S) activities of various Army or-ganizations. It provides phone numbers, office symbols and addresses to assist users inmaking initial contact. This appendix also provides information on Model and Simula-tions: Army Integrated Catalog (MOSAIC). Figure B-1 depicts the relationship of organi-zations active in the Army’s development, management and use of M&S in acquisition.

Figure B-1: Army Organizations Active in M&S for Acquisition

POLICY AND GUIDANCE

Army Model and Simulation Management Office (AMSMO): The AMSMO promul-gates Army M&S policy; publishes guidance and administers the Army Model Improve-ment Program (AMIP) and Simulation Technology Program (SIMTECH); develops andpublishes the Army Model and Simulation Master Plan and implementing procedures suchas the verification, validation and accreditation (VV&A) of Army models andsimulations.(The AMSMO will be the focal point for establishing and maintaining themedia by which to identify and share information on agencies, organizations and activities

Army Organizations Active in M&S for Acquisition

DCSINTDCSPER

DCSLOG

Secretaryof the Army

ASA(FM) ASA(RDA) ASA(RDA)

TRADOC COE SSDC ISA,C

TEC WES

AWC

CAA OPTEC

CSA

AMSMO

MICOMATCOM CECOM TACOM TECOM IOC

RDEC RDEC RDEC RDEC CSTAAMSAA

STRICOM

NSC TRACBATTLELABS

CEAC

PM

PEO DSCOPS

B-2

in the Army community who use or develop M&S.) The point of contact (POC) is:

U.S. AMSMO1725 Jefferson Davis Highway, Suite 808

Arlington, VA 22202Phone: (703)607-3375

POCS FOR ANALYSIS MODELS (BY FORCE LEVEL)

System Performance Models, Item level: Focusing on a single weapon system or pieceof equipment. Examples include: Air Defense Air-to-Ground Engagement (ADAGE), Ar-tillery Force Simulation Model (AFSM), Simplified Artillery, Reliability Growth model(e.g. SESAME), Projectile Effectiveness Model (e.g. ARTQUIK), TANKWARS, NATOReference Mobility Model. The POC is:

U.S. Army Materiel Systems Analysis Activity (AMSAA)Special Studies and Activities Office

ATTN:AMXSY-DAAberdeen Proving Ground, MD 21005-5071

Phone: (410)278-6576DSN: 298-6576

Brigade and task force level, and below: Focusing on combined arms forces and singlefunctional elements; they are represented as an integral part of combined arms and ser-vices activities. Examples include: Combined Arms and Support Task Force Evaluation(CASTFOREM), JANUS(T), ELANT, American Canadian Australian British Urban Game(ACABUG). The POC is:

Commander, U.S. Army TRADOC Analysis Command (TRAC)Brigade/Battalion Modeling and Analysis Support Center

ATTN:ATRC-WEWhite Sands Missile Range, NM 88002-5502

Phone: (505)678-1012DSN: 258-1012

Corps and Division Level: Focusing on single and multi-division levels of operation withor without a supervising corps headquarters. Examples include: EAGLE, Corps BattleAnalyzer (CORBAN), Vector-in-Commander (VIC), JIFFY. The POC is:

Commander, U.S. Army TRADOC Analysis Command (TRAC)Corps/Division Modeling and Analysis Operations Analysis Directorate Support Center

ATTN:ATRC-OACFt. Leavenworth, KS 66027

Phone: (913)684-2276DSN: 552-2276

B-3

Theater level and above: Focusing on all force levels at echelons above corps; includesmulti-corps, regional and global models and simulations. Examples include: Force Evalu-ation Model (FORCEM), Concepts Evaluation Model V (CEM V), Force Analysis Simu-lation of Theater Administrative and Logistic Support (FASTALS), Transportation Model(TRANSMO). The POC is:

Commander, U.S. Army Concepts Analysis Agency (CAA)Research and Analysis Support Directorate

ATTN:CSCA-RS8120 Woodmont AveBethesda, MD 20814Phone: (301)295-1692

DSN: 295-1692

POCs by M&S Functional Application

Engineering Models: Information focusing on models and simulations which augmentdesign, engineering and testing in various stages of the materiel acquisition process. Mod-els and simulations are used in investigating mechanical, electrical and physical phenom-ena associated with the functioning of an item or system. Examples include hardware-in-the-loop (HWIL), six degree-of-freedom (6-DOF), physics-of-failure simulations, computer-aided design (CAD), computer-aided manufacturing (CAM) and phenomenology models. ThePOCs at the various Research, Development and Engineering Centers (RDEC) are:

Commander, U.S. Army Communications-Electronics Command (CECOM)Research, Development and Engineering Center (CRDEC)

ATTN:AMSEL-RD-ST-CE-MFt. Monmouth, NJ 07703-5203

Phone: (908)544-4708DSN: 992-4708

Commander, U.S. Army Close Combat Armament CenterLight Armament DivisionATTN:SMCAR-CCL-E

Picatinny Arsenal, NJ 07806-5000Phone: (201)724-6054

DSN: 880-6054

Commander, U.S. Army Fire Support Armament CenterFire Control Division

ATTN:SMCAR-FSF-BDPicatinny Arsenal, NJ 07806-5000

Phone: (201)724-7920DSN: 880-6054

B-4

Commander, U.S. Army Missile Command (MICOM)ATTN:AMSMI-RD-SS

Redstone Arsenal, AL 35898-5252Phone: (205)876-4271

DSN: 746-4271

Commander, U.S. Army Tank-Automotive Command (TACOM)ATTN:AMSTA-RYA

Warren, MI 48397-5000Phone: (313)574-8633

DSN: 786-8683

Commander, U.S. Army Aviation and Troop Command (ATCOM)ATTN:AMSAT-R-B

4300 Goodfellow Blvd.St. Louis, MO 36120-1798

Phone: (314)263-1333DSN: 693-1333

Commander, U.S. Army Edgewood RD&E CenterATTN:SCBRD-RTM

Aberdeen Proving Ground, MD 21010-5423Phone: (410)671-1774

DSN: 584-1774

Commander, U.S. Army Natick RD&E CenterATTN:SATNC-AAM

Kansas StreetNatick, MA 01760-5015Phone:(508)651-4881

DSN: 256-4881

Director, U.S. Army Research Laboratory (ARL)2800 Powder Mill Road

Adelphi, MD 20783-1145Phone:(301)394-4650

DSN: 290-4650

Cost Models: The U.S. Army Cost and Economic Analysis Center (CEAC) develops andscreens models that pertain to financial management (e.g. cost analysis); designs and imple-ments the cost methodology for program office estimates and component cost analysisthat shape the Army cost position. The CEAC develops data bases, cost models and costestimating relationships for major materiel systems; tracks operating and support costs;captures demand data and scales it to activity or use; uses M&S to cost real time sustain-ment cost during the battlefield simulations; develops and screens M&S pertaining to man-

B-5

power costs; and tracks the personnel costs to operate the force. The POC is:

Director, U.S. Army Cost and Economic Analysis Center (CEAC)ATTN:SFFM-CA-CR

5611 Columbia Pike, Room 436Falls Church, VA 22041-5050

Phone: (703)756-8732DSN: 289-8732

Logistics Models: The U.S. Army Logistics Support Agency (LOGSA) develops andmanages techniques/models for performing logistics support analysis (LSA) and logisticssupport analysis reports (LSAR); is responsible for policy, techniques/ models and dataelements for level of repair analysis (LORA); is proponent, developer and configurationmanager of three of the currently designated Army standard LORA techniques/models;performs LORA on designated weapon systems; and provides consultation and guidanceon performing LORA for all Army weapon systems. The POCs are:

Executive Director, AMC LOGSAATTN:(See Below)

Redstone Arsenal, AL 35878-7466Phone Hotline: 1-800-553-0764/0769

For LORA: ATTN:AMXLS-ALFOR LSA/LSAR: ATTN:AMXLS-ALD

Manpower Models: The Office of the Deputy Chief of Staff for Personnel (DCSPER),Director for Manpower and Personnel Integration (MANPRINT) has oversight for devel-opment of M&S for soldier survivability and manpower, personnel and training integra-tion. The POC is:

HQDA, ODCSPER, Director for MANPRINTATTN: DAPE-MR

The PentagonWashington, DC 20310-0300

Phone:(703)697-9213DSN: 227-9213

Manufacturing Models: Since these models are usually weapon system, material andtechnology specific, a central POC is:

Commander, U.S. Army Materiel Command, Manufacturing Sci-ence and Technology Office

ATTN:AMCRD-IEM5001 Eisenhower Avenue

Alexandria, VA 22333-0001Phone: (703)274-9437

DSN: 284-9437

B-6

TESTING:

Developmental Testing: The various RDECs possess inherent capabilities to conduct tech-nical testing at various stages of the acquisition process. To support formal developmentaltesting, the U.S. Army Test and Evaluation Command (TECOM) develops and supportsM&S for vibration and environmental testing; uses M&S for meteorology studies for testplanning and to drive simulation targets in moving target simulators; and is developing avirtual test range in support of virtual prototyping and other simulation exercises. ThePOC is:

Commander, TECOM, Technology Development DivisionATTN:AMSTE-CT-T

Aberdeen Proving Ground, MD 21005-5055Phone:(410)278-1479

DSN: 298-1479

Operational Testing: The U.S. Army Operational Test and Evaluation Command (OPTEC)is the Army’s lead agent for operational test and evaluation; using M&S throughout thetesting process, OPTEC applies cost effective techniques to situations of limited testabil-ity; and incorporates the notion of distributed interactive simulation (DIS) through the useof constructive and virtual simulations in the live simulations (operational tests) that itconducts. The POC is:

Commander, U.S. Army OPTEC, Office of Policy and MethodologyATTN:CSTE-MP

Park Center IV, 4501 Ford AvenueAlexandria, VA 22302-1458Phone:(703)756-1685/1688

DSN: 289-1685/1688

Training: The Simulation, Training and Instrumentation Command (STRICOM) man-ages the development and acquisition of simulations to support training, exercises, andmilitary operations. The various Program Managers are: PM – Training Devices (PM-TRADE); PM – Instrumentation, Targets and Threat Simulators (PM-ITTS); PM – Com-bined Arms Tactical Trainer (PM-CATT); and PM – Distributed Interactive Simulation(PM-DIS). Central POC is:

Commander, U.S. Army STRICOM, Director for ManagementATTN:AMSTI-M

12350 Research ParkwayOrlando, FL 32826-3276

Phone:(407)380-8234DSN: 960-8234

B-7

M&S DATA SOURCES

Weapon Systems Performance Data: Item level weapon systems performance data forU.S. and threat systems, and characteristics data for U.S. systems. Data focusing on thelowest level system, such as a gun with its crew or a tank with its crew (includes reliabilityand supportability). The POC is:



Phone:(410)278-6576DSN: 298-6576

Threat Systems Performance Data: Item level weapon systems operational and charac-teristic data for threat systems focusing on lowest level threat systems, such as an airdefense gun or tank. The POC is:

HQDA, Office, Deputy Chief of Staff for Intelligence (ODCSINT)ATTN:DAMI-FIT

The PentagonWashington, DC 20310-1088

Phone:(703)614-8121DSN: 224-8121

Weather Data: The POC is:


Adelphi, MD 20783-1145Phone:(301)394-4650

DSN: 290-4650

Terrain Data: The POC is:

Director, U.S. Army Topographic Engineering Center (TEC)7701 Telegraph Road

Alexandria, VA 22310-3864Phone:(703)355-3176

DSN: 345-3176

B-8

Smoke Data: The POC is:


Adelphi, MD 20783-1145Phone:(301)394-4650

DSN: 290-4650

OTHER ORGANIZATIONS WITH SPECIFICM&S RESPONSIBILITIES AND CAPABILITIES

U.S. Army Materiel Systems Analysis Activity (AMSAA): The AMSAA uses M&S toprovide U.S. and threat systems performance data for use in cost and operational effective-ness analyses (COEA), in Army studies, and in support of the acquisition of systems;helps accredit and provides certified systems performance data characteristics and data tothe Army community; provides VV&A support to U.S. Army Materiel Command (AMC)and other agencies; promulgates VV&A and data certification policy throughout AMC;and, in conjunction with TRADOC, standardizes data and algorithms within the Armycommunity. The AMSAA maintains configuration control on M&S for item level perfor-mance, one-on-one system performance, few-on-few and many-on-many combat and largewar gaming simulation. The POC is:



Phone:(410)278-6576DSN: 298-6576

U.S. Army Industrial Operations Command (IOC): Formerly the U.S. Army Arma-ment, Munitions and Chemical Command. The IOC’s Cost and Systems Analysis Direc-torate provides and uses M&S to assess manufacturing and flexible computer integratedmanufacturing and repair operations at the macro and micro level within the arsenals, thedepots and the ammunition plants. The POC is:

Commander, Industrial Operations CommandATTN:AMSMC-AN

Rock Island, IL 61299-6000Phone:(309)782-5262

DSN: 793-5262

National Simulation Center (NSC): The TRADOC proponent for all battle commandtraining simulations. The NSC sponsors, designs, maintains and provides configurationmanagement (CM) of all Army battle command training M&S; and coordinates, manages,

B-9

and executes training simulation V&V. The POC is:

Commander, Combined Arms CenterATTN:ATZL-NSC

Ft. Leavenworth, KS 66027-7305Phone:(913)684-8101

DSN: 552-8101

TRADOC Battle Labs: The Battle Labs use models and simulations, as well as othertools, to improve battlefield capabilities by deriving insights across doctrine, training, leaderdevelopment, organization design and materiel (DTLOM). Central POC is:

Commander, U.S. Army TRADOCATTN:ATCD-B

Ft. Monroe, VA 23651-5000Phone:(804)727-4283/4284

DSN: 680-4283/4284

U.S. Army Space and Strategic Defense Command (USASSDC): The USASSDC ap-plies M&S in conducting research and development (R&D) of missile defense simula-tions. The POC is:

Commander, Space and Strategic Defense CommandATTN:CSSD-CR

PO Box 1500Huntsville, AL 35807-3801

Phone:(205)955-1354DSN: 654-1354

U.S. Army War College (AWC): The AWC’s Center for Strategic Leadership uses M&Sin support of students and curriculum. The Center uses M&S for war gaming, simulationsupport, and studies; analysis for other Army and DoD commands and agencies at theoperational and strategic levels of war; and provides limited development of M&S educa-tion decision tools. The POC is:

Commandant, U.S. Army War CollegeATTN:AWC-AW

Carlisle Barracks, PA 17013-5050Phone:(717)245-4281

DSN: 242-4281

B-10

THE MODELS AND SIMULATIONS: ARMY INTEGRATED CATALOG (MOSAIC)

The MOSAIC provides potential model developers and users the opportunity to peruse thearray of existing M&S and query the hypertext system for all information of interest tothem in their proposed application. The official registry of M&S, MOSAIC contains allactive and developmental M&S.

In AR 5-11, Army M&S proponents are directed to:

• Enter information about their M&S into the catalog

• Keep that information current by providing updates at least every two years

To become and remain a valid entry in MOSAIC, a model or simulation must:

• Be computerized

The POC for MOSAIC is:

Chief, Army Model and Simulation Management Office (AMSMO)ATTN:SFUS-MIS (MOSAIC System Administrator)

1725 Jefferson Davis Highway, Suite 808Arlington, VA 22202Phone:(703)607-3383

DSN: 327-3383

ARMY POLICY AND PROCEDURE REFERENCES

Army Regulation 5-11, Army Model and Simulation Management Program (AMSMP):Prescribes policy for Army M&S management and formalizes the Army’s program formanagement of models and simulations.

Department of The Army Pamphlet 5-11, Verification, Validation, and Accreditationof Army Models and Simulations: Provides procedures to assist model developers, pro-ponents and sponsors to conform to the policies in AR 5-11.

Army Model and Simulation Master Plan: Promotes the adoption of standards and com-mon tools and processes in building and populating models and simulations, for use in allapplications throughout the Army.

OPTEC Handbook 73-21: Written primarily for OPTEC’s evaluators, test officers andanalysts. It assists OPTEC personnel in coordinating more proficiently with external par-ties with respect to the use of M&S in support of OT&E. Likewise, it allows managers tobe pro-active in their M&S planning efforts.

C-1

APPENDIX CDEPARTMENT OF THE NAVY SOURCES

OF INFORMATION FOR MODELING AND SIMULATIONIN WEAPON SYSTEMS ACQUISITION

This appendix provides information on various organizations within the Department ofthe Navy (DON) that are involved with modeling and simulation (M&S) activities. It pro-vides phone number, office codes and addresses to assist users in making initial contact.

Assistant Secretary of the Navy (Research, Development and Acquisition)

The ASN(RDA) has overall responsibility for Navy acquisition. It provides guidance ondevelopment and use of models in the acquisition process. ASN(RDA) is a member of theDON M&S Oversight Council. The point of contact (POC) is:

Assistant Secretary of the Navy (Research, Development and Acquisition)The Pentagon

Washington, DC 20350-1000Phone: (703) 695-2843

DSN: 225-2843

Chief of Naval Operations, Space and Electronic Warfare Directorate (N6)

The Navy’s Policy and Coordination Office for Modeling & Simulation is expected to beestablished in this directorate during the summer of 1994. The specific office had not beendetermined as of May 1994. The POC is:

Chief of Naval Operations (N6)The Pentagon


DSN: 225-4379

D-1

APPENDIX DMARINE CORPS SOURCES OF INFORMATION

FOR MODELING AND SIMULATIONIN WEAPON SYSTEMS ACQUISITION

The information contained in this appendix draws heavily upon the list of organizationscontained within the Marine Corps Modeling and Simulation Master Plan (Draft, dated 16March 1994.

MARINE CORPS MODELING AND SIMULATIONMANAGEMENT OFFICE (MCMSMO)

The MCMSMO is the focal point for modeling and simulation (M&S) in the Marine Corps.It serves as the management and coordinating activity for all M&S related activities andprovides limited technical support to M&S users. The MCMSMO supports developmentof and manages the Marine Corps Modeling and Simulation Master Plan and long rangeinvestment strategies. The point of contact (POC) is:

MCCDC, T&E Div, MCMSMO (Code: C46MS)2006 Hawkins Ave

Quantico, VA 22134Phone: (703) 640-2520

Training and Education (T&E) Division, MCCDC: The T&E division is responsiblefor the development and implementation of policy and programs for training and educa-tion of all regular and reserve Marine Corps personnel and units. They function, amongother things, as the proponent for non-standard training devices and simulators, validatetraining device requirements, and develop and refine the Marine Corps Ground RangeProgram. This division also functions as the proponent for MTWS and provides construc-tive modeling support to the Fleet Marine Force (FMF), Marine Corps University (MCU)and other agencies. The POC is:

MCCDC, T&E Div (Code: C46)1019 Elliot Rd.


Marine Corps Operational Test and Evaluation Activity (MCOTEA): The MCOTEAis the Marine Corps principal operational test organization. The MCOTEA is an activemember of the MCMSWG and provides the necessary test perspective to Marine Corps

D-2

M&S policies, procedures and guidelines. The POC is:

MCCDC, MCOTEA3035 Barnett Ave


Marine Corps Tactical Systems Support Activity (MCTSSA), Marine Corps SystemsCommand (MARCORSYSCOM): This activity supports tactical software developmentby conducting testing and providing maintenance through the software life cycle. TheMCTSSA’s integration of M&S in its testing process is a critical component in accom-plishing many of the Marine Corps M&S end states. The POC is:

MARCORSYSCOMMCTSSA

2033 Barnett Ave, Suite 315Quantico, VA 22134-5010

Phone: (703) 640-3792DSN: 278-2411

Amphibious Warfare Technology Directorate (AWT), MARCORSYSCOM: Researchand demonstration of future technologies and their applicability to the Corps is the respon-sibility of AWT. A significant element in AWT activities is M&S related technology. TheAWT also provides technology insights which support MCMSWG activities. The POC is:

MARCORSYSCOMAWT

2033 Barnett Ave, Suite 315Quantico, VA 22134-5010

Phone: (703) 640-3792DSN: 278-2411

Wargaming and Combat Simulation Division (WCSD): The WCSD provides wargamingand assessment support for HQMC, the operating forces (active and reserve), and the sup-porting establishment including the Marine Corps Combat Development Command(MCCDC). The POC is:

MCCDC, WCSD, (Code: C471)2076 South St.


D-3

MAGTF/Expeditionary Training Center (M/ETC): The M/ETC mission is to strengthenand improve the coordination and integration of training opportunities among MCAGCC,MAWTS, MWTC, LFTCs and other Service Training Centers in a Naval ExpeditionaryWarfare context. The M/ETC will compliment training through the use of models andsimulators that support mission preview and rehearsal, and battle staff training. Addition-ally, M/ETC will use the Global Grid to participate in Joint and CINC level exercises andincrease training opportunities for geographically dispersed active and reserve MAGTFelements. The POC is:

MCCDC, MSTP (C466)2024 Barnett Ave, Suite 202


DSN: 278-3279

Marine Corps Computer and Telecommunications Activity (MCCTA), Headquar-ters Marine Corps (HQMC) (C4I2): The MCCTA is a major activity pursing and imple-menting Marine Corps requirements relating to establishing the Global Grid. Additionally,MCCTA participates in developing M&S verification, validation and accreditation (VV&A)and configuration management policies, procedures and guidelines as a member of theMarine Corps Modeling and Simulation Working Group (MCMSWG). The POC is:

CMC, HQMCMCCTA

2 Navy AnnexWashington, DC 20380-1775Phone: (703) 614-2443/2604

DSN: 224-2443/2604

ADS Demonstration Sites: Two ADS demonstration sites, the MCAGCC ADS Demon-stration Site located at Twenty-nine Palms and the proposed DMSC at Quantico providean operational setting to demonstrate and test M&S components. The POC is:

MCCDC, T&E Div, MCMSMO (Code: C46MS)2006 Hawkins Ave.Quantico, VA 22134

Phone: (703) 640-2498/2520DSN: 278-2498

D-4

Program Support Directorate (PS), MARCORSYSCOM: The PS directorate is a ma-trix organization supporting the program manager (PM) in engineering, logistics, techni-cal manual documentation, operations analysis and life cycle cost analysis. The PS hasbeen designated the focal point for M&S within MARCORSYSCOM. The POC is:

MARCORSYSCOMProgram Support Directorate (Code: PSA)

2033 Barnett Ave., Suite 315Quantico, VA 22134

Phone: (703) 640-4451DSN: 278-4451

C-2

Deputy Chief of Naval Operations (Resources, Warfare Requirements and Assess-ment), Assessment Division, Assessments and Affordability Branch, Modeling andAnalysis Section (N812D)

This office is the focal point for Navy modeling and simulation. The Navy M&S functionis expected to transition to N6 during the summer of 1994. The POC is:

Chief of Naval Operations (N812D)The Pentagon


DSN: 227-5242

Space and Naval Warfare Systems Command (SPAWAR 31)

SPAWAR 31 leads the Navy’s M&S Technical Support Group. The POC is:

Commander, Space and Naval Warfare Systems Command, (SPAWAR 31)2451 Crystal Drive

Arlington, VA 22245-5200Phone: (703) 602-4540

DSN: 332-4540

USERS AND/OR DEVELOPERS

The following are users and/or developers of models and simulations. There may be anumber of groups within each organization which deal with M&S. However, a single pointof contact (code) is listed for each organization.

Naval Air Systems Command POC is:

Commander, Naval Air Systems Command(PMA-205)

1421 Jefferson Davis HighwayArlington, VA 22243

Phone: (703) 604-2245, ext.3046DSN: 664-2245 ext. 3046

C-3

Naval Air Warfare Center/Aircraft Division (Warminster) POC is:

Commander, Naval Air Warfare Center/Aircraft Division (Code 30B)PO. Box 5152

Warminster, PA 18974Phone: (215) 441-1534

DSN: 441-1534

Naval Air Warfare Center/Aircraft Division (Patuxent River) POC is:

Naval Air Warfare Center/Aircraft DivisionFlight Test and Engineering Group (Code SY02C)

Patuxent River, MD 20670Phone: (301) 826-6009

DSN: 326-6009

Naval Air Warfare Center/Training Systems Division POC is:

Commander, Naval Air Warfare Center/Training Systems Division(Code PDB8)

12350 Research ParkwayOrlando, FL 32826

Phone: (407) 380-8367DSN: 960-8367

Naval Air Warfare Center/Weapons Division POC is:

Commander, Naval Air Warfare Center/Weapons Division(Code C0243)

China Lake, CA 93555Phone: (619) 939-2353

DSN: 437-2353

Naval Sea Systems Command POC is:

Commander, Naval Sea Systems Command(SEA-91W1, Combat Systems Training)

2351 Jefferson Davis HighwayArlington, VA 22242-5160

Phone: (703) 602-1782DSN: 332-1782

C-4

Naval Surface Warfare Center/Headquarters POC is:

Commander, Naval Surface Warfare Center(NSWC-04M)

2531 Jefferson Davis HighwayArlington, VA 22242-5160

Phone: (703) 602-0632DSN: 332-0632

Naval Surface Warfare Center/Carderock Division POC is:

Naval Surface Warfare Center/Carderock Division(Code 21)

Bethesda, MD 20084-5000Phone: (301) 227-1013

DSN: 287-1013

Naval Surface Warfare Center/Crane Division POC is:

Naval Surface Warfare Center/Crane Division(Code 604), Bldg. 2045

300 Highway 361Crane, IN 47522-5001Phone: (812) 854-3966

DSN: 482-3966

Naval Surface Warfare Center/Dahlgren Division POC is:

Naval Surface Warfare Center/Dahlgren Division(Code A08)

Dahlgren, VA 22448-5000Phone: (703) 663-7369

DSN: 249-7369

C-5

Naval Surface Warfare Center/Dahlgren Division/Coastal Systems Station POC is:

Commanding Officer, Naval Surface Warfare Center/Dahlgren Division/Coastal Systems Station

(Code 04)Panama City, FL 32407Phone: (904) 234-4200

DSN: 436-4200

Naval Surface Warfare Center/Indian Head Division POC is:

Naval Surface Warfare Center/Indian Head Division(Code 64C4)

Indian Head, MD 20640-5000Phone: (301) 743-4397

DSN: 354-4397

Naval Surface Warfare Center/Port Hueneme Division POC is:

Naval Surface Warfare Center/Port Hueneme Division(Code 4L12)

4363 Missile WayPort Hueneme, CA 93043-4307

Phone: (805) 982-7023DSN: 551-7023

Hydrodynamics/Hydroacoustics Technology Center (NAVSEA-03HT) POC is:

Carderock Division, NSWCBuilding 17, Room 120

Bethesda MD 20084-5000Phone: (301) 227-3827

DSN: 287-3827

C-6

Naval Command & Control and Ocean Surveillance Center/RDT&E Division POC is:

Commander, Naval Command & Control and Ocean Surveillance Center/RDT&E Division

(Code 78)San Diego, CA 92152Phone: (619) 553-1637

DSN: 553-1637

Naval Undersea Warfare Center POC is:

Commander, Naval Undersea Warfare Center Detachment, New London(Code 63)

39 Smith St.New London, CT 06320Phone: (203) 440-4059

DSN: 636-4059

Naval Research Laboratory POC is:

Commander, Naval Research Laboratory(Code 5550)

4555 Overlook Drive, S.W.,Washington, DC 20375Phone: (202) 767-3162

DSN: 297-3162

COMOPTEVFOR POC is:

COMOPTEVFOR(Code 332)

7970 Diven St.Norfolk, VA 23505-1498Phone: (804) 445-0292

DSN: 565-0292

C-7

Center for Naval Analysis POC is:

Center for Naval Analysis4401 Ford Avenue

PO. Box 16268Alexandria, VA 22302-0268Phone: (703) 824-2352/2998

DSN: 289-2352

Naval Center for Cost Analysis POC is:

Naval Center for Cost Analysis1111 Jefferson Davis Highway

Suite 400 West TowerArlington VA 22202-4306

Phone: (703) 604-0293DSN: 664-0293

NAVY MODELING AND SIMULATION CATALOG

The Navy Modeling and Simulation Catalog contains information on a number of Navymodels ranging from engineering to campaign level. The catalog focuses on models usedfor Joint Mission Area and Support Area Assessments performed by the Chief of NavalOperations. The POC is:

Space and Naval Warfare Systems Command(SPAWAR 312)

2451 Crystal DriveArlington, VA 22245-5200

Phone: (703) 602-4541DSN: 332-4541

C-8

DEPARTMENT OF THE NAVY DOCUMENTS

In addition to the DON Instructions on Modeling and Simulation described in Chapter 3,the following documents concerning acquisition and M&S may be useful to the reader.

OPNAV Instruction 5000.42D. (19 April 1993). OPNAV Role and Responsibilities in theAcquisition Process.

SECNAVINST 5000.2A. (9 Dec 92). Implementation of Defense Acquisition Manage-ment Documentation and Reports.

PRINCIPLES FOR VERIFICATION, VALIDATION, AND ACCREDITATION(VV&A) OF NAVY MANAGED MODELS AND SIMULATIONS (M&S) , dated 15December 1993, prepared by SPAWAR 31. This document identifies principles and tech-niques to enhance V&V of Navy managed models and simulations and describes methodsto facilitate accreditation. It describes three levels of accreditation and discusses the re-quirements for each level, including model documentation, verification plans & tests, vali-dation plans & tests, configuration management, data, and personnel qualifications.

E-1

APPENDIX EDEPARTMENT OF THE AIR FORCE SOURCES

OF INFORMATION FOR MODELING AND SIMULATIONIN WEAPON SYSTEMS ACQUISITION

This appendix provides information regarding various Air Force organizations’ M&S ac-tivities. It provides office symbols, phone numbers and addresses to assist users in makinginitial contact.

POLICY AND GUIDANCE

Headquarters U.S. Air Force, Directorate of Modeling, Simulations, and Analysis (HQUSAF/XOM): The HQ USAF/XOM promulgates Air Force M&S policy; publishes guid-ance and administers the development of AFPD 16-10, Modeling and Simulation Man-agement (draft, May 1994), AFI 16-1001, Verification, Validation, and Accreditation (draft),and AFI 16-1002, M&S Management (draft). The HQ USAF/XOM is the single point ofcontact for M&S issues and activities within the Air Force. They also represent the AirForce in joint, multi-service and multi-agency M&S efforts.

POINTS OF CONTACT:

This list is a continuation of very good information provided in AFMCP 800-66, Atch 7,dated 1 July 1993. This pamphlet is also a good reference for specific model information;description, OPR and phone numbers. This list is not a complete listing of all M&S con-tacts, however, it is a good starting point in locating assistance.

HQ USAF/XOM Room 4C1059 Phone: (703) 695-1835/18471400 Air Force, Pentagon DSN: 225-1847Washington D.C. 20330-1400 FAX: (703) 695-1161

HQ USAF/XOME Phone: (202) 504-5333Evaluation Support Division DSN: 285-5333624 9th Street NW, Suite 300Washington D.C. 20001-6303

HQ USAF/XOMT Phone: (202) 504-5339Technical Support Division DSN: 285-5339(same address as XOME)

HQ USAF/XOMW Phone: (202) 504-4441Warfighting Support Division DSN: 285-4441(same address as XOME)

E-2

AFSAA Room 1E388 Phone: (703) 695-9048Air Force Studies and Analysis DSN: 225-90481400 Air Force, PentagonWashington D.C. 20330-1400Other organizations with M&S experience:

HQ AFMC/XRX Phone: (513) 257-4914Director for Requirements DSN: 787-49144375 Chidlaw Rd. Suite 6Wright-Patterson AFB, OH 45433-5001

HSC/XR (AFMC) Phone: (512) 471-3406Human Systems Center DSN: 240-3406Brooks AFB TX 78235-5000

SMC/XR (AFMC) Phone: (310) 336-4613Space & Missile Systems Center DSN: 833-4613Los Angeles AFB CA 90009-2960

ESC/XRP (AFMC) Phone: (617) 377-6554Electronic Systems Center DSN: 478-6554Hanscom AFB MA 01731-5000

ASC/XR (AFMC) Phone: (513) 255-4656Aeronautical Systems Center DSN: 785-4656Wright-Patterson AFB OH 45433-5000

Armstrong Laboratory (AFMC) Phone: (512) 471-2424Brooks AFB TX 78235-5000 DSN: 240-2424

Phillips Laboratory (AFMC) Phone: (805) 846-1737Kirtland AFB NM 87117-6008 DSN: 246-1737

Rome Laboratory (AFMC) Phone: (315) 330-7701Griffiss AFB NY 13441-5700 DSN: 587-7701

Wright Laboratory (AFMC) Phone: (513) 255-4840Wright-Patterson AFB OH 45433-6553 DSN: 785-4840

OAS/XROffice of Aerospace Studies Phone: (505) 846-8322Kirtland AFB, NM 87117-6008 DSN: 246-8322

E-3

AFSPACECOM/CNA Phone: (719) 554-5196HQ Air Force Space Command DSN: 692-5196Peterson AFB CO 80914-5001

AFIC/DOA Phone: (210) 977-2877Air Force Intelligence Cmd DSN: 969-2877Kelly AFB TX 78243-5000

AFCC/XR Phone: (618) 256-5541HQ Air Force Communications Cmd DSN: 576-5541Scott AFB IL 62225-5000

ACC/XP-JSG Phone: (804) 764-5751HQ Air Combat Command DSN: 574-5751Langley AFB VA 23665-5520

AMC/XPY Phone: (618) 256-5560HQ Air Mobility Command DSN: 576-5560Scott AFB IL 62225-5001

ATC/XPC Phone: (210) 652-2640Air Training Command DSN: 487-2640Randolph AFB TX 78150-5000

AFOTEC/SANAF Operational Test & Eval Ctr Phone: (505) 846-1357Kirtland AFB NM 87117-6008 DSN: 246-1357

AWS/XTX Phone: (618) 256-4598Air Weather Service DSN: 576-4598Scott AFB IL 62225-5008

STRATCOM/J53 Phone: (402) 294-2355HQ US Strategic Command DSN: 271-2355Force Assessment DivisionOffutt AFB NE 68113-5001

AFDTC/XRP Phone: (904) 882-4188AF Developmental Test Center DSN: 872-4188Eglin AFB, FL 32542-5495

AFFTC/XRX Phone: (805)277-3837AF Flight Test Center DSN: 527-3837Edwards AFB CA 93542-1036

E-4

AEDC/DOT Phone: (615)454-6508Arnold Engineering Development Ctr DSN: 340-6508Arnold AFB TN 37389-9011

HQ AFMC/STXP Phone: (513)257-7850AFMC Science & DSN: 787-7850 Technology DirectorateWright-Patterson AFB OH 45433-5006

AFOSR/XPP Phone: (202)767-6010Air Force Office of DSN: 297-6010 Scientific ResearchBolling AFB DC 20332-0001

HQ AFMC/LGP Phone: (513)257-5610AFMC Logistics Directorate DSN: 787-5610Wright-Patterson AFB OH 45433-5006

OC-ALC/FMPM Phone: (405)739-2519Oklahoma City Air Logistics Center DSN: 336-2519Tinker AFB OK 73145-3056

OO-ALC/FMPM Phone: (801)777-5851Ogden Air Logistics Center DSN: 924-5851Hill AFB UT 84056-5038

SA-ALC/FMPB Phone: (201)925-6726San Antonio Air Logistics Center DSN: 945-6726Kelly AFB TX 78241-6435

SM-ALC/FMPM Phone: (916)643-6162Sacramento Air Logistics Center DSN: 633-6162McClellan AFB CA 95652-1060

WR-ALC/FMPX-1 Phone: (912)926-3202Warner Robins Air Logistics Center DSN: 468-3202Robins AFB GA 31098-1640

AGMC/FM(2) Phone: (614)522-7643Aerospace Guidance & Metrology Ctr DSN: 346-7643Newark AFB OH 43057-5260

E-5

CATALOGS/SOURCE DOCUMENTS:

Catalog of Simulation Models and Wargames Used for Unit and Leader Training.(1987).(2nd ed.) Orlando FL: Training and Performance Data Center.

Catalog of War Games, Training Games, and Combat Simulations.(1983). WashingtonDC: Deputy Under Secretary of the Army (Operations Research).

Department of Defense Catalog of Logistics Models.(1990). Fort Lee VA: Defense Logis-tics Studies Information Exchange.

1990 Catalog of Computer Simulation Tools.(1990). Washington DC: Air Force Center ofStudies and Analysis.

Major Military Models Written in SIMSCRIPT II.5. (3rd ed.) La Jolla CA: C.A.C.I.

A Summary of Analysis Methodologies Used in the Directorate of Mission Analysis, (1987).Wright-Patterson AFB OH: Deputy for Development Planning, Aeronautical SystemsCenter.

Survey of Models/Simulations at RL (Rome Laboratory). (1986). Vol. 3. Griffiss AFB NY:Rome Air Development Center.

Science Advisory Board (SAB) ad-hoc Committee on Modeling and Simulation, Decem-ber 1991.

Science Advisory Board (SAB) Report for Theater Missile Defense (TMD),(1993).

AFMCP 800-66. (July 1993). AFMC Models and SimulationS (M&S) Guide. Wright-Patterson AFB, OH: HQ Air Force Materiel Command.

HQ USAF/XOM. Newsletter: Issues in Air Force Simulation and Analysis, (January 1994).

F-1

APPENDIX FADDITIONAL SOURCES OF INFORMATION FOR MODELING

AND SIMULATION IN WEAPON SYSTEMS ACQUISITION

This appendix provides information on the Modeling and Simulation Information System(MSIS) and the DoD Information Analysis Centers (IAC). These are additional sources ofinformation which may assist program managers in their search for models, simulations ordata.

THE MODELING AND SIMULATION INFORMATION SYSTEM

1. Name of System: Modeling & Simulation Information System (MSIS).

2. Proponent / Sponsor / Operator: Defense Modeling & Simulation Office (DMSO).

3. Purpose: Serve as an information clearing house for M&S and related information.Provide a means of disseminating and coordinating M&S related information.

4. System Contents: Menu driven system containing catalogs to provide summary in-formation on models, simulations and applicable data (including the J-8 and Service spe-cific catalogs); documents and reports pertaining to activities and developments in M&Salong with related and supporting areas; glossary of terms; calendar of events; remoteaccess to other M&S information repositories. Key word / topic search capability in cata-logs and other modules. All data are unclassified, with unrestricted distribution. The sys-tem also provides a number of support utilities and functions, including e-mail.

5. Account Availability: Individuals with interests in Modeling & Simulation and re-lated areas.

6. Requirements for Use: Users must register (see POC information, below) to obtainan account on the system. User must provide necessary hardware and software to commu-nicate with the system, e.g., PC with modem and terminal emulation/communications soft-ware or workstation with internet access. Direct dial (703 area code) and tymnet (localnumber access, nation-wide) access are provided. Users may also connect over the internetvia telnet. Users with systems running a gopher client may connect to the system throughthe gopher client. As of July 1994 there is no charge for connect time.

7. Point of Contact (POC) for information or user accounts:Modeling and Simulation Information System

ATTN: Administrative Support1901 N. Beauregard St., Suite 510, Alexandria, VA 22311

Phone: (703) 379-3770 Fax: (703) 379-3778E-mail: [email protected] - e-mail

F-2

CATALOGS LISTED IN THE MSIS (AS OF JULY 1994):

1. J-8 Catalog of Models and Simulations (12th edition)

2. TRANSCOM System Model Catalog (Sep 89)

3. Catalog of War Games, Training Games and Combat Simulation

4. MOdels and Simulations: Army Integrated Catalog (MOSAIC) (Dec 93)

5. Navy�s Catalog of Models and Simulations (5 Oct 93)

6. U.S. Air Force Rome Laboratory Models and Simulations Catalog (Oct 93)

7. DMSO - Catalogs of Models and Simulations

8. Institute for Simulation and Training (IST) Catalog of Models and Simulations Docu-ments, Videos, Files, etc.

NOTE: The content and format of the entries in the individual catalogs vary.

THE DOD INFORMATION ANALYSIS CENTERS

The DoD IACs provide information to users which allows them to benefit from experi-ences of counterparts in comparable fields of endeavor, and increase productivity and qualityof research.

The IACs are established under DoD Regulation 3200.12-R-2, Centers for the Analysis ofScientific and Technical Information, dated 17 January 1985. Their primary mission is tocollect, analyze, synthesize and disseminate worldwide scientific and technical informa-tion in clearly defined, specialized fields or subject areas.

There are 26 IACs supported by DoD, maintaining comprehensive knowledge bases whichinclude historical, technical, scientific and other data and information collected on a world-wide basis. Information collections include a wide range of unclassified, limited distribu-tion and classified information appropriate to the requirements of sponsoring technicalcommunities.

The IACs also collect, maintain, and develop analytical tools and techniques includingdata bases, models and simulations. Their collections and products represent intensiveevaluation and screening efforts to create authoritative sources of evaluated data.

For the purposes of this guide, the subject matter covered by existing DoD IACs is relatedto military functions, roles/missions and to key technologies. Figures F-1 and F-2 display

F-3

the association of each IAC based on this organization.

Figure F-1 maps DoD IACs for users based on military mission or function. The IACs areidentified by their abbreviations, which will be expanded later in this appendix.

For example, a user involved in planning, analysis or acquisition in support of SpecialOperations Forces (SOF) will find IACs with a · in the column labeled �SOF� more likely(but not exclusive) providers of appropriate and useful information products and services.

Figure F-1. IACs Associated with Military Missions/Functions

IACs Associated with MilitarIACs Associated with MilitarIACs Associated with MilitarIACs Associated with MilitarIACs Associated with Military Missions/Functionsy Missions/Functionsy Missions/Functionsy Missions/Functionsy Missions/Functions

Com

mun

icat

ions

Logi

stic

s

Trai

ning

Inst

alla

tions

Inte

llige

nce

SO

F

Nuc

lear

Forc

es

Gro

und

Forc

es

Nav

alFo

rces

Tact

ical

Air

Forc

e

MilitarMilitarMilitarMilitarMilitary Missions/Functionaly Missions/Functionaly Missions/Functionaly Missions/Functionaly Missions/FunctionalArArArArArea of Potential Userea of Potential Userea of Potential Userea of Potential Userea of Potential User

IACIACIACIACIACAPMIAC (Airfields, Pavements, & Mobility)

CBIAC (Chemical/Biological Warfare)

CEIAC (Coastal Engineering)

CIAC (Ceramics)

CPIA (Chemical Propulsion)

CRISTIAC (Cold Regions)

CSERIAC (Crew Systems Ergonomics)

CTIAC (Concrete Technology)

DACS (Data Analysis Center for Software)

DASIAC (Nuclear Effects)

GACIAC (Guidance & Control)

HEIAC (Hydraulic Engineering)

HTMIAC (High Temperature Materials)

IRIA (Infrared)

MIAC (Metals)

MMCIAC (Metal Matrix Composites)

MTIAC (Manufacturing Technology)

NTIAC (Nondestructive Testing)

PLASTEC (Plastics)

RAC (Reliability)

SMIAC (Soil Mechanics)

SURVIAC (Survivability/Vulnerability)

TWSTIAC (Tactical Warfare Simulation & Technology)

F-4

Although likely to find useful M&S products within the TWSTIAC, a user would alsopossibly find useful M&S products addressing survivability within the SURVIAC.

Figure F-2 illustrates the subject matter focus of IACs as they relate to DoD Key Tech-nologies. Again, a · in a Key Technology column highlights areas of special interest orfocus within the IAC program. A key feature of the DoD IACs is that they work together toensure that user needs are met.

Figure F-2. IACs Associated with DoD Key Technologies

IACs Associated with DoD Key TIACs Associated with DoD Key TIACs Associated with DoD Key TIACs Associated with DoD Key TIACs Associated with DoD Key Technologiesechnologiesechnologiesechnologiesechnologies

Com

pute

r

Com

mun

icat

ions

Net

wor

king

Ele

ctro

nic

Dev

ices

Env

ironm

enta

lE

ffect

s

Mat

eria

ls a

ndP

roce

sses

Ene

rgy

Sto

rage

Pro

puls

ion

and

Ene

rgy

Con

vers

ion

Des

ign

Aut

omat

ion

Sof

twar

e

Sen

sors

Hum

an-S

yste

mIn

terf

ace

Key TKey TKey TKey TKey Technology Arechnology Arechnology Arechnology Arechnology Area of Interea of Interea of Interea of Interea of Interestestestestestto Potential User Interto Potential User Interto Potential User Interto Potential User Interto Potential User Interestestestestest

IACIACIACIACIACAPMIAC (Airfields, Pavements, & Mobility)

CBIAC (Chemical/Biological Warfare)

CEIAC (Coastal Engineering)

CIAC (Ceramics)

CPIA (Chemical Propulsion)

CRISTIAC (Cold Regions)

CSERIAC (Crew Systems Ergonomics)

CTIAC (Concrete Technology)

DACS (Data Analysis Center for Software)

DASIAC (Nuclear Effects)

GACIAC (Guidance & Control)

HEIAC (Hydraulic Engineering)

HTMIAC (High Temperature Materials)

IRIA (Infrared)

MIAC (Metals)

MMCIAC (Metal Matrix Composites)

MTIAC (Manufacturing Technology)

NTIAC (Nondestructive Testing)

PLASTEC (Plastics)

RAC (Reliability)

SMIAC (Soil Mechanics)

SURVIAC (Survivability/Vulnerability)

TWSTIAC (Tactical Warfare Simulation & Technology)

F-5

FEES FOR SERVICES

Financial support for basic IAC operations is provided by the Defense Technical Informa-tion Center (DTIC). However, the IACs offset costs incurred in collecting, analyzing anddisseminating information by a service charge structure implemented in accordance withguidance provided by DoD. However, no charges are incurred without the explicit agree-ment of the customer.

CONTACTING THE DOD IACS

Users may contact the appropriate IAC(s) directly for specific information. For generalinformation on the DoD IAC program, or for assistance in locating an IAC to service aparticular need, users can contact:

Defense Technical Information CenterATTN: DTIC-AICameron Station

Alexandria, VA 22304-6145Phone: (703) 274-6260

DSN: 284-6260FAX: (703) 274-0980

E-mail: [email protected]

ALPHABETIC LISTING OF DOD IACS

Airfields, Pavements and Mobility Information Analysis Center (APMIAC):

Subject coverage � Airfields, pavements, vehicle mobility, and terrain, as relevant prima-rily to military needs. Specific areas of vehicle on- and off-road mobility, ground flotationand terrain evaluation. The POC is:

Commander and DirectorU.S. Army Engineer Waterways Experimentation Station

ATTN: CEWES/GM-L3909 Halls Ferry Road

Vicksburg, MS 39180-6199Phone: Tech/Biblio Inquiries:

Mobility/Terrain: (601) 634-2734Airfields/Pavements: (601) 634-3304

Computerized Database/Library: (601) 634-4120FAX: (601) 634-3068

F-6

Ceramics Information Analysis Center (CIAC)

Subject coverage � Source of engineering and technical data, and research and develop-ment information on monolithic ceramics and ceramic composites, hybrids, laminates andcoatings used in Defense systems. The POC is:

CIAC/CINDASPurdue University2595 Yeager Road

West Lafayette, IN 47906-1398Phone: (317) 494-9393FAX: (317) 496-1175

Chemical Propulsion Information Agency (CPIA)

Subject coverage � Acquisition, analysis and dissemination of information in the areas ofmissile, space and gun propulsion technology. Hardware of interest includes rocket mo-tors, rocket engines, air breathing missile propulsion systems, electric and nuclear spacepropulsion systems, electric and conventional guns, gas generators, mines and torpedoes,and their inert and energetic components. The POC is:

The Johns Hopkins UniversityChemical Propulsion Information Agency10630 Little Patuxent Parkway, Suite 202

Columbia, MD 21044-3201Phone: (410)992-7306FAX: (410)730-4969

Chemical Warfare/Chemical and Biological Defense (CW/CBD) Information Analy-sis Center (CBIAC):

Subject coverage � DoD focal point for technical information related to CW/CBD. TheCBIAC collects, analyzes, summarizes and stores CW/CBD information available fromboth domestic and foreign sources. The POC is:

Battelle Edgewood OperationsATTN: CBIAC

2113 Emmorton Park Road, Suite 200Edgewood, MD 21040-1037

Inquiries, Products, Publications: (601) 634-2734FAX: (410) 676-9703

Technical Area Tasks: (410) 676-0200FAX: (410) 676-8862

F-7

Coastal Engineering Information Analysis Center (CEIAC):

Subject coverage � Coastal engineering, coastal regions, beaches, shore erosion, coastalenvironments, oceanography, ocean waves tides, inlets and hydrodynamics. The POC is:


ATTN: CEWES-CV-I3909 Halls Ferry Road

Vicksburg, MS 39180-6199Phone: (601) 634-2012FAX: (601) 634-3433

Cold Regions Science and Technology Information Analysis Center (CRSTIAC)

Subject coverage � Hydrology, climatology, civil engineering, meteorology, military op-erations, mobility, construction, materials in cold temperatures and environmental issuesrelating to cold regions. The POC is:

U.S. Army Cold Regions Research and Engineering Laboratory72 Lyme Road

Hanover, NH 03755-1290Phone: (603) 646-4221FAX: (603) 646-4712

Concrete Technology Information Analysis Center (CTIAC)

Subject coverage � Concrete, reinforced concrete, reinforcing materials, cements, mix-tures, construction materials, loads (force), fracture (mechanics), deformation, degrada-tion, chemical analysis, repair, evaluation, maintenance and rehabilitation. The POC is:


ATTN: CEWES/SV-Z3909 Halls Ferry Road


F-8

Crew Station Ergonomics Information Analysis Center (CSERIAC)

Subject coverage � Scientific and technical knowledge and data concerning human char-acteristics, abilities, limitations, physiological needs and tolerances, performance, bodydimensions, biomechanical dynamics and physical strength. Also includes engineeringand design data concerning equipment intended to be used, operated, maintained or con-trolled in sea, land, air and space environments. The POC is:

CSERIAC Program OfficeAL/CFH/CSERIAC, Bldg 248

2255 H StreetWright-Patterson AFB, OH 45433-7022

Phone: (513) 255-4842 or DSN 785-4842FAX: (513) 255-4823 or DSN 785-4823

Data and Analysis Center for Software (DACS)

Subject coverage � Supports the development, testing, validation, transitioning of soft-ware engineering technology to the defense community, industry and academia. Includesthe entire software life cycle (requirements definition, design, coding, integration, testingand post-deployment support). The POC is:

Data & Analysis Center for SoftwareKaman Sciences Corporation

P.O. Box 120Utica, NY 13503-0120Phone: (315) 734-3696FAX: (315) 734-3699

DoD Nuclear Information Analysis Center (DASIAC)

Subject coverage � Nuclear weapons explosion phenomena, effects on military strategicand tactical systems and components; survivability, vulnerability and hardening, militarydoctrine and operations, nuclear weapon effects testing. The POC is:

DASIAC2560 Huntington Avenue, Suite 500

Alexandria, VA 22303-1490Phone: (703) 960-4774FAX: (703) 329-7198

F-9

Guidance and Control Information Analysis Center (GACIAC)

Subject coverage � Dissemination and exchange of technical information related to theguidance and control of weapons. These include missiles, rockets, bombs, submunitions,projectiles, mines and munition dispensing canisters. The POC is:

IIT Research InstituteGACIAC

10 West 35th StreetChicago, IL 60616-3799

Phone: (312) 567-4345/4492FAX: (312) 567-4889

High Temperature Materials Information Analysis Center (HTMIAC)

Subject coverage � Central source of engineering data and technical information on hightemperature materials, properties and laser effects; especially in the critical technologyareas of aerospace structural composites and metals, infrared detector materials and coat-ings. The POC is:

HTMIAC/CINDASPurdue University2595 Yeager Road


Hydraulic Engineering Information Analysis Center (HEIAC)

Subject coverage � River, harbor and tidal hydraulics; flow through pipes, conduits, chan-nels and spillways as related to flood control and navigation; hydraulic design and perfor-mance of dams, locks, channels and other structures. The POC is:


ATTN: CEWES/HV-Z3909 Halls Ferry Road


F-10

Infrared Information Analysis Center (IRIA)

Subject coverage � Electro-optics technology of interest to DoD including: sources ofelectromagnetic radiation from the ultraviolet through far infrared spectral regions; radia-tion characteristics of natural and man-made targets; optical properties of materials; detec-tion materials and elements; information processing as it pertains to sensory collection ofdata; imaging, detecting, searching, homing, tracking and ranging subsystems. The POCis:

Environmental Research Institute of MichiganATTN: The IRIA Center

P.O. Box 134001Ann Arbor, MI 48113-4001

Phone: (313) 994-1200 ext. 2302FAX: (313) 994-5550

Manufacturing Technology Information Analysis Center (MTIAC)

Subject coverage � Collection, analysis and dissemination of manufacturing technologyand data. Technology information is acquired in the following defense-related fields; ma-chine tools and manufacturing equipment, robots and special machines, material handlingequipment, controls, software and databases, communication lines and networks, sensors,inspection or checkout procedures, factory automation, computer-integrated manufactur-ing, specific defense-related products, and the management aspects of manufacturing tech-nology. The POC is:

MTIACIIT Research Institute10 West 35th Street

Chicago, IL 60616-3799Phone: (312) 567-4732FAX: (312) 567-4736

Metal Information Analysis Center (MIAC)

Subject coverage � Central source of engineering and technical data and research anddevelopment information on monolithic metals, metal alloys, intermetallic compoundsand coatings used in defense systems. The POC is:

MIAC/CINDASPurdue University2595 Yeager Road


F-11

Metals Matrix Composites Information Analysis Center (MMCIAC)

Subject coverage � Central source of engineering and technical data, and research anddevelopment information on metal matrix composites (MMC). Information pertaining toaspects of MMC applications in air, sea, land and space-based systems necessary for sup-port of DoD basic and applied research. The POC is:

MMCIAC/CINDASPurdue University2595 Yeager Road


Nondestructive Testing Information Analysis Center (NTIAC)

Subject coverage � Information pertaining to non-destructive testing, non-destructive evalu-ation, and non-destructive inspection. Coverage includes techniques and processes. Theuse of nondestructive sensors for manufacturing and materials process control, and forintelligent or adaptive control applications, is also within the purview of NTIAC. ThePOC is:

Texas Research Institute Austin415A Crystal Creek DriveAustin, TX 78746-6201Phone: (512) 263-2106FAX: (512) 263-3530

Plastics Technical Evaluation Center (PLASTEC)

Subject coverage � Acquisition, evaluation and exchange of technical information relatedto plastics, adhesives, and organic matrix composites. Coverage includes technology fromapplied research through fabrication, with emphasis on properties and performance. ThePOC is:

Plastics Technical Evaluation Center (PLASTEC)U.S. Army Armament, Munitions and Chemicals Command

Picatinny Arsenal, NJ 07806-5000Phone: (201) 724-4222/5859

FAX: (201) 361-7378

F-12

Reliability Analysis Center (RAC)

Subject coverage � Reliability, maintainability and quality of devices and systems. Reli-ability and failure mode/mechanism information that is generated during all phases ofcomponent fabrication, testing, equipment assembly and operation. Information and datainclude process control, quality assurance practices, screening and burn-in, qualificationand environmental screening, failure analysis, reliability prediction and demonstrations,field testing and mission deployment. Reliability effects of electrical overstress and elec-trostatic discharge (EOS/ESD) on semiconductors is a specialty. The POC is:

Reliability Analysis Center (RAC)IIT Research Institute

P.O. Box 4700Rome, NY 13440-8200Phone: (315) 337-0900FAX: (315) 337-9932

Soil Mechanics Information and Analysis Center (SMIAC)

Subject coverage � Soil mechanics, engineering geology, rock mechanics, soil dynamics,earthquake engineering, earth and rockfill dams, levees, earth retaining structures and build-ing foundations, and laboratory testing of soils and rocks. The POC is:


ATTN: CEWES/GV-Z3909 Halls Ferry Road


Survivability/Vulnerability Information and Analysis Center (SURVIAC)

Subject coverage � The DoD focal point for non-nuclear survivability/vulnerability data,information, methodologies, models and analyses relating to U.S. and foreign aeronauti-cal and surface systems. Information and data covers the survivability of all allied andother non-adversary systems to threat weapons as well as the effectiveness of U.S. weap-ons against foreign systems. The POC is:

SURVIACBooz-Allen & HamiltonWL/FIVS/SURVIAC

2130 Eighth St., Bldg 45, Suite 1Wright-Patterson AFB, OH 45433-7542

Phone: (513) 255-4840/9509 or DSN 785-4840FAX: (513) 255-9673 or DSN 785-9673

F-13

Tactical Warfare Simulation and Technology Information and Analysis Center (TWSTIAC)

Subject coverage � Principal focus is to meet operational requirements and the underlyingacquisition and training needs associated with non-nuclear roles and missions for all tacti-cal forces of the DoD. The TWSTIAC is the DoD resource for assistance in developingand implementing M&S to operate in the DoD DIS environment. It is specifically man-dated to perform primary research; develop and apply advanced and integrated M&S;develop standardized processes to collect, analyze, evaluate, and synthesize test data. ThePOC is:

Battelle Memorial InstituteATTN: TWSTIAC505 King Avenue

Columbus, OH 43201-2693Phone: (614) 424-5047FAX: (614) 424-4874

Tactical Warfare Technology: (614) 424-7871Distributed Interactive Simulation (DIS) and related

M&S information requirements: (407) 658-5014

MILITARY SERVICE-SPONSORED INFORMATION CENTERS

Several DoD components have established information centers which provide productsand services comparable to those provided by the DoD IACs. Points of contact for thesecenters are provided below. Users should contact these organizations directly for furtherinformation.

U.S. Air Force Aerospace Structures Information Analysis Center (ASIAC):

Subject Coverage � A central agency for the collection and dissemination of informationon aerospace structures. It provides state-of-the-art solutions to small complex structureproblems and distributes structural computer programs not available at other dissemina-tion centers. The POC is:

Aerospace Structures Information and Analysis CenterATTN: WL/FIBRA/ASIAC

Wright-Patterson AFB, OH 45433-6553Phone: (513) 255-6688 or DSN: 785-6688

FAX: (513) 476-4682

F-14

U.S. Navy Shock and Vibration Information Analysis Center (SAVIAC):

Subject Coverage � Research, analysis and testing related to the structural dynamics, me-chanics and physical environmental effects on vehicles, structures, equipment, compo-nents and humans under operational and combat conditions. This encompasses technicalareas of vibration, shock, blast, crash, impact, penetration, vibroacoustics and mechanicalenvironments along with supporting areas of software, sensors, instrumentation and dy-namic material properties. The SAVIAC operates under the direction of a multiple-agencyTechnical Advisory Group with members from the Army, Navy, Air Force, DNA, NASAand DOE laboratories. The POC is:

Shock and Vibration Information Analysis Center2711 Jefferson Davis Highway #600

Arlington, VA 22202-4158Phone: (703) 412-7570FAX: (703) 412-6555

U.S. Air Force Supportability Investment Decision Analysis Center (SIDAC):

Subject Coverage: Acquire, improve and apply existing analysis methods, models, tech-niques and enabling services for all aspects of weapon system supportability. It focuses inthe areas of logistics support, logistics research and development, technology insertionand supportability investment information. The POC is:

Supportability Investment Decision Analysis Center (SIDAC)ATTN: SIDAC Program Director

Battelle Technical Support Operations, Dayton5100 Springfield Pike, Suite 311

Dayton, OH 45431Phone: (513) 258-6711 or 1-800-547-4322

FAX: (513) 254-9575

G-1

APPENDIX GMODELING AND SIMULATION TEMPLATES FOR USE

IN WEAPON SYSTEMS ACQUISITION ACTIVITIES

This appendix contains sample templates for the application of modeling and simulation(M&S) to some of the activities described in Chapter 5. These templates serve only as ageneral guideline to help acquisition managers in planning how they will use models andsimulations in support of their programs. They are meant to stimulate ideas for planningand not to be directly applied to a particular program. The activities and M&S tools usedwill vary from one program to another. Therefore, such templates must be tailored specifi-cally for each individual acquisition program including the specific models and simula-tions planned to support each activity or document preparation within the program. Thefunctional activities addressed within this appendix include: requirements definition; pro-gram management; design and engineering; manufacturing; test and evaluation; training;and logistics support.

In these templates, readers will notice that models and simulations span across multiplephases of a program and that the same type of models and simulations are often used tosupport multiple acquisition activities. Recognizing this, acquisition managers should:

� Plan on M&S reuse to avoid duplicative M&S development efforts and to improveconsistency across the acquisition process;

� Consider how the various functional disciplines can share the information from suchmodels and simulations, to facilitate Integrated Product and Process Development (IPPD);and

� Plan for the transition of developmental models and simulations to support activitiesin later phases, such as to support weapon system upgrades or the training environment.

G-2

Figure G-1. Modeling and Simulation Application in Requirements Definition

MO

DE

LS &

SIM

ULA

TIO

NS

MS

MS

M

S

M

S

M

S 0

I

I

I

III

IV

Pre

-MS

0

Con

cept

Exp

l & D

ev

Dem

/Val

E

ng &

Mfg

Dev

Pro

d &

Dep

loy

Ope

r. &

Sup

port

CO

ST

(AN

ALO

GY

&

C

OS

T (

PA

RA

ME

TR

IC &

C

OS

T (

EN

GIN

EE

RIN

G)

O&

S C

OS

T T

RA

CK

ING

AN

D P

RE

DIC

TIO

NP

AR

AM

ET

RIC

)

E

NG

INE

ER

ING

)

PR

OG

RA

M P

LAN

NIN

G

R

EF

INE

/UP

DA

TE

S(S

CH

ED

ULE

, RIS

K, P

LAN

S)

PR

OG

RA

M A

CT

IVIT

IES

/DO

CU

ME

NT

S❷

CO

EA

❷ C

AR

D❷

IPS

:

- A

CQ

. ST

RA

TE

GY

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

AT

E-

EN

VIR

ON

ME

NT

AL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TUR

E-

CO

OP

ER

AT

IVE

O

PP

OR

TU

NIT

IES

D

OC

UM

EN

T

- A

CQ

. ST

RA

TE

GY

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

AT

E-

EN

VIR

ON

ME

NT

AL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TUR

E-

CO

OP

ER

AT

IVE

O

PP

OR

TU

NIT

IES

D

OC

UM

EN

T

❷ C

OE

A❷

CA

RD

❷ IP

S:

❷ C

OE

A (

UP

DA

TE

)❷

CA

RD

❷ IP

S:

❷ C

OE

A (

UP

DA

TE

)❷

CA

RD

❷ IP

S:

- A

CQ

. ST

RA

TE

GY

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

AT

E-

EN

VIR

ON

ME

NT

AL

A

NA

LYS

IS

- P

RO

GR

AM

ST

RU

CTU

RE

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

- A

CQ

. ST

RA

TE

GY

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

AT

E-

EN

VIR

ON

ME

NT

AL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TUR

E

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Req

uir

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Req

uir

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Req

uir

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Req

uir

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Req

uir

emen

ts D

efin

itio

nem

ents

Def

init

ion

emen

ts D

efin

itio

nem

ents

Def

init

ion

emen

ts D

efin

itio

n

G-3

Figure G-2. Modeling and Simulation Application in Program Management

MO

DE

LS &

SIM

ULA

TIO

NS

MS

MS

M

S

M

S

M

S 0

I

I

I

III

IV

Pre

-MS

0

Con

cept

Exp

l & D

ev

Dem

/Val

E

ng &

Mfg

Dev

Pro

d &

Dep

loy

Ope

r. &

Sup

port

CO

ST

(A

NA

LOG

Y &

CO

ST

(P

AR

AM

ET

RIC

&

CO

ST

(E

NG

INE

ER

ING

)

O

&S

CO

ST

TR

AC

KIN

G A

ND

PR

ED

ICT

ION

PA

RA

ME

TR

IC)

EN

GIN

EE

RIN

G)

PR

OG

RA

M P

LAN

NIN

G

R

EF

INE

/UP

DA

TE

S(S

CH

ED

ULE

, RIS

K, P

LAN

S)

PR

OG

RA

M A

CT

IVIT

IES

/DO

CU

ME

NT

S❷

CO

EA

❷ C

AR

D❷

IPS

:

- A

CQ

. STR

AT

EG

Y

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

ATE

- E

NV

IRO

NM

EN

TAL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TU

RE

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

- A

CQ

. STR

AT

EG

Y

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

ATE

- E

NV

IRO

NM

EN

TAL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TU

RE

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

❷ C

OE

A❷

CA

RD

❷ IP

S:

❷ C

OE

A (

UP

DA

TE

)❷

CA

RD

❷ IP

S:

❷ C

OE

A (

UP

DA

TE

)❷

CA

RD

❷ IP

S:

- A

CQ

. STR

AT

EG

Y

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

ATE

- E

NV

IRO

NM

EN

TAL

A

NA

LYS

IS

- P

RO

GR

AM

ST

RU

CT

UR

E

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

- A

CQ

. STR

AT

EG

Y

- R

ISK

AS

SE

SS

ME

NT

- A

FF

OR

DA

BIL

ITY

- P

RO

GR

AM

LIF

E C

YC

LE

CO

ST

ES

TIM

ATE

- E

NV

IRO

NM

EN

TAL

A

NA

LYS

IS-

PR

OG

RA

M S

TR

UC

TU

RE

- C

OO

PE

RA

TIV

E

OP

PO

RT

UN

ITIE

S

DO

CU

ME

NT

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Pr

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Pr

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Pr

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Pr

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Pr o

gra

m M

anag

emen

to

gra

m M

anag

emen

to

gra

m M

anag

emen

to

gra

m M

anag

emen

to

gra

m M

anag

emen

t

G-4

Figure G-3. Modeling and Simulation Application in Design and Engineering

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Des

ign

an

d E

ng

inee

rin

gM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in D

esig

n a

nd

En

gin

eeri

ng

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Des

ign

an

d E

ng

inee

rin

gM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in D

esig

n a

nd

En

gin

eeri

ng

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Des

ign

an

d E

ng

inee

rin

gM

OD

ELS

&S

IMU

LAT

ION

SM

S

M

S

MS

MS

M

S 0

I

II

III

IV

Pre

-MS

0

C

once

pt E

xpl &

Dev

Dem

/Val

E

ng &

Mfg

Dev

P

rod

& D

eplo

y

Ope

r. &

Sup

port

EN

GIN

EE

RIN

G M

&S

(TE

CH

NO

LOG

Y &

PH

EN

OM

ON

OLO

GY

)

VIR

TU

AL

PR

OT

OT

YP

ES

(TE

CH

NO

LOG

Y &

CO

NC

EP

TS

)

LIV

E T

EC

HN

OLO

GY

DE

MO

NS

TR

AT

ION

S

PR

OG

RA

M A

CT

IVIT

IES

/DO

CU

ME

NT

S❷ O

RD

❷ S

EM

P

❷ S

YS

TE

M/S

EG

ME

NT

❷

SP

EC

IFIC

AT

ION

S

❷ C

RIT

ICA

L S

YS

TE

M

❷

CH

AR

AC

TE

RIS

TIC

S

EN

GIN

EE

RIN

G D

ES

IGN

(SY

ST

EM

, SU

BS

YS

TE

M &

PH

EN

OM

ON

OLO

GY

)

CA

D/C

AM

RIS

K, S

UR

VIV

AB

ILIT

Y/

VU

LNE

RA

BIL

ITY

,S

PE

CIA

LTY

EN

GIN

EE

RIN

G

VIR

TU

AL

PR

OT

OT

YP

ES

(PR

OT

OT

YP

ES

YS

TE

MS

)

HW

/SW

IL

ST

IMU

LAT

OR

S

PH

YS

ICA

L M

OD

ELS

EN

GIN

EE

RIN

G D

ES

IGN

(SU

BS

YS

TE

M &

CO

MP

ON

EN

TS

)

CA

D/C

AM

VIR

TU

AL

PR

OT

OT

YP

ES

HW

/SW

IL

ST

IMU

LAT

OR

S

PH

YS

ICA

L M

OD

ELS

CA

D/C

AM

HW

/SW

IL (

DE

SIG

NC

HA

NG

ES

)

ST

IMU

LAT

OR

S

EN

GIN

EE

RIN

G D

ES

IGN

(MO

DIF

ICA

TIO

N D

ES

IGN

)

HW

/SW

IL (

MO

DIF

ICA

TIO

ND

ES

IGN

AN

D P

3 I)

ST

IMU

LAT

OR

S

❷ O

RD

❷ S

EM

P

❷ D

EV

ELO

PM

EN

T❷ S

PE

CIF

ICA

TIO

NS

❷ C

RIT

ICA

L S

YS

TE

M❷ C

HA

RA

CT

ER

IST

ICS

❷ O

RD

❷ S

EM

P

❷ P

RO

DU

CT

, MA

TE

RIA

L,❷ P

RO

CE

SS

SP

EC

IFIC

AT

ION

S

❷ C

RIT

ICA

L S

YS

TE

M❷ C

HA

RA

CT

ER

IST

ICS

❷ C

RIT

ICA

L S

YS

TE

M

CH

AR

AC

TE

RIS

TIC

SD

ES

IGN

RE

VIE

WS

& A

UD

ITS

❷ D

EF

INE

CO

NC

EP

TS

&

ALT

ER

NA

TIV

ES

❷ S

YS

TE

M D

ES

IGN

❷ S

UB

SY

ST

EM

&

CO

MP

ON

EN

T D

ES

IGN

❷ E

NG

INE

ER

ING

D

ES

IGN

CH

AN

GE

S

SR

RS

DR

SS

RP

DR

CD

RT

RR

FC

AP

CA

G-5

Figure G-4. Modeling and Simulation Application in Manufacturing

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or tttt t

G-6

Figure G-5. Modeling and Simulation Application in Test and Evaluation

MO

DE

LS &

SIM

ULA

TIO

NS

MS

MS

M

S

M

S

MS

0

I

II

III

IV

Pre

-MS

0

C

once

pt E

xpl &

Dev

Dem

/Val

E

ng &

Mfg

Dev

P

rod

& D

eplo

y

O

per.

& S

uppo

rt

CA

MP

AIG

N/T

HE

AT

ER

(OU

TC

OM

ES

)

MIS

SIO

N/B

AT

TLE

(EF

FE

CT

IVE

NE

SS

)

EN

GA

GE

ME

NT

(EF

FE

CT

IVE

NE

SS

)

VIR

TU

AL

PR

OT

OT

YP

ES

PR

OG

RA

M A

CT

IVIT

IES

/DO

CU

ME

NT

S

❷ ID

EN

TIF

Y M

OE

’s

❷ M

OP

’s

❷ C

RIT

ICA

L S

YS

TE

M❷

CH

AR

AC

TE

RIS

TIC

S

EN

GIN

EE

RIN

G(P

ER

FO

RM

AN

CE

)

PH

YS

ICA

L M

OD

ELS

LIV

E P

RO

TO

TY

PE

TE

ST

ING

TH

RE

AT

SIM

ULA

TO

RS

SU

RV

IVA

BIL

ITY

/V

ULN

ER

AB

ILIT

Y

PH

YS

ICA

L M

OD

ELS

LIV

E P

RO

TO

TY

PE

TE

ST

ING

EN

GIN

EE

RIN

G (

E.G

.H

W/S

WIL

)

ST

IMU

LAT

OR

S

DIS

TR

IBU

TE

D S

IMU

LAT

ION

LIV

E T

ES

TIN

G

EN

GIN

EE

RIN

G (

E.G

.H

W/S

WIL

TO

SU

PP

OR

TP

AT

&E

)

ST

IMU

LAT

OR

S

❷ P

RE

LIM

INA

RY

❷ T

EM

P❷

TE

MP

LIV

E T

ES

TIN

G

EN

GIN

EE

RIN

G (

TO

SU

PP

OR

T M

OD

IFIC

AT

ION

TE

ST

ING

)

❷ T

EM

P

❷ D

OT

&E

RE

PO

RT

❷ ❷ LF

T R

EP

OR

T

❷ T

EM

P

❷ M

OD

IFIC

AT

ION

TE

ST

ING

❷ F

OT

&E

❷ D

T&

E

❷ P

AT

&E

❷ F

OT

&E

❷ C

OM

PO

NE

NT

DT

&E

❷ E

AR

LY O

PE

RA

TIO

NA

L❷

AS

SE

SS

ME

NT

❷ S

UB

SY

ST

EM

&

SY

ST

EM

DT

&E

❷ LF

T&

E

❷ O

PE

RA

TIO

NA

L

AS

SE

SS

ME

NT

❷ IO

T&

E

UP

DA

TE

SA

S R

EQ

’DU

PD

AT

ES

UP

DA

TE

SU

PD

AT

ES

AS

RE

Q’D

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

TM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in T

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

TM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in T

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Tes

t an

d E

valu

atio

nes

t an

d E

valu

atio

nes

t an

d E

valu

atio

nes

t an

d E

valu

atio

nes

t an

d E

valu

atio

n

G-7

Figure G-6. Modeling and Simulation Application in Logistics Support

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Lo

gis

tics

Su

pp

or tttt t

G-8

Figure G-7. Modeling and Simulation Application in Training

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

TM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in T

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

TM

od

elin

g a

nd

Sim

ula

tio

n A

pp

licat

ion

in T

Mo

del

ing

an

d S

imu

lati

on

Ap

plic

atio

n in

Tra

inin

gra

inin

gra

inin

gra

inin

gra

inin

g

MO

DE

LIN

G &

MO

DE

LIN

G &

MO

DE

LIN

G &

MO

DE

LIN

G &

MO

DE

LIN

G &

SIM

UL

AS

IMU

LA

SIM

UL

AS

IMU

LA

SIM

UL

AT

ION

TIO

NT

ION

TIO

NT

ION

H-1

APPENDIX HABBREVIATIONS AND ACRONYMS USED THROUGHOUT

THIS MODELING AND SIMULATION GUIDEBOOK

6-DOF Six Degree-of-FreedomACAT Acquisition Category

ACE-IT Automated Cost Estimating - Integrated ToolsACMC Assistant Commandant of the Marine Corps

ADM Acquisition Decision MemorandumADS Advanced Distributed Simulation

AFAM Air Force Acquisition ModelAFMC Air Force Materiel Command

AFSAA Air Force Studies and AnalysisALM Airlift Loading ModelAMC Army Materiel CommandAMIP Army Model Improvement Program

AMSAA US Army Material Systems Analysis ActivityAMSEC Army Modeling & Simulation Executive Council

AMSMO Army Model & Simulation Management OfficeAMSMP Army Model & Simulation Management Program

AOR Area of ResponsibilityAPB Acquisition Program Baseline

APMIAC Airfields, Pavements, and Mobility Information Analysis CenterAR Army Regulation

ARPA Advanced Research Projects AgencyASD(PA&E) Assistant Secretary of Defense (Program Analysis and Evaluation)

ASN(RDA) Assistant Secretary of the Navy (Research, Development andAcquisition)

ATD Advanced Technology DemonstrationsATFM&S Acquisition Task Force on Modeling & Simulation

AWC US Army War CollegeAWSIM Air Warfare Simulation Model

BAT Brilliant Anti-TankBFTT Battle Force Tactical TrainingCAA US Army Concepts Analysis Agency

CAAM Composite Area Analysis ModelCAD Computer-Aided DesignCAE Computer-Aided Engineering

CAIG Cost Analysis Improvement GroupCALS Continuous Acquisition and Life-cycle SupportCAM Computer-Aided Manufacturing

CARD Cost Analysis Requirements DocumentCASTFOREM Combined Arms and Support Task Force Evaluation Model

H-2

CATIA Computer-aided Three Dimensional Interactive ApplicationCBIAC Chemical Warfare/Chemical and Biological Defense Information

Analysis CenterCCTT Close Combat Tactical TrainerCEAC US Army Cost and Economic Analysis Center

CECOM US Army Communications-Electronics CommandCED Concept Exploration and Definition

CEIAC Coastal Engineering Information Analysis CenterCEM V Concepts Evaluation Model V

CIAC Ceramics Information Analysis CenterCINC Commander-in-ChiefCJCS Chairman of the Joint Chiefs of Staff

CM Configuration ManagementCOEA Cost and Operational Effectiveness AnalysisCPIA Chemical Propulsion Information Agency

CRSTIAC Cold Regions Science and Technology Information Analysis CenterCSEAL Combat System Engineering and Analysis Laboratory

CSERIAC Crew Station Ergonomics Information Analysis CenterCTIAC Concrete Technology Information Center

DA Department of the ArmyDAB Defense Acquisition Board

DACS Data and Analysis Center for SoftwareDAE Defense Acquisition Executive

DASIAC DoD Nuclear Information and Analysis CenterDDR&E Director of Defense Research and Engineering

DDTO Deputy Director of Technical OperationsDemVal Demonstration and Validation

DepSecDef Deputy Secretary of DefenseDIA Defense Intelligence AgencyDIS Distributed Interactive Simulation

DMS Distributed Models and SimulationsDMSO Defense Modeling & Simulation Office

DoD Department of DefenseDON Department of the NavyDPG Defense Planning GuidanceDSB Defense Science Board

DSMC Defense Systems Management CollegeDT Developmental Test

DT&E Developmental Testing and EvaluationDTIC Defense Technical Information Center

DUSA(OR) Deputy Under Secretary of the Army for Operations ResearchEADSIM Extended Air defense Simulation

ECP Engineering Change ProposalEMD Engineering and Manufacturing Development

ENWGS Enhanced Naval Warfare Gaming System

H-3

EOS/ESD Electrical Overstress and Electrostatic DischargeESAMS Enhanced Surface-to-Air Missile Simnulation

EXCIMS Executive Council on Modeling & SimulationFMF Fleet Marine Force

FOT&E Follow-on Operational Test and EvaluationGACIAC Guidance and Control Information Analysis Center

GWEF Guided Weapons Evaluation FacilityHQMC Headquarters Marine Corps

HTMIAC High Temperature Materials Information Analysis CenterHW/SWIL Hardware/Software-In-The-Loop

HWIL Hardware-In-the-LoopIAC Information Analysis CenterIDA Institute for Defense Analysis

IG Inspector GeneralIOT&E Initial Operational Test and Evaluation

IPPD Integrated Product and Process DevelopmentIPR In Process ReviewIPT Integrated Product Team

IRIA Infrared Information Analysis CenterIWSM Integrated Weapon System Management

JADS/JFS Joint Advanced Distributed Simulation Joint Feasibility StudyJM&S Joint Models and Simulations

JMSEP Joint Modeling and Simulation Executive PanelJROC Joint Requirements Oversight Council

JTCG/AS Joint Technical Coordinating Group on Aircraft SurvivabilityJTCTS Joint Tactical Combat Training System

LAR Launch Acceptability RegionsLCC Life-Cycle CostLFT Live Fire TestingLMI Logistics Management Institute

LOGSA U.S. Army Materiel Command Logistics Support AgencyLORA Level of Repair Analysis

LOS-F-H Line-of-Sight Forward, HeavyLRIP Low Rate Initial ProductionLSA Logistics Support Analysis

LSAR Logistics Support Analysis RecordsM&S Modeling and Simulation

M/ETC MAGTF/Expeditionary Training CenterMAA Mission Area Assessment

MAGTF Marine Air-Ground Task ForceMAP Mission Area Plan

MASS Mobility Analysis Support SystemMCCDC Marine Corps Combat Development Command

MCMSMO Marine Corps Modeling & Simulation Management Office

H-4

MCMSWG Marine Corps Modeling & Simulation Working GroupMDA Milestone Decision Authority

MDAP Major Defense Acquisition ProgramMIAC Metals Information Analysis Center

MICOM U.S. Army Missile CommandMMC Metal Matrix Composites

MMCIAC Metals Matrix Composites Information Analysis CenterMNA Mission Need AnalysisMNS Mission Need StatementMOE Measures of EffectivenessMOO Measures of OutcomeMOP Measures of Performance

MOP 77 Memorandum of Policy No. 77MOSAIC MOdels & Simulations: Army Integrated Catalog

MOU Memorandum of UnderstandingMRS-BURU Mobility Requirements Study-Bottom-Up Review Update

MS Milestone (as in MS 0, MS I, etc.)MS&A Modeling & Simulation and Analysis

MSIS Modeling and Simulation Information SystemMTIAC Manufacturing Technology Information Analysis CenterMTWS Marine Air-Ground Task Force (MAGTF) Tactical Warfare

SimulationNAVAIR Naval Air Systems Command

NDAA Non-Developmental Airlift AircraftNDI Non-Developmental Item

NLOS Non Line-of-SightNTIAC Nondestructive Testing Information Analysis Center

O&S Operations and SupportOAS Office of Aerospace Studies (Air Force Materiel Command)OFP Operational Flight Program

OPTEC U.S. Army Operational Test and Evaluation CommandORD Operational Requirements DocumentOSD Office of the Secretary of Defense

OT Operational TestOT&E Operational Test and EvaluationPA&E Program Analysis and EvaluationPDM Periodic Depot Maintenance

PE Program ElementPEO Program Executive Office

PLASTEC Plastics Technical Evaluation CenterPM Program Manager

PMD Program for Management DevelopmentPMOs Program Management Offices

PMWS Program Manager’s Work StationPOC Point of Contact

H-5

POM Program Objective MemorandumPPBS Planning, Programming & Budgeting System

PRIMES Preflight Integration of Munitions and Electronic SystemsR&D Research and DevelopmentRAC Reliability Analysis Center

RAM Reliability, Availability, and MaintainabilityRCM Requirements Correlation Matrix

RD&A Research, Development and AcquisitionRDEC Research, Development and Engineering Center

RFP Request for ProposalS&A Studies and AnalysisSAB Air Force Scientific Advisory Board

SAFMA Strategic Airlift Force MixSAIC Science Applications International Corporation

SEMP Systems Engineering Management PlanSIDAC Supportability Investment Decision Information Analysis Center

SIMNET Simulation NetworkSIMTECH Simulation Technology Program

SIMWG Simulation Working GroupSMIAC Soil Mechanics Information and Analysis Center

SOF Special Operation ForcesSOW Statement of Work

SPAWAR Space and Naval Warfare Systems CommandSSP Simulation Support Plan

SSTORM Structured Scenario Torpedo Operational requirements ModelSTAF Simulation/Test Acceptance Facility

STRICOM Simulation, Training and Instrumentation CommandSURVIAC Survivability/Vulnerability Information Analysis CenterSYSCOM Systems Command

T&E Test and EvaluationT&E Training and Education Division, Marine Corps Combat

Development CommandTACOM U.S. Army Tank-Automotive Command

TARDEC U.S. Army Tank Automotive and Armament Research, Development& Engineering Ctr

TECOM U.S. Army Test and Evaluation CommandTEMP Test and Evaluation Master Plan

TLD Top Level DemonstrationsTPFDD Time-Phased Force and Deployment DataTPIPT Technical Planning Integrated Product TeamTRAC TRADOC Analysis Command

TRADOC U.S. Army Training and Doctrine CommandTRIMS Technical Risk Identification and Mitigation System

TWSTIAC Tactical Warfare Simulation and Technology Information andAnalysis Center

H-6

USAF/CVA Assistant Vice Chief of Staff, U.S. Air ForceUSAF/XOM U.S. Air Force Directorate of Modeling , Simulation and AnalysisUSD(A&T) Under Secretary of Defense for Acquisition and Technology

USD(AR) Under Secretary of Defense for Acquisition ReformV&V Verification and Validation

VV&A Verification, Validation and AccreditationWBS Work Breakdown Structure

WEPTAC Weapons and Tactics Center

I-1

APPENDIX IGLOSSARY

Acceptability criteria: A set of standards that a particular M&S must meet to be accred-ited for a given use. (DA PAM 5-11)

Accreditation: The official certification that a model or simulation is acceptable for usefor a specific purpose. (DoDD 5000.59)

Ada: A high order computer language designed and developed to DoD requirements formodular standard language. While the original focus was for real-time embedded soft-ware, Ada has also been used for a variety of other software systems including some simu-lation systems.

Advanced Distributed Simulation (ADS): A concept which applies a common core ofadvanced technologies (including; computer, display, communication and simulation) toprovide a mix of live, constructive and virtual simulation methods across the spectrum ofDefense uses, from training and readiness through requirements generation throughprototyping through fielding. ADS and DIS are synonymous. (DSB)

Aggregate Level Simulation Protocol (ALSP): A protocol that permits the integration ofdistributed simulations (or Actors). The protocol is currently being used to integrate AWSIMand CBS. The protocol synchronizes the advancement of simulation time among the simu-lations, provides mechanisms for interaction among combat entities (e.g., direct or indi-rect fire engagements) across simulations, and the update of state attributes of those com-bat entities.

Air Warfare Simulation Model (AWSIM): The simulation currently used at the WarriorPreparation Center and Blue Flag to conduct battle staff training.

Algorithm: A prescribed set of well-defined, unambiguous rules or processes for the so-lution of a problem in a finite number of steps. (Webster Computer)

Appended Trainers: Combat Vehicle Appended Trainers are a family of deployable train-ers designed to support individual and full-crew mission training for tanks, armored ve-hicles and assault vehicles. These trainers include the visual and aural cues which immersethe operator in a near-real battle environment, and have provisions for recording events forpost-exercise crew performance analysis.

Architecture: The high-level organization of hardware or software systems. (Krueger)

Artificial Intelligence: The effort to automate those human skills that illustrate our intel-ligence (e.g., understanding visual images, understanding speech and written text, prob-

I-2

lem solving and medical diagnosis). (Krueger)

Brawler: TAC Brawler is an engagement level air-to-air combat simulation tool used pri-marily for the evaluation of avionics, weapons and tactics. It has also been used as a targetgenerator for manned simulators. (J-8 Catalog).

C-Plus-Plus (C++): A high order computer language used extensively in commercial soft-ware. C++ is an object oriented extension to the C language.

Classes of Simulation:(DSB)

Live - The live component of simulation involves operations with real forces and realequipment in the air, on the ground, on and below the sea.

Constructive - A class of simulation typified by wargames, models and analytical tools.

Virtual - A class of simulation where systems are simulated both physically and elec-tronically.

Code Verification: A rigorous audit of all compilable code to ensure that the representa-tions of vertical logic have been properly implemented in the computer code. (DA PAM 5-11)

Combined Arms and Support Task force Evaluation Model (CASTFOREM): This isthe Army’s highest resolution, lowest echelon (up through and including brigade) com-bined arms combat simulation model. Often run in conjunction with JANUS(T) to per-form weapon systems analysis for a COEA (MOSAIC).

Common-use M&S: M&S applications, services or materials provided by a DoD compo-nent to two or more DoD components. (DoDD 5000.59)

Compatibility: The capability of a functional unit to meet the requirements of a specifiedinterface. (ANSI X3.172-1990)

Computer Generated Forces (CGF): A collection of unmanned battlefield entities undercontrol as a unit. CGF replace or supplement friendly, enemy or neutral manned simula-tors during a specific session. The SIMNET program uses the term “semi-automated forces”(SAFOR) for CGF. (DIS Glossary)

Computer War Game: A technique by which different concepts, different pieces of hard-ware or different military plans can be investigated in a multi-sided confrontation using acomputer to generate displays of the battlefield and perform computations of outcomes.

Concepts Evaluation Model (CEM): CEM is used to analyze force effectiveness at the-ater level warfare. It is used as a tool to assess the effectiveness of different mixes of forces

I-3

or resources and estimates of ammunition, equipment and personnel requirements. (MO-SAIC)

Concurrent Engineering: Concurrent engineering is a systematic approach to the inte-grated, concurrent design of products and their related processes, including manufactureand support. This approach is intended from the outset to cause developers to consider allelements of the product life cycle from conception through disposal, including quality,cost, schedule and user requirements. (IDA)

Confederation of Models: A set of simulation methods (or Actors) operating in an inte-grated manner using the ALSP protocols.

Continuous System: A system for which the state variables change continuously withrespect to time. (Law and Kelton)

Cooperative Development: A project in which two or more DoD components share indomain research, technical studies or technology development, but that may result in dis-similar M&S applications. (DoDD 5000.59)

Corps Battle Simulation (CBS): A simulation used by the Army simulation centers totrain battle staffs at Corps and echelons below. Previously known as the Joint ExerciseSimulation System (JESS).

Cross-functional Integration: The melding of acquisition functions (such as design analy-sis with logistics analysis) involving shared modeling and simulations data and informa-tion. (ATFM&S)

Data base Management System: A set of computer programs that provides convenientand efficient means to retrieve and store data in a database.

Data base: A collection of data.

Defense Simulation Internet (DSI): A communication network under development byARPA that provides secure, packet-switched, data, voice and video services. Current sys-tem provides communication links at 1.5 megabits per second rates. DSI should not beconfused with DIS: DSI may be used as a communication network over which DIS simu-lators are linked. However, DSI is not limited to supporting the linking of DIS simulators,nor are DIS simulators limited to using the DSI as a communication network.

Deterministic Algorithm: A process that yields a unique and predictable outcome for agiven set of inputs. (Harris)

Discrete System: A system for which the state variables change instantaneously at sepa-rated points in time. (Law and Kelton)

I-4

Distributed Interactive Simulation (DIS): 1. A time and space coherent representationof a virtual battlefield environment, measured in terms of the human perception and thebehaviors of warfighters interacting in free play with other warfighters and/or with com-puter generated forces. DIS provides a structure by which independently developed sys-tems may interact with each other in a well managed and validated combat simulationenvironment during all phases of the development process and in subsequent training. DISand ADS are synonymous. 2. The class of simulations defined by the DIS Architecture andassociated standards. (DIS Glossary)

DoD Modeling and Simulation (M&S) Executive Agent: A DoD component to whomthe USD(A&T) has assigned responsibility and delegated authority for the developmentand maintenance of a specific area of M&S application, including relevant standards anddatabases, used by or common to many models and simulations. (DoDD 5000.59)

Eagle: Corps/Division level combat model that simulates the operations level of war andincludes joint and combined operations. It is used for assessments, combat development,as an exercise driver and as a staff trainer. (MOSAIC)

Enhanced Naval Warfare Gaming System (ENWGS): A multi-platform, multi-war-fare, real-time wargaming and training simulation used for battle staff training and strate-gic and operational wargaming and planning. (Navy M&S Catalog)

Enhanced Surface-to-Air Missile Simulation (ESAMS): Generates one-on-one probabilityof kill for aircraft versus surface-to-air missiles. Its results are used in higher level survivabilityanalyses to evaluate weapon system and subsystems effectiveness. (AFSAA Catalog)

Entity: An object by which the system can be defined (i.e., a component of the systemrepresented in the model). (McQuay)

Executive Council for Modeling and Simulation (EXCIMS): An organization estab-lished by the Under Secretary of Defense (Acquisition) (USD(A)) (now USD(A&T)) andresponsible for providing advice and assistance on DoD modeling and simulation issues.Membership is determined by the USD(A&T) and is at the Senior Executive Service, flagand general officer level. Chaired by the DDR&E. (DoD 5000.59)

Extended Air Defense Simulation (EADSIM): EADSIM is used primarily to analyzeextended air-defense scenarios. It is used to evaluate effectiveness and efficiency of weaponsystems against targets and to evaluate the value of different mixes of forces or resources.(J-8 Catalog)

Fidelity: The degree to which aspects of the real world are represented in the M&S. (See“Resolution”)(DA PAM 5-11)

General-use M&S Applications: Specific representations of the physical environmenteffects used by, or common to, many models and simulations; e.g. terrain, atmospheric or

I-5

hydrographic effects. (DoDD 5000.59)

Hardware/Software-in-the-Loop: This hybrid simulation includes actual system or sub-system hardware and software in conjunction with mathematical (computer) models andexternal stimuli to demonstrate the capability to operate within an environment simulatingactual operating conditions.

Hierarchy: A hierarchy of models and simulations is a taxonomy which is used to de-scribe the various levels of models and simulations. Assorted taxonomies may be found inthe literature; this guidebook describes four levels of models and simulations: engineer-ing, engagement, mission/battle and theater/campaign.

Hybrid Simulation: A simulation that combines multiple classes of simulations, such ascombining computer (constructive) simulations with actual system hardware and software(live). An example is a hardware/software-in-the-loop simulation.

Integrated Product and Process Development (IPPD): IPPD is an approach to systemsacquisition which brings together all of the functional disciplines required to develop, design,test, produce and field a system. This is essentially the same as Concurrent Engineering.

Integrated Product Team (IPT): Integrated Product Teams are a means to achieve concur-rent engineering or IPPD. They are multidisciplinary teams consisting of representatives fromall disciplines involved in the system acquisition process, from requirements developmentthrough disposal. Having the participation of all the appropriate disciplines, IPTs are oftenempowered to make decisions to achieve successful development of their particular product.

Interface: The interconnection between two pieces of hardware or software. A device orpiece of software that accomplishes such a connection. (Krueger)

Interactive Models: Models that require human participation are sometimes called inter-active or human-in-the-loop. Human participation can include decision making withincomputer wargaming models for tactics development and battle staff training as well ashuman-in-the-loop weapon system simulators and trainers.

Interoperability: The capability of two or more systems to exchange and use informa-tion. (ANSI X3.172-1990)

JANUS: JANUS is a multi-purpose near-real-time interactive wargame used to examinethe relationships of combat and tactical processes. IT is used for weapon system perfor-mance, test planning, test augmentation, scenario evaluation and exercises. It can modelentities down to the individual soldier or system. (MOSAIC)

Joint M&S: Modeling and simulation representations of joint and Service forces, capa-bilities, materials and services; used in the joint environment of by two or more Services.(DoDD 5000.59)

I-6

Legacy Model: A model developed in the past which is still in use that was not imple-mented using today’s standards (e.g., software, communication, DIS, ALSP, etc.). Somelegacy models have been modified with interfaces to some of the current standards extend-ing their usefulness and interoperability with newer, standards based models.

Marine Air Ground Task Force (MAGTF) Tactical Warfare Simulation (MTWS):MTWS is a computer-assisted exercise support tool designed to support training of Ma-rine Corps commanders and their staffs. It is designed to be used in Command Post Exer-cises in which combat forces, supporting arms, and combat results are modeled by thesystem. It will also be used in field exercises in which all, or part, of the combat forces areactual military units. MTWS provides a full spectrum of combat models required to sup-port Marine Corps exercises.

Measure of Effectiveness (MOE): A quantitative expression which compares the effec-tiveness of alternatives in meeting an operational objective or need. (DA PAM 5-11)

Measure of Performance (MOP): A defined metric of a component which contributes tobasic system effectiveness as described by an MOE. MOPs relate to specific performancecharacteristics from which data can be actually collected. (DA PAM 5-11)

Measures of Outcome (MOO): Metrics that define how operational requirements con-tribute to end results at higher levels, such as campaign or national strategic outcomes.

Model: A physical, mathematical or otherwise logical representation of a system entity,phenomenon or process. (DoDD 5000.59)

Model and Simulation (M&S) Interoperability: The ability of a model or simulation toprovide services (data and functionality) to and accept services from other models andsimulations; and to use the services so exchanged to enable them to operate effectivelytogether. (DoDD 5000.59)

Modeling and Simulation (M&S) Investment Plan: A DoD plan, published under theauthority of the Under Secretary of Defense (Acquisition & Technology) and with thecoordination of the DoD components, that established short-term (present to six years)and long-term (beyond six years) programs and funding for joint and common use M&Sto achieve the specified goals and objectives outlined in the DoD M&S Master Plan. (DoDD5000.59)

Model-Test-Model: An integrated approach to using models and simulations in supportof: pre-test analysis and planning; conducting the actual test and collecting data; and post-test analysis of test results along with further validation of the models using the test data.

Monte-Carlo Algorithm: A statistical procedure that determines the occurrence of proba-bilistic events or values of probabilistic variables for deterministic models, i.e., make arandom draw. (Neelamkavil)

I-7

Object-Oriented Language: A language which best suits an object-oriented decomposi-tion of software and which provides the capability to implement classes and objects. Di-rectly supports data abstraction and classes, and provides additional support for inherit-ance as a means of expressing hierarchies of classes. (Booch)

Object-Oriented Programming: A method of implementation in which programs areorganized as cooperative collections of objects, each of which represents an instance ofsome class, and whose classes are all members of a hierarchy of classes united via inherit-ance relationships. (Booch)

Object: Physical or logical structures (models) that keep their characteristics and behav-ior together.

Open System: A system in which the components and their composition are specified in anon-proprietary environment, enabling competing organizations to use these standard com-ponents to build competitive systems. There are three perspectives on open systems: port-ability-the degree to which a system component can be used in various environments,interoperability-the ability of individual components to exchange information, and inte-gration-the consistency of the various human-machine interfaces between an individualand all hardware and software in the system. (Nutt)

Parallel Processing: Multiple processes running on multiple processors simultaneously.

Protocol: A set of rules used to control/regulate the interaction between entities in a sys-tem (e.g., computers communicating on a network). Often implemented as hierarchy of“layers” in which each level provides a defined set of services to the layer above.

Real-Time System: A system that computes its results as quickly as they are needed by areal-world system. Such a system responds quickly enough that there is no perceptibledelay to the human observer. In general use, the term is often perverted to mean within thepatience and tolerance of a human user. (Krueger)

Resolution: The degree of detail and precision used in representation of real world aspectsin the M&S. (See “Fidelity) (DA PAM 5-11)

Requirements Correlation Matrix (RCM): A prioritized list of system requirementswith associated performance thresholds and goals.

Semi-Automated Forces (SAFOR): Simulation of friendly, enemy and neutral platformson the virtual battlefield in which the individual platform simulation are operated by com-puter simulation of the platform crew and command hierarchy. The term “semi-automated”implies that the automation is controlled and monitored by a human who injects com-mand-level decision making into the automated command process. For the purpose of theDIS architecture, the term Computer Generated Forces (CGF) replaces SAFOR. (DIS Glos-sary)

I-8

Scenario: The entire spectrum of environmental considerations that have interaction withsystems(s) under analysis or those of interest for training purposes. The spectrum includesphysical environment, threat conditions, rules of engagements, and systems performanceand effectiveness.

SIMNET: ARPA/Army simulation network where simulators can be connected over localand wide-area networks to create a simulated battlefield. SIMNET was the forerunner tothe DIS architecture.

Simulation: A method for implementing a model over time. Also a technique for testing,analysis or training in which real-world systems are used, or where real-world an concep-tual systems are reproduced by a model. (DoDD 5000.59)

State Variables: The collection of variables necessary to describe a system at a particulartime, relative to the objectives of the study. (Law and Kelton)

Stimulation: Stimulation is the use of simulations to provide an external stimulus to asystem or subsystem. An example is the use of a simulation representing the radar returnfrom a target to drive (stimulate) the radar of a missile system within a hardware/software-in-the-loop simulation.

Stochastic Process: Any process dealing with events that develop in time or cannot bedescribed precisely, except in terms of probability theory. (Webster Computer)

Stochastic: Probabilistic or random, as opposed to deterministic.

Structured Scenario Torpedo Operational Requirements Model (SSTORM): SSTORMis a high fidelity Monte-Carol torpedo simulation capable of modeling the performance ofU. S. and threat weapons. It includes submarine and torpedo kinematics as well as coun-termeasures. (Navy M&S Catalog)

Synthetic Environments: Representations of present or future, factory-to-battlefield, en-vironments generated by models, simulations and wargames. May include a mix of realand simulated objects accessible from widely dispersed locations. One of the Science andTechnology Thrust areas.

Thunder: Thunder is a two-sided theater level model designed to simulate conventional,air-land combat. It determines contribution of weapon systems to the combat outcomes forkey operational objectives. (AFSAA Catalog)

Virtual Prototype: A computer-based simulation of systems and subsystems which ex-hibits both geometric and functional realism. This three-dimensional virtual mockup maybe used to evaluate prototypes or concepts and provides a common platform from whichall functional disciplines (design, test, manufacturing, logistics, training, operations, etc.)can work.

I-9

Virtual Reality: 1. Also referred to as artificial reality or synthetic reality. Virtual realityperceives a participant’s action in terms of the body’s relationship to a graphic world andgenerates responses that maintain the illusion that his actions are taking place within thatworld. (Krueger) 2. The application of integrated technologies to enable a participant tosense that he or she is occupying, to some degree, an environment other than which he orshe physically occupies. (Loftin)

Validation: The process of determining (a) the manner and degree to which a model is anaccurate representation of the real-world from the perspective of the intended uses of themodel, and (b) the confidence that should be placed on this assessment. (DoDD 5000.59)

Verification: The process of determining that a model implementation accurately repre-sents the developer’s conceptual description and specifications. (DoDD 5000.59)

Weapons and Tactics Center (WEPTAC): WEPTAC is an interactive wargaming modelused for many-on-may, multi-platform scenarios. It is used for development of fleet tac-tics, as well as evaluating the military worth of future weapon systems. (Navy M&S Catalog)

REFERENCES

DA PAM 5-11: Department of the Army Pamphlet 5-11, Verification, Validation and Ac-creditation of Army Models and Simulations, Headquarters, Department of the Army,Washington, DC, October 15, 1993.

DoDD 5000.59: Department of Defense Directive 5000.59, DoD Modeling and Simula-tion (M&S) Management, January 4, 1994.

DSB: Office of the Under Secretary of Defense for Acquisition. (1993). Impact of Ad-vanced Distributed Simulation on Readiness, Training and Prototyping. Washington,DC: Report of the Defense Science Board Task Force on Simulation, Readiness, andPrototyping.

Webster Computer: Webster’s New world Dictionary of Computer Terms, 3rd Ed., Simon& Schuster, 1988.

Krueger, M. W. Artificial Reality II (2nd Ed.), Addison-Wesley, Reading, Massachusetts,1990.

Distributed Interactive Simulation (DIS) Glossary Update, Rev. 2.0, October 2, 1992.

ANSI x3.172-1990: American National Standard for Information Systems - Dictionary forInformation Systems. New York, New York: American National Standards Institute (1991).

I-10

Law, A.M. & Kelton, W. D. Simulation Modeling & Analysis. (2nd Ed.) New York: McGraw-Hill, Inc. (1991).

Harris, G., Maj. “Computer Models, Laboratory Simulators, Test Ranges Meeting theChallenge of Estimating Tactical Force Effectiveness in the 1980’s”, 1979.

McQuay, W. K. “Computer Simulation Methods for Military Operations Research”, AFALTR 73-341, 1973.

IDA: IDA Report R-338, “The Role of Concurrent Engineering in Weapons System Ac-quisition”, Institute for Defense Analysis, December 1988.

Neelamkavil, F. Computer Simulation and Modeling. John Wiley & Sons, 1988.

Booch, Object-Oriented Design with Applications

Nutt, G.J., Open Systems Prentice-Hall, 1992.

Loftin, R.B., “Virtual Environments for Astronaut Training” Synthetic Environments Con-ference, March 30, 1993.

Catalog of Wargaming and Military Simulation Models (1992), (12th Ed.). Washington,DC: Force Structure, Resource, and Assessment Directorate, Technical Support andOperations Division (J-8/TSOD), The Joint Staff.

Navy Modeling and Simulation Catalog, October 1993.

AFSAA Catalog: Air Force Studies and Analysis Catalog.

MOSAIC: MOdels and Simulations: Army Integrated Catalog.

ATFM&S: Acquisition Task Force on Modeling and Simulation, Final Report, June 1994.

Date post:	03-Oct-2014
Category:	Documents
Upload:	rudwhy
View:	103 times
Download:	3 times