+ All Categories
Home > Documents > 942134 Tdpm:Ane2004.lpc 6/12/09 3:21 PM Paed 7 A Framework ... · 942134_Tdpm:Ane2004.lpc 6/12/09...

942134 Tdpm:Ane2004.lpc 6/12/09 3:21 PM Paed 7 A Framework ... · 942134_Tdpm:Ane2004.lpc 6/12/09...

Date post: 24-Nov-2018
Category:
Upload: buidieu
View: 214 times
Download: 0 times
Share this document with a friend
7
July/August 2009 www.stsc.hill.af.mil 7 T he level of complexity needed to develop spacecraft systems and other emerging technologies require programs to develop risk management and risk plan- ning techniques that can potentially identi- fy schedule and cost risks as early as pos- sible during the acquisition life cycle. According to the Government Accountability Office (GAO), studies have shown that there has been an increase in schedule and cost overruns that involve complex systems and emerg- ing technologies. This is often the case when projects exceed scheduled activity durations, resulting in frequent budget overruns. There are a plethora of risks that factor into inaccurate schedule esti- mates, including the elusive emerging requirements to the lack of process under- standing. Unfortunately, it is common to observe how requirements established during the earlier stages of an acquisition life cycle are changed to accommodate customer requests, thus impacting sched- ule and delivery costs. These impacts invariably affect scheduled activities from the design through the development and production of the complex system. To improve implementation and the under- standing of the life-cycle processes for the complex system, it is essential that the development of a work breakdown struc- ture (WBS), or an architecture and its interfaces within the appropriate hierar- chical levels of decomposition, be accu- rately structured. The structure of this architecture coincides with the development and implementation of activities that are required for the design, development, and production stages of the life cycle. As a result, the developed schedule activities influence the cost of delivery. Since sched- uled activities impact the cost to develop complex systems, it can be shown that there is an inherent relationship between the complex systems architecture and the process activities required for schedule and cost estimating. As a result, project schedule and cost estimation play an important role in driving key acquisition life-cycle decisions. Developing and delivering complex systems requires the management of com- plex risks such as uncertainty usage, schedule uncertainties, uncertainties asso- ciated with technology maturity, manufac- turing maturity, technical design, and tech- nical complexity [1]. Acquisition life-cycle decisions can be potentially flawed if the systems engineering development model isn’t appropriately matched to the com- plex system being developed. To address the challenge of selecting a candidate sys- tems engineering development model, the Committee on Human-System Design Support for Changing Technology recom- mended implementation of the ICM as a reasonably robust framework for the “progressive reduction of risk through the full life cycle of system development, to produce a cost-effective system that meets the needs of all the stakeholders” [1]. The ICM integrates key strengths or attributes of other models into an integrated frame- work while introducing risk decision anchor points throughout the life cycle [2]. To complement implementation of this model, our research provides a hypotheti- cal example that incorporates maturity risk drivers—technology readiness levels (TRL) and manufacturing readiness levels (MRL)—within a notional ICM frame- work as an approach to assess schedule and cost risks during the development of a complex system. The method chosen to evaluate sched- ule and cost risk drivers for this research is known as Monte Carlo simulation. This method is used to model probabilistic behaviors of activities throughout the acquisition life cycle. The term Monte Carlo has been used interchangeably with prob- abilistic simulation since it is a technique used to randomly select numbers from a probability distribution or sampling. For our work, Monte Carlo simulation mod- eled the behaviors of schedule and cost uncertainties while providing traceability and consequence between the risk drivers and activities within the acquisition life cycle. Specifically, the simulation modeled the behaviors of tasks and activities derived from the spacecraft systems (the WBS) after mapping various risk drivers to those tasks [3]. This allows for exploration of the likelihood and consequences that these risk drivers have scheduled task activities throughout the acquisition life cycle. The ICM Framework The ICM was developed to ensure the flexibility of implementing one or more frameworks throughout each stage of an acquisition life cycle. The model was built upon five key principles that are crit- ical for system development: 1) customer satisfaction; 2) incremental growth of system definition and stakeholder com- mitment; 3) iterative system definition and development; 4) concurrent system definition and development; and 5) man- agement and project risk [2]. These prin- ciples are also proven strengths of other models such as the Waterfall, iterative, Rational Unified Process (RUP), and spi- ral development frameworks. The ICM is unique in that it merges these key principles into one framework [2] to provide a process model that is robust with a central focus to progressively reduce potential risks throughout the entire life cycle, culminating in the development of A Framework for Systems Engineering Development of Complex Systems In developing complex systems, evaluating potential schedule and cost risks is essential. With the Incremental Commitment Model (ICM), schedule tasks can be evaluated against manufacturing and technology risk drivers. In this article, these risk drivers are analyzed using a comprehensive approach with emphasis being placed on quantitative risk analysis through Monte Carlo simulation. Through modeling the behavior of a hypothetical project schedule for a notional spacecraft system, the authors show how the ICM framework is implemented in complex system development. The result is a repeatable, inherent, risk-driven commitment process that can stabilize and synchronize systems engineering and acquisition processes. Dr. Karl L. Brunson, Dr. Jeffrey Beach, Dr. Thomas A. Mazzuchi, and Dr. Shahram Sarkani George Washington University There are a plethora of risks that factor into accurate schedule estimates ...
Transcript

July/August 2009 www.stsc.hill.af.mil 7

The level of complexity needed todevelop spacecraft systems and other

emerging technologies require programsto develop risk management and risk plan-ning techniques that can potentially identi-fy schedule and cost risks as early as pos-sible during the acquisition life cycle.According to the GovernmentAccountability Office (GAO), studieshave shown that there has been anincrease in schedule and cost overrunsthat involve complex systems and emerg-ing technologies. This is often the casewhen projects exceed scheduled activitydurations, resulting in frequent budgetoverruns. There are a plethora of risksthat factor into inaccurate schedule esti-mates, including the elusive emergingrequirements to the lack of process under-standing.

Unfortunately, it is common toobserve how requirements establishedduring the earlier stages of an acquisitionlife cycle are changed to accommodatecustomer requests, thus impacting sched-ule and delivery costs. These impactsinvariably affect scheduled activities fromthe design through the development andproduction of the complex system. Toimprove implementation and the under-standing of the life-cycle processes for thecomplex system, it is essential that thedevelopment of a work breakdown struc-ture (WBS), or an architecture and itsinterfaces within the appropriate hierar-chical levels of decomposition, be accu-rately structured.

The structure of this architecturecoincides with the development andimplementation of activities that arerequired for the design, development, andproduction stages of the life cycle. As aresult, the developed schedule activitiesinfluence the cost of delivery. Since sched-uled activities impact the cost to developcomplex systems, it can be shown thatthere is an inherent relationship betweenthe complex systems architecture and theprocess activities required for schedule

and cost estimating. As a result, projectschedule and cost estimation play animportant role in driving key acquisitionlife-cycle decisions.

Developing and delivering complexsystems requires the management of com-plex risks such as uncertainty usage,schedule uncertainties, uncertainties asso-ciated with technology maturity, manufac-turing maturity, technical design, and tech-nical complexity [1]. Acquisition life-cycledecisions can be potentially flawed if the

systems engineering development modelisn’t appropriately matched to the com-plex system being developed. To addressthe challenge of selecting a candidate sys-tems engineering development model, theCommittee on Human-System DesignSupport for Changing Technology recom-mended implementation of the ICM as areasonably robust framework for the“progressive reduction of risk through thefull life cycle of system development, toproduce a cost-effective system that meetsthe needs of all the stakeholders” [1]. TheICM integrates key strengths or attributesof other models into an integrated frame-work while introducing risk decisionanchor points throughout the life cycle [2].To complement implementation of thismodel, our research provides a hypotheti-cal example that incorporates maturity riskdrivers—technology readiness levels(TRL) and manufacturing readiness levels(MRL)—within a notional ICM frame-work as an approach to assess scheduleand cost risks during the development of

a complex system. The method chosen to evaluate sched-

ule and cost risk drivers for this research isknown as Monte Carlo simulation. Thismethod is used to model probabilisticbehaviors of activities throughout theacquisition life cycle. The term Monte Carlohas been used interchangeably with prob-abilistic simulation since it is a techniqueused to randomly select numbers from aprobability distribution or sampling. Forour work, Monte Carlo simulation mod-eled the behaviors of schedule and costuncertainties while providing traceabilityand consequence between the risk driversand activities within the acquisition lifecycle. Specifically, the simulation modeledthe behaviors of tasks and activitiesderived from the spacecraft systems (theWBS) after mapping various risk drivers tothose tasks [3]. This allows for explorationof the likelihood and consequences thatthese risk drivers have scheduled taskactivities throughout the acquisition lifecycle.

The ICM FrameworkThe ICM was developed to ensure theflexibility of implementing one or moreframeworks throughout each stage of anacquisition life cycle. The model wasbuilt upon five key principles that are crit-ical for system development: 1) customersatisfaction; 2) incremental growth ofsystem definition and stakeholder com-mitment; 3) iterative system definitionand development; 4) concurrent systemdefinition and development; and 5) man-agement and project risk [2]. These prin-ciples are also proven strengths of othermodels such as the Waterfall, iterative,Rational Unified Process (RUP), and spi-ral development frameworks.

The ICM is unique in that it mergesthese key principles into one framework [2]to provide a process model that is robustwith a central focus to progressively reducepotential risks throughout the entire lifecycle, culminating in the development of

A Framework for Systems EngineeringDevelopment of Complex Systems

In developing complex systems, evaluating potential schedule and cost risks is essential. With the Incremental Commitment

Model (ICM), schedule tasks can be evaluated against manufacturing and technology risk drivers. In this article, these risk

drivers are analyzed using a comprehensive approach with emphasis being placed on quantitative risk analysis through Monte

Carlo simulation. Through modeling the behavior of a hypothetical project schedule for a notional spacecraft system, the

authors show how the ICM framework is implemented in complex system development. The result is a repeatable, inherent,

risk-driven commitment process that can stabilize and synchronize systems engineering and acquisition processes.

Dr. Karl L. Brunson, Dr. Jeffrey Beach, Dr. Thomas A. Mazzuchi, and Dr. Shahram SarkaniGeorge Washington University

“There are a plethoraof risks that factor into

accurate scheduleestimates ...”

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 7

Process Replication

8 CrossTalk The Journal of Defense Software Engineering July/August 2009

complex systems that are both schedule-and cost-effective. The ICM frameworkalso provides the decision-maker with theflexibility to recognize potential risks thatcoincide with system’s maturity and com-plexity of scope. To accommodate theensuing consequences of potential risk dri-vers, the ICM framework implements a riskmanagement principle that associates risk-driven tasks and activities for each stage ofthe acquisition life cycle.

The construct of the ICM frameworkis comprised of two major stages. Stage I,the Incremental Definition, entails the ini-tial design stages of the system where theconceptual definition and feasibility stud-ies are conducted for a better understand-ing of the system and stakeholder com-mitment. Stage II, Incremental Develop-ment and Operations, is where the incre-ments of operational capability are devel-oped and integrated into schedules thatcorrelate to the development and evolu-

tion of the complex system. The activitieswithin each stage are risk-driven toaccount for process agility and rigor toensure that the system objectives are metthroughout the systems development lifecycle [2]. The concepts that the ICMframework is built upon include:• Early verification and validation con-

cepts of the V-model.• Concurrency concepts of the concur-

rent engineering model.• Concepts from Agile and Lean mod-

els.• Risk-driven concepts of the spiral

model.• Phases and anchor points of the RUP.• Systems of systems acquisition con-

cepts of the spiral model.Synergistic structuring of one or more

process models within the ICM frame-work provides the tailoring flexibility toaccommodate the varying maturity char-acteristics of any complex system; Barry

Boehm and Jo Ann Lane provide a moredetailed discussion regarding the ICMframework in [2]. An illustration of anintegrated DoD/ICM life-cycle frame-work is provided in Figure 1. This viewaligns milestones A, B, and C, representingthe designated commitment point of keystages. A more detailed discussion ofanchor points can be found in both [1]and [4].

Our work focused on how the ICMframework is implemented in the develop-ment of a complex system by modelingthe behavior of a hypothetical projectschedule of a notional spacecraft system[3]. The modeling technique, Monte Carlosimulation, will be used to help the deci-sion-maker evaluate schedule durationsand cost estimates that are impacted byrisk drivers. It will also aid the decision-maker with establishing preliminary riskmanagement assessments.

Spacecraft System WBS The WBS shown in Figure 2 illustrateshow the notional spacecraft system isdefined, developed, and maintainedthroughout the acquisition life cycle [4].The hierarchical breakdown of the space-craft is used to understand the productscontained within each level of decomposi-tion. The hierarchical levels that comprisethe products provide the work structurenecessary to develop tasks and activitiesthroughout the acquisition life cycle. Theproducts include hardware, software, doc-uments, and processes. However, the WBSin this case is only focused on the hard-ware of the spacecraft at a third level ofdecomposition. Each level of decomposi-tion was used to identify all of the subsys-tems and components that influence thepreceding level and are aggregated to thetop of the hierarchy through functionalrelationships.

Spacecraft System Project ScheduleThe spacecraft system’s project schedule

Figure 1: ICM and DoD Milestone Traceability

Figure 2: WBS of Notional Spacecraft System

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 8

A Framework for Systems Engineering Development of Complex Systems

July/August 2009 www.stsc.hill.af.mil 9

was used to provide key activity dates anddurations that are associated with theproducts of the WBS. The WBS sets thefoundation of all scheduled activities,thus impacting cost estimates. The dura-tion of the scheduled activities was dri-ven by many factors such as the technicalcomplexity of work to be performed,manufacturing availability of compo-nents to be developed, and the technicalmaturity of components to be assem-bled. These factors were risk drivers thathad an effect on the uncertainties of theproject schedule.

The project schedule illustrates twokey elements: the influence that risk dri-vers have on schedule tasks, and the influ-ence that tasks and activities have on eachother through precedence relationships.For example, let’s say that Task 1 must endbefore Task 2 can begin or Task 3 cannotbegin until Tasks 1 and 2 have ended,respectively [5]. Because of the sharedinterrelationships of the schedule’s tasks,it is inevitable that any overrun in sched-uled activities will most likely impact theduration of other tasks and activities ofthe project schedule—thus increasing thelikelihood of cost overrun. Table 1 illus-trates a sample project schedule for thedevelopment of the spacecraft systemimplementing the ICM framework.

Risk Management To evaluate the proposed ICM frameworkeffectively, it is important to organize a riskmanagement approach that ensures theidentification and quantification of risksand uncertainties that may impact a pro-ject’s schedule and cost [6]. Because of the

increasing complexity of the developmentof spacecraft systems, it is likely that a pro-ject’s schedule tasks are ultimately interre-lated and associated with cost [7]. In thecontext of system development programs,schedule and cost risks may determinewhether or not the program will completethe systems development on schedule andon budget. If the program successfullymeets or exceeds the schedule and budgetexpectations of the customer, it will likelybe due to the effective implementation ofrisk management processes [8].

The ICM model implements risk man-agement anchor points throughout eachstage of the life cycle in order to improvethe possibility of success for the develop-ment of the complex system [2]. The pro-ject schedule of the notional spacecraftsystem illustrated similar risk anchorpoints and was modeled with the appro-priate stage tasks and activities againstassociated risks. Cost and schedule risksco-exist because of inherent uncertaintiesregarding the time and costs required tocomplete tasks of a project’s schedule [9].To understand the uncertainties of sched-ule and cost risks, a risk mitigation strate-gy must be implemented to minimize theimpact of these risks.

The risk mitigation strategy to beimplemented with our research includesthe following steps:1. Risk Identification. Evaluation of a

risk’s probability of occurrence andthe impacts or consequence of risksagainst schedule and costs.

2. Risk Assessment. Quantification ofthe information acquired from riskidentification to assess project sched-

ule, cost, and technical risks.3. Risk Analysis. Quantification of risk

data in terms of probability of occur-rence and the eventual consequence(s)if a risk does occur.

4. Risk Mitigation. Determination ofactions to be implemented to reduceschedule and cost risks.These steps are supported by empirical

data that show how cost estimates areoften linked to activity durations viaschedule risk results. This is often seenwhen schedule risk analysis results areused as input to cost risk analysis and isprimarily implemented to identify theuncertainty in activity durations in orderto assess cost risks [3].

Risk Identification The risk identification process begins byevaluating key risks (independent vari-ables) and their respective uncertainimpacts, throughout the acquisition lifecycle, on the project’s schedule and costvariables. The risk identification process isimplemented by a team of experts whoevaluate project tasks and activities againstthe categorized risks that have varyingprobabilities of occurrences within eachstage of the life cycle. It should also benoted that the risk identification processcan be implemented with empirical datafor the complex system being developed.

Our research identified maturity risks(TRL and MRL) as technical risks to beevaluated against project events. A briefdescription for TRL and MRL ratings areprovided in Table 2 (see next page) [10].The risks were evaluated independentlyagainst tasks and activities that would

Table 1: Sample Spacecraft Project Schedule With Notional ICM Framework

942134_Text:Aug2004.qxd 6/12/09 7:53 PM Page 9

Process Replication

10 CrossTalk The Journal of Defense Software Engineering July/August 2009

impact subsystems and components ofthe spacecraft’s WBS. This approachallows a quick and simple explanation ofhow these risk drivers impact the finalschedule and cost results of the projectschedule.

The maturity risk drivers (TRL andMRL) are independent variables that carrya correlating metric weighting system thatestablishes the readiness to implementbased upon factors such as the stability,technical complexity, and maturity of aspacecraft’s systems and subsystems. Hadthe research been more detailed, theresults of the weighting metric systemwould be used to establish an entrance andexit criteria for identifying the mostappropriate framework or frameworkcombinations to be implemented through-out the life cycle. However, that step wasbypassed for this study; the given ICMframework was considered baseline for a

nominal matured spacecraft system. Thehypothetical impact for both TRL andMRL risk drivers are shown in Figure 3with a brief explanation. The exampleprovided for that figure summarizes theratings for each risk driver in a pre- andpost-mitigated format. However, thefocus of this research is to demonstratehow a system or subsystem may have dif-ferent maturity ratings throughout theWBS and illustrate the impacts on tasksand activity durations.

Figure 3 illustrates the process used toidentify risks that affect the projectschedule (generated using [5]). Each riskis evaluated against a task where they’rerated based upon the probability ofoccurrence and their severity effects onschedule and cost. This example demon-strates a pre-and-post mitigation evalua-tion—quantified after running the MonteCarlo simulation—to address the sched-

ule’s behavioral uncertainties. The firststep in the Monte Carlo simulation devel-opment required a qualitative risk assess-ment of the defined risk drivers. Thequalitative risk assessment is performedwith the use of the WBS and the identifi-cation of an expert familiar with the tasksand activities of the development andproduction phases. The input from theexpert will be used to establish the quali-tative risk assessment with the followingimportant steps:1. Develop a working list of risk drivers

that pose a threat to the project sched-ule, cost, or development performance.

2. Develop a risk ranking guide thatestablishes the probability of riskoccurrence. This study uses very low(VL), low (L), medium (M), high (H),very high (VH) probabilities of occur-rence for activities and tasks of theWBS affecting schedule and cost.

3. Identify the impacts or consequencesof the risk drivers by evaluating theprobability of occurrence and themagnitude of impact on the schedule,cost, and development performance.Establish qualitative descriptions toidentify the assessed risk drivers withthe use of a risk matrix.

4. Establish a final risk score for each riskdriver after completing steps 1-3, fol-lowed by populating the risk matrix toillustrate the magnitude of all identi-fied risks. Figure 3 illustrates how steps1-4 were implemented and ranked forMRL-1.

Risk AssessmentRisk assessment is the process of classify-ing risks into categories characterized bytheir frequency of occurrence and theseverity of their consequences. The riskassessment can be performed througheither qualitative or quantitative evalua-tions as well as through a comprehensiveevaluation combining both assessmenttypes. Qualitative risk assessment is con-sidered to be the process of prioritizingrisks based upon the risk’s probability ofoccurrence ranging from unlikely to mostlikely. The second aspect of the qualitativerisk assessment is when the risks are pri-oritized based upon the risk’s severity of con-sequence. Quantitative risk assessment isconsidered to be the process of prioritiz-ing risks using statistical techniques toestimate the project’s numerical outcome(schedule/cost behaviors) based uponidentified project risks through the use ofprobability distributions. Monte Carlosimulation is commonly used to modelproject behavior [11].

For our research, the quantitative risk

Table 2: TRL and MRL Rating Descriptions and Relationships

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 10

A Framework for Systems Engineering Development of Complex Systems

July/August 2009 www.stsc.hill.af.mil 11

assessment was based upon the projectschedule that was developed for the space-craft system. The goal was to understandthe levels of uncertainty inherent withintasks and activities of a project’s schedule.These uncertainties (task durations) wereevaluated by probability distributions viathree-point estimates and are acquiredeither through empirical data or expertjudgment. Duration values associated witheach task or activity risk are representedby subjective bounds least likely, most likely,and optimistic, and are analyzed after theMonte Carlo simulation is run.

Monte Carlo simulation is designed toiterate the project schedule’s tasks andactivities multiple times by randomlyselecting task or activity duration valuesfor each iteration from the probability dis-tribution type chosen. The outcome resultsof the simulation were then used to pro-vide the possible end dates of all tasks andactivities based upon the respective associ-ated risk drivers for the spacecraft’s projectschedule.

Risk Analysis and ResultsRisk analysis was conducted using MonteCarlo simulation to illustrate how the ICMframework can be implemented for thedevelopment of a spacecraft system [3].The benefit of using this technique is thatit generates schedule and cost estimates foruncertain input values through the use ofprobability distributions. It does this byrandomly generating values and iterativelymodeling the behavior of tasks and activi-ties of a project schedule.

The methodology was used to demon-strate how TRL and MRL risk drivers canbe mitigated within the ICM framework,and to understand and assess whether theproject schedule will meet the requiredcompletion date without budget overruns.In order to properly model the behavior

of the project schedule, the ICM frame-work was baselined with the DoD life-cycle stages [3]. Within each stage,planned tasks and activities were devel-oped for a spacecraft’s development andtraced against the TRL and MRL risk dri-vers through risk identification. An exam-ple of the mapping process, illustratingthe relationship of the risk drivers and thelife-cycle stages, is provided in Figure 4.

Once the risk drivers were mapped tothe appropriate activities, the uncertaintyof the risks were pre- and post-mitigatedthrough qualitative analysis with theoptions of using expert opinion andempirical data. Although our work washypothetical, the data collection methodincluded both expert opinion and empiri-cal data. The probability distribution cho-sen to be simulated throughout the lifecycle is frequently used to model expertopinion or empirical data. The distribu-tion used was adapted so that the expertcan provide three-point estimates thatrepresent pessimistic, most likely, andoptimistic inputs. The estimates corre-spond directly to the time estimates used

as input variables for the Monte Carlosimulation. Therefore, the triangular dis-tribution was chosen because it is the mostcommonly used distribution for modelingexpert opinion and empirical data. The tri-angular distribution is also used whenthere is very little information knownabout the parameters outside the approxi-mate estimate of its pessimistic, most like-ly, and optimistic variables. In addition,the uniform distribution was not chosenbecause it is known to be a very poormodeler of expert opinion and empiricaldata (since all of the values within itsrange have equal probability density).Thus, the density falls sharply to zero atthe pessimistic and optimistic endpointestimates.

The example (provided in Figure 3)identifies MRL-1 as the risk driverimposed upon the concept studies task ofthe Exploration and Valuation stages ofthe life cycle. The MRL-1 risk driver wasthen categorized and identified with amedium impact and low probability ofoccurrence. An illustration of the risk rat-ing is provided in the risk matrix of Figure

Figure 4: TRL and MRL Maturity Trace to DoD Acquisition Life Cycle

Figure 3: Risk Identification of MRL-1

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 11

Process Replication

12 CrossTalk The Journal of Defense Software Engineering July/August 2009

3, in addition to detailing pre- and post-mitigation definitions for that risk driver.The risk identification and mitigationprocess is a critical step in the event thatcontingency scenarios need to be formu-lated to successfully complete the project.

Now that all steps have been taken todevelop the spacecraft’s project model, thesimulation is run and the results are evalu-ated to be realistic or unrealistic. Afterrunning the pre-mitigated project sched-ule, the statistical completion dates areprovided in the project schedule resultswithout the imposed maturity risks shownin Figure 5. The results of the spacecraft’sproject schedule were then used to estab-lish the baseline model for the ICM acqui-sition life cycle. The next step, as outlinedin the Risk Identification section, requiresfactoring the risk drivers into the projectschedule’s tasks and activities, followed byrunning another simulation to determinethe final schedule outcome. The modeledbehavior of the risk-driven schedule isprovided in the project schedule with theimposed maturity risks of Figure 5.

Figure 5 also illustrates the spacecraft’scost result for non-risk- and risk-imposedsimulations. The non-risk-imposed proba-bility distributions for both schedule andcost outcome aided in the evaluation ofthe uncertainties of the maturity risk dri-vers that are incorporated into the risk-imposed results. Thus, it is clear that therisk-imposed simulation results haveincreased in cost when compared to thepre-mitigated results.

Risk Mitigation andConclusionsThe objective of this study was to evaluatethe implementation of the ICM frame-work within a DoD acquisition life cyclewhile imposing TRL and MRL risk dri-vers. This was accomplished by the imple-mentation of qualitative and quantitativestatistical techniques and the use ofMonte Carlo simulation to predict theprobability of meeting the program’s pro-jected schedule and cost estimates. Theproject schedule activities developed forthis study correlated to activities that com-prised a notional ICM framework. As aresult, the notional ICM framework estab-lished the baseline project schedule to beevaluated against technology and manu-facturing maturity throughout the systemdevelopment life cycle.

The evaluation was successfully per-formed using a step-by-step risk manage-ment process that was quantified throughsimulation. The triangular probability dis-tribution was chosen to be used through-out the Monte Carlo simulation. This dis-tribution type was chosen because three-point estimates (pessimistic, most likely,and optimistic) were used to representworkflow or activity durations of the lifecycle. It should be noted that although life-cycle-critical paths were not identified anddiscussed in this study, the Monte Carlosimulation generated random variables topredict activity durations from each criticalpath probability distribution; this was ulti-mately used to develop the overall proba-

bility distribution of the system’s projectschedule. Because of the project schedule’ssimulation, insight was given to the deci-sion-maker that revealed the consequencesof the imposed maturity risks against thepredicted schedule and cost estimates.

The project results contain qualitative(expert opinion) and quantitative (empiri-cal) data. It is assumed that the degree ofsubjectivity associated with an expert’sinput is consistent with the expected dataof the project schedule. However, if thesubjective inputs are inaccurate, the resultsof the simulation can be very sensitive andreflect inaccurate schedule and cost esti-mates. Therefore, since the risk scoreswere developed by experts with somedegree of subjectivity, it is important toconsider evaluating the credibility of thoseexperts in order to quantify their input.This evaluation can be performed with atechnique called the classical method [12].Although not used within our study, it is acredible approach to consider when vali-dating subjective inputs.

The results of the Monte Carlo simula-tion demonstrated that the technology andmanufacturing maturity risks influence theoverall schedule and cost to develop thecomplex system; this is explicitly shown inthe final schedule and cost results ofFigure 5. The pre-mitigated status of theproject schedule represents a simulationrun where the maturity risk drivers are notapplied to the activities of the projectschedule. The post-mitigated status of theproject schedule represents a simulationrun where the maturity risk drivers areapplied to the activities of the projectschedule and a risk-mitigation strategy isdeveloped but not implemented. This isconsistent with the schedule slip and thecost increase illustrated in Figure 5.However, a simulation run that imple-ments the risk-mitigated strategy shouldshow improvement in the final projectschedule and cost metrics. It should alsobe noted that the simulated results wereused to illustrate two important elements:1. There may be schedule and cost con-

sequences when applying maturity riskdrivers to the project schedule of theacquisition life cycle.

2. The analysis can provide a level ofconfidence in meeting the projectedschedule and cost estimates of theoverall project. As a result of the risk management

process developed for this study, decision-makers have a significant alternative miti-gation strategy that can be implemented inorder to minimize potential schedule andcost overruns.

For future research, this study estab-

Figure 5: Spacecraft Schedule and Cost Results of Non-Risk Imposed vs. Risk Imposed Simulations

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 12

A Framework for Systems Engineering Development of Complex Systems

July/August 2009 www.stsc.hill.af.mil 13

lishes a framework for evaluating theimpacts of maturity risks against scheduleand cost and produces results that quanti-fy a hypothetical baseline ICM framework.It also establishes the risk assessmentapproach to quantitatively evaluate andcompare the metrics of other life-cyclemodels, further identifying the strengthsand tailorability of the ICM framework.�

References1. Pew, Richard W., and Anne S. Mavor,

eds. Human-System Integration in theSystem Development Process. Wash-ington, D.C.: The National AcademiesPress, 2007.

2. Boehm, Barry, and Jo Ann Lane.“Using the Incremental CommitmentModel to Integrate System Acqui-sition, Systems Engineering, andSoftware Engineering.” CrossTalk

Oct. 2007.3. Brunson, Karl L., et al. “Toward a

Framework for Implementing SystemsEngineering Development for Com-plex Systems.” Proc. of the 23rd Inter-national Forum on COCOMO andSystems/Software Cost Modeling andICM Workshop 3. Los Angeles: Uni-versity of Southern California, 2008.

4. DoD. MIL-HDBK-881A. Washing-ton, D.C.: DoD, 2005.

5. Primavera Pertmaster Software. “Pert-master Tutorial.” Primavera Pertmas-ter, 2008.

6. van Dorp, Johan R., et al. “A RiskManagement Procedure for the Wash-ington State Ferries.” Risk Analysis21.1 (2001): 127-142.

7. Loureiro, Geilson, Paul G. Leaney, andMike Hodgson. “A Systems Engineer-ing Framework for IntegratedAutomotive Development.” SystemsEngineering 7.2 (2003): 153-167.

8. Robinette, G. Jeffrey, and Janet S.Marshall. “An Integrated Approach toRisk Management and Risk Assess-ment.” Incose Insight 4.1 (2001).

9. GAO. Space Acquisitions DoD Needsto Take More Action to AddressUnrealistic Initial Cost Estimates ofSpace Systems. Washington D.C.:GAO, 2006.

10. DoD. Technology Readiness Assess-ment Deskbook. Washington, D.C.:Deputy Under Secretary of Defensefor Science and Technology, 2005.

11. NASA. Exploration Systems RiskManagement Plan. Risk ManagementPlan. Washington, D.C.: NASA, 2007.

12. Bedford, Tim, and Roger M. Cooke.Probabilistic Risk Analysis: Founda-tions and Methods. New York:Cambridge University Press, 2006.

About the Authors

Thomas A. Mazzuchi,

D.Sc., earned his doctor-al and master’s degrees inoperations research atGeorge Washington Uni-versity, and a bachelor’s

degree in mathematics at GettysburgCollege. He is chair of the Departmentof Engineering Management andSystems Engineering at George Wash-ington University where he is also pro-fessor of both operations research andengineering management.

School of Engineeringand Applied ScienceThe George Washington UniversityDept. of Engineering Management and Systems Engineering1776 G ST NWSTE 101Washington, D.C. 20052Phone: (202) 994-7424Fax: (202) 994-0245E-mail: [email protected]

Shahram Sarkani, Ph.D.,

is faculty adviser andhead of George Wash-ington University’s (GWU)engineering managementand systems engineering

off-campus programs office. He hasserved as professor of engineering man-agement and systems engineering since1999, and is the founder and director(since 1993) of GWU’s Laboratory forInfrastructure Safety and Reliability.Sarkani received his doctorate in civilengineering from Rice University, andhis master’s and bachelor’s degrees incivil engineering from Louisiana StateUniversity.

School of Engineeringand Applied ScienceThe George Washington UniversityDept. of Engineering Management and Systems Engineering2600 Michelson DRSTE 750Irvine, CA 92612Phone: (949) 724-9695Fax: (949) 724-9694E-mail: [email protected]

Karl L. Brunson, Ph.D.,

is currently senior staffaerospace engineer atLockheed Martin wherehis professional experi-ence spans modeling and

simulation through systems engineeringand architecture development. Hereceived a master’s degree in aerospaceengineering with a specialization in spacesystems from the University ofMaryland, and a bachelor’s degree inaerospace engineering from MississippiState University. Brunson recently com-pleted his doctorate at George Wash-ington University in systems engineering.

Lockheed Martin Corporation13200 Woodland Park RDSTE 9066Herndon, VA 20171 Phone: (703) 984-7229Fax: (703) 984-7393E-mail: [email protected]

Jeffrey Beach, D.Sc., isthe head of the Struc-tures and CompositesDepartment at the Car-derock Division of theNaval Surface Warfare

Center, where he has worked since 1969.He received his bachelor’s and master’sdegrees in aerospace engineering fromthe University of Maryland, and receivedhis doctorate in engineering managementand systems engineering from GeorgeWashington University.

School of Engineeringand Applied ScienceThe George Washington UniversityDept. of Engineering Management and Systems Engineering20101 Academic WYSTE 227-AAshburn,VA 20147Phone: (703) 726-8260Fax: (703) 726-8251 E-mail: [email protected]

942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 13


Recommended