Date post: | 31-Mar-2015 |
Category: |
Documents |
Upload: | rene-haler |
View: | 214 times |
Download: | 1 times |
November 8, 2007
Conference Brief
RISC-IQ: Risk Integrated w / Schedule & Cost – Intelligent Quantification Applications for Systems Engineers
This document is confidential and is intended solely for the use and information of the client to whom it is addressed.
International Council on System Engineering
2
International Council on System Engineering Booz Allen Proprietary
The need for Risk Management to be Integrated with Systems Engineering is well understood
In a recent Department of Defense briefing titled “Technical Excellence through Systems Engineering”, it was determined that there was a lack of understanding in programs on systems engineering. One of the reasons sighted was:
– Cost and schedule estimation and risk management processes inconsistently aligned with systems engineering processes
26% of surveyed systems engineers listed “Integration with other program management” as a way to drive technical rigor back into programs. This included:
– Linkage with acquisition strategy, IMP, IMS, logistics, testing, and risk management
As a result, a recommended strategy for driving technical rigor back into programs was to have risk management integrated, effective, and well resourced
3
International Council on System Engineering Booz Allen Proprietary
So what?
The question isn’t “Should we do risk management?”
The question is, as a Systems Engineer:
– What do I do with the risk program I have?
– Is the risk program providing a good return for the expense?
– What, if anything, are the risk program outputs telling me?
– Can I use the risk program to communicate more effectively?
The rest of this discussion supposes a mature risk program exists:
– Roles and Responsibilities are defined (i.e. a Risk Mgt plan exists)
– Training has been provided
– Resources have been provided (tools, staff)
– Risk Management products are reviewed at regular intervals in formal reviews (Risk Review Boards)
4
International Council on System Engineering Booz Allen Proprietary
RISC-IQ was developed to help extract actionable information from traditionally stove-piped data streams
RISC-IQ enables critical decision makingRISC-IQ enables critical decision making
Risk Exposure?
Impact Relationships?
Goals Too Risky?
Which Design?
More Reserves?
Major Drivers?
Adequately Mitigated?
5
International Council on System Engineering Booz Allen Proprietary
RISC-IQ works by establishing a structured, repeatable process that integrates the potential impacts of risk on program baselines
Risk Integrated with Schedule and Cost – Intelligent QuantificationRisk Integrated with Schedule and Cost – Intelligent Quantification
Objective: Assess completeness of baseline documentation and understand existing processes.
Objective: Assess completeness of baseline documentation and understand existing processes.
Objective: Compare program risk profile at mitigation completion to determine outstanding risk exposure.
Objective: Compare program risk profile at mitigation completion to determine outstanding risk exposure.
Objective: Provide insight into where risks affect the program and uncover their true impacts.
Objective: Provide insight into where risks affect the program and uncover their true impacts.
Mitigation Strategy Revision
Mitigation Strategy Revision
RiskAnalysis
RiskAnalysis
ScheduleDevelopment
ScheduleDevelopment
CostEstimating
CostEstimating
Program Risk
Baseline
Program Risk
Baseline
Risk-WBS MappingRisk-WBS Mapping
Risk Impact AnalysisRisk Impact Analysis
Integrated Master
Schedule
Integrated Master
Schedule
Most Likely Cost
Estimate
Most Likely Cost
Estimate
Pre-Mitigation Cost EstimatePre-Mitigation Cost Estimate
Economic Effectiveness AnalysisBaseline Product Development
Pre-Mitigation Schedule Estimate
Pre-Mitigation Schedule Estimate
Mitigation Plan Development
Mitigation Plan Development
Mitigation Resource Analysis
Mitigation Resource Analysis
Economic Effectiveness Assessment
Economic Effectiveness Assessment
Post-Mitigation Cost Estimate
Post-Mitigation Cost Estimate
Initial Risk Accounting
Schedule Factor
Analysis
Schedule Factor
Analysis
Cost Factor Analysis
Cost Factor Analysis
Post-Mitigation Schedule Estimate
Post-Mitigation Schedule Estimate
6
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
7
International Council on System Engineering Booz Allen Proprietary
RISC-IQ uses explicit, quantified statements for risk probability and impact to determine risk scores
Probability
5 Very High
4 High
3 Medium
2 Low
1 Very Low
Impact
5 Very High
4 High
3 Medium
2 Low
1 Very Low
Probability
5 >70%
4 50-70%
3 30-50%
2 10-30%
1 <10%
Cost Impact
1 >70%
2 50-70% of Project Budget
3 30-50% of Project Budget
4 10-30% of Project Budget
5 <10% of Project Budget
The right-hand set of tables:
– Communicates the leadership’s sensitivity to risk
– Reduces personal bias
– Creates a foundation for quantitative analysis
VS
8
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
9
International Council on System Engineering Booz Allen Proprietary
RISC-IQ can be used to help select and support multiple types of development cycle
Waterfall Development Model & Derivatives
Assists with estimates of overall cost and schedule estimates
Maps risk mitigation activities to development phases
Quantified risk score can be used to establish threshold for entering the next development phase
Spiral Type Develop Model
Enables capability vs. risk trade study for design features
Economic analysis of risk help apportion development resources to highest need areas
Tracking of risk profile useful in determining if the current spiral met its goals
10
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
11
International Council on System Engineering Booz Allen Proprietary
RISC-IQ is useful for evaluating and communicating the impacts of design selection on risk exposure
As part of design trade activities, RISC-IQ provides input to the engineering processes by analyzing design options based on risk profiles
The risk team helps determine the risk variability across the options and the dependency of any one risk to key features/designs within an option
RISC-IQ applies a “Dependency” framework for this activity.
– Design Independent Risks-Design Independent Risks- Risks that remain relatively constant when key aspects of the system are altered
– Design Dependent Risks-Design Dependent Risks- Risks that vary significantly based on changes to the system design
Design Independent Risk Example
Option 1 Option 2 Option 3
32.50 32.00 33.50Risk Score
Design Dependent Risk Example
Option 1 Option 2 Option 3
11.23 5.48 55.75Risk Score
Little to no change in the risk score as the design or Little to no change in the risk score as the design or option changesoption changes
Significant change in the risk score as the design or Significant change in the risk score as the design or option changesoption changes
12
International Council on System Engineering Booz Allen Proprietary
RISC-IQ provides a simple analytical framework for quickly highlight what risks drive any given design option
Dependency analysis is performed for every risk to determine whether the risk applies to a given option and the strength of dependency by design parameter
Shown below is a sanitized example from a recent client engagement
Risk # Prob Mit Sev Score Risk # Prob Mit Sev Score Risk # Prob Mit Sev ScoreProbability Variance
Mitigation Variance
Severity Variance
Score Variance Risk
Extent of Risk WBS
1 0.37 0.5 22 4.07 1 0.37 0.5 22 4.07 1 0.37 0.5 22 4.07 0 0 0 0 Weight issue None 1.3.55 0.37 0.25 22 2.04 5 0.37 0.25 22 2.04 5 0.37 0.25 22 2.04 0 0 0 0 Deployment None 1.3.36 0.87 0.5 43 18.17 6 0.87 0.5 43 18.17 6 0.37 0.5 43 7.96 0.5 0 0 10.75 Staff resource Moderate 1.3.4
7 0.37 0.5 43 7.96 7 0.37 0.5 43 7.96 0 0 0 0 Space availability Minimal 1.3.425 0.37 0.75 22 6.11 25 0.37 0.75 22 6.11 0 0 0 0 Deployment Minimal 1.2.2.3.3
48 0.98 0.75 43 31.61 48 0.98 0.75 43 31.61 48 0.63 0.5 22 6.93 0.35 0.25 21 24.675 Weight issue Significant 1.3.449 0.63 0.25 22 3.47 0 0 0 0 Deployment Minimal 1.2.3
54 0.13 0.75 36 3.51 54 0.63 0.75 36 17.01 0.5 0 0 13.5 Staff resource Moderate 1.2.2.3, 1.255 0.13 0.75 36 3.51 55 0.37 0.75 36 9.99 0.24 0 0 6.48 Space availability Moderate 1.2.2.4
62 0.63 0.75 36 17.01 0 0 0 0 Deployment Moderate 1.2.3.263 0.37 0.75 75 20.81 0 0 0 0 Weight issue Moderate 1.2.2.264 0.13 0.75 75 7.31 0 0 0 0 Deployment Minimal 1.2.3
70 0.13 0.75 95 9.26 70 0.63 0.75 75 35.44 70 0.63 0.75 75 35.44 0.5 0 20 26.175 Staff resource Significant 1.2.3.471 0.13 0.75 95 9.26 71 0.63 0.75 75 35.44 71 0.37 0.75 75 20.81 0.5 0 20 26.175 Space availability Significant 1.2.3.475 0.13 0.5 75 4.88 75 0.13 0.5 75 4.88 75 0.13 0.5 75 4.88 0 0 0 0 Weight issue None 1.2.2.3.178 0.87 0.25 75 16.31 78 0.63 0.25 75 11.81 78 0.63 0.25 75 11.81 0.24 0 0 4.5 Space availability Moderate 1.2.180 0.37 0.75 50 13.88 80 0.37 0.5 22 4.07 80 0.37 0.50 22 4.07 0 0.25 28 9.805 Weight issue Significant 1.2.2.2.5
91 0.37 0.5 75 13.88 91 0.37 0.5 75 13.88 0 0 0 0 Staff resource Minimal 1.2.392 0.13 0.25 75 2.44 0 0 0 0 Space availability Minimal 1.2.2.498 0.87 0.5 62 26.97 98 0.87 0.5 62 26.97 98 0.37 0.5 62 11.47 0.5 0 0 15.5 Deployment Moderate 1.2
100 0.37 0.75 36 9.99 0 0 0 0 Weight issue Minimal 1.2.3103 0.13 0.75 43 4.19 103 0.37 0.75 22 6.11 0.24 0 21 1.9125 Deployment Significant 1.2104 0.87 0.75 50 32.63 104 0.87 0.75 50 32.63 0 0 0 0 Space availability Moderate 1.2.2.2.4
106 0.13 0.25 29 0.94 106 0.13 0.25 29 0.94 106 0.63 0.75 95 44.89 0.5 0.5 66 43.945 Staff resource Significant 1.2.4.4157 0.87 0.75 95 61.99 157 0.87 0.75 95 61.99 0 0 0 0 Space availability Significant 1.2.4.4162 0.87 0.75 50 32.63 162 0.87 1 50 43.50 0 0.25 0 10.875 Deployment Moderate 1.2.5
175 0.87 0.5 95 41.33 175 0.87 0.5 95 41.33 175 0.87 0.5 95 41.33 0 0 0 0 Weight issue None 1.2.2.5
Design Option 1 Design Option 2 Design Option 3
Results
Sam
ple
13
International Council on System Engineering Booz Allen Proprietary
The RISC-IQ design dependency approach establishes a method to evaluating requirements and overall risk tolerance
Using the quantitative risk scores we can compare a single risk area across platforms, or the variability of a single risk within a given platform
Ris
k L
ev
el
Weight RiskSatellit
e A optio
n
Satellite B
option
Satellite D
option
Satellite C
option
Medium
Low
High
Critical
Ris
k L
ev
el
Communications Risk
Option 1
Option 2
Option 4
Option 3
Medium
Low
High
Critical
(Placing the options in order of complexity, i.e. performance or technology, allows the program to understand the ‘risk curve’ as the options become more aggressive or challenging in their profile.)
14
International Council on System Engineering Booz Allen Proprietary
Once the analytical framework and quantitative scoring is established, risk can then be treated as a design variable
Risk can be treated as an independent variable and applied in a similar fashion to cost in CAIV methodologies; “Risk As An Independent Variable”“Risk As An Independent Variable”
Ris
k L
ev
el
Technology Design Variable/Option
(1) Risk curves are generated for designs or aspects of each design using dependency model
Pe
rfo
rma
nc
e
Me
tric
Technology Design Variable/Option
(2) Performance Analysis working group determines relationship between the design variable and system performance
Variable
Risk
Metric
Variable
Effectiveness
Metricƒ(Variable) = X X
Eff
ec
tiv
en
es
s
Performance Metric
(3) Utility Analysis working group performs effectiveness analysis to show relationship to performance metric
Risk Sensitivity
Performance Sensitivity
Effectiveness Analysis
15
International Council on System Engineering Booz Allen Proprietary
RISC-IQ then uses this analytical framework show when additional performance isn’t worth the additional risk
By combining these curves, it’s possible to identifying the point of diminishing risk tolerance for any design variable or option
– Point of Diminishing Risk Tolerance- “The point at which the increase in additional design capability, or the selection of a proposed option, reduces effectiveness by increasing risk to the program.”
Ris
k T
ole
ran
ce
Design Variable/Option
Effectiveness
Risk=
Point of Diminishing Risk Tolerance
16
International Council on System Engineering Booz Allen Proprietary
RISC-IQ takes the results of the design trades and evaluates them by source category and the work to satisfy the requirements
WB
S #
WB
S E
lemen
t
Co
mm
s
Co
ncu
rrency
Co
st
Desig
n
Develo
per
Lo
gistics
M&
S
Mg
mt
Perfo
rman
ce
Pro
du
ction
Req
uirem
ents
Sch
edu
le
T&
E
Tech
no
log
y
Th
reat
To
tal
1.0 Prog Mgmt 1 1
2.0 Network 1 1 1 3
3.0 Mission 2 1 1 1 5
4.0 Ground 4 1 1 6
4.1 GSE 3 3
4.2 Program Management 1 1
4.3 Subcontractor 2 3 1 1 2 2 1 1 12
4.4 TT&C 2 1 2 1 1 7
4.5 Communications
5.0 Launch 1 2
6.0 Space PMO
7.0 Space SEIT 1 1
8.0 Bus 1 5 2 2 2 1 1 2 1 3 20
9.0 Payload 5 5 10
10.0 Spacecraft AI&T 1 1 1 1 4
Total 3 6 11 8 3 9 3 15 7 7
A review by WBS elements shows which areas of the program need additional risk
identification.
A review by WBS elements shows which areas of the program need additional risk
identification.
Critical Risk Categories with Few Risks Identified
A review by categories show holes in existing risk areas that should be
investigated further.
A review by categories show holes in existing risk areas that should be
investigated further.
The following is an example from a commercial satellite project analysis
17
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
18
International Council on System Engineering Booz Allen Proprietary
RISC-IQ uses a convention of aligning risks to the Work Breakdown Structure (WBS) to help define resource requirements Aligning risks to the program WBS serves as the basis for allocating cost and schedule impacts
throughout the program
During the risk analysis phase, WBS elements are correlated to each other and risks mapped to WBS elements
Mapping provides an easily understood foundation that is repeatable, traceable, and defensibleMapping provides an easily understood foundation that is repeatable, traceable, and defensible
19
International Council on System Engineering Booz Allen Proprietary
After mapping, any WBS element may have multiple risks that affect it
The combined affect of the risks on the cost element is related to the type of the risk, its likelihood and severity, and the cost of mitigation actions that could avert the risk
Using probabilistic models, the combined affect is determined and applied to the base cost estimate
RISC-IQ uses this WBS mapping to show the impacts of risk on the original estimate
WBS
1 3 2 11.1.2.3
Baseline Estimate
Risk-Adjusted Estimate
$1,179,000
50% 50%
20
International Council on System Engineering Booz Allen Proprietary
This activity helps justify mitigation requests and can highlight when mitigation is no longer effective
This activity helps justify mitigation requests and can highlight when mitigation is no longer effective
Represents the initial baseline cost distribution without any included risk effects
Represents the cost distribution with potential risk effects BEFORE any mitigation
Represents the cost distribution AFTER successful mitigation, PLUS the required mitigation investment.
As a result of a $500,000 investment in risk mitigation, the program avoided a potential increase of $645,000.
$645,000
Mitigation Investment= $500,000
RISC-IQ then applies the WBS mapping and pre- and post- mitigation scores to guide mitigation activity
The final determination of cost considers the remaining risk exposure AFTER mitigation, plus the mitigation investment required
21
International Council on System Engineering Booz Allen Proprietary
Relationships between risks and schedule elements for critical path determination must account for:
– Predecessor-Successor logic
– Parent-Child
– Parallel-Effort
– Conditional Dependencies
The same process can be performed on any unit of measure. Risk Adjusted Schedule is particularly complex
Cum
ulat
ive
Fre
quen
cy
Hits
0
24
48
72
96
120
Distribution (start of interval)
13/Jul/05 26/Jul/05 08/Aug/05 21/Aug/05 03/Sep/05 16/Sep/05
5% 29/Jul/05
10% 02/Aug/05
15% 04/Aug/05
20% 06/Aug/05
25% 08/Aug/05
30% 09/Aug/05
35% 11/Aug/05
40% 12/Aug/05
45% 14/Aug/05
50% 15/Aug/05
55% 17/Aug/05
60% 18/Aug/05
65% 20/Aug/05
70% 21/Aug/05
75% 23/Aug/05
80% 25/Aug/05
85% 26/Aug/05
90% 29/Aug/05
95% 02/Sep/05
100% 18/Sep/0500017 - Review production performance results 46%
00008 - Design and order new equipment 45%
00005 - Confirm product specification 33%
00018 - Identify long term production requirements 32%
00003 - Confirm product safety 25%
00021 - Determine if product is meeting customer cost/performance expect 21%
00004 - Re-evaluate manufacturing processes 17%
00020 - Affirm decision to continue commercialization program 15%
00019 - Recalculate product economics 14%
00022 - Finish 7%
00009 - Install and test new equipment or equipment modifications 3%
00002 - Confirm sales and marketing strategies 3%
00012 - Confirm manufacturing capabilities 3%
00016 - Perform internal product testing 3%
00014 - Start production of commercial product quantities 2%
00010 - Test failure 2%
00011 - Test pass 0%
00013 - Develop and review production trial plan (resolve logistics issu 0%
00006 - Confirm customer specification acceptance and trial participatio 0%
00015 - Monitor customer acceptance testing 0%
Risk Adjusted Delivery Likelihood Schedule Sensitivity By Task
22
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
23
International Council on System Engineering Booz Allen Proprietary
RISC-IQ can help determine if the engineering activity is proceeding as planned
As a “snapshot” of project health, RISC-IQ can reveal whether or not the engineering activity is likely to deliver the expected results
In the example below, a RISC-IQ analysis demonstrated a previously unknown risk exposure of $148M and potential project cost growth of 25%
Incorporating RISC-IQ throughout a project’s lifecycle results in better initial planning and helps answer questions like:
– Are prototype or testing activities reducing risk as planned?
– Have risks been reduced sufficiently to pass entry / exit criteria for established technical reviews?
– Are new risks been uncovered faster than they are being closed?
– What are the impacts of “mid-stream” requirements changes?
24
International Conference on System Engineering Booz Allen Proprietary
Agenda
Project / Program Initiation
Design Cycle Selection
Engineering Trades
Defining Resources
Measuring Execution
Managing Groups of Projects
25
International Council on System Engineering Booz Allen Proprietary
RISC-IQ practices for a single project are extensible to groups of projects or systems of systems the must interact to deliver capability…
0
10
20
30
40
50
60
70
GroupAverage
Prog 1 Prog 2 Prog 3 Prog 4 Prog 5 ClassifiedProg Ave
CommercialProg Ave
Low
Med
High
Critical
Average Composite Risk Ranking by Program
Ave
rag
e R
isk
Sco
re
By standardizing risk analysis across all multiple programs, we provide insight into high risk programs and overall trends
26
International Council on System Engineering Booz Allen Proprietary
400000
600000
800000
1000000
1200000
1400000
1600000
Prog 1: Pre-Mitigation
Prog 1:Post-
Mitigation
Prog 2: Pre-Mitigation
Prog 2:Post-
Mitigation
Prog 3: Pre-Mitigation
Prog 3:Post-
Mitigation
Prog N: Pre-Mitigation
Program N:Post-
Mitigation
… and program-by-program risk exposure analysis provides comparative insights what programs can be improved
Pro
gram
Cos
t (in
thou
sand
s)
$39,100
Initial Program Estimate
Revised Estimate
Budget at Risk
Mitigation Investment
Reduced Cost Exposure
Analysis will lead to a cumulative assessment of the total risk exposure, the investment allocated to reduce it, and the resultant investment benefit.
Analysis will lead to a cumulative assessment of the total risk exposure, the investment allocated to reduce it, and the resultant investment benefit.
27
International Council on System Engineering Booz Allen Proprietary
Added DurationContract Length Total Risk Exposure
In a matrixed organization where Systems Engineering is a shared resource, using RISC-IQ to align risk-adjusted program schedules helps prioritize the timing of resources and risk reduction
Post-Mitigation Delivery
Pre-Mitigation Delivery
Contract Delivery
Program1
Program4
Program2
Program3
ProgramN
T0 T3 T18T6 T9 T12 T15T2 T5 T20T8 T11 T14 T17T1 T4 T19T7 T10 T13 T16
The overlap of additional program duration requires allocation of program
and/or enterprise resources
The overlap of additional program duration requires allocation of program
and/or enterprise resources
Understanding most probable program delivery allows S&IS to:
Risk exposure is the amount of schedule slip reduced through successful risk
mitigation
Risk exposure is the amount of schedule slip reduced through successful risk
mitigation
Inform clients of most probable schedule Plan for resources in advance Address contract issues Evaluate effectiveness of proposed mitigation Understand when enterprise reserves will be
needed
28
International Council on System Engineering Booz Allen Proprietary
In summary, risk management provides the Systems Engineer with quite a bit of “so what”
Concept Development
Design Planning & Execution Enterprise
RISC-IQ can help established risk tolerance and incorporate risk as variable benefit trades
RISC-IQ can help established risk tolerance and incorporate risk as variable benefit trades
RISC-IQ provides unique insight into project cost and schedule realism, reserve requirements and health during execution
RISC-IQ provides unique insight into project cost and schedule realism, reserve requirements and health during execution
RISC-IQ highlights what specific requirements or technologies are project drivers
RISC-IQ highlights what specific requirements or technologies are project drivers
RISC-IQ at the enterprise level highlights overall reserves needs, resource shortfalls due to slips, and supports benchmarking
RISC-IQ at the enterprise level highlights overall reserves needs, resource shortfalls due to slips, and supports benchmarking
Cum
ula
tive F
requency
Hits
0
21
42
63
84
105
Distribution (start of interval)
15/May/12 18/Jun/12 22/Jul/12 25/Aug/12 28/Sep/12 01/Nov/12
5% 12/Jun/12
10% 21/Jun/12
15% 27/Jun/12
20% 03/Jul/12
25% 09/Jul/12
30% 12/Jul/12
35% 17/Jul/12
40% 20/Jul/12
45% 25/Jul/12
50% 31/Jul/12
55% 03/Aug/12
60% 08/Aug/12
65% 14/Aug/12
70% 20/Aug/12
75% 24/Aug/12
80% 31/Aug/12
85% 10/Sep/12
90% 19/Sep/12
95% 28/Sep/12
100% 05/Nov/12 Analysis
Simulation: Latin HypercubeIterations: 1000
Convergence
Plan Finish Date:Converged in 200 iterations(variation < 1% over 100 iterations)Total Plan Cost:Converged in 200 iterations(variation < 1% over 100 iterations)
Statistics
Minimum: 15/May/12Maximum: 05/Nov/12Mean: 01/Aug/12Max Hits: 111Std Deviation: 33.62
Selected Confidence75%: 24/Aug/12Deterministic Finish: 03/Apr/12Probability (less than 1%)
System Engineering & IntegrationEntire Plan : Finish Date
Booz Allen Hamilton
29
International Council on System Engineering Booz Allen Proprietary
For more information on how risk management can be applied to your specific challenges, please contact:
Tweed RossColorado Springs, [email protected]
Robert MakarSan Diego, [email protected]
Michael LopezLos Angeles, [email protected]