10-1 GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS Selection should be based on the complexity of the program and the phase of development (modification). Participants Type and Number Include tasking and performing activity personnel responsible for the area or item being reviewed, key representatives for lower level items, and other personnel who have a stake in the specific objectives of the review. Objectives 1. Demonstrate progress 3. Verify the expected maturity 2. Ensure issues have been resolved 4. Confirm that risks are acceptable
Transcript
10-1
GENERAL GUIDANCE ON CONDUCT OF
TECHNICAL REVIEWS
Selection should be based on the complexity of the program and the phase of development (modification).
Participants
Type and Number
Include tasking and performing activity personnel responsible for the area or item being reviewed, key representatives for lower level items, and other personnel who have a stake in the specific objectives of the review.
Objectives
1. Demonstrate progress 3. Verify the expected maturity2. Ensure issues have been resolved 4. Confirm that risks are acceptable
10-2
• Confirm that issues addressed during previous reviews have been satisfactorily resolved.• Only areas requiring close scrutiny should be addressed in detail. Proper integration and management of interim, subsystem, and functional reviews (as planned in SEMP) should make a detailed total evaluation unnecessary.• Opportunity to modify program emphasis for the next phase based on risk levels.
In General
FOCUS IN MAJOR REVIEWS
10-3
TYPES OF REVIEWS
Major: Demonstrate that risk levels are acceptable and provide an opportunity to modify program emphasis for next phase/effort. (Examples: ASR, SRR, SFR, PDR, CDR, SVR, PCA)
Functional: System wide review conducted by involved disciplines to address progress and issues associated with one functional area. (Examples: Development, Training, Support, Verification, Manufacturing, Disposal, etc.)
Subsystem: Multidisciplinary reviews to assess progress in defining and satisfying subsystem requirements.
Interim: Reviews held between major reviews that cover spectrum of system, segments, and configuration items.
10-4
Integrated Technical Reviews (Cont.)
Major Technical Reviews include:
- Alternate System Review (ASR): The ASR forms the basis for determining which system concepts should be continued as candidate development programs. It provides the data for recommendations to eliminate potential systems concepts.- System Requirements Review (SRR): The SRR forms the basis early in System Development and Demonstration whether customer requirements are understood. It should confirm the progress and direction of technology verifications and demonstrations and convergence on viable system requirements
10-5
Integrated Technical Reviews (Cont.)
Major Technical Reviews include:
- System Functional Review (SFR): The SFR (formally called the SDR) forms the basis to establish and verify an appropriate set of functional and performance requirements for the complete system.- Preliminary Design Review (PDR): The PDR forms the basis for determining whether a preliminary physical architecture and design approach for configuration item or aggregation of configurations (including software) is ready to start detailed design. A configuration item development spec must be complete and ready for use in detailed design
10-6
Integrated Technical Reviews (Cont.)
Major Technical Reviews include:
- Critical Design Review (CDR): The CDR confirms the detailed design for each configuration item (including software), each aggregation of configuration items, and that requirements are satisfied.- System Verification Review (SVR): The SVR demonstrates the total system (people, products, and processes) satisfies functional and allocated configuration documentation requirements and confirms readiness for production, support, training, deployment, operations, an disposal. The Production Readiness Review (PRR) is a part of the SVR.
10-7
Customary System Level Review Accomplishments
SRR - Review system functional and performance requirements.
SDR/SFR - Review allocation of system requirements to configuration item (CI) aggregates. Assess maturity
of system spec. and CI aggregates.
PDR - Review design progress for each CI and aggregate of CIs: 1. Review predicted performance 2. Review physical and functional interfaces 3. Verify integration of specialty eng. req. 4. Evaluate risk resolution
10-8
Customary System Level Review Accomplishments (Cont.)
CDR - Review maturity of design for each CI and aggregate CIs:
1. Verify that functional and performance requirements of CI specs are satisfied.2. Verify that engineering specialty requirements have been satisfied.3. Verify compatibility between CIs and hardware, software, facilities and personnel.4. Review preliminary product specs.5. Assess producibility risks.
10-9
Customary System Level Review Accomplishments (Cont.)
SVR - Confirms readiness to enter full rate production:
1. Verify work on all CIs – all functions & performance levels met.
2. FCA conducted for CIs.See page 107 of Sys. Eng. Fundamentals for
additional accomplishments.
10-10
Systems Engineering Master Schedule (SEMS)
Purpose:
• Contractual Agreement Between Contractor and Customer on Critical Tasks Required for Completing Each Major Program Milestone and Defining the Criteria for Successful Completion of Those Tasks.
10-11
Role of SEMS in Acquisition Planning
The SEMS Outlines the Integration of All Major Tasks Necessary to Design, Develop, Test, Produce, Deploy and Support the System.
Support Training
System
Prime Mission Equipment
Performance
Producibility
Reliability
Cost
Schedule
RiskManagement
Balanced Process/Productto Satisfy User Needs
10-12
SEMS Key Tasks
• Typical Tasks• Develop Test Plans• Develop Software Plans
• Tasks Define Interim Steps to Developing and Producing System
• Tasks Should Include Verification Criteria
10-13
SEMS Accomplishment Guidelines
• Typical Accomplishment for Sample Tasks– Test Plan Complete– Software Development Plans Complete
• Accomplishment Should Be Easily Measurable• Example of Difficult to Measure Accomplishment
– Test Plan 85% Complete
10-14
SEMS Sample Accomplishment
Milestone 3 - Test Readiness Review (TRR)
Accomplishments
3.1 Complete Software Test Procedures3.2 Complete System Integration Test Procedures3.3 Complete All Major Subsystem TRRs3.4 Complete All Action Items from Milestone 2
10-15
10-16
10-17
Section L - Instructions, Conditions, and Notices to Offerors
Systems Engineering Master Schedule(SEMS)
volume 3 attachment 1
Systems Engineering Master Schedule(SEMS)
volume 3 attachment 1
10-18
Section L - Instructions, Conditions, and Notices to Offerors (Cont.)
3.0 SEMS PROGRAM EVENTS
Listed below are the Major Program Events and their scheduled time of achievement. Each event shall be considered achieved upon successful demonstration of criteria accomplishments. Criteria for each event is defined in Section 4.0.Event Number Event Title Event Date
1. System Hardware Preliminary 5 MACADesign Review(PDR)
2. System Software PDR 7 MACA
3. ILS Management Team (ILSMT) 9 MACAMeeting
4. System Hardware Critical Design 11 MACAReview (CDR)
5. System Software CDR 13 MACA6. System Test Readiness Review 22 MACA7. Ground Durability/Qualification 24 MACA
Test Release
10-19
Section L - Instructions, Conditions, and Notices to Offerors (Cont.)
Event Number Event Title Event Date
8. (Reserved)9. RF-4C #1 Development Flight 29 MACA
Test Release10. RF-4C DT&E/IOT&E start 29 MACA11. UARS #1 Development Flight 29 MACA
Test Release12. UARS DT&E/IOT&E start 29 MACA
13. F/A-18 DT&E/IOT&E Start 29 MACA14. FSD Maintainability Demonstration 34 MACA15. Production Readiness Review 35 MACA16. System Function Configuration 36 MACA
28. Delivery Per Table 4-2 As Required*MACA: Months After Contract Award
10-21
10-22
10-23
10-24
10-25
10-26
10-27
10-28
10-29
EXERCISE #2
Exercise: Develop the portion of a Systems Engineering Master Schedule (SEMS) for the AJS design and development tasks. Assume the system level schedule for Peace Whey includes a SRR, SFR, PDR, CDR, and PRR. Your SEMS should identify a list of AJS program reviews with accomplishment criteria.
• Part 1: Referring to the system level Peace Whey reviews, identify and list the major reviews which need to be supported by inputs from the AJS design and development tasks. Based upon this list, determine a set or reviews for the AJS development effort required to support the Peace Whey program.
• Part 2: Identify the accomplishment criteria for one AJS review.
You will have 20 minutes to complete this exercise.
10-30
IMP/SEMS & IMS/SEDS Relationships
Integrated Master Plan (IMP)
Integrated Master Schdule(IMS)
ProgramOffice
Contracts
Security
System
Engineering
Master
Schedule
(SEMS)
ProgramOffice
Contracts
Security
System
Engineering
Detailed
Schedule
(SEDS)
10-31
IMP/IMS Development Steps
A
SYSTEM PRODUCT HIERARCHY
SPEC TREEWBSIPT
CB D E
L MKJ
IHGF
SPEC TREEWBSIPT
SPEC TREEWBSIPT
SPEC TREEWBSIPT
ProductEvent No.
AccomplishmentCriteria
IMS Task
A NN NN NN NN
1. Define Events, Accomplishments (RFP “Shalls”), and Criteria
2. Define a Product Oriented Management Structure (SPEC Tree, WBS, and IPT Organization)
3. Build a Top Down, Layered IMP/IMS with Activity Numbers as Shown Below
10-32
Integrated Master Plan (IMP) and Integrated Master Schedule (IMS) Terms
• Product - Hardware, Software, Facilities, Data, or Materials
• Event - Decision Point at End of Major Project Activity (EX. - CDR)
• Accomplishment - Desired Result at Specified Events
• Criteria - Measure of Meeting Accomplishment
• Task - Specific Activity to Complete a Criteria
10-33
Top Level Task Relationships
RequirementsFor Design
RequirementsFor
Integrated Test
SoftwareImplementation
SoftwareDesign
Hardware Design
Suppliers
Component& SystemTesting
RequirementsDevelopment
• Primary • Derived
Production
• Inventory Control• Parts Production• Assembly
WeaponSystem
Integration& Testing
FEEDBACK
• Requirements• Design
Product DefinitionDelivered Product
Parts Assemblies
Product
10-34
Network Chart Example
Network Layout
Conversion to Milestone Chart
K
2 4 6 8 10 12 14
J
C G
A D
BE H
F I
S C
(D) 6 Weeks
(A) 4 Weeks (E) 3 Weeks
(I) 4 Weeks
(H) 2 Weeks
(C) 1 Week (G) 4 Weeks
(J) 5 Weeks
(B) 2 Weeks (F) 8 Weeks
(K) 8 Weeks
10-35
Integrated Master Schedule(IMS) and Network Schedules
• IMS Show Dates for Major Tasks and Milestones- Usually Presented as Gantt (Bar) Chart- Shows Tasks Corresponding to Entry/Exit Criteria- Must Include All IMP Milestones
• Network Schedules Are Essential to the IMS- Helps Determine if IMS Dates and IMP Events Are Achievable- Critical-Path Method (CPM) Schedule Software is Essential- Essentials: Tasks, Relationships, Durations, Resources
10-36
IMP/IMS Summary
• Negotiated IMP Will Be Place on Contract, Changes Will Require a Contract Change Proposal• IMS Shall Support the IMP. IMS Updates Shall Be Approved Prior to Implementation of
Changes. IMS is Time-Based• Customer May Provide Government IMP in RFP. Government IMP Should Provide Detailed
Information for Next Acquisition Phase to Identify Specific Events, Accomplishments and Criteria Necessary to Satisfy Planned and Required Technical Exit Criteria
• IMP Is a Key Control Element of SE Process. IMP Can Be Used as the Basis for Quantitative Requirements for Award Fees
10-37
Exercise # 3
Exercise:Referring to the SEMS from Exercise #2, develop a System Engineering Detailed Schedule (SEDS) for the AJS design and development task. The AJS Safety of Flight unit must be available in month 26 and first production aircraft in month 36. Be prepared to report your results to the class. You will have 20 minutes for this exercise.
Hint – Use building block tasks on next page.
10-38
ElectronicWarfare
10-39
Backup
10-40
Progressive Subsystem
Informal Subsystem
Formal Subsystem
Limited Formality
Formal
Issues
Roll-Up
10-41
Integrated Technical Reviews (Cont.)
Technical reviews are a series of systems engineering activities by which the technical progress of a program is demonstrated relative to its technical or contractual requirements. They are conducted at logical transition points in the development effort to reduce risk by assessing progress against SEMS requirements and identifying and correcting problems/issues resulting from the work completed before the program is disrupted or delayed. They provide a method for the contractor and government to determine if the development of a system and/or configuration item and its documentation have met contract requirements.
10-42
Planning Guidance from DoDD 5000.1(Defense Acquisition)
Part 2.B.3 Acquisition Strategies, Exit Criteria, and Risk Management
“Event driven acquisition strategies and program plans must be based on rigorous, objective assessments of a program’s status and the plans for managing risk during the next phase and the remainder of the program. The acquisition strategy and associated contracting activities must explicitly link milestone decision reviews to events and demonstrated accomplishments in development, testing and initial production. The acquisition strategy must reflect the interrelationships and schedule of acquisition phases and events based on logical sequence of demonstrated accomplishments not on fiscal or calendar expediency.”
10-43
Integrated Technical Reviews
The government assesses the status, maturity, risk, and verification status by conducting reviews that will:
• Be adequately planned prior to the initiation technical work• Have event-driven entry and exit criteria with defined accomplishments and success metrics via the SEMS.• Be conducted incrementally (as appropriately) to measure progress towards achieving major accomplishments and obviate unexpected issues resulting from incomplete or inconclusive development activities.• Be conducted by multidisciplinary teams to ensure that all the system's functions are addressed and all system elements are integrated and balanced across the system.
10-44
DoD Risk ManagementPolicies and Procedures
10-45
Risk Assessment and Management (DoD 5000.1)
“Program Managers and other acquisition managers shall continually assess program risks. Risks must be well understood, and risk management approaches developed, before decision authorities can authorize a program to proceed into the next phase of the acquisition process. To assess and manage risk Program Managers and other acquisition managers shall use a variety of techniques, including technology demonstrations, prototyping, and test and evaluation. Risk management encompasses identification, mitigation, and continuous tracking, and control procedures that feed back through the program assessment process to decision authorities. To ensure an equitable and sensible allocation of risk between government and industry, Program Managers and other acquisition managers shall develop a contracting approach appropriate to the type of system being acquired.”
10-46
Cost, Schedule, and Performance Risk Management (DoD 5000.2-R)
“The Program Manager shall establish a risk management program for each acquisition program to identify and control performance, cost, and schedule risks. The risk management program shall identify and track risk drivers, define risk abatement plans, and perform periodic assessments to determine how risks have changed. Risk reduction measures shall be included in cost-performance tradeoffs, where applicable. The risk management program shall plan for back-ups in risk areas and identify design requirements where performance increase is small relative to cost, schedule, and performance risk. The acquisition strategy shall include identification of the risk areas of the program and a discussion of how the Program Manager intends to manage those risks.”
• Process has two segments– Implementing a comprehensive and active strategy to continuously identify,
mitigate and track program risks• Who does it• What do they do• When do they do it• How risk is shared
– Documenting risk elements of program activities
How do I getthere from here?
10-50
Risk Assessment
• Process of identifying and analyzing program risks to increase the chances of meeting performance, schedule, and cost objectives
• Two segments– Risk Identification– Risk Analysis
10-51
Risk Identification
• Process of specifying, describing and documenting program risks and their sensitivities to other risks
– Internal– External
10-52
Risk Analysis
• Process of evaluating program risks for their impacts to performance, cost, and schedule objectives• Process includes assessing each risk’s:
– Probability of occurrence, and– Consequences of failure to mitigate the risk
10-53
Risk Handling
• Process that identifies, evaluates, selects, and implements risk handling options– to set risk at acceptable levels– give program constraints
• Typical risk handling strategies can include:– replan to eliminate the identified risk– avoid risk by changing requirements– transfer the risk– control the risk through active steps– assume the risk without special efforts
10-54
Risk Monitoring
• Process that systematically tracks and evaluates the performance of risk mitigation actions
- against established metrics throughout the acquisition* process, and- develops further risk handling options as appropriate
* Acquisition includes any procurement from government or contractor sources within all phases from early research through logistics, operations, support, and disposal
10-55
Risk Management Process
Evaluate RiskHandling Options
•Avoid• Information Gathering• Transfer• Assume• Control
Program Requirements
AssessRisk
• Identify• Analyze• Quantify Risk
Evaluate Subcontractor
Risks
Analyze Impacts
• Performance• Cost• Schedule
Manage Risk
• Review Indicators• Abatement Actions
10-56
Assess Risks
Program Requirements
AssessRisks
Evaluate RiskHandling Options
EvaluateSubcontractor
Risks
Establish Cost,Schedule, &Performance Impacts
ManageRisks
EstablishApproach
DevelopTeam
IdentifyRisk
Areas
AnalyzeRisk
Create RiskManagement
Tools
10-57
Risk Identification
The WBS is normally used to organize and ensure completeness of the risk identification effort.
Identification is generally performed at the 3rd or 4th level of the WBS.
10-58
10-59
10-60
Risk Categories and Consequences
• Risk Categories– Requirements - Are the Necessary Requirements ( Operational or Design) Fully Defined? Is the
Basis for the Requirements Stable (e.g. No Expected Threat Change)– Technology - Is the Technology Available Proven in Previous Use?– Engineering - How Much New Design is Needed to Achieve Requirements?– Manufacturing - Are the Required Manufacturing Processes, Facilities, and Sources of Materials
Known and Available?– Support - Are the Required Support Resources Defined and Available?– Management - Are the Processes, Resources, and Experience Available to Successfully Perform
this Program?• Risk Consequences
– Performance - Can the Item Meet Its Requirements (Operational, Support and Manufacturing)?– Cost - Can the Item Be Developed and Operated within the Funding Allocated to It?– Schedule - Can the Item Be Developed and Deployed with the Time Allocated to It?
10-61
Sample Risk Identification
WBS 113 - Guidance System
Issues: New Design – Uses new chipset from CHIPLEAP program. Some concern exists on producibility and thermal characteristics.
Assumptions: Megalith Corporation will design guidance system. They were a participant in the CHIPLEAP effort.
10-62
Additional risk categories such as customer satisfaction or customer expectation can be developed for identification purposes.
Risk assessment templates (shown on later charts) for subcategories can be developed and maintained by organizations.
Organization of Risk Assessment
10-63
10-64Ref. – Best Practices
10-65
Funding
ComputerAided Design
Built –In Test
Parts and MaterialsSelection
Trade Studies
Design Process
Design RefMission Profile
SubcontractorControl
DesignReviews
LogisticsSupport AnalysisDesign
Requirements
Design Policy
Design Analysis
SoftwareDesign
Design For Testing
ConfigurationControl
Design Release
IntegratedTest
Failure Reporting
System
UniformTest Report
Software Test
Design Limit
Life
Test AnalyzeAnd Fix
Field FeedbackManufacturing
Screening
Computer-AidedMFG
Special Test Equipment
Tool Planning
Defect Control
Piece PartControl
Quality MFGProcess
ManufacturingPlan
Factory Improvements
Productivity Center
Manufacturing Strategy
ProductionBreaks
Modernization
ManpowerAnd Personnel
Support andTest Equipment
Training MaterialAnd Equipment
Personnel Requirements
Data Requirements
Technical Risk Assessment
Design Test Production Facilities Logistics Management
Product
Funding
Money Phasing
Spares
TechnicalManuals
Transition PlanRef. – Best Practices
10-66Ref. – Best Practices
10-67Ref. – AFMC 63-101
10-68
10-69Ref. – AFMC 63-101
10-70
Probability (Likelihood)
1
0
Consequences
Traditional Risk Analysis
Performance
Cost
Schedule
Potential Degradation
Sys Reqt not Achieved
Element Increase > 50%
System Increase > 40%
Element Increase
Element Increase < 10%
x
x
x
Probability is Assigned
Consequences are Estimated
x
10-71
Probability (Likelihood)
1
0Consequence
Traditional Risk Analysis
Performance
Cost
Schedule
Potential Degradation
Sys Reqt not Achieved
Element Increase > 50%
System Increase > 40%
Element Increase
Element Increase > 50%
x
x
x
x
x
High Risk – Severe disruption expected to performance, cost, and / or schedule even with risk mitigation plans in place.
Moderate Risk –Expected disruption to performance, cost, and / or schedule can be overcome by implementing risk mitigation plans.
Low Risk – Little disruption expected to performance, cost, and / or schedule.
10-72
Weaknesses of Traditional Risk Analysis Process
1. Roll-up of risks characterized as high, moderate, or low at the 3rd or 4th WBS level are difficult. Example – Are 10 low risks and 1 high risk at WBS Level 4 elements expressed as low, moderate, or high at the parent WBS at Level 3?
2. Characterizing a risk as high, moderate, or low alerts the customer to the severity of the outcomes without giving insight into the likely capability of the delivered product.
![Research Articledownloads.hindawi.com/journals/complexity/2020/8231243.pdf · sented in literature [6–32]. In particular, latest comparative studies and reviews of most commonly](https://static.documents.pub/doc/80x56/5e8ba5595c1b80665040f884/research-sented-in-literature-6a32-in-particular-latest-comparative-studies.jpg)
