Kapruch steve

NASA Engineering of Systems NASA Engineering of Systems Excellence InitiativeExcellence Initiative

S J KapurchS J KapurchProgram Executive OfficerProgram Executive Officer

Systems EngineeringSystems Engineering7 Feb 20077 Feb 2007

PurposePurposeOverview – OCE Initiatives– Policy in context of other effortsMulti-Faceted Approach– Highlight key elements– Integration of Related Efforts

The EnvironmentThe EnvironmentAcquisition streamlining of the 90’s– The Good, the Bad, & the Ugly

Nature abhors a vacuum– ISO-15288, IEEE-1220, EAI-632=Standards quagmire

Maj. Gen. Craig Cooning, the Air Force's Director of Space Acquisition at the Pentagon– "We walked away from some of the things that served us very

well. When you did away with that, you did away with the common language that engineers spoke. So, what we are trying to do is to reinstate those processes that served us well--not to go overboard--but to do it selectively." (21 October 2004)

Like other organizations, NASA tossed out many processes & procedures during acquisition streamlining.

System Engineering Issues System Engineering Issues – Failure Review Boards have consistently pointed to lack of SE

CAIB– “Organizational causes of this accident … cultural traits … detrimental to safety …

allowed to develop including: …reliance on past success as a substitute for sound engineering practices …” -CAIB, Executive Summary

Other Recent NASA Projects – “…., the root cause was not caught by the processes implemented by the project – CDR was too high level to adequately assess design…too little time to perform an

adequate assessment– ….Although training was widely available, poor requirements are still common– “… likely cause for the failure of the system was a faulty design …switches …

improperly installed on a circuit board”– “ the mishap occurred mainly because of failures in SE process….and is known to

be the cause of several recent failures”

Complexity is a Major IssueComplexity is a Major IssueIntegration of systems create a major problem with complexity– As more systems are added, the interfaces

grow in a non-linear fashion– Many of the existing systems were not built

for these interfaces– Conflicting or missing interface standards

make it hard to define interface interactionsSystems engineering must deal with this complexity– End-to-end systems engineering is needed,

including “reengineering” of legacy systems– Robust M&S, verification and validation testing

are A must

Need: Consistency in basic approach to systems Need: Consistency in basic approach to systems engineering.engineering.

Need:Need: Agency wide framework of recognized best Agency wide framework of recognized best practices that guides the engineering of program practices that guides the engineering of program and project products and capabilities.and project products and capabilities.

Need:Need: Common systems engineering terminology and Common systems engineering terminology and definitions to enhance communication and definitions to enhance communication and collaboration among engineering teams across the collaboration among engineering teams across the Agency and with external partners and customers.Agency and with external partners and customers.

Need: Basis for assessing and continuously improving Need: Basis for assessing and continuously improving systems engineering capabilities. systems engineering capabilities.

Specific Needs Specific Needs

Response: Systems Engineering Excellence Initiative

• Systems engineering working group (SEWG)- POC at centers and MD’s for SE efforts- Plan, develop and execute the Initiative.- Coordinate project products within members’ centers

• Engineering management board (EMB) - provide project oversight and approvals.

• Mission directorates & centers- Provide members of the SEWG.- Provide needed support for reviews, pilots and assessments.- Verify suitability for accomplishing programs and projects.

• Customers • NASA engineering community• Advanced technology teams• Payload developers• OCE• MD and Center management• Program and project managers • External partners

SEWG CHARTERThe SEWG:The SEWG:Is chartered by EMB, Is chartered by EMB, in support of Strategic in support of Strategic Plan to develop and Plan to develop and implement a common implement a common framework for framework for systems engineering systems engineering in NASAin NASA

TERMS of REFERENCE

Response: The License. . .Response: The License. . .

“…“… This This Framework will Framework will describe the describe the requirements for SE requirements for SE processes required processes required to engineer products to engineer products and capabilities and capabilities …”…”

GoalsGoalsStimulate and enable the development and advancement of a sound systems

engineering capability necessary for success in fulfilling NASA Mission’s

Ensure continuous improvement of the NASA engineering workforcethrough relevant education, training and work experiences.

Ensure sound and effective discipline and systems engineering Agency-wide.

Provide value-added cross-cutting products and services that enable the infusion of technology, knowledge, and capabilities to support innovation in engineering and push the state of the art.

Increase participation, membership, and leadership in recognized national and international engineering organizations.

Integration of Software.

Enable and foster excellence in systems engineering capabilities to:

– Formulate feasible program and project concepts.– Deliver required products and services to NASA customers. – Make timely acquisition of enabling products and critical technologies.– Reduce risk in system development and deployment.

Enable more effective communications and collaboration within NASA and with external partners and customers.Conduct effective assessment and improvement of systems engineering capabilities.Change the culture to represent the needs of one NASA, and not the unique needs of a particular Center.Develop strategic focus for advanced engineering environments.

Expected Benefits

What is Systems EngineeringWhat is Systems Engineering

“Common People Separated by a different language” Winston Churchill

Integrated Baseline Review


System Integration System Demonstration Low-Rate Initial Production

DefenseAcquisition

System(event driven)

Planning,Programming,

Budgeting, & Execution

Process(biennial calendar

driven)

MilitaryDepartments &

Defense Agencies

Cost

Integrated Defense Acquisition, Technology, & Logistics Life Cycle Management Framework

FOCIOC

Full-Rate Production/DeploymentMSB

System Development & Demonstration Phase Operations & Support Phase

PostDeploymentPerformance

Review

Three DODDecision Support Systems

Effective Interaction is Essential

Sustainment

National Military Strategy FYDPupdated

ISSUES

BudgetCommittees

Congress Congress

DoDBudget

On Year

Off Year

POM/Budget FormulationPCP/BCP Prep

FinalPBDs

Actual CostsEngineeringCost Estimation

MethodsParametric

Analogy

Technology Development PhaseConcept Refinement Phase

National Security Strategy

DoD Testimony

Strategic Planning Guidance

Off Year Optional

Fiscal GuidancePresident’sBudget toCongress

Joint Programming Guidance

Off Year Optional

WhiteHouse

OSD &Joint Staff

OMB

SLRG Reviews

Joint Planning Document

PBD Cycle

PMO Budget Estimate

PMO POM Input

FullFunding in

FYDP

CDD

MSC

POM/Budget SubmitPCP/BCP Submit

FYDPupdated

MSA

The Milestone Decision Authority may authorize entry into the acquisition process at any point, consistent with phase specific entrance criteria and statutory requirements

Economic Analysis(MAIS Only)

AoA Plan

System Threat Assessment

InformationSupport

Plan

CPDService/JROCValidation &

Approval

J-6 Interoperability& Supportability Cert. Validated and approved CDD and CPD for each increment of an

evolutionary acquisition

Oversight &Review

Clinger-Cohen Act- Compliance (all IT)- Certification (MAIS)

Net-ReadyKPP

KPPs

CPD

System Threat Assessment

InformationSupport

Plan

Service/JROCValidation &

Approval

J-6 Interoperability& Supportability Cert.

Net-ReadyKPP




ADMExit

CriteriaMet

DAB/ITAB MDAADMDAB/

ITAB MDA ADMExit

CriteriaMet

DAB/ITAB MDAAPB ADM

ExitCriteria

MetDAB/ITAB MDA ADM

ExitCriteria

MetDAB/ITAB MDAAPB ADM

ExitCriteria

MetDAB/ITAB MDAAPB

MBI

Allocation

Apportionment

AffordabilityAssessment

CARD(Designated Programs)

POE CCA ICEAffordabilityAssessmentPOE CCA ICE


Contracting Acq Plan

Source Selection

Plan

DraftRFPRFP

Acq Plan

DraftRFPRFP

Source Selection

Plan

Production & Deployment Phase

Increment IIIB C

DRR FRP

Increment IIB C

DRR FRP

KPPs

Joint CapabilitiesIntegration &

Development SystemVCJCS OversightCJCSI 3170.01D

DefenseAcquisition

SystemUSD(AT&L) Oversight

DoDD 5000.1

Planning,Programming,

Budgeting & ExecutionDEPSECDEF Oversight

MID 913

Purpose of LRIP:• Establish Production Base• Ramp to Production Rate• Produce systems for IOT&E

Low-Rate Initial Production

Systems

FCA

Prototypes/EngineeringDev Models

TechnologyDevelopment Strategy• Program Strategy• Cost, schedule &performance goals &exit criteria for first tech demo

• Test Plan

Acquisition Strategy• Program Structure• Acquisition Approach• Capability Needs• T&E Considerations• Risk Management• Resource Management• Life-Cycle Considerations• Business Considerations

System Development& Demonstration

Contract


LRIPContract

AoA

BestMateriel

Approach(es)

Draft ver. 4.7. May 21, 2004

StudyContracts

Initiate Evolutionary Acquisition Strategy Evolutionary Acquisition Strategy

Refine initial concept. Develop Technology Development Strategy

Reduce technology risk and determine appropriate set of technologies to integrate into a full system.

Develop a system or increment of capability; reduce integration and manufacturing risk; ensure operational supportability; reduce logistics footprint; implement human systems integration; design for producibility; ensure affordability and protection of critical program information; and demonstrate system integration,

interoperability, safety, and utility.

Achieve operational capability that satisfies mission needs.

Execute a support program that meets operational support performance requirements and sustains the system in the most cost-effective manner over its

total life cycle. Dispose of the system in the most cost-effective manner at the end of its useful life.

TechDemos

PreliminaryIntegrated

Architecture

Acq Plan

DraftRFPRFP

Source Selection

Plan

TechnologyDevelopment

Contract

Supports O&M Budget Review

Acq Plan

DraftRFPRFP

Source Selection

Plan

In-ServiceReview

October April / May

August

September - November

October - November November December

JanuaryFebruary (1st Monday)

February - September

DoD Appeals

Disposal

This chart is a classroom aid for Defense Acquisition University students. It provides a notional illustration of the interfaces among the three major decision support systems used to develop, produce, and field a system for national defense. Defense acquisition is a complex process, with many more activities than shown here, and many concurrent processes that cannot be properly displayed on a two-dimensional chart. Supporting information is on the back of this chart. For more detailed information see the Acquisition, Technology & Logistics Knowledge Sharing System (http://akss.dau.mil).

SystemPerformance

Spec

Prototypes/EngineeringDev Models

InitialProduction

Baseline

CARD – Cost Analysis Requirements DescriptionCCA – Component Cost AssessmentICE – Independent Cost EstimateMAIS – Major Automated Information SystemPOE – Program Office EstimateRDT&E – Research, Development, Test & Evaluation

Cost Acronyms

BCP – Budget Change ProposalsFYDP – Future Years Defense ProgramMBI – Major Budget IssueOMB – Office of Management & BudgetPBD – Program Budget Decision

PCP – Program Change ProposalsPDM – Program Decision MemorandumPOM – Program Objectives MemorandumSLRG – Senior Leadership Review Group

Planning, Programming, Budgeting & Execution Acronyms

Joint Capabilities Integration & Development

System(need driven)

ADM – Acquisition Decision MemorandumAoA – Analysis of AlternativesAPB – Acquisition Program BaselineCD – Concept DecisionDAB – Defense Acquisition BoardDRR – Design Readiness ReviewFOC – Full Operational Capability

FRPDR – Full-Rate Production Decision ReviewIOC – Initial Operational CapabilityITAB – Information Technology Acquisition BoardLRIP – Low Rate Initial ProductionMAIS – Major Automated Information SystemMDA – Milestone Decision Authority

Oversight & Review Acronyms

RFP – Request for Proposal

Joint Capabilities Integration & Development System - AcronymsCDD – Capability Development DocumentCJCSI – Chairman, Joint Chiefs of Staff InstructionCPD – Capability Production DocumentDOTMLPF – Doctrine, Organization, Training,

Materiel, Leadership, Personnel, and Facilities

DAB – Defense Acquisition BoardICD – Initial Capabilities DocumentIOC – Initial Operational CapabilityJROC – Joint Requirements Oversight CouncilKPP – Key Performance Parameter

Draft CDD

AoAupdated

Full-RateProduction

SystemsMajor

Products

AppropriationCommittees

AuthorizationCommittees

Authorization/AppropriationActs Passed

PDM(s)

Appropriated Funds To Support Contracts

FOC

AffordabilityAssessmentPOE CCA ICE


CARD(Designated Programs)

Post Production Software Support ContractsSustainment Contracts

Operations &Maintenance

Types ofFunds RDT&E – Management & Support RDT&E – Management & SupportRDT&E – Management & Support

RDT&E – Adv Component Dev & Prototypes RDT&E – Systems Development & Demonstration

Decision Points/Milestones CD DRR FRPDR



Procurement

AoAupdatedn/a MAIS

AoAMAIS only

PreferredSystemConcept

FinalProduction

Baseline

ICD

Post Independent Analysis

FunctionalArea Analysis

FunctionalNeeds Analysis

Joint Operations Concepts

DOTMLPF

Joint Operating ConceptsJoint Functional ConceptsJoint Integrating ConceptsIntegrated Architectures

DoD StrategicGuidance

Functional Solution Analysis

DOTMLPF Changes(CJCSI 3180)

Service/JROCValidation &

ApprovalMaterielChanges

(CJCSI 3170)

Ideas forMateriel

Approaches

Analysis ofMateriel

Approaches

Alternative 1Alternative 2Alternative N

RDT&E – Advanced Technology Development


ProductionContract

Joint Functional ConceptJoint Integrating Concept

Integrated Architecture


Integrated ArchitectureThreshold/objective tradeoffs

– Revised Performance Attributes

Threshold/objective tradeoffs – Revised Performance

Attributes


Integrated Architecture

FOT&E

LFT&EWaiver

(if appropriate)

TechnicalSystems EngineeringTest & EvaluationSupportability

Analyze/Assess Concepts Versus

Defined User Needs

Develop Concept Performance (& Constraints)

Definition & VerificationObjectives

Interpret User Needs,Analyze Operational

Capabilities &Environmental Constraints

Decompose ConceptPerf into Functional

Definition &Verification Objectives

Assess/AnalyzeConcept System

Versus FunctionalCapabilities

Decompose Concept Functional Definition into

Component Concepts/Assessment Objectives

Develop Component Concepts, i.e., Enabling/Critical

Technologies, Constraints & Cost/Risk Drivers

Assess/AnalyzeEnabling/Critical

Components VersusCapabilities

Interpret User Needs.Analyze Operational

Capabilities & Environmental Constraints

Demo & Validate SysConcepts & Technology

Maturity VersusDefined User Needs

Develop System Perf(& Constraints) Spec &Enabling/Critical Tech

&Verification Plan

Develop FunctionalDefinitions for Enabling/Critical Technologies &

Associated Verification Plan

Demo SystemFunctionalityVersus Plan

Decompose FunctionalDefinitions into CriticalComponent Definition

& Tech Verification Plan

Demo Enabling/Critical Technology

ComponentsVersus Plan

Develop System Concepts,i.e., Enabling/Critical Technologies,

Update Constraints & Cost/Risk Drivers

TRR

PDR

SRR

Interpret User Needs, Refine System

Performance Specs &Environmental Constraints

Develop SystemFunctional Specs &

System Verification Plan

Evolve FunctionalPerformance Specs into CI Functional (Design to)

Specs and CI Verification Plan

Evolve CI FunctionalSpecs into Product

(Build to) Documentationand Inspection Plan

Fabricate, Assemble,Code to “Build-to”

Documentation

Integrated DT&E, LFT&E & EOAs Verify Performance

Compliance to Specs

Individual CIVerification

DT&E

SFR

CDR

INPUTSINPUTS•ICD•AoA Plan•Exit Criteria•Alternative Maintenance & Logistics Concepts

INPUTS OUTPUTS OUTPUTS OUTPUTS

Trades Analyze

Assess/AnalyzeConcept & Verify System Concept’s

Performance

Trades Analyze

Demo/ModelIntegrated System Versus

Performance Spec

Trades Analyze

System DT&E, LFT&E & OAs,Verify System Functionality& Constraints Compliance

to SpecsMonitor and Collect

All ServiceUse Data

DevelopCorrective

Action

Analyze Data toDetermine

Root Cause

DetermineSystem Risk/

Hazard Severity

Integrate & TestCorrective Action

Implement andField

• Process Change –Hardware/Support• Materiel Change

INPUTS•Service Use Data•User Feedback•Failure Reports•Discrepancy Reports•SEP

OUTPUTS

AnalyzeTrades

Assess Risk of Improved System

Combined DT&E/OT&E/LFT&EDemonstrate System toSpecified User Needs &

Environmental Constraints

•ICD & Draft CDD•Preferred Sys Concept•Exit Criteria •T&E Strategy•Support & MaintenanceConcepts & Technologies

•AoA•SEP•TDS

•Sys Performance Spec•Exit Criteria•Validated Sys Support &Maintenance Objectives &Requirements•APB •CDD •SEP •ISP •TEMP

Analyze DeficienciesTo Determine Corrective

Actions

OUTPUTS

Modify Configuration(Hardware/Software/Specs)

To Correct Deficiencies

Logistics & Technical AcronymsASR – Alternative Systems ReviewBLRIP – Beyond Low Rate Initial ProductionCDR – Critical Design ReviewCI – Configuration ItemDT&E – Developmental Test & EvaluationEDM – Engineering Development ModelEOA – Early Operational AssessmentESOH – Environmental, Safety & Occupational HealthFCA – Functional Configuration AuditFMECA – Failure Mode Effects & Criticality AnalysisFOT&E – Follow-on Test & EvaluationFTA – Failure Tree AnalysisIOT&E – Initial Operational Test & EvaluationISR – In-Service ReviewJITC – Joint Interoperability Test CommandLFT&E – Live Fire Test & EvaluationLCC – Life Cycle CostsLORA – Level of Repair AnalysisLRIP – Low Rate Initial ProductionMTA – Maintenance Task AnalysisOA – Operational Assessment

OT&E – Operational Test & EvaluationOTRR – Operational Test Readiness ReviewPESHE – Programmatic Environment, Safety &

Occupational Health EvaluationPDR – Preliminary Design ReviewPCA – Physical Configuration AuditPRR – Production Readiness ReviewPPP – Program Protection PlanRMS – Reliability, Maintainability &

SupportabilitySEP – Systems Engineering PlanS&T – Science & TechnologySFR – System Functional ReviewSRR – System Requirements ReviewSTA – System Threat AssessmentSVR – System Verification ReviewT&E – Test & EvaluationTEMP – Test & Evaluation Master PlanTDS – Technology Development StrategyTRA – Technology Readiness AssessmentTRR – Test Readiness Review

LeastAcceptable

MostAcceptable

Recycle/Reuse

Reprocessing

Disposal Landfill

Disposal

Logistics/Sustainment

•Test Results •Exit Criteria•APB • CPD • SEP •TEMP•Product Support Package

INPUTS

Operations & Sustainment•Peacetime•Training

•Joint Operations •Crises

Evaluate ProductSupport

Capabilities

Performance Based Agreements

Business CaseAnalysis

Product Support Integrator/Product Support Provider

LFTEReport to Congress

BLRIPReport to Congress

AA CCCDCD DRRDRR FRPFRPDRDRBB

Demonstrate Product Support Capability

•Footprint Reduction •Supply Chain Management•Product Support Elements

Total LifeCycle SystemsManagement

Verify & ValidateProduction

Configuration

Define Supportability

Objectives

MTA

FMECA

FTA

RCM

LORA

•Prelim Sys Spec•T&E Strategy•SEP•Support & MaintenanceConcepts & Technologies•Inputs to:-draft CDD-AoA-TDS-Cost/Manpower Est.

ASR

•Sys Performance Spec•LFT&E Waiver Request•TEMP•Validated Sys Support &Maintenance Objectives & Requirements

•SEP •PESHE •PPP •TRA•Inputs to:

-IBR -ISP -STA -CDD-Acq Strategy-Affordability Assessment-Cost/Manpower Est.

SVR PRRSRR

•Initial Prod Baseline•Test Reports•TEMP•Elements of ProductSupport•Risk Assessment•SEP •TRA •PESHE•Inputs to:

-CPD -STA -ISP -Cost/Manpower Est.

•Data for In-Service Review•Input to CDD for next increment•Modifications/upgrades to fielded systems•SEP

Independent IOT&E

•Production Baseline•Test Reports•TEMP • PESHE • SEP •Input to:

- Cost/Manpower Est.

Full-Up System Level LFT&E

J-6 Interoperability& Supportability Validation

Product Support Package/PBL Implementation•Product Support Elements •Support and Cost Baseline

•Supply Chain Management

•Contract for Sustainment (organic & commercial)

OTRR

JITC Interoperability Certification Testing

PCA

Develop Initial Product Support Strategy•Interoperability•Supply Chain Mgmt•LCC Optimization

•Footprint reduction•Product Support Elements

Performance Based Logistics (PBL) Strategy (Preferred Product Support Approach)

Refine Supportability

Objectives/Constraints

Set Product Support

Strategy

Pre-IOC & Post IOC Supportability Assessments

•Continuous Tech Refreshment•Obsolescence Management•Configuration Control•Data Management

Product Support/PBL Management•Public-Private Partnering•PBA Modifications•Assessment of PSI/PSPs•Supply Chain Management

Product Support Plan•Statutory/Regulatory•Source of Support•Legacy Considerations

-Supply Support -Training-Maintenance -Support Data

-Manpower & personnel

•Product Support Elements

Engineering Excellence Framework

Consistent Approach at All Levels

Workforce– Experienced – Well Trained – Application

Concepts & Processes Tools & Methods

Continuous Improvement – Metrics

Knowledge& Skill of Workforce

Concepts and Processes

Tools &Methods

Capab

ility

Capab

ility

Multi-Dimensional problem requires 3-D solution

And we will get it right later

And fix it with software

Concepts & ProcessesConcepts & Processes

Develop a NASA Systems Engineering Policy to:– Provide consistency across agency– Advance practice in agency

Take advantage of lesson’s learned from other organizations

– Address findings and results from numerous studies and FRB’s/Mishaps

CAIB/DiazNIATOthers

Policy ApproachPolicy ApproachDevelop Requirements for NPR

– Detailed Research Into the Failure Review Board Reports, NIAT, Etc…– Analyzed Results of the Pre-assessment Study– Review SE at Other Organizations Such As LMI, Boeing, RAYCO, N-G,

NGA, DoD, SMC, NRO, USAF, USN, INCOSCE, NDIA….Assess Systems Engineering Best PracticesEvaluate Systems Engineering at NASAIntegrate the Best SE Practices for NASAModeled After and Integrated with 7120.5C, 8700 Series and SW NPR

ProcessConduct Four Workshops Hold Technical Exchanges and Conduct Reviews (Internal and External)

ProductsNPR As Well As Others As Required, …

1.1. Make it a Make it a Value Added PropositionValue Added Proposition not an Overhead Burden to the Projects, not an Overhead Burden to the Projects, Missions, Centers, Agency PractitionersMissions, Centers, Agency Practitioners

2.2. Do it right upDo it right up--front, the first timefront, the first time……instead of instead of ““heroes saving the dayheroes saving the day””

Objective of NPRObjective of NPR

Develop Agency NPR To:– Transform Systems Engineering From a Task

Performed by Individuals With the Title Systems Engineer to -- A Logical Systems Approach Performed by Mutli-discipline Teams to Engineer and Integrate NASA’s Systems.

– Provide Consistency Across Agency– Advance Practice in Agency

Take Advantage of Lesson’s Learned From Center’s and Other Organizations

– Address Findings and Results From Numerous Studies and FRB’s/mishaps

NASA’s policy is to establish, document, and promulgate internal NASA requirements where necessary to fulfill the Agency’s vision, mission, and external mandates. (NPD 1400)Agency Level– Ensure consistency– Ability to work between centers

Written requirements establish the baseline for:– Performing activities– Measuring compliance and effectiveness of that performance– Capture and disseminate corporate knowledge– Codify lessons learned

Verba Volent – Scripta Manent(What is spoken flies – What is written remains)

Requirements Philosophy and ObjectivesRequirements Philosophy and Objectives

NPR 7123 ApproachNPR 7123 Approach

Preparation Briefings

SE NPRObjectives

Off-Site 1: Baseline team, Define Objectives and draft annotated outline

Draft Outline

Off-Site 2: Meat on the Bones

SE Process

Revised Outline

Elements for Rough Draft

75% 75% DraftDraftNPRNPR

NODIS NODIS

Red & Comp Red & Comp Teams/Teams/PrePre--NodissNodiss

Update 6105

Update Update 61056105

Off-Site 3 Partial Draft

Run Scenarios

DevelopProcess &Models

Updated Processes& Models

Off-Site 4: Review draft

Other Products

CompleteSections, Interate Models

Updated Draft

ApprovedApprovedNPRNPR

App.App.StdsStds

Workshop IWorkshop IOCE approvalWorkshop I: Senior ExpertsCaptured valuable nuggets

– DoD and Industry are developing robust SE processes

Enforced and supported by leadership– NASA has many disconnected pockets

Each center has their own wayBut, similarities and commonalities

– NASA needs a top-down directed approach– An Enterprise (OCE) Architecture

It is important that PM, software and systems engineering be well integrated

– Crucial for development of complex systems – Converging on a new enterprise process,

framework, and consistent processes across NASA represents significant culture change

Change of this magnitude takes time and persistence

WS I Attendees (partial list)• OSD - Mark Schaffer• USAF - Col. Mike Holbert• NRO- Rob Klotz• USAF SMC - Col. Rocky Dewan• NSSO- Capt. Marsden• USN - Zig Rafalek• Raytheon/NDIA- Robert Rossa• Lockheed- Paul Robataille• Boeing- Dev Banerjee• Northrop-Grumman- James van Gaasbeek• INCOSE- John Snoderly• NASA HQ (OCE) - Rex Geveden• NASA ESMD - Ellen Stigberg• NASA HQ (OSMA) - Wilson Harkins• NASA Stennis - Christine Powell• NASA HQ (SOMD) - Stan Fishkind • NASA JPL – Steve Wall• NASA LaRC - Al Motley• NASA MSFC - Herb Shivers• NASA JSC - Linda Bromley• NASA NESC- Peggy Chun• NASA APIO- Steve Cavanaugh• USAF Col Mike Holbert• Consultants- Jerry Lake/SMi, Jalal Mapar/SAIC

WS I Attendees (partial list)WS I Attendees (partial list)•• OSD OSD -- Mark SchafferMark Schaffer•• USAF USAF -- Col. Mike HolbertCol. Mike Holbert•• NRONRO-- Rob KlotzRob Klotz•• USAF SMC USAF SMC -- Col. Col. Rocky DewanRocky Dewan•• NSSONSSO-- Capt. MarsdenCapt. Marsden•• USN USN -- Zig RafalekZig Rafalek•• Raytheon/NDIARaytheon/NDIA-- Robert RossaRobert Rossa•• LockheedLockheed-- Paul RobataillePaul Robataille•• BoeingBoeing-- Dev BanerjeeDev Banerjee•• NorthropNorthrop--GrummGrummanan-- James van GaasbeekJames van Gaasbeek•• INCOSEINCOSE-- John SnoderlyJohn Snoderly•• NASA HQ (OCE) NASA HQ (OCE) -- Rex GevedenRex Geveden•• NASA ESMD NASA ESMD -- Ellen StigbergEllen Stigberg•• NASA HQ (OSMA) NASA HQ (OSMA) -- Wilson HarkinsWilson Harkins•• NASA Stennis NASA Stennis -- Christine PowellChristine Powell•• NASA HQ (SOMD) NASA HQ (SOMD) -- Stan Fishkind Stan Fishkind •• NASA JPL NASA JPL –– Steve WallSteve Wall•• NASA LaRC NASA LaRC -- Al MotleyAl Motley•• NASA MSFC NASA MSFC -- Herb ShiversHerb Shivers•• NASA JSC NASA JSC -- Linda BromleyLinda Bromley•• NASA NESCNASA NESC-- Peggy ChunPeggy Chun•• NASA APIONASA APIO-- Steve CavanaughSteve Cavanaugh•• USAF Col Mike HolbertUSAF Col Mike Holbert•• ConsultantsConsultants-- Jerry Lake/SMi, Jalal Mapar/SAICJerry Lake/SMi, Jalal Mapar/SAIC

A Change in Culture is Required that A Change in Culture is Required that Promotes and Works Towards Using a Promotes and Works Towards Using a

Systems Approach by All Disciplines not Just Systems Approach by All Disciplines not Just Systems Engineers!Systems Engineers!

Workshop IIWorkshop II-- IVIVNASA-only deliberationsCriteria for NPR developedPreliminary - Final Results of CMMI Pre-AssessmentAdopted 17 Processes as basis for engineFirst cycle of Validation Process for NPR– The NPR draft was tested against the four scenarios (representing

types of projects/science)– Evaluated how NPR will apply to each type– Tests raised some fundamental questions resulted in rewriting of

sections– Design reviews– Applicability of concepts– Product line life cycle

Context for the document and training to accompany NPR– Make Buy discussions

Leverage other activities (APPEL)– Online

““What We FoundWhat We Found””Lack of Uniform UnderstandingLack of Uniform Understanding

of SE in the Communityof SE in the Community--atat--LargeLargeNo single definition or agreement on the scope of SELack of common understanding of how SE is implemented on programs– Is SE done by the systems engineer?– Does the systems engineer lead the SE effort?

No uniform understanding of what makes a good systems engineerNo consistent set of metrics or measures to quantify the value of SECost and schedule estimation and risk management processes inconsistently aligned with SE processesResistance to harmonization of multiple standards and modelsMultiple practioner communities not aligned– Software - Hardware– Information Technology - Aircraft vs. Rocket Developers– Telecommunications - Submarine Propulsion vs. Ship Designers

1. Consistent with existing and emerging NASA policy2. Should have a track record of successful implementation to

engineer systems3. Within the purview of SE4. Written at a level that states “what” not “how” (not overly

prescriptive)5. Within the scope of responsibility of [actors]6. Must be a “requirement” denoted by a “shall” rather than

guidance/informative7. Must be verifiable with objective evidence*8. Allows appropriate tailoring (appendices to define tailoring)9. Improves engineering performance10. Supports the NASA SE initiatives and ongoing improvement

activities 11. Reasonable confidence that projects can accomplish 12. Necessary for consistency across the Agency

NPR CriteriaNPR Criteria

Key Strategies in NASA SE PolicyKey Strategies in NASA SE Policy

Recognize differences in types of NASA projects.

Define a standard design review approach within a common life cycle definition.– Based on product line

Standard SE process that can be applied to any system regardless of scope & scale.

SE NPR

The NPR is a high level NASA Policy The NPR is a high level NASA Policy and Requirements document to support and Requirements document to support Program and Project Management. Program and Project Management.

Process orientedProcess oriented““What to doWhat to do”” vice vice ““how tohow to””

Technical input Technical input Flow down to center directivesFlow down to center directives

PARADIGM SHIFT REQUIRED: Profile PARADIGM SHIFT REQUIRED: Profile of Software NPR target audienceof Software NPR target audience

Early Adopters

ProgressiveUsers

Slow Adopters

EntrenchedResisters

Advances

IEEE 1220 Application & Management of the SE Process

Leve

l of D

etai

l

Breadth of Scope

Scope: SE Related Standards

ISO/IEC 15288System Life Cycle Processes

Envisioned NASA NPGEnvisioned NASA NPG

MIL-STD-499B & EIA IS 632Systems Engineering

ANSI/EIA 632 Processes for Engineering a System

CMMI – includes SW engineering

7123 Structure7123 StructurePreface : Describes applicability, scope, authority, and referencesPrologue: Describes the purpose and vision for this SE NPR.Chapter 1 : Describes the SE framework and introduces the SEMP.Chapter 2 : Describes institutional and programmatic requirements, including roles and responsibilities. Chapter 3 : Describes core set of common technical processes and requirements for engineering NASA system products throughout the product life cycle.

– Appendix C contains supplemental amplifying material.Chapter 4 : Describes activities and requirements to be accomplished by NASA technical teams or individuals (NASA employees and their service support contractors) when performing technical oversight of a prime or external contractor. Chapter 5 : Describes tech reviews throughout the SE life cycles with clear differentiation between management reviews and engineering reviews.

– Exit and Entrance CriteriaChapter 6 : Describes the SEMP, including the role, functions, and content.

– Appendix D provides details of a generic SEMP and annotated outline.Appendices

– A: Definitions– B: Acronyms– C: Practices– D: SEMP– E: Hierarchy– F: Tailoring– G: Technical Reviews– H: Templates– I: Additional reading– J: Index

Requirements Development 1. Stakeholder Expectations

Definition Process2. Technical Requirements

Definition Process

Technical Solution Definition 3. Logical Decomposition

Process4. Design Solution Definition

Process

Product Transition9. Product Transition

Process

Evaluation7. Product Verification

Process8. Product Validation Process

Design Realization5. Product Implementation

Process6. Product Integration Process

Technical Planning10. Technical Planning

Process

Technical Control11. Requirement Mgmt Process12. Interface Management Process13. Technical Risk Mgmt Process14. Configuration Mgmt Process15. Technical Data Mgmt Process

Technical Assessment16. Technical Assessment

Process

Technical Decision Analysis

17. Decision Analysis Process

Requirements Flowdown From WBS Model Above

or From User

Requirements Flowdown From WBS Model Above

or From User

Requirements Flowdown To WBS Models Below or

To Implementation Process

Requirements Flowdown To WBS Models Below or

To Implementation Process

Realized End Products FromWBS Models Below or From

Implementation Process

Realized End Products FromWBS Models Below or From

Implementation Process

Realized End Products To WBS Model Above

or To User

Realized End Products To WBS Model Above

or To User

Applied to each WBS

Model

Applied to each WBS

Model

WBS Model

WBS Model

WBS ModelWBS ModelWBS Model

WBS ModelWBS ModelWBS ModelWBS Model

WBS Model WBS ModelWBS ModelWBS ModelWBS Model

WBS ModelTop Down

System DesignBottom Up

Product Realization

System StructureWBS ModelWBS Model

WBS ModelWBS Model

WBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS Model

WBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS Model

WBS ModelWBS Model WBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS ModelWBS Model

WBS ModelWBS ModelTop Down


Product Realization

System Structure

STS

Orbiter ET SRB

Propulsion

Galley

Crew Cabin Payload Bay Aft Skirt Nose EtcO2 Tank H2 Tank

ECLSS Avionics Etc

TransponderComputers Antenna Etc

17 Common Processes 17 Common Processes ––Flow ExampleFlow Example

Etc Etc Etc Etc EtcEtc Etc

Etc Etc Etc

Etc Etc Etc Etc

Top Down System DesignBottom Up Product Realization

WBS Model

WBS Model




WBS ModelTop Down


Product Realization


WBS ModelWBS Model






Product Realization

System Structure

-

ImplementationImplementationFormulationFormulationProjectProject

-


-


WBS Model

WBS Model




WBS ModelTop Down


Product Realization


WBS ModelWBS Model






Product Realization

System StructureWBS Model

WBS Model




WBS ModelTop Down


Product Realization


WBS ModelWBS Model






Product Realization


WBS Model




WBS ModelTop Down


Product Realization


WBS ModelWBS Model






Product Realization


WBS Model




WBS ModelTop Down


Product Realization


WBS ModelWBS Model






Product Realization

System Structure

Generic System Life Cycle

Iterate Processes Throughout The Life Cycle

Program

Project

Subsystem Subsystem

Component Component

Project

System of Interest

System of Interest

System of Interest

System of Interest & Hierarchy of System of Interest & Hierarchy of SystemsSystems

The systems engineering process can be applied atany level of the systems hierarchy.





Continuous Improvement – Assessments



Tools &Methods

Capab

ility

Capab

ility


AssessmentsAssessmentsTo Establish a systems engineering capability

assessment methodology that enables continuous process improvements in the engineering of systems agency wide.

– Metrics – Maturity models– Pilots– Assessments– Monitor– Feedback results– Updates as required

Background: PreBackground: Pre--Assessment Assessment Response to TaskingResponse to Tasking

SEWG Assessment Sub-group– Mission

To establish a systems engineering capability assessment methodology that enables continuous process improvements in the engineering of systems Agency-wide, with validation and documentation throughout the lifecycle

– Evaluate current SE and SWG capability assessment efforts for CAIB/DIAZ action

Subgroup developed approach, objectives and plans to conduct SE Pre-assessments IAW SE Framework– Researched Industry and other Government Agencies

Approaches – Developed model selection methodology and criteria

A tailored CMMI model selected for pre-assessments

Background: Models ExaminedBackground: Models ExaminedISO 15504 – The international standard assessment methodology for systems engineering

EIA/IS 731 – An Electronic Industries Alliance (EIA) standard that brings together the EPIC Systems Engineering Capability Maturity Model (SE CMM) and the INCOSE Systems Engineering Capability Assessment Model (SECAM) into a single capability model to minimize confusion within the industry and to relate the resulting capability model to the EIA-632 Standard, Processes for Engineering a System.

SE-CMM – The Carnegie Mellon University (CMU) Software Engineering Institute (SEI) capability maturity model for systems engineering

FAA-iCMM, v2.0 – FAA’s own CMMI-based model

CMMI v1.1 SE/SW – This is the latest CMU/SEI capability maturity model that integrates systems engineering and software engineering

Objectives of the PreObjectives of the Pre--assessment assessment To acquire data to determine the state of system engineering as practiced across the Agency in terms of consistency across centers, effectiveness and efficiency

To determine gaps/deficiencies in the system engineering discipline as practiced at the Agency relative to the pre-assessment approach based on a tailored CMMI model

To form a baseline against which future assessments can determine overall trends and provide input/guidance for system engineering process improvements

To determine if the CMMI model meets future Agency system engineering continuous improvement requirements

Class C• Documentation Review

•Identify Gaps• Estimate of Goal Satisfaction

• No rating

Class B• Documentation

• Interviews• Identify Gaps

• Estimate of Goal Satisfaction• No rating

Class A: SCAMPI• Full, comprehensive method• Thorough model coverage• Provides maturity and/or

capability levels

CMMI CMMI AppraisalAppraisal ClassesClasses

Three projects were assessed at each center during a five day period

A tailored CMMI model was used– 13 of 25 Process Areas– Over 260 Practices looked at– The focus was on systems engineering practices

Discovery-based with objective evidence– The findings were based on objective evidence to the question “Can you demonstrate that

you do a practice?” rather than just asking question “Do you do a practice?”– Evidence was provided by the project and examined by a pre-assessment team during

interviews to demonstrate whether or not the practice was performed.

Rules for evidence and corroboration were relaxed from a formal appraisal to enable completion with minimal impact to projects and still enable meeting NASA objectives

Questions were asked of project leads and systems or other engineers to determine their perspectives on systems engineering. These were a supplement to the tailored CMMI model discovery approach.

Overview of PreOverview of Pre--AssessmentAssessment

Process AreasProcess Areas

1 Performed

Requirements ManagementProject PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

BasicProject

Management2 Managed

Organizational Process FocusOrganizational Process DefinitionOrganizational Training Integrated Project ManagementRisk ManagementRequirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidationDecision Analysis and Resolution

ProcessStandardization3 Defined

Organizational Process PerformanceQuantitative Project Management

QuantitativeManagement

4 Quantitatively Managed

Organizational Innovation and DeploymentCausal Analysis and Resolution

Continuous Process Improvement5 Optimizing

Process AreasFocusMaturity Level Quality

Risk &Rework

Detailed AnalysisDetailed AnalysisOverall Agency Performance

Performance in the Specific Practices

Performance in the Generic Practices

Performance in the Generic Practices for Generic Goal 2

Performance in the Generic Practices for Generic Goal 3

Tier One of the Analysis Results (a single over all radar chart) approximates an indicator for the Over All Agency Performance

Tier Two of the Analysis Results (separate SP and combined GPs radar chart) approximates an indicator for how much of the Over All Agency Performance is attributed to people actually doing the "right things" (the SPs) and how much of what they are doing was driven by structured prior planning (the GPs)

Tier Three of the Analysis Results (two separate GG2 and GG3 radar charts) approximates an indicator for how much of that structured prior planning was driven by Project level planning (GG2) and by Center level or Agency level policies and procedures (GG3)

Tier Four of the Analysis Results used qualitative information of gaps and deficiencies as basis for developing recommendations

Pareto charts provide basis for recommendations

How To MethodologiesHow To Methodologies–– NASA NASA BOKBOK

Update and expand Existing SP 6105 – Lesson’s Learned– References

Best practicesStandardsGuides

–TemplatesE-Book





Continuous Improvement – Assessments



Tools &Methods

Capab

ility

Capab

ility


Office of the Chief Engineer Vision for Office of the Chief Engineer Vision for Systems EngineeringSystems Engineering

Vision:Vision: A premier systems engineering capabilityA premier systems engineering capability widely widely recognized for its leadership and expertise in the engineering recognized for its leadership and expertise in the engineering

of systems and subsystems to enable NASA to provide of systems and subsystems to enable NASA to provide leading edge aerospace research, products and servicesleading edge aerospace research, products and services

Mission:Mission: Develop and implement a common SE framework,Develop and implement a common SE framework,and promote the environment for excellence and the and promote the environment for excellence and the

revolutionary advancement of the system engineering revolutionary advancement of the system engineering capability tocapability to anticipateanticipate andand meet the needs of NASAmeet the needs of NASA

Programs and Projects.Programs and Projects.

QuestionsQuestions

Date post:	12-May-2015
Category:	Technology
Upload:	nasapmc
View:	13,870 times
Download:	0 times