Preconstruction Planning forLarge Science Infrastructure Projects
William L Miller, Ph.D.AAAS Science and Technology Policy Fellow
National Science Foundation
[email protected], 703-292-7886
A B
A comparative analysis of practices and challenges
at DOE, NASA and NSF
dSdt
≥ 0
NRC Board on Physics and Astronomy April 24, 2010
Large Science Infrastructure: a highly varied class…
W. Miller 2Preconstruction Planning at DOE, NASA and NSF
Working definition and portfolios studied
DOE Office of Science
• Science user facilities (mainly at national labs)
NASA Science Missions Directorate (SMD)
• Robotic science missions (mainly GSFC, JPL)
NSF Science & Engineering Directorates, OPP
• Large facilities (receive special construction funds)
• Ground-, sea-, air-, or space-based
• Defined projects (not ongoing fleets, programs)
Large multi-user research platforms intended to serve significant
segments of scientific disciplines for discovery-oriented research.
W. Miller 3Preconstruction Planning at DOE, NASA and NSF
Preconstruction Planning at DOE, NASA and NSF 4W. Miller
Major science platforms – key differences from research programs
• Expensive investments: ~ $100 millions to several billions
• Long time scales from planning (10-15 years) to
construction (3-5 yr) to operations (10 - ∞ years)
• Out-year “committed” budgeting over decades = mortgage
• Big Science Big $$ Big decisions
On the OMB/Congress/Community radar screens
Often need to prioritize (across disciplines)
Require a lot of oversight policy and process
Often need to partner
Preconstruction Planning at DOE, NASA and NSF 5W. Miller
Many stakeholders are engaged…
Large ScienceInfrastructure
ScienceEnterprise
TechnologyEnterprise
NationalPriorities
InteragencyPartnerships
InternationalCooperation
Lots of interest in process, performance
and outcomes…
Year-long cross-agency study: Approach
Desired outcomes:
Indications for partnering?
Improvements to NSF practices?
Useful document for developers and stakeholders at NSF & beyond.
W. Miller 6Preconstruction Planning at DOE, NASA and NSF
Questions:
1. How do agencies compare in
their infrastructure enterprises?
2. How do their development
processes align?
3. Commonalities and differences
in practices and challenges?
Sources:
• Agency documents,
other reports & analyses
(GAO, NRC, Rand, industry…)
• Look “under the hood”:
~45 stakeholder interviews,
design reviews, site visits,
advisory committee meetings, …
Comparative anatomy: Differences and similarities…
W. Miller 7Preconstruction Planning at DOE, NASA and NSF
Comparative anatomy: Agency structures for science
FPD Federal Project Director
OECM Office of Engineering and Construction Mgmt
OPA Office of Project Assessment
SRB Standing Review Board
LFO Large Facilities Office
NSB National Science Board
S&E Science and engineeringPolicy and process
Independent review
$26.6B
$4.9B $4.5B
$18.7B
$2.1B
$6.9B
$5.7B
Budgets: FY 2010 approps, from FY2011 Congressional Requests
Oversight bodies
DOESecretary
Program Offices
Laboratories
Projects
OPA
Programs
Centers
Projects
Science Missions
Directorate
NSFDirector
Programs
Awardees
Projects
S&E Directorates
& Offices
NASAAdministrator
Chief
Engineer
NSB
SRBs PanelsDivisions
Divisions
UnderSec
Divisions
42 U.S.C.§1873b
(“shall not operate
laboratories”)
Ad
min
istr
atio
nP
rog
ram
sP
roje
cts
FPDs
Ofc of Science
Management
Support Office
OECM
Budget, Finance
& Award Mgmt
LFO
W. Miller 8Preconstruction Planning at DOE, NASA and NSF
Key agency character differences
Internal Resources
• DOE and NASA have infrastructure-intensive MISSONS, and in-house project
resources (Labs, Centers).
• NSF mainly funds research grants, and “shall not operate laboratories” (by statute,
42 U.S.C.§1873b).
Project-related factors
• DOE has extensive legacy infrastructure (National Labs): can leverage for new
projects, but cost a lot to maintain….
• NASA spacecraft operate remotely in harsh environments, with high risk of
catastrophic failure ~ Drives extremes of planning, test … and oversight.
• NSF large facilities are highly varied – giant telescopes to distributed sensor
networks – and in disciplines new to multi-user facilities (e.g. Seismology, Biology).
W. Miller 9Preconstruction Planning at DOE, NASA and NSF
Preconstruction Planning at DOE, NASA and NSF 10W. Miller
Development process framework focus on planning
Early
acquisitions
Early
operations
Project
CloseoutTerminate
Facility
Development
Conceptualization Construction DOperations
Begin major
Investment
Conceptual
Preconstruction Planning
PreliminaryFinal
design
Ongoing community input, horizon planning,
reprioritization, and portfolio adjustment
Preconstruction planning (also called definition & design, formulation…)
• Plan Management/governance plans, WBS assemble project team
• Design Goals, requirements iterative design bring to readiness
• Invest R&D, necessary technologies bring to readiness
• Estimate Effort, cost, schedule, reserves, risks refine to believability
• Govern Stand-up and start oversight & decision-making structures/processes
Development policies & procedures are captured formally
Title ID Originator Released
DOE
“Program and Project
Management for the
Acquisition Of Capital Assets”
Order
413.3A
Office of Engineering and
Construction ManagementJuly 2006
NASA
“Space Flight Program &
Project Management
Requirements”
NPR
7120.5D
Office of the
Chief Engineer
Mar 2007(under
revision)
NSF “Large Facilities Manual”NSF
10-12Large Facilities Office
Nov 2009(update from
May 2007)
Top-level “best practices” for science platform development,
issued by project policy/assurance offices
All are recent and still evolving (e.g. NASA, NSF)
Processes interpreted and implemented by the sponsoring programs
W. Miller 11Preconstruction Planning at DOE, NASA and NSF
12Preconstruction Planning at DOE, NASA and NSF
ImplementationFormulation
Concept
Studies
Prelim Design &
Tech Completion
Final Design &
Fabrication*
Assembly,
Integ & Test,
LaunchOperations
Concept &
Tech Devel
Pre-Phase A Phase A Phase B Phase C Phase D Phase E
KDP-A KDP-B KDP-C KDP-D KDP-ENASA-SMD Ref: NASA NPR 7120.5D
Development process crosswalk: Terminology and alignment
MCR MDR
SRR
PDR CDR SIR
ORR
FRR
LRRPLAR
CERRCDR Critical Design Review
CERR Critical Events Readiness Review
FRR Flight Readiness Review
KDP Key Decision Point
LRR Launch Readiness Review
MCR Mission Concept Review
MDR Mission Definition Review
ORR Operational Readiness Review
PDR Preliminary Design Review
PLAR Post-Launch Assessment Review
SDR System Definition Review
SIR System Integration Review
SRR System Requirements Review
*elongated to visually align NASA, NSF & DOE
equivalent events. Sequences on this chart do not
represent typical or relative phase durations.
Approve Implementation at KDP-C
CDR PDR FDR Ops
NSB Approved
Preliminary
DesignFinal Design Construction Operations
Horizon planning and
Conceptual Design
Readiness
NSFRef: NSF 0738
CDR Conceptual Design Review
FDR Final Design Review
PDR Preliminary Design Review
Ops Operations Review
Approvals
Post-CDR Approve advance to Readiness
Post-PDR Approve submission to Nat. Science Board (NSB)
Post-FDR Congress appropriates MREFC funds
Post-Ops Approve Operations start
IPR** CDR EIR
PDR
FDR
IPR/EIR**
ORR/RA
**CD-0 IPR and CD-3 EIR for >$750M projects
CDR Conceptual Design Review
EIR External Independent Review (OECM)
FDR Final Design Review
IPR Independent Project Review (SC)
ORR Operations Readiness Review
PDR Preliminary Design Review
RA Readiness Assessment
Critical Decision (CD) approvals
CD-0 Approve mission need
CD-1 Approve Alternatives selection & cost range
CD-2 Approve Performance baseline
CD-3 Approve Construction start
CD-4 Approve Operations start
DOE-SCRef: DOE O 413.3A
Execution
Pre-conceptual
Planning
Trans/Closeout
OperationsConceptual
Design
CD-0 CD-1 CD-2 CD-4CD-3
Initiation Definition
Preliminary
DesignFinal Design Construction
DecisionReview
W. Miller
Process On-Ramps (when does a project start?)
DOE &
NASA
• Projects may be identified in long-range strategic/science plans1,2
based on community indications (e.g. Decadal Surveys, roadmaps)
• Establish “mission need” & feasibility at milestone #1 (CD-0, KDP-A)
NSF• Peer-reviewed (un)solicited proposals, workshops, studies, etc.
• Evolved concepts may be brought to development: ~ CDR
• “Mission-driven” – projects determined via strategically-defined goals and priorities
1. Facilities for the Future of Science, A twenty year outlook, DOE/SC-0078, Dec 2003; and Four Years Later: an Interim
Report on Facilities for the Future…, Aug 2007. 2. NASA Strategic Plan, 2006; and NASA Science Plan 2007–2016
• “Community-driven” – projects “bubble up” from the scientific disciplines
Horizon concepts,
prioritization Conceptual Preliminary Final
Preconstruction Planning
?
W. Miller 13Preconstruction Planning at DOE, NASA and NSF
• Agencies usually “commit” after PDR (but momentum is needed to get there…)
Governance, oversight and approval
Independent
Review
Internal decision
support
Sign Off
(e.g. Implement Project)
NASA
PDR
by Standing Review
Board
Program Mgmt Council
• Program Manager
• Center Mgmt Council
• Technical Authority
• Project Manager
KDP-C
Decision Authority (DA)(AA for Cat 1, otherwise MDAA)
Approve Implementation
(Enter Phase C – Final Design & Fab)
NSF
PDR
by Review Panel
NSF Director
MREFC* Panel
• Program Officer
• Directorate/Division
• DDLFP, CFO
NSB Approval
for inclusion in a future
budget request
DOE
PDR
by OPA
“Lehman Review”
(also: EIR by OECM)
Energy Systems Acquisition
Advisory Board (ESAAB)
• OPA briefs AE, calls ESAAB
• Federal Project Director
CD-2
Acquisition Exec (AE)
(DepSec, US Sci, SC Dir/AD)
Approve Performance Baseline
(Enter Final Design Stage)
* Major Research Equipment and Facilities Construction W. Miller 14Preconstruction Planning at DOE, NASA and NSF
Funding streams
MO&DAProgram Funds, project line-item
Mission Operations
and Data Analysis
Execution
Pre-conceptual
PlanningTrans/Closeout
Operations
Conceptual
Design
CD-0 CD-1 CD-2 CD-4CD-3
Initiation Definition
Preliminary
DesignFinal Design Construction
NSB Approved
Preliminary
DesignFinal Design Construction Operations
Horizon planning and
Conceptual Design
Readiness
R&RAProgram Funds (Research & Related Activity, R&RA) MREFC
Major Research Equipment
and Facilities Construction
Concept
Studies
Prelim Design &
Tech Completion
Final Design &
Fabrication
Assembly,
Integ & Test,
LaunchOperations
Concept &
Tech Devel
Pre-Phase A Phase A Phase B Phase C Phase D Phase E
KDP-A KDP-B KDP-C KDP-D KDP-E
Formulation Implementation
Requires separate appropriation
Program Ops FundsLIC Prelim Engr & Design (PED)
Program Funds
Two modes for intramural projects: Line Item Construction (LIC) or
Major Item of Equipment (MIE, no major construction)
Major Item of Equipment (MIE)
LIC Construction
NASA-SMD
NSF
DOE-SC
• Separate funds support research (Research and Analysis, R&A)
• Same funds support research (Research & Related Activities, R&RA)
• Separate funds support research
W. Miller 15Preconstruction Planning at DOE, NASA and NSF
Challenges: Estimating project cost
• When to state:
Recent reports
on NASA data
by GAO, NASA
and Industry
Desire to state
costs LATER
Pressure to state
costs EARLIER
• Strategic Plans, Surveys etc.
• Manage portfolios
• Request appropriations
• Understand project
• Improve estimates
• Track realistically
Planning stage
• Unpredictable external factors: Funding cuts and delays. Partner delivery and contribution issues. Fuel costs for distributed systems. Exchange rates.
• Unpredictable internal factors: Reprioritizations, overruns of other projects can impact available funds and resources. (Delays may add useful time but at a cost).
• Substantial cost increases can occur late in project:– Overoptimistic technology readiness
– Unmaterialized savings from use of “heritage” designs
– Underestimated difficulties during system I&T and verification
– instrument development slow or unmanaged
W. Miller 16Preconstruction Planning at DOE, NASA and NSF
Challenges: Portfolio management analysis
• For large portfolios, problems on one project can ripple through others, and out-years are hard(er) to predict.
• Challenge increases with portfolio size and project complexity.
• Need to do quantitative (retrospective and prospective) analyses, e.g.:
– Spending by stage?, is this changing?
– Funding profile dependence on project type, complexity, cost….
– Outcomes vs. initial baselines, cost estimates
Portfolio analysis can
– more fully describe the portfolio
– inform decisions on future projects, and
– serve as comparison with other agencies.
W. Miller 17Preconstruction Planning at DOE, NASA and NSF
Portfolio analysis: Advantages of a good database…
Source: David Bearden, Presentation to NASA/GSFC, June 3, 2008
W. Miller 18Preconstruction Planning at DOE, NASA and NSF
Other challenges…
• Workforce training new generation
o Another Federal dip (many retirements after years of flat budgets –
recall the mid-1980s) Losing corporate memory.
o Scarcity of seasoned program/project managers for large projects*
o NASA APPEL and annual PM Challenges are great models…
• Collecting and disseminating Lessons Learned
o NASA – required for all development activities (NPR 7120.6);
public database: http://llis.nasa.gov/offices/oce/llis/home/
o DOE – on-line LL repositories, project closeout reports
o NSF – some activities by programs and awardees
• Can these activities be leveraged through sharing?
*e.g. DOE whitepaper on Federal Project Directors, www.er.doe.gov/opa/pdf/SC%20FPD%20Leadership%20Skills%20rev5.pdf
W. Miller 19Preconstruction Planning at DOE, NASA and NSF
Preconstruction Planning at DOE, NASA and NSF 20W. Miller
Challenges: Partnering
Inter-agency partnering: Joint Oversight Groups (JOGs)
• DOE-NASA: FERMI, Joint Dark Energy Mission
• NSF-DOE: U.S. Large Hadron Collider (LHC)
International partnering:
• A higher dimension of “opportunity cost”
Preconstruction Planning at DOE, NASA and NSF 21W. Miller
-Source: R. Staffin, 14Feb2006, FY06 presentation to HEPAP, www.er.doe.gov/hep/files/pdfs/HEPAPFeb142005Staffin.pdf
Challenges: Partnering impedance (mis)matches
Strategic
valuation
Technical
domains
Management
practices
Capabilities
& Practices
Better planning could involve assessment of COMPLEMENTARITY across
capabilities, practices, lessons learned, etc…)
Preconstruction Planning at DOE, NASA and NSF 22W. Miller
Synthesis of best practices for partnering…
• Early engagement. Agree on clear goals, timeframe, and effective membership rules and governance structure.
– Clear leadership. Having a dominant partner may work best…
– Complementarity assessment. Learn partner enterprises and realities. Identify technical domains required and respective partner(s) capabilities.
• Clear (shared) process. Agency processes align well, but ideally want an integrated review/decision process (i.e. more than a JOG).
– Strong project management team. Aim for a single team, in place before funding begins, with effective budget authority (not just a coordinator).
– Transparent budgeting. Adopt standard costing and budgeting techniques.
• Open knowledge exchange. Identify effective interfaces at all levels. Maintain open, frequent and honest communication.
Cross-agency training? Shared lessons learned?
Reference material
DOE Lifecycle Process – from O413.3A
Source: S. Meador, Feb 2009, based on DOE Order 413.3A
W. Miller 24Preconstruction Planning at DOE, NASA and NSF
25W. MillerPreconstruction Planning at DOE, NASA and NSF
Conceptual Design Stage Readiness Stage Board Approved Stage Construction
Concept development – Expend approximately
1/3 of total pre-construction planning budget
Develop construction budget based on
conceptual design
Develop budget requirements for advanced
planning
Estimate ops $
Preliminary design
Expend approx 1/3 of total pre-
construction planning budget
Construction estimate based on
prelim design
Update ops $ estimate
Final design over ~ 2 years
Expend approx 1/3 of total pre-
construction planning budget
Construction-ready budget &
contingency estimates
Preliminary Design
Develop site-specific preliminary
design, environmental impacts
Develop enabling technology
Bottoms-up cost and contingency
estimates, updated risk analysis
Develop preliminary operations cost
estimate
Develop Project Management Control
System
Update of Project Execution Plan
Final Design
Development of final construction-
ready design and Project Execution
Plan
Industrialize key technologies
Refine bottoms-up cost and
contingency estimates
Finalize Risk Assessment and
Mitigation, and Management Plan
Complete recruitment of key staff
Conceptual design
Formulation of science questions
Requirements definition, prioritization,
and review
Identify critical enabling technologies and
high risk items
Development of conceptual design
Top down parametric cost and
contingency estimates
Formulate initial risk assessment
Initial proposal submission to NSF
Initial draft of Project Execution Plan
Construction per
baseline
Pro
ject e
vo
lutio
nB
ud
ge
t e
vo
lutio
nO
ve
rsig
ht e
vo
lutio
n
Merit review, apply 1st and 2nd ranking
criteria
MREFC Panel briefings
Forward estimates of Preliminary Design
costs and schedules
Establishment of interim review schedules
and competition milestones
Forecast international and interagency
participation and constraints
Initial consideration of NSF risks and
opportunities
Conceptual design review
NSF Director approves Internal
Management Plan
Formulate/approve Project
Development Plan & budget;
include in NSF Facilities Plan
Preliminary design review and
integrated baseline review
Evaluate ops $ projections
Evaluate forward design costs
and schedules
Forecast interagency and
international decision
milestones
NSF approves submission to
NSB
Apply 3rd ranking criteria
NSB prioritization
OMB/Congress budget
negotiations based on Prelim
design budget
Semi-annual reassessment of
baseline and projected ops
budget for projects not started
construction
Finalization of interagency and
international requirements
Final design review, fix
baseline
Congress appropriates
MREFC funds & NSB
approves obligation
Periodic external review during
construction
Review of project reporting
Site visit and assessment
MREFC $
Expenditure of budget and
contingency per baseline
Refine ops budget
MR
EF
C P
an
el re
co
mm
en
ds a
nd
NS
F D
ire
cto
r
ap
pro
ve
s a
dva
nce
to
Re
ad
ine
ss
NS
F a
ppro
ves s
ubm
issio
n to N
SB
Congre
ss a
ppro
priate
s funds
Funded by R&RA or EHR $
NSF oversight defined in Internal Management Plan, updated by development phase
Proponents development strategy defined in Project Development Plan Described by Project Execution Plan
Source: NSF Large Facilities Manual
NSF vs. Doe Process
Conceptual Design
Preliminary Design
Final Design
Construction
Operations
R&RA $ R&RA $ R&RA $ R&RA $
Approximate DOE Translation:
CD 0 CD 1 CD 2 CD 3 CD 4
Approve
mission need
Approve
alternate
selection and
cost range
Approve
performance
baseline
Approve
construction
start
Approve
operations
start
CDR PDR FDROperations
Review
Science
Review
Renewal
Review,
etc.
MREFC $
-- Source: M. Coles, NSF
W. Miller 26Preconstruction Planning at DOE, NASA and NSF
NASA Project Life Cycle Process – from NPR 7120.5D
W. Miller 27Preconstruction Planning at DOE, NASA and NSF
Authorities and bodies engaged in project oversight
W. Miller 28Preconstruction Planning at DOE, NASA and NSF
Oversight: Project categorization sets governance level
DOE Cost categories: 5-20, 20-100, 100-400, 400-750, >750 ($M)
• Approval level depends on cost category
• Highest category requires DOE Deputy Secretary approval
NASA Cost Categories: based on Life Cycle Cost Estimate (LCCE):
(1) > $1B (or nuclear powered or human spaceflight)
(2) $250M to $1B
(3) < $250M (High Priority projects may be Cat 2)
• Approval level depends on project category
• Highest category requires NASA Administrator approval
NSF Either MREFC (eligibility based on cost*) or non-MREFC
• MREFC approvals by Director, National Science Board
• Non-MREFC projects governed within Directorates (but may
require NSB approval depending on cost)
*estimated cost exceeds 10% of a directorates budget
W. Miller 29Preconstruction Planning at DOE, NASA and NSF
Governance and Oversight: Independent review
DOE Expert Panels
• Run by SC Office of Program Assessment (OPA)
• Chair always a DOE federal employee (usually from OPA)
• Evolving membership of DOE staff, contractors, other experts; usually a core
group, but introduce “new blood” as project evolves
NASA Standing Review Board (SRB)
• Coordination: Independent Program Assessment & Oversight Office (IPAO)
• Chair vetted and agreed by project, program, Decision Authority
• Fixed membership of NASA staff, contractors, other experts (selected by chair)
follows entire lifecycle, including subsystem reviews. Add experts as needed.
NSF Expert Panels
• Assembled for each review by Program Officer with assistance of LFO
director.
• Changing membership of external technical and scientific experts, but often try
to keep a core group for institutional memory.
W. Miller 30Preconstruction Planning at DOE, NASA and NSF
Funding and acquisitions
DOE NASA NSF
Ownership Primarily
owner/operator
Primarily
owner/operator
Not owner/operator (except
polar)
Expenditure vehicles
In-house
projectsContracts
Contracts.
CAs often for ops
Contracts
(Polar Programs)
External
projects
Recent CA with MSU for
FRIB developmentSame as in-house
CAs for development and
operations
Carryover Funds are “no year”Funds are 2-year
(reduced for FY 2010?)
No R&RA carryover
MREFC construction funds
are “no year”
Acquisition
Start
DOE and NASA can acquire “long-lead items” during
initial development (after CD-1 and KDP-B,
respectively), per approved project plans.
No acquisition until MREFC
funds are appropriated, NSB-
approved and awarded
(after FDR)
CA – Cooperative Agreement
W. Miller 31Preconstruction Planning at DOE, NASA and NSF
Acronyms
CD Critical Decision (DOE)
CDR Conceptual Design Review (DOE, NSF)
Critical Design Review (NASA)
DDLFP Deputy Director for Large Facility Projects, head of LFO (NSF)
EIR External Independent Review (DOE OECM)
IPAO Independent Program Assessment & Oversight Office (NASA)
JOG Joint Oversight Group
KDP Key Decision Point (NASA)
LFO Large Facilities Office (NSF)
MO&DA Mission Operations and Data Analysis funding account (NASA)
MREFC Major Research Equipment and Facilities Construction funding account (NSF)
NAR Non-Advocate Review (PDR, NASA)
OECM Office of Engineering and Construction Management (DOE)
OPA Office of Program Assessment (DOE/SC)
PED Preliminary Engineering and Design funding account (DOE)
PDR Preliminary Design Review
PNAR Preliminary Non-Advocate Review (MDR, NASA)
R&RA Research and Related Activities funding account (NSF)
R&A Research and Analysis funding account (NASA)
SC Office of Science (DOE)
SRB Standing Review Board (NASA)
SMD Science Missions Directorate (NASA)
W. Miller 32Preconstruction Planning at DOE, NASA and NSF