8/12/2019 coles_lfp_11_04

1/25

Planning and Oversight of NSFs Large Facility

Projects

Mark Coles

Deputy Director for Large Facility Projects

Office of Budget, Finance and Award Management

8/12/2019 coles_lfp_11_04

2/25

2

Introduction

National Academies study Setting Priorities for Large Research FacilityProjects (Brinkman report) makes specific recommendations forprioritizing, planning, and overseeing the construction and operation oflarge facilities supported by NSF.

The report cites the need to make greater resources available for pre-award planning of a projects workscope, and

Implement rigorous post-award oversight through periodic externalreview

National Science Board and National Science Foundation joint responseembraces this recommendation.

Purpose of this presentation is to discuss howthis should be implemented and solicit input

8/12/2019 coles_lfp_11_04

3/25

3

Summary of the NAS Study Recommendations

Development of a 10-20 year facility roadmap

Three levels of criteria for ranking:

Scientific and technical merit

Agency Strategic Criteria National Criteria

New starts ranked in annual budget request using clear rationale basedon roadmap

Enhanced project pre-approval planning and budgeting

Greater independent oversight and review needed Review effectiveness of Deputy Director for Large Facilities in two years

OSTP should coordinates roadmaps across agencies

NSF Leadership and NSB should pay careful attention toimplementation of proposed reforms

8/12/2019 coles_lfp_11_04

4/25

4

NAS Study Findings

There is a lack of funding for disciplines toconduct idea-generating and project-ranking

activities and, once ideas have some level ofapproval, a lack of funding for conceptualdevelopment, planning, engineering, anddesigninformation needed when judging

whether a project is ready for funding in lightof its ranking and for preparing a project forfunding if it is selected.

8/12/2019 coles_lfp_11_04

5/25

5

NSB defines a new process for approvingreadiness of a project:

Project plans are judged to be construction ready bythe Large Facility Deputy,

The budget for construction and for operations costs

has been justified to the satisfaction of the ChiefFinancial Officer

From: Attachment 5 to NSB-04-92, NSB-04-97

What are the criteria for ascertaining readinessof plans and budget?

8/12/2019 coles_lfp_11_04

6/25

6

Readiness certification process needs to be defined

Goal: NSB approves projects that are well definedand that have thorough cost estimates

Present situation:

Project baseline, including well scrutinized bottoms-upcontingency estimate, is established after construction.

Many projects are de-scoped during construction (examples:LIGO, Gemini, IceCube, likely ALMA)

Development of a project baseline (planning, costing,enabling R&D, PMCS, risk assessment, staffing)requires ~5-10% of total project cost.

8/12/2019 coles_lfp_11_04

7/25

7

Recommendation: Develop project baselineprior to approval of construction award

Project baseline:

Detailed technical workscope flowdown from sciencerequirements

Bottoms-up budget estimate

Resource loaded schedule with realistic budget profile andcritical path

Thorough risk assessment and item-by-item contingencyestimate

Robust project execution plan for project management, andstaffing commitments for key work package managers

8/12/2019 coles_lfp_11_04

8/25

8

Benefits of defining project baseline prior toconstruction award

Development of a robust project baseline reduces uncertainty inproject workscope.

Need to make an earlier commitment of significant developmentfunding to a project, with expectation that construction approvalis likely if pre-defined development criteria are satisfied.

Internal Management Plan (aka Project Development Plan)could lay out predefined criteria that must be satisfied. Forexample: Completion and successful review of preparatory research

Satisfactory demonstration of enabling technology Develop of detailed bottoms-up cost and contingency estimates,

resource loaded schedule, recruitment of key personnel, ProjectManagement Plan, and other components of Project Baseline,followed by successful external baseline review.

8/12/2019 coles_lfp_11_04

9/25

9

More planning resources will:

Reduce uncertainties in construction and operations budgets

Reduce uncertainty in construction schedule

Reduce likelihood of de-scoping

Thorough pre-construction risk assessment will create a more robustProject Execution Plan

Create an appropriate framework for cooperation for activities involvinginternational and inter-agency partnerships as an integral part of in-depth planning.

Make project oversight during construction more straightforward, add

more definition to the proposed workscope and decision points. Allow more in-depth consideration of transition to operations and

greater certainty in predicted O&M costs.

Avoid a funding hiatus.Overlap some development activities beyondNSB approval and inclusion in annual NSF budget submission.

8/12/2019 coles_lfp_11_04

10/25

10

Other factors to consider

Greater involvement by NSF OD, NSB, OMB, Congress, in theearly stage of a project - prior to entry into Readiness Stage.

Need to provide visible, but not risk-free, level of commitment

to a project during the readiness phase so that a project canrecruit engineering and managerial staff and undertake enablingR&D, detailed plans, cost estimates.

Need to set an expectation that a project will go forward intoconstruction from the readiness phase if it meets the criteria

established in its Internal Management Plan (and that if can fallout, or terminate, if unsuccessful)

8/12/2019 coles_lfp_11_04

11/25

11

Suggested Implementation

Prior to entering Readiness phase a project musthave: Scientific merit satisfying the first two NAS criteriascientific

merit and NSF priority

Preliminary Project Plan: Top down estimates of technical scope, budget, contingency,

Detailed plan for enabling R&D, planning, technicaldemonstrationsProject Development Plan

Detailed schedule with identified decision point and plan fordetermining construction readiness.

Prioritization of readiness activities with window for constructionapproval, to provide latitude for timing of construction approval

Approval by Facilities Panel of Project Development Plan

Approval by MREFC Panel of overall activity, timing, decisionpoints, etc.

8/12/2019 coles_lfp_11_04

12/25

12

Implementation:Requirements to enter Readiness Phase

Review and approval of: Preliminary technical scope,

Top-down budget and contingency estimates,

Project development plan

Project development schedule, including proposed decision point window Presentation and approval to NSF OD and NSB of project for entry into

Readiness phase

Determination by NSF OD and NSB that project meets third level NAScriterianational importance

Inform OMB and Congress of:

Projects potential construction scope

Development schedule

Window for construction approval (and allow for tuning to meet futurebudget envelope expectations)

8/12/2019 coles_lfp_11_04

13/25

13

While in the Readiness Phase

Regular status reports to NSF and NSB

Periodic updates to OMB and Congress of project status

External reviews of development milestones as appropriate

Baseline review to establish construction readiness and budgetreadiness

Request to NSF OD and NSB to exit Readiness Phase uponsuccessful completion of baseline review

Inform OMB and Congress of successful completion of baseline

review and request inclusion of projects construction budgetwithin window

Projects that fail to meet development milestones may bedelayed or terminated, as appropriate.

8/12/2019 coles_lfp_11_04

14/25

14

Exiting Readiness Phase

Construction activity begins

Construction oversight by NSF begins:

Monthly status reports

Periodic external reviews of construction activity

Development activities conclude

Periodic updates to inform NSF OD, NSB, OMB,Congress of project status

8/12/2019 coles_lfp_11_04

15/25

15

Summary Impacts on NSF

Greater commitment of R&RA funds for extensiveplanning

Involvement of NSF Director, OMB, Congress frombeginning of the intensive planning stage, rather thanat the end

Implementation of formally approved projectdevelopment plans

Greater responsibility for evaluation of projecttechnical plans and budget by CFO and DeputyDirector for Large Facilities

8/12/2019 coles_lfp_11_04

16/25

16

Oversight during construction

Practices for post-award oversight vary greatly across foundation

External review by slowly varying panels with a great breadth ofrelevant experience are used mostly in Physics.

Other disciplines rely on self-assessment, or narrowly constitutedadvisory panels from the target research field

How can we promote implementation of periodic external review by apanel of scientific, engineering, management, and technical experts?(As recommended by NAS report)

8/12/2019 coles_lfp_11_04

17/25

17

Need for further detailed advice

NSFs Facilities Management and Oversight Guide willbe revised as part of NSF/NSB response to NAS study

Other findings from NAS study also present

opportunities to improve NSFs capabilities: Post award oversight

Full life-cycle cost accounting

Need for guidance on how to develop and implementthese policies, particularly in areas where there hasnot been a tradition of: Construction of large facilities

Use of power tools for technical and financial reporting

Periodic post-award external review

8/12/2019 coles_lfp_11_04

18/25

18

Facilities Subcommittee

With approval of Business and OperationsCommittee, propose to establish FacilitiesSubcommittee as a subsidiary of the Business and

Operations Committee to provide specific guidanceon these policy issues

Facilities Subcommittee will report its findings to thefull Business and Operations Committee

Subcommittee will meet twice per year

Plan to have five members of subcommittee, all withsignificant experience with management of largescientific facilities

8/12/2019 coles_lfp_11_04

19/25

19

Facilities Subcommittee Charter

The Facilities Subcommittee will examine the business practicesthe NSF applies to planning and executing major researchfacility projects, including: NSF policies and procedures for pre-construction planning and

evaluation of proposed large facility projects, Non-technical aspects of the oversight practices applied by NSF

during construction of large facilities, and

Practices used to monitor business operations aspects of thefacilities.

The Subcommittee will make recommendations to the Businessand Operations Advisory Committee as to the adequacy ofcurrent practices, and will make recommendations for ways thatthe NSF can improve current practices to enhance theireffectiveness.

8/12/2019 coles_lfp_11_04

20/25

20

Proposed Facilities Subcommittee Membership

The following individuals have agreed to serve on theSubcommittee if approved by the Business and OperationsCommittee: Tom Kirk (Chair of Subcommittee and member of B&O Committee)

Associate Laboratory Director of High Energy and Nuclear Physics,Brookhaven National Laboratory

Jim Yeck Project Director, IceCube Project, Univ. of Wisconsin, Madison

Janos Kirz Acting Director, Advanced Light Source, Lawrence Berkeley National

Laboratory Paul Gilbert

Chairman Emeritus, Parsons Brinckerhoff Quade & Douglas, Inc.

Larry Ramsey Professor of Astronomy and Astrophysics at Penn State, and Project

Scientist on Hobby-Eberly Large Telescope

8/12/2019 coles_lfp_11_04

21/25

21

How the B&O Committee can help

Recommend business processes for ascertainingreadiness of budget and construction plans

Recommend how best to promote theimplementation of periodic external review duringconstruction

Approve charter and membership of FacilitiesSubcommittee

8/12/2019 coles_lfp_11_04

22/25

Backup materials

8/12/2019 coles_lfp_11_04

23/25

23

NSF/NSB Response:

The National Academies Report properly calls attention to thenecessity for considerable pre-approval funding for planning anddevelopment when it questions whether there is sufficient NSFsupport for this "bottom up" process.NSF endorses the Reports

recommendations to provide researchers access to fundingsufficient to develop compelling research agendas, to refine andprioritize their facility requirements, and to complete researchand development on facility designs and needed technologies.The level and form of funding for planning and development willbe reviewed, and an evaluation will be made of how project

funds are best invested to attain robust plans and scheduleswith better cost projections, so that only well-defined andthoroughly-costed projects are brought forward forconsideration by the Board.

8/12/2019 coles_lfp_11_04

24/25

24

More NSF/NSB Response:

The Director and the Board recognize the need tostrengthen oversight of the implementation of largefacility projects, which will require increased

investments of NSF staff time and travel funds. TheReport emphasizes the importance of initial planningand definition of technical scope, budget, andschedule, followed by periodic post-award status

reviews held on-site by external experts, withimplementation of a transparent process formanagement of changes to a projectsimplementation plan.

8/12/2019 coles_lfp_11_04

25/25

25

And more NSF/NSB response

On the basis of merit-reviewed proposals, NSF willfund these planning and project developmentefforts.

In accepting the Facility Plan the NSB will concurthat each of the Readiness List projects has attainedthat status by an appropriate process