Closeout Report on the DOE/SC Status Review of the...review to eliminate issues with LFD in the...

OFFICE OF

SCIENCE

1

Closeout Report on the

DOE/SC Status Review of the

Proton Improvement Plan (PIP-II)

Fermi National Accelerator LaboratoryDecember 4-6, 2018

Kurt Fisher

Committee Chair

Office of Science, U.S. Department of Energy

http://www.science.doe.gov/opa/

http://www.science.doe.gov/opa/

OFFICE OF

SCIENCEReview Committee

Participants

2

Kurt Fisher, DOE/SC, Chairperson

SC1 SC2 SC3 SC4

Accelerator Upgrades,

Accelerator Systems Linac and I&C SRF and Cryogenics Conventional Facilities

* Ali Nassiri, ANL * Thomas Roser, BNL * Matt Howell, ORNL * Brad Bull, MSU

Chris Adolphsen, SLAC Peter Ostroumov, MSU Brian DeGraff, ORNL Gary Bloom, ORNL

Matthew Bickley, TJNAF Marion White, ANL Ting Xu, MSU

Charles Reece, TJNAF

SC5 SC6 SC7

Env, Safety and Health Cost and Schedule Project Management

* Dave Rodgers, LBNL * Ethan Merrill, DOE/OPA * Mark Reichanadter, SLAC

Andrew Ackerman, BNL Frank Gines, DOE/ASO Greg Hays, SLAC

Lanson Oukrop, PNNL Joe Ingraffia, retired ANL

Robbie Leftwich-Vann, LBNL

Steve Meador, DOE/OPA

Observers LEGEND

Mike Procario, DOE/HEP SC Subcommittee

Ted Lavine, DOE/HEP * Chairperson

Mike Harrison, DOE/HEP [ ] Part-time Subcom. Member

Adam Bihary, DOE/FSO

Pepin Carolan, DOE/FSO Count: 23 (excluding observers)

OFFICE OF

SCIENCE

3

Charge Questions

1. Is the project making adequate technical progress to ensure that the completed project will

perform as planned and meet the key performance parameters?

2. Will execution of PIP-II design plans and planned R&D program activities ensure most major

technical risks will be appropriately mitigated or retired prior to CD-3?

3. Has the project made adequate progress on its resource-loaded schedule to complete it by the

time of CD-2?

4. Are preparations for defining, documenting, and managing the international in-kind

contributions suitable to ensure their timely delivery and technical fidelity?

5. Is the proposed CD-2 timeline reasonable and consistent with the current project status?

6. Is ESH&Q being handled appropriately?

7. Are the proposed risk mitigation strategies reasonable and are the proposed contingencies

acceptable?

8. Has the project satisfactorily responded to the recommendations from previous reviews?

9. Are there any other significant issues that require HEP or project’s attention?

OFFICE OF

SCIENCE

4

2.1 Accelerator Systems

C. Adolphsen (SLAC), M. Bickley (TJNAF)

and A. Nassiri (ANL) Subcommittee 1

1. Is the project making adequate technical progress to ensure that the

completed project will perform as planned and meet the key

performance parameters? Good progress has been made on the technical

front since CD-1. However the TRS’s are not adequately supported by

physics specification documents. In the near term, the project should

focus more on shoring up the design for CD-2 than production level

activities for post CD-2.

2. Will execution of PIP-II design plans and planned R&D program

activities ensure most major technical risks will be appropriately

mitigated or retired prior to CD-3? Yes.

8. Has the project satisfactorily responded to the recommendations from

previous reviews? The team has addressed some and others are to be

done by CD-2.

9. Are there any other significant issues that require HEP or project’s

attention? DOE should review the appropriateness of the KPPs in light

of other improvement programs that are aimed at increasing the MI

beam power.

OFFICE OF

SCIENCE

5


A. Nassiri, ANL / Subcommittee 1

Findings

• The PIP-II 800 MeV, SC linac in combination with the Booster, RR

and MI is to provide 1.2 MW of proton beam power before the start

of DUNE.

• The linac is also being built to support future CW beam operation

for multi-MW, few-GeV physics programs.

• CW RF operation with pulsed beams was suggested at the last

review to eliminate issues with LFD in the narrow-band SC cavities

– it is now the baseline design.

• The PIP-II linac utilizes both room-temperature and

superconducting technology – there are three classes of SC cavities:

HWR, SSR and Elliptical.

• An arrangement of four beam scrapers will be used in SRF linac to

eliminate transverse beam tails beyond ±3σ. The longitudinal beam

phase space is well within the Booster acceptance.

OFFICE OF

SCIENCE

6



Findings

• The H- ion source is capable of producing up to 15 mA of beam

current.

• Beam tracking studies were performed from the exit of the RFQ to

the end of SRF linac.o A 6D Gaussian distribution of 1M macro particles was simulated,

corresponding to bunch current of 5mA.

o 3D space charge forces were included.

• Synchronous phases in the cavities along the linac are kept

sufficiently large to accommodate a 6σ longitudinal phase space

variation.

• The final normalized RMS transverse emittance is below 0.25 mm

mrad.

• The MEBT absorber is capable of handling 21 kW of beam power.

OFFICE OF

SCIENCE

7



Findings

• Fermilab will provide circulators, RF distribution systems, controls

and interlocks for all linac systems.

• Fermilab is providing RF amplifiers for the warm cavities (6) and

HWRs (8).

• India will provide amplifiers for SSR1 (16), SSR2 (35), LB650 (33)

and HB650 (24).

• A working prototype 3 kW, 325 MHz amplifier from India has been

sent to FNAL for evaluation.

• A prototype 7kW, 325 MHz amplifier has been completed and an

order has been placed with a vendor (ECIL) for 9 production units.

• India will build prototypes and ECIL will ‘industrialize’ the design

using standard production techniques.

OFFICE OF

SCIENCE

8



Findings

• Circulators and waveguides are specified to handle four times the

power capability of the amplifiers. All circulators are tested CW

with full reflection at the rated power levels of the amplifiers.

• Preliminary designs of the magnets are proceeding based on the

beam physics requirements as outlined in the CDR.

• India will provide forty-four quadrupole magnets and twenty x-y

correctors magnets - prototypes are expected in Jan 2020.

• The LLRF system for each cavity is designed to provide rms

amplitude stability of 0.01% and rms phase stability of 0.01°.• The LLRF post down-mix modules are the same for all frequencies

and are housed in racks that independently regulate four cavities.

• The Controls Group is evaluating the use of EPICS versus

upgrading ACNET.

OFFICE OF

SCIENCE

9



Comments

• The accelerator team to be congratulated for its substantial progress

since CD-1.

• The team is highly experienced, competent and fully committed to

PIP-II Project.

• The LLRF Group has done well in leveraging its experience from

its work in developing the LCLS-II LLRF system.

OFFICE OF

SCIENCE

10



Comments

• The linac beam optics evaluation is fairly mature and simulations

will be expanded to include various beam loss scenarios and

magnet misalignment errors.

• There is a concern as to whether upgrading the global control

system will be complete by the start of the beam commissioning.

• The linac has hundreds of single-point failure sources that will

shutoff the beam so component reliability needs to be very high.

• Long term testing of the SSAs should be done as much as possible

to identify failure modes before full production begins.

OFFICE OF

SCIENCE

11



Comments

• We fully support and encourage using the PIP-II designed LLRF

systems for HWR and SSR1 tests at PIP2IT.

• It is imperative that the PIP-II staff have full access to all in-kind

design and fabrication documents, test results and vendor

inspections.

• We encourage further development of resonance control techniques

and laser profile monitor.

• A presentation should be made at the next DOE Review

summarizing how and when supporting R&D will address

remaining risks (e.g., microphonics and component reliability).

• The Project team should identify which tasks are R&D that reduce

CD-2 risks, and which tasks are pre-production aimed at reducing

CD-3 risks.

OFFICE OF

SCIENCE

12



Recommendations

1. Start immediately to better document physics-driven requirements

for the various sub-systems, consistent with the PDR – to be

evaluated for each sub-system before the CD-2 IPR.

2. Prior to the CD-2 IPR:

• Complete an overall availability assessment of the accelerator

sub-systems and include the results in the appropriate

requirement and specification documents.

• Complete linac beam tracking simulations including realistic

errors.

• Validate linac beam optics at PIP2IT as much as possible.

• Complete the 325 MHz, 7 kW amplifier evaluation.

• Finalize the controls modernization path.

OFFICE OF

SCIENCE

13

2.2 Accelerator Upgrades,

Linac and I&C

T. Roser, BNL / Subcommittee 2

Subcommittee members:

Peter Ostroumov (MSU), Thomas Roser (BNL), Marion White (ANL)

1. Is the project making adequate technical progress to ensure that the completed project

will perform as planned and meet the key performance parameters? The scope for the

Objective KPPs is well defined and technical progress towards meeting the

Objective KPPs is adequate. The performance of the completed project (1.2MW

beam power on neutrino production target) and the full scope to achieve it,

however, is not as well defined. Achieving this level of performance also requires

additional accelerator upgrades planned for the 900-kW NuMI upgrade.

2. Will execution of PIP-II design plans and planned R&D program activities ensure

most major technical risks will be appropriately mitigated or retired prior to CD-

3? Yes.

8. Has the project satisfactorily responded to the recommendations from previous

reviews? Generally yes. The recommendation from the CD-1 review “Review

Intensity Limits in Booster, Recycler and Main Injector before CD2.” was not fully

addressed.

9. Are there any other significant issues that require HEP or project’s attention? No

OFFICE OF

SCIENCE

14


Linac and I&C


Findings

● The PIP2IT commissioning was very productive and resulted in:

○ Successful delivery of beam to the dump at nominal parameters (5 mA,

0.55 ms, 20 Hz) with low losses and all aperture restrictions

○ Characterization of transverse and longitudinal beam parameters

○ Demonstration of bunch-to-bunch chopping

○ Reliable CW operation of the RFQ

○ Successful operation of beam optics devices developed and built in India

● Currently, the PIP2IT is being prepared for the acceleration of 25-MeV H-minus

beam at design parameters (2 mA, 20 Hz, 550 msec) through the HWR and SSR

cryomodules.

● Overall the planning for the Linac installation and commissioning is progressing

well.

● Lessons from PIP2IT installation are taken into account and provide valuable

guidance and training for the installation of PIP-II.

● Installation concepts are evolving in parallel with facility and system design,

commissioning planning, and Resource Loaded Schedule development

● Beam commissioning at PIP-II is foreseen to occur in parallel with installation

activities in 6 stages

OFFICE OF

SCIENCE

15


Linac and I&C


Findings

● Design work under the installation WBS is focused on cryomodule tunnel

transport system, warm unit structures and requirements for interface definition.

● 3D CAD modeling efforts and mechanical design tasks have been initiated and are

proceeding well.

● Building infrastructure scope deals with the design and purchase of materials to

extend utilities further into the building and interface with machine

infrastructure.

● Planning and developing layouts of utilities, racks, cables, and cable trays for the

LINAC Complex, PIP-II Utility Building (PUB), and Beam Transfer Line (BTL)

are underway.

● Team in place to advance designs of mechanical system, electrical system, interior

layouts, and cable database.

● The Test Infrastructure is at an advanced stage of the design with the fabrication

and installation of the infrastructure for these facilities already in progress. The

future conversion of PIP2IT into a cryomodule test stand for 650 MHz and

SSR1/SSR2 cryomodules, and the construction of the HTS-2 facility are at a

conceptual stage of design maturity.

OFFICE OF

SCIENCE

16


Linac and I&C


Findings

● The upgrade of the Accelerator complex consists of the 800 MeV Booster injection

system, the upgrade of the Booster repetition rate to 20 Hz, an upgrade of the

Recycler cavities and a power upgrade of the Main Injector rf system.

● The charge exchange injection girder includes a pulsed four-bump (ORBUMP)

and a dedicated absorber for the unstripped and neutralized Hminus.

● The neighboring combined function defocusing magnets will be replaced with

shorter magnets with larger aperture.

● Design for the injection area has started with a PDR scheduled for April 2019.

● Little activity on the Booster repetition rate upgrade since last IPR. The main

issues for the upgrade are understood.

● The Recycler scope includes removing the current Recycler 53 MHz RF stations

and replacing them with new ones.

● The MI Upgrade scope includes delivery of 20 upgraded MI RF stations together

with their corresponding modulators and solid state drivers. A preliminary design

review is foreseen for 6/19.

● This scope is necessary to achieve the objective KPP of circulating beam in the

booster.

OFFICE OF

SCIENCE

17


Linac and I&C


Findings

● Studies of the cavity with two power amplifiers look promising; plans are to

continue test stand studies and eventually beam studies if all goes according to

plan.

● These studies, if successful, will help retire the risk that the 2-amplifier solution

may not work

● Scope of the transfer line and absorber WBS includes defining and maintaining

the Lattice of the Beam Transfer Line and the Beam Absorber Line, specifying

Beam Line components, and design, procurement, fabrication, and testing of

Beam Transfer Line collimators and beam dumps. A 5-kW beam dump in line

with the Linac and the main 50-kW beam dump in the Beam Absorber line are

also included.

● There are no risks associated with transfer line or absorber in the register.

● No schedule or cost information was presented to the subcommittee.

OFFICE OF

SCIENCE

18


Linac and I&C


Comments

● For the limited scope to reach the Objective KPPs the preparation for CD-2 is

making good progress.

● The final goal of the PIP-II project is to enable the delivery of 1.2 MW beam to the

neutrino target by the time LBNF starts operation.

● Additional accelerator upgrade projects are needed to reach 1.2 MW beam power

on target. These upgrades are also needed to reach 900 kW beam power on target

for NuMI. The completion of these additional upgrades should be tracked as they

are critical for the PIP-II project success.

● An overall project parameter optimization could be useful. As an example the

performance of the booster could substantially benefit from a shorter injection

time. It might be possible to operate the linac at higher peak current to shorten the

injection time into the Booster.

● The Objective KPP on the delivery of the number of particles per pulse should

also include pulse length.

OFFICE OF

SCIENCE

19


Linac and I&C


Comments

● The warm front-end test stand successfully demonstrated required rf and beam

performance, assembly, integration, installation, and commissioning of the

complex front end.

● The CW beam development at the front end was stopped. Continuation of this

effort would be beneficial.

● The PIP2IT is using the FNAL-style control system and development of additional

software is expected with the installation of HWR and SSR cryomodules. The

project should make a decision on the PIP-II control system as soon as possible to

avoid substantial re-work effort if the EPICS toolkit will be adopted.

● The project should update the present plan for needed Booster modifications to

cope with the higher intensity and repetition rate required by PIP-II. This effort

should be based on detailed simulations and analysis of beam losses in the Booster.

The simulations should be benchmarked with presently observed beam losses

during Booster operation.

OFFICE OF

SCIENCE

20


Linac and I&C


Comments

● A detailed beam commissioning plan is being developed based on the experience of

other similar facilities such as SNS. Currently, the commissioning team envisions 6

stages of beam commissioning, which will occur in parallel with installation

activities. It is encouraged to plan external ARRs prior to each of the six stages of

the beam commissioning to fully address safety requirements, hardware readiness

and efficiency of the commissioning.

● The project should be commended on the development of the extensive test

infrastructure, which includes the warm front end, beam test of two types of

cryomodules. The same facility will be modified for the testing of the production

cryomodules. This is efficient use of personnel that gained experience with testing

of LCLS II cryomodules.

● The project should be commended for laying out the Linac and the building

infrastructure using 3D CAD software. This work should continue.

● The Linac lattice needs to be finalized as soon as possible since many design

details depend on it.

OFFICE OF

SCIENCE

21


Linac and I&C


Comments

● There should be a full simulation of the charge exchange injection process to

optimize the design before the component specifications are finalized.

● First tests of the second rf power amplifier connected to a MI cavity are very

promising. The Committee supports the plan to install the 2 PA cavity in the MI

and test with beam.

● The design of the BTL and absorbers seems to be well advanced for this stage of

the project.

● A resource allocation plan will be needed to ensure that PIP-II, LBNF beam line,

NuMI upgrades and facility operation can proceed in parallel.

● The Committee supports the project’s plan to use the removable foil changing

device that is in operation at SNS.

OFFICE OF

SCIENCE2.2 Accelerator Upgrades,

Linac and I&C


Recommendations

1. Update the plan for needed Booster modifications to cope with the higher

intensity and repetition rate based on a detailed analysis of beam losses in the

Booster, benchmarked with present beam losses, and present the plan at the

next IPR.

2. Ensure that all the additional accelerator upgrades needed for 1.2 MW beam

power on the neutrino production target are completed before the start of

LBNF operation.

3. Perform an overall project parameter optimization before CD-2.

22

OFFICE OF

SCIENCE

23

2.3 SRF and Cryogenics

M. Howell, ORNL / Subcommittee 3

Subcommittee Members:

Charles Reece (TJNAF), Ting Xu (FRIB/MSU) & Brian DeGraff (ORNL)

1. Is the project making adequate technical progress to ensure that the completed

project will perform as planned and meet the key performance parameters?

Yes, with the current proposed KPPs.

2. Will execution of PIP-II design plans and planned R&D program activities

ensure most major technical risks will be appropriately mitigated or retired

prior to CD-3? Yes, however the SSR2 and LB650 prototype cryomodules will

be tested two years after CD-3. Three shipping tests will also occur after CD-3.


reviews? Yes, but some recommendations are yet to be closed.


Resource requirements to achieve CD-2/3a by Q3 2019 are much higher than

the current allocated resource level.

OFFICE OF

SCIENCE

24



Findings

• Changes since last review:

OFFICE OF

SCIENCE

25



Findings

• The SSR2 cryomodules are on the critical path. However, the LB650 and HB650

cryomodules are close to critical path.

• The project has made it a priority to accelerate the cryo plant and SRF development

schedules.

• There are four prototype cryomodules added to this project. Installation of the prototype

cryomodules in the linac is not part of the baseline plan but it is not eliminated as a

possibility.

• Cryomodules coming from India are supplied as kits. These kits are assembled at

FNAL.

• FNAL will supply the niobium specification to the partner institutes.

• It is not known whether Indian laboratories will build the SSR1 cavities or if the cavities

will be procured from an industrial vendor.

• Repair and rework costs for the cryomodules during production period are not part of the

base cost estimate. Any repairs would be paid for using contingency.

OFFICE OF

SCIENCE

26



Findings

• New project team has been in place after CD-1. Project annex with INFN, Italy has

been signed 12/2018. New project organization tailored to address communication and

coordination of international partners. In-kind contribution scope is still varying in the

SRF scope.

• Cavity resonance control is considered a medium technical risk and extensive

test/development work is foreseen as part of each prototype cryomodule testing.

• HWR cryomodule from ANL is scheduled for delivery 3QFY19 and testing in PIP2IT

4QFY19. HWR module shipping vendor engaged with FNAL/ANL to develop and

evaluate shipping plan, preliminary Transport Readiness Review held, 14 August 2018.

• Cryogenic Distribution System (CDS) ICD are well developed.

• HB CM design is at an early stage. PDR scheduled for 2019.

• HB650 cavity treatment for robust Q & gradient R&D is underway.

• LB650 cavity RF design has been frozen, mechanical designs being refined. LB650 will

be partially tested at partner labs. All cryomodules will be cold tested in the test bunker

before installing into tunnel

OFFICE OF

SCIENCE

27



Findings

• An integrated design team for the 650 CMs has been formed. Two different supporting

structures have been proposed and analyzed for transportation loads. Three international

shipping tests of 650 MHz cryomodules have been added as lesson learned from LCLS

II project. The first test is planned FY2022.

• The current cryogenics plant design does not account for upgrades to the SCL. The

threshold KPP is 600 MeV instead of 700MeV reported at CD-1 review.

• The cryogenic plant is designed to operate the linac at 2K and provide sufficient cooling

at either pulsed or continuous wave operation.

• There is no LN2 precooling in the cold box

• Impact of insufficient heat load capacity from the single plant is to add a small (~$3M

4.5K liquefaction plant)

• Spare compressor and spare recovery compressor are not part of the vendor contract.

• Cryoplant delivery is planned for Q3 of 2021.

OFFICE OF

SCIENCE

28



Findings

• Cryoplant vendor is required to provide all necessary hardware to validate the capacities

of the box at all temperature levels, including 2K.

• Recovery compressor will be a Mycomm compressor and will not have backup power.

The recovery compressor will serve as the main compressor in the event of a need for a

supplemental 4.5K cold box. In the event of a power outage, helium is vented.

• The change from a primary pulsed machine to a primary CW machine has changed the

needed cryogenic plant capacity with appropriate margins. At CD-1 the requested plant

capacity given to vendors was 2 kW at 2K. The new required 2K capacity is about 2.2

kW. This has yet to be communicated to the cryoplant vendors. It is hoped that after the

Indian plant is ordered, that the PIP-II plant can have some design modifications to

increase the 2K capacity by about 200 W.

• SSR2 cavity design is on-going and PDR is planned Mar 2019. No progress on SSR2

cryomodule design is reported this time.

OFFICE OF

SCIENCE

29



Comments

• It is estimated that it will require 45 FTEs to advance the SRF/Cryo design to CD-2/3a

maturity by June 2019. Currently, approximately 30 FTEs are working on this scope.

Manpower requirements must be managed and integrated with competing projects.

• The provision of four prototype cryomodules early enough to provide feedback for

refinement of production is to be commended as this will significantly retire technical

risk, including possible microphonic resonance control issues. However, two prototype

cryomodules are completed approximately two years after CD-3. The resource and cost

impact should be carefully evaluated.

• Installed cryomodule performance is vulnerable to particulate contamination residual

from all processing, assembly, installation, and maintenance activities, and from

migration within beamline vacuum. Common QA standards for “particle free” assurance

are indicated for all cryomodule preparation, assembly, installation, and maintenance

activities, as well as all other linac beamline elements.

• Provisions have been made for a spare compressor and recovery compressor, however

they are not in the current project baseline scope. Meeting the availability requirements

without these spare compressors will be challenging.

OFFICE OF

SCIENCE

30



Comments

• The recommendation from CD-1 to “Define operational gradient margin and

cryomodule maintenance strategy to meet the performance specification of 90%

reliability with only 8 weeks of maintenance per year by CD-2” is ongoing.

• The newly installed PIP-II SRF staff have little prior project experience. Active

management of the complex set of documentation required to attain CD-2 will be

difficult, especially in view of the network of partners/suppliers. Organizational links

with partners are setup. In some cases, partners are more experienced than FNAL staff.

The high communication overhead needs to be reflected in labor allocation plans.

• Including bayonet cans for the two upgrade cryomodule slots in CDS baseline scope

would facilitate future upgrades to increase beam energy for negligible cost and

schedule impact.

• If the beam energy KPP is not met, rebuilding existing cryomodules or adding additional

cryomodules in the upgrade cryomodule slots could be done to increase beam energy.

OFFICE OF

SCIENCE

31



Comments

• With international partners and procurement of the cryo plant advancing, integrating the

instrument and controls into the FNAL controls architecture may be an issue.

• The level 3 managers were not able to quickly answer resource requirements questions

to get to where they are going by CD-2/3a. Some of this is because there have been

many recent changes to the project team. This is a concern.

• International scope is not in the EVMS system. A system to track the progress of

international partner scope will be beneficial to the project.

• The cryoplant and CDS are relying on matrixed personnel to meet the required design

level before CD2. Adding staffing is most likely needed.

• No cost and RLS has been presented to the committee. Additional external review for

the cost and schedule will help vet the resources required before CD2/3a.

• Partner laboratory Intellectual Property barriers prevented the committee from seeing the

technical requirements specification for the cryogenic plant. This prevented the

committee from being able to completely assess technical details.

• Expectation is this documentation would be available to the committee by CD-2/3a

review

OFFICE OF

SCIENCE

32



Comments

• The shipping test for the LB650 occurs late in the current schedule. A detail schedule

should be created and reviewed to verify that time is still available to correct any

problems this test reveals.

• Cryo plant and cryo plant building infrastructure requirements and interfaces shall be

fully detailed before CD3a.

• To work towards the same CAD software for the integrated design team is a good

approach and will significantly reduce the risk to have design and fabrication issues due

to file translation and facilitate the ECO process.

• All four types of cryomodules production are planned to be in parallel and finish very

close to each other. Additional float has been built into the schedule due to the concern

of lack of the experience of some partner labs. The committee encourages the project

team to provide support to partner labs in certain areas.

OFFICE OF

SCIENCE

33



Recommendations

1. Reconcile the schedule and the required experienced resources of the SRF/Cryo team to

advance the design to the PDR level by CD-2/3a.

2. Develop a comprehensive particulate contamination management plan that ensures

consistent specifications, procedures, and QA for cavity preparation, string assembly,

CM assembly, CM testing, installation, operation and maintenance across all systems

that include linac beamline vacuum elements by CD-2/3a.

3. Conduct a design failure mode analysis of the cryo plant including the “in-kind”

contribution and the FNAL contribution to ensure the plant meets the reliability and

availability requirements of the project prior to CD-2/3a.

OFFICE OF

SCIENCE

34

3. Conventional Facilities

B. Bull, MSU and Gary Bloom ORNL

Subcommittee 4

1. Is the project making adequate technical progress to ensure that

the completed project will perform as planned and meet the key

performance parameters? Yes, but the underground utility routing

for the cryoplant needs to be expedited so they can be included in

60% design documents

8. Has the project satisfactorily responded to the recommendations

from previous reviews? Yes


attention? No

OFFICE OF

SCIENCE

Findings

• A total of 8 recommendations have been satisfactorily addressed from the previous

reviews and the recommendations have been closed.

• Cryoplant construction is scheduled for completion in 2 quarter of FY2021

• Bids for the Site Preparation package have been received

• Conceptual Design Complete for Utility Plant Building (WBS 121.06.04), Linac

Complex (WBS 121.06.05), Booster Connection (WBS 121.06.06)

• Construction delivery method is planned to be firm fixed price contracts

• Currently there are 0 High Risks, 15 Medium Risks and 31 Low Risks for CF

• Cost Estimates are updated at 60% and 90% design stage

• A/E responsible for estimating the construction costs but has in turn retained a

national construction manager to provide this service

• Level 2 manager has extensive experience at the laboratory

• Utilizing Fermilab guidance for escalation with a base rate of 3.4% but the difference

between the base rate and actual rate is carried at the project level as a risk

35



Subcommittee 4

OFFICE OF

SCIENCE

Findings

• A total of 8 recommendations have been satisfactorily addressed from the previous

reviews and the recommendations have been closed.

36



Subcommittee 4

OFFICE OF

SCIENCE

Comments

• The late validation of building requirements from the cryoplant vendor could cause

significant design and construction changes to a firm fixed price construction

contract. Consider delaying bidding or at least finalizing the construction contract

until the building requirements are validated.

• The project should continue to evaluate and update the cost estimates, and technical

requirements.

• Consider developing a list of technical construction activities that would typically be

completed by construction trades to evaluate whether some should be included in

the CF bid packages.

• Using an IDIQ like type of contract for A/E design services allowing the release of

work by task order is a good practice for the phased construction anticipated

• Consider carrying the escalation difference between project escalation rate and

annual construction forecast escalation rate in the base construction package

estimate.

37



Subcommittee 4

OFFICE OF

SCIENCE

Comments

• The risk Cost Impact and/or Schedule Impact for some of the conventional facilities

risks appears to be low. Consider reviewing the cost and schedule Impact level for

the CF risks.

• Consider developing a commissioning plan that accounts for the installation of the

technical equipment after substantial completion of the CF work and before the CF

equipment will see loads.

• Consider completing a construction market analysis to determine if there are any

advantages to awarding to a construction manager or general contractor that could

also deliver the balance of the project by task order .

• Advancing site clearing to avoid environmental impact issues is commendable.

38



Subcommittee 4

OFFICE OF

SCIENCE

Recommendations

• The project should detail routing of all final connections from CF systems to technical

equipment but at a minimum for all underground or concealed routes prior to 60%

design completion for each package

• Interfaces between CF and technical groups need to be fully defined, including

installation responsibility, stay clear zones and treaty points prior to 60% design

completion for each package

• CF group should add additional dedicated staff that reports up through the project

prior to releasing cryoplant building for bidding

39



Subcommittee 4

OFFICE OF

SCIENCE

Subcommittee members:

Dave Rodgers (LBNL); Andrew Ackerman (BNL)

6. Is ESH&Q being handled appropriately?

Yes. EHS&Q is well integrated into the project.

8. Has the project satisfactorily responded to the recommendations from

previous reviews?

All recommendations have not been completed but are on track for

completion by 3/31/2019.


attention?

No.

40

4. Environment, Safety and Health

D. Rodgers, LBNL / Subcommittee 5

OFFICE OF

SCIENCE

41



Findings

• CD-1 recommendations are either done or on track to be completed

prior to CD-2:

o Prepare the CD-1 required Environmental Compliance Strategy

consistent with existing programs and documentation. Complete

Posted v2 1/17/18, Schedule to be added when EA complete.

o Continue NEPA process/strategy as briefed, and complete in

accordance with a schedule that supports CD-2. In progress;

FONSI expected 12/31/18.

o Perform an overall document review for consistency and

integration across programs and project functions prior to CD-2.

In progress, expected 3/31/19.

o Per project documents, ensure Quality Assurance processes and

oversight for international and domestic providers that mitigate

risks are included in the baseline for procurements. In progress,

expected 3/31/19.

OFFICE OF

SCIENCE

42



Findings (cont.)

• The QA manager is performing on an acting basis but this may turn out

to be permanent.

• QA engineer position is posted to support L2/L3 managers, starting with

SRF and Cryo Systems.

• A Safety by Design process has been developed and implemented just

begun.

• Formal program for integrating Lessons learned into project has being

developed.

• EHS and QA requirements will be integrated into Project Planning

Documents both for international partners and participating DOE labs.

The plans will be reviewed and approved by Fermilab staff and the

partners.

• EHS staffing is 0.5 FTE but able to be increased as needed.

• ODH monitors will be installed.

• The project plans for cryomodule transport testing with prototypes

being sent back and forth to Europe.

OFFICE OF

SCIENCE

43



Findings (cont.)

• Shielding assessment awaits baseline design at CD-2 and the process

resides with the Accelerator Division. Done in-house.

• Air activation not expected to be a problem but more detailed analysis

will be performed during shielding assessment.

• Plans for compliance with the Accelerator Safety Order (ASO) include a

revised Safety Analysis Document (SAD), Accelerator Safety Envelope

(ASE) and three planned Accelerator Readiness Reviews (ARRs).

• Preliminary shielding assessment is complete for the cryo-building/linac

penetration.

• Electrical AHJ is traveling to India to inspect equipment.

• A USI process was used to address the addition of PIP-II to Fermilab

complex and resulted in need for Shielding Assessment/SAD/ASE.

OFFICE OF

SCIENCE

44



Comments

• Progress on the responses to the CD-1 recommendations has not been

completed. The planned completion schedule seems appropriate for

finish prior to CD-2.

• Lessons learned program is ambitious and can provide significant value

to project.

• The QA manager is fully involved in implementing appropriate project

QA processes.

• EHS/QA are well integrated into the project. Management can help

support these roles by including these managers in senior planning

meetings.

• Providing a dedicated QA person embedded at the L2/L3 levels is an

excellent practice.

• Plans for integrating EHS and QA requirements into Project Planning

Documents seems well developed and a good practice, but full

implementation will be challenge prior to CD-2.

OFFICE OF

SCIENCE

45



Comments (cont.)

• The Safety by Design process is an admirable initiative and is a new

process for Fermilab.

• The plans for Accelerator Safety Order compliance utilizing three ARRs

should be reconsidered to better match the planned six stage

commissioning plans with expected installation schedules.

• The plans for testing cryo module transportation by shipping prototypes

to Europe and back should provide maximum assurance that

production modules will be shipped without equipment damage.

OFFICE OF

SCIENCE

46



Recommendtions

1. Close out remaining CD-1 recommendations prior to CD-2:

• Perform a comprehensive document review for consistency and

integration.

• Complete the Environmental Assessment process by receiving a

FONSI.

• Update the QAP and implementing processes to better address

international partners.

2. Finalize and post the HAR prior to CD-2.

3. Finalize the Project Planning Documents providing for EHS and QA

flowdown to the international and domestic partners, prior to CD-2.

OFFICE OF

SCIENCE

47

5. Cost and ScheduleE. Merrill, DOE/OPA, L. Oukrop, PNNL, F. Gines, DOE/ASO

Subcommittee 6

3. Has the project made adequate progress on its resource-loaded schedule to

complete it by the time of CD-2? The project has made good progress but the

RLS is not complete. Completing a credible RLS in time for a June 2019 CD-

2 IPR will be challenging.

7. Are the proposed risk mitigation strategies reasonable and are the proposed

contingencies acceptable? The proposed preliminary mitigation strategies

appear reasonable; however, an analysis of the cost and schedule contingency

values are not feasible at this time as the performance measurement baseline

and risk register are still in development.


reviews? Yes.


Continue to monitor progress in developing project documentation supporting

the Basis of Estimate documentation, RLS, finalized time-phased risk register

and revised probabilistic risk assessment, and updated cost and schedule

contingency analysis.

OFFICE OF

SCIENCE

48


Subcommittee 6

• Findings

• The PIP-II Project is currently maintaining the CD-1 cost range of $653M - $928M with

preliminary Total Project Cost point estimate of $721M including $191M (37%) in cost

contingency (from the PPEP). The TPC is being revised and was not presented at this

review.

• The PIP-II Tier 4 CD-4 date is Q3 FY27. The Tier 1 CD-4 date is Q1 FY30, providing

30 months of schedule contingency.

• The current DOE-HEP funding profile between FY16 and FY27 supports the $721M

TPC which is based on the CD-1 point estimate.

• The project cost and schedule development methodology follows the DOE Cost

Estimating Guide and employs standard estimating and scheduling tools (P6, Cobra,

Primavera Risk Analysis).

• The project risk register contains 109 open and approved entries (101 threats, 7

opportunities and 1 uncertainty). The project risk severity rated as 15 high, 49 medium,

and 45 low.

OFFICE OF

SCIENCE

49


Subcommittee 6

• Findings

• The project is reliant on international deliverables estimated at approximately $250M in-

kind contributions.

• Cost risks associated international in-kind contributions are not held by the project.

• The project has obligated approximately $60.7M since January 2016.

• The project cost to date is $56.2M.

• The project is proposing to freeze the project baseline in February 2019 and begin

exercising the FRA EVMS in March 2019.

• The project’s procurement durations are based on guidance from the Fermilab

Procurement Office.

• Critical Path goes through SSR2 prototype/production and LINAC

installation/commissioning. LB650 & HB650 productions are near Critical Path.

• Schedule tailoring plans CD-2/3a for Q4FY19 (Jun 2019) and CD-3b in Q4FY20.

• CD-3a – Site Preparation & Cryoplant Building Construction

• CD-3b – Civil construction of LINAC Complex, cryoplant installation, SSR1 power

coupler, and High Bay LCW system. Civil construction of booster connection

maybe added to give schedule flexibility.

OFFICE OF

SCIENCE

50


Subcommittee 6

• Findings

• The RLS contains 6841 activities, 243 T5 milestones, 141 Design review milestones (53

PDR, 88 FDR), 211 International milestones (64 major design reviews, 147 major

deliverables), and 95 Major procurements.

• BOE being develop by 24 CAMs for 116 Control Accounts (CAs) identified, one BOE

for each CA.

OFFICE OF

SCIENCE

51


Subcommittee 6

• Comments

• The PIP-II project controls team is experienced and employing best practices and

lessons learned to execute the PIP-II Project. The project should consider augmenting

the project controls staff as quickly as possible.

• The process of preparing a revised TPC, project schedule, bases of estimate

documentation, contingency estimates, and an updated risk register to reflect changes

in the project organization, project scope, and execution strategy implemented since

the CD-1 IPR will require a comprehensive effort to complete prior to CD-2.

• The project maintains a credible Basis of Estimate (BOE) development process down

to the WBS Level 5 incorporated into a preliminary, technically constrained project

schedule.

• The project maintains a credible risk management process; however, the risk register

provided needs to be tied to schedule activities to facilitate the risk management

process (appropriate time phasing of risk retirement).

• The project should consider developing a draft scope deduct list prior to CD-2/3a.

OFFICE OF

SCIENCE

52


Subcommittee 6

• Comments

• The project plans to be ready for a DOE baseline review in June 2019, to achieve this

the baseline will need to be locked down in February 2019. EVMS implementation

will begin in March 2019. The subcommittee is concerned that this schedule is very

aggressive and may not be achievable.

• The overall cost and schedule contingency appeared adequate for the project at CD-1;

however, the subcommittee was unable to evaluate the appropriateness of current

contingency estimates. The aggressive schedule to baseline could result in inadequately

defined scope and associated risk and uncertainties.

• The project’s cost and schedule contingency should be re-evaluated for adequacy prior

to CD-2.

• Although some data was available for this review, the following project information (at

a minimum) should be developed and available for the CD-2 review.

• Performance Measurement Baseline (cost and schedule), BA/BO Profile, RAM,

planned value by FY at control account level, FTE profile by FY at the control

account level, preliminary scope deduct list, monthly reports, updated risk register

with qualitative/ quantitative analysis performed, and cost estimate/basis of

estimate documentation.

OFFICE OF

SCIENCE

53


Subcommittee 6

• Comments

• The Level 3 Managers (L3M) demonstrated familiarity with their cost estimates as

evidenced by three cost drill down sessions. However, the current cost estimates are in

various stages of revision.

• The updated WBS appears reasonable and complete to better define and control

planned work although impact to the TPC is uncertain.

• The project should consider revising the procurement schedule to reflect HCA/IRB

approval of contract values greater than $25M.

• HCA/IRB approval required before issuing RFP (~55 working days)

• HCA/IRB approval to award (~40 working days)

• Consider repricing resource rates in FY19 dollars prior to setting baseline.

• The project should evaluate the impact of concurrent projects to identify and mitigate

any potential conflicts (i.e., resources/labor/interfaces) to ensure PIP-II Project

deliverables can be achieved on the schedule proposed.

• The project should evaluate the current work packages to ensure they appropriately

reflect the revised project scope. Some conversion back to planning packages may be

required. Further discrete planning can be performed using rolling wave planning as

scope maturity increases throughout the life-cycle of the project (planning package to

work package conversion).

OFFICE OF

SCIENCE

54


Subcommittee 6

• Recommendations

1. Complete development of all the necessary baseline documentation (RLS, Basis of

Estimate, risk assessment, etc) and other related items in the comments above and in

comments made by other subcommittees prior to the CD-2 Director’s Review.

OFFICE OF

SCIENCE

55


Subcommittee 6

PROJECT STATUS – November 2018

Project Type Line Item

CD-1 Planned: Q2FY19 Actual: 7/23/2018

CD-2/3a Planned: Q3FY20 Actual:

CD-3b Planned: TBD Actual:

CD-3 Planned: Q4FY21 Actual:

CD-4 Planned: Q1FY30 Actual:

TPC Percent Complete Planned: ___N/A__% Actual: __10.6___%

TPC Cost to Date $56.2M

TPC Committed to Date $60.7M

TPC (TPC being revised, not presented at this review) $721M @ CD-1

TEC $435M @ CD-1

Contingency Cost (w/Mgmt Reserve) $191M @ CD-1 _ _37__% to go

Contingency Schedule on CD-4 (Apr2027-Dec2029) ____30__months __ 30__% to go

CPI Cumulative N/A

SPI Cumulative N/A

OFFICE OF

SCIENCE

56

4. Are preparations for defining, documenting, and managing the international in-

kind contributions suitable to ensure their timely delivery and technical fidelity?

Partially, the project has defined and drafted a number of governance documents to

coordinate the international contributions, however the implementation processes,

procedures and staffing to assert needed oversight by the host lab are not yet

implemented to ensure timely delivery and technical fidelity.

5. Is the proposed CD-2 timeline reasonable and consistent with the current project

status?

The Committee found the timeline to CD-2 very aggressive and will be a challenge to

deliver a quality product. See follow-on comments in the body of the report.


reviews?

Yes, the staffing plan recommendation still needs a few critical hires.


RLS, transition to operations, operations budget dependencies were not presented.

Preliminary KPPs may require additional definition. Cryomodule production spares

should be evaluated in light of the last two SC projects which have experienced >10%

production loss.

Management/Subcommittee 7G. Hays (SLAC), J. Ingraffia (ANL), R. Leftwich-

Vann (LBL), S. Meador (DOE), M.

Reichanadter, SLAC

OFFICE OF

SCIENCE

57



Reichanadter, SLAC

Findings

• The PIP-II Project proposes to deliver a high proton energy beam (up to 1.2MW), thus

providing an intense neutrino beam to the LBNF and DUNE experiments (near and far

sites).

• PIP-II will be upgradeable to multi-MW capability and compatible with CW-operations,

greatly increasing the linac output and providing for a full modernization of the front-

end of the Fermilab accelerator complex (new SC Linac and upgrades to Booster,

Recycler, and Main Injector). A preliminary set of KPPs have been proposed which will

be finalized at CD-2.

• PIP-II was fully endorsed by the HEPAP P5 Report, “to provide proton beams of >1 MW

by the time of first operation of the new LBNF.”

• The PIP-II Project is hosted by Fermilab with in-kind contributions from four

international partners (UK, France, Italy, India, additional partners are under

consideration). Currently, “in-kind” contributions comprise ~25% of the PIP-II project.

• “In-kind” contributions from international partners (includes design, hardware, labor and

delivery to FNAL) are managed through Project Planning Documents (PPDs).

• Five PPDs are planned (DAE, INFN, CEA, IN2P3, UKRI) to be completed by May 2019

and signed/approved by September 2019 (before CD-2/3A ESAAB). PPDs are not

legally binding.

OFFICE OF

SCIENCE

58

Findings

• PIP-II received CD-1 approval in July 2018, with a cost range is $653M - $928M and

point estimate of $721M. The project team has been working on an updated

performance baseline, however the updated baseline along with a risk-based

contingency assessment was not available for this review.

• The project has not identified a significant list of scope contingency.

• Outside project dependencies are defined and captured in a project assumptions

document and an Interface Control Document (ICD)

• DOE program, international institution and Fermilab (host lab) support are in place.

Regular governance meetings at the agency, international and project levels are

beginning to function.

• PIP-II proposes a tailored CD strategy, starting with a CD-2/3A IPR in June 2019 to

establish commitment by DOE-HEP, its international partners and the project team.

• $10M of OPC funding is planned in the last two years for commissioning according to

the CD-1 profile.

• The project is considering a phased transition-to-operations plan to transfer equipment

when it has achieved its project objectives.



Reichanadter, SLAC

OFFICE OF

SCIENCE

59

Findings

• Since CD-1, a new management team has been formed with a significant number of

new hires, the WBS has been restructured, and many of the necessary governance

documents and management plans have been drafted.

• A Project Scientist role has been filled and will lead the development of scientific

requirements and specifications for the PIP-II technical and conventional systems.

• A new PIP-II Procurement Manager has recently been hired. The project will matrix

additional procurement staff from the FNAL procurement department. A few critical

hires remain to be filled (PMCS, some QA, etc.).

• An acting risk manager is assigned to the project, but is not located permanently within

the US.

• An acting quality assurance manager is assigned to the project. Two requisitions are

being opened for full-time quality assurance engineers to support the project.

• A FONSI is expected on the Environmental Assessment later in December 2018.

• Prototype cryomodules are being considered as operational spares.

• Currently, shutdown schedules between PIP-II and LBNF do not align.



Reichanadter, SLAC

OFFICE OF

SCIENCE

Comments

• The project team has made significant improvements since CD-1. A capable project

management team is in place, and the processes, tools and staff to successfully deliver

the PIP-II Project have made substantial progress. Nice job!

• However, a significant amount of work has yet to be accomplished in a short time

assuming CD-2/3A IPR in June 2019. From the committee’s perspective, a partial list

of deliverables not yet complete to support a CD-2/3A Director’s Review (in May

2019) are:

• RLS in alignment with funding profile guidance provided by the program

• All BOEs complete with supporting WBS dictionary

• Risk-based (cost and schedule) contingency analysis

• Transition to Operations (TTO) Plan and Operation budget dependencies

• List of scope contingency

• Updated PEP, PMP, Procurement Management Plan, HAR, QAP, APPs

• Off-project commitments documented in an MOU with time phasing, resources,

and stakeholder signatures

• Finalized Assumption and Interface Control Documents

• Preliminary Design Report w/ independent committee recommendations

• 1-month minimum of EVMS reporting on a stable baseline60



Reichanadter, SLAC

OFFICE OF

SCIENCE

Comments

• PIP-II management team will track progress on international deliverables using an

extensive set of milestones. However, there is not yet enough oversight of other

international project functions, particularly in the areas of procurement and QA.

• While PIP-II is an international collaboration, Fermilab is the host and ultimate owner

of its partners’ in-kind hardware. With this in mind, it is important the PIP-II Project

Office provide sufficient oversight of its in-kind partners (particularly in the

procurement and QA) and conduct project management partnering sessions to ensure

the entire organization clearly understands their respective roles and responsibilities.

• The project should identify explicitly which international partner milestones trigger risk

mitigation plans.

• There are a number of key staffing roles which need to be hired or strengthened prior to

CD-2/3A. Project Controls Lead, Risk Management and Procurement Specialists in

particular need to be staffed with clear R2A2s across the project organization.

Procurement oversight on international deliverables would be a good place to start. In

addition, Fermilab should consider hiring a full-time risk manager and matrix at least

50% during the development of the PIP-II CD-2/3A package.

61



Reichanadter, SLAC

OFFICE OF

SCIENCE

62

Comments

• Many PIP-II senior leaders are new to the project and ~half of the CAMs are new to

CAM responsibilities. Given the new project processes to be rolled-out, a steep

learning curve for FNAL and its international partners should be expected. Conduct

early on project management sessions to ensure clear R2A2s and process familiarity.

• PIP-II proposes to use technical configuration and baseline configuration processes

which are layered and complex. Consider simplifying to only one TCB and CCB with

the PIP-II PO, L2 Managers and international partner leads. The goal is for PIP-II to

minimize the effects of boundaries between its partners. Common and simple

processes can facilitate this goal.

• To mitigate the risk of a late international deliverables due to inadequate funding,

investigate using an investment bond. This helped moderate cash flow issues on the

ESS Project.

• Multiple international partners provide the same components (e.g., HB650

Cryomodules). Sourcing same components from multiple international sources could

increase cost, result in quality (fit/finish) issues and overload vendor(s). Consider

bundling large, critical procurements at one institution with drop-ship to other partners.



Reichanadter, SLAC

OFFICE OF

SCIENCE

63

Comments

• For critical technical components with limited vendor base, the project should consider

defining approved vendors for international partners. In some cases, consider

embedding expert project staff at major international production facilities.

• Off project dependencies should be expanded to define dependency owners, resources,

time frame of dependency by quarter and fiscal year, and should be signed in

agreement by all stakeholders. The document should also be updated annually.

• The project should consider a TTO plan prior to CD-2 and determine if an update to

the funding profile is needed to permit early TTO plans for equipment.



Reichanadter, SLAC

OFFICE OF

SCIENCE

64

Recommendations1. Review the Committee list of CD-2 deliverable yet to be completed and develop a

schedule for completion leading up to a CD-2/3A Director’s Review. Once

complete present this “CD-2/3A Timeline” to the PIP-II IPT and Level 2s (G9) no

later than January 15, 2019. This timeline should be used as the basis to

determine when the project is ready for CD-2/3A.

2. Consider streamlining the project’s configuration and change control processes

and rolling out in project management training sessions in line with the

committee’s comments.

3. After settling on the PIP-II international project management processes, conduct

project management sessions to ensure clear R2A2s and process familiarity.

4. Hire the PMCS Lead, clarify the role of the Procurement Manager and obtain

adequate Risk Management support as soon as possible.

5. Review the commissioning portion of the PIP-II performance baseline in the next

iteration.



Reichanadter, SLAC

Date post:	24-Apr-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times