1
Project Closeout Report
for the
Muon g-2 Project
Project # SC-30YC
at
Fermi National Accelerator Laboratory
Office of High Energy Physics
Office of Science
U.S. Department of Energy
Date Approved:
April 2017
Month/Year
3
Project Closeout Report for the Muon g-2 Project
at the Fermi National Accelerator Laboratory
1. EXECUTIVE SUMMARY ........................................................................................................................ 4 2. INTRODUCTION ...................................................................................................................................... 5 3. ACQUISITION APPROACH ................................................................................................................... 5 4. PROJECT ORGANIZATION .................................................................................................................. 6 5. PROJECT BASELINE .............................................................................................................................. 7 6. CLOSEOUT STATUS ............................................................................................................................. 16 7. LESSONS LEARNED ............................................................................................................................. 16 8. PHOTOS ................................................................................................................................................... 18 9. PROJECT DOCUMENT ARCHIVES AND LOCATIONS ................................................................ 28 10. APPENDICES .......................................................................................................................................... 29
APPENDICES
A. Detailed Technical Performance Achieved B. Major External Reviews C. Project Risk Registry D. Detailed Technical Lessons Learned from CD-0 through CD-3 E. Detailed Technical Lessons Learned from CD-3 through CD-4 F. Transition to Operations
4
1. EXECUTIVE SUMMARY
The Muon g-2 Project was a DOE project to fabricate an experiment that seeks to improve the
measurement of the muon anomalous magnet moment, which is sensitive to new physical
interactions through interactions with virtual particles emerging from vacuum interactions. In
particular, Muon g-2 provides unique sensitivity to beyond Standard Model theories such as
supersymmetry and dark matter by measuring indirectly their impact on the muon spin
precession in a magnetic field. The project repurposed a storage ring from the E821 experiment
at Brookhaven National Laboratory (BNL) with the transfer of the BNL storage ring
accomplished in fiscal year 2013. Upgrades to the ring injection subsystems, detectors,
electronics, data acquisition, and field monitoring equipment were needed to utilize the high
intensity proton beam available at Fermilab, and control systematics at a level commensurate
with the 21-fold increase in statistics compared to prior measurements. A number of
modifications of the former Fermilab anti-proton source were required to deliver a high purity,
intense beam of muons to the storage ring. Some of the critical detector systems were provided,
in part, by in-kind contributions from non-DOE sources including National Science Foundation
(NSF) and international contributions, particularly from the UK and Italy. The Muon g-2
experiment offers a strategic opportunity to search for new physics that may be inaccessible to
the Large Hadron Collider (LHC). The project was successfully completed ahead of schedule
and under budget.
The Muon g-2 Project was completed within its baselines. The project has fully met its
Objective KPPs 18 months ahead of schedule and with $195 thousand of contingency remaining.
Construction of the Muon g-2 detector components and subsystems was completed in November
2017.
In addition, the Project, using contingency, was able to fund development and a portion of the
fabrication of a new inflector to potentially increase muon injection efficiency and provide for a
spare to this critical device. The new inflector was the highest priority on the optional scope list.
As contingency funds became available for optional scope (based on the retirement of project
risks), the new inflector was funded incrementally. The superconducting shielding material used
for fabricating the existing inflector in Japan, approximately 25 years ago, was so specialized
that the project could not locate anyone who knew how to re-make it. The project was able to
obtain the last remnants of the superconducting shielding material and superconductor cable of
the type used on the existing inflector. Since all of the requirements for CD-4 had been achieved
by November 2017, DOE-OHEP wanted the work on the new inflector to be transitioned to
operations at that time for greater operational efficiency. All but a relatively small amount of the
work needed for the new inflector was able to be accomplished on the project. Since the end of
the equipment fabrication project, Fermilab has resume the final winding of the coils and plans to
complete the assembly of the cold mass.
5
2. INTRODUCTION
This is the Project Closeout Report for the Muon g-2 Project, a DOE project, which will enable
the Muon g-2 Experiment to take place at Fermilab.
Muon g-2 will use Fermilab's powerful accelerators to explore the interactions of short-lived
particles known as muons with a strong magnetic field in "empty" space. Scientists know that
even in a vacuum, space is never empty. Instead, it is filled with an invisible sea of virtual
particles that, in accordance with the laws of quantum physics, pop in and out of existence for
incredibly short moments of time. Scientists can test the presence and nature of these virtual
particles by observing the rate at which the magnetic moment of a fundamental particle rotates
around the magnetic field, a phenomenon known as spin precession. This spin precession has
been measured to even higher precision with electrons, but the heavier mass of the muon makes
it uniquely sensitive to the presence of higher mass particle that appear in quantum field loops
around the muon.
The Muon g-2 experimenters will examine the precession of muons that are subjected to a
magnetic field. The main goal is to test the Standard Model's predictions of this value by
measuring the precession rate experimentally to a precision of 0.14 parts per million, which
requires 2x1011 muons to be stored and observed in the experiment. If there is an inconsistency,
it could indicate the Standard Model is incomplete and in need of revision.
3. ACQUISITION APPROACH
DOE acquired the Muon g-2 Project through the operating contractor of Fermilab, Fermi
Research Alliance (FRA). Construction of the conventional facility to house the Muon g-2
detector was carried out as an independent General Plant Project (GPP) as a multi-use facility.
The building is called the MC-1 Building.
The technical design and specifications for Muon g-2 Project were developed by the Muon g-2
collaboration, under the project management at Fermilab. Procurements followed the Fermilab
rules for contract awards and competitive bids.
In addition, the detector systems include in-kind contributions of scope from NSF funded
university groups, valued nominally at $3.2 million, for the construction of 24 PbF2
calorimeters, along with their associated electronics, and the data acquisition for the experiment.
This contribution from NSF is not part of the DOE scope or Total Project Cost (TPC).
Some components were delivered as in-kind contributions from international collaborating
institutions—notably, a consortium of UK universities primarily funded by the STFC provided
$2.25 million in funding for the construction of the straw tracker system, and a consortium of
Italian groups funded by INFN provided $0.5 million in materials and the effort needed to
construct a sophisticated laser monitoring system for the calorimeters. In the original Project
Execution Plan, these contributions were estimated as a $1 million contribution.
6
DOE DES Project Director
Fermi Site Office ort
Center
Fermi National Accelerator Laboratory
Office of Science (Acquisition Executive)
Office of High Energy Physics
BTeV Program Manager
Muon g-2
Federal Project Director
Fermi Site Office Integrated Support
Center
Fermi National Accelerator Laboratory
Office of Science
(Acquisition Executive)
Office of High Energy Physics
Muon g-2 Program Manager
Muon g-2 Project Manager
Project Management
Group
Muon g-2
Technical Board
Muon g-2
Collaboration
Finally, a DOE Early Career award for Brendan Casey in the amount of $2.5M over 5 years was
also used primarily to advance the straw tracker hardware and analysis and is not considered part
of the TPC associated with the Muon g-2 Project. In the original Project Execution Plan, it was
estimated that the Early Career Award would provide a $900k savings to the project construction
costs.
Commissioning with beam was not part of the Muon g-2 Project.
4. PROJECT ORGANIZATION
Figure 4.1 shows the management chain for the Muon g-2 Project, from the DOE Office of
Science to the Project Manager. The Project Management Group, which met monthly, was a
mechanism used by Fermilab to monitor the progress and address the needs of the project in a
timely fashion. Figure 4.2 shows the organization chart for the project itself. The project was
organized into five work breakdown structure (WBS) areas, as shown.
Figure 4.1: Project Management Chain
7
5. PROJECT BASELINE
Scope Baseline
The scope of the Muon g-2 Project was to deliver the existing superconducting storage ring
magnet from BNL to Fermilab, and to provide the accelerator modifications, injection system
upgrades, improved field monitoring, and detector systems not covered by in-kind contributions.
A set of General Plant Projects (GPPs) and Accelerator Improvement Projects (AIPs), known as
the Muon Campus GPPs and AIPs, were set up to provide some of the multiuse infrastructure
needed for the Muon g-2 and Mu2e Experiments. Each of these projects had a defined scope
and was managed under formal change control. Interface milestones were used to coordinate
key events between the Muon Campus GPPs and AIPs and the Muon g-2 Project and Mu2e
Project. Final ring magnet shimming after the vacuum chambers were installed and beam
commissioning was not included in the scope of the project as these are areas that will continue
to improve throughout the lifetime of the experiment as it operates over the next few years.
Details of the technical performance of these project elements are given in Appendix A and the
documents referenced within.
The Project met the scope baseline when the apparatus was demonstrated to be functioning by
achieving the Key Performance Parameters (KPP) listed below in Table 5.1. The KPPs
demonstrate performance capability of the system. The Threshold KPPs were the minimum
technical requirements for successful project completion. Objective KPPs were to be
accomplished by the project if sufficient project funds was available and if external factors
beyond the control of the project did not prevent it.
WBS 1.0
Muon g-2 Project
WBS 1.1
Project Management
WBS 1.3
Ring
WBS 1.2
Accelerator
WBS 1.4
Detectors
WBS 1.5
BNL Equipment
Transfer
Figure 4.2 Project Work Breakdown Structure
8
Table 5.1 Key Performance Parameter
Threshold Performance Objective Performance Results
Accelerator All accelerator components of Beamlines M2, M3, M4
and M5 are ready for
installation, dependent on
external factors. (Installation
of Beamline M2 or M3
components requires
accelerator shutdown for
personnel access, impacting
NOvA operations.
Installation of Beamline M4
or M5 components requires
the Beamline Enclosure
GPP.)
All other accelerator
components are installed and
ready for commissioning
with beam at nominal
voltages and currents,
represented by:
• Target Station
Momentum Selection
Magnet (PMAG) Pulsed
Power Supply achieving
15.3 kA peak current;
• Target Station Lithium
Lens Pulsed Power
Supply achieving 19 kA
peak current;
• Delivery Ring Extraction
Lambertson Magnet and
Power Supply achieving
1.13 Tesla-m integrated
field strength; and
• Delivery Ring Extraction
C-Magnet and Power
Supply achieving 1.68
Tesla-m integrated field
strength.
All accelerator components
are installed and ready for
commissioning with beam at
nominal voltages and
currents.
Objective KPP
achieved. (See
Appendix A for
technical details.)
Storage Ring Storage ring yoke pieces, pole pieces, and superconducting
coils have been cooled and
powered to full 1.45T field.
Objective KPP
achieved. (See
Appendix A for
technical details.)
9
Storage Ring
Subsystems
Storage ring subsystems,
including the electrostatic
quadrupoles, pulsed
electromagnetic kickers, and
inflector, are ready to install.
Storage ring subsystems,
including the electrostatic
quadrupoles, pulsed
electromagnetic kickers, and
inflector, are installed and
ready for commissioning
with beam at nominal
voltages and currents.
Objective KPP
achieved. (See
Appendix A for
technical details.)
NMR
Systems
Nuclear magnetic resonance
(NMR) systems for
monitoring magnetic field,
including fixed probes,
plunging probes, and NMR
trolley, are ready to install.
Nuclear magnetic resonance
(NMR) systems for
monitoring magnetic field,
including fixed probes,
plunging probes, and NMR
trolley, are installed and ready
for commissioning with beam
at nominal values.
Objective KPP
achieved. (See
Appendix A for
technical details)
Detector All calorimeters and trackers that have been received by
the time that the accelerator
system is ready to be
commissioned with beam
have been installed.
All calorimeters and trackers
have been installed.
Objective KPP
achieved. (See
Appendix A for
technical details)
In addition, the Project was able to use contingency funds for the development and partial
fabrication of a new inflector that could potentially increase the muon injection efficiency and
serve as a spare to this critical device.
Cost Baseline
DOE funding for the Muon g-2 Project came through Fermilab. The sum of the Fermilab
funding ($46.4 million) constitutes the project’s TPC. The Muon g-2 Project was finished with
an estimated final cost of $46.20 million and with $195 thousand remaining in available
contingency.
Table 5.2 shows the original CD-2 baseline. The major area with large contingency usage was
for beamline magnets and specialized power supplies. The contingency was used primarily for
additional engineering and designer resources needed to complete the project. Of the total of
$7.1 million of contingency used, almost all was on labor at Fermilab. As discussed in the Scope
Baseline section above, the Project was able to save enough contingency to fund the
development of a new inflector and a portion of the fabrication. The project contingency usage
is shown in Table 5.3, Table 5.4 and Figure 5.1.
10
Table 5.2 DOE CD-2 Cost Estimate by WBS Element ($ in Millions)
WBS
Element Item
DOE
OPC
DOE
TEC
DOE
TPC
1.1 Project Management 2.6 1.5 4.1
1.2 Accelerator 4.9 12.9 17.8
1.3 Ring 4.4 7.8 12.3
1.4 Detectors 0.1 0.4 0.5
1.5 BNL Equipment Transfer 4.2 0.0 4.2
Subtotal of above 16.2 22.6 38.9
DOE Contingency† 7.5
DOE Total 46.40
Table 5.3 Project Contingency Usage
Date or End
of Fiscal
Year
% Project
Complete TPC ($M)
Actual
Costs to
Date ($M)
Contingency
Remaining
($M)
Estimate to
Complete
($M)
FY2015 57% 46.4 23.36 6.95 11.33
FY2016 87% 46.4 37.65 3.08 5.67
Sept 2017 99.6 46.4 46.12 0.13 0.15
Dec 2017 100% 46.4 46.20 0.19 0
*$0.19 million remained in contingency at the end of the project.
Figure 5.1 Project Contingency History
11
Table 5.4 Contingency Use in Detail
WBS
Element Item
CD-2 Base
Cost
Estimated
Final
Cost*
Cont.
used*
1.1 Project Management 4.1 3.67 -0.43
1.2 Accelerator 17.8 21.76 3.96
1.3 Ring 12.3 15.95 3.65
1.4 Detectors 0.5 0.64 0.14
1.5 BNL Equipment Transfer 4.2 4.18 -0.02
DOE Total at Completion 38.9 46.21 7.31
Unused Contingency 7.5 0.19
DOE TPC 46.40 46.40
*Numbers based on end of December 2017 EVMS data
Schedule Baseline
The Muon g-2 Project was baselined with a CD-4 milestone date of June 2019. All the project
deliverables were provided by the end of November 2017. The CD-4 Review was conducted on
November 14, 2017.
The project was baselined with twenty-four months of schedule contingency. CD-4 was
approved on January 16, 2018, leaving 17.5 months of schedule float remaining. The major
elements of the project which took longer than originally scheduled included the assembly and
powering of the magnet prior to CD-2, the removal of the AP0 beam absorber, the design and
upgrades of existing pulsed power supplies in the target station, the construction of conventional
magnets for the beamline, the assembly and alignment of trolley rails and quadrupole plates
within the storage ring vacuum, and the in-kind production of the straw trackers.
In general, the completion of the Ring and Detector L2 branches stretched by approximately six
months from the original plan. Within the Accelerator L2, the extraction and replacement of the
AP0 beam absorber also took six months longer than planned. In the end, a number of the
subsystems all came together just in time for the fiscal year 2017 engineering run that started
early in June 2017. Production of the beamlines, which was the critical path and thought to be a
major schedule risk at CD-3, did not take longer than scheduled. Table 5.5, Table 5.6 and Table
5.7 summarize the major milestones and dates.
12
Table 5.5 Critical Decisions Milestones
Milestone Description Preliminary
Baseline Date
0.0 CD-0: Approve Mission Need Sep. 18, 2012 (A)
0.1 CD-1: Approve Alternative Selection and Cost Range Dec. 19, 2013 (A)
0.2 CD-2: Approve Performance Baseline Aug. 20, 2015 (A)
0.3 CD-3: Approve Start of Construction Aug. 20, 2015 (A)
0.4 CD-4: Approve Project Completion Jan. 16, 2018 (A)
Table 5.6 Level 1 Milestones
Milestone Description Baseline Date
1.1 KPP—Storage Ring Threshold KPP Achieved Aug 30, 2016
1.2 KPP—Storage Ring Subsystems Threshold KPP Achieved Aug 22, 2017
1.3 KPP—NMR Systems Threshold KPP Achieved Oct 4, 2017
1.4 KPP—Accelerator Threshold KPP Achieved Feb 27, 2018
1.5 KPP—Detector Threshold KPP Achieved Mar 27, 2018
13
TABLE 5.7 LEVEL 1 AND 2 MILESTONES
L1 Milestones:
L2 Milestones:
14
Work Breakdown Structure
The WBS for the Muon g-2 Project to level 2 is defined in Table 5.8 below.
Table 5.8 Muon g-2 Work Breakdown Structure Dictionary
WBS Title Description
1.0 Muon g-2 Project
1.1 Project Management Project Management support
1.2 Accelerator Accelerator modifications
1.3 Ring Storage ring magnet and associated systems
1.4 Detectors Calorimeters, trackers, auxiliary detectors, electronics, DAQ
1.5 BNL Equipment Transfer Transport of storage ring magnet and other reusable equipment
Funding Profile
The funding tables below present the funding profile planned at CD-2 and the actual funding
profile.
Table 5.9 CD-2 Planned Funding
FY 2012
FY 2013
FY 2014
FY 2015
FY 2016
FY 2017 Total
OPC-Other 0.601 2.742 3.2 6.543
OPC-Design 3.108 5.2 4 12.308
TEC-MIE 2.0 9 10.2 6.349 27.549
Total 0.601 5.850 10.4 13 10.2 6.349 46.400
Table 5.10 Actual Funding
FY 2012
FY 2013
FY 2014
FY 2015
FY 2016
FY 2017 Total
OPC-Other 0.601 2.742 3.2 6.543
OPC-Design 3.108 5.2 4 12.308
TEC-MIE 2.0 9 10.2 6.349 27.549
Total 0.601 5.850 10.4 13 10.2 6.349 46.400
15
Staffing Profile
The figure below shows the actual staffing profile for the Muon g-2 Project in hours per fiscal
year.
Environmental Requirements/Permits
The NEPA Categorical Exclusion (10CFR1021, Subpart B, Appendix B1.30, B1.31, and B3.10)
for the Muon g-2 Project was approved on December 20, 2012. No environmental issues were
encountered during this project.
Safety Record
Standard Fermilab safety procedures and policies were followed throughout the project,
including the use of Integrated Safety Management principles, clear Job Hazard Analysis
standards, and standing safety committees which conducted Operational Readiness Reviews of
custom equipment and frequent safety walk-throughs/inspections of the work areas.
Throughout the course of the Muon g-2 Project construction, there were a total of three notable
safety incidents. The first occurred when one of 24 pieces of steel (26T each) that form the
storage ring return yoke was dropped due to a failure in rigging equipment. The root cause was
traced to a new commercial softener that was used to protect the straps and failed to protect the
16
sling from being cut by the sharp 90 degree edge of the steel. The softener was brought to the
lift by a technician working for the project and had not been reviewed by either the rigging
supervisor or project engineering. The sling failed after one end of the steel had lifted several
inches from the deck of the flatbed truck. The load fell back onto the truck deck without any
damage to property or injuries to personnel. A stop work was issued; a root cause analysis was
conducted; procedures were reviewed; and the project took the opportunity to provide additional
training to the entire project team to emphasize safety ahead of schedule.
The second notable accident occurred as a technician was taking down a temporary barrier that
was used to discourage access to the storage ring during the time it was stored outside while
work on the building finished. The worker stepped in a hole and twisted an ankle resulting in
some time lost from work.
The final notable safety incident occurred near the end of the project as engineers and
technicians were testing the pulsed Li lens power supply used to focus the primary proton beam
onto the target. During the testing of one capacitor bank in the power supply, a second capacitor
bank was inadvertently charged due to a wire that had been previously connected and then
overlooked. When the tests concluded, the technician noticed a wire on the second capacitor
bank still needed to be reconnected and touched the wire to the charged bank anode. This
resulted in an accidental discharge of the stored energy, which vaporized the wire. The
technician suffered a small laceration on his finger due to quickly retracting his hand from the
cabinet. Proper PPE likely saved more serious injury as a portion of the vaporized wire
splattered onto the technician’s safety glasses. A stop work was issued; a root cause analysis
was completed that found some violations of best practice procedures; the procedures were
reviewed; and special training was provided for the department responsible for this type of work
at the laboratory.
6. CLOSEOUT STATUS
The financial closeout of the project will be completed following the CD-4 ESSAB.
Operations and Commissioning: Planning and Readiness
An accelerator readiness review was held on March 14, 2017, and beamline commissioning
began in April 2017. An engineering run served as an integration test of all the subsystems from
June 5, 2017 through July 7, 2017. Commissioning was resumed in November 2017, following
the completion of the summer accelerator shutdown. Further details are contained in Appendix
F: Transition to Operations.
7. LESSONS LEARNED
Some lessons learned are briefly presented below. Details for each of these lessons learned are
presented in Appendix E: Detailed Technical Lessons Learned from CD-3 through CD-4.
Management
17
Custom project management tools made work more efficient and effective.
Management team composition led to coherent, effective team.
Transition to operations of the Cryo Plant AIP should have been better planned.
Coordinated management of Muon Campus and Muon g-2 aided in accomplishing projects efficiently and effectively.
Funding
Project and scientific collaboration worked together effectively to achieve project goals.
Reuse of the anti-proton complex and other Tevatron equipment saved time and money.
Integrated approach to coordinating effort with detector deliverables funded through an NSF MRI were successful.
Cost
Cost overrun on conventional magnet construction were due to inefficiencies at lab.
A number of factors lead to cost overrun on Li lens and PMAG power supplies.
Estimation and planning of alignment and metrology effort lead to more efficient implementation.
Schedule
Custom project management tools made work more efficient and effective.
Tracking uncosted scientific effort ensured accurate progress measurement.
Technical
Costs overran on target station beam absorber.
Safety and Quality
Need to pay close attention to prevent overworking personnel when schedules become tight.
18
8. PHOTOS
Photo 2 Removal of Superconducting Coils from Building at BNL. Red Steel is Support
Structure for the Coils, 6/22/2013
Photo 1 Muon g-2 Experiment at BNL, 12/22/2005
19
Photo 3 Rings Traveling Across the BNL Site, 6/22/2013
Photo 5 Barge Traveling on the Mississippi River
by the Gateway Arch in St. Louis, 7/19/2013
Photo 4 GPS Tracking of Muon g-2 Rings on
Barge, 6/22-7/26/2013
20
Photo 6 Rings Going Under an Overpass in Illinois, 7/24/2013
Photo 7 Rings Traveling on I-88 in Illinois, 7/25/2013
21
Photo 8 Rings Traveling across the Fermilab Site, 7/26/2013
Photo 9 Rings on Display at a Public Event at Fermilab, 7/26/2013
22
Photo 10 Rings Moving into Experiment Hall,
7/30/2014 Photo 11 Rings Moving into Experiment Hall, 7/30/2014
Photo 12 Rings Lowered onto Yoke Steel, 8/2014 Photo 13 Storage Ring Assembled for Initial
Powering , 6/2015
Photo 14 Computer Display Showing that Storage
Ring Achieved Operating Temperature , 6/2015
Photo 15 Display Showing that Storage Ring
Achieved Full 1.45 T Field , 10/2015
23
Photo 16 Beamline Instrumentation Testing,
2/13/2017
Photo 17 Removal of Old Beam
Absorber and Module, 11/16/2016
Photo 18 Delivery Ring D-30 Straight Section, 3/1/2017
24
Photo 20 Kicker Plates Installed in Vacuum
Chamber, 3/2017
Photo 22 Trolley Drive Installation 4/2017 Photo 23 Fiber Harp (Beam Profile Monitor),
5/2017
Photo 21 Calorimeter Installation, 5/2017
Photo 19 Kicker Installation, 4/2017
25
Photo 24 Fiber harp monitor (7 horizontal fiber
optic cables seen in the center of the
photograph) installed in the Muon g-2 storage
ring. The monitor is in the inserted
measurement position. 2/2017
Photo 25 Tracker modules installed in the Muon g-2 storage
ring. 2/2017
26
Photo 31 Last Vacuum Chamber Installed, 1/2017 Photo 30 Remote Operations in ROC-West Tested,
10/2016
Photo 27 Quad Plates in Vacuum Chamber, 8/2016
Photo 26 High Voltage Feedthrough, 4/2017
Photo 29 Tracker Installation 5/2017 Photo 28 Calorimeter sled
2/2017
27
Photo 32 Muon g-2 Experiment in MC-1 Building at Fermilab 8/28/2017
28
9. PROJECT DOCUMENT ARCHIVES AND LOCATIONS
Project documents are archived by the Muon g-2 Project in a Fermilab-maintained document
database: http://gm2-docdb.fnal.gov; review access is available upon request.
http://gm2-docdb.fnal.gov/
29
10. APPENDICES
Appendix A: Detailed Technical Performance Achieved
In this section the elements required to demonstrate the completion of the KPP for the Muon g-2
Project are described in detail.
Threshold Performance Objective Performance
Accelerator All accelerator components of Beamlines M2,
M3, M4 and M5 are ready for installation,
dependent on external factors. (Installation of
Beamline M2 or M3 components requires
accelerator shutdown for personnel access,
impacting NOvA operations. Installation of
Beamline M4 or M5 components requires the
Beamline Enclosure
GPP.)
All other accelerator components are installed
and ready for commissioning with beam at
nominal voltages and currents, represented by:
• Target Station Momentum Selection Magnet
(PMAG) Pulsed Power Supply achieving
15.3 kA peak current;
• Target Station Lithium Lens Pulsed Power
Supply achieving 19 kA peak current;
• Delivery Ring Extraction Lambertson
Magnet and Power Supply achieving 1.13
Tesla-m
integrated field strength; and
• Delivery Ring Extraction C-Magnet and
Power Supply achieving 1.68 Tesla-m
integrated field strength.
All accelerator components are
installed and ready for
commissioning with beam at
nominal voltages and currents.
Storage Ring Storage ring yoke pieces, pole pieces, and
superconducting coils have been cooled and
powered to full 1.45T field.
Storage Ring
Subsystems
Storage ring subsystems, including the
electrostatic quadrupoles, pulsed
electromagnetic kickers, and inflector, are ready
to install.
Storage ring subsystems,
including the electrostatic
quadrupoles, pulsed
electromagnetic kickers, and
inflector, are installed and
ready for commissioning with
30
beam at nominal voltages and
currents.
NMR
Systems
Nuclear magnetic resonance (NMR) systems for
monitoring magnetic field, including fixed
probes, plunging probes, and NMR trolley, are
ready to install.
Nuclear magnetic resonance
(NMR) systems for monitoring
magnetic field, including fixed
probes, plunging probes, and
NMR trolley, are installed and
ready for commissioning with
beam at nominal values.
Detector All calorimeters and trackers that have been
received by the time that the accelerator system
is ready to be commissioned with beam have
been installed.
All calorimeters and trackers
have been installed.
Achievement of Accelerator Objective KPP:
All accelerator components are installed and ready for commissioning with beam at nominal
voltages and currents, thus achieving this Objective KPP.
The accelerator installation was completed in the spring of 2017. An Accelerator Readiness
Review was held on March 14 through 16, 2017, and permission to run beam was granted on
April 5, 2017. Although not part of the KPPs, commissioning-quality beam was established to
the g-2 muon storage ring on May 31, 2017. Some details of the technical accomplishments are
presented below.
The target station pulsed power supplies and beam absorber (a.k.a. beam dump) were ready
when needed for beam commissioning. The old beam absorber which had cooling water leaks
was removed. This required more effort than planned due to a broken bolt at the attachment of
the beam absorber to the shielding module, both of which were highly radioactive. A new
module and a new beam absorber were fabricated and installed.
All g-2 beamlines (M1, M2, M3, M4, and M5) were installed, including magnets, vacuum
systems, power supplies, controls, and instrumentation. In the M1 line, magnets used for the
final focus of the beam on the target were rearranged from their 120-GeV configuration to
provide a smaller spot size with 8-GeV beam. The M2 and M3 lines were created from the
former antiproton-source transport lines, with additional quadrupole magnets to better capture
muons from pion decay. The Delivery Ring was reconfigured for injection from the M3 line and
extraction to the M4 line. The new M4 and M5 beamlines were designed and installed to
transport muons from the Delivery Ring to the g-2 muon storage ring.
31
Figure A1 shows the current from the PMAG power supply during the 2017 commissioning
period. The supply was tested at currents above 15.3 kA, but only needs to run at 14.7 kA
Figure A1 also shows the current from the lithium lens power supply during the 2017
commissioning period. Currents above 19 kA have been achieved and provide better focusing of
particles from the target.
Figure A2 shows the Lambertson power supply operating at 1,400 amps which corresponds to an
integrated field of 1.16 T-m. Fermilab’s Technical Division tested the magnet as high as 1.32 T-
m, as shown in Fig. A3 [gm2-doc-8957].
Figure A2 shows the Extraction C-Magnet power supply operating at 1,040 amps which
corresponds to an integrated field of 1.69 T-m. Technical Division tested the magnet as high as
2.09 T-m, as shown in Fig. A4 [gm2-doc-8960].
The Muon g-2 Project re-used or re-purposed a significant amount of technical components and
enclosure including:
1km of tunnel complete with electrical infrastructure, cable trays, cooling water distribution system and safety interlocks;
former Pbar (antiproton) Target Station including target, lithium lens, pulsed momentum-selection magnet, collimation system, target vault, cooling systems, hot work cell, and
tunnel access points with overhead crane coverage -Service buildings with HVAC,
cooling water, controls communication infrastructure, extensive electrical infrastructure,
electronics racks, access roads and parking lots, etc
~250 Pbar magnets (>$20M) and ~30 BNL magnets plus 505m former Pbar Debuncher Ring used in-place as Delivery Ring;
power supplies repurposed where practical (new supplies are smaller and more energy efficient); and
Secondary Emission Monitors and other instrumentation from the Antiproton Source.
32
Figure A2: Power Supply Currents for the Lambertson (red)
and C-Magnet (green) in April and May 2017 when the
Delivery Ring was set up for 8 GeV Beam, which is when the
Highest Fields are Required.
Figure A1: Lithium Lens (green) and PMAG (red) Power
Supply Currents during the 2017 Commissioning Period.
33
Figure A4: Magnetic Field Strength as a Function of Current During C-
Magnet Testing.
Figure A3: Magnetic Field Strength as a Function of
Current during Lambertson Magnet Testing.
34
Achievement of Storage Ring Objective KPP:
The storage ring yoke pieces and pole pieces have been assembled, and superconducting coils
have been cooled and powered to full 1.45T field. This was accomplished in November 2015.
This satisfied the KPP. The magnet was used from November 2015 and for most of 2016 for
magnetic field shimming purposes. There was no separate Objective KPP since the magnet
being fully-functional was required midway through the project in order to complete the rough
shimming of the field before other equipment could be installed in the magnet gap.
The magnet was warmed up in the winter/spring of 2016-2017 to repair a helium leak and for the
removal of three loose charcoal panels in the lower inner cryostat. These flaws required the
magnet be powered down after 12 hours so that the charcoal could be regenerated. Figure A5
shows the removal of loose charcoal from the lower inner cryostat. The loose pieces have poor
thermal contact to 5K, and periodically release helium, which increases heat load on the magnet.
Six more charcoal panels remain in the upper and outer rings, but these are well secured to their
respective mandrels.
After the helium leak repair and charcoal removal, the magnet could be operated for weeks
without requiring powering down for maintenance. The magnet has been in regular operation
ever since. Improvements are continuing to be made, on the operations side to guard against
scenarios where quenches can be caused by power outages.
Figure A5: One of 3 charcoal panels removed from the lower inner
cryostat.
35
Achievement of Storage Ring Subsystems Objective KPP:
The inflector, quadrupoles, and kickers have all met their threshold and objective KPPs, as
described below.
In November 2016, the inflector was cooled to 5K and powered for eight hours at the design
current. The superconducting shield was also shown to be operational. This satisfies the
threshold KPP. Figure A6 shows the inflector ramped to full current.
The inflector was operated during June 2017 for the first engineering run. The new power
supply tripped several times in the initial part of the run. This is understood now to be very
infrequent 100 Amp current glitches from the power supply, which crosses the quench detection
threshold. This was mitigated by increasing the quench detection threshold from 25 mV to 120
mV, the same value used by BNL E821. This satisfies the objective KPP for the inflector.
The kicker system was installed in spring 2017. During the June 2017 engineering run, it could
only operate at about 70 percent of the design strength, and there was severe sparking, which
also affected the quadrupole operation. Since each spark deposits up to 125 joules into the
vacuum, and possibly various electrical insulators, it likely caused damage. The kicker HV
vacuum feedthrough was refabricated using a new design and was installed in September 2017.
Subsequently, it was commissioned to full power (55 KV) and at the required repetition rate, and
was operated with a negligible spark rate. Figure A7 shows the secondary transformer voltages
of the three kickers, reaching the designed kick. This satisfies the objective KPP for the kicker
system.
Figure A6: the inflector current being ramped to 2840 Amps.
36
The electrostatic quadrupole system was fully installed in spring 2017 (Figure A8). During the
June 2017 engineering run, it was operated in the range of 15-20kV, below the designed field
index. Above these voltages, the quads could not operate for long term without sparking. This is
now understood to be due to a combination of factors: the SRV vacuum value did not reach the
designed value and there was sparking in the kicker system, causing the quads to also spark.
Following the kicker HV feedthrough upgrade, and better vacuum, the quadrupoles were
demonstrated to achieve robust operation at 20.5 KV and at 25.4 KV. These are the voltages
expected to give optimal beam dynamics for g-2. The spark rate was sufficiently low, and
estimated to affect the data-collection efficiency at the few percent level. This satisfies the
objective KPP. The table below summarizes the quad spark statistics achieved in September-
October 2017.
Figure A7: Secondary transformer voltages for each of the 3 kickers, indicating operation
at the designed 55 KV.
37
Achievement of NMR Systems Objective KPP:
The fixed probes, plunging probe, and the NMR trolley comprise the systems relevant for the
NMR Systems KPP. All systems were installed and ready for commissioning beam at their
nominal values as of June 20, 2017. The Objective KPP was completed on March 29, 2017
when all systems were ready for installation in the ring. Full operational readiness clearance
approval for remote operation of these systems was subsequently granted after installation for:
the fixed probes on April 21, 2017; trolley on April 26, 2017; and the plunging probe on
September 5, 2017.
The fixed probe systems and the trolley were used throughout the June 2017 engineering run to
monitor and tune the magnetic field. While all systems were ready to install in late March 2017,
the plunging probe installation was deferred until a natural break point in the engineering run in
order to maximize the efficacy of the run and allow for independent testing in the storage ring.
All systems have operated well since meeting the KPP.
The fixed probes are embedded in the upper and lower surfaces of the vacuum chambers, and
their installation was required prior to the installation of the SRV chambers into the magnet gap.
Figure A9 shows a close-up of the probes installed in the vacuum chamber (left), and a view of a
vacuum chamber ready to install in the ring (right). The fixed probe system was fully
commissioned prior to the engineering run, and was used to monitor and stabilize the magnetic
field during that period. Figure A10 shows the Data Quality Monitor displaying information
Figure A8: Electrostatic Quadrupole Spark Rate measured during Sept-October 2017.
38
from a subset of the fixed probes during the run (left) as well as the stabilization of the magnetic
field based on these probes (right).
The trolley was installed in the ring in April 2017. The trolley can be seen through a storage ring
vacuum port during a transit around the ring in Figure A11 (left). Full maps of the magnetic
field were also produced based on this data, and were utilized to further shim the magnet during
the summer shutdown. Figure A12 demonstrates the azimuthally averaged magnet field, and the
average field vs. azimuth after the shutdown shimming period.
The plunging probe was installed during the June 2017 run. The successful measurement of the
field at nominal value in the ring demonstrated the objective KPP on June 20, 2017. Figure A11
(right) shows a view of the plunging probe from its vacuum port, while Figure A13 shows a
readout of the signal from this probe.
Figure A9: (Left) NMR Probes embedded in the surface of a vacuum chamber
and (Right) the vacuum chamber ready to install in the ring.
Figure A10: (Left) Data Quality Monitor shows the configuration of fixed probes on
Yoke C, as well as snapshots of signals from those probes. (Right) shows the ability
to stabilize the magnetic field by providing feedback to the main magnet power
supply.
39
Figure A11: (Left) Installed trolley visible through a vacuum
port during transit. (Right) The cylindrical plunging probe
with a copper holder is visible in the background. In the
foreground the 3D translation stage is visible through its port
on the storage ring vacuum, installed in the ring.
40
Achievement of Detector Objective KPP:
All calorimeters and trackers have been installed, thus achieving the Objective KPP for the
Detectors.
This KPP was structured such that the Project could still achieve the Threshold KPP even if
external partners were late in delivering the detectors. The KPP was structured this way because
the calorimeters were funded by a separate NSF MRI, while the trackers were funded through a
combination of Early Career grant and a large In-Kind Contribution from the STFC. In the end,
the partnerships proved reliable, and the equipment from separate funding source were delivered
in time to meet the project installation schedule.
The calorimeters were all delivered in time for the fiscal year 2017 engineering run. The Figure
A14 below shows the data acquired in each of the 24 calorimeters during the fiscal year 2017
engineering run that ended on July 7, 2017.
Figure A12: (Left) The magnetic field deviations from nominal, averaged over all ring
azimuth. The full range of +/-2 part-per-million (ppm) demonstrates excellent
uniformity, and the individual contributions to the moments in the table are all well
under 1 ppm. (Right) The average field at each point around the ring. This achieved
uniformity is 3-4x the uniformity achieved at BNL.
Figure A13: The free induction decay signal from the plunging probe, measuring
in the ring on June 20, 2017.
41
The first tracker station with 8 modules was also installed during the engineering run and Figure
A11 shows an example of how an event passing through the tracker modules appeared in the
FY17 engineering run.
The final 8 modules required for the second tracking station were delivered by the end of
October 2017 and installed just before the CD-4 review.
Figure A15: Straw hits as a particle passes through the 8 modules
of the 1st tracker during the FY17 engineering run.
Figure A14: Snapshots for each of the 24 calorimeters showing the energy spectrum of
decay electrons measured in each detector during the FY17 engineering run.
42
Appendix B: Major External Reviews
DOE CD-4 Review November 14, 2017
DOE Status Review April 6, 2016
DOE CD-2/3 Follow-up Review June 25 – 26, 2015
DOE CD-2/3 Review July 29 – 31, 2014
Director's CD-2/3 Review & IDR June 17 – 19, 2014
DOE CD-1 Review September 17 – 18, 2013
Director's CD-1 Review July 23 – 25, 2013
Independent Design Review June 5 – 7, 2013
Ring Transport Safety & Engineering Review May 21, 2013
http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=875http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=697http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=495http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=490http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=316http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=304http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=269http://gm2-docdb.fnal.gov:8080/cgi-bin/DisplayMeeting?conferenceid=266
43
Appendix C: Project Risk Registry
The table presented below contain select columns for High and Moderate risks from the Muon g-
2 Risk Registry. The complete risk registry and risk log are available upon request. Red in the
tables presented below is for threats that were realized or opportunities missed, and green is for
opportunities realized or threats avoided.
Prior to CD-1 Review
Muon g-2 Realized Opportunities & Threats
Risk
ID#Type
Potential Problem/Opportunity
(short description)Mitigation Action
Threat Feasible transport for superconducting coils or
catasthophic damage during transport
Significant R&D and test welds were performed before the original welds in
the interconnects were cut. No damage has been seen. R&D and test welding
will continue to perfect the procedures before the reconnection is made. If a
failure occurs, a detailed lessons learned analysis will be conducted before a
splice is attempted.
2204 Threat Inability to complete 12Hz Lens pulse testing or
conclude projections of lens operating at 12Hz with a
high degree of confidence could lead to complete
change of requirements, specifications and design of the
entire target station.
Redesign target station
1101 Threat NSF MRI funding falls through Keep lines of communications open with NSF, invite to reviews,
reinforcement from DOE that the project is going forward
2403 Threat AP30 Cryo Duct incompatibility Build new communication ducts between MI8 manholes and AP30 service
building. Requires going through MI8 line berm.
1102/
1103
Threat NSF MRI funding was reduce by $300k in award,
universities not fully able to make up for difference
Work with universities to understand what essential pieces need additional
funding
1107.1 Threat Project is delayed in FY14 due to reduced funding
profile (Recognized in FY13 reduced guidance delayin
Final Design by 3.5 mos)
Keep OHEP informed, monthly tracking of interface milestones to GPP and
AIPs, focus on keeping critical path activities on track
1107.2 Threat Project is delayed in FY14 due to reduced funding
profile (Recognized in FY13 reduced guidance delayin
Final Design by 3.5 mos)
Keep OHEP informed, monthly tracking of interface milestones to GPP and
AIPs, focus on keeping critical path activities on track
Prior to CD-1 Approval
Muon g-2 Realized Opportunities & Threats
Risk
ID#Type
Potential Problem/Opportunity
(short description)Mitigation Action
1111 Opportunity Project is able to schedule a protion of the additional
$1.7M of final design work needed in FY14 to regain
technically-driven schedule
Keep OHEP/laboratory informed, work with universities on forward funding
strategies, and deliver preliminary design work at the base estimate to earn
contingency.
1112 Opportunity Project is able to schedule a portion of additional $6.0M
of Implementation work needed in FY15/16 to regain
technically-driven schedule
Keep OHEP/laboratory informed, work with universities on forward funding
strategies, and deliver preliminary design work at the base estimate to earn
contingency.
Prior to CD-2/3 Review
Muon g-2 Realized Opportunities & Threats
Risk
ID#Type
Potential Problem/Opportunity
(short description)Mitigation Action
1114 Opportunity UK consortium joins with funding support from STFC
to support trackers, reduces project exposure to $250k
tracker risks in registry
Supporting letters from laboratory/OHEP to UK as needed
1105 Threat Early Career funding for trackers not sufficient Early Career funded trackers will be statused by project EVMS so that
problems can be recognized for early intervention
1115 Threat A new inflector is needed (added cost but major
technical opportunity)
Encourage UK collaborators to pick this up as a substantial technical and
intellectuatl contribution, carefully bench test old inflector
44
1116 Opportunity UK consortium joins and takes on new inflector as
contribution
Supporting letters from laboratory/OHEP to UK as needed. Continued
communication with UK collaborators to understand possibilities.
1118 Threat MC-1 Building does not complete its scope identified as
optional
Keep change request on the MC-1 GPP to a minimum since there is currently
$1M contingency remaining. Work with the Directorate to see if the GPP for
the MC-1 building can be authorized to spend at $9.5M instead of the current
$9.0M target.
1119 Threat Magnet test stand needed at BNL for quadrupole tests Fund R&D at BNL in FY14 to determine safety factor in absence of field.
Schedule full field tests as soon as storage ring is powered at FNAL.
1117 Threat Dedicated MRI magnet constructed at Fermilab (aded
cost but major technical opportunity)
Such a facility would be multi-purpose and is thus a good candidate for an
Early Career award. Looking into reusing LANL magnet used in the muonium
hyperfine experiments
2303 Opportunity Simplified "cross-over" scheme to the M3 line, located
further upstream.
Use the new scheme.
2304 Threat Quadrupole733 in M2 quadrupole may require new
type.
Find magnet or change design.
2310 Threat The new magnet designs, the Lambertson and C-magnet,
may prove to be more expensive to design and build
than anticipated.
Mature engineering design will show whether the original cost estimates were
valid or not.
2311 Threat Extraction devices, particularly the Lambertson and C-
Magnet, may require complicated radiation hardening
and shielding to withstand Mu2e operation.
Mu2e completes shielding plan so that the impact is known.
2413 Threat Not enough viable spare vacuum cans. Design and build new vacuum cans to house the SWICs.
2414 Threat Commissioning is not completed during FY'14 studies
period.
Implement device at a later time when funding and engineering support
become available.2202 Threat Default target does not provide required pion yield for
experiment after beam study period summer 2013 or
beginning of run 2016
If the default target in not capable of delivering the required pion production
the alternative design for the target will need to implemented
2207 Opportunity Once preliminary lens operational power supply is
complete, there may be no need to install 13.8Kv 480v
3phase transformer
don't install hardware
2206 Threat Final phase of 12 Hz lens testing shows significantly
more engineering will be needed to run at required rep
rate
Finish bench-testing ASAP in FY14 to mitigate risk so lens is not on critical
path
2308 Threat The design for the in tunnel shielding for Mu2e can have
great bearing on the plan for reconfiguring the 30
straight section. Demands for shielding can change
plans for reconfigurations.
Track and understand the Mu2e shielding design and plan accordingly for the
30 straight section reconfiguration
2408 Threat Fixed BNL SWICs with vacuum window breaks cannot
adequately measure beam profiles or create too many
losses due to scattering.
Build new FNAL SWICs in place of the planned BNL SWICs in the M5 line.
3807 Threat absolute calibration probe is damaged The probe is stored in a safe location and will be carried by hand from BLL to
Fermilab, A alternate, more modern absolute calibration scheme based on He
3 is being developed as a backup option should it be required.
3808 Threat plunging probe is damaged As we refurbish existing probes, we will make spares.
3202 Threat The pole pieces have developed surface imperfections
that take them out of the tight flatness tolerances
The pole pieces were disassembled very carefully and coated to prevent any
further oxidation during shipment and storage at Fermilab. The surfaces may
need to be repolished if they are out of spec. This could require extensive
setup and machining time
4202 Threat Final cost at time of order exceeds preliminary quotes Purchase fewer spares at slightly greater operational risk. Delay some fraction
of purchase. We will make this purchase as quickly as possible using off-
project funds and in-kind contributions.
4204 Threat transient signals are present that cause problems with the
precision magnetic field
These devices can be shielded from the main magnet. All low voltage and
signal lines will be run in parallel to avoid bare currents. However, the plan
must be tested. A field test stand will be set up to measure all DC and transient
fields produced by prototype modules and final boards. If necessary, the
mechanical enclosures might be revised.
4207 Threat Amplifier board performance problems We have made 7 prototypes. We are working with EE faculty now and our
own engineer on a next-generation design. We are following the literature
closely. We are prioritizing this task so revisions can be made rapidly and
prior to the milestone to commit to a final design.
4404 Threat 1 GSPS ADCs become necessary without a significant
reduction in component cost
AMC13 and host µTCA platform can already accommodate the increased data
rates. Timing circuitry local to WFD AMC
3506 Threat Tubes fail to trigger or do not hold off voltage Swap in spare
3507 Opportunity A supply of thyratrons is available from E821 that can
be reused if tests prove operational
45
1121 Opportunity Baseline plan assumed project would pay for vacuum
chamber mods in CD-2/3 Dir schedule, but Early Career
funds should be able to pay
Move funding type to Early Career after review is complete
2401 Threat Inability to complete cable pull Build additional communications ducts between the Central Utility Building
and CMH34
Through CD-23 Follow-up
Muon g-2 Realized Opportunities & Threats
Risk
ID#Type
Potential Problem/Opportunity
(short description)Mitigation Action
3602 Threat Old pulser does not work Test on bench to determine if now pulser has to be manufactured.
3605 Threat Muon losses higher than spec. B-field shimming, active monitoring of radial magnetic field (Br)
1108 Threat Project is delayed by 3 months in FY15 due to:
- Inadequate funding profile
- Delay in delivery of cryogens from Cryo Plant AIP
- Delay in construction schedule of Beamline Enclosure
GPP
- Slip in critical path schedule
- FNAL divisions unable to provide resources
- An injury or ES&H incident results in delay
Keep OHEP/laboratory informed, monthly tracking of interface milestones to
GPP and AIPs, focus on keeping critical path activities on track
1124 Threat FY14 laboratory overheads increase (known at this time
that a $400k increase in FY14 is in the pipeline)
Not much that the project can do to mitigate this risk. Some discussion
ongoing about giving projects a fixed rate.
1104 Opportunity In-kind contribution from INFN for laser system comes
through
Supporting letters from laboratory/OHEP to INFN as needed
2313 Threat Tunnel GPP may not pay for tunnel cable trays. Will not be known until Tunnel GPP is nearing completion.
2401 Threat Inability to complete cable pull Build additional communications ducts between the Central Utility Building
and CMH34
2404 Threat Do not get approval for safety rated PLC for ODH. Obtain approval in as timely a fashion as possible.
2406 Threat Preamps are not sufficient to see the beam. Build new SWICs
2407 Threat SEM hardware not functional Build new SWICs
3201 Threat The yoke is in several pieces and it may not fit back
together on the first try.
The disassembly of the steel was done carefully and was well documented.
Fermilab staff was on site at BNL to understand the procedure. The steel will e
reassembled as soon as it can to ensure it does not become a schedule driver if
problems arise
4210 Threat Original design may not control the noise to the desired
level
Make prototype as early as possible. Work closely with Washington electronics
engineer group who are designing the summing and amplifier board to use
best practices jointly to reduce noise. If necessary, develop some level of
filtering to reduce noise on the low voltage lines.
4211 Threat Delay in selecting SiPMs impacts low-voltage design
timeline
Ideally we will design the low-voltage to be independent of final SiPM vendor
and compatible with any devices. This will introduce a two-stage approach
with a bias offset and a tunable final stage.
4201 Threat Vendor delay in fabrication and growth of crystals We have a local collaborating university in Shanghai, near the Vendor can
keep close track of the production and help to resolve any delays quickly.
4203 Threat instantaneous rate is too high and a gain adjustment
needs to be made during injection
We are performing in-lab flash tests that simulate the effect of high rate at
injection to determine if our boards can recover. We are continuing to
simulate the injection to gauge the expected rates. If necessary, our engineers
will have to introduce a switching circuit in the bias supply
4205 Opportunity SiPM performance improvement We will wait to the last moment to obtain the best products and we will be
testing SiPMs from various vendors to make compare technically to obtain the
best performance solution
4206 Opportunity SiPM cost reduction We are planning to make our readout boards modular and ready for a variety
of standard footprint SiPMs from different manufacturers so that we do not
lock in on only one company. This will allow for a competitive bid for
pricing.
4213 Threat Insufficient light to flash entire system A system involving multiple primary lasers, all synchronized and coordinated
to direct light flashes into a common mixer on the upstream side will have to
be employed.
4214 Threat System stability is insufficient or inconsistent among end
branches
As early as possible, we will built a prototype system with as many branches as
will be required for the full system. Additionally, we will carefully select
Monitor detectors and place them throughout the system.
46
4303 Threat instantaneous rate is too high and a gain adjustment
needs to be made during injection
We will continue to improve our full simulation of the entire accelerator
complex form the target to the ring. This will be verified with accelerator
studies that determine the beam composition. We will continue to simulate
and pin down the maximum acceptable rate in the straws. If necessary, the
circuit will be added.
4306 Threat get a batch of gas with impurities Purity specs will be well documented and communicated to the vendor before
each shipment. Purity will be tested on arrival and monitored in-situ. An
undetected problem could severely degrade the lifetime of the entire system.
4406 Threat required footprint for each channel does not allow 4
channels per board
A full layout will be completed before baselining. If the package does not fit,
more boards will be required.
CD-2/3 Approval to Project Compete
Muon g-2 Realized Opportunities & Threats
Risk
ID#Type
Potential Problem/Opportunity
(short description)Mitigation Action
3607 Threat baseline design of Q1 outer plate technically unfeasible Perform R&D studies and testing of a prototype at early stages of the Project.
Consider DC HV option for Q1 outer.
1113 Threat After first cooldown in FY15, yoke has to be
disassembled to modify or repair cryostats resulting in
more ring PM due to schedule delay. Poles may need
another round of alignment.
Make sure initial alignment is meeting spec. Quality assurance/control during
assembly.
4307 Threat tracker production experiences delay The design for the modules has been simplified so that only one geometry is
being produced which will allow for faster production with only one module
type. Priority will be given two getting the first two trackers installed and it
would not compromise the physics results if the 3rd tracker was installed in the
first shut-down
3611 Opportunity reuse existing insulators Test the system for holding the HV
1120 Opportunity Opportunity to advance accelerator construction
schedule by 2 +/- 1 months
with an addition $2-4M in funds advanced from FY17 to
FY15/16. Current technically-driven schedule has to be
held out to meet funding profile.
Continue to accomplish work as close to baseline cost as possible in order to
move contingency into schedule work. Keep OHEP and lab informed of
difference between technically and financially-driven schedule
1131 Threat Although this is a cost threat, it is really driven by a
technical opportunity. The injection efficiency into the
storage ring could be doubled with a new inflector
Project and scientific collaboration will continue to look for means of alternate
funding, although there are no prospects at this time.
2416 Threat Bayonet and antivacuum can not able to hold required
vacuum levels.
Redesign the antivacuum box to provide a better vacuum seal.
2418 Threat Bayonet and antivacuum can not able to hold required
vacuum levels.
Redesign the antivacuum box to provide a better vacuum seal.
3606 Opportunity Reuse old insulators Test on bench to determine if quads hold voltage with old insulators, measure
leak current
4302 Threat system alignment out of spec Alignment will be done in a beam test and using lost muons. Individual straws
will be X-rayed to determine wire positions. A stand-alone alignment system
using for example hall probes may be needed.
4405 Threat Clock distribution via AMC13 does not meet
experimental requirements of stability, particularly for
clock frequency or phase drift over muon fill
1) Clock distribution performance will be characterized on test stand, currently
under assembly. 2) Option for separation of clock and control signals, with
dedicated clock input on new AMC13 mezzanine card; 3) Include front-panel
clock input on WFD AMC, bypassing AMC13 and µTCA distribution
1109 Threat Project is delayed by 3 months in FY16 due to:
- Inadequate funding profile
- Slip in critical path schedule
- FNAL divisions unable provide resources
- An injury or ES&H incident results in delay
- The He leak in the magnet degrades such that
shimming is delayed
Keep OHEP/laboratory informed, monthly tracking of interface milestones to
GPP and AIPs, focus on keeping critical path activities on track. The magnet
is being monitored for evidenc that the leak might be degrading, no evidence
so far. If the He leak did increase, then the pumping speed could be roughly
doubled in the area of the leak with relative ease.
4308 Threat Straws do not meet QC spec on diameter We are purchasing much more material than we need to allow for a low
throughput. Raw material will be converted into straws In 4 batches and QC
will be made after each batch.
1123 Threat Funding constraints push the start date for much of the
accelerator work into FY16. This leads to a significant
risk of schedule slip that is worth considering separately
from the schedule risks summarized in risks 1109/1110
Use any contingency that might be liberated in FY15 to start accelerator work
earlier
1132 Threat Some problem develop with the storage ring that
operational budgets are unable to pay.
Working with PPD to ensure operations budgets remain adequate. Minimizing
risk to storage ring damage with proper, regulary updated procedures and
operator training.
47
2402 Threat HRM Incompatibility Establish CAMAC links at MC-1
3811 Threat higher order multipoles exist that are out of dynamic
range of the shimming kit.
The field uniformity requirements are well specified and communicated. The
detector geometries are chosen to minimize higher multipole fields. The muon
beam is round to minimize the effects of higher order multipoles. In the event
that something does occur, either the source would need replacement or a
solution would need to be devised using a combination of the passive and
active shields.
4401 Threat Environmental noise, particularly noise correlated with
fill structure, introduces timing shift
1) Clock distribution performance will be characterized on test stand, currently
under assembly. Testing will include sensitivity to prototype kicker operation
to mimic a potential fill-correlated noise source. 2) Base design on integrated
components like TI LMK04010, rather than discrete components, to limit
sensitivity to noise; 3) Include clock monitoring within WFD
2302 Threat Desired spot size is not achieved or simulations
overstated improved pion yield from smaller spot size.
None
3806 Threat Current trolley frequency measurement (onboard
electronics) need to be replaced entirely.
Careful handling to prevent any damage. Careful evaluation of systematic
errors related to the frequency measurement in the test solenoid as soon as
possible to investigate this risk. Test solenoid is being established at Argonne
and past E821 experts are meeting there in June to test the full system.
3809 Threat Delayed CENPA electronics shop availability. Some projects in the shop's queue could be outsourced or additional part time
personnel could be hired.
2315 Threat Until design and constructions work is 100% complete
there is some risk that the kicker power supply might be
more expensive than anticipated
Complete design work at 100% level as quickly as possible, advance
contruction schedule if project funds allow
1110 Threat Project is delayed by 3 months in FY17 due to:
- Inadequate funding profile
- Slip in critical path schedule due to large amount of
accelerator work scheduled in FY17
- FNAL divisions unable to provide resources
- An injury or ES&H incident results in delay
Keep OHEP/laboratory informed, monthly tracking of interface milestones to
GPP and AIPs, focus on keeping critical path activities on track
2417 Threat Ion Chamber and Digitizer are not sensitive enough to
see 10^5 beam
Design a low intensity intensity measurement device based on scintillators.
3402 Threat Required pumping speed is much greater than the
pumping speed achieved at E821
We will collaborate closely with the quad and tracker groups to ensure any
changes in requirements are communicated as soon as possible. Secondary
containment vessels may be necessary to increase pumping speed locally. For
a drastic change, a new system may need to be designed and implemented.
3610 Threat ESQ system alignment out of spec Identify potential problems in advance (in R&D), upgrade problem
components, develop control and monitoring procedure
4403 Threat Transient currents that perturb precision magnetic field All DC and transient fields produced by µTCA crates, prototype modules and
final boards will be measured in the magnetic field test stand. Shielding
strategies will first be exploited if necessary, followed by locating the crates
farther from the field (longer signal cables) If these actions insufficient, a
large design iteration may be needed.
48
Appendix D: Detailed Technical Lessons Learned from CD-0 through CD-3
Three most significant “success” lessons for this project
Lessons Learned
Description, Impacts, and Solutions
Building a strong
and successful
project management
team
The project management team for the Muon g-2 project was assembled over a 2 year period. Key members of the team were actively recruited
for over a year. The team was augmented prior to CD1 by a deputy
project manager with significant project management experience.
The management team has become strong and successful and works together to lead the collaboration through the design and fabrication
process and through the associated project reviews. It has been
particularly important to have such a talented team to navigate reviews,
project execution, and the transition to operations.
Preparation of the project plan and all documentation for each DOE CD
review was preceded with a “dry run” managed by the Laboratory. This
enabled the project to benefit from feedback and make corrections in
the development of the project baseline prior to the formal DOE
review.
Early
implementation of
an Earned Value
Management
System
Starting the roll-out of the EVM system in the months prior to CD-1 allowed the project to assess very early where cost & schedule
problems were developing pre-baseline. It also allowed the project
team to benefit from a significant amount of time to learn how to use
EVM as it evolved from a basic analysis at CD-1 to the full EVM and
project management process deployed by the baseline review.
Project tasks were created with durations that were trackable on a monthly time frame so that progress could be monitored in conjunction
with the monthly budget reports.
For project management and engineering oversight, “level of effort” tasks are appropriate for managing the budget for these resources.
Project analysts needed to work closely with the technical managers to understand activity sequences before developing the schedule logic.
It is very important
to select high
quality vendors.
The major vendor contract for the g-2 project entailed the safe
transportation of the BNL storage ring from Long Island to Fermilab
Emmert International was selected even though they were the high bid due to the strong value added.
Safety was a key part of their company culture and they led meetings at BNL, FNAL and with the transportation agencies to ensure Emmert
was compliant with all safety protocols.
Emmert understood the delicate nature of the storage ring and worked with the project team to ensure damage to the ring would be avoided at
all steps of the transport.
Emmert also worked well with project and communications personnel to turn the transport of the ring into a very successful PR event for the
field.
49
Three areas of potential improvement and how they have or might have
impacted the project.
Lessons Learned
Description, Impacts, and Solutions
Estimating the cost of
refurbishment of old
equipment can benefit
from more
assessment in the
final design phase.
Estimating the cost of refurbishing equipment is difficult. The repeated
lesson seems to be that as soon as implementation work starts, new
information is discovered, and the plan for old equipment is likely to
change. Redefining the initial stages of that work to be final design and
completed ahead of baselining would be advantageous. Examples
include
Once the storage ring was moved into the building and could really be taken apart, it became clear that much more refurbishment of
valves and seals would be required, which also resulted in much
more leak checking.
Dismantling of the inflector lead box and valve can revealed that the corrosion and piping not meeting ASME standards could be repaired
more cost effectively than rebuilding a new system from scratch.
Extracting vacuum cages from the chambers and doing as-founds emphasized difficulties in meeting the alignment spec for quad plates
and trolley rails.
The original plan assume the electrostatic quadrupole pulsers would be reused with minor refurbishment. Initial analysis from engineers
determined that the cost would be prohibitive, the lack of
documentation would make it very difficult, and the end product
would not meet current standards.
Other areas where
costing lessons were
learned.
Drafting was an area where estimates were consistently low. For beamline installation it was found that the design needed to produce
a defensible beamline final design was about 1/3 of the final drafting
needed to be construction ready.
Alignment time, especially with a device like the g-2 storage ring where alignment errors directly generate systematics, was initially
underestimated by a factor of 2 to 3.
Beamline installation requires a degree of oversight from engineers and engineering physicists that was not in the original cost model.
50
Technical reviews Throughout the DOE CD review process the project had trouble communicating all of the technical details to the reviewers during the
short timescale associated with the review. The project learned of
the importance of contacting the DOE reviewers several weeks in
advance to start the dialogue as soon as possible. A much better
model is to conduct multiple, project-commissioned reviews with a
small panel of experts that are given 1-2 months to study the issues.
In the year between the first CD-2/3 review in summer 2014 and the
follow-up CD-2/3 review in 2015, the project conducted 3-4
technical reviews of particularly complicated subsystems. Having
these mini-review reports available to the DOE reviewers made their
job much easier and allowed the project to much more effectively
communicate the degree of construction readiness.
Other lessons learned.
Lessons Learned
Description, Impacts, and Solutions
L3 managers and
scientists with technical
expertise.
Accelerator L3 managers and installation coordinator each had more than 20 years of experience operating accelerators at
Fermilab.
Iteration on BoEs between L3 managers and engineers helped to ensure that all scope was accounted for.
Experience with beamlines being reconfigured had a positive
influence on new beamline design in avoiding known pitfalls and
developing creative solutions.
Strong engagement of the experimental research collaborators, supporting the project team, added much value to the quality of
project baseline development and execution.
Selecting an appropriate
number of CAMs The project entered the CD-2 review cycle with roughly one
CAM per L3 WBS element. A reexamination of the amount of
work in different WBS elements led to a redistribution of control
accounts with 7 CAMs, thus making the project more
manageable.
51
The risk-mitigating
strategy of delaying
construction approval
until the storage ring
could be tested had both
positive and negative
impacts on the project
Advantages:
Beneficial for minimizing project risk by requiring a full demonstration of the g-2 storage ring prior to full approval for
funding the implementation phase.
Helped to keep the ring work as a high priority at the laboratory.
Interim funding through special authorizations from the DOE Project Management Executive allowed the ring work to progress
($2.7M) and critical path accelerator implementation ($2.9M)
kept the schedule largely on track.
Disadvantages:
The next stage of funding being so closely tied to the ring demonstration created a very high pressure situation for the
management and engineers.
The finite ‘special authorization’ funding was set initially and was based on needing to use estimate uncertainty, but did not
cover all of the risk.
A couple of risks were realized during this time period including needing to repair a known cold leak from the BNL era and a
serious schedule delay that arose from discovering a poor indium
connection in the lead can.
The refurbishment of the ring cryo system and construction of new cryogenic lines was greatly underestimated by more than
50%.
In general, the combination of the finite funding running out (prior to CD-2/3) and the schedule delay conflicted with taking
the time to carefully implement quality control. As a specific
example, more time spent on the analysis of the resistance
measurements of the reconnected indium joint might have
revealed the need to repair this joint prior to the first cool-down.
52
Appendix E: Detailed Technical Lessons Learned from CD-3 through CD-4
WBS 1.0 Project Management
Lesson Learned
Description, Impacts, and Solutions
Success:
Custom project
management tools made
work more efficient and
effective.
The project office constructed a number of custom PM tools
that worked well with positive feedback from the managers,
many of which have had experience with tools used in other
projects. In particular, a standardized Basis of Estimate form
was created in Excel to assess costs and collect the activity-
level descriptions for constructing the P6 schedule. A second
extremely useful tool was a shared Google spreadsheet that was
used on a monthly basis to collect the status of all of the
activities for generating the monthly EVMS. Finally, a
spreadsheet based risk registry that performed Monte Carlo
calculations using Excel macros was used for tracking
remaining cost risk, but had the drawback of not calculating
remaining schedule risk. In general, Google shared documents
were used for a number of other areas including issue tracking
and compiling specifications. The simplicity of the tools
enabled broad buy-in and prompt response from managers,
including university-based collaborators.
Success:
Management team
composition led to coherent,
effective team.
The project team included a number of early career scientists
with technical skill and enthusiasm that made up for their initial
inexperience with the formalities of DOE project management.
The Project Manager, an early-career Wilson Fellow, had
extensive subject-matter expertise, having been a graduate
student on the previous Muon g-2 experiment at Brookhaven.
He and the other early-career scientists in key leadership
positions (Level-1 and Level-2 managers) inspired and
empowered the entire team. To compensate for their initial
dearth of management inexperience, a senior scientist and
skilled manager served as the Deputy PM prior to CD-2 and
helped guide and mentor the team during the project-
development phase, with additional help from the Fermilab
Office of Program and Project Support (OPPS) and
management training classes delivered by the Laboratory.
53
Success:
Project and scientific
collaboration worked
together effectively to
achieve project goals.
Throughout the course of the project, reviewers commented
about how effectively the project and the scientific
collaboration worked together. There were a number of factors
that contributed to this. One was having a project manager that
was also a subject matter expert, which made it easier to reach
consensus on priorities. It also helped that the spokespeople
and project leadership had weekly meetings to make sure that
everyone at the top of both organization charts were constantly
in agreement on the best course of action. The project actively
sought out ways to bring additional scientific effort from the
collaboration to assist with the project deliverables. This
included technical work in the construction and assembly,
simulations to inform design decisions, assignment of
management positions within the project, and R&D and testing,
and quality control. This had the two-fold benefit of reducing
the project cost compared to relying solely on laboratory
resources and helped to keep a large portion of the collaboration
positively engaged with the project.
Success:
Tracking uncosted scientific
effort ensured accurate
progress measurement.
The project relied on effort from the scientific collaborators,
much of which is not costed to the project, per DOE HEP
guidance. Understanding the effort needed, and measuring
progress on tasks performed using that effort, was a priority for
the project. The Project measured Earned Value for these tasks
by including all activities needed to complete project
deliverables, with estimates of how much uncosted labor was