Operational Efficiency
Walter Venturini Delsolaro
LHC Beam Operation Workshop - Evian 7-9 December 2010
Outline• Quick run through the run• Definitions and method• Fault statistics• Dump statistics• Technical stops• Tools• Operational efficiency
“Now, what I want is, facts. Teach these boys and girls nothing but Facts. Facts alone are wanted in life. Plant nothing else, and root out everything else. You can only form the minds of reasoning animals upon Facts: nothing else will ever be of any service to them. (…) Stick to Facts, sir!”
"a little inaccuracy saves a world of explanation"
Quick run through the run• 1-30 March: ramp commissioning,
first collisions• 1-16 April: squeeze commissioning,
then Physics • May: increasing Nb and kb, Physics• Intense summer: pushing & Physics• Canicular Physics • September: resuming commissioning • Bunch harvest• Heavy November
Method and Definitions • From 1 March to 30 November: 6600 hours• From the logbook, cross checked with status
reports of coordination and Timber for the beam presence
• Grid: Setup no beam (grey), beam setup (silver), stable beams (gold), TS/HC (purple), Fault (red)
• Availability = Beam presence + Setup no beam
• Downtime=Fault + TS/HC– Additional setup time after faults credited to the
faulty system– Not straightforward (coupled faults)
• … Shake and pie it up
Initial commissioning (March pie )
beam setup42%
Stable beams1%
setup no beam10%
Technical stop/HC19%
Fault29% 53%
availability
March faults distribution
CryogenicsQPS
EL+UPSPC
LBDSInjectorsVacuum
Access systemQ, Qp Feedbacks
RFControls
BLMExperiments
CVIQCMKI
0 1 2 3 4 5 6 7 8
April is the cruellest month…?
beam setup40%
Stable beams17%
setup no
beam9%
Technical stop/HC
13%
Fault22%
65% availability
April faults distribution
CryogenicsQPS
PCEL+UPS
LBDSInjectors
Q, Qp FeedbacksOP
ControlsAccess system
MKIVacuum
NOFBLM
QuenchIT
RFCollimators
Experiments
0 1 2 3 4 5 6
What the Thunder said
beam setup41%
Stable beams15%
setup no
beam8%
Tech-nical stop/HC6%
Fault30%
64% availability
All May faults
EL+UPSCryogenics
QPSPC
InjectorsLBDSBLM
CVControls
OPCollimators
MKIExperiments
Q, Qp FeedbacksRF
BICSoftIQC
0 2 4 6 8 10 12
June, change of Tune
beam setup44%
Stable beams4%
setup no beam8%
Technical stop/HC9%
Fault36%
56% availability
June faults
Cryogenics
QPS
NOF
Controls
RF
LBDS
CV
OP
MKI
Experiments
0 2 4 6 8 10 12
Another pie…July
beam setup35%
Stable beams18%
setup no beam
7%
Techni-cal
stop/HC15%
Fault24%
61% availability
…other testimony of summer nights
QPSPC
EL+UPSCollimators
Access systemOP
InjectorsControls
RFCryogenics
BLMQ, Qp Feedbacks
IQCMKI
LBDSBIC
Experiments
0 1 2 3 4 5 6
The summers corny crownbeam setup
31%
Stable beams29%
setup no
beam6%
Technical stop/HC7%
Fault26%
66% availability
Usual suspects
QPSCryogenics
PCEL+UPS
CVInjectors
Access systemBLM
RFPIC
settingsCollimators
MKIControls
Q, Qp FeedbacksBPM
VacuumOP
0 1 2 3 4 5 6
September trains
beam setup51%
Stable beams10%
setup no beam
11%
Technical stop/HC7%
Fault21%
72% availability
September faults distribution
QPS
PC
Injectors
Access system
EL+UPS
Controls
PIC
MKI
LBDS
Q, Qp Feedbacks
0 1 2 3 4 5 6
October slices
beam setup41%
Stable beams18%
setup no
beam10%
Techni-cal
stop/HC12%
Fault19%
69% availability
October faults distribution
CryogenicsInjectors
QPSPC
EL+UPSExperiments
BLMLBDS
TimingOPBICRF
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
Heavy ending
beam setup44%
Stable beams26%
setup no
beam11%
Techni-cal
stop/HC1%
Fault19%
80% availability !
Heavy faults
QPSPC
InjectorsAccess system
NOFLBDS
CVEL+UPS
CollimatorsQ, Qp Feedbacks
RFCryogenics
ControlsExperiments
OPMKIIQC
0 1 2 3 4 5 6 7
All faults downtime distribution
0 1 2 3 4 5 6
QPSCryogenics
PCEL+UPS
InjectorsAccess sys tem
LBDSCol l imators
ControlsRFOP
BLMCV
Q, Qp FeedbacksExperiments
NOFMKI
VacuumBICPIC
Alarm-fi re IT
IQCsetti ngs
BPMTiming
SoftQuench
Equipment type Faults Qty. Availability[1] [%] MTBF [hours]
Quench heater power supplies 26 6076 99.998 1145760
Quench detection systems 19 10438 99.999 3362135
DAQ caused by radiation (SEU) 12 1624 99.997 828240
DAQ other causes than radiation 8 2532 99.999 1936980
DAQ all faults combined 20 2532 99.997 774792
EE600 6 202 99.988 206040
EE13 kA 5 32 99.939 39168
QPS wins by a neck…
R. Denz
0
10
20
30
40
50
60
70
80
90
100
25 01 22 02 22 03 19 04 17 05 14 06 12 07 09 08 06 09 04 10 01 11 29 11
Avai
labi
lity
Daily "Global LHC Cryo" weekly AVG Monthly AVG Scheduled Stops
LHC Cryo global availability
Powering tests
Perturbations: clogging sub-atm circuits-CV891-instrumentation-Shaft seals-VFD/MB-24V
Learning spring Fantastic since summer !
Results for 2010 above expectations, thanks as well to periodic technical stops
S. Claudet
Dump statistics during ramp
BLM
Coll
MP test
SIS COD/orbit
OP mistake
software
LBDS fault
Exp
controls
0 2 4 6 8 10 12
74 protection dumps
Dump statistics during squeeze
PC tripOP switch
BLMCryoQPS
BPM IP6BPM
LBDS faultBISCollOFB
OP mistake
0 2 4 6 8 10 12 14
53 protection dumps
Dump statistics: from stable beams
BLMEL pert
QPS PC trip
LBDS faultOP switch
CryoBLM (spurious)
WaterBPM IP6
Exp (spurious)Access
OFBBPM
SIS CODRF (cryo)
Coll
0 2 4 6 8 10 12 14
66 protection dumps
Technical Stops• A total of 6 were done as scheduled • First started on March 15• Pattern: 4-36-3-31-4-45-5-37-4-45-4-
40+• Naive question: is the machine
availability more or less after a TS ? • Consider the 72 hours preceding and
the 72 hours following a TS, and compare pies...
• Compare faults for the various systems
BEAM (%)STABLE Beams
Setup no beam TS-HC Access FAULT (%)
before TS 75.00 0.00 13.89 8.33 6.94 5.56after TS 55.63 0.00 14.79 0.00 8.45 30.99DELTA -19.37 0.00 0.90 -8.33 1.51 25.43before TS 70.14 48.61 9.03 0.00 11.11 22.22after TS 49.65 0.00 11.81 0.00 24.31 42.71DELTA -20.49 -48.61 2.78 0.00 13.19 20.49before TS 61.81 1.39 9.03 0.00 6.94 29.86after TS 51.39 5.56 15.28 0.00 5.56 34.72DELTA -10.42 4.17 6.25 0.00 -1.39 4.86before TS 63.19 28.47 5.56 0.00 5.56 31.94after TS 45.14 0.00 27.78 0.00 11.11 28.47DELTA -18.06 -28.47 22.22 0.00 5.56 -3.47before TS 64.58 39.58 8.33 0.69 22.22 29.17after TS 54.86 0.00 14.58 0.00 0.00 31.94DELTA -9.72 -39.58 6.25 -0.69 -22.22 2.78before TS 67.71 15.28 20.14 0.00 2.78 14.93after TS 80.56 29.86 16.67 0.00 0.00 4.17DELTA 12.85 14.58 -3.47 0.00 -2.78 -10.76
All faults: after TS- before TS
Trends of “TS messing up” effect...
0 1 2 3 4 5 6 7-15
-10
-5
0
5
10
15
20
25
30
Tools• Logbooks to get the fault attributions• Measurement and Logging DB: a lot of information,
JAVA API available to do specific searches, already used by some people (C. Roderick)
• Web-based Post Mortem Data Extraction (M. Zerlauth)
For next year: try at least to copy SPS: automatic entries in the logbook when there is a fault, for LHC it is more difficult, one has to take into account machine modes, etc. (more in Oliver’s talk). Also we need to agree on conventions.
Several people are working to applications for the Fill statistics (e.g. SUPERTABLE, and others), using the same API used by TIMBER
Wrap up • Machine availability for the run: 65%• Faults: 25% (TS 10%)• Beam presence: 56%, setup no beam: 9%• Stable beams: 15.7 % (e1)• e2=Physics/Available: 23.7%• For most of the 2010 run, e2 is not a good indicator of
operational efficiency, as it rejects all the beam commissioning time
• Last two weeks of August: e2 ~ 50%. • Max e2 = 83 % (with 10.6 hs fill time and minimum
turnaround)– With 65% machine availability and only trying to do phyics:
Max e1 = 54% , or 32% if we had the same efficiency as in August
Conclusions• 2010 run was driven by commissioning, not
physics• Machine Availability was satisfactory and steadily
increasing• Equipment performs above expectations (MTBF
etc)• Equipment groups are aware of the weak points
and are working to improve them• Less mixing of Physics and beam commissioning
in 2011• TS to be reviewed for 2011, can probably be less
frequent• More tools for statistics to be developed, also with
discussions with equipment groups • Margin to improve operational efficiency ( see
Stefano’ talk)