Eddy Jans 1Vertex2013
Operational aspects of the
VELO cooling system of LHCb
Eddy Jans (Nikhef)on behalf of the LHCb VELO group
• Introduction• Main components and operation principle of the system• Issues: how to prevent and to tackle them• Keep the detectors cold 24/7• Summary & outlook
19 September 2013
Eddy Jans 2
Introduction• VELO-module is double-sided (300 mm, oxygenated, n+-on-n) and operated in vacuum,• Strip closest to the beams is at 8.2 mm,• Per double-sided module: 2x16 frontend chips that together dissipate ~20 W,• 4 NTCs give temperature readings• Two movable detector halves with 21 VELO + 2 PileUp modules each (400 W/side)
Vertex2013
NTCs
19 September 2013
Vacuum tank
One detector half
Detectors are operated in a secondary vacuum absolutely no leaks allowed.Orbital welding or vacuum brazing of the pipes. All tested at 170 bars.
cooling block frontend chips
module base iskept at +20 oC
CO2 connections
Eddy Jans 3Vertex201319 September 2013
0.3 mm thick RF-box
• in 1999 it was proposed to use CO2 as refrigerant for the vertex detector of LHCb [LHCb note 99-046], • CO2 is radiation hard,• CO2 has a high latent heat value can use small diameter capillaries small amount of dead material in the acceptance,• stainless steel capillaries: Finner=1 mm, wall thickness 0.25 mm• system uses bi-phase CO2 via the accumulator controlled method.
Some cooling considerations
cooling blocks
Eddy Jans 4Vertex201319 September 2013
Pressure x Enthalpy diagram of CO2
At -30 oC: 300 J/g
mass flow: 10 g/sper module: 0.43 g/sfull evaporation130 W
liquid CO2-speed: 28 cm/s all-gas speed: 240 cm/s
bi-phasearea
pre
ssu
re [
MP
a]
enthalpy [kJ/kg]
bi-phasearea
-30 oC
gasphase
liquidphase
pre
ssu
re [
bar]
criticalpoint
vapour quality 0 1
isothermal cooling
Eddy Jans 5Vertex201319 September 2013
liquid vapor
bi-phase
EnthalpyP
ressu
re
1
2 3
4 56P7
The accumulator controlled cooling cycle
12: pump increases the pressure of the sub-cooled liquid23: heat exchange in the transfer line brings evaporator pre-expansion per definition right above saturation point, since (E2-E3) = -(E5-E6)34: pressure drop in restriction and expansion in capillary brings CO2 in cooling blocks in bi-phase state,45: isothermal cooling via evaporation56: warming up of incoming sub-cooled liquid61: condensation and cooling of the CO2
R5
07
a c
hille
rs
Con
den
ser
pump
evaporator
restriction
accumulator
Heat
in
Heat
in
Heat
ou
t
Heat
ou
t
1
2 34
5
6
P7
55 m transfer line
insulationf=80 mmsub-cooled liquid inbi-phase return
only passivecomponentsin the radiationzone
Eddy Jans 6Vertex201319 September 2013
Main components of the system
Air-cooled SB chiller
Water-cooled SA chiller
Velovacuum tank with silicon modules, module base and CO2
evaporator.
TLR – CO2 system TR – CO2 systemTL – CO2 system
TL – CO2 bridge
TR t
rans
fer
line
TL t
rans
fer
line
TR – CO2 bridge
Liquid Pump
Electronic 3-way valve
Electronic 2-way valve
Manual restriction valve
No return valve
Electric heater
Damper with heater
Heat exchanger
PLC and electronics rack
Cooling plant at UXA03
VELO
• evaporative CO2 cooling system• “independent” system for either side • PLC-controlled• 2.5 kW water-cooled chiller at –40 oC• 1 kW air-cooled backup chiller at -25 oC• 55 m CO2 transfer lines• 10 heat exchangers• 8 actuators• 9 heaters• 31 pressure sensors• 192 temperature sensors• 350 parameters monitored in PVSS• only passive components at VELO• 2*400 W heat load of detectors• 2*12 kg CO2
Performance @ detector• main chiller: -28 oC operational CO2 temp. LV on: sensors @ -7 oC• backup chiller: LV off: @ -8 oC• stability < 0.1 oC
Eddy Jans 7Vertex201319 September 2013
Design considerations and operational experience
• redundancy of crucial components• insulation• clogging filters• superheated CO2
• dependence on electrical power• dependence on chilled water• safety measures to prevent overheated detectors• keeping the system 24/7 cold
Eddy Jans 8Vertex201319 September 2013
Redundancy in the design
To minimise down time the system has a few redundant crucial components:• 3 CO2 pumps, where 2 are needed,• 2 chillers, water-cooled and an air-cooled as backup,• for controls crucial temperature and pressure sensors are two-fold implemented,• possibility to interconnect the two sides by hand,• PLC is on a 1500 VA/1000 W UPS,• PLC, backup chiller and CO2 pumps are connected to a diesel generator.
Eddy Jans 9Vertex201319 September 2013
InsulationLiquid pumped system cold transfer lines good thermal insulation required. This seems trivial, but turned out not to be so in practice.Originally CERN safety regulations forced us touse Armaflex NH.Glued surfaces started to delaminate after 2 years.
Renewed insulation of the transfer lines and most of the cooling plant during Winter shutdown ‘10-’11.Now foamglass covered by an Aluminium protection shield and Armaflex AF, respectively.
Eddy Jans 10Vertex201319 September 2013
Filters
piece of Armaflexthermalinsulationonce completelyblockeda restrictionvalve
Throughout the system eleven 15 mm filters are installed. (5 (CO2-plant), 2(@VELO), 2(manifold), 1(main chiller), 1(backup chiller)).
In one detector half we have experienced a few times clogging filters.
Replacement procedure is tricky and risk of additional dirt in the system due to difficult accessible filter houses.
2 months
pressure [bar]
15
16
17
Eddy Jans 11Vertex201319 September 2013
Post mortem analysis of the filters
0 5 10Energy (keV)
0
5
10
15
20
25
cps
O
Cr
Fe
Si CrCr
Fe
Fe
Ni
Many >15 µm orange objects have been observed inside the filter. They mainly contain Fe and O.Before, particles containing Cl had been observed. Possibly due to connections soldered with flux for a testbeam experiment risk in terms of corrosion. Work extremely clean from construction to installation.
Scanning Electron Microscope image
Energy Dispersive Spectrometer analysis of an orange particlefound inside the filter
75 mm
Eddy Jans 12Vertex201319 September 2013
Superheated CO2 after startup
After startup we occasionally observe in a varying number (a fewall)of cooling blocks the phenomenon of superheated CO2.
Issue: cooling performance is very bad because liquid cooling has much less cooling power than evaporative cooling.
Eddy Jans 13Vertex201319 September 2013
Superheated CO2 after warmup and cooldown
DT=3 oC-30
-22
-14
time
Tsilicon
[oC]LV off
warmupof thecoolingplant
cooldown
30 minutes
Eddy Jans 14Vertex201319 September 2013
Superheated CO2 after warmup and cooldown
DT=3 oC-30
-22
-14
time
Tsilicon
[oC]
startaddingheat
30 minutes
Remedy: add heat by means of a dedicated heater to bring the incoming CO2 in the liquid+gas state.
LV offliquid vapor
bi-phase
1
2 3
4 56
Enthalpy
Eddy Jans 15Vertex201319 September 2013
Pre
ssu
re
Not all cooling blocks behave the same way:- not all show superheating- when adding heat they don’t start boiling at the same moment
Some more superheated CO2
startaddingheat
silicon temperatures of 4 modules
DT4 oC
Eddy Jans 16Vertex201319 September 2013
Power cutsPLC and backup chiller are connected to the power of a diesel generator of LHCb and the PLC also to its own UPS (1500 VA/1000 W)
When the power gets cut the switch-over from main to backup chiller is handled automatically by the PLC.
After switching back to the mainchiller the system is stable after ~20 minutes.After switching on the LV the sensors are at their operational temperatures after 10 minutes. half an hour recovery time.
-10
-20
-30
sensor temperature
10 minutes
LV onEddy Jans 17Vertex201319 September 2013
Failure of chilled water supply
Chilled water supply, that cools the main chiller, sometimes gets interrupted.If so, the PLC switches on the air-cooled backup chiller.However, this causes the LV to be switched off also.
Eddy Jans 18Vertex201319 September 2013
Safety
1. HW-based: interlock system
Beam Conditions
Monitor
Cooling-PLC
Vacuum-PLC
Module temperatur
esTemp-boards
Conditions
LV Off
HV Off
Retract VELO
Cooling Off
Actions
2. SW-based: warning and interlock system
132 cooling parameters monitored
3 levels each: warning, error and fatal
Emergency button
Combined information of 4 NTCs per module are input to the FPGA,which can interlock the LV.
Operation in vacuum requires immediate reacting safety systems. Three levels.
Eddy Jans 19Vertex201319 September 2013
3. Human-based
Emergencybutton inthe LHCb control roomto power offthe VELO.
Eddy Jans 20Vertex201319 September 2013
Keep the detectors cold 24/7At the tip the received fluence is 2x1014 neq/cm2 and type inversion has
taken place, so the sensors should always be kept cold, (below -8 oC), in order to prevent the Vdepletion to increase due to reverse annealing.
The beneficial annealing budget amounts to a handful of weeks at room temperature. We try to save it till we really need it. But the conditions of the LongShutdown1 period at LHCb make it hard to do so.
Vdepl NC + DNeff (=effective space charge density)
• Short term: “Beneficial annealing (NA)” • Long term: “Reverse annealing (NY)” - time constant depends on temperature: ~ 500 years (-10°C) ~ 500 days ( 20°C) ~ 21 hours ( 60°C)
NC
NC0
gC eq
NYNA
1 10 100 1000 10000annealing time at 60oC [min]
0
2
4
6
8
10
N
eff [
1011
cm-3
]
[M.Moll, PhD thesis 1999, Uni Hamburg]
Eddy Jans 21Vertex201319 September 2013
The challenge is to keep the system operational 24/7.
Under normal conditions the PLC deals with common problems.
Goal: minimize the warm time due to• scheduled maintenance of crucial components,• repair of malfunctioning components,• unexpected problems during LS1 and shutdown periods.
Eddy Jans 22Vertex201319 September 2013
Regular maintenanceYearly maintenance of the R507a chillers is performed by a specialized external company. Downtime ~0.5 day / chiller.
Yearly maintenance of the 3 CO2 pumps is done by Nikhef-technicians. Pump is unavailable for >24 hours.Effective downtime of the system: 2 hours / pump.
Repair of failing componentsSo far no component had to be replaced, although a (redundant) pressure gauge stopped working in 2012, but miraculously reincarnatedafter 6 months.
Eddy Jans 23Vertex201319 September 2013
Unexpected problemsHow do you know a serious problem occured, causing the cooling to go off and the detector to warm up ? Especially during LS1.Can’t rely on a PVSS-script sending a mail or sms.A modem and sms-routine have been installed in the PLC.
modem withSunrise sim card, so worksunderground.
antenna
When a problem occurs every half hour a text message is sent to alist of phones numbers, until the cooling system is again in a proper state.Acted 6 times since Feb. ‘13 due to failing services.
Eddy Jans 24Vertex201319 September 2013
Integrated warm time in 2012: ~1 day
0
-30
Eddy Jans 25Vertex201319 September 2013
year 2012
= 8 hours
18
0
-10
Summary• cooling system is continuously operational since >4 years,• performance is stable and according to specs,• redundancy of crucial components has shown to pay off,• clogging filters are annoying,• good thermal insulation is less simple than it seems,• superheated CO2 can be dealt with,• the warning system that sends sms-es is a great tool,• thus far the integrated warm time has been ~1 day/year, so ……….
Eddy Jans 26Vertex201319 September 2013
Eddy Jans 27
Outlook
Eddy Jans 27Vertex201319 September 2013
lets keep it cool till LS2, when the new VELO pixel detector goes in.