Changes in Powering InterlocksI.Romera, TE-MPE
MPP Workshop – 11th-13th March 2013
Thanks to: P.Dahlen, S.Gunther, R.Mompo, S.Ravat, H.Thiesen, M.Zerlauth
I.Romera - TE/MPEMPP Workshop March 2013
PowerInterlock
Controllers
BeamInterlockSystem
Beam Dumping System
Quench Protection System
Power Converters
Cryogenics Auxiliary Controllers
Warm Magnets
Experiments
Access System
Beam Loss Monitors (Arc)
Collimation System
Radio Frequency System
Injection Systems
Vacuum System
Access System
Beam Interlock System
Control System
Essential Controllers
General Emergency Stop
Uninterruptible Supplies
Discharge Circuits
Beam Loss Monitors (Aperture)
Beam Position Monitor
Beam Lifetime Monitor
Fast Magnet Current Changes
Beam Television
Control Room
Software Interlock System
TimingSystemPost Mortem
Safe Machine Parameters
HTS temperature interlock
Access vs Powering
Outline
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
PowerInterlock
Controllers
BeamInterlockSystem
Beam Dumping System
Quench Protection System
Power Converters
Cryogenics Auxiliary Controllers
Warm Magnets
Experiments
Access System
Beam Loss Monitors (Arc)
Collimation System
Radio Frequency System
Injection Systems
Vacuum System
Access System
Beam Interlock System
Control System
Essential Controllers
General Emergency Stop
Uninterruptible Supplies
Discharge Circuits
Beam Loss Monitors (Aperture)
Beam Position Monitor
Beam Lifetime Monitor
Fast Magnet Current Changes
Beam Television
Control Room
Software Interlock System
TimingSystemPost Mortem
Safe Machine Parameters
HTS temperature interlock
Access vs Powering
PowerInterlock
Controllers
BeamInterlockSystem
Beam Dumping System
Quench Protection System
Power Converters
Cryogenics Auxiliary Controllers
Warm Magnets
Experiments
Access System
Beam Loss Monitors (Arc)
Collimation System
Radio Frequency System
Injection Systems
Vacuum System
Access System
Beam Interlock System
Control System
Essential Controllers
General Emergency Stop
Uninterruptible Supplies
Discharge Circuits
Beam Loss Monitors (Aperture)
Beam Position Monitor
Beam Lifetime Monitor
Fast Magnet Current Changes
Beam Television
Control Room
Software Interlock System
TimingSystemPost Mortem
Safe Machine Parameters
HTS temperature interlock
Access vs Powering
I.Romera - TE/MPEMPP Workshop March 2013
Sensitivity to electrical perturbations• Electrical network glitches are one of the main dump causes from magnet powering systems• 24 FMCM triggers last year due to glitches in the electrical network distribution• A large fraction do not result in equipment trips and only seen by FMCMs!• High sensitivity (<3x10e-4) on FMCMs is required to avoid effects on circulating beams• Being RD1 and RD34 most sensitive circuits which tripped in most of the cases• RD1.LR5 is about 40% more sensitive than RD1.LR1 despite similar input amplitude
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Sensitivity to electrical perturbationsProposed mitigations
2012
FMCM dump cause Systems affected by EL perturbationRD1 RD34 RQ5 RQ4 RBXWTV RMSD
0%
10%
20%
30%
40%
50%
60%
70% 63%56%
15%10%
13% 13%
Likelihood to trip
I.Romera - TE/MPEMPP Workshop March 2013
Mitigations put in place in 2012• FMCM review carried out to study possible mitigations to the sensitivity problem [1]• Thresholds relaxed by factor 3 on RBXWTV and RD34 still maintaining specification values[2]• Simulations on RD1 shown that there is no margin for threshold change• No significant availability improvement seen after mitigations as always in the shadow of RD1• Other mitigations were studied: Improving regulation characteristics of the PC, changing circuit
impedance by adding SC solenoid and replacing thyristor-based by switch mode PC…
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Sensitivity to electrical perturbationsMitigations in place in 2012Post L1 operation
Courtesy of T.Baer
Effect of 100mA current change on orbit in the arc
I.Romera - TE/MPEMPP Workshop March 2013
Post LS1 operation• EPC will soon initiate a collaboration with EPFL to better understand the effects of such glitches• After tests started in 2012, EPC is confident to provide a better rejection of elec. perturbations
by only changing the regulation characteristics of the RD1 and RD34 power converters• We expect that… ~40% of minor perturbation trips will be avoided in the future…
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Sensitivity to electrical perturbationsMitigations in place in 2012Post LS1 operation
23-1005-0924-0823-0804-0808-0730-0608-0603-0620-0507-0507-0512-0412-040.0
5.0
10.0
15.0
20.0
25.0
RD1.LR1RD1.LR5RD34.LR3RD34.LR7RQ5.LR3RQ4.LR3
EL perturbations where only FMCM tripped (2012)
Exce
ed fa
ctor
in F
MC
Ms
I.Romera - TE/MPEMPP Workshop March 2013
~7% beam loss in B2,almost nothing in B1
I_MEAS
V_MEAS
Trip of power converter
400um in B1H
200um in B1V
200um in B2H
100um in B2V
Late dump by CMS solenoid trip - 10.08.2012• CMS solenoid trip due to a cooling problem which had an effect on beams [3]• Beams dumped only when I_MEAS at 40% of nominal value (3 minutes after solenoid trip)• Orbit deviation up to 400um seen by interlock BPMs• Very slow and distributed losses not reaching BLM thresholds• Beams dumped on high voltage BLM interlock (via the SIS)
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converters60A orbit correctors
I.Romera - TE/MPEMPP Workshop March 2013
Late dump by LHCb trip - 19.08.2012• LHCb trip led to slight beam orbit deviations before beam dump [3]• Orbit changed by 20um during the last 200 turns• Magnet Safety System (MSS) takes 30ms to send request to Beam Interlock System
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converters60A orbit correctors
I.Romera - TE/MPEMPP Workshop March 2013
Foreseen changes to Magnet Safety System• MSS upgrade to MSS2: NI cRIO FPGA platform to replace obsolete fuse programmable FPGA• Interlock will be provided in case of a Fast Discharge (CMS trip…)• Optocoupler will replace the output safety relay and time to interlock reduced to 16ms
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converters60A orbit correctors
Power Converter Fault
Optocouplers
cRIO FPGA Processing
Optocouplers
CIBU
13 ms
1 ms
1 ms
< 1 ms
MSS2
Total delay ≈ 16ms
Power Converter Fault
Optocouplers
FPGA Processing
Safety relays
CIBU
13 ms
1 ms
15 ms
< 1 ms
MSS
Total delay ≈ 30msCourtesy of S.Ravat
I.Romera - TE/MPEMPP Workshop March 2013
Late dumps by LHC Power Converters• 2 events last year on inner triplet power converters • On 15.06.2012 beams dumped due to high beam losses in SR7 [4]• A broken diode inside RTQX2.L2 provoked voltage oscillations which put in conduction the
FWD in parallel with Q3 magnet • Large circulating currents between the 3 converters and RTQX1.L2 tripped by I_OUT_OVER
CURRENT only 300ms after first current excursions observed• Better sensitivity achieved by reducing I_ERR and I_POS limits in the FGC
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converters60A orbit correctors
RQX[8kA/8V]
RTQX2[6kA/8V]
RTQX1[±600A/±10V] FWDFWT
KEK MQXA6450A/91mH
Q2a Q2b Q3Q1
Fermilab MQXB11390A/19mH
Fermilab MQXB11390A/19mH
KEK MQXA6450A/91mH
++
+-
-
-Courant d’oscillation
RTQX1.L2
I_OUT OVER CURRENT
I_AI_MEAS
I_REFV_REF
I.Romera - TE/MPEMPP Workshop March 2013
Late dumps by LHC Power Converters• On 07.09.2012 beams were dumped on beam losses due to trip of RTQX2.R5• Interlock was not sent to the PIC due to radiation induced latch-up• PIC interlock only received ~2 seconds after the dump by VS_FAULT on RTQX1• During LS1 the 2s FGC watchdog will be removed as part of FGC2 CPLD upgrade!
Operation with FMCMs post LS1Late dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converters60A orbit correctors
I.Romera - TE/MPEMPP Workshop March 2013
Late dump by loss of 60 Powering Permit • No hardware signals between Power Converters and interlock system• Magnet protection ensured by Power Converters and interlocks just to avoid
unnecessary quenches of magnets and current leads and to help operations• On 25.10.2012 communication problem with PIC => BLMs triggered in SR7 [5]
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converter interlocks60A orbit corrector interlocks
Cryogenics
Powering Interlock
Controllers
PICPVSS LHC Timing
CRYO_OK_PP60A(PVSS -> PVSS)
POW_SUB_OFFCOMM ALIVE
PC_PERMIT_60A(CMW)
EPC gateway(s)
LHC Timing cable
Telegram groupPP60A
Cryogenics
Powering Interlock
Controllers
PICPVSS LHC Timing
CRYO_OK_PP60A(PVSS -> PVSS)
POW_SUB_OFFCOMM ALIVE
PC_PERMIT_60A(CMW)
EPC gateway(s)
LHC Timing cable
Telegram groupPP60A
~ 16:20:38 ~ 3min
~ 1sec
I.Romera - TE/MPEMPP Workshop March 2013
Late dump by loss of 60 Powering Permit • PC_PERMIT_60A would have never been lost with the correct PVSS logic due to a
communication problem affecting a single PIC…• PVSS logic will be corrected during LS1
Logic implemented…What we saw…
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Experimental magnetsLHC Power Converter interlocks60A orbit corrector interlocks
I.Romera - TE/MPEMPP Workshop March 2013
Renovation of interlocks in SPS• Last June a problem with ring-line interlocks in SPS caused 16 hours of downtime! [6]• Existing magnet protection system relies on 3 interlock systems (dating from 70’s)• Spares for ring-line and auxiliary lines but no spares available for main interlocks• Consolidation works done during 2012 and full renovation will be done during LS1 [7]• Electromagnetic solution to be replaced by standard WIC interlock system
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Renovation of SPS interlocksChanges in Access PoweringOperational improvements
I.Romera - TE/MPEMPP Workshop March 2013
Courtesy of L.Ponce – see Chamonix 2010
Changes in Access vs Powering• Software-based solution put in place after the incident in September 2008• A more dependable implementation will replace the existing one during LS1 [8]• New PIC-PLC to be installed by the LHC Access Control System (LACS) in the CCR• Communication between LACS and PIC-PLC via hardwired signals• PIC-PLC will publish access status to CMW and SIS will subscribe to the data
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Renovation of SPS interlocksChanges in Access PoweringOperational improvements
Courtesy of J.Wenninger
I.Romera - TE/MPEMPP Workshop March 2013
Operational improvements• Protection mechanisms to avoid quench propagation to neighboring magnets• During HWC campaigns such global protection is a bottleneck for testing!• Proposal is to make the Global protection mechanisms maskable via PVSS• Masking status clearly visible from PIC SCADA system
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Renovation of SPS interlocksChanges in Access PoweringOperational improvements
I.Romera - TE/MPEMPP Workshop March 2013
Operational improvements• To allow for some flexibility, not all circuit trips will directly request beam dumps:
• RCD, RCO, ROD, ROF, RQS, RSS… do NOT directly dump the beam• Review of critical circuits for operation in view of new ATS optics… • In 2012, quadrupole correctors RQSX3 were included in maskable configuration
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Renovation of SPS interlocksChanges in Access PoweringOperational improvements
I.Romera - TE/MPEMPP Workshop March 2013
And some additional changes…
• PICs:• 25% of the controllers to be relocated within R2E project (UL14/16 &UL557) [9]• Re-commissioning of UPS links following UPS renovation (see V.Chareyre talk)• HTS room temperature interlocks in RR53 routed through the PIC
• FMCM:• FMCM in UJ56 relocated to USC55 within R2E project [10]• Controls renovation in SPS transfer lines (removal of obsolete ROCS-MUGEF)
WIC:• Ti2/Ti8 - Installation of 800 interlock boxes
to allow remote testing of magnet interlocks
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
Renovation of SPS interlocksChanges in Access PoweringOperational improvements
I.Romera - TE/MPEMPP Workshop March 2013
Summary• High sensitivity of FMCMs against electrical network perturbations:
• Thresholds relaxed on RBXWTV and RD34 during 2012 • Voltage regulation improvements on the power converters during LS1
• Redundant protection to the BLMs will be restored after LS1:• PC_PERMIT_60A logic will be corrected in PVSS• Upgrade on Magnet Safety System• FGC2 CPLD upgrade to remove the 2s watchdog in case of FGC crash
• No SEUs observed on PICs last year, thanks to temporary relocation in 2011
• SPS magnet interlocks were consolidated and will be fully renovated during LS1
• More dependable approach for the Access Powering interlocks
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS
I.Romera - TE/MPEMPP Workshop March 2013
Thank you for your attention!• References:
• [1] FMCM review, TE-Technical meeting• [2] ECR-Change of Protection Threshold for FMCMs installed on the ALICE
compensators and the twin aperture warm separation circuits, EDMS 1213337• [3] 67 - MPP meeting, Minutes • [4] 63 - MPP meeting, Minutes• [5] 75 - MPP meeting , Minutes• [6] Problem with SPS interlock crate, MPE-COMS-167• [7] TE-MPE-Technical meeting, Slides• [8] ECR-Change of the Interlocking of Powering and Access systems, EDMS 1246780 • [9] ECR-Activités R2E au point 1 durant LS1, EDMS 1263553• [10] ECR-Activités R2E au point 5 durant LS1, EDMS 1263595
FMCM operationLate dump detection by magnet powering systemsForeseen changes to powering interlocks of the LHC-SPS