EPAC 2008, Genoa, Italy, June 23-27, 2008
Post-mortem diagnostics for the Taiwan Light Source
K.H. Hu, Jenny Chen, C.Y. Wu, Demi Lee, C.H. Kuo, P.C. Chiu, S.Y. Hsu, C.J. Wang, K.T. Hsu, M.S. Yeh, Ch. Wang
Presented by K.T. Hsu
NSRRC, Hsinchu 30076, Taiwan
2EPAC 2008, Genoa, Italy, June 23-27, 2008
Outline
NSRRC accelerator facility
Taiwan Light Source (TLS) and Taiwan Photon Source (TPS)
Operation status of the TLS (poster WEPC058)
Tools to capture data for post-mortem analysis
Some trip scenarios analysis
Summary
3EPAC 2008, Genoa, Italy, June 23-27, 2008 Hsinchu Science Park
NSRRC accelerator facility
Hsinchu
Taiwan
(Courtesy of Google Earth)
4EPAC 2008, Genoa, Italy, June 23-27, 2008
NSRRC accelerator facility – cont.
11
2233
44
55
6677
88
99
國家同步輻射研究中心
中心設施 增能環館 機電館
鄰近單位 99
11
1010
大門 22 行政大樓 33 研光大樓 44 儀光大樓 55 儲存環館 77 88 招待所國家高速網路與計算中心 交通大學
66
Taiwan Photon Source
Taiwan Light Source
Chiao-Tung UniversityTsing-Hua University
Hsinchu Science Park
5EPAC 2008, Genoa, Italy, June 23-27, 2008
TPS officeActivity center & guest house &
Utility building no.3
Taiwan Photon Source project
Taiwan Photon Source (TPS)
3 GeV, C=518.4 mCommissioning : 2013+
Taiwan Light Source(TLS)
1.5 GeV, C=120 mDedication in 1993
Aerial view
North
TPS related poster:
TUPC054TUPC120TUPC127TUPD007TUPP025WEPC004WEPC005WEPC006WEPC017WEPC082WEPC104WEPC113WEPC114WEPC142THPC005THPC041THPC120THPP118THPP143
6EPAC 2008, Genoa, Italy, June 23-27, 2008
TLS beamline status
Soft X-ray Hard X-ray
20
19
1817
1615 1413
12
11
1009
08
07
0605
04030201
2122
23
24
SW
U5
EPU5.6
W20
U9
SWLS
01A 01B X-Ray Microscopy - SWLS01C
White X-ray - SWLS
EXAFS, Powder Diffraction - SWLS
03A Gas Phase (HF-CGM) - BM
04A 04B
Beam Diagnosis - BM
05A Spin-Polarized PES, PEEM - EPU 05B
Soft X-ray Scattering - EPU
07AX-ray Scattering - IASW
08B
XPS, UPS (L-SGM) - BM 08A
Beam Diagnosis - BM
09ASPEM, XPS - U5
11AMCD, XAS (Dragon) - BM11B
13B13C
13A SW6 - X-ray ScatteringSW6 - Protein CrystallographySW6 - Protein Crystallography
14A BM - IR Microscopy
15A
16A
IASW
17B W20 - X-ray Scattering
17C W20 - EXAFS
18A BM - Beam Diagnosis18B BM - LIGA
19A BM - X-ray Lithography19B BM - Photo Stimulated Desorption
20A20B
BM - (H-SGM) XAS, XPSBM - X-ray Instrumentation
21A21B
U9 - (White Light) Chemical Dynamics
23A IASW - SAXS
24A BM - (WR-SGM) XPS, UPS
Gas Phase (Seya) - BM
Beam Diagnosis - BM
U9 - (CGM) Angle-Resolved UPS, Gas Phase
Under construction
: 9: 10
NSRRC Beamlines
IR, VUV : 5
or planning
IASW
IAS
W
IASW
SP12U Inelastic X-ray Scattering
SP12B X-ray Absorption, Diffraction
12NSRRC Beamlines
Hard X-ray : 2
at SPring-8, JapanProtein Crystallography,
( )
( )
( )
BM - Tender X-ray Absorption, Diffraction
( )
( )
at Hsinchu, Taiwan
17A W20 - X-ray Diffraction
: Participating Research Group ( )
SWLS: Superconducting Wavelength Shifter
EPU5.6: Elliptically Polarized UndulatorIASW: In Archomatic Superconducting Wiggler
U5: Undulator (5 cm, 76 Periods)
W20: Wiggler (1.8 T, 27 Poles)U9: Undulator (9 cm, 48 Periods)
SW: Superconducting Wiggler (3.2 T, 32 Poles)
( )
Beam Diagnosis : 6
10B Beam Diagnosis - BM
( )( )
Insertion Devices
XAS, XPS ( M-AGM) - BM 08B
SP12D HE Photoemission
June 2006
ID Status
IDs:W20U5U9EPU5.6EPU4.6 (2009)
Superconducting IDs:SWLSSW6IASW6-R6IASW6-R2 (2009)IASW6-R4 (2009)
7EPAC 2008, Genoa, Italy, June 23-27, 2008
301 mA Top-Up operation ~ 0.05% photon flux constancy were achieved routinely> 98 % availability achieved
Keeping high stability and high availability is our commitment to the users
How to achieve this?
TLS operation status
@ 2008
8EPAC 2008, Genoa, Italy, June 23-27, 2008
Bad experience – one of an example
Problem of master oscillator related problem
0
5
10
15
20
25
30
35
2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8
Num
ber
of B
eam
Los
s
2000 2001 2002 2003
What happened ?
Shutdown
9EPAC 2008, Genoa, Italy, June 23-27, 2008
Trilogy of the abnormal high trip rate in (February - December, 2000)
LLRFRF Low Level Electronics
RFTransmitter
Power distribution panel,bad contact
(Voltage drop to about 70 ~80 V in a few power line cycle)
Control board malfunction ,(always in by-pass mode,
no indication)
Output coaxial switch was open when line
voltage dropped to less than 90 V
RF Loss(< 0.1 sec)
Beam Trip
Rule of Thumb : Don’t Trust “Common Sense”
Bad experience – one of an example - cont.
UPSCircuit Breaker Master Oscillator
115 V60 Hz
If some trip diagnostic tools existed, it might be solved soon!
10EPAC 2008, Genoa, Italy, June 23-27, 2008
久病成良醫 (三折肱成良醫 )Failure is the Mother of Success (失敗是成功之母)Experience is the Mother of Wisdom (經驗是智慧之母)Fall into a moat and you will gain wisdom from the experience (吃一塹長一智)Fall Down Seven Times, Get Up Eight (七転八起)Success comes from experience; but experience comes from failure (Mark Twain)
Proverb
Strategies to improve or to keep high availability
Preventative maintenanceIdentify weaknesses of the system and remove themOperation automationAnalyze each trip event to identify where is the problem
=> Fault diagnosticCapture data before and after specific event happened
=> Post-mortem analysisMany measures were done after identify the real reasons for further improvement!
11EPAC 2008, Genoa, Italy, June 23-27, 2008
High time resolution
100 msec 10 msec 400 nsec
Oscilloscope, Fast digitizer
Beam intensityBeam positionBeam instabilityBeam trip triggerKicker waveform
…….
Control system archiverBeam currentBeam positionBeam loss distributionSuperconducting insertion device related signals….…
Vision XP,High density cPCIdigitizer
Beam currentBeam trip triggerSRF interlockSuperconducting insertion device related signals…….
BPM electronicsTurn-by-turn beam positionbeam phasebeam intensity …….
Low time resolution~ nsec
Various tools for beam trip data acquisition
All tools are trigged by the sum signal: beam trip + superconducting IDs quench + SRF trip + predefined trigger conditions
12EPAC 2008, Genoa, Italy, June 23-27, 2008
AND
100 μsecMonostable
50 μsecMonostable
50 μsecMonostable
Beam trip trigger
Ref IcBeam intensity50 mA/V
How to detect beam loss?
Beam
DCCT(Beam current) Ref Ia
Ref Ib
Beam tripNon beam trip0 mA
300 mA
13EPAC 2008, Genoa, Italy, June 23-27, 2008
Superconducting insertion device quench detection
MainPower Supply
+/- 350 A
TrimmingPower Supply
+/- 20 A
TrimmingPower Supply
+/- 20 A
Comparator Latch
Comparator Latch
WindowDetector Latch
1
4
7
Superconductingcoils
HTC lead
HTC lead
Diff. Amp.
14EPAC 2008, Genoa, Italy, June 23-27, 2008
Recorder, oscilloscope, and b-by-b feedback processor, t-by-t BPM
Recorder
Data memory in bunch-by-bunch feedback processor Post-mortem buffer of BPM electronics
Oscilloscope
15EPAC 2008, Genoa, Italy, June 23-27, 2008
Control Ethernet
cPCI based high density digitizer
Beam trip trigger
SRF related signals
SWLS related signals
SW6 related signals
IASW-R6 related signals
IASW-R2 related signals
IASW-R4 related signals
PC/LinuxFile Server
1. Record data before and after the trigger event for several seconds
2. The server saves all data for further analysis
3. Automatically identify faulty signals
96 channels/board
Sampling rate: up to 500 kS/secRecord length: msec to 10 sec (depending on sampling rate)
Embedded EPICS IOC
Power Lines
16EPAC 2008, Genoa, Italy, June 23-27, 2008
Maximum Sampling Rate:
100 MS/sec500 MS/sec
…
8 Ch14 Bit
100 Ms/secADC
Trigger
Kicker waveform
Gating (optional)
8 Ch14 Bit
100 Ms/secADC
Trigger
DataGating
Up to 8 boards (Maximum 64 Channels/Link)
PC/LinuxConfiguration,Data archive,
Analysis,User interface
Control Ethernet
VME based fast digitizer
Kicker trigger
Fast beam current detector
17EPAC 2008, Genoa, Italy, June 23-27, 2008
Kicker trigger
VME based fast digitizer – cont.
Beam trip
Triggeroutput
Trigger output
Beamcurrent
Kickerwaveform
Beam loss due to kicker misfired, …
Beam loss due to kicker misfired, SID trip, SRF trip, Transmitter trip, …
Kickertrigger
Live display for every top-up injection cycle
Store data if beam loss occurred
Injection time Non injection time
18EPAC 2008, Genoa, Italy, June 23-27, 2008
Klystron Anode Current
Klystron Arcing
Linac klystron stand arcing diagnostics
Simple pinhole CCD camera is also very useful!
This symptom is just the precursor of isolation breakdown!
10 μsec
100 msec
19EPAC 2008, Genoa, Italy, June 23-27, 2008
Diagnostic tools for the SRF system
- Try to understand the reasons of every RF/SRF fault
• What we experience a lot…– SRF “quench” due to fast change of beam behaviors…
• Fast beam dump or partial beam loss due to various reasons;• Fast beam dump due to kicker misfired;• Orbit change;• …
– LLRF’s feedback loop becomes unstable due to heavy beam loading…– False alarm by window arcing…– We experience with a “new” kind of trip event every few months and
usually solve it by reducing various feedback loop gains.– Modified many LLRF electronics to avoid false alarm.
• What we never or seldom experience…– Multipacting in the cold surface waveguide (Pf < 70 kW);– Real quench or field emission (rf gap voltage < 1.6 MV);– Vacuum burst due to hydrogen desorption (vacuum better than 0.7 nTorr);
20EPAC 2008, Genoa, Italy, June 23-27, 2008
Beam currentTuning angle
Pr
Pf
Pt phase
Beam phase
Interlock faultKlystron phase
SRF trip due to mismatch of beam loaded cavity
21EPAC 2008, Genoa, Italy, June 23-27, 2008
SRF trip due to mismatch of beam loaded cavity – kicker misfired
Spontaneous kicker fired(bad grounding contact)
22EPAC 2008, Genoa, Italy, June 23-27, 2008
SRF Trip due to mismatch of beam loaded cavity – SW6 fault
23EPAC 2008, Genoa, Italy, June 23-27, 2008
Feedback loop oscillations due to heavy beam loading
Amp loop
RF Gap voltage
Full scale = +/- 3.6o
Tuner loop
287 mA
24EPAC 2008, Genoa, Italy, June 23-27, 2008
Kicker #2 no fire
Kicker #2 miss fire during Top-up Injection.
Beam tripBeam Current
Kicker Trigger
SRF Interlock Sum
Tuning Angle
Phase Angle
Beam Phase Detector Output
The beam trip caused by kicker misfired
25EPAC 2008, Genoa, Italy, June 23-27, 2008
SWLS main coil PS failed
T1 (8.5oK)
T7 (6.5oK)T2 (6.5oK)
030 10/19/2006 17:52
Trip caused by the SWLS main power supply failed
10 Hz archived data, SWLS MPS drop lead beam current drop later slightly!
26EPAC 2008, Genoa, Italy, June 23-27, 2008
1 2 3 4 5 6 7 8 9 10024
Vol
tSWLS Trip Event on 19-Oct-2006 17:52:41
1 2 3 4 5 6 7 8 9 10024
Vol
t
1 2 3 4 5 6 7 8 9 1005
10
Vol
t
1 2 3 4 5 6 7 8 9 10-0.2
00.2
Vol
t
1 2 3 4 5 6 7 8 9 100
102030
Vol
t
1 2 3 4 5 6 7 8 9 100
102030
Vol
t
Sec
Beam Trip
Beam Current
SWLS MPSI
SWLS Vdiff
SWLS QD Out
SWLS PLC Int
302 mA0 mA
Trip caused by the SWLS main power supply failed – cont.
SWLS MPS tripped firstly (Vision XP data)。Reason: SWLS main power supply failed – control firmware problem!
Time (sec)
SWLS Trip Event on 19-Oct-2006 17:52;41
27EPAC 2008, Genoa, Italy, June 23-27, 2008
SW6 and IASW-R6 quench - I
SW6 Quench@ 14:03:51
IASW-R6 Quench@ 14:04:34
Partial Beam Loss
SID Trip Event on 25-Sep-2007
28EPAC 2008, Genoa, Italy, June 23-27, 2008
SW6 and IASW-R6 quench – II (SW6 quench)
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9
300
302
SID Trip Event on 25-Sep-2007 14:03:51m
A
Beam Current
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 LS218 Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 Coil Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 Pressure Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
100200300
A
SW6 MPSI RD
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9
-2-10
Vol
t
SW6 Vdiff
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
SW6 QD OUT
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Sec
Vol
t
SW6 PLC INT
Partial beam lossSID Trip Event on 25-Sep-2007 14:03:52
Time (sec)
29EPAC 2008, Genoa, Italy, June 23-27, 2008
SW6 and IASW-R6 quench – III (SW6 quench)
4.4 4.5 4.6 4.7 4.8 4.92.5
3
3.5SID Trip Event on 25-Sep-2007 14:03:51
Vol
t
IASW-R6 He Pressure
4.4 4.5 4.6 4.7 4.8 4.9298
300
302
304
mA
Beam Current
4.4 4.5 4.6 4.7 4.8 4.9
0
2
4
Vol
t
Inj Sync
4.4 4.5 4.6 4.7 4.8 4.90
2
4
6
Vol
t
IASW-R6 N2 Pressure
4.4 4.5 4.6 4.7 4.8 4.90
100
200
300
A
SW6 MPSI RD
4.4 4.5 4.6 4.7 4.8 4.9
-2
-1
0
1
Vol
t
SW6 Vdiff
4.4 4.5 4.6 4.7 4.8 4.90
10
20
Vol
t
SW6 QD OUT
4.4 4.5 4.6 4.7 4.8 4.90
10
20
Sec
Vol
t
SW6 PLC INT
Interlock active
Coil quench
Non injection time
Partial beam loss
SID Trip Event on 25-Sep-2007 14:03:52
30EPAC 2008, Genoa, Italy, June 23-27, 2008
SW6 and IASW-R6 quench – IV (IASW-R6 quench)
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9
260
280
SID Trip Event on 25-Sep-2007 14:04:34m
A
Beam Current
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 LS218 Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 Coil Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW-R6 Pressure Trip
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
100200
A
IASW MPSI RD
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.9-505
Vol
t
IASW Vdiff
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Vol
t
IASW QD OUT
4.4 4.45 4.5 4.55 4.6 4.65 4.7 4.75 4.8 4.85 4.90
1020
Sec
Vol
t
IASW PLC INT
SID Trip Event on 25-Sep-2007 14:04:34
Time (sec)
31EPAC 2008, Genoa, Italy, June 23-27, 2008
SW6 and IASW-R6 quench – V (IASW-R6 quench)
4.4 4.5 4.6 4.7 4.8 4.9
3.3
3.4
3.5
3.6SID Trip Event on 25-Sep-2007 14:04:34
Vol
t
IASW-R6 He Pressure
4.4 4.5 4.6 4.7 4.8 4.9
260
270
280
290
mA
Beam Current
4.4 4.5 4.6 4.7 4.8 4.90
2
4
Vol
t
Inj Sync
4.4 4.5 4.6 4.7 4.8 4.9
4.974.984.99
55.01
Vol
t
IASW-R6 N2 Pressure
4.4 4.5 4.6 4.7 4.8 4.90
100
200
A
IASW MPSI RD
4.4 4.5 4.6 4.7 4.8 4.9-4-20246
Vol
t
IASW Vdiff
4.4 4.5 4.6 4.7 4.8 4.90
10
20
Vol
t
IASW QD OUT
4.4 4.5 4.6 4.7 4.8 4.90
10
20
Sec
Vol
t
IASW PLC INT
Partial beam loss
Coil quench
Trip MPS
Interlock active
He Pressure increase
SID Trip Event on 25-Sep-2007 14:04:34
Time (sec)
Injection
32EPAC 2008, Genoa, Italy, June 23-27, 2008
25-Sep-2007 14:03:51 SW6 quench. * SW6 quench happened.* MPS output dropped to 0 (before PLC interlock tripped MPS).
25-Sep-2007 14:04:34 IASW-R6 quench.* Partial beam loss happened during injection instance.* IASW-R6 quench.* PLC interlock tripped MPS.
Unknown reason caused SW6 quench, then partial beam loss at every injection instance, then IASW-R6 quench.
Many improvements were done: coil winding (poster WEPC114), liquid He filling procedure, interlock logic modification…
SW6 and IASW-R6 quench – VI
33EPAC 2008, Genoa, Italy, June 23-27, 2008
Beam loss induced IASW-R6 trip (2008/04/15 06:57)
IASW-R6
Partial beam loss
RF system trip
Coil quench
Summary
SWLS
SW6
Reason: one vertical corrector PS failed
34EPAC 2008, Genoa, Italy, June 23-27, 2008
Kicker diagnostic
Kicker diagnostic (VME digitizer working in segmented capture mode)
Beam Loss
Normal top-up injectionMaintenance test
35EPAC 2008, Genoa, Italy, June 23-27, 2008
BPM post-mortem data for large kicker jitters
Libera Brilliance post-mortem data (turn-by-turn)Diagnostics node (10 KHz rate for ~ 10 sec) is in implementation
K1 trigger -150ns
SRF Interlock Sum Active
Beam Loss
R1BPM3 Spectrogram
Short-time Fourier Analysis
36EPAC 2008, Genoa, Italy, June 23-27, 2008
BPM post-mortem data for IASW-R6 trip
Vertical tune jumps due to IASW-R6 trip
37EPAC 2008, Genoa, Italy, June 23-27, 2008
What we have achieved?
Several different tools are available.Some occasions have helped to identify some peculiar events.However no all events can explanation.
=> further efforts are essential !Simple automatic signal analysis are supported.
Our current efforts
Better integration of various diagnostic tools with the control system.Archive the events.Develop automatic analysis tools to search various signals.Develop better user interface.Automatic report generation.
38EPAC 2008, Genoa, Italy, June 23-27, 2008
Summary
Taiwan Light Source (TLS) is current operated at 301 mAtop-up operation. Top-up operation at 360 mA was commissioning.=> Poster WEPC043 (stable beam up to 400 mA has been tested)
Photon flux constancy can be kept around ~ 0.05%.
Availability is achieved 97 ~ 99% routinely.
To provide user better service, reduce trip rate is an important routine job.
Improving reliability, availability and serviceability is still on going.
All of the experiences will help the reliability and availability design for the newly proposed Taiwan Photon Source.
=> Reliability design for various subsystem.
=> Post-mortem buffer are expected embedded in BPM electronics, superconducting insertion devices’ controller, LLRF system, power supply controller, FOFB diagnostic node, … etc.