Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

1

TE-MPE Stand-By Service 2010 – 2011 lessons learnt

Gert Jan Coelingh on behalf of TE-MPE Stand-By team,

Team members

Description of the TE-MPE Stand-By Service

Statistics 2010 + 2011

SEU

DQHDS main switch

“IN THE SHADOW”

Personal touch

Extrapolate 2011 – Improvements?

Conclusion & Outlook

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

2

Team Members

– Giorgio D'Angelo EI ELQA & Heating Syst. C.L

– Richard Mompo EI ELQA & Heating Syst. C.L

– Grzegorz Seweryn EI ELQA & QPS Heating Syst. C.L

– Steven Pemberton EI ELQA & Heating Syst. C.L

– Kevin Priestnall CP QPS – Retirement 2011

– Vincent Froidbise CP QPS

– Joaquim Mourao CP QPS

– Gert-Jan Coelingh CP 600A EE, 13 kA & QPS

– Help of Mathieu Favre, Noel Fournier and Bozhidar Panev (CP)

Reiner is responsible of the service, and Gert-Jan is the linkman

Stand-By members:

In 2011, 8 members, with specific knowledge:

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

3

Team Members: Experts

Specialists: support for Stand-By members

Very limited specialists (in number ):

– Knud Dahlerup-Petersen (EE & QPS)

– Reiner Denz (QPS)

– Zinur Charifoulline (nQPS informatics).

– Jens Steckert (nQPS).

• CP section: Other specialists on Stand-By

– Current leads specialist all specialists on Stand-By

Now under EI responsibility;.

– DFB specialists: are part of Stand-By members, Mateusz Bednarek since November 2010!!

– MP3 members Stand-By

• All Experts are on BEST-EFFORT

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

4

MPE Stand-By mandate:

1. Mission:

– To put back in service, as rapid as possible, a faulty installation within the framework (QPS, Energy Extraction Systems and electrical components of the DFBs) during LHC run.

2. Actions:

– Intervene after a call from the Control Centre CERN (CCC),

– Diagnostics and if possible solve the problem remotely

– Organize the intervention with the operator/EIC CCC (RP)

– If needed ask the “Expert” (if reachable) for support

– Archive the intervention in the Logbook (QPS Logbook).

3. Responsibility:

– He must assure his own security as well as those he asked helping him and assure the correct way of intervening in the tunnel

– The technician is responsible for the intervention. If he cannot find a solution he must inform the group hierarchy.

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

5

MPE Expert mandate (1/2)

The Expert (based on best effort) :

1. Mission:

– Support the first line intervention in case of complex problems

2. Actions:

– If judged intervene to help the Stand-By technician

– Ask another Stand-By team member to join for intervention

3. Indirectly related actions:

Decrease LHC machine down-time where possible

– Implement solutions for problems

Firmware upgrades

Preventive maintenance

DQHDS main switch exchange campaign

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

6

MPE Expert mandate (2/2)

3. Indirectly related actions (cont.):

– Not to be underestimated in hours and workload

– Example:

Arkadiusz Gorzawski developed an interface

logging database – 600A EE signals

- permits preventive maintenance on breakers with too many closing failures

- Currently extended with intervention history per system

Direct result = no breaker exchange so far this year

HUGE EFFORT not visible in Stand-By statistics

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

7

MPE Stand-By organization:

Documentation:

– Each equipment specialist prepared a documentation on how to solve a typical problem.

– An updated copy is always available in two Stand-By cars

Regular meeting and training:

– Weekly meeting on Monday morning to transmit the “Piquet role” and share experience over the past week.

– Training are organized in order to be as multidisciplinary as possible.

– Two Stand-By cars available, equipped with tooling, docs and spare parts.

– Storage place in Build.30, 281 and 936 (in front of CCC) Yearly planning:

On our repository, we have a yearly planning with foreseen vacations, official holidays, night and day “Stand-By” turn-over.

– Day calls: transparent to CCC (same number 16.8801), assures intervention during normal hours. This allow night piquet to recover.

– Night calls: outside normal hours and w-e, official holidays included.

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

8

Intervention Flowchart

Emergency call from CCC to Stand-By duty:Stand-By: 16.8801

Emergency call from CCC to Stand-By duty:Stand-By: 16.8801

End of intervention

End of intervention

N

Stand-By person to analyze

max. delay (1hour)

Stand-By person to analyze

max. delay (1hour)

Information exchange via telephone

Stand-By / CCC operatorRemote Analysis

Information exchange via telephone

Stand-By / CCC operatorRemote Analysis

Need more than onePerson?

Assistance via:Stand-By support

Operator CCC*

Assistance via:Stand-By support

Operator CCC*

N

Y

Y

Repair and restart the equipment or,

feedback situation with CCC.

Repair and restart the equipment or,

feedback situation with CCC.

Call equipment Specialist or Responsible TE/MPE

Call equipment Specialist or Responsible TE/MPE

Problem Solved remotely?Stand-By / CCC

Difficult problem or

Time needed > 3 hours ?

N

Repair and

restart the equipment

Repair and

restart the equipment

Y

*(en dehors des heures ouvrables uniquement )

Fill in QPS Logbook

Fill in QPS Logbook

End of intervention

Fill in QPS Logbook

Fill in QPS Logbook

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

9

TE-MPE Stand-By service for the LHC

Gert Jan Coelingh on behalf of TE-MPE Stand-By team,with inputs from R. Denz

Team members


Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

10

66

151.5

41

50

33

18

0

20

40

60

80

100

120

140

160

180

200

220

240

Total number of interventions Total duration [hours]

Interventions from MPE Piquet in 2010 (39 Weeks)

Outside normal working hours Inside normal working hours Remote (day & night)

Some statistics from 2010/2011: (1/..)

Total number of interventions (Remote / Local) and Total duration

140 Interventions

219 hours

87 Interventions87 hours

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

11


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

12


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

13


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

14

Some statistics from 2010: (5/..)

26

19

12

8

20

65

43

37

6076 10438 1624 2532 2532 202 32 32 1298

Quench heater power supplies

Quench detection systems

DAQ caused by radiation (SEU)

DAQ other causes than radiation

DAQ all faults combined

EE600 EE13 kA DFB Instrumentation

Current Lead Heating system

Others

Faults distribution per equipment type

0

5

10

15

20

25

30

35

40

1819

17

22

39

1 12

1

6

Faults up to June 20, 2011

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

15



Team members



SEU

DQHDS

“IN THE SHADOW”

Personal touch

Improvements


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

16

SEU – increasing number

17 treated by Stand-by team so far this year

– Symptoms: QPS_OK lost (fill can be continued until beam dump)

• continuous sending PM files or loss of communication

– Action:

• locally power cycle the crate – (and depending on time of intervention)

• Update locally the firmware or– change board for one with new firmware)

– Specialists actions: upgrade of firmware

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

17



Team members



SEU

DQHDS main switch

“IN THE SHADOW”

Personal touch

Improvements


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

18

DQHDS main switch

18 cases treated by Stand-by team so far

– 26 treated in 2010 – we are NOT better in number but much better in time of intervention; almost always IN THE SHADOW causing no downtime to LHC operation

1650 (26%) out of 6200 have been replaced

– All DQHDS of all MQs done and regular campaigns on-going

• Software interlock adapted to MB. One out of four DQHDS of a MB can be down. No immediate intervention needed.

• Why not better in numbers? …

» plastic is getting too hard?? We should hurry up

No faults in the new switches

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

19



Team members



SEU

DQHDS

“IN THE SHADOW”

Personal touch

Improvements


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

20

« IN THE SHADOW »

30 out of 87 interventions “IN THE SHADOW”

• less downtime for LHC operations

Mainly because

– of upgraded Interlocking software in case of a MB DQHDS failure.

– SEU can finish the fill before power cycling the crate

– Preventive maintenance

– Coordination between Equipment Responsible and Machine Coordination in case of longer downtime

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

21



Team members



SEU

DQHDS

“IN THE SHADOW”

Personal touch

Improvements


TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

22

Personal Touch (updated until June 20)

3 x

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

23



Team members



SEU

DQHDS

“IN THE SHADOW”

Personal touch



TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

24

Extrapolate 2011 – improvement?

+170%

+178%

0%

+17%

- 21%

- 41%

189 interventions = +35% 189 hours = -14%

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

25



Team members



SEU

DQHDS main switch

“IN THE SHADOW”

Personal touch


Outlook

Conclusion

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

26

Outlook:

Improvement ongoing:

– TUNNEL INTERVENTIONS

• Upgrades on iQPS firmware on SEU

• Switches DQHDS replacement – 1600 done + 500 4-9 July

– Software tools on QPS

• Remote restart – done

• Upgrade Firmware of new Busbar detector 4-9 July

• Interlocks regarding DQHDS – interventions “in-the-shadow”

Potential bottlenecks:

– More radiation to equipment SUE

– Resources are limited, both experts and technicians

– MP3 Stand-By canceled in 201x? Who takes over?

– Back-up for Stand-By team based on best effort

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

27

Conclusion :

Conclusion so far as in 2010

– Very efficient: < 2 hours/intervention on site, including access time !!

– High reliability of the systems

– No accident !

Stand-By service interventions seems to be decrease in numbers of hours

BUT..

A lot of work behind the scenes to get there

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

28

MPE STAND-BY on DFS

All information about the Stand-By service can be found on the DFS folder:

Dfs:\\cern.ch\Departments\TE\Services\MPE_on-call-service

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

29

QUESTIONS ?

THANK YOU !!!

LPCT

Local Power Cycle Tool

The absence of safety shoes and helmet is not representative for the MPE Stand-By team

TE

-MP

E,

Gro

up M

eetin

g -

G.J

. C

oelin

gh -

23

Jun

e 20

11

30

Buffer zone for pieces from LHC machine

Les pièces sont contrôlées par un

technicien DGS/RP

Sécurité Radiologique Opérationnelle: Lignes directrices générales• Tout matériel quittant les zones de rayonnement CERN doit être considéré comme potentiellement radioactif,• Tout matériel doit être contrôlé par DGS/RP,• Seul DGS/RP, est autorisé à déclarer du matériel comme non radioactif,• En cas de doute, contactez un technicien DGS/RP

Repair work shop

TE/MPEBat. 30

Bat. 281Bat. 287External

Zone de Transit

TE/MPE for non radio

active pieces

Bat. xx

Labo de Réparation dédié aux

équipements faiblement radioactifs

TE-EPC

Who?: Equipment responsible

1) Routing of faulty pieces from a TE/MPE installation towards a repair workshop

LHC Machine (Zone Opérationnelle

contrôlée)

LHC Machine (Zone Opérationnelle

contrôlée)

UA(Zone Machine

Supervisée)

UA(Zone Machine

Supervisée)

When: Weekly, when lower limit is reached

-

RR/UJ(Zone Opérationnelle

contrôlée)

RR/UJ(Zone Opérationnelle

contrôlée)When?: Only in cases of an intervention

-

Quand?: Uniquement lors d’une intervention

Quand?: Uniquement lors d’une interventionWhen?: Only in cases of an intervention

Radioactif ?

Yes

Broken Modules

Rack

Who?: MPE on call Technician

Who?: MPE on call Technician‘Picomur’

‘Picomur’

‘Picomur’

When?: Only in cases of an intervention

Who?: MPE on call Technician

Date post:	16-Jan-2016
Category:	Documents
Upload:	kieve
View:	39 times
Download:	0 times

Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU

Documents