Qweak Target Training Requirements

• For all new Qweak TO’s:

– Oral Training course

– Practical either:

• 4h in CR looking over a TO’s shoulder, OR

• 8h as SL/Worker

• After > 6 months as TO:

– Online training course

• With a test

– Practical either:

• 4h in CR looking over a TO’s shoulder, OR

• 8h as SL/Worker

• After > 3 mos, but < 6 months as a TO:

– Online training course

• With a test

• Caveats:

– 1st week of run, and 1st

week after 6 month down:

• Rules relaxed

• Man shifts with experienced TOs only, no novices

– GRS sign off on these “brown belts”

Essential Responsibilities

5/24/2010 Qweak G. Smith: 2

• At start of your shift:

– Verify power settings, configuration, program

– Verify alarm handler & logger are functional

– Make sure alh visible in all workspaces

– Complete the checklist

• Log alarms, configuration changes

• Service alarms from lowest level in tree

• Contact MCC before & after tgt motion

• Know the essential Qweak “How-tos” at

www.jlab.org/Hall-C/ in the Cryogenic Target link

5/24/2010 Qweak G. Smith: 3

Your Job (Do’s)

• Keep tgt safe & within specs – ~55 liters of LH2!!!

• Monitor! Watch! Listen! Think!

• Service & log alarms

• Change targets (thru MCC)

• Respond to ESR changes with JT– Keep ~150 W reserve heater power for PID

• Check lumi widths, halo, beam position, vacuum, cameras, etc.

• Log target changes, alarms, etc.

• Fill out the checklist.

5/24/2010 Qweak G. Smith: 4

Don’ts• Cooldown• Warmup

– unless asked to by an expert

• Manipulate gas panel• Change alarm setpoints

– Unless asked to do so by a tgt expert

• Leave tgt console unattended• Fail to acknowledge alarms• Let system go sub-atmospheric or more

than 3K below setpoint• Move tgt while beam is on• Panic

5/24/2010 Qweak G. Smith: 5

Experts

Name Work Pager Cell Home

Greg Smith 5405 584-5405 871-4371 565-9883

Silviu Covrig 6410 584-5036 (843) 697-1753 394-1187

Dave Meekins 5434 327-5491 874-4750

Chris Keith 5878 327-5527

Josh Pierce 5036 327-5478

Jim Dunne 6928 (662) 325-2929

If you need help, call these people in the order listed.

(PAGER FIRST)

Please don’t call home numbers except in emergencies!

5/24/2010 Qweak G. Smith: 6

Basic Components• LH2 cell & re-circulation loops• Re-circulation pump (30 Hz)

– Bypass valve to WR

• 2.5 kW (max) heater (on PID) to replace Pbeam

• 3 kW (Helium) CF Hybrid heat exchanger:

– JT controller on He Supplies. Change slowly.

• T&P instrumentation. • Control System• Gas Panel, Ballast tank, Electronics

Source T (supply) T (return) P (supply) P (return)

ESR 15 K 20 K 12 atm 3 atm

CHL 4 K 20 K 3 atm 1.3 atm

Loop Anatomy

HX

Relief Tee

8” SS Support Pipe

Cell Manifold

(inlet)

Cell

Pump

Pump Inlet

Adapter

Heater

Brackets

Cell Adjuster

5/24/2010 Qweak G. Smith: 8

Qweak Solid Targets

• 35 cm LH2 transverse flow cell (X0~4.0%)– US window ~ 4 mils Al 7075– DS window ~ 5 mils AL 7075

• Dummy targets: (horizontally centered tgts underlined)– US: 1%, 2%, & 4%– DS: 2%, 4%, & 8%

• Optics tgts: Five 1 mm thick Al foils.• 6 combos: Foils 3, 1&5, 1&3&5, 1&2&3&4&5, and 2 others• Foil ~locations: 1 at 0 cm, 2 at 8.75 cm, 2 at 17.5 cm

• Other tgts: – Viewers: BeO high current & AlO/Cr low current with graticule.– Hole targets: 2 mm US & DS, 6 mm square, 0.5” (halo). – Thick (~10 mm) and thin (2.5 mm) C.– Pure Al– Thick Be.– Key.– Empty frame.

5/24/2010 Qweak G. Smith: 9

LH2 Cell

Beam

5/24/2010 Qweak G. Smith: 10

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

0 500 1000 1500 2000 2500

5/24/2010 Qweak G. Smith: 11

LH2 Target Specifications

• T = 20 0.01 K – (TF = 13.8 K, TB = 23.6 K)

• P~30-35 psia (20 psig)

• Pbeam ~ 1.8 kW @ 150 µA

• Ptotal ~ 2.3 kW @ 150 µA– Reserve power ~ 150 W

• Fan = 30 Hz

• 5x5 mm2 uniform raster

LH2 (35 cm cell):

Heaters must provide Pbeam + Preserve ~ 1.9 kW (@ 150 µA)

Coolant (JTs) must provide Pbeam + Preserve + Ppump + Pother ~ 2.3 kW

Co

ola

nt

ma

ss f

low

(g

/s)

Cooling Power (W) for a 20 K Target

4.5 K

15 K

0

500

1000

1500

2000

2500

3000

0 50 100 150 200

5/24/2010 Qweak G. Smith: 12

Power vs Current (35 cm LH2)

Beam Current (µA)

Bea

m P

ow

er

(W)

Beam Power

Heater Power

Cooling Power

Po

wer

(W)

5/24/2010 G. Smith: 13Qweak

5/24/2010 Qweak G. Smith: 14

New Recovery HX in ESR

General advice for

an off-normal event:1. Think!

2. Respond if you think you understand what the problem is and what the proper response is.

3. You can almost always control the tgtmanually with the manual heater, or the JTs, to keep it safe.

4. Reboot (rebooting solves many problems).

5. Call an expert unless you are 100% positive the problem has been dealt with successfully. For all but routine alarms, call expert anyway.

6. Log the problem.

5/24/2010 Qweak G. Smith: 15

High Loop Temperature Alarm

1. Check other TS’s, & analog temp. r.o.1. If just PID TS change PID TS

2. LH2 Pump tripped? Restart.

3. HPH should be @ 0W. If not turn it off.

4. SC vacuum OK? 1. Turbopump tripped? Reset ASAP in hall.

5. Coolant Tsupply abnormally high?1. Close offending JT, open other JT more.

2. May be ESR/CHL or xfer line IV failing.

6. Reboot IOC.5/24/2010 Qweak G. Smith: 16

No Danger. Just delays.

Low Loop Temperature Alarm

1. Check other TS’s, & analog temp. r.o.1. If just PID TS change PID TS

2. HPH should be railed. PID working?1. Use manual HPH to restore temps.

1. Look at analog temp readout

2. May need to slowly close JTs some.

3. Coolant (ESR) Tsupply abnormally low?1. Close ESR JT some.

4. Reboot IOC.

5/24/2010 Qweak G. Smith: 17

Danger: freezing the H2

Low Loop Pressure Alarm

1. Diurnal pressure variation?1. Check archiver to see recent history.

2. PV12 closed? System can’t breathe:1. If SC vac OK open PV12

2. If SC vac NOK do not open PV12!1. Turbopump tripped? Reset.

2. Potential H2 leak. BAD.

3. Reboot IOC.

5/24/2010 Qweak G. Smith: 18

Danger: going sub-atmospheric & freezing air in relief path

Tgt

Tgt Supply

Ballast Tanks

High Loop Pressure Alarm

1. Diurnal pressure variation?1. Check archiver to see recent history.

2. Temps high? 1. System is warming up.

2. Warm (storage) pressure ~ 66 psia.

3. See High temp alarm procedures:1. Probably the LH2 pump has tripped, or else a

vacuum pump has tripped..

3. Reboot IOC.

5/24/2010 Qweak G. Smith: 19

Danger: Bursting cell window

Pt TS Alarms

1. Solid target ladder (high) alarm?1. Reduce solid tgt beam current asap.

2. Phytron motor (high) alarm?1. Stop hor. motion. Brake release gets hot.

2. Let it cool before moving again.

3. LH2 pump (high) alarm?1. Open bypass more.

2. Lower pump speed.

4. Hor. motion table (low) alarm? 1. Call an expert. Bad.

5. Reboot IOC.5/24/2010 Qweak G. Smith: 20

Danger: melting solid tgts, burning up motors.

Vacuum Alarms

1. Monitor vacuum stripcharts over ~ days.1. SC vacuum should be < 5x10-7 Torr

2. Turbopumps:1. Scattering chamber & vacuum dumptank

2. Upstream & Downstream beamline

3. Can close GVs to isolate & diagnose problem

3. If one tripped, reset ASAP:1. Escorted, immediate access with ARM

2. GVs close automatically at ~ 5x10-5 Torr

3. Can develop ice on tgt, ruin expt.1. May have to warmup to eliminate the ice

4. At few Torr, SC vac switch kills pump, HPH, etc.

5. Other possibility: H2 leak or o-ring leak. Call expert.

6. Reboot IOC.5/24/2010 Qweak G. Smith: 21

No Danger. Just delays & headaches.

H2 Sensor (sniffer) Alarm

1. Located above gas panel & target:1. Have always been false alarms so far.

2. Have failed to detect real leaks so far.

2. Clear tgt & gas panel area if personnel in hall.

3. Check system pressure for H2 leaks:

1. Use archiver to look for slow leaks.1. Fold in effect of diurnal P variations

2. If a real leak seems possible, use extreme caution!1. H2 flame is odorless & colorless

4. Call experts:1. Will sniff with a portable sensor

2. Will check if sniffer is OK or not.

5/24/2010 Qweak G. Smith: 22

Danger: System leak, hydrogen fire.

Non-Target fire in Hall C

1. Sound the fire alarm & evacuate Hall C.

2. Call 911.

3. Call 4444.

4. Call MCC.

5. Close GVs on both sides of the SC.

6. Close JT valves.

7. Initiate target warmup procedure.

5/24/2010 Qweak G. Smith: 23

Danger: Obvious

LH2 Pump Trip

5/24/2010 Qweak G. Smith: 24

1. Will test this to refine response

2. In the meantime:

1. Expect T to drop at first, then rise

2. Thermal siphoning should help some

3. Takes time (15-20 min) to leave VP curve (phase transition)

4. Reset pump controller asap

1. May require an escorted entry

5. If pump is toast (eg bearing failure) so are we. Call an expert, warmup tgt.

No Danger. Just delays.

Breach of system in Hall C

(H2 release into Hall C)

1. Evacuate Hall C.

2. Shut off pump and heater

3. Do not operate lifter, JTs, etc.

4. Call cryo to shut off coolant

5. Call experts to pump & purge system, …

5/24/2010 Qweak G. Smith: 25

Danger: System leak, hydrogen fire or explosion.

EXTREME HAZARD!

Power Failure

1. System is designed to handle this event safely, passively.

1. GVs will close, generates an FSD, kills beam.

2. Pump & heater will stop.

3. Vacuum will begin to spoil.

4. TS’s, JT’s, & control computer are on UPS.

5. Close JTs. Very important if ESR or CHL are still up & sending coolant to the target.

1. If you can’t, call MCC to get cryo on call to stop coolant at source in case it has not already stopped.

5/24/2010 Qweak G. Smith: 26

Danger: Freezing H2, blocking relief path

Heater Failure

1. There are two redundant relief paths:1. On either side of HX.

2. If both were to become blocked, loop would fail through exit window into SC & vent line (safely).

2. PID problem? Reboot.

3. If heater PS fails, use secondary (manual) heater power supply

1. Power cycle the primary HPH PS- this usually fixes problem

4. If neither PS works, heater coil may be shorted.

5. Then close JT valves. Can control tgt temp with reduced JTs alone, without a heater (manually).

1. Call expert. May have to warmup and repair.5/24/2010 Qweak G. Smith: 27

Danger: Could freeze the H2, & turn the HX into an ice cube.

Computer failure

1. Common occurrence.

2. Reboot IOC1. Ask MCC for beam off

2. Use manual HPH PS & analog temperature readout to control tgt temp during reboot

3. Should happen less frequently now1. Our IOC is now in the CH, not in Hall C

5/24/2010 Qweak G. Smith: 28

Danger: You are blind, don’t know what is going on.

Outer limit switches hit

1. Hard stops will prevent a catastrophe.1. But we do not want to rely on them.

2. If you have hit the outer limit switches something has gone very wrong.

1. Target motion is inhibited.

2. Can only be restarted by an expert.

3. Call an expert

5/24/2010 Qweak G. Smith: 29

Danger: You may damage the tgt or the motion system.

5/24/2010 Qweak G. Smith: 30

Hall C Layout

Target

Target Electronics

Target gas panel

Key room

Beam

5/24/2010 Qweak G. Smith: 31

Target

Electronics

Fan controller

JT control

Motion controllers

Tachometer

HPH power supplies

P readouts

Temps

Temps

Cryogen Temps

5/24/2010 Qweak G. Smith: 32

Ballast TanksH2

D2

G0

6000 gallons

across 3

tanks

Warm storage

pressure is

66 psia

5/24/2010 Qweak G. Smith: 33

• Ballast tank outside, and P T.

– Ex: ΔT=15C (5%) ΔP=1 psi

• Systematic effect on target ρ negligible.

23 psia LH2 density vs temp

y = -0.2495x + 76.507

R2 = 0.9981

67

68

69

70

71

72

73

74

75

76

77

1515

.5 1616

.5 1717

.5 1818

.5 1919

.5 2020

.5

Temp (K)

de

ns

ity

(k

g/m

^3

)

23 psia

25 psia

27 psia

Linear (23 psia)

24h

1 psi

Diurnal ΔP

Questions?

Use the

archiver!

Counting Room

• Target computer (controls, charts)

• TV camera looking at tgt

• TV cameras looking at Hall C (iceballs…)

• Target IOC (iocctarg) – Not in Hall C anymore, we hope…

• Manual heater controls

• Target motion kill switch

• Documents (how-tos, phone #’s, etc.)

5/24/2010 Qweak G. Smith: 35

Getting Started

• log into gzerol2.jlab.org as cvxwrks– PW is ____________

• cd $GUI (/u/group/poltar/HallC/QWeak/Screens)

• Type ./tgtgui

• This will launch medm & the main GUI.– From main GUI, launch alh, strip charts, etc.

• Insure alh visible everywhere (each workspace), alh & logger running. Heartbeat OK?

• Crack secondary GUIs from main GUI

5/24/2010 Qweak G. Smith: 36

Qweak Main GUIHeartbeatAlarm Handler

Strip Charts

Secondary

GUIs

Coolant

Params

Secondary T & P GUIs

5/24/2010 Qweak G. Smith: 37

35 36.3 1.3 35

Expected values

when H2 condensed

Expected value

when H2 condensed:

20K (loop)

Heater GUI

5/24/2010 Qweak G. Smith: 38

Qweak: 1.2 Ω

20

HPH PS:

60V, 50A

5/24/2010 Qweak G. Smith: 39

Adjusting the Heat Load

• Pbeam~ 1750 W at 150 µA, Preserve~ 150 W

• With beam on, need ~ 150 W on HPH

• With beam off should have ~ 1900 W on HPH

– (if off a long time, can reduce)

• HPH power adjusted automatically

– (in PID looking at Tloop & Ibeam)

• To adjust, use JT:

– Open JT to increase HPH power

– Close JT to lower HPH power

JT Controls

5/24/2010 Qweak G. Smith: 40

Change :

< 2 % per minute

Change :

< 4 % per minute

Change :

Experts only

Old GUI

1% 2%

Hybrid 3 kW HX

5/24/2010 Qweak G. Smith: 41

• Uses 4K from CHL (20K return)

• Uses 15K from ESR (20K return)

• Danger: 4K < 14K. H2 freezes at 14K!

• Danger: changing 4K JT too quickly can crash the CHL

½” Cu fin tube,

16 fpi

Downstream Geometry

5/24/2010 Qweak G. Smith: 42

Scattering Chamber

16” Al extended

stroke

Gate ValveSpool Piece

Scattering Chamber

wagon wheel

Window

Tungsten Plug

(inside collimator)

Collimator Frame

Scattering Chamber Window

Have three tested

35 mil thick windows.

Gas

Panel

5/24/2010 Qweak G. Smith: 44

Ballast

(storage)

tanks

Target &

Vent Line

Schematic

5/24/2010 Qweak G. Smith: 45

Gas Panel

5/24/2010 Qweak G. Smith: 46

Vent Line

Vents H2 outside Hall C in case

primary relief fails, or primary

containment fails.

Purged continuously with

N2 gas.

LH2 Centrifugal Pump GUI

5/24/2010 Qweak G. Smith: 47

Parameters :

30 Hz

15 l/s

(1.08 kg/s, 238 gpm)

1.3 psid (13m)

3.3 m/s in a 3” pipe

0.3 hp (225W)

1.2 NM (168 oz-in)Coolant bypass for cooling pump motor

Coolant GUI

5/24/2010 Qweak G. Smith: 48

Cartoon needs to change to reflect

Qweak loop.

However, coolant parameters will

carry over.

Can drill down to ESR parameters

with the ESR button.

Beam Current & Raster Size Alarms

5/24/2010 Qweak G. Smith: 49

These should only be changed by a target expert.

If either the beam current limit or the raster limit is

violated for a given target, an alarm will be generated.

Ask MCC to turn off the beam immediately!

5/24/2010 Qweak G. Smith: 50

Alarm Handler• Alarm Handler GUI must be visible in each

workspace• If not running, from main GUI click:

– Charts & Alarms in upper right corner– Choose Start Alarm Handler– Load appropriate alarm file (usually ctarg.alh)

• Alarm Colors:– Red: major, ouside wide limits– Yellow: minor, outside narrow limits– White: readout error

• Alarm Color States:– Steady: alarm condition was transient– Flashing: alarm condition is active

5/24/2010 Qweak G. Smith: 51

Servicing Alarms

• Click on alh bar to open alarm tree GUI • Click lit main branch (left side) text button• Click lit sub-branch (right side) “P” button• On this new GUI look at the alarming sub-branch.

Shows current value and all the alarm limits.– OK? click lit button left of sub-branch to clear the

alarm. Transients can happen. – Not OK?

• May be telling you something (adjust JT?). Think!• If can’t understand why it’s alarming, call an expert.

• Log the event• Change alarm limits only if asked to by an expert

5/24/2010 Qweak G. Smith: 52

Alarm Handler GUI1

23

0

5/24/2010 Qweak G. Smith: 53

StripTool

• Watch out: vertical scale only appropriate for selected color. Have to click on desired variable to see its scale!

• Horizontal Scale: Buttons in lower left corner of the graph

• If you zoom, you may also want to right justify!

• Vertical Scale: Right-click inside graph to bring up controls dialog

5/24/2010 Qweak G. Smith: 54

Check Logger

• From main GUI:– Type ps –e | grep ArchiveEngine on a control computer

xterm• OK? Then 3 instances will be listed

• Not OK? You get nothing back. This should never happen, since cron checks for it. But if it does:

– Then, type cd /Archives

– Then, type ./start_archiver

• When making hclog entries, use keyword “target:”

• Archiver: http:/jeffylab.jlab.org/~cvxwrks/– Very useful to see history of any target parameter

– Query epics name: right-click in gui, choose “PV info”, then left-click on value whose name you want

5/24/2010 Qweak G. Smith: 55

Checklist

• Use ksnapshot

• Put a picture of the main guiand the stripcharts into hclog

• Note any unusual conditions

• Do this at least once per shift

5/24/2010 Qweak G. Smith: 56

Changing Targets

• Call MCC: beam off, mask tgt motion FSD, explain which tgt you’re going to

• From main GUI, click desired tgt: – Observe motion on TV. – Hit panic button if necessary.– Note: light switch on main console…– Wait for the green light to come back, check you are

where you should be. • Check resistive encoders jive with stepper encoders

– Close the relevant GUIs

• Call MCC: beam & FSD on, new Ib & tgt– check Ibeam limit & raster OK! Use operational

restriction web page.– Check bpms & halo

• Log configuration change• Check hole tgts from time to time

5/24/2010 Qweak G. Smith: 57

• Why? Heartbeat lost, stripcharts flat, white medm readouts, etc.

• Don’t panic. You can do this!• Call MCC for beam off.• Set manual HPH pot for appropriate

power• Monitor analog temps, fine-tune HPH• Reset IOC in counting room• Wait 2 min. Check PID working again.• Turn off manual HPH. Beam back. Log it.

Rebooting the Target IOC

5/24/2010 Qweak G. Smith: 58

Cryotarget Web Page

From Hall C Web page:

www.jlab.org/Hall-C

Cryotarget link

Contents:

• Design document

• How-tos

• Contacts

• This talk

• Goal parameters

• Thicknessess

• Documentation

• Etc.

5/24/2010 Qweak G. Smith: 59

A Typical Shift

(what you’ll really wind up doing)

• Monitor tgt parameters, vacuum, heater

• Service the occasional alarm

• Log target parameters & changes

• Change targets

• Adjust JTs in response to changes at ESR & CHL

• Occasional tgt IOC reboot

That’s all!

5/24/2010 Qweak G. Smith: 60

Done!

• Remember to sit ~½ shift with a trained operator to complete your practical

• Then email smithg@jlab.org that you have completed your training

• Read documentation available from the Hall-C Cryotarget website

• Stay cool!