Shifter Instructions

Instructions for the SCT ACR Shifter

Version 0.4 – February 2010

ContentsPart I – Shifter Duties

Part II – ACR environment

Part III – Run Control & DAQ

Part IV – Beam

Part V – DCS

Part VI – Data Quality Monitoring

Part VII – Problems

2SCT Shifter Manual

Shifter Duties

Part I - Instructions for the SCT ACR Shifter

Before your shift

1. Make sure your training is up to date : courses Level 1, Level 2, Level 3 and Level 4A and SCT Shifter Qualification

2. Your access rights should be valid (ATL_CR) and SCT:shifter rights.

3. Read documentation on Twiki page : http://atlasop.cern.ch/twiki/bin/view/Main/AtlasOperation and go to shifter manual.

4SCT Shifter Manual

Beginning of your shift

1. Come to the ACR at least 15 minutes before your shift and consult with the previous shifter and check:

1. Current ATLAS conditions

2. Particular item under watch?

3. Where we stand respect to run plan?

4. LHC status and plan for the coming shift

2. Your access privileges will be granted automatically 2 hours before the beginning your shift

3. Check the run plan• http://atlasop.

cern.ch/atlas-point1/wmi/current/Run%20Status_wmi/AtlasRunCom.html

4. Read previous shifter e-log and start your new entry

5. Log in to the DCS, FSM Alarms and Run Com (Check-List Utility) 5SCT Shifter Manual

During your shift1. Do the checks requested in the Check-List

2. Monitor DCS alarms on the DCS Alarm Panel

3. Look for ERROR and FATAL messages at the DAQ coming from the SCT

4. Check Monitoring Histograms and DQM status flags

5. Check beam conditions are stable

6. Monitor Cooling status and alarms

7. Document in the e-log (SCT and IDGen) problems and happenings

8. In case of problems contact relevant experts and/or Run Coordinator

6SCT Shifter Manual

End of shift1. Submit e-log entries SCT and IDGen (log out of the e-log)

2. Inform the next shifter on the current situation of the running and about any special conditions or plans.

3. Log out of the Check-List, DCS and FSM Alarms

4. Do not leave if following shifter did not arrive (after 30 minutes call Run Coordinator if shifter has not arrived).

7SCT Shifter Manual

ACR environmen

t

Part II - Instructions for the SCT ACR Shifter

ACR Layout

SCT Shift DeskACR, CERN, Meyrin

Shift Leader DeskACR, CERN, Meyrin

9SCT Shifter Manual

ACR Desktop

TDAQ Screen DQM Screen DCS and Alarm Screens

1. Desks have 4 Screens to be used by the shifter

2. Get familiar with the environment

3. Some useful Utilities can be found in the General tab

4. Example of ScreenShot of the ACR Desktop

10SCT Shifter Manual

SCT Shifter Manual

RunCom1. In the General tab find

RunCom

2. Look for the SCT tab and log yourself in

3. Log out of the Check-List and FSM

4. Read the checks and comments, some of the requirements have an information page

5. DQ checks will appear every hour

Relevant for ATLAS runs but NOT for SCT Standalone 11

ATLAS e-log1. e-log can be found under the tab General

2. To be able to submit entries you have to login with your P1 login/password

3. Make entries if problems occur do not wait until the end of your shift

4. Do a Shift Summary (and select ID-Gen and SCT) for every shift

Remember to select all options and attach plots (if any) before submitting

12SCT Shifter Manual

ATLAS e-log Sample

13

Short summary…..

*******************************************15.00: start of shift

15.04: Recovered module Crate 22 Ch 4 which showed a trip error.

17.03: end of run 139303

17:05 Expert takes over DAQ to investigate XXX18:30 DAQ tests concluded. The result is that blah blah blah... and more blah, blah, blah… Changes have been applied, blah, blah, blah…

18.40: SCT on stand-by19.17: Start of Physics run 13933419.30: SCT on during LB 6.

20.00: Monitoring the number of errors, stable and around 20-30,see separate report from MON shifter.…21.12: Run 139336. Try a short cosmics run again(8 bins, 1000 triggers, timeout 60s)Result status -> no fail, but quite a few problematic modules?

… SCT Shifter Manual

SCT Shifter Manual

Atlas Operation1. Information for shifters will be posted in the Operations page.

• From outside P1 : http://atlasop.cern.ch/twiki/bin/view/Main/AtlasOperation

SCT Links

General Information

14

SCT Shifter Manual

SCT WhiteBoard1. Special announces, SCT DAQ Panel Configuration and SCT Phones

• From outside P1 : http://atlasop.cern.ch/twiki/bin/view/Main/SCTDetectorWhiteBoard

• From P1 : ID Menu SCT WhiteBoard

Important Announcements

Phones

15

SCT Shifter Manual

SCT Shifter Manual

1. Information for your shift• From outside P1 : http://atlasop.

cern.ch/twiki/bin/view/Main/SctShifterManual

• From P1 : Link from the Atlas Operation Twiki

16

pplications1. DAQ Panel go to TDAQ tab

2. FSM Alarms and DCS go to ID SCT tab

3. RunCom go to General tab

4. e-log go to General tab

5. konqueror go to General tab

6. ksnapshot go to General tab

17SCT Shifter Manual

Run Control & DAQ

Part III - Instructions for the SCT ACR Shifter

DAQ Panel

Log Window

Options

Applications

19SCT Shifter Manual

DAQ PanelSelect the setup script (/det/tdaq/scripts/setup_TDAQ.sh will always point to the production release)

1

Select the partition db file (i.e., /atlas/oks/tdaq-02-00-03/combined/partitions/ATLAS.data.xml)

2

Get the partition(s) described in db(the Part Name list will be automatically filled)

3

Tell the DAQPanel to acquire the information

4

20SCT Shifter Manual

DAQ Panel : Main1. Monitor Partition: Use it to

Spy on the ATLAS Partition status

2. Log Manager: Browse the log messages produced by the applications

3. MRS: It starts a viewer of the MRS (Message Reporting System) messages flowing through the system

4. Busy: It starts the Busy Presenter. Use it to understand whether/why the system is in a busy state

21SCT Shifter Manual

Monitor Partition GUI

22

RootController

CommandsTab Panels

MRS Log Window

Run Information & Settings

SCT Shifter Manual

MRS

23

Change subscription

Message format

Clone table

Start from DAQPanel – Main tab – MRS button

SCT Shifter Manual

Busy Monitor

24

1. No data flowing through the system? • The system may be in a BUSY state

2. Start the Busy Presenter from the DAQPanel: Main tab, BUSY• RED: the corresponding subsystem is busy• PINK: the corresponding subsystem is busy but out of the

PARTITION• GREY: the corresponding subsystem is not part of the run

If SCT is Busy and do not know what to do CALL DAQ EXPERT (162034) for instructions

SCT Shifter Manual

SCT Gui

25

1. DAQPanel in the SCT Desk has an additional tab called SCT 2. Use the button to launch the SctGui. Use the GUI to:

• Run Mode Physics scans• Alternatively check DCS conditions• Probe and check modules : if OUT of ATLAS running and you

have been instructed to do it

SCT Shifter Manual

SCT Gui : Short Tour

26

Data shows different options depending on the kind of display selected (Module Configuration in this case)Views: Geometry and ROD cratesList of: Barrels/Wheels or Crates in the configuration

Different module groups

Each of the squares is a module

SCT Shifter Manual

SCT Shifter Manual

SCT Gui : Short Tour

27

Example: SCT View

SIDE

Coordinates of the module under the mouse

0 – No Data1 – Bad Data2 – CLK/2a – Corrupt DataA – ABCD Error codeC – Config Readback modeE – Returning eventsJ – Junk DataO – Other

SCT Gui : Short Tour

28

You can access to the different menus using the toolbar or with a right click on the empty part of the screen

- Module Configuration: Change modules in the configuration- Probe Data: Check for modules with errors and recover them. Using this display you can perform actions like Power Cycle, Reset, Configure…- Data Manager: Allows you to access to results and plots from a test- Test Results: Show test results for all the modules- DCS Conditions: Several of the DCS parameters related with the modules are available through the SctGui (Temperature, LV, HV,...)

SCT Shifter Manual

SCT Gui : Short Tour

29

How to check the ROD which is BUSY?

• Right Click at the TIM Module and select TTC Status

- If Right Click on any of the Crates a menu will show three possibilities:- Clear Latch - Mask ROD Busy - Raise ROD error (manual stop-less removal) : Use this one if a

busy appears and is not picked up by the automatic removalOnly do this if DAQ EXPERT (162034) ask you to do it

TTC Status

Physics Mode Scan

30

- In SctGui : Tools Menu Physics Mode Scan - Need to be in Physics mode (standalone) or in ATLAS partition- Uses triggers to acquire events, number of events and crates can

be selected- Results are displayed in the SctGui – Test Results

Physics Mode

Remember…

31

I. Important to be specially vigilant at the beginning of the run. Problem most likely will happen at the point. Look for:a) FATAL / ERRORS from the SCT appearing in the MRSb) Check BUSY status (and TTC Status). If no triggers are coming

through one of the systems could be busyc) Stop-less removal could happen at this stage. Ask to restart

the run if:• 3 Barrel RODs or 7 RODs (Barrel + EndCap) get disabled

d) Always write Information on SctGui Run comment window (pop-ups at the start and end of the run)

e) Run a Physics Mode Scan and check for errors

II. During the run :a) Keep checking on disabled RODsb) Check ROS Monitoring every hour. Look for errors and

anomalous values in the resultsc) Continue to check for errors in MRS d) Document everything

SCT Shifter Manual

Beam

Part IV - Instructions for the SCT ACR Shifter

33

Also available on the WEB @http://atlasop.cern.ch/atlas-point1/operRef.php?subs=dcs/dcs/process.php?page=ATL_LHC

Beam Intensity From LHC (P4)

<#protons> / bunch

Beam IntensityFrom Beam Pick Up <#protons> / bunch

SCT Shifter Manual

ATLAS Stand-by Mode

34

• ATLAS should ALWAYS be in stand-by mode except when:

• STABLE BEAMS have been declared Physics Mode

• Beam is not possible due to LHC conditions LHC NO BEAM Mode

• In ALL OTHER situations ATLAS MUST be in stand-by mode

• "Stand-by Mode" is state of ATLAS hardware• it is decorrelated from the data aquisition state • it is possible to take data (not physics quality) with

ATLAS in stand-by

• SCT in Stand-by : LV ON and HV at 50 V

SCT Shifter Manual

Fill Pattern

35

Beam Injection: I

36

1. ATLAS Shift Leader receive LHC INJECTION WARNING via the LHC handshake panel. Big Sound coming from the DCS Desk!

2. ATLAS shift leader replies ATLAS INJECTION = PREPARE, within few seconds, otherwise phone call from CCC

3. ATLAS shift leader asks the shifters to prepare for for INJECTION:• Make sure that SCT is in Standby and Beam

Protection is ACTIVE• Check the INJECTION PERMIT mask. SCT

should NOT be MASKED

LHC Page

37

How to see if SCT is in STANDBY?

Masks

Link to the LHC Handshake page

SCT Beam Page

38

SCT Status : Injection Permit and

HV

Link to the LHC Handshake page

Beam Protection OFF / ON

SCT Beam Page

39

SCT Status : Injection Permit and

HV

Beam FlagProtection OFF /

ONLHC Status

Beam Injection : II

40

4. ATLAS Shift Leader turns the Injection Key5. Shift Leader uses hand-shake page and declares

ATLAS INJECTION READY6. LHC will inject Pilot bunch...

During Injection

41

• Monitor carefully beam losses (Backgrounds) o ATLAS BCM and BLM readingso LHC BLMs

• Monitor Beam parameters, position, ... o BCM, LUCID, MBTS, L1Calo rates, Tile

integrated currents

• Inform Shift Leader if some problem is found.

Instructions and limits will be set soon

ATLAS in Beam Mode

42

• Only with STABLE BEAM• All systems should have Low Voltages (Front end powered,

system can be read out)• All systems should have High Voltages / Bias Voltages, systems

are providing signals from particles

• SCT ON only with STABLE BEAM and GOOD BEAM Conditions (TBD)

Stable Running

43

• When STABLE BEAM Flag becomes TRUE. Shift Leader will ask systems to go ON

➔ Check STABLE BEAM Flag AND Beam BACKGROUNDS : 1. Inform SCT Run Coordinator and ask for approval2. HV can go ON3. Inform Shift Leader that system is ON

➔ Keep Checking Beam Conditions at ALL TIMES during your shift.

Removal of Permit

44

1. If the Beam Mode= STABLE BEAMS, Injection Permit will be removed AUTOMATICALLY.

2. If a Beam Dump is requested anywhere around the LHC, our BCM receives a post mortem request which removes the ATLAS Injection Permit.

• Without beam: If the LHC Page 1 indicates that the beam will be absent from LHC for a period of a few hours “Quite Time”.

SCT will be informed that some “Quiet Time” is available. SCT Run Coordinator should be informed by shifter and/or ATLAS Run Coordinator.

Beam Dump

45

1. Planned Dump :• LHC send an ADJUST/ DUMP warning via the hand-

shake• SCT will go to Standby automatically when BEAM

DUMP state goes to INMINENT• Shift leader ask shifters PIX, SCT, MUO, LArg to put

their systems in safe state. Send SCT to Standby if not already

2. Non planned Dump :• Send system to Standby if not already• Ask Shift Leader for an explanation and post in the e-

log as much information as you can gather. Do not wait up to the end of your shift to post it!

• Inform Run Coordinator and DAQ On-call expert (162034)

• If qualified perform the calibrations for a Non planned dump, if not wait for the On-call expert

Operation Flow-Chart

46

NO BEAMShort Interruption

INJECTION PERMIT=FALSE

PROTECTION ACTIVESTABLE BEAM=False

NO BEAMLong Interruption

INJECTION PERMIT=FALSE

PROTECTION OFFSTABLE BEAM=False

INJECTION

INJECTION PERMIT=True

PROTECTION ACTIVESTABLE BEAM=False

STABLE BEAM

INJECTION PERMIT=False

PROTECTION ACTIVESTABLE BEAM=True

RAMP/SQUEEZE/..

INJECTION PERMIT=True

PROTECTION ACTIVESTABLE BEAM=False

Injection handshake

warningInject and dump

cycleor lose the beam

(no dump)

LHC Page 1 announces NO BEAM

for > 8hours

NO

YES

1) SL notifies ARC and shifters in the ACR2) Shifters inform their SRC if instructed to3) SRC requests to ARC to override STABLE BEAM flag4) ARC checks LHC plan, gives permission for STABLE BEAM flag override5) SRC can override STABLE BEAM flag and notifies SL

SL Removes Injection Permit

Injection handshakewarning

1) SL notifies ARC2) Remove STABLE BEAM override

LHC Page 1: Beam <2hours

1) SL notifies ARC2) Remove STABLE BEAM override3) SL goes through Injection handshake

Beam

dum

p

ARC= ATLAS Run Coordinator.SRC= Sub system Run CoordinatorSL = Shift Leader

SL goes through Injection Handshake & brings ATLAS to STDBY, gives Injection Permit

Beam

dum

p

Acronyms

ATLAS in Beam Mode

47

1

2

3

4 5

6

7

8

LBDSBeam-1

LBDSBeam-2

MKIBeam-2CIBC

CIBC.R3

CIBC.L3

CIBC.L4

CIBC.R4 CIBC

.L5CIBC.R5

CIBC

CIBC.L6

CIBC.R6

CIBC.L7

CIBC.R7

CIBC.L8

CIBC.R8 CIBC

CIBC.R1

CIBC.L1

CIBC.R2

CIBC.L2

CIBC

MKIBeam-1

INJECTION PERMIT= is Injection into LHC Allowed?

If INJECTION PERMIT is FALSE:• Beam cannot be injected• Ongoing injection is STOPPED or PAUSED

But Beam is NOT dumped

Remember...

48

I. In NON SAFE situations SCT must be in Standby :o Single or Double beam operation during the commissioning

phaseo During beam injection and while STABLE BEAMS = False o During fills even if there is no beam in the machineo Standby : LV ON and HV 20 V

II. Special Configuration applies for Beam Commissioning :o Splash Configuration : LV ON, EndCaps at 20 V and Barrel

at 5 V (OUT of the data taking)

This is the SAFE configuration up to further notice

III. Only SCT Run Coordinator or someone designated by the Run Coordinator is allowed to give you permission to switch ON or change running conditions (Commissioning Configuration Standby Configuration)o ON Configuration : LV ON and HV 150 V

DCS

Part V - Instructions for the SCT ACR Shifter

FSM

50

State machine hierarchyDetector hardware and hierarchical structure represented by FSM entities

State & command propagationStates upwards (using programmed logic with flexible rules)Commands downwards

StatusError handling upwards

PartitioningAsynchronous operation of sub-trees possible

State/Status/Command propagation can be inhibitedAccess control

Alarm Screen

51

1. Active and non-acknowledged alarms2. Acknowledgement:

• Only for some alarms• Operator needs to acknowledge

to remove it3. For each alarm:

• Get details for experts• Mask alarm on UI level until

the alarm condition goes• Insert alarm to Elog with comment• Display trend plot of value• In dev: alarm help on TWiki

4. Summary alert:• Hides several individual alerts in a single alarm entry• Access to individual alerts via “Details”

Log in as yourself !

DCS Screen

52

Log in as yourself !

Beam Flags Summary

SCT Status

Beam Status

Alarms

FSM Modes

53

• Padlocks:No control, Free

No control, Included

Exclusive, Owner

Shared, Owner

Exclusive, NOT Owner

Shared, NOT Owner

No control, Locked Out

• Padlock background:Tree complete

Excluded children(partitionable trees)

Disabled children(non-partitionable )

• Control:No propagation, NO commands

Propagation, NO commands

Propagation (if Included), Commands by you

Propagation (if Included), Commands by all

Propagation (if Included), NO commands

Propagation (if Included), Commands by all

No propagation, NO commands

Tree complete

Excluded children(partitionable sub-trees)

Disabled children(non-partitionable sub-trees or devices)

FSI Monitor in FSM

54

ON/OFF flag shows when FSI laser is ON (light in SCT), in the Barrel and/or both End-caps. Note FSI laser is usually ON, even when FSI partition is not running.

Access under: SCT -> INFRASTRUCTURE -> FSI

History plots show peak and mean power in each SCT region:fibres 0 & 1 are Barrel, fibres 2 & 3 are both End-caps.

FSI Monitor in FSM

55

1. Persistent HV trips on certain SCT end-cap modules.• Very rarely, overpower of the laser may cause

persistent HV trips on certain end-cap modules. The watch list below shows which modules are expected to trip and in which order, if the laser power were to exceed upper safety levels.

• 4114, 5929, 7834, 7937,4231, 3743, 6731, 7926

2. If persistent HV trips are seen on the above modules, first check whether FSI is ON using the laser power monitor in the FSM (previous slide). If so call the FSI expert with a list of which modules have tripped and in which region (barrel or end-cap). The FSI expert will reduce the laser power to this region, so the modules can be recovered.

3. If the ID cooling is about to be stopped (e.g. imminent power intervention) or restarted, please notify the FSI expert who will ensure the alignment system is running to record any interesting thermal motions.

ROD Racks

56

Access to ROD

Crates

Power

Remember...

57

I. Use TOP Level of the Tree to send detector to ON (GOTO_READY) and Standby (GOTO_STANDBY)

II. Inform the Shift Leader when going to ON or STBY. On their side they should ask you to do the transition. BE AWARE of the beam STATUS.

III. Cooling loop 186 is OUT. This loop has a leak and can not be repaired without access to the detector volume.

IV. Do not try to take OUT modules of the configuration without expert advice.

V. Check that the FSM tree is complete. Nothing should be out of the Tree (grey). But be aware that a few modules will be out and one of the cooling loops.

VI. Document problems in the e-log

VII. Call the DCS expert (162035) if you get ALARMS. Investigate the WARNINGS.

Data Quality Monitoring

Part VI - Instructions for the SCT ACR Shifter

Check-List

59

DQ Check-List • Every hour the list will pop-up for you to fill • Read the comments and look to the documentation if have any question

Comment

Help

IF Problem found : • Check in the List of known problems and monitoring documentation• Call Monitoring expert (160037) if : plots stopped updating, monitoring processes crashed,... and you do not have instructions on how to act

Accessing Apps

60

Main Monitoring Applications. From the Main tab in the DAQ Panel :

• DQM Display : this application displays a sub-set of the histograms with tests applied to them.• OHP : a larger set of histograms is available in it. Results of the check of the DQM are also shown there.

OHP Configuration

Buttons to OHP and DQM

Main tab

OHP

61Online Histogram Presenter – OHP

If NULL monitoring could be not working or between updates

Colors show the result of the Checks from DQM

Links to different tabs

DQM Display

62

Data Quality Monitoring Display – DQMD• Colors : GREEN, YELLOW, RED show status of the checks• Grey : Out of the Configuration or Masked (Link Masking +

Stopless removal)

SCT Tree SCT tab

Bad Module - RED

Histogram Display

63

Displays the content of the Information Server (IS) • Histograms will be found under ATLAS partition• Providers are called : SCT-PT-X while SCT-PT will be the gathered information from all of them • SCT-PT is the important one for the DQM. If this is not updated DQM will not update• Contains all histograms created by Monitoring. Useful for further debugging

Mon Advanced tab Last Update (60 s)

Press to get last update

MDA

64

Browse results for previous runs• Directory structure is the same as the one in the Monitoring files and in IS• For a given run files are saved every X (60) Luminosity Blocks (LB). An additional file is saved at the end of the run EOR• Useful to check if things are new or known

Mon Advanced tab

Summary Histogram

65

Displays a general view of the status of the system• Bins show an average over the current run

> 3

5 h

its

> 7

0 %

err

or

rate

< 9

0 %

effi

cient

> 0

.00

1 N

O

OU

T +

Mask

ed O

nlin

e

Large Number of Entries (>100) on ANY of the bins is BAD!

Number of modules OUT of the configuration is around 30, rest is Masked which should be small.

During WARM START, Noise and Inefficient will be high.

Remember…

66

I. Look at the Status of the DQM, if it turns red for Barrel or End-Caps, something really bad happened, if no clear reason – call DAQ expert (162034)

II. Check DQM if several modules (around 50) go RED. Investigate the problem – call DAQ expert (162034)

III. Loop over the histograms in OHP especially:a) Summary histograms – Look at contents of each of the bins,

check for increases. • BAD! If an increase of more than 30

b) Error summaries – Some modules could show some errors but a low rate, check the scale.• BAD! If more than 30 modules with around 100% error rate

c) Noise Occupancy – Check for a lot of modules with high occupancy (red) or white modules. Few white modules are worse than few red ones. • BAD! White modules after enough statistics or lots of red

ones.IV. Document problems and read documentation during or previous to

your shift. If found a problem call expert.

Problems

Part VII - Instructions for the SCT ACR Shifter

Problems – I

68

I. SCT sets a BUSY:Stop-less removal will start and a window will appear in the Run Control Desk. If this does not happen you can use the TTC menu. Ask Shift Leader to restart a run if more than 3 Barrel RODs or 7 RODs (Barrel + EndCap) get disabled.

II. Frequency Veto: A bunch of messages like these or similar appear in MRS:

Something is not working properly in the Trigger. Ask the RunControl shifter to check it. Good practice would be set triggers to HOLD while investigating.

III. Monitoring stops updating:Easiest way is to restart the whole monitoring segment, but probably restarting the Gatherer would work.

Problems – II

69

I. Cooling goes OFF:Call the Cooling expert (162449) and the SCT Run Coordinator. Document the alarms and errors from the FSM. Follow the error documentation procedure described in:

Shifter Manual page Cooling Infrastructure Failures

II. Power Cut:Same as above.

III. ….