The DØ Control System

J. Frederick Bartlett

For

The DØ Controls Group

Outline

Fermilab and the D0 Experiment The DØ Control/Monitoring System IOC extensions Process Variable Naming Conventions Database Management Significant Event System Detector Configuration Management Operator Display Support Library

The Fermilab Collider Complex

DØCDF

DØ Detector

Scintillating Fibers

Liquid Argon/Uranium

Calorimeter

Central MuonLayer A

Silicon Tracker

Superconducting Solenoid

Central MuonLayer B

Central MuonLayer C

Forward MuonLayer C

Muon Toroid

DØ Run II Control System

EPICS-based controls Field buses

VME MIL/STD1553B CANBUS (Run IIB)

Size ~15 host-level processors ~100 IOCs (Input/Output Controllers)

MVME2301, MVME2304, MVME5500 ~6600 high-level devices ~138000 process variables (records)

DØ Run II Control System 15 major detector sub-systems Host-Level processes written in Python Source management - CVS Operating systems

Host processors - Linux IOC processors – vxWorks

Software versions Now - EPICS 3.14.6 Next - EPICS 3.14.7 with patches, using SNS linux build tools

Controls staff Core system - 3 FTEs (4 people) Detector-specific components - ~2 FTEs

Primarily from other institutions

Devices and Records

Detector Devices Records

Argon/Uranium Calorimeter 844 9219

Scintillating Fiber Tracker 445 35900

Muon Spectrometer 1572 20027

Silicon Microstrip Tracker 2235 45706

Controls 440 3775

Trigger Systems 694 21644

Other Systems 468 2693

Total 6653 138964

EPICS IOC Extensions

Channel Access

Database Access

Record Support

Device Support

Driver

VME + MIL/STD1553

Scanners

DB Load

Ethernet LAN

IOC Database

CustomizingElements

MIL/STD1553Bus and CANBUS

VME Bus AccessWith Retries

HV GeneratorState Machine Off

On

Offline Disabled

Average

Holding

Tripped

Paused Locked

Start

Begin Ramp

End Ramp

Mid Ramp

Gateways to Other Systems

Accelerator Gateway link to ACNET system Bidirectional Data access only (no control)

Cryogenics and Gas Gateway link to DMACS system Read-only Data access only (no control)

ACNET Gateway

xmlrpc Server

ToACNET

FromACNET

ToFromITC

ToEPICS

FromEPICS

CACHE

REMOTE

EXPORT

xmlrpc

xmlrpc

ITCCLIENTS

CA

CA

EPICSRECORDS

CLIENTS xmlrpcCLIENTS

D0 BD

DØ Naming Convention

Name elements Detector <det> CAL Sub-det <sub> N Device <dev> VBD Locator <loc> 01 Attribute <attr> STATUS I/O <io> W Field <field> SCAN

Template<det>[<sub>]_<dev>_<loc>/<attr>[:<io>] [.

<field>]

CALN_VBD_01/STATUS:W.SCAN

Frequently used in filters

Many utility tools use this field for device-specific

actions

Database Management

Basic EPICS uses flat text files Template definition file

Multiple record definitions with substitution parameters Template generator file

Create a group of records (instance of a template) with values for parameters

Record instance file Instances of records read by IOC to create record

database

Relational Database System EPICS database creation from text files

Record structure defined by DBD files Template structure defined by template definition

files Template instances defined by template generator

files or from Browser GUI IOC record instances (record instance file) are

extracted from database Database support programs written in Python

Relational Database System

OracleHardwareDatabase

TemplateDefinitions

DBCreation

RecordField

Definitions

RecordExtract

TemplateGenerators

List ofRecordsFor IOC

IOC Id

TemplateExtract

TemplateGenerators

For IOC

InstanceCreation

Significant Event System

The significant event philosophy Alarms are a only a sub-set of the significant

events EPICS does not generate all of the significant

events of interest Alarm utilities enhance reliability

Detect impending failures and fix them before they fail Minimize the time to correct failures

Archive all event transitions The archive is a history of the state transitions of the

experiment Tools available to search the event archive What was your system doing last month?

Alarm/Event Distribution

SES is a server-based, event (alarm) system: IOCs and user processes connect to and send

alarm transitions to the server Pushed by sources not pulled by the server or client

Server holds the current experiment (alarm) state Server has one or more filters for each receiving

client Makes use of name structure

Rapid display startup of receiving clients User processes may also declare events via API

(C, C++, Python) Written in Python

Significant Event System

Significant Event Server

EPICS IOC ProcessWatcher

PeriodicHeartbeat

AlarmDisplay

F

AlarmWatcher

F

RunSuspend

Run Control(COOR)

F

F FilterFilter

SE MessageFiltered Message

seBrowser

seLogger

F

EPICSIOC

Process

SES Alarm Display

Select Guidancebutton

Events in line selected by filter

Select minor alarm button

Detector Configuration Management

EPICS Save-and-Restore is adequate for accelerators Accelerators are tuned for an operation mode and

the configuration is saved for future restoration Some complex devices like HEP detectors do not

work this way There are too many distinct modes to be saved Tuning may not be required

Settings are often predictable from trigger selection Many detector component settings are taken from

separate calibration measurements

Comics Server

Manages the configuration of the detector Receives load requests from:

Run control Expert GUIs Program APIs

Load map is a directed graph (tree) Tree node (intermediate)

Establishes a layered hierarchy Establishes an execution order

Action node (leaf) Issues EPICS CA requests

Comics Server

Constructed on the server model with multiple clients sending commands

Server may be activated independently for configuring detector component

Coded in the Python language

Configuration Management Tree

IntermediateTree Node

CFTSMTCAL MUO

DEV2 DEV3

S0 S2S1

DEV0 DEV1 T0

Root Node

Configuration Data

Action NodeUses EPICS

Channel Access

Links determine the

order of execution

DØ

Comics Expert GUI

Operator Display Library

Standard, process flow (synoptic) displays do not adapt well to the monitoring of HEP detectors Like accelerators, there are many similar devices

that make up the typical detector component (cells in a calorimeter)

The components are not related in a serial fashion like a cryogenic plant

Tabular (spread-sheet designs) are more natural Similar properties for different devices are easily

compared Deviations are apparent

Operator Display Library

DØ has developed a series of Python display classes for building tabular displays that collect information from EPICS process variables

Resource Display

Device Window

Basic Frame with Menu Bar, Status Window, and Button Bar

Tabbed Pages with CA

Connect and Disconnect

Device Analog Property

Device Status (Bit-Encoded)

Property

Right-Click on Name Field