Controls and Monitoring Implementation Plan J. Leaver 03/06/2009.

Controls and Monitoring Implementation Plan

J. Leaver

03/06/2009

04/18/23 Imperial College 2

Implementation Issues

• Organisation & responsibilities

• General EPICS infrastructure

– EPICS server / client organisation

– Unification of control systems

• Remote access

– Monitoring

– Controls

• Configuration database

• Schedule


Organisation of Control Systems

• Original plan was for Daresbury Lab (DL) to provide all controls for the experiment

– DL responsible for many existing C&M systems (excellent quality)

• Unfortunately, recent funding issues have limited collaboration’s ability to pay DL for new work

– DL to continue with current projects (where possible)

– MICE community to take responsibility for additional C&M systems



• MICE Online Group (MOG) created in January

– Aim: Organise data acquisition, C&M & online reconstruction

• Controls & Monitoring Leader (JL)

– Identify control requirements for each section of MICE

– Decide on most appropriate solution

– Coordinate the effort of those involved in implementing agreed solution



• MOG directly responsible for C&M infrastructure

– Network/hardware organisation

– Integration of control systems (with each other & the rest of MICE)

– User experience (i.e. how operators interact with ‘global’ C&M system)

• For individual projects, each group within MICE should be responsible for own system(s)

– Contributing either EPICS development effort or funds for a 3rd party (e.g. DL) to complete required work

• Where necessary, MOG contributes developer effort

– However, very limited resources available (~1.5 man years per year)

– Currently seeking additional support within the community


EPICS Client / Server Overview

micetkpc1- AFEIIts- AFEIIt In frastructure

miceopi1 + miceopi2- O perator in terfaces

miceiocpc1- FN AL BPM s - N etwork S tatus- B -F ie ld P robes - C herenkov- D ATE S tatus

target1ctl- Beam Loss M onitors- Target C ontro ller

- C A G atew ay

miceioc1 + miceioc5- D ecay Solenoid + C ryo

miceioc4- Beam line M agnets

miceioc2 + miceioc3- Target D rive

vxW

orks

vxW

orks

vxW

orks

vxW

orks

Win

32 P

C

miceioc?- Spectrom eter Solenoid

miceioc?- H ydrogen System

miceioc?- R F System

miceioc?- Absorber Focus C oils

miceioc?- C avity C oupling C oils

miceiocpc2- TO F - EM R- D iffuser- KL C alorim eter

miceecserv- A ll c lient applications- A larm H andler - C hannel A rchiver

- C A G atew ay

TBD

EP

ICS

Ser

vers

('Har

d' IO

Cs,

'Sof

t' IO

Cs,

Por

tabl

e C

A S

erve

rs)

EP

ICS

Clie

nts

Controls Netw ork

DAQ Netw ork

Linu

x P

CLi

nux

PC

PPD Netw ork

To heplnw 17(RAL Gatew ay)

DATE Interface

Linu

x P

C

vxW

orks

vxW

orks

Linu

x P

C

vxW

orks

Linu

x O

RR

TE

MS

Linu

x O

RR

TE

MS

Linu

x P

C

Linu

x P

C

Linu

x P

C


EPICS Server / Client Organisation

• Wide variety of EPICS server applications permitted

– Typically connect to physical hardware

• Impossible to enforce common interface/processor/OS specifications

– Each server maintained by ‘owner’ of respective control system

• Strict central administration unnecessary – ‘end user’ only concerned with availability of PVs on network

• EPICS clients also varied, but must be uniformly accessible

– Users should not have difficulty finding/launching clients

– Applications should be consistently organised/updated

– MOG responsibility


EPICS Client Organisation

• All client-side applications run on miceecserv

– Central installation repository greatly simplifies configuration/maintenance/backup

– MOG collates individual applications, applies updates when available from control system ‘owners’

miceecserv miceopi1 miceopi2

EPICS IOC

Portable CA Server

Portable CA Server

EPICS IOC

EPICS IOC

Controls Network

EPICS server applications

EPICS client applications


EPICS Client Organisation

• Client control/monitoring GUIs viewed directly on miceecserv, or one of 2 ‘Operator Interface’ PCs

– OPI PCs act as ‘dumb terminals’, running displays from miceecserv via SSH

miceecserv miceopi1 miceopi2

EPICS IOC

Portable CA Server

Portable CA Server

EPICS IOC

EPICS IOC

Controls Network

EPICS server applications

EPICS client applications


Unification of Control Systems

• At user level: Simple ‘wrapper’ GUI provides menu for launching individual client applications

• At system level: Employ 2 standard EPICS tools (running as background services on miceecserv)

– Alarm Handler

• Monitors all servers & warns operators of abnormal/dangerous conditions

– Channel Archiver

• Automatically records PV parameters to disk & provides several visualisation options

• See P. Hanlet’s talk


User Interface

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser

Spectrom eter Solenoid

B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

H2 Absorbers Focus Coils

Hydrogen System

RF Cavities Coupling C oils

RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers


FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

User Interface

Large wall-mounted display

Alarm Handler

Message log

Any important parameters for

current run


User Interface

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

Client application launcher

Standard desktop monitor

Client GUI


FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

User Interface

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

Connected to miceecserv


FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

User Interface

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

FNAL BPM Sm all

Large

TO F 0

1

2

CKO V 1

2

Tracker 1 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos

KL Calorim eter

Tracker 2 D iffuser


B-F ie ld P robes

AFEIIts

AFEIIt C ryos


Hydrogen System


RF System

Drive

Contro ller

Beam Loss

Target

Beam line M agnets

Decay Solenoid

Absorbers

Connected to miceopi1

Connected to miceopi2


Remote Monitoring: General Principles

• Remote users should have simple, easily accessible interface for routine monitoring

• ‘Expert’ remote users should have access to monitoring displays which match those in MLCR

• No machine on Controls Network should be directly accessible over the internet

• System load generated by remote monitoring should have minimal impact on control & monitoring services


Remote Monitoring: Web Server

miceecserv

EPICS IOC

Portable CA Server

Portable CA Server

EPICS IOC

EPICS IOC

RAL Gateway

Channel Archiver

Web Server

PV Archive

Data Server

CGI Export

Controls Network

Java Archive

View

erW

eb browser

NFS Mount

PPD Network

Internet


Remote Monitoring: Direct PV Access

• Could recreate normal client displays using web interface, but would involve impractical development overheads

– Provide direct read only access to PVs so actual client GUIs may be run remotely

miceecserv

EPICS IOC

Portable CA Server

Portable CA Server

EPICS IOC

EPICS IOC

RAL Gateway

Controls Network

Standard client GUI running on remote

PC (read only)CA Gateway (read only)

CA Gateway (read only)


Remote Monitoring: Direct PV Access

• CA Gateway makes PVs available across subnets (with full access control), while minimising load on underlying servers

• To simplify end-user support, virtual machine disk image containing EPICS + all client applications will be made available

miceecserv

EPICS IOC

Portable CA Server

Portable CA Server

EPICS IOC

EPICS IOC

RAL Gateway

Controls Network

Standard client GUI running on remote

PC (read only)CA Gateway (read only)

CA Gateway (read only)


Remote Control

• Where possible, operations affecting the state of any MICE system should only be performed within MLCR

– Remote users accessing controls can lead to unknown/unexpected running conditions – should be discouraged

• If necessary, off-site experts will be permitted to run control client applications on miceecserv, via SSH through RAL Gateway

– Each expert will have an account on miceecserv which only contains client applications for their designated system


Configuration Database• Necessary to integrate control systems with central MICE

Configuration Database

1) Read set point values from database

2) Upload PV values to EPICS servers

3) Modify PVs with client GUIs

4) Download PV values from EPICS servers

5) Write new set point values to database

• For (2) & (4), could use standard EPICS Backup & Restore Tool (BURT)

– Backup/restore PV values to/from ‘snapshot’ files

• However, interfacing snapshot files with database introduces significant overheads

– Propose creation of custom backup/restore client



• Simple client application

– Read/write PV values via MICE C++ wrapper for CA C-bindings

• XML configuration file specifies PV names, correct sequence for write operations

– Import/export sets of PV values from/to XML string

– Read/write XML string from/to database via Configuration Database API

• Manual backup/restore

– State tagged with time, user-generated identification string, etc.

• Monitoring of DATE DAQ state

– Automatic backup at start of each run



• Additional requirements

– Throughout each DAQ run, all set point values should be held in state defined by the last Configuration Database ‘snapshot’

• If values change, system in unknown state

Cannot perform automated analysis of run data

– While DAQ in run state, client monitors all set point values

– If any parameters are modified

• Set PV to indicate invalid run state (read into DAQ stream)

• Set warning on Alarm Handler display



• Configuration Database interface still in early design stages – work not commenced

– J. Leaver/P. Hanlet to develop EPICS client

– D. Forrest to implement database API functions for parsing/formatting EPICS set point XML strings

• Details of run state PV monitoring to be confirmed


Infrastructure Schedule

Date post:	19-Dec-2015
Category:	Documents
View:	217 times
Download:	0 times

Controls and Monitoring Implementation Plan J. Leaver 03/06/2009.

Documents