EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 1IDM UID: YCA7A8
Configuration and Deployment of
production-level services for
CODAC operation at ITER
A.Bustos, B.Bauvir, R.Lange
ITER Organization
Disclaimer: The views and opinions expressed herein do not necessarily reflect those of the ITER Organization
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 2IDM UID: YCA7A8
ICH/
ECHCooling water
Control building
Backup control
building
Electrical
Coil power supplies Tokamak
complex
Cryoplant
Neutral beamHot cell
Infrastructure: Systems Requiring Control
Motivation
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 3IDM UID: YCA7A8
Architecture Recap: ITER Control System Architecture
Motivation
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 4IDM UID: YCA7A8
• Breakdown in 18 ITER control groups (CBS level 1) covering 28 PBS
• An ITER control group contains many Plant System I&C, total no. 171 (CBS level 2)
• A Plant System I&C is a deliverable from a procurement arrangement (IN-KIND)
• A procurement arrangement delivers a part, one or many Plant System I&C
CB
S 1
CB
S =
Co
ntr
ol B
reakd
ow
n S
tru
ctu
re
CB
S 2
Architecture Recap
Motivation
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 5IDM UID: YCA7A8
Motivation
Most of ITER Applications are built outside ITER (ITER shall provide standardization)
171 Plant System I&C to be integrated
• Follow STRICT HW and SW standards
• Follow STRICT Software Configuration Control
• minimal human intervention in deployments
• capability for disaster recovery
• capability to inspect configuration version in production
• capability for reinstallation of a known/verified configuration
• Follow STRICT deployment procedures and reduced numbers of actors involved
• Support incremental changes during integration and test
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 6IDM UID: YCA7A8
Scope
CODAC provides centralized
services to Plant Systems during
installation, testing, integration and
operation.
• CODAC Services– BOY Operator Interface
– BEAST Alarm System
– BEAUTY Archiving System
– OLOG Electronic Logbook
• CODAC Machines– Infra. Servers and Switches
– Operator Terminals
– Central Servers and DB
– Fast Controllers
– Plant System Hosts
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 7IDM UID: YCA7A8
• The selected operating system is Red Hat Enterprise Linux v7.4
for the x86-64 architecture (RHEL x86_64) and, optionally, real-
time extension MRG-R
• The infrastructure layer is EPICS v7.02 (Experimental Physics and
Industrial Control System) @ CCSv6.1
• The CODAC services layer is Eclipse based Control System
Studio v4.6.2 including HMI, alarming, archiving etc.
• ITER specific software such as configuration (self-description data),
state handling, drivers, networking, etc.
• Distribution over Internet to registered user organization
Integration strategy – CODAC Core System
CCS Technologies
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 8IDM UID: YCA7A8
I&C Project Structure
Software project structure adheres to Maven recommended structure.src/main/epics is allocated for EPICS projects
{unit_name}
├── pom.xml (project information)
├── doc/ (Directory for documentation)
└── src/ (Directory for source files)
├── main/ (Directory for source files of product)
│ ├── beast/ (CSS alarm configuration files)
│ ├── beauty/ (CSS archive configuration files)
│ ├── boy/ (CSS boy opi files)
│ ├── c++/ (C++ application sources)
│ ├── databrowser/ (databrowser files)
│ ├── epics/ (EPICS application sources)
│ ├── java/ (Java application/library sources)
│ ├── plc/ (PLC program sources)
│ ├── python/ (Python application/library sources)
│ ├── resource/ (application/library resources)
│ └── scan/ (CSS scan command files)
├── test/ (Directory for source files of test)
│ ├── ...
└── target/ (Build results)
├── ...
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 9IDM UID: YCA7A8
CCS Approach to Packaging
Red Hat Linux as a base platform
RPM as a packaging tool, with YUM as a distribution mechanism
Build system is controlled by Apache Maven (clean, compile, …)
Maven is also used to automate frequent I&C tasks ( start / test / stop
sequence).
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 10IDM UID: YCA7A8
Software Units and RPM packages
deployment Artifacts
Unit
«deployment spec»
RedHat Package
Management RPM
component
«deployment spec»
Application package
«deployment spec»
CODAC Core System
package
«deployment spec»
Host Role
«deployment spec»
Host meta-package
Serv ers::
CODAC Serv er
Project Object
Model (POM)
Aplication
Sources
Aplication
ConfigurationAplication
Binaries
1..*
1
1..* 1
1
0..*
1
0..*
1 1
1..*
1
• One-to-One CODAC Server - Host Metapackage
• Host metapackage pulls all required CODAC Core System RPMs and Application RPMs
• Extra configuration of Server and Services contained in metapackage’s deployment
scripts
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 11IDM UID: YCA7A8
SW Configuration Control Workflow
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 12IDM UID: YCA7A8
Packaging I&C Integration Modulescmp Configuration Model
Serv ers::Central
Version Control
Repository
Serv ers::Build
Serv er
Serv ers::
Satellite Serv er
Configuration RO
(from
Actors)
Integration RO
(from
Actors)
Supplier
(from
Actors)
«deployment spec»
Artifacts::RedHat Package
Management (RPM)
component
Unit
+ Application Configuration
+ Application Sources
+ Project Object Model (POM)
(from Artifacts)
Generate deployable artifacts
from version controlled sources
host
deliver
«use»
develope
branch
import
use
«use»tag
hostinstantiate
own
• Fix Nomenclature on the generated RPM
<<codac_version>>+<<module_name>>+<<module_version>>+<<build_date>>+<<svn_ci number>>
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 13IDM UID: YCA7A8
Deployment Artifacts
deployment Deployment Model CODAC INTEGRATION
Integration RO
(from
Actors)
«deployment spec»
Artifacts::RedHat Package
Management (RPM)
component
Serv ers::
CODAC Serv erServ ers::
Satellite Serv er
Serv ers::
Ansible
Configuration
Management
DB
use
«use»install
host
verify
use
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 14IDM UID: YCA7A8
ITER Control System Model
Deployment Modules: Pre-Production
Validate CODAC Scalability
Provide isolate test environment
Provide Simulators for testing CODAC Tech. & PS I&C Modules
Provide environment to Operator Training
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 15IDM UID: YCA7A8
Archive4501AS-SRV-3601
Alarm4501AS-SRV-3602
Logbook4501AS-SRV-3603
SUP4501AS-SRV-3604
ArchiveDB4501DS-SRV-3601
AlarmDB4501DS-SRV-3602
LogbookDB4501DS-SRV-3603
Archive4503BU-SRV-3601
Alarm4503BU-SRV-3602
CA Gateway4503BU-SRV-3603
SUP4503BU-SRV-3604
Archive4503UT-SRV-3601
Alarm4503UT-SRV-3602
CA Gateway4503UT-SRV-3603
SUP4503UT-SRV-3604
IOC CUBM
IOC CORE
IOC SYSM
Workstation4501WS-CC-3601
ArchiveDB4501DS-SRV-3601-XArchiveDB
4501DS-SRV-3601-X
AlarmDB4501DS-SRV-3602-XAlarmDB
4501DS-SRV-3602-X
LogbookDB4501DS-SRV-3603-XLogbookDB
4501DS-SRV-3603-X
Workstation4501WS-CC-3602
PSH6336BS-PSH-0001
PSH43CC00-PSH-0001
BUIL UTIL
CTRL
XPOZ EXT
XPOZ INT
B72/B07
B36-L1-02
ArchiveDB4501DS-SRV-3601-RArchiveDB
4501DS-SRV-3601-R
LogbookDB4501DS-SRV-3603-RLogbookDB
4501DS-SRV-3603-R
AlarmDB4501DS-SRV-3602-RAlarmDB
4501DS-SRV-3602-R
CMN
PLC develop4501AS-SRV-3606
CCS develop4501AS-SRV-3605
B36-L1-07 Cubicle 450536-CU-0002
DataWrapper4501DS-SRV-1003-X
WebServer4501AS-SRV-0003-X
CA Gateway450300-SRV-1000
UDA Serverio-ls-udafe01
IT Shared Filesystem
B72
TCR VM 4501WS-CC-7201
DataWrapper4501DS-SRV-1002-X
DataWrapper4501DS-SRV-1002-X
PON Replicator4501DS-SRV-0011
NX
IEC 61850
WinCC-OAGateway
CUBM PLC450536-PLC-0002CUBM PLC
450536-PLC-0002
WinCCLocal HMI
PSH43CCLV-PSH-0001
UTIL-HVUTIL-LV2BUIL-B36
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
UTIL-LV2 PLC43CCLV-PLC-2000
BUIL-B36 PLC6336BS-PLC-0001
Profinet
HW
RIORIO Gateway
Profinet
HW Jbus RS422/SerialRS232/BACnet/Dynet
IEDIEDFielddevice
IEDIEDFielddevice
Modbus TCP Modbus TCP
WinCC-OALocal HMI
IEDIEDFielddevice
IEDIEDFielddevice
B72
Workstation4501WS-CC-7202
Workstation4501WS-CC-7201
Workstation4501WS-CC-3603
Deployment: Production TCRs
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 16IDM UID: YCA7A8
Archive4501AS-SRV-3601
Alarm4501AS-SRV-3602
Logbook4501AS-SRV-3603
SUP4501AS-SRV-3604
ArchiveDB4501DS-SRV-3601
AlarmDB4501DS-SRV-3602
LogbookDB4501DS-SRV-3603
Archive4503BU-SRV-3601
Alarm4503BU-SRV-3602
CA Gateway4503BU-SRV-3603
SUP4503BU-SRV-3604
Archive4503UT-SRV-3601
Alarm4503UT-SRV-3602
CA Gateway4503UT-SRV-3603
SUP4503UT-SRV-3604
IOC CUBM
IOC CORE
IOC SYSM
Workstation4501WS-CC-3601
ArchiveDB4501DS-SRV-3601-XArchiveDB
4501DS-SRV-3601-X
AlarmDB4501DS-SRV-3602-XAlarmDB
4501DS-SRV-3602-X
LogbookDB4501DS-SRV-3603-XLogbookDB
4501DS-SRV-3603-X
Workstation4501WS-CC-3602
PSH6336BS-PSH-0001
PSH43CC00-PSH-0001
BUIL UTIL
CTRL
XPOZ EXT
XPOZ INT
B72/B07
B36-L1-02
ArchiveDB4501DS-SRV-3601-RArchiveDB
4501DS-SRV-3601-R
LogbookDB4501DS-SRV-3603-RLogbookDB
4501DS-SRV-3603-R
AlarmDB4501DS-SRV-3602-RAlarmDB
4501DS-SRV-3602-R
CMN
PLC develop4501AS-SRV-3606
CCS develop4501AS-SRV-3605
B36-L1-07 Cubicle 450536-CU-0002
DataWrapper4501DS-SRV-1003-X
WebServer4501AS-SRV-0003-X
CA Gateway450300-SRV-1000
UDA Serverio-ls-udafe01
IT Shared Filesystem
B72
TCR VM 4501WS-CC-7201
DataWrapper4501DS-SRV-1002-X
DataWrapper4501DS-SRV-1002-X
PON Replicator4501DS-SRV-0011
NX
IEC 61850
WinCC-OAGateway
CUBM PLC450536-PLC-0002CUBM PLC
450536-PLC-0002
WinCCLocal HMI
PSH43CCLV-PSH-0001
UTIL-HVUTIL-LV2BUIL-B36
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
IEDIEDIEDIEDIED
UTIL-LV2 PLC43CCLV-PLC-2000
BUIL-B36 PLC6336BS-PLC-0001
Profinet
HW
RIORIO Gateway
Profinet
HW Jbus RS422/SerialRS232/BACnet/Dynet
IEDIEDFielddevice
IEDIEDFielddevice
Modbus TCP Modbus TCP
WinCC-OALocal HMI
IEDIEDFielddevice
IEDIEDFielddevice
B72
Workstation4501WS-CC-7202
Workstation4501WS-CC-7201
Workstation4501WS-CC-3603
Deployment: Production TCRs
i.e. New Version of PS I&C module
for UTIL-HV-S400!
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 17IDM UID: YCA7A8
Conclusion
CCS is based on fix releases (patch or new release needed
for updates) vs PS I&C SW Support incremental changes
during integration and test.
Deployment in production only configuration control
versioned resources
Support incremental changes (~30 versions of m-UTIL-HV-S400/ integrated in 7 months)
11 TCRs in 6 Buildings to come
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 18IDM UID: YCA7A8
EXTRA Support Slidescourtesy Denis Stepanov
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 19IDM UID: YCA7A8
Packaging Instructions for IOCspom.xml snippet:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-iter-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<packaging>
<package name="TEST-S7-PSH0CORE-ioc">
<include type="ioc" name="TEST-S7-PSH0CORE" runlevels=“345" />
</package>
<package name="TEST-S7-PSH0PLC-ioc">
<include type="ioc" name="TEST-S7-PSH0PLC" runlevels=“345" />
</package>
<package name="TEST-S7-PSH0SYSM-ioc">
<include type="ioc" name="TEST-S7-PSH0SYSM" runlevels=“345" />
</package>
<package name="ioc">
<dependency version="current">%{codac_rpm_prefix}-${project.artifactId}-TEST-S7-PSH0CORE-ioc</dependency>
<dependency version="current">%{codac_rpm_prefix}-${project.artifactId}-TEST-S7-PSH0PLC-ioc</dependency>
<dependency version="current">%{codac_rpm_prefix}-${project.artifactId}-TEST-S7-PSH0SYSM-ioc</dependency>
<include type="script" scriptType="initd" file="plc-sample-iocs" />
</package>
</packaging>
</configuration>
</plugin>
</plugins>
</build>
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 20IDM UID: YCA7A8
EPICS Base Packaging
ITER made a substantial effort to package EPICS itself in
accordance with Red Hat Linux practices. This results in
approx. 90 RPMs covering the following areas:
- EPICS base
- EPICS extensions and tools
- EPICS device support modules
Whenever possible,
runtime and devel
packages are separated.
ITER-specific
EPICS packages
are clearly
separated too.
EPICS Collaboration Meeting, ITER HQ, France, 3-7 June 2019
© 2019, ITER Organization
Page 21IDM UID: YCA7A8
EPICS and CODAC profiles
It is possible to have several CODAC versions installed on
one machine, hence, several EPICS versions too.
<meta name="ct">
<summary>CODAC Core System ${codac.version} CODAC
Terminal</summary>
<provides codac="true">system-role</provides>
<!-- client/user applications -->
<requires codac="true">css</requires>
<requires codac="true">css-jms-send</requires>
<requires codac="true">css-opi</requires>
<requires codac="true">epics</requires>
<requires codac="true">python-cachannel</requires>
<requires codac="true">python-pyepics</requires>
<requires codac="true">system-tests</requires>
<requires codac="true">psps-fatsat-editor</requires>
</meta>
<meta name="pcf">
<summary>CODAC Core System ${codac.version} PCF
Operation</summary>
<provides codac="true">system-role</provides>
...
<!-- epics modules -->
<requires codac="true">epics-irio</requires>
<requires codac="true">epics-mcoreutils</requires>
<requires codac="true">epics-nisync</requires>
<requires codac="true">epics-nisync-general-time</requires>
<requires codac="true">epics-pxi6259</requires>
<requires codac="true">epics-pxie6368</requires>
<requires codac="true">epics-pxi6528</requires>
<!-- libraries -->
<requires codac="true">irio</requires>
<requires codac="true">dan-daq</requires>
<requires codac="true">log-lib</requires>
...
</meta>
CODAC has profiles for different types of control system
machines, which dictate which part of EPICS has to be
installed