ACS at ANKA – Status 2007

Igor Kriznar,Klemen Zagar

ACS Workshop 2007

Contents

• Introduction of ANKA facility• Status of control system at ANKA• Upgrade to ACS 4.1.3• Technical details (separate presentation

with discussion)

Introduction to ANKA

• Located at FZK - Forschungszentrum Karlsruhe GmbH

• Run by ISS - Institut für Synchrotronstrahlung• Operational

since 2003• Research with

synchrotron radiation: Condensed matter, Nano and Micro Technologies, Actinide Research and Environmental Research, Synchrotron Technology and Instrumentation

The Machine

0 5 10 m

storage ring2,5 GeV/200 mA

booster500 MeV

microtron50 MeV

Beamlines at ANKA

FluoTopoFluorescence,Topography

WERA Soft X-ray Analytical Device

INE Actinide Research

SUL-X

MPI-MF Surface Diffraction

IR InfraRed-Spectroscopy

PX Protein-Crystallography

DIFF Diffraction

XAS Absorption

Lithography

The ANKA Storage Ring

Circumference: 110,4 m

Diameter: ca. 35 m

Electron energy: 2,5 GeV

Electron current: ca. 200 mA

Life time: > 20 h

ANKA Machine CS Datasheet

• 9 device server machines• 28 different device types (IDLs)• 43 device servers• 507 devices• more than 1500 data channels (BACI

properties)

ANKA Machine CS Diagram

Device Server Machines

• Windows XP machine with LonWorks interface card.– LonWorks drivers available only for

Windows• ACS device server (4.1.3 Java Container)• local copy of dll-s and configuration

database maintained with CVS• remote access with VNC• First microIOC was introduced with ACS on

embedded Linux SBC computer

ACS and microIOC at ANKA

• At the moment only ACS Component in C++ at ANKA

• Integrates one GPIB and one RS232 device to ANKA control system, used for FastFeedback system.

• Running on microIOC SBC computer with embedded Debian Linux.– ACS 4.1.3 reduced to 66MB

Java Clients - Simple

Java Clients - OrbitCorrection

• High level complex client

• Calculated physical model of electrons in storage ring

• Steers electron beam

• Connects to all BPM devices (96 BACI prop.) and PS (over 150 BACI properties) in storage ring

• MUST run reliably during beam operation or experiments get corrupted

Java Clients - PSTable

• PS devices in tabular form

• Heavy client, connects to up to 300 BACI properties at the time

• Runs daily

Plans at ANKA

• Small upgrades of Java clients• Introduction of new devices (e.g. insertion

devices) to ACS based machine control system

• Machine-non-related devices integrated with VPSS scada control system

• Gradually replace outdated devices on LonWorks and Windows with microIOC and Linux based device servers

ACS Upgrade to 4.1.3

• Upgraded from ACS 1.1 to 4.1.3 successfully finished in January 2006

• Central ACS services run on SuSE 10 from binary distribution without problems

• Device servers (Containers/Components) in Java on Windows machines, JNI used to access native C drivers

• Java clients kept same with minor adjustments (BACI API mature)

Upgrade Goals

• Successfully meet regarding performance and stability improvement

• Failed to stay with ACS HEAD, changes were necessary in ACS source code because of:– Java Container and Java BACI were in

prototype state– ACS was used in a way which was not

tested before– Some bugs in ACS

Upgrade Conclusions

• Definitely worth upgrading• Thanks to ACS team for support• Some of improvements were build into ACS

HEAD and released in later versions (e.g. jManager improvements)

• Some changes are still outside ACS HEAD branch (will be discussed at Technical Details presentation)

ACS at ANKA: Technical Details

• ACS 4.1.3 is used because it was last stable at the time of decision

• Java BACI and containers were used even though they were in prototype stage because there was no other choice (MS Windows constraint)

• ACS 4.1.3 was changed for use at ANKA. Changes are not big but we would like to be with HEAD

ACS at ANKA - Changes

• Changes due to missing implementation at Java Containers and Java BACI– Added persistence– Improved event dispatching

• Bugs in ACS (mostly integrated back to ACS)– jManager improved

• Changes in ACS design• Changes for improved operator's

experience with ACS

Added ACS Implementation at ANKA- Not in HEAD -

• jBACI

– BACIAction: ANKA also uses CB<type>::working(), which is not supported by jbaci;

– BACIDispatchAction

• DEFAULT_FAILURE_COUNT_LIMIT can be configured using JVM properties

• reportQueueSize prints warning log (if queue gets full)

• queue.clear() on failed (queue is cleared if dispatch fails - Matej needs to check if this is OK)

– alma.ACS.impl

• CharacteristicModelWrapper – added to allow to plugin other-than-CDB implementation of database access.

• Changes due to persistence added

• jCont

– ContainerServices – added support for persistent POA

– AcsCorba – persistence POA implementation added

Fixed ACS bugs – In HEAD -

• Manager– because of misconfigured threadpool

executor only one thread was working.• Acsjlog

– Could not be configured, when replaced with classes from ACS 5.0 it was better.

Operator's Experience Improvements - Not in HEAD -

• Operators want to see only important messages in logging console: when component is started, when is closed and errors. Logging level on INFO should provide this. Changes made:– Jcont: AcsContainer, ContainerServicesImpl– Acsjlog: ClientLogManager

• Startup script acsStartJava changed to local time (western European time). Should be settable from config files.

Improvements To Be Done

• BACIAction: implement like in C++• BACIDispatcher: POOL_THREADS should be

configurable using JVM property• Manager:

– thread pool size should be configurable– Ghost reference problem

Design Level Problems

• Changes in ACS distribution folders

– acsexmplPowerSupply.idl and acsexmplRampedPowerSupply.idl files were removed. PowerSupply and RampedPowerSupply are colliding with ANKA IDL names even that prefix and module name were different.

– AcsStartManager – when running -r option added to maciManagerJ (instead -n), to enable recovery by default. Should be settable from config files.

– acsStartJava - -Duser.timezone was changed to local time (western European time). Should be settable from config files.

• “cable-out” problem (described in next slide)

– in baci.idl all oneway declarations are removed, to callbacks at ANKA are twoway.

– Instead of 3 retries only 1 retry is set. alma.ACS.jbaci.BACIDispatchAction.DEFAULT_FAILURE_COUNT_LIMIT=1. After callback fails queue in alma.ACS.jbaci.BACIDispatchAction is cleared.

Cable-Out Problem Description

• If there is client connected to all components from one or several containers and if this client is misbehaving (connection is broken, network cable is unplugged, client computer's CPU is overloaded etc.) than this can crash container, because dispatching of monitor events to callbacks does not discard misbehaving client from the monitors.

– How to reproduce #1:• start container with about 20 components (PowerSupply kind of container will do)

• start client which connects to all this components and creates monitor listener (monitor are on 1 second update) to all BACI properties. Write arrived callbacks to console. (At ANKA we use Table application)

• start on different computer another such client which does the same.

• Unplug network cable from first client computer.

• Monitor callbacks will stop coming to both clients and will never resume. JacORB will crash on container computer after several minutes.

– How to reproduce #2: (after persistence for monitors was implemented):• start container as before

• start two client as before (one is enough, second is juts to check)

• shutdown container

• close one client

• start again container

• Monitors on not closed client will never recover, JacORB will crash the container.

Cable-Out Problem Analysis

• Callback working and done methods are oneway in IDL. Calling them in JacORB will never fail and dispatcher will never report failure to monitor. This way monitor will never remove filthy callback from dispatch queue. Call to filthy callback stays in JacORB or TCP/IP dispatch buffer for period of timeout. Because monitor does not know callback is filthy, it makes new dispatch call every second and puts to dispatching buffer per second more calls that are delivered. This eventually crashes JacORB, probably due to TCP/IP stack overflow. Could be that this is only problem of JacORB on Windows, but will have to be tested.

• Could be that this appears only on windows computer with JacORB CORBA (as it was observed). We did not try to test is on Linux or TAO CORBA.

• Partial fix was baci.idl: all oneway declarations are removed in order to become twoway.

• This does not remove problem altogether. We had to make ACS dispatching remove filthy callbacks faster still for containers with many components:

– Instead of 3 retries only 1 retry is set. alma.ACS.jbaci.BACIDispatchAction.DEFAULT_FAILURE_COUNT_LIMIT=1.

– After callback fails then queue in alma.ACS.jbaci.BACIDispatchAction is cleared.

• To be done

– Test if appears also on C++ TAO CORBA

– Find design solution or improvement

– Depends on size of a system: less probable on small or fast systems. Dispatching should be fine-tuned from configuration files for particular ACS installation or container.

Timeouts

• Changed JacORB socket factory so that its on connect delay is configurable:jacorb.net.socket_factory=anka.ACS.impl.CorbaSocketFactory

• Delay configurable with new property:jacorb.connection.client_connect_timeout = 3000

• Without this, the following scenario would last for a LONG time:– container crashes– before the container is restarted also the client (PSPanel)

crashes– container is restarted and starts to recover, but the client is

dead– so the connections timeout very slowly (it took "forever" for

PSPanel)

General Observations about ACS at ANKA- Developers Perspective -

• Creation of new device complicated, a lot of tasks (IDL, makefile, XML schema, etc.). A lot of code is copy&paste, could be generated.

• XML configuration human unfriendly

– Writing XML schema complicated as such, Simpler and less strict schema would be adequate

– Not possible to simply add arbitrary parameter to property or device configuration

– Folder structure too fragmented (e.g. device/device.xml)

– Missing option to set default device/property values for group or globally (with templates or redirects, possibly outside schema)

• Not possible to build only C++ or Java binaries (especially IDL compilation)

• Not supported creation of lightweight distribution with (Java) clients only or with particular component only

• More configuration possibilities for fine tunning (thread pools, timeouts, retries, dispatching)

• Simple things should be possible to be done simple way, like logging or config file editing (changing default introot or Java locations is a challenge)

General Observations about ACS at ANKA- Operators Perspective -

• Better control over logging and better log messages desired

• Persistence desired at all levels– Also restarting ACS CORBA central services

should recover manager connections (if manager computer is restarted all containers must be restarted: mostly tiresome)

• Possibility to set XML values for device/property group in one place

• Knowledge required on expert level for wide range of skills for simple maintenance. Operators are generally not programmers but technicians.

Ideas

• Support for EPICS, TINE or TANGO device drivers on DevIO level– Reuses wide range of their supported HW

• Building Java sources with Java tools– Maven can be nicely integrated with

introot concept• Wizard (in eclipse?) for creating new device

from IDL