Grid Computations without the ComputingElement: Interfacing Control Systemsfor On-Line Computations
Roberto Borghes, Roberto Pugliese, George Kourousias, Alessio Curri,Milan Prica, and Andrea Del Linz
Abstract Traditionally the computations on the Grid take place on the ComputingElement. In the same line, the Instrument Element is meant to Grid-enableinstrumentation. In this chapter we introduce a non-classical use of the InstrumentElement where it serves as a virtual instrument for performing a computational task.Specifically it has been used as the interface to a Control System that executesa series of High Throughput Computing tasks in an On-Line manner. This hadto be done in order to meet the special requirements of an application in theSynchrotron Radiation Facility Elettra. The instrument control in such institutionsis often done through Distributed Control Systems. Such a system is TANGOand the Synchrotron Radiation Facility (SRF) Elettra among other synchrotronsis heavily based on it. The application was for a beamline working in medicalimaging (SYRMEP) and aimed to be an improvement of an established ComputedTomography workflow. The task was the generation, in parallel, of sinograms of aspecific data format based on the acquired X-ray absorption data. The target was theavailability of the complete sinogram data set in a Storage Element by the time ofthe completion of the CT scan. The Grid related latencies, like job submission andqueuing, would have been an issue given the near-real-time requirements. Moreoverthe inclusion of a set of TANGO devices was necessary and a generic gLite WNwould not have been as suitable as a dedicated system. Besides the avoidanceof certain Grid parts, the Grid Security infrastructure was required to be fully
This works has been partially supported by the DORII EU project (European Commissionwithin the 7th Framework Programme (FP7/2007-2013) under grant agreement no. RI-213110)www.dorii.eu
The authors would like to thank Diego Dreossi, a senior SR X-ray CT and Medical Imagingspecialist working for the SYRMEP@Elettra beamline, for his valuable feedback and cooperationduring all the phases of the project.
R. Borghes • R. Pugliese (�) • G. Kourousias • A. Curri • M. Prica • A.D. LinzScientific Computing Team, IT Group, Sincrotrone Trieste S.C.p.A., Strada Statale 14 - km 163,5in AREA Science Park, 34149 Basovizza, Trieste, Italye-mail: author’[email protected]
utilised in the final solution. The design followed a bottom-up approach: (a) designand preparation of a dedicated system based on virtualisation, (b) developmentof a parallel sinogram generator, (c) deployment of suitable TANGO devices forcontrolling the data acquisition, the generator, and the On-Line progress, (d) aTANGO-to-IE bridge to export the devices as IM, (e) utilising a Grid Web Portal(VCR) in order to serve as the end-user GUI for the application. In this contribution(I) we introduce a novel concept where computation may take place outside theCE, (II) we design an architecture where a Distributed Control System is pilotedby an Instrument Manager through the Grid, and (III) we discuss a workingimplementation of the system.
1 Introduction
The domain of the application is that of scientific computing for a large multi-disciplinary Physics research establishment, a Synchrotron Radiation Facility [3].The proposed architecture has been realised to satisfy certain computational needsin the third generation Synchrotron Radiation Facility Elettra.1 Specifically thedeployment was for a beamline/laboratory that specialises in advanced ComputedTomography and Medical Imaging.
The SciComp team of Elettra has successfully utilised the Grid Computingparadigm [9] in various occasions [1, 19]. The power of the Grid has been proveduseful but it has various limitations [11] that will be discussed later in this chapter.In order to overcome certain Grid-related problems we utilise a new architecturethat is only-partially Grid based.
In general, the computation on the Grid is done on Worker Nodes (WN) ofa Computing Element (CE). There are various “logistics” in the process that themiddleware, in our case gLite [4], takes care of. It has been observed that there arequite high latencies during the job submission phase. These latencies for certainapplications are not a problem but for others they are. The proposed solution carriesthe computation in an atypical Grid manner while maintaining other Grid featureslike security and I/O to storage.
Elettra, like many other Synchrotrons, bases its control on a Distributed ControlSystem (DCS) [2]. This control core manages more than 10,000 devices, includingsensors, detectors, PSUs, and motors. The DCS of choice is TANGO [10]. The pro-posed solution assumes that a set of TANGO devices are part of the system.
In order to blend TANGO with the Grid technologies we used the InstrumentElement (IE) [5, 14], a middleware extension that allows instrument integration onthe Grid. The IE has been utilised in a non-standard way, so it does not control anactual instrument but a software-controlling TANGO device. This has been possibledue to the extensible nature of its architecture.
1URL:http://www.elettra.trieste.it, note that the establishment also includes a fourth generationlight source based on a free-electron laser, FERMI@Elettra.
In order to satisfy the specific needs of the application a suitable set ofVirtualisation [21] technologies was deployed. The actual imaging algorithm andapplication were designed to take advantage of this setup, yielding satisfactoryperformance. The user interaction with the application was of good standards sincethe final solution is based on a modern Grid-portal, the VCR [13].
2 The Application
2.1 Beamlines and Online Processing
A beamline (BL) in a Synchrotron is a kind of specialised laboratory. Many ofthe experiments that take place in such environments have specialised and oftenhigh computing needs. Such needs may include the design of novel algorithms,software, hardware, control, and user interaction. The SRF Elettra has 24 beamlinesproviding more than 105,000 h of user time to visiting and in-house scientists. Amode of operation that is in demand is that of “on-line” processing (Fig. 1), whichtakes place during the acquisition phase of the experiment. Due to long duration ofcertain experiments, like >4 h scans, this mode of operation can provide an earlyfeedback that allows the specialist to preliminarily evaluate the correctness of theexperiment’s setup.
2.2 Imaging for a Tomography Workflow
The requested application aimed to serve as an improvement to an establishedComputed Tomography (CT) [8] workflow for a Medical Imaging beamline. An es-tablished workflow implies that the beamline is already operating and functional:there are various stages of processing (including few Grid-based [6]), other sequen-tial other parallel, and data formats and instrumentation control is already in-place.
Fig. 1 ONLINE takes place during the experiment and ends with it, while OFFLINE starts whenthe dataset is complete. Where the situations permit, ONLINE is a preferred mode since it savestime
6 R. Borghes et al.
Fig. 2 Non-normalised X-ray absorption and the sinogram line that is contributing. PSGen isprocessing all the lines in parallel
This increased the challenge since the improvements had to be a transparent part thatwould integrate well and would yield advantages. An example technical challengewas the on-the-fly conversion of Lempel-Ziv-Welch [22] compressed 12-bit datasetsof integers to a RAW 16-bit format. The conversion may have not been an optimalengineering solution2 but had been strictly requested so that the I/O formats wouldbe exactly the same as prior to the introduction of our solution. This resulted in anundisturbed workflow where the data processing procedures, before and after thenew system, remained the same.
The mode of operation was identified from the early System Analysis stagesthat would be Online. Before our solution the specific process in the beamline wasoperating in an Offline mode. In Fig. 1, we see that this can provide a completeoutput dataset by the end of the experiment. Eventually, the online system that thispaper is describing saves 2 h per dataset; therefore, more data can be acquired for aspecific experiment.
The application had to collect X-ray Absorptions data during a CT scan andprocess and produce the corresponding Sinogram dataset (Fig. 2). The resulted ap-plication was PSGen, a Parallel Sinogram Generator. This through the architecturedescribed in the following sections has resulted in a rapid sub-second Absorptiondata processing including parallel I/O to multiple hundreds of Sinogram files, witha total I/O of approximately 5 GB per dataset; all in near-real-time.
3 The Distributed Control System
TANGO is an open source object oriented control system developed mainly forcontrolling Accelerators and experiments. It can be used for almost any kind ofhardware but deployment on a standard gLite WN may be difficult. The system isactively developed by a consortium of synchrotron radiation institutes: Alba, Desy,Elettra, ESRF, and Soleil.
2For large scientific datasets modern formats like HDF5 should be considered.
Grid Computations without the Computing Element 7
Fig. 3 There are many ways to control and monitor the TANGO devices. But even the popularGUI based tool JIVE would not be user-friendly enough for the end-user, the beamline scientist
TANGO uses the omniORB implementation of CORBA [7] as its networkprotocol in order to provide network access to hardware. Hardware can rangefrom single bits of digital input/output up to sophisticated detector systems orentire plant control systems. Hardware access is programmed in a process calleda Device Server. The device server implements device classes which implementthe hardware access. At runtime the device server creates devices which representlogical instances of hardware. The object model in TANGO supports methods, at-tributes and properties. The client-server communication model can be synchronous,asynchronous or event driven. Clients and Device Servers can be written in CCC,Java or Python. The choice for the devices of PSGen was CCC in order to achievethe optimal performance.
For the online-processing application, two TANGO device servers have been de-veloped: the Watchdog and the Worker. The main task of the Watchdog is signallingthe presence of new data to the Worker device. The Worker, after receiving thesignal, performs the data processing step. The system is highly configurable andprovides at all stages of the processing feedback and status information.
The devices have such an open and extensible architecture that can be deployed inother online-processing application with minimal effort. Example settings for sucha device include the Watchdog frequency, dataset size indicator, and self-stoppingparameters. In Fig. 3, a generic interaction is achieved by using JIVE, a graphicaltool for viewing and modifying the TANGO db and testing devices3.
3The GUI tool JIVE for TANGO we are referring to, should not be confused with the homonymEU FP7 project.
8 R. Borghes et al.
4 Where Computation Takes Place
The TANGO system that has been described in the previous section hasrequirements that the typical gLite Worker Node (WN) may not have or maybe impractically difficult to be satisfied. These requirements usually are OS relatedlike user privileges and kernel tuning, and network related like having inboundconnectivity. In addition to the TANGO requirements, PSGen itself requires ahardware and software setup that is not present in a general gLite CE deployment.These include custom and cutting-edge versions of libraries and Dynamic MemoryProvisioning. An additional design decision was to provide the application with adirect, fast and low latency access to the storage which at all times can be accessedsecurely through the GridFTP protocol [20].
The system we designed for deploying the application coupled with the TANGOdevices, is hosted in the Elettra Cloud Platform. This infrastructure is based on aVirtualisation solution, XEN Hypervisor. It provides a flexible way to add, remove,upgrade, and change resources like CPUs, memory, and Network connectivity.In the future, the application may see an increase to its computational requirements(i.e. as CCDs get larger) so scalability and upgradability is necessary [16]. The sys-tem, that we named “Dynamite”, has been configured in a paravirtualised mode andthe mixed PSGen-specific performance hit has been measured to be less than 4.2%.The current setup of Dynamite is a 64-bit OS (CentOS 5.4) with many upgradedlibraries. The 64-bit OS permits access to the 14GB of available memory, allowingmassive RAM caching and large array in-memory operations. The available CPUcores that can be dedicated are 10 but the setup can easily scale and be upgraded.The current performance satisfies the processing needs of PSGen resulting to thecalculation of an approximate 15:6 � 106 16-bit Integers per second.
An additional consideration of the system’s performance is that of the I/O. Thesystem does not use local storage at any stage. The input and output datasets arestored in a high performance Storage Area Network (SAN). The connection betweenDynamite and the front-end server of the SAN is 4Gb/s. The available storage forthe specific beamline/lab is 20TB. The overall performance, including calculationand I/O, has been fast and stable enough for near-real-time computation (Fig. 4).
Fig. 4 In theory, each PSGen cycle for a given dataset should be the same. The actual numbersconfirm the stability and constant performance of Dynamite, showing an insignificant variation.The initial cycles reflect cached I/O
Grid Computations without the Computing Element 9
5 The IE not for Instrument Control
The Instrument Element (IE) is an advanced Grid middleware extension that enablesthe inclusion of Instrumentation to the Grid. The exact details of it are beyondthe scope of this contribution but the reader may refer to the suitable literature forbetter understanding [14]. In general, it is a set of technologies that permit the easyinclusion of instruments like sensors, motors and cameras to a gLite infrastructure.The main objective is the remote operation [15, 18] and control of the instrumentthrough a secure channel by respecting the Grid security protocols.
The Instrument Element is an open source middleware implemented in Java asan Axis SOAP web service that runs in the Tomcat web server. To interface theactual instruments and sensors it uses the so-called Instrument Managers (IM).An IM is a suitably structured Java client code that communicates with the physicalinstrument or better, its control system. Instrument managers have to run insidethe IE framework which provides a number of common services. In particular, theauthentication and authorisation mechanism based on user credentials and VOMSroles accordingly to the GSI/gLite schema and a common interface to all instrumentsattached (Fig. 5).
The architecture of the IE is extensible and the scope of it may be stretchedbeyond its original objectives. Specifically, the IE can be seen as an IntegrationElement by exploiting the abstraction of the concept instrument [17, 19]. When theconcept is generalized it may include software application too; and in our case weassume as an instrument the underlying TANGO device server that manages PSGen.Thus, the full control of the PSGen application and its accompanying TANGO
Fig. 5 The Instrument Element architecture. It is an extensible and expandable model so it can beused for purposes beyond its original scope, like steering the TANGO devices that control PSGen
10 R. Borghes et al.
Fig. 6 The IE for TANGO Watchdog of PSGen can be controlled from a web portal through anautomatically generated graphical panel
devices can be interfaced and provided as an IE via an Instrument Manager. At thispoint, the IE controls a Computing entity, the PSGen application, without accessinga traditional CE. This irregular use of the IE demonstrates a case where it mayeven compute or, more precisely, steer a computation. Alternative Web Servicesbased technologies could be used as well, but other than the overhead of the newdevelopment, the main disadvantage would be the disconnection of the applicationfrom the rest of Grid infrastructure that it profits from (like the SE) (Fig. 6).
6 The User Interaction
The Scientific Computing team of the IT Group of the Synchrotron Elettra, for thepast years has been involved in R&D for technologies that aim to realise singlesing-on and Web based access to various resources. One of the latest products isthat of the VCR. The VCR was initially designed to serve as a portal for easy accessto Grid resources. During a project, DORII [12], the VCR was further developed toinclude Workflow Management, Remote Rendering (GVid), collaboration featuresand other.
The latest version of the VCR had to be suitably updated in order to accommodatethe novel hybrid approach for applications that was described in the previous
Grid Computations without the Computing Element 11
Fig. 7 The end-user, the beamline scientist, interacts with the application through a web browserin a simple manner
chapters. An updated Application Manager allowed an easy integration of theTANGO subsystem and rapid application deployment. PSGen, at this point a VCRApplication, can be accessed by the users that have access to portal, taking intoaccount most of the fine Grid-related security control, like Certificates, VOs, andthe VCR-side TAGS4. The PSGen user can start the application at the beginning ofthe experiment and receive feedback at any stage; even after logoff and re-login. Theinput and output datasets are stored in Storage Elements (SE) that can be browsedfrom the web portal. In addition to the above, the user may start other Applicationsthat make full use of the Grid in standard manner. These may include online andoffline, single, parametric, MPI, and interactive and any combination permitted fromthe gLite, the Common Library, and the rest of the technologies developed duringthe DORII project. Finally, the scientists may use features like the logbook fornote-taking purposes while using PSGen. Additional work is in progress in orderto allow richer GUI elements that will permit richer interaction with the remoteapplication (Fig. 7).
7 Conclusions
The Grid can be useful in many scenarios but there are cases where its limitationsare obvious. One such limitation is constituted by the various latencies. They usuallyoccur during the job submission phase. There is a large family of applications wherethese latencies are not a problem but there are other cases that can be limited fromthese. The presented work was for an application with near-real-time requirementsfor on-line processing in a Tomography lab of a Synchrotron. The presentedarchitecture and the related R&D demonstrated a hybrid approach where storage
4TAGS in the VCR domain can be seen as flags that allow a flexible association of a specific useror user-group with a set of applications.
12 R. Borghes et al.
(I/O on the SE), interaction (VCR portal) and security is done on the Grid while thecomputation is done in a non-CE based infrastructure controlled by an IE. Such ahybrid approach is novel in the field and in the future may be extended towards otherdirections.
References
1. R. Pugliese, M. Prica, G. Kourousias, A. Del Linz, A. Curri, Integrating Instruments in the Gridfor On-line and Off-line Processing in a Synchrotron Radiation Facility, COMPUTATIONALMETHODS IN SCIENCE AND TECHNOLOGY 15 (2009), no. 1, 21–30.
2. A. Butkovskiı, Distributed control systems, Elsevier Publishing Company, 1969.3. E. Koch, G. Marr, T.Sasaki, H Winick, Handbook on synchrotron radiation, (1983).4. E. Laure, S. Fisher, A. Frohner, C. Grandi, P. Kunszt, A. Krenek, O. Mulmo,F. Pacini, F. Prelz,
J. White, and others, Programming the grid with glite, Computational Methods in Science andTechnology 12 (2006), no. 1, 33–45.
5. E. Frizziero, M. Gulmini, F. Lelli, G. Maron, A. Oh, S. Orlando, A. Petrucci,S. Squizzato,S. Traldi, Instrument Element: a new Grid component that enables the control of remoteinstrumentation, doi:10.1109/CCGRID.2006.146), 2006.
6. F. Brun, G. Kourousias, D. Dreossi, L. Mancini, An improved method for ring artifactsremoving in reconstructed tomographic images, IFMBE World Congress on Medical Physicsand Biomedical Engineering, vol. 24, Springer, 2009, pp. 926–929.
7. M. Henning, The rise and fall of CORBA, Queue 4 (2006), no. 5, 34.8. G.T. Herman, Fundamentals of computerized tomography: Image reconstruction from projec-
tions, Springer Verlag, 2009.9. I. Foster, C. Kesselman, The grid: blueprint for a new computing infrastructure, Morgan
Kaufmann, 2004.10. J.M. Chaize, A. Gotz, W.D. Klotz, J. Meyer, M. Perez, E. Taurel, Tango-an Object oriented
control system based on CORBA, Proceedings of the International Conference on Acceleratorand Large Experimental Physics Control Systems (ICALEPS99), 2000.
11. Lovro Ilijasic, Computational Grids as Complex Networks, Master’s thesis, Scuola di Dottoratoin Scienza e Alta Tecnologia, 2010.
12. M. Płociennik, D. Adami, A. David, G. Barcelo,I. Coterillo, F. Davoli, P. Gamba, R. Keller,D. Kranzlmuller, I. Labotis, DORII–Deployment of Remote Instrumentation Infrastructure,Relation 10 (2009), no. 1.109, 800.
13. M. Prica, R. Pugliese, A. Del Linz, G. Kourousias, A. Curri, D. Favretto, F. Bonaccorso,An advanced web portal for accessing grid resources with virtual collaboration features, 5thEGEE User Forum (Uppsala, Sweden), EGEE User Forum, April 2010.
14. M. Prica, R. Pugliese, A. Del Linz, A. Curri, Adapting the instrument element to supporta remote instrumentation infrastructure, Remote Instrumentation and Virtual Laboratories:Service Architecture and Networking (2010), 11.
15. M. Prica, R. Pugliese, C. Scafuri, L.D. Cano, F. Asnicar, A. Curri, Remote operations of anaccelerator using the grid, Grid Enabled Remote Instrumentation (2009), 527–536.
16. R. Pugliese, G. Kourousias, A. Curri, A quantitative method for the projective approximationof computational requirements, Tech. Report 090500:01, Sincrotrone Elettra, 2009.
17. R. Pugliese, G. Kourousias, M. Prica, A. Del Linz, A. Curri, An infrastructure for theintegration of geoscience instruments and sensors on the grid, Geophysical Research Abstracts,vol. 11, EGU, 2009.
18. R. Pugliese, M. Prica, Remote operations of an accelerator using the grid, InternationalConference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS),WOPA02, 2007, pp. 303–306.
Grid Computations without the Computing Element 13
19. R. Pugliese, M. Prica, G. Kourousias, A. Del Linz, A. Curri, The grid as a softwareapplication provider in a synchrotron radiation facility, Remote Instrumentation Services onthe eInfrastructure, vol. X, Springer, 2009.
20. I.J. Taylor, From P2P to web services and grids: peers in a client/server world, Springer-VerlagNew York Inc, 2005.
21. T.L. Borden, J.P. Hennessy, J.W. Rymarczyk, Multiple operating systems on one processorcomplex, IBM Systems Journal 28 (1989), no. 1, 104–123.
22. T.A. Welch, A technique for high performance data compression, IEEE Computer 17 (1984),no. 6.
