K. HarrisonCERN, 25th September 2003

GANGA: GAUDI/ATHENA AND GRID ALLIANCE

- Project news- Ganga release 1- Work towards Ganga release 2- Interaction with other projects- Conclusions

25th September 2003 2

Project news (1)

- Ganga tutorial given at BNL (K.Harrison and C.L.Tan) Ganga session included in one-day software tutorial at US-Atlas Computing and Physics Meeting (BNL, 27-29 August) Tutorial attended by about 50 physicists Not able to cover everything in short-time available, but all participants successfully launched Ganga GUI, and were able to submit jobs to local LSF queue

- Ganga workshop held at LBNL, 1-6 September Contributors from US and UK brought together for extended discussions and some hands-on sessions Started along path towards Ganga release 2 With D.Adams, agreed first steps for Ganga-DIAL integration

25th September 2003 3

Project news (2)

- ITR (Information Technology Research) proposal from LBNL turned down by US National Science Foundation LBNL will be unable to maintain current high level of involvement in Ganga C.E.Tull has been a driving force in defining the project; hope that we don’t lose completely his expertise W.Lavrijsen will continue to work on AthASK and the Python bus, both directly relevant to Ganga

- Ganga presentation given by R.W.L.Jones at UK eScience All-Hands Meeting (Nottingham, 1-2 September) - Ganga Savannah Portal for bug reporting and tracking set up: https://savannah.cern.ch/projects/ganga

25th September 2003 4

Ganga release 1: availability

- Ganga release 1 was placed in Gaudi CVS at beginning of July - Several small modifications and improvements have been made, leading to minor releases (latest tagged version is Ganga-01-03-00)

- Code can be obtained using: cvs –d :pserver@lhcbcvs.cern.ch:/local/gaudicvs login [password: CERNuser] cvs –d :pserver@lhcbcvs.cern.ch:/local/gaudicvs co –P Ganga

- GUI is launched by executing: Ganga/GangaMain/src/Ganga.py (Requires that wxPython be available – no problems on lxplus)

25th September 2003 5

Ganga release 1: functionality

- Ganga release 1 includes: GUI Command-line access to underlying tools (but not user oriented) Job-options editor (optimised for Atlfast) Job submission to different batch systems (localhost, LSF, PBS) Mechanism for splitting/cloning jobs (usability to be improved) Job catalogue Automatic monitoring On-line help

25th September 2003 6

Ganga release 1: first usage

- Ganga has been installed and run at a number of sites: Birmingham, BNL, Cambridge, CERN, Imperial, Oxford, LBNL

- Ganga has been used to submit jobs for a variety of applications Atlfast, where Ganga specialisations help user Atlsim and Athena reconstruction, currently without Ganga specialisations Applications of LHCb and BaBar

- Test users have provided valuable feedback to developers

25th September 2003 7

Basic GUI

Job tree

Main panel

Toolbar

Pythoninterpreter

25th September 2003 8

Job creation

25th September 2003 9

Job-parameters panel

25th September 2003 10

Job-options editor: sequences

25th September 2003 11

Job-options editor: options

25th September 2003 12

Job submission

25th September 2003 13

Examination of job output

25th September 2003 14

Ganga release 2: refactorisation- Plans for Ganga release 2 include a refactorisation Improve breakdown into components, to simplify the integration in Ganga of external components, and the use externally of Ganga components Collect information that may be used by Ganga in a layer of databases or database-like structures Ensure clear mapping onto services identified by LCG RTAG11 (ARDA)

- Python bus will be introduced to mediate communication between components, as opposed to simply using the Python interpreter Improve configurability: bus can determine component to load on basis of functionality requested Prototype (PyBus) has been written (W.Lavrijsen)

25th September 2003 15

Ganga release 2: refactorisation diagram

25th September 2003 16

Ganga release 2: other improvements

- High priority will be given to user-oriented CLI- Unit testing will be introduced Help ensure quality of software- Interfaces to Magda and AMI will be added Python bindings have been written (C.L.Tan)- Job-options handling will be more sophisticated Editor will be more generic; options validator is to be added- Remote job submission will be enabled First tests of procedures for issuing commands remotely from Python have been carried out (K.Harrison,W.Lavrijsen)- Ganga will become more Grid-oriented Submission to EDG is already possible in principle, but procedure needs to be checked; submission to NorduGrid is being added (A.Soroko)

25th September 2003 17

Interaction with other projects (1)- AthASK W.Lavrijsen will be migrating most (all?) AthASK functionality into PyBus components that can be harvested by Ganga

- DIAL DIAL, with focus on interactive analysis, and Ganga, with focus on batch processing, are complementary (even if divide between interactive and batch processing is hazy) First steps for Ganga-DIAL integration agreed with D.Adams Idea is to wrap C++ classes of DIAL using SWIG or Boost.Python, and so make them available from Ganga User will be able to use DIAL directly, or will be able to use it transparently from Ganga

25th September 2003 18

Interaction with other projects (2)- Pacman Meeting on installation and packaging held this morning Will start looking at ways to take advantage of Pacman, in particular for packaging user code on the client side and unpackaging the code at a remote (Grid) site

- AtCom and DC2 C.L.Tan is a member of DC2 Task Force; other Ganga people also following developments Role Ganga might play in DC2 still to be defined: Production request tool? Production definition tool? Supervisor? Other?

25th September 2003 19

Conclusions

- A lot of work has been done on Ganga since the May software workshop, with a strong emphasis on dealing with Atlas-specific requirements- Ganga release 1 is available from Gaudi CVS repository This isn’t production quality, but gives a feeling of how things should work- Ganga has been used to run Atlsim, Atlfast and Athena reconstruction at both European and US sites- Work has started on interfacing Ganga to other Atlas tools: AMI, DIAL, Magda, Pacman- Good progress is being made towards Ganga release 2 (refactorised design, remote submission, other improvements)