Ganga Developments

Karl Harrison(University of Cambridge)

18th GridPP MeetingUniversity of Glasgow, 20th-21st March 2007

http://cern.ch/ganga

20 March 2007 2/15

Ganga basics• Depending on context, Ganga can be any of:

(A) a Hindu goddess

(B) an hallucinogenic drug

(C) a job-management framework (Gaudi/Athena and Grid Alliance)that simplifies running jobs on the Grid

• Anyone expecting a presentation on (A) or (B) is probably at the wrong meeting

20 March 2007 3/15

Ganga releases• Ganga has well-developed release procedure, with built-

in quality checks(1) Code committed to CVSby package maintainer(s)

(2) Packages are tagged for release, with changes documented

(3) Suite of over 100 tests is run,with results available on web (4) Release made, using purpose-written tool

• Ganga 4.2.0 released October 2006, with subsequent minor releases• Ganga 4.3.0 at beta testing phase, with public release foreseen for April

20 March 2007 4/15

Core developmentRemote repository

• Job information stored onremote server• Certificate-based authentication

Easier access to job files

• Locating job files in Ganga workspacecan quickly become tiresome • File access simplified by peek() method

• j.peek( “stdout” )

• Validity of Grid proxy and AFStoken checked in monitoring loop • User prompted to renewcredential when validity fallsbelow configurable minimum

Monitoringloop

Credential monitoring

Usage monitoring

• Information sent to MonALISArepository every time Gangais started

20 March 2007 5/15

Developments for LHCb

• Logical file names for input data can be used for jobs run on local batch system as well as for Grid jobs– Run exactly the same job locally and on the Grid

• Added possibility of selecting input data using dataset browser

• New type of job splitting implemented, allowing creation of subjobs that differ from one another in terms of one or more job option

• Introduced possibility to set CPU limit for DIRAC jobs– Take advantage of short queues not used by production jobs

• Migrated to new system for setting environment for LHCb applications

• Example Ganga scripts for running LHCb applications now included in LHCb software releases

20 March 2007 6/15

Developments for ATLAS

• Support provided for both legacy and current datasets– Latest (DQ2) datasets can be used both for jobs run on local batch system and for Grid jobs

• Introduced possibility of fast preselections based on TAG analysis– Also have first version of interface to TAG Navigation Tool

• User allowed to choose between shipping binaries and compiling code at worker node

• Added job monitoring based on ARDA dashboard technology• Extended range of backends for ATLAS applications:

NorduGrid and PANDA for analysis, local resources for user production

• Introduction to Ganga included in ATLAS Workbook– Workbook is main point of entry for physicists wanting to learn to use ATLAS software

20 March 2007 7/15

Applications and backends for ATLAS and LHCb

PBS OSG NorduGridLocal LSF PANDA

US-ATLAS WMS

LHCb WMS

ExecutableAthena

(Simulation/Digitisation/Reconstruction/Analysis)

AthenaMC(Production)

Gauss/Boole/Brunel/DaVinci(Simulation/Digitisation/Reconstruction/Analysis)

LHCb Experiment neutral ATLAS

Available in Ganga 4.2

Work in progress

New in Ganga 4.3

20 March 2007 8/15

LHCb Ganga Tutorials

• 3rd LHCb-UK Software Course held in Edinbugh, January 2007• Like previous editions, course provided overview of LHCb

applications and running them on the Grid, with focus on physics analysis– Very successful approach

• Ganga, largely a demo tool in 1st LHCb-UK Software Course, is now a key component– Participants were introduced to Ganga on first day, then used it throughout the course to build code and run jobs

• Online LHCb analysis tutorial now based on use of Ganga

3rd LHCb-UK Software Course, Edinburgh, January 2007

20 March 2007 9/15

ATLAS Ganga Tutorials

• Upsurge of interest in Ganga in ATLAS, following good publicity in presentation (J.Catmore) on use of Ganga by B-physics group

• Six ATLAS Distributed Analysis Tutorials held since September– Milan (September 2006, February 2007), CERN (October 2006), Mumbai (December 2006), Edinburgh (February 2007), Lyon (March 2007)

• Ganga is main focus, but tutorials also introduce DQ2 data-management system, metadata catalogue and TAG navigation tool

ATLAS Distributed Analysis Tutorial, Edinburgh, February 2007

20 March 2007 10/15

User survey• User survey carried out in December 2006, with

responses from 27 physicists: 12 from ATLAS, 15 from LHCb

User types

Experienced user46%

Novice user27%

Non user27%

Experienced user50%

Novice user50%

Non user0%

ATLAS

LHCb

DIRAC60% Condor

30%LSF10%

LCG92%

Local8%

ATLAS

Backends used

Mode of use

ATLAS LHCb

LHCb

Country distribution

20 March 2007 11/15

Feedback from user survey

Favourite features: top 4

Most-wanted features: top 3

Documentation and user support

• Job splitting• Ease of use• Data management• Monitoring

ATLAS

• General job handling• Job splitting• Grid/Local switching• All functionality

LHCb

• Enhanced/easier data management• Faster response• Job resubmission

ATLAS

• Output merging• Control over output destination• Additional applications

LHCb

• Web-based documentation andtutorials good for getting started• Useful to have more examples and FAQ• High level of support from developers(locally or via e-mail) very muchappreciated

20 March 2007 12/15

Usage monitoring• MonALISA monitoring shows more than 300 Ganga users

since beginning of year• Typically have 40-50 users per work day (peaks for

tutorials)

20 March 2007 13/15

Ganga beyond ATLAS and LHCb: Geant 4

• Regression tests run for Geant 4 major releases - twice per year– Search for differences in calorimeterobservables

– Need a few years of CPU, concentratedin short period

• Geant 4 team now runs these tests on the Grid, using Ganga in conjunction with the DIANE job-distribution framework

20 March 2007 14/15

Ganga beyond ATLAS and LHCb: Cambridge Ontology

• Cambridge Ontology is developingsolutions for image retrieval based onanalysis of image content– combines state-of-the-art computervision with machine learning,natural-language processing, andinformation-retrieval methods

• Large amounts of CPU time needed for image processing

• Mini-PIPSS project set up to enable processing to be done on the Grid

• Ganga is used for the job submission

- Specialised Ganga components for Cambridge Ontology applications

20 March 2007 15/15

Conclusions

• Continued excellent progress with core and experiment-specific Ganga developments

• Number of successful tutorials held over past six months– Demand for tutorials especially high in ATLAS

• Feedback from user survey has been very positive• MonALISA monitoring shows that Ganga has had more than

300 users since the beginning of the year• Ganga used to good effect in applications beyond ATLAS

and LHCb– Geant 4 regression tests, Cambridge Ontology image processing

• Clouds on the horizon with anticipated loss of members of Ganga development team at end of GridPP2– Lose vital expertise just when LHC is about to start taking data and user support is essential