EGEE-III INFSO-RI-222667
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite are registered trademarks EGEE-III INFSO-RI-222667
Astro-Wise and EGEE
Fokke DijkstraDonald Smits Center for Information TechnologyUniversity of Groningen, NLIn collaboration with OmegaCEN, Kapteyn institute, University of Groningen, NL
23 September - EGEE conference Istanbul
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Introduction
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Astro-Wise setup
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
TARGET diagram
Extreme data lineage!
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Astro-WISE: Services
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Lofar in short
• Large distributed radio telescope• In NL 2008: initial roll out 20 stations
– Full scale aperture synthesis array, extends 100 km
• Fibre network, Software Correlator• Run as a broad
common-user observatory– Making the transition now from
“project” to “observatory”
• Extended with European partners
• Up to 4 Petabytes/year to archive
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Grid userEGEE
BlueGene
Tier 0
Nearline storage & processing
Tier 1
New Tier 1 locationJülich - Germany
AmsterdamCIT / Tape
OracleDBExport by file
Meta data and Files
Tier 2
Astro-WISE view
CIT – Groningen
Lofar Communities
SASArchive
Codebase
Euro VO
Astro-WISE
Communities- Groningen- Amsterdam- Leiden- Nijmegen- Jülich
Global FS view
LoWise
C=IT/O=INFN /L=CNAF/CN=Pinco Palla/CN=proxy
VOMSpseudo-cert
C=IT/O=INFN /L=CNAF/CN=Pinco Palla/CN=proxy
VOMSpseudo-cert
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Astro-Wise and Grid
• User authentication– VOMS proxies– Proxy server– Preferably client installation
Python only! Web portal (should work)
– Astro-Wise system will obtain delegated proxy– Extend to Oracle database
• Job submission– Manage collection of tasks
• Storage– Support SRM
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Astro-Wise job submission
Astro-Wise
DatabaseAstro-Wise
user
tasks
DPU
Proxy server
credentials
credentialstasks
connects
•gLite UI
tasksLinux cluster
VOMS server
VO
MS
attr
ibut
es
connects
connects
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Problems encountered
• Preferrably no gLite-UI installation for all users• Currently only myproxy-init really necessary
– Python implementation needed OMII-UK (NDG) seems to have one, earlier tests unsuccessful
– Other option java webstart version CCLRC version works
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Job submission
• Image processing inherently parallel– e.g. Processing per single CCD image– One step requires global communication
Communication through database
• MPI job type used for processing– No real MPI program used– Independent processes started using ssh
Advantages Disadvantages
• Reduces the amount of datathat is shipped around
• Single job, easier management
• No problem with varying system speeds
• MPI support limited• Shared directory necessary
(currently)• Depends on password-less
ssh between WNs
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Future plans
• Investigate current state of python myproxy implementation as used by NDG
• Use of SRM for Lofar storage (StoRM on global file system)
• Further integration of GSI with Astro-Wise and Oracle• Allow Astro-Wise to store data through SRM
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Final remarks
• Grid job submission relatively easy• Credential management stumble block for users
– Obtaining certificate– Managing (proxy) certificates– Renewing certificates
• Lack of portable tools– Cannot ship current gLite UI to everyone– UI portals may be a solution
• Advantages of Grid– Standards!
Job submission Data management Security
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667
Acknowledgments
• Kor Begeman (implementation in Astro-Wise)• Willem Jan Vriend (implementation in Astro-Wise)• Danny Boxhoorn• Andrey Belikov
All from:OmegaCEN
Kapteyn Institute
University of Groningen, NL
