Post on 15-Jan-2016
description
transcript
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite are registered trademarks
Towards Seamless Grid ComputingThe EGEE Experience on Interoperable Grid Infrastructures
Erwin Laure
EGEE-II Technical Director
Erwin.Laure@cern.ch
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 2
eScience
• Science is becoming increasingly digital, needs to deal with increasing amounts of data and computational needs
• Simulations get ever more detailed– Nanotechnology – design of new materials from
the molecular scale– Modelling and predicting complex systems
(weather forecasting, river floods, earthquake)– Decoding the human genome
• Experimental Science uses ever moresophisticated sensors to make precisemeasurements Need high statistics Huge amounts of data Serves user communities around the world
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 3
Scientific trends
Scientific advances are more and more based on simulations using “virtual laboratories”
Ulf Dahlsten, Former Director Emerging Technologies and Infrastructures, EU, predicts that “in five years 80 percent of all scientific papers in all areas will be made in virtual laboratories. Fifty percent of social science documents will go the same way in five to ten years.”
The size of data an organization owns, manages, and depends on is dramatically increasing:–Ownership cost of storage capacity goes down–Data generated and consumed goes up–Network capacity goes up–Distributed computing technology matures and is more widely adopted
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 4
EGEE
Main Objectives• Operate a large-scale,
production quality grid infrastructure for e-Science
• Attract new resources and users from industry as wellas sciences
– Flagship grid infrastructure project co-funded by the European Commission
– Now in 2nd phase with 91 partners in 32 countries
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 5
EGEE – What do we deliver?• Infrastructure operation
– Sites distributed across many countries Large quantity of CPUs and storage Continuous monitoring of grid services & automated site
configuration/management Support multiple Virtual Organisations from diverse
research disciplines
• Middleware– Production quality middleware distributed under
business friendly open source licence Implements a service-oriented architecture that virtualises
resources
Adheres to recommendations on web service inter-operability and evolving towards emerging standards
• User Support - Managed process from first contact through to production usage– Training– Expertise in grid-enabling applications– Online helpdesk– Networking events (User Forum, Conferences etc.)
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 6
250 sites48 countries50,000 CPUs13 PetaBytes>5000 users>200 VOs>140,000 jobs/day
ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…
32%
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 7
Users and resources distribution
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 8
EGEE Grid Management Structure
• Operations Coordination Centre (OCC)
– management, oversight of all operational and support activities
• Regional Operations Centres (ROC)
– providing the core of the support infrastructure, each supporting a number of resource centres within its region
– Grid Operator on Duty
• Resource centres – providing resources (computing,
storage, network, etc.);
• Grid User Support (GGUS)
– At FZK, coordination and management of user support, single point of contact for users
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 9
Example: GridMap Monitoring Visualization
9
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 10
Registered Collaborating Projects
Applicationsimproved services for academia,
industry and the public
Support Actionskey complementary functions
Infrastructuresgeographical or thematic coverage
25 projects have registered as of September 2007: web page
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 11
EGEE working with related infrastructure projects
GIN
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008
LHC Use of Multiple Grid Infrastructures
12
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008
Why is Interoperability difficult?
• Grid infrastructures use different technologies– And even if same technologies are used they are usually heavily
customized
• Only a few widely adopted standards– gridFTP, X.509 (but used differently!)– Prototypes: BES, JSDL, …
Production Grids are difficult to change – adopting standards takes time Standards need to be stable before adoption
• Apart from technological differences, access policies also differ– Dialog among major Grid infrastructure providers started at last OGF22.
• Strong interactions between infrastructures and application community needed– HEP was driving interop efforts for LHC– Other applications can build on these experiences
13
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 14
Access to Compute Resources
PBS/Torque
LSF
Condor
Load Leveler
Sun Grid Engine
ARCCREAM
NAREGI
Unicore
OSG
Nordugrid
Naregi
DEISA
EGEE
Teragrid
GRAM
v2/v4
GRAM
v4WS
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 15
How to Start
• Understanding the differences – Compatibility matrix
• Domains that have to be linked for interoperability– Security– Information Services – Job Management– Data Management
• For interoperation you have to add– Monitoring– Accounting – Operational links and joint policies– Trouble ticket systems – Operational security
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 16
Interoperability Matrix
1. Understand both middleware stacks
2. Identify the “common” interfaces
3. Create an interoperability matrix
SRMSRMSRMStorage Control Protocol
GSI/VOMS
GridFTP
GLUE v1
LDAP/GIIS
GRAM
OSG
GSI/VOMSGSI/VOMSSecurity
GridFTPGridFTPStorage Transfer Protocol
GLUE v1.2ARCSchema
LDAP/BDIILDAP/GIISService Discovery
GRAMGridFTPJob Submission
EGEEARC
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 17
Different Strategy
• Long term solution– Common interfaces– Standards
• Medium term solutions– Gateways – Adaptors and Translators
• Short term solutions– Parallel Infrastructures
User driven Site driven
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 18
Parallel Infrastructures
• User Driven– The user joins both grids
Uses different clients• Depending on which interface
– More work for the User Required for each infrastructure
– Keyhole approach Restricts functionality
– Method initially used by ATLAS Split workload between grids
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 19
Parallel Infrastructures
• Site Driven– The site joins both grids
Deploys both interfaces
– User only sees their grid interface
– More work for the site Can only be supported by large sites
• Reduced resources
– Use By FZK Participating in EGEE, Nordugrid and D-Grid
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 20
Gateway
• A gateway is a bridge between grid infrastructures– Single point of failure– Gateway breaks, grid disappears– Scalability bottleneck– All the load through one service
• Useful as a proof concept and to demonstrate the need• NAREGI approach using glite-CE
Gateway
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 21
Adaptors and Translators
• Adaptors allow connection• Translators understand/modify information• They are built into the middleware
– The middleware can then work with both interfaces Useful feature even when using standards!
• Requires modification to the grid middleware– Existing service interfaces can still be used
• Using in the GIN information System; most portals A
PI P
lug
inP
lug
in
EuroVO Workshop - April 2008
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 22
Worldwide Grids
APAC
DEISA
EGEE
Naregi
NDGF
NGSOSG
Pragma
Teragrid
GIN
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 23
How mature are we?
Gartner Group
Grid on the Computing in HighEnergy Physics conferences timeline
Padova2000
Beijing2001
San Diego2003
Interlaken2004
Mumbai2006
Victoria2007
Slide courtesy of Les Robertson, LCG Project Leader
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
From e-Infrastructures to Knowledge Infrastructures
• Network infrastructure connects computing and data resources and allows their seamless usage via Grid infrastructures
• Federated resources and new technologies enable new application fields: – Distributed digital libraries– Distributed data mining– Digital preservation of cultural heritage– Data curation
→ Knowledge Infrastructure
NETWORK .
INFRASTRUCTURE
GRID .
–INFRASTRUCTURE
KNOWLEDGE .
INFRASTRUCTURE
24EuroVO Workshop - April 2008
••• 25
ICT for Science: e-Infrastructures
Linking at the speed of the lightLinking at the speed of the light
Sharing computers, instruments and applicationsSharing computers, instruments and applications
Sharing and federating scientific dataSharing and federating scientific data
. . . . . .
.
Astrophysics
community
WeatherForecast
community
Biomedics
community
Connecting the finest mindsSharing and federating the best scientific
resourcesBuilding global virtual communities
Mario Campolargo Acting Director Emerging Technologies and Infrastructures, EU European Information Space: Infrastructures, Services and Applications Workshop, Rome, 29-30 October 2007
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 26
Evolution
European e-Infrastructure
Testbeds Utility ServiceRoutine Usage
National
Global
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 27
European Grid Initiative
• Need to prepare permanent, common Grid infrastructure
• Ensure the long-term sustainability of the European e-Infrastructure independent of short project funding cycles
• Coordinate the integration and interaction between National Grid Infrastructures (NGIs)
• Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688 EuroVO Workshop - April 2008 28
Summary
• EGEE provides a dependable production quality Grid infrastructure to a wide variety of scientific disciplines.
• Collaborations on technical and political topics are key to implement a truly world-wide infrastructure
• Need to cover full spectrum: from individual sites, small scale Grids to world-wide infrastructures
• Grids are increasingly becoming an essential part of the scientific computing infrastructure – sustainability needs to be ensured