Post on 04-Jun-2020
transcript
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.eu EGI-InSPIRE RI-261323
A Federated Cloud Infrastructure for European
Science Steven Newhouse
EGI.eu Director
13/12/2011 CloudCom 2011 1
www.egi.eu EGI-InSPIRE RI-261323
EGI
• European – Over 35 countries
• Grid – Secure sharing
• Infrastructure – Computers – Data – Instruments – …. and beyond!!
13/12/2011 CloudCom 2011 2
www.egi.eu EGI-InSPIRE RI-261323
What is EGI?
• European Grid Infrastructure (EGI) – Federation of National Grid Infrastructures – Represented by EGI.eu (Dutch foundation)
• National Grid Infrastructures (NGIs) – Sole national point of contact for ‘Grids‘ – Integrates individual resource centres
• Current Status: – 35+ NGIs & 350+ resource centres – 300,000 CPUs & 200PB+ storage
13/12/2011 CloudCom 2011 3
www.egi.eu EGI-InSPIRE RI-261323
What is e-Infrastructure?
Electronic Infrastructure to enable research (Established by the European Commission)
GEANT
Pan-European High Performance Networking
European Grid Infrastructure
(EGI)
Partnership for Advanced Computing in Europe
(PRACE)
Infrastructure for Data (EU-DAT project)
13/12/2011 CloudCom 2011 4
www.egi.eu EGI-InSPIRE RI-261323
C21st: Digital Research
CloudCom 2011
Extracting Knowledge from the Data Deluge
5 13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
The Data Deluge
• The Grid goes beyond High Energy Physics – Over 40 European research infrastructures
• Medical Records – Genome $3B & 10 years $5,000 & 24hrs – Scans, Xrays, etc. – Mobile medical monitors
• From Humanity – TV, radio, documents, videos, etc. – Analysis and understanding
13/12/2011 CloudCom 2011 6
www.egi.eu EGI-InSPIRE RI-261323
The Challenge for Europe
• 1.8M+ public & private sector researchers – They are the European Research Area (ERA)
• Reduce time to market for new innovation – Key to assure Europe’s future competitiveness
• Having to cope with the data deluge – Yet working practices still mostly C20th
• E-Infrastructures have been early adopters
13/12/2011 CloudCom 2011 7
Need to bring the whole ERA on-line
www.egi.eu EGI-InSPIRE RI-261323
Digital Agenda for Europe • Borderless Services • Interoperability • Supporting Innovation
CloudCom 2011 8
Digital Agenda for Europe
EGI’s role in EU2020: http://go.egi.eu/317
“Europe should also build its innovative advantage in key areas through reinforced
e-Infrastructures (i.e. GEANT & EGI)”
removing barriers to the free movement of
knowledge
13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
Which direction(s) to take?
13/12/2011 CloudCom 2011 9
New Services New Technologies
Sustainability
European Structures National Priorities
Increasing Use & Users
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.eu EGI-InSPIRE RI-261323
Technology and Services
13/12/2011 CloudCom 2011 10
www.egi.eu EGI-InSPIRE RI-261323
Disruptive Technologies
• The Cloud has changed IT provision – User focused – Service oriented – On demand
• Underlying technology is virtualisation – Greater flexibility – Better utilisation – Commoditised
13/12/2011 CloudCom 2011 11
www.egi.eu EGI-InSPIRE RI-261323
Everything as a Service
13/12/2011 CloudCom 2011 12
Infrastructure
Platforms
Software
gLite UNICORE dCache ARC Globus
Core Services Support Community
Virtual Research
Communities (Users)
EGI.e
u Co
ordi
natio
n
NGI NGI NGI EIRO NGI NGI NGI
Delivering services to individual communities
Structural development and implementation funded by the
European Commission Established EGI Community
www.egi.eu EGI-InSPIRE RI-261323
A vehicle for Innovation • Deploying Technology Innovation
– Distributed Computing continues to evolve • To include: Grids, Desktops, Virtualisation, Clouds, …
• Enabling Software Innovation – Provide reliable persistent technology platform
• Tools built on gLite/UNICORE/ARC/Globus
• Supporting Research Innovation in Europe – Infrastructure for data driven research
• Support for international research (e.g. ESFRI)
13/12/2011 CloudCom 2011 13
www.egi.eu EGI-InSPIRE RI-261323
What do we provide now?
13/12/2011 CloudCom 2011 14
Core Infrastructure
Services
www.egi.eu EGI-InSPIRE RI-261323
A Virtuous Service Cycle
CloudCom 2011
Used by Researchers
Gathering New
Requirements
New Technology
Assessed
Deployed Infrastructure
Services
15
EGI.eu
National Resource Providers
Current User Communities
New User Communities Technology
Providers
13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
Service Evolution • ‘Previous’ (pre-EGI) Model
– Single Platform – Single Technology Islands (gLite & ARC) – Single Computing Model (HTC)
• HTC: High Throughput Computing
• ‘Current’ (EGI) Model – Single Platform – Integrating Technology Islands (EMI & IGE) – Two Computing Models (HTC & HPC)
• HPC: High Performance Computing 13/12/2011 CloudCom 2011 16
www.egi.eu EGI-InSPIRE RI-261323
Different Computing Models • Computing models tied to a community
– The services they use – Where the data is consumed or produced – Distributed but linked by networks
• Different models for different users On the same e-Infrastructure
13/12/2011 CloudCom 2011 17
www.egi.eu EGI-InSPIRE RI-261323
Use Emerging Technology
• Virtualisation – Adoption reduces deployment costs
• Federation – There will never be one EU provider
• Integration – Integrate commercial & private resources
13/12/2011 CloudCom 2011 18
www.egi.eu EGI-InSPIRE RI-261323
Federated EU Cloud
13/12/2011 CloudCom 2011 19
Core Infrastructure
Services
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.eu EGI-InSPIRE RI-261323
European & National Structures
13/12/2011 CloudCom 2011 20
www.egi.eu EGI-InSPIRE RI-261323
European Federation
• Build upon strong national activities – National Research and Education Networks – Public investment in data centres
• European coordination & representation – Dedicated legal entity – National stakeholder representatives – Weighted voting
13/12/2011 CloudCom 2011 21
www.egi.eu EGI-InSPIRE RI-261323
EGI.eu • Coordination for European Grid resources
– Established 8 February 2010 – Central policy & services needed to run EGI – Sustainable small coordinating organisation
• Based in Amsterdam – Coordinating core (21 people) in Amsterdam – Technical services from partners (~20 FTEs)
• Governed by a Council with national reps.
CloudCom 2011 22
EGI and EGI.eu: Supported by the EGI-InSPIRE project 13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe
A 4-year project with €25M EC contribution
– Project cost €72M – Total Effort ~€330M – Effort: 9261PMs
Project Partners (50) EGI.eu, 38 NGIs, 2 EIROs
Asia Pacific (9 unfunded partners)
CloudCom 2011 23 13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
Project Objectives
• A sustainable production infrastructure – Resource providers in Europe and worldwide – With new technologies as they mature
• Support structured international research – Sustain current domain specific services – Attract new user communities (e.g. ESFRI)
13/12/2011 CloudCom 2011 24
www.egi.eu EGI-InSPIRE RI-261323
NIST Deployment Scenarios
Within trust boundary
Crossing trust boundary
Centralised (within an administrative domain)
Private Cloud Commercial Cloud
Decentralised (across administrative domains)
Federated Cloud (Out of bound trust)
Hybrid Cloud (Negotiated Trust)
13/12/2011 CloudCom 2011 25
EGI
www.egi.eu EGI-InSPIRE RI-261323
Federated Cloud Task Force
• 52 members including: – 15 Resource Providers across Europe – 6 Technology Providers
• Bridge different technologies (currently) – 47% (7) deploy OpenNebula – 20% (3) deploy StratusLab – 20% (3) deploy OpenStack – 7% (1) deploys WNoDeS – 7% (1) deploys proprietary stack
13/12/2011 CloudCom 2011 26
www.egi.eu EGI-InSPIRE RI-261323
EGI’s Cloud Usage Scenarios
1. Running a pre-defined VM Image (VMI) 2. Using My VMIs & My Data in the Infrastructure 3. Running across multiple Resource Providers 4. Accounting across Resource Providers 5. Reliability & Availability of Resource Providers 6. VM/Resource state change notification
http://go.egi.eu/435
CloudCom 2011 27 13/12/2011
www.egi.eu EGI-InSPIRE RI-261323
The need for standards • Federation implies integrated diversity
– Different sites have different policies – Different providers have different solutions
• Need to integrate across interfaces for: – Virtual Machine Management (e.g. OCCI) – Access to data (e.g. CDMI) – Guaranteed networking (e.g. NSI)
• Other standards for: – Identity federation (e.g. X.509) – Data movement (e.g. GridFTP) – SLAs (e.g. WS-Agreement)
13/12/2011 CloudCom 2011 28
www.egi.eu EGI-InSPIRE RI-261323
Capabilities & standards
13/12/2011 CloudCom 2011 29
Capability Standard(s) Status Comments
VM Management OCCI 1.1 Confirmed Interop testing underway
Data management CDMI 1.0.1 OVF 1.1.0 Proposed
Information Discovery
LDAP3/LDIF, GLUE2+ Proposed Extensions to GLUE2 are
expected
Accounting UR+, APEL based Proposed
UR as basis agreed, extensions discussed.
APEL
Monitoring Nagios, Nagios plugins Confirmed Strict external interface
testing.
Notification Messaging based Research
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.eu EGI-InSPIRE RI-261323
Sustainability & Growth
13/12/2011 CloudCom 2011 30
www.egi.eu EGI-InSPIRE RI-261323
EGI Ecosystem
CloudCom 2011 31
Public Funding Bodies
European Commission
National Research Councils
Service & Resource Providers
EGI.eu
Resource Infrastructure
Providers
Technology Providers
Open Source Providers
Commercial Providers
User Community
13/12/2011
Technology providers bring research results to a production quality
www.egi.eu EGI-InSPIRE RI-261323
Sustaining the ecosystem?
• National Resource Providers – Integrate new national research groups – Increase the flexibility to offer different services
• User Communities – Excellence supported through peer review funded – Need to budget for the cost of infrastructure
• Technology Providers – Innovation funded nationally or by EC – Maintenance by the technology consumers
13/12/2011 CloudCom 2011 32
www.egi.eu EGI-InSPIRE RI-261323
Innovative Technology
• Work needed for two distinct groups: – Infrastructure providers – Consuming user communities
• Federating the virtualised infrastructure – Proven grid technologies to integrate clouds – Scale across trust & admin boundaries
• User facing services – Supporting distinct different computing models
13/12/2011 CloudCom 2011 33
www.egi.eu EGI-InSPIRE RI-261323
The European Cloud
13/12/2011 CloudCom 2011 34
NGI NGI NGI EIRO Commercial Commercial EGI.e
u Co
ordi
natio
n Co
re S
oftw
are
& S
uppo
rt
Individual communities get the environments they require
gLite UNICORE dCache ARC Globus Community Platform
Community Services
Community Services Globus
Com
mun
ity sp
ecifi
c se
rvic
e pl
atfo
rms
Federated Virtual Machine Managers
A cloud for EU researchers Public & commercial resources Complement EC investment in innovation with commercial
expertise
www.egi.eu EGI-InSPIRE RI-261323
Grid/Cloud vs. Cloud/Grid
• Grid is a platform, deployed on a cloud – Many grid services are user facing – Applications dealing with data analysis – These services can sit in a cloud
• A federated cloud is a grid – A grid provides controlled access to resources – Similar issues around monitoring and accounting – Use grid technology to federate clouds
13/12/2011 CloudCom 2011 35
www.egi.eu EGI-InSPIRE RI-261323
Research Challenges • Provisioning
– I want to deploy my services in my configuration near my data sources and sinks on the resources I have access to
• Autonomic Management – I want these services to keep operating despite
failures for the next 14 days as I analyse my data • Service Level Agreements
– I want my use of compute, storage & networking resources to be assured during that time
• Standards
13/12/2011 CloudCom 2011 36
www.egi.eu EGI-InSPIRE RI-261323
Summary
• After a decade the system works well – Scaled up with increasing use & users
• An active & engaged community – Consensus based policy & procedures
• Sustainability for decades to come – EC funds innovation and structuring – Stakeholders fund the operation they benefit from – By growing the number and diversity of end-users
13/12/2011 CloudCom 2011 37
www.egi.eu EGI-InSPIRE RI-261323
Summary
13/12/2011 CloudCom 2011 38
• We have done a lot – But there is more to do!
• Challenges everywhere – Software for new users – Provisioning at scale – Autonomic Management
• Sustainability – 10 year not a 2 year view