A Federated Cloud Infrastructure for European Science€¦ · EGI-InSPIRE RI-261323 EGI-InSPIRE A...

www.egi.eu EGI-InSPIRE RI-261323

EGI-InSPIRE


A Federated Cloud Infrastructure for European

Science Steven Newhouse

EGI.eu Director

13/12/2011 CloudCom 2011 1


EGI

• European – Over 35 countries

• Grid – Secure sharing

• Infrastructure – Computers – Data – Instruments – …. and beyond!!

13/12/2011 CloudCom 2011 2


What is EGI?

• European Grid Infrastructure (EGI) – Federation of National Grid Infrastructures – Represented by EGI.eu (Dutch foundation)

• National Grid Infrastructures (NGIs) – Sole national point of contact for ‘Grids‘ – Integrates individual resource centres

• Current Status: – 35+ NGIs & 350+ resource centres – 300,000 CPUs & 200PB+ storage

13/12/2011 CloudCom 2011 3


What is e-Infrastructure?

Electronic Infrastructure to enable research (Established by the European Commission)

GEANT

Pan-European High Performance Networking

European Grid Infrastructure

(EGI)

Partnership for Advanced Computing in Europe

(PRACE)

Infrastructure for Data (EU-DAT project)

13/12/2011 CloudCom 2011 4


C21st: Digital Research

CloudCom 2011

Extracting Knowledge from the Data Deluge

5 13/12/2011


The Data Deluge

• The Grid goes beyond High Energy Physics – Over 40 European research infrastructures

• Medical Records – Genome $3B & 10 years $5,000 & 24hrs – Scans, Xrays, etc. – Mobile medical monitors

• From Humanity – TV, radio, documents, videos, etc. – Analysis and understanding

13/12/2011 CloudCom 2011 6


The Challenge for Europe

• 1.8M+ public & private sector researchers – They are the European Research Area (ERA)

• Reduce time to market for new innovation – Key to assure Europe’s future competitiveness

• Having to cope with the data deluge – Yet working practices still mostly C20th

• E-Infrastructures have been early adopters

13/12/2011 CloudCom 2011 7

Need to bring the whole ERA on-line


Digital Agenda for Europe • Borderless Services • Interoperability • Supporting Innovation

CloudCom 2011 8

Digital Agenda for Europe

EGI’s role in EU2020: http://go.egi.eu/317

“Europe should also build its innovative advantage in key areas through reinforced

e-Infrastructures (i.e. GEANT & EGI)”

removing barriers to the free movement of

knowledge

13/12/2011

http://go.egi.eu/317�




Which direction(s) to take?

13/12/2011 CloudCom 2011 9

New Services New Technologies

Sustainability

European Structures National Priorities

Increasing Use & Users


EGI-InSPIRE


Technology and Services

13/12/2011 CloudCom 2011 10


Disruptive Technologies

• The Cloud has changed IT provision – User focused – Service oriented – On demand

• Underlying technology is virtualisation – Greater flexibility – Better utilisation – Commoditised

13/12/2011 CloudCom 2011 11


Everything as a Service

13/12/2011 CloudCom 2011 12

Infrastructure

Platforms

Software

gLite UNICORE dCache ARC Globus

Core Services Support Community

Virtual Research

Communities (Users)

EGI.e

u Co

ordi

natio

n

NGI NGI NGI EIRO NGI NGI NGI

Delivering services to individual communities

Structural development and implementation funded by the

European Commission Established EGI Community


A vehicle for Innovation • Deploying Technology Innovation

– Distributed Computing continues to evolve • To include: Grids, Desktops, Virtualisation, Clouds, …

• Enabling Software Innovation – Provide reliable persistent technology platform

• Tools built on gLite/UNICORE/ARC/Globus

• Supporting Research Innovation in Europe – Infrastructure for data driven research

• Support for international research (e.g. ESFRI)

13/12/2011 CloudCom 2011 13


What do we provide now?

13/12/2011 CloudCom 2011 14

Core Infrastructure

Services


A Virtuous Service Cycle

CloudCom 2011

Used by Researchers

Gathering New

Requirements

New Technology

Assessed

Deployed Infrastructure

Services

15

EGI.eu

National Resource Providers

Current User Communities

New User Communities Technology

Providers

13/12/2011


Service Evolution • ‘Previous’ (pre-EGI) Model

– Single Platform – Single Technology Islands (gLite & ARC) – Single Computing Model (HTC)

• HTC: High Throughput Computing

• ‘Current’ (EGI) Model – Single Platform – Integrating Technology Islands (EMI & IGE) – Two Computing Models (HTC & HPC)

• HPC: High Performance Computing 13/12/2011 CloudCom 2011 16


Different Computing Models • Computing models tied to a community

– The services they use – Where the data is consumed or produced – Distributed but linked by networks

• Different models for different users On the same e-Infrastructure

13/12/2011 CloudCom 2011 17


Use Emerging Technology

• Virtualisation – Adoption reduces deployment costs

• Federation – There will never be one EU provider

• Integration – Integrate commercial & private resources

13/12/2011 CloudCom 2011 18


Federated EU Cloud

13/12/2011 CloudCom 2011 19

Core Infrastructure

Services


EGI-InSPIRE


European & National Structures

13/12/2011 CloudCom 2011 20


European Federation

• Build upon strong national activities – National Research and Education Networks – Public investment in data centres

• European coordination & representation – Dedicated legal entity – National stakeholder representatives – Weighted voting

13/12/2011 CloudCom 2011 21


EGI.eu • Coordination for European Grid resources

– Established 8 February 2010 – Central policy & services needed to run EGI – Sustainable small coordinating organisation

• Based in Amsterdam – Coordinating core (21 people) in Amsterdam – Technical services from partners (~20 FTEs)

• Governed by a Council with national reps.

CloudCom 2011 22

EGI and EGI.eu: Supported by the EGI-InSPIRE project 13/12/2011


EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe

A 4-year project with €25M EC contribution

– Project cost €72M – Total Effort ~€330M – Effort: 9261PMs

Project Partners (50) EGI.eu, 38 NGIs, 2 EIROs

Asia Pacific (9 unfunded partners)

CloudCom 2011 23 13/12/2011


Project Objectives

• A sustainable production infrastructure – Resource providers in Europe and worldwide – With new technologies as they mature

• Support structured international research – Sustain current domain specific services – Attract new user communities (e.g. ESFRI)

13/12/2011 CloudCom 2011 24


NIST Deployment Scenarios

Within trust boundary

Crossing trust boundary

Centralised (within an administrative domain)

Private Cloud Commercial Cloud

Decentralised (across administrative domains)

Federated Cloud (Out of bound trust)

Hybrid Cloud (Negotiated Trust)

13/12/2011 CloudCom 2011 25

EGI


Federated Cloud Task Force

• 52 members including: – 15 Resource Providers across Europe – 6 Technology Providers

• Bridge different technologies (currently) – 47% (7) deploy OpenNebula – 20% (3) deploy StratusLab – 20% (3) deploy OpenStack – 7% (1) deploys WNoDeS – 7% (1) deploys proprietary stack

13/12/2011 CloudCom 2011 26


EGI’s Cloud Usage Scenarios

1. Running a pre-defined VM Image (VMI) 2. Using My VMIs & My Data in the Infrastructure 3. Running across multiple Resource Providers 4. Accounting across Resource Providers 5. Reliability & Availability of Resource Providers 6. VM/Resource state change notification

http://go.egi.eu/435

CloudCom 2011 27 13/12/2011

https://documents.egi.eu/document/435�

https://documents.egi.eu/document/435�


The need for standards • Federation implies integrated diversity

– Different sites have different policies – Different providers have different solutions

• Need to integrate across interfaces for: – Virtual Machine Management (e.g. OCCI) – Access to data (e.g. CDMI) – Guaranteed networking (e.g. NSI)

• Other standards for: – Identity federation (e.g. X.509) – Data movement (e.g. GridFTP) – SLAs (e.g. WS-Agreement)

13/12/2011 CloudCom 2011 28


Capabilities & standards

13/12/2011 CloudCom 2011 29

Capability Standard(s) Status Comments

VM Management OCCI 1.1 Confirmed Interop testing underway

Data management CDMI 1.0.1 OVF 1.1.0 Proposed

Information Discovery

LDAP3/LDIF, GLUE2+ Proposed Extensions to GLUE2 are

expected

Accounting UR+, APEL based Proposed

UR as basis agreed, extensions discussed.

APEL

Monitoring Nagios, Nagios plugins Confirmed Strict external interface

testing.

Notification Messaging based Research


EGI-InSPIRE


Sustainability & Growth

13/12/2011 CloudCom 2011 30


EGI Ecosystem

CloudCom 2011 31

Public Funding Bodies

European Commission

National Research Councils

Service & Resource Providers

EGI.eu

Resource Infrastructure

Providers

Technology Providers

Open Source Providers

Commercial Providers

User Community

13/12/2011

Technology providers bring research results to a production quality


Sustaining the ecosystem?

• National Resource Providers – Integrate new national research groups – Increase the flexibility to offer different services

• User Communities – Excellence supported through peer review funded – Need to budget for the cost of infrastructure

• Technology Providers – Innovation funded nationally or by EC – Maintenance by the technology consumers

13/12/2011 CloudCom 2011 32


Innovative Technology

• Work needed for two distinct groups: – Infrastructure providers – Consuming user communities

• Federating the virtualised infrastructure – Proven grid technologies to integrate clouds – Scale across trust & admin boundaries

• User facing services – Supporting distinct different computing models

13/12/2011 CloudCom 2011 33


The European Cloud

13/12/2011 CloudCom 2011 34

NGI NGI NGI EIRO Commercial Commercial EGI.e

u Co

ordi

natio

n Co

re S

oftw

are

& S

uppo

rt

Individual communities get the environments they require

gLite UNICORE dCache ARC Globus Community Platform

Community Services

Community Services Globus

Com

mun

ity sp

ecifi

c se

rvic

e pl

atfo

rms

Federated Virtual Machine Managers

A cloud for EU researchers Public & commercial resources Complement EC investment in innovation with commercial

expertise


Grid/Cloud vs. Cloud/Grid

• Grid is a platform, deployed on a cloud – Many grid services are user facing – Applications dealing with data analysis – These services can sit in a cloud

• A federated cloud is a grid – A grid provides controlled access to resources – Similar issues around monitoring and accounting – Use grid technology to federate clouds

13/12/2011 CloudCom 2011 35


Research Challenges • Provisioning

– I want to deploy my services in my configuration near my data sources and sinks on the resources I have access to

• Autonomic Management – I want these services to keep operating despite

failures for the next 14 days as I analyse my data • Service Level Agreements

– I want my use of compute, storage & networking resources to be assured during that time

• Standards

13/12/2011 CloudCom 2011 36


Summary

• After a decade the system works well – Scaled up with increasing use & users

• An active & engaged community – Consensus based policy & procedures

• Sustainability for decades to come – EC funds innovation and structuring – Stakeholders fund the operation they benefit from – By growing the number and diversity of end-users

13/12/2011 CloudCom 2011 37


Summary

13/12/2011 CloudCom 2011 38

• We have done a lot – But there is more to do!

• Challenges everywhere – Software for new users – Provisioning at scale – Autonomic Management

• Sustainability – 10 year not a 2 year view

Date post:	04-Jun-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

A Federated Cloud Infrastructure for European Science€¦ · EGI-InSPIRE RI-261323 EGI-InSPIRE A...

Documents