The EGI Federated Cloud, 7 months of production

www.egi.euEGI-InSPIRE RI-261323

EGI-InSPIRE


The EGI Federated Cloud, 7 months of production

David Wallom

University of Oxford

Chair, EGI Federated Cloud


• Why• Principles• What• Who• For whom• Future


Growth of Providers

• Grid– Limited academic resource providers

• Cloud– Diverse of resource providers


Growth of Userbase

20k researchers 2M researchers!

Few related use cases

Many diverse use cases


HardwareHardware

HardwareHardware

Hardware

Cloud ManagementCloud Management

Cloud ManagementCloud Management

Cloud Management

User CommunitiesUser Communities

User Communities

Federated interfaces

Federated services

• Standards and validation: Recommended and common open standards for the interfaces and images – OCCI, CDMI, OVF, GLUE2.

• Resource integration: Cloud Computing to be integrated into the existing production infrastructure.

• Heterogeneous implementation: no mandate on the cloud technology.

• Provider agnosticism: the only condition to federate resources is to expose the chosen interfaces and services.

Principles of Federation


Value proposition

The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud Services to researchers in Europe and worldwide

A single cloud system able to • Scale to user needs• Integrate multiple different providers to give resilience• Prevent vendor lock-in • Enable resource provision targeted towards the research community

Standards based federation of IaaS cloud: • Exposes a set of independent cloud services accessible to users

utilising a common standards profile• Allows deployment of services across multiple providers and

capacity bursting


EGI Cloud Infrastructure

EGI Core Platform

Federated AAI

ServiceRegistry Monitoring Accounting

EGI Cloud Infrastructure Platform

Instance Mgmt

7

Information Discovery

GSIGLUE2

Cloudinit

Storage Management

CDMI

Cloud Management Stack(OpenStack, OpenNebula, Synnefo, …)

SAM UR

Help and Support

Security Co-ordination

Training and

Outreach

EGI C

olla

bora

tion

Tool

s

EGI A

pplic

ation

D

BIm

age

Repo

sito

ryEG

I Clo

ud S

ervi

ce M

arke

tpla

ce

OVF

Sustainable Business Models

OCCI


VM Instance Management through rOCCI + Contextualisation with Cloudinit

• Contextualisation, the process of installing, configuring and preparing software upon boot time• All uploaded Fedcloud images should support Cloudinit• All supported CMF OCCI implementations extended to support


Accounting

9

~384k VMs launched across 25 clouds

Usage Record Content• Site• User: local user/group, Digital Certificate

DN• VMUUID (VM instance identifier)• Start (and End) Time• Status• Wall and CPU Duration• VM source image• Cloud Type


Status

10


Federated AAI Inherited from the EGI AAI framework• Scalable with many users• Uniform authentication across multiple service providers• Users manage their community via Virtual Organisations


AppDB – Virtual Appliance catalogue

12


Compatibility for technology providers

Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI

OpenStack Yes Yes* Yes Yes Yes Yes

OpenNebula Yes Yes* Yes Yes Yes Yes

Synnefo Yes Yes Yes - Yes -

StratusLab* Yes Yes Yes - Yes - Cloudstack Yes

Emotive Yes Yes

13

Example documentation for a CMF:-https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack

https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack




EGI Marketplace

• Rich and diverse set of individual offerings• Application/VM Image repository• User community choice of provider at all levels• Academic and commercial resource & service providers• User community from the public and private sector


Federated Cloud Services

Federated IaaS and STaaS Cloud

EYR TC monthly meeting 15

Tier 1:Reliable

Infrastructure Cloud

Tier 4:Zero ICT

Infrastructures

Tier 3:Platform as a Service

Tier 2:General-purpose platform services

PaaS

PaaS

DB

aaS

Had

oop

aaS

VRE

Secure storage

Key

Mgm

t

Encr

yptio

n

ACL

mgm

t

Virtual eLaboratory

17 Feb 2014


Partnership• Resources

– 12 countries provide 19 certified resources• Czech Republic, Germany, Greece, Hungary,

Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom

– 4 countries currently integrating• France, Finland, Portugal, South Korea*

– 5 new countries interested+

• Bulgaria, Denmark, Israel*, The Netherlands, Switzerland

– Worldwide interest• South Africa* (SAGrid)• United States* (NIST, NSF Centres)• Australia* (NeCTAR)

• People– >155 individuals participating and

contributing to the Federated Cloud group– >150 conference calls (Sep 2011 – August

2014)

• Utilisation– 98’860 VM instantiations in the last year

* Not shown on map


User Model• Total control over deployed

applications • Elastic resource consumption

based on real needs • Immediately processed

workloads – no more waiting time

• An extended e-Infrastructure across resource providers in Europe

• Service performance scaled with elastic resource consumption

• Single sign-on at multiple, independent providers


EGI FedCloud Communities 9/2014

• Ecology – BioVeL: Biodiversity Virtual e-Laboratory

• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology

• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)

• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for

volcano and earthquakes monitoring (Collaboration with Helix Nebula)

• Software Engineering – SCI-BUS: simulated environments for portal testing

• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems

• Software Engineering – Catania Science Gateway Framework

• Musicology – Peachnote: dynamic analysis of musical scores

• Earth Observation – ENVRI: Common Operations of Environmental Research

infrastructures (collaboration with EISCAT3D)

• Geology – VERCE: Virtual Earthquake and seismology Research

• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem

Research

• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula

More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users


New EGI FedCloud Communities since launch

• Education – Cranfield University distributed systems course

• Cultural Heritage – DCH-RP management of preservation services in the cloud

• Hydrological Modelling – Running Hydrological models to support real time analysis

• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud

environment

• Systems implementations – deployment of FTK developed tools and services and data

preservation

• Internet of Things – Smart Grid systems investigation

• Software Development – deployment of research PaaS

• RNA Sequencing – deployment of analysis engines in the cloud

• Physiological Modelling – Calibration, scenario mapping and development

More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users


WeNMR e-Infrastructure (VirtualCing)

EGI Federated Cloud Infrastructure

…

NMR Protein DB

ToPoS

Virtual CINGVirtual

CINGVirtual Cing

Virtual CINGVirtual

CINGVirtual Cing

Virtual CINGVirtual

CINGVirtual Cing

OCCI client

Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR, has been deployed via the EGI Federated Cloud


ESA HN Flagship on the EGI Federated Cloud

The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federated Cloud:

1. Use SlipStream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET)

2. Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring

21

Automatic Cluster Deployment

EGI Federated CloudComputing Cluster

Cluster Head

Working Nodes

OCCI

Globus

Job submission

BLUE BOX


EISCAT 3D PoC

EISCAT_3D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space.

The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data.


Peachnote

04/15/2023 23

Peachnote is a music score search engine and analysis platform.

Hundreds of thousands of music scores are being digitized by libraries all over the world. In contrast to books, they generally remain inaccessible for content-based retrieval and algorithmic analysis.

There is no analogue to Google Books for music scores, and no large corpora exists that can empower advanced analysis on music scores.

Peachnote want to help change that providing visitors and researchers access to a massive amount of symbolic music data.

EGI Federated Cloud

OMR Worker

PDF Splitter

OMR Feeder

OMR Worker

…


OpenModeller on the Biovel Portal

04/15/2023 24

The Ecological Niche Modeling (ENM) Workflow takes as input a file containing species occurrence points to create a model with the openModeller Web Service.

• The EUBrazilOpenBio ENM service is exposed through an extended openModeller Web Service interface• Multi-staging and multi-parametric oM experiments are implemented through COMPSs that dynamically creates the virtual resources to execute the operations.• An OCCI connector is used for the VMs management while data management supports CDMI endpoints.

ENM Service (OMWS2)

VENUS-C Cloud Middleware

COMPSs Workflow Orchestrator

COMPSs Workflow OrchestratorOCCIOCCI CDMICDMI

EGI Federated Cloud

Service available athttps://portal.biovel.eu/

https://wiki.biovel.eu/display/doc/Ecological+Niche+Modelling+(ENM)+Workflow

http://www.bsc.es/compss

https://portal.biovel.eu/


BioSTIF - Biodiversity Spatio-temportal interface on the BioVel Portal

04/15/2023 25

BioSTIF provides an easy GIS interface to vizualize and quality control biodiversity data in time and space

The interface contains; • a map widget (displaying

geographical data)• a timeline widget

(displaying data on a time scale)

• a table widget (displaying data in table format).

Service available athttps://portal.biovel.eu/

https://wiki.biovel.eu/display/doc/BioSTIF+User+Manual;

https://portal.biovel.eu/

www.egi.euEGI-InSPIRE RI-26132304/15/2023 26

Running GNU Octave and R on the CHAIN-REDS Science Gateway

26

CHAIN-REDS Science Gateway-CHAIN-REDS RI-306819

http://science-gateway.chain-project.eu/


Usecase Integration Workflow

Status Description

Pre-assessment FCUS members identified a potential use case that can profit from the EGI Federated Cloud services

Assessed The use case requirements are assessed with relevant information added to a dedicated wiki page, to manage the full porting & integration process

Preparatory The user & FCUS team setup the environment to execute tests on the FedCloud (configuration of client environment, identify resource providers, upload VM images, ...)

Test & Integration

Users are accessing the FedCloud through the fedcloud.egi.eu VO to execute tests while integrate their applications to the infrastructure

Pre-production Test & integration phases were successfully completed. Create a production Virtual Organization (VO) or join an already existing VO

Production Use case is being used in production. The use case completed all tests and is regularly making use of the Federated Cloud using a production level VO

Closed Test & integration successfully completed. Use case either does not foresee moving into production.

Cancelled Test & integration did not successfully completed. User cancelled the use case.

10 6411 3

6


Strengthening the underpinning platform

Technology Evolution• Broader support for open standards in Cloud

management frameworks– Utilisation of rOCCI for interfaces to commercial cloud

frameworks

• New feature additions to foundational tools depending on requests– Accounting, monitoring, service discovery, Application DB

• Innovate deployments of opensource PaaS• Large improvements on documentation for all 3

stakeholder groups

28

Valu

e ad

ded

serv

ices

for U

ser

Com

mun

ities


EGI’s Cloud Vision for 2020To support the digital European Research Area through a pan-

European research infrastructure based on an open federation of reliable services that provide uniform access to computing and

data resources provided by the public and private sector.

EGI Federated Cloud capability vision 10M cores Cloud compute

1 EB Cloud storage


Conclusions• 40 individual usecases from 27 communities have

used the fedcloud activities since launch– ‘Standard’ and innovative utilisation of cloud including

research and teaching

• Paving the way for a global federated cloud marketplace– Established best practice– Illustrating leadership– Open standards, open technology– Open membership, open processes– Open competition

30

Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh Nguyen, Björn Hagemeier, Boris Parak, Boro

Jakimovski, Cal Loomis, Daniele Cesini, Daniele Lezzi, David Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos, Jan Meizner, John Gordon, Kostas Koumantaros, Malgorzata Krakowian, Marios

Chatziangelou, Marco Verlato, Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel

Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge, Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi, Soonwook Hwang, Steven Newhouse,

Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah.


Thank you!

Questions?

32

Task Force resources

Mailing List: [email protected]

Wiki site: http://go.egi.eu/tf-fedclouds

GitHub: https://github.com/EGI-FCTF

Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112

Date post:	04-Aug-2015
Category:	Documents
Upload:	david-wallom
View:	63 times
Download:	2 times

The EGI Federated Cloud, 7 months of production

Documents