EDGI Brings Desktop Grids To Distributed Computing Interoperability

The EDGI project receives Community research funding 1

EDGI Brings Desktop Grids To Distributed Computing

Interoperability

Etienne URBAH [email protected] OGF29 Chicago

LAL, Univ Paris-Sud, IN2P3/CNRS, Orsay, France

EDGIEDGI

Bringing Desktop Grids To Bringing Desktop Grids To Distributed Computing InteroperabilityDistributed Computing Interoperability

Author : E. Urbah v1.0 2

• EDGI project : Context– EGEE = Service Grid focused on Secure Data Sharing– DG = Desktop Grid = Loose opportunistic grid using

idle resources– EDGeS project :

Presentation, 3G Bridge, Infrastructure, Gflop/s

• EDGI project : Goal = Grid + Cloud– Bridging Security and Info– Useful Official and De facto Standards– Interfaces : Diagrams – Scope of JSDL + BES

Summary

EDGIEDGI



EDGI project : Context

Academic Clouds

OGF PGI

EDGeS

EDGI

EDGI

Cluster based service grids (SGs)(EGEE, NDGF, ...)

Supercomputer based SGs(DEISA, ...)

Desktop grids (DGs)(Volunteer DGs – Home computers,

Organizational DGs – Institutional desktops)

Achievements of EDGeS

• Modular design of 3G Bridge with Application Repository, permitting integration between various DGs and SGs

• 3G Bridges in full production for SGs implementing gLite :

– BOINC and XWHEP <--> EGEE,

– OurGrid <--> EELA2.

• ATTIC = Proof of Concept of Data Distribution

EDGIEDGI



EGEE = Service Grid focused on Secure Data Sharing

Grid User

X509 proxy

VOMS Extensions

VOMS Admin

Manages VO

Site Admin

Manages Site

Accesses Data with VOMS proxy

VOMS Server

SiteStorage Resource

VOMS Proxy =X509 proxy with

VOMS extensions

Trust Anchor = IGTF

Publishes the Root CA Certificates

Interoperability with another Grid Infrastructure is possible only if it uses

IGTF as Trust Anchor

EDGIEDGI



EGEE = Service Grid focused on Secure Data Sharing

Grid User

X509 proxy

VOMS Extensions

Submits Job with VOMS proxy

Publishes available Resources

Pushes Job

Log

Log

VOMS Admin

Manages VO

Site Admin

Manages Site



Gives Job Status

Gives Accounting and Auditing

VOMS Server

AccountingLogging & Bookkeeping

Meta-scheduler(WMS)

SiteComputing Resource

SiteStorage Resource

Grid Admin

Sends back Output Sandbox Sends back

Output Sandbox

VOMS Proxy =X509 proxy with

VOMS extensions

EDGIEDGI



DG = Desktop Grid = Loose opportunistic grid using idle resources

WU = Work Unit = Application + Input DataWU = Work Unit = Application + Input Data

Grid User

Submits input data for an application

Requests WU

Sends WU

Application Manager

Certifies Application

Resource Owner(often volunteer)

Owns Resource

Sends back results

Accepts or Refuses an application on his resource

Grid Server withApplicationRepository

Computing Resource

(often Desktop Computer) Sends back results

Currently, for BOINC, both roles of ‘Application Manager’ and ‘Grid User’ are fulfilled by ‘BOINC Project Owners’.

EDGIEDGI



EDGeS project : Presentation

FP7 project started in 2008 and just ended

• Integrate Service Grids and Desktop Grids

• Enable very large number of computing resources (100K-1M processors)

• Attract new scientific communities

• Provide a Grid application development environment

• Provide application repository and bridges for the execution in the SG-DG system

WLCG (CERN)

EDGeS

gLite(EGEE, EELA,

GridPP)

ARC(NDGF)

Boinc(Berkeley)

XWHEP(INRIA/IN2P3)

OurGrid(Brazil)

Unicore(DEISA)

VDT(OSG)

Operational

Future

Prototype with BES

GT4(NGS)

OperationalFuture

EDGIEDGI



EDGeS 3G Bridge : Modular Design

EDGeS 3G Bridge

EGEE

EGEE

Sou

rce

Grid

H

andl

er

Inte

rfac

e

JobDatabase

Queue Manager

Tar

get

Grid

P

lugi

n In

terf

ace

BOINC Plugins

(DC-API)

EGEEPlugins

Scheduler

Control path WU Job

BOINCHandlers

EGEEHandlerLCG-CE

for EDGeS

gLite WMS

BOINC Server

UserJob with

VOMS proxy

Work Unit

BOINC Server

gLite WMS

WU

Job

Control path Job WU

Job

WU

XWHEP Plugins XWHEP

Server

BESPlugins

NDGF

DEISA

EDGIEDGI



EDGeS Infrastructure : Principle of Operation

EDGIEDGI



EDGeS operational EGEE DG infrastructure

EDGeS VO of EGEE

SZTAKI CE

16 cpus

CNRS / IN2P3 CE

1.600 cpus

CIEMAT CE

20 cpus

BDII VOMS

MyProxy

WMS LB

EGEE Users

EDGeS

BOINC EGEE bridge

EGEE XWHEP bridge

BOINC-based Desktop Grids

SZDG (public)80.000 PCs

IberCivis (public)28.000 PCs

AlmereGrid (public)1.800 PCs

UoW (local)3.100 PCs

EDGeS@ home (public)3.300 PCs

BOINC Project Owners

XWHEP-based Desktop Grids

LAL (local)80 PCs

IN2P3 (public)30 PCs

AlmereGrid (local)10 PCs

XWHEP Users

EGEE BOINC bridge

Application Repository

XWHEP Agent

EDGIEDGI



EDGeS@Home – GFLOP/s (7 days)

EDGIEDGI



EDGI project : Goals

Academic Clouds

OGF PGI

EDGeS

EDGI

EDGI

Cluster based service grids (SGs)(EGEE, NDGF, ...)

Supercomputer based SGs(DEISA, ...)

Desktop grids (DGs)(Volunteer DGs – Home computers,

Organizational DGs – Institutional desktops)

• On the model of IberCivis, creation of a self-sustainable EuroCivis

• Integration between DGs and SGs implementing BES (ARC, Unicore)

• Bridge to Academic Clouds for Jobs needing QoS

• Long term support of the EDGI bridges by the NGIs

• Secure Data Distribution using ATTIC

EDGIEDGI



BOINC EGEE Bridging Security

EGEE

WMS

EDGeS 3G bridge

EGEE Plugin

1 for each (BOINC Project Owner, EGEE VO) pair

Queue Manager & Job DB

BOINC Handler1 for each (BOINC server,

BOINC Project Owner, EGEE VO) triple

WUi+1

WUi+2

WUi+3

Job

Jobi+1

Jobi+2

BOINC Server

Work Unit

BOINC Project Owner

Submission

MyProxy trusting EDGeS

3G bridgeMedium term X509 proxy

Config. file

DN of X509 proxy

Short term X509 proxy

VOMS Server

VOMS extensions

Sou

rce

G

rid

Han

dler

In

terf

ace

Tar

get

Grid

P

lugi

n In

terf

ace

BOINC jobwrapper client (simulating

a large BOINC computing resource)

3G job-wrapper

3G job-wrapper

VOMS proxy Retriever

EDGIEDGI



BOINC Server

EGEE

LCG-CE

for EDGeS

EDGeS3G bridge

EGEE BOINC Bridging Info

Gets EXE

Reports resourcesand performance

Checks EXE

Adds jobWatches

job

Gets VOMS proxy

BOINC

Computing

Resource

BOINC plugin (DC-API)

EDGeS

Application

Repository

Information

provider

GRAM Job

Manager

for EDGeSEGEE VOMS

EGEE User

Queue Manager

& Job DB

Generic Job WS Handler

BOINC Service

Watches

Pushes jobSubmits Job

Logs events

Logs events

EGEE

BDII

gLite

WMS

EGEE LB

Sends output

Gets output

Gets Infos

EDGIEDGI



Grid + Cloud : Useful Official and De facto Standards

EDGIEDGI



Interfaces for Grid + Cloud : Abstract Diagram

EDGIEDGI



Interfaces for Grid + Cloud : Detailed Diagram – Scope of JSDL + BES

Date post:	13-Jan-2016
Category:	Documents
Upload:	adila
View:	36 times
Download:	0 times

EDGI Brings Desktop Grids To Distributed Computing Interoperability

Documents