Status of Tsukuba, KEK, 21 September 2010 Sylvain Reynaud.

Status ofStatus of

Tsukuba, KEK, 21 September 2010

Sylvain Reynaud

JSAGA 2

The SAGA specificationThe SAGA specification

SAGA : the specification– a recommendation of Open

Grid Forum

– a Simple API

– for using Grid middleware• execution management

– job submission, monitoring...

• data management– transfer, list files, search…

• service discovery• …

– in user Applications• implementations (or wrappers)

exist for C++, Java and Python

…in a uniform way

S A

G

A

use middlewares in a uniform way(independent of middlewares)

use functionalities in a uniform way(consistent "look-and-feel")

use language bindings in a similar way

(language-independent specification)

JSAGA 3

SAGA vs. gLite User InterfaceSAGA vs. gLite User Interface

Source: S Jha, T Kielmann, A Merzky

user

gLite-UIgLite-UI

VOMS,VOMS,MyProxMyProx

yy

WMS,WMS,LCG-CE,LCG-CE,

CREAM-CECREAM-CE

SRM,SRM,gsiftpgsiftp

LFC,LFC,AMGAMG

AABDIIBDII

JSAGA 4

Simple API for Grid ApplicationsSimple API for Grid Applications

Java code example (data management)

Session session = SessionFactory.createSession(true);URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/");

// create SAGA object with a factory

NSDirectory dir = NSFactory.createNSDirectory(session, url);

// use the created SAGA objectList<URL> result = dir.list();for (URL r : result)

System.out.println(r);

list content of directory

gsiftp://…/tmp/

JSAGA 5

Simple API for Grid ApplicationsSimple API for Grid Applications

Java code example (execution management)

Session session = SessionFactory.createSession(true);URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…");

// create SAGA object with a factory

JobService svc = JobFactory.createJobService(session, url);

// use the created SAGA objectJob job = svc.runJob("/bin/date --utc");job.waitFor();System.out.println(job.getState());

submit a job and wait for

the end of its execution

JSAGA 6

The JSAGA implementationThe JSAGA implementation

a Java implementation of the SAGA specification

focuses on– uniform usage of middleware…. (beyond what they natively support)– ease of extension……………….– efficiency and scalability……….– control of the API behavior........ (configure binding rather than trying)– operating-system independency (tested on )

Under LGPL license

(thanks to the design of adaptor interfaces)

JSAGA 7

user applicationsuser applications

VOM

S

WM

S

gsiftp

Gatekeep.

Globus

JSAGAJSAGA

GridFTPGridFTP

Example – uniform usageExample – uniform usageSAGA

jobSAGA

job

out.txtout.txt

out.txtout.txt

JSAGA 8


VOM

S

WM

S

gsiftp

Example – what happens behindExample – what happens behind

EGEE proxy + attributes translate submit job poll status delegate transfer

DGrid proxy adapt + translate submit job receive status transfer

– to any protocolG

atekeep.

Globus

JSAGAJSAGA

CREAM-CECREAM-CELCG-CELCG-CE UnicoreUnicore

WMSWMS

Globus GKGlobus GK

RSL

VOMSVOMS

modified

jobwrapper

script

GridFTPGridFTP

SAGAjob

D-GridEGEE JDL

out.txtout.txt

out.txtout.txt

JSAGA 9

Layered software architectureLayered software architecture

JSAGAJSAGA

interfaceinterface

implementationimplementation

JSAGA 10


Used by end usersUsed by end users– web portals, GUI, CLI…web portals, GUI, CLI…

Used by application developersUsed by application developers– a single uniform interfacea single uniform interface

– object oriented, high-levelobject oriented, high-level

Implemented by adaptorsImplemented by adaptors– each "way" to implement a each "way" to implement a

given feature has 1 interfacegiven feature has 1 interface

– service oriented, low-levelservice oriented, low-level

Implemented by middlewareImplemented by middleware– each technology has its own each technology has its own

interfacesinterfaces


middleware APIsmiddleware APIsmiddleware APIsmiddleware APIs

core enginecore engineJSAGAJSAGA

adaptorsadaptorsJSAGAJSAGA

adaptors interfacesadaptors interfacesJSAGAJSAGA

SAGASAGA Java bindingJava binding

interfaceinterface

implementationimplementation

JSAGA 11

listen status

changestransfer

output



GridFTPGridFTP Globus GKGlobus GK

adaptor

getstatus

getoutput

uploaddata

JSAGA JSAGA core enginecore engine

JSAGA JSAGA adaptor interfacesadaptor interfaces

adaptor









JSAGA 12




adaptoradaptor

getstatus

getoutput

uploaddata











listen status

changestransfer

output

JSAGA 13




adaptoradaptor

getstatus

getoutput

uploaddata



listen status

changes

Middleware API can be used Middleware API can be used more efficientlymore efficiently– engine selects best interface for engine selects best interface for

each user requesteach user request

Developing new adaptors is Developing new adaptors is easiereasier– just wrap functionalities just wrap functionalities

supported by middleware APIsupported by middleware API

transfer

output

JSAGA 14

adaptorsadaptorsinterfacesinterfacesJSAGAJSAGA

As many interfaces as ways to implement each functionality

Example: ways to monitor jobs– API mode

• poll job status• listen to notifications about job status

changes

– API granularity• individual jobs• list of jobs• jobs filter (e.g. by user, by date, by

tag…)

planned

construction

done

Job monitoringJob monitoring

ga

teke

ep

er

gL

ite-W

MS

wsg

ram

un

icore

6

crea

m

na

reg

i

Design of adaptors interfacesDesign of adaptors interfaces

ssh

loca

l

SAGASAGA

querystatus forindividual

job

listenstatus forindividual

job

querystatus for

filteredjobs

getState

waitFor

JSAGA 15

adaptorsadaptorsinterfacesinterfacesJSAGAJSAGA

Design of adaptors interfacesDesign of adaptors interfaces

Optional interfaces for optimization

Example: ways to copy a file– data read/write methods

• either stream methods• or get/put methods

– data copy• e.g. enable third-party transfer

– delegated transfer

planned

construction

done

PhysicalPhysicalrbyte

io

gsiftp

srm

srb

irod

s

http

http

s

sftp

ftp

file

zip

cach

e

SAGASAGA

datareader

datawriter

datacopy

copy

JSAGA 16

LogicalLogical PhysicalPhysical

Supported technologiesSupported technologies

Minimal dependencies on external libraries external tools

– e.g. no gLite-UI operating system

– tested on

planned

construction

done

DataData

ExecutionExecution

SecuritySecurity

InM

em

Cre

d

Glo

bu

s

G. L

eg

acy

G. R

FC

82

0

MyP

roxy

VO

MS

JKS

SS

H

Lo

gin

/ pw

d

X5

09

ga

teke

ep

er

gL

ite-W

MS

wsg

ram

un

icore

6

crea

m

na

reg

i

ssh

loca

l

rns

lfn

srb

irod

s

cata

log

rbyte

io

gsiftp

srm

srb

irod

s

http

http

s

sftp

ftp

file

zip

cach

e

Grid

techno.

Grid

techno.

Grid

techno.

Others

Others

Others

JSAGA 17

Latest developmentsLatest developments

New JSAGA adaptors– for data management

• gLite-LFC, dCache (SRM)

– for execution management• OGSA-BES (Unicore, ARC…)

Improvements in some gLite adaptors and their libraries– features

– thread-safety

– memory leaks

– …

JPySAGA is now compliant with– reference Python binding for

SAGA (as user interface)– reference Java binding for

SAGA version 1.1 (as implementation interface)

JSAGA has a new configuration engine– enable dynamic configuration– support 1 configuration/session– detect conflicts at config.-time

JSAGA 18

Security context selectionSecurity context selection

Try all security contexts

attractive for beginners

can be inefficient– waste of time (while trying)

– waste of computing resource (risk of output staging failure)

can be unreliable– may create files with wrong

ownership

– may block an account with too many failed connection attempts

Bind contexts with URL patterns

predictable/controllable behavior

requires some knowledge about– used infrastructures– JSAGA configuration capabilities

JSAGA has a new configuration engine– enable dynamic configuration– support 1 configuration/session– detect conflicts at config.-time

JSAGA 19

Software qualitySoftware quality

Adaptors– validated by a middleware-

independent SAGA test suite

– maven 'archetype' to generate skeleton of new adaptor project

Automated build process– download and install

• build tools• external libraries

– generate source code– execute test-suites

• unitary tests• integration tests

– generate project web site• documentation• reports

– generate installer GUI• extract library dependencies

from maven project description

# SAGA protocols test-suite configurationgsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/gsiftp.context=OpenPlast_proxy

https.base=http://grid.in2p3.fr/html/Private/https.context=Web_X509

file.base=file:///c:/tmp/file.base2=file:///c:/

JSAGA 20

Installer GUIInstaller GUI

JSAGA 21

LicensesLicenses

LGPL license– for the core engine and for

most adaptors

Optional licenses– for adaptors having external

dependencies, which license is not compatible with LGPL

– then, end-user must…• either accept the terms of the

license agreement,• or go back to previous screen

and uncheck the adaptor(s)

JSAGA 22

Example applicationsExample applications

Pandora Gateway

– the grid platform of the

company

• a leading Grid/Cloud Computing

Business Services Provider

– used in several grid projects,

including

SimExplorer• software for simulation

experiment management• includes a workflow engine that

uses JSAGA to run simulations on grids

• used by experiences in– biology

– computer science

– physics

– food industry

JSAGA 23


jCAE (Java CAE)• Computer Aided Engineering

with support for distributed computing facilities

Elis@• web portal for industrial

and academic production grids

• Elis@ is used in several grid projects, including

//

Calcul Intensif pour les LOgicielsde CAO Electronique et lesapplications embarquées

BioInformatique(protein sequence alignement...)

Plasturgy (simulation of injection…)

JSAGA 24

Other interoperability tools – JJSOther interoperability tools – JJS

Java Job Submission

tool to submit jobs on EGEE/EGI grid infrastructure– efficient and scalable

– resource selection based on the QoS observed while submitting jobs

http://cc.in2p3.fr/docenligne/269

uses JSAGA in order to hide middleware evolutions for– security

• globus proxy VOMS proxy

– data management• gridFTP SRM

– execution• LCG-CE CREAM-CE• WMS

used by D0 experiment to submit 15 000 jobs/day on– EGEE/EGI (mainly Europe)– Open Science Grid (USA)

JSAGA 25

Other interoperability tools – JUXOther interoperability tools – JUX

Java Universal eXplorer

multi-protocols file explorer

extensible– for protocols

• just put the JSAGA

plug-ins into the

directory "lib/".

– for viewers

http://cc.in2p3.fr/docenligne/821

JSAGA 26


JSAGA provides command line interfaces for…– security

• jsaga-context-init• jsaga-context-info• jsaga-context-destroy

– execution management• jsaga-job-run• jsaga-job-status• jsaga-job-cancel• jsaga-job-output

Also useful as code examples to start with SAGA

– data management• jsaga-cat• jsaga-cp• jsaga-ls• jsaga-mkdir• jsaga-mv• jsaga-rm• jsaga-rmdir• jsaga-stat• jsaga-test• jsaga-logical

JSAGA 27

Interoperability of SAGA implementationsInteroperability of SAGA implementations

Java SAGAJSAGASAGA-C++

Java Bindings for SAGAC++ Bindings SAGA

several implementations of SAGA– different languages (not necessarily)– different characteristics

• features (e.g. automatic/configurable context selection, timeout…)

• optimizations (e.g. caching of connections, meta-data, states…)

• design choices (e.g. high-level / low-level adaptor interfaces)

– different supported API extensions (e.g. SD, Advert, Messaging)– different supported technologies (adaptors)

how to use all these API & technologies simultaneously ?

JSAGA 28

Interoperability of Interoperability of JavaJava implementations implementations

Java SAGAJSAGA

Java Bindings for SAGA

System.setProperty("saga.factory", IMPL_NAME);

File f = FileFactory.createFile(url); // old codeFile f = FileFactory.getInstance().createFile(url); // new codeFile f = FileFactory.getInstance(IMPL_NAME).createFile(url); // new (thread-safe)

SAGA-C++

C++ Bindings SAGA

Application

+ getInstance()

+ getInstance() + getInstance()

+ getInstance()Application

JSAGA 29

Interoperability of Interoperability of PythonPython implementations implementations

C PythonC Python JythonJython

JPySAGA

Python Bindings for SAGA (PySAGA)

Java SAGAJSAGA


SAGA-C++

C++ Bindings SAGA

JySAGA

Application Application

CppySAGA

CppSAGA (legacy)

JSAGA 30

Interoperability : all together…Interoperability : all together…

C PythonC Python JythonJython

JPySAGA

Python Bindings for SAGA (PySAGA)

Java SAGAJSAGA


SAGA-C++

C++ Bindings SAGA

JySAGA

Application

CppySAGA

CppSAGA (legacy)

+ getInstance()

+ getInstance() + getInstance()

+ getInstance()S A G A

JSAGA 31

PerspectivesPerspectives

Develop plug-ins for batch systems– DRMAA-based

• Grid Engine

– CLI-based, with ssh• torque

• Grid Engine

Develop API extensions– Grid-RPC (from SAGA core)– Service Discovery API (a

SAGA extension)• with plug-ins for

technologies– BDII

• with plug-ins for schemas– GLUE– CIM

JSAGA 32

PerspectivesPerspectives

Job submission– principle

• describe user's job• submit it to the grid

– middleware select the execution host

– OGF standards• BES, DRMAA, SAGA

– examples

Remote Procedure Call (RPC)– principle

• deploy user's service on grid• call a procedure

– middleware select the service instance

– OGF standards• Grid-RPC, SAGA

– examples

job Grid-RPC

JSAGA 33

Questions ?Questions ?

Date post:	28-Dec-2015
Category:	Documents
Upload:	chad-simpson
View:	218 times
Download:	0 times

Status of Tsukuba, KEK, 21 September 2010 Sylvain Reynaud.

Documents