COMP3019 Coursework:Introduction to GridSAM

Steve Crouch

s.crouch@software.ac.uk, stc@ecs

School of Electronics and Computer Science

A Reminder…

For materials (inc. slides, software, materials, handout):– http://www.ecs.soton.ac.uk/~stc/

COMP3019/

Coursework deadline: March 27, 2014 at 4pm

GridSAM

Job Submission and Monitoring

GridSAM Overview

What is GridSAM?– A Job Submission and Monitoring Web

Service– Allows compute resources to be

accessible securely over the internet– Uses Job Submission Description

Language (JSDL) to describe requirements for jobs

GridSAM Overview

What is GridSAM to the resource owners?– A Web Service to expose heterogeneous

execution resources uniformly Single machine through Forking or SSH Condor Portable Batch Scheduler (PBS) Load Sharing Facility (LSF) Sun Grid Engine Globus

– Acts as a client to these resources

GridSAM Overview

What is GridSAM to end-users?– A set of end-user tools and client-side APIs

to interact with GridSAM Web Services Submit and start jobs Monitor jobs Terminate jobs File transfer Client-side submission scripting Client-side Java API

GridSAM Server and Client Server comprised of:

– Web Services Container Tomcat/Axis - Web Services provider/container Optional WS-Security - handles authentication

(contains X509 certificate for security) Can be run over HTTP or HTTPS

– GridSAM Java web service

Client comprised of:– Client security ‘container’.

Contains Configuration for security (including X509 certificate security credentials)

Used by client applications to interact with services on OMII server

– GridSAM Java client

Web Service Definition

“A service is the logical manifestation of some physical or logical resources (databases, programs, devices, humans, etc) and/or some application logic that is exposed to the network”

Service interaction is facilitated by message exchanges

Web Services (WS) I

XML: Platform neutral mechanism to describe data

SOAP: Mechanism to describe message exchange

– Simple Object Access Protocol Not simple and nothing to do with Objects!

– Service Oriented Access Protocol Re-engineering of acronym to fit current use!

WSDL: Defines the service interface– Web Services Definition Language

Web Services (WS) II

Services have to reside in a supporting environment:– Called: hosting environment or container

e.g. Axis on Tomcat

– Marshals requests into and response out of the service– Service can discover local configuration parameters– Provides a standard infrastructure for service developers

Processing incoming requests & outgoing responses– Called: Message handlers– Manipulates elements of the message header

Primarily the SOAP header

– Handlers can be applied to message traffic into or out of the whole container or a specific service

Architecture

WS-Security

AXIS

TOMCAT

GridS

AM

…

Client

…

…

GridSAM – Publications & Enabled Activities

GridSAM Architecture A staged event-driven architecture

– Submission pipeline is constructed as a network of stages connected by event queues

– Each stage performs a specific action upon incoming events

Example Pipeline: Condor

Scenario: Condor Pool

GSIFTPGSIFTPFTPFTP WEBDAVWEBDAV HTTPHTTP…

Example using File Staging Objectives: submit simple job with

data input and output requirements and monitor progress

OMII GridSAM

Client

OMII GridSAM

Server

submit JSDL

monitor

OMII GridSAM

FTP Server1 output file

2 input files

JSDL Example <omii_client_home>/gridsam/data/examples/remotecat-staging.jsdl:

<JobDescription>

<JobIdentification> … </JobIdentification>

<Application>

<POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl-posix">

<Executable>bin/concat</Executable>

<Argument>dir2/subdir1/file2.txt</Argument>

<Output>stdout.txt</Output>

<Error>stderr.txt</Error>

<Environment name="FIRST_INPUT">dir1/file1.txt</Environment>

</POSIXApplication>

</Application>

…

JSDL Example<DataStaging>

<FileName>bin/concat</FileName>

<CreationFlag>overwrite</CreationFlag>

<Source>

<URI>ftp://ftp.do:55521/concat.sh</URI>

</Source>

</DataStaging>

<DataStaging>

<FileName>dir1/file1.txt</FileName>

<CreationFlag>overwrite</CreationFlag>

<Source>

<URI>ftp://ftp.do:55521/input1.txt</URI>

</Source>

</DataStaging>

<DataStaging>

<FileName>dir2/subdir1/file2.txt</FileName>

<CreationFlag>overwrite</CreationFlag> <Source> <URI>ftp://ftp.do:55521/input2.txt</URI> </Source></DataStaging><DataStaging> <FileName>stdout.txt</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</

DeleteOnTermination> <Target> <URI>ftp://ftp.do:55521/stdout.txt</URI> </Target></DataStaging></JobDescription></JobDefinition>