1

Condor BirdBathSOAP Interface to Condor

Charaka GoonatilakeDepartment of Computer Science

University College Londonc.goonatilake@cs.ucl.ac.uk

2

Background

3

Motivations Adding Web Service interfaces to Condor daemons

Facilitate the development of third-party applications capable of interacting with Condor (remotely).– E.g. build higher-level application specific scheduler that submits jobs to

multiple Condor pools based on application semantics

– These can be built using a wide range of languages/SOAP packages

– BirdBath has been tested on:• Java (Apache Axis, XSUL)• Python (ZSI)• C# (.Net)• C/C++ (gSOAP)

Condor accessible from platforms where its command-line tools are not supported/installed

4

The (Hypothetical) Big Picture

Schedd Scheddflocking

Schedd

SOAP

Globus/gridSAMCondor-G

Client

Site A Site B Site C

Site D

SOAP

5

Condor SOAP Interfaces

Central manager

Execution machine

Submission machine(s)

CollectorSchedd

Execution machine

Execution machine

Execution machine

Begin/commit/abort transaction Submit Job Obtain Job ClassAds Get/Send file Removed/hold/release job

Obtain resource ClassAds Obtain all daemon ClassAds

Shadow

Negotiator

Now available in Condor 6.7.5

6

Clients for BirdBath

WSDLs available that describe each daemon interface

SOAP libraries (stub code) can be generated automatically from WSDLs

Custom Client

holdJob()

SOAP library

Submission machine(s)

Schedd

7

Example: Query Collector

Obtain information (ClassAds) about all resources that have over 512MB RAM

collectorLocator = new CondorCollectorLocator();

collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”);

classAds = collector.queryStartdAds(“Memory>512”);

8

Example: Job Submission

1. Create New Transaction2. Create New JobId3. Send Files ({input files, binaries})4. Create Job Description5. Submit Job6. Commit Transaction

Transaction based process

Uses two phase commit to promote reliability and robustness

Files transferred in chunks – Base64 encoded SOAP attachments

DAGMan jobs supported

9

OMII Collaboration: gridSAM plugin Job Submission & Monitoring Web Service that submits to range of

resource managers (e.g. Condor, SGE, Globus) Uses JSDL (Job Submission Description Language) for

standardised job description

GridSAMGridSAM

CondorCondor SGESGE ForkFork ……

ClientClient

SOAP

JSDLJSDL

<?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://…"> <JobDescription> <Application> <Executable>/bin/echo</Executable> <Argument>hello world</Argument> </Application> </JobDescription></JobDefinition>

GridSAMGridSAM

CondorCondor SGESGE ForkFork ……

ClientClient

SOAP

JSDLJSDL

10

gridSAM plugin Existing Condor plugin

uses shell-based submission via command-line tools and monitoring of log files

GridSAMGridSAM

CondorCondor SGESGE ForkFork ……

ClientClient

Condor-WSCondor-WS Our plugin accesses Condor via the SOAP interfaces

Submissionmachine(s)

Schedd

Middle-tier

JSDLJSDLSOAP

SOAPClassAdClassAd

Building JSDL to ClassAd conversion libraries

11

Future Work Bundle plug-in into gridSAM

– Also provide separate JSDL to ClassAd conversion library

Extensions to other daemons (what functionality to expose?)– For example, allow checkpoint data to be retrieved through the shadow

in order to migrate jobs across pools

Roll out daemons on e-minerals mini-grid