The Design, Usage, and Performance of GRUBER: A Grid Usage Service Level Agreement based BrokERing Infrastructure Catalin L. Dumitrescu & Ioan Raicu & Ian Foster Received: 9 March 2006 / Accepted: 4 November 2006 # Springer Science + Business Media B.V. 2007 Abstract We present GRUBER, a Grid Resource Usage service level agreement (uSLA) based BrokERing infrastructure, aimed at addressing the challenging issues that can arise within virtual organizations (VOs) that integrate participants and resources span- ning multiple physical administrative domains. In such environments, participants delegate to one or more VOs the right to use certain resources subject to local policy and service level agreements; each VO then uses those resources subject to VO policy. GRUBER supports the explicit representation, enforcement, and management of service level agreements (SLAs) concerning resource usage (uSLAs) that can serve as an objective organizing principle for controlled re- source sharing in distributed systems. uSLAs express how resources must be used over various time intervals and represent a novelty for the Grid domain. This paper provides a detailed overview of the GRUBER infra- structure, the evolution of its design to improve scalability, specifically the distribution of the resource brokering service, and the extended support for dynamic environments. We also present various results achieved over time that demonstrate both the utility and performance of GRUBER under various application workloads and scenarios. Key words controlled resource sharing . Grid computing . resource management resource and service brokering . usage service level agreements (uSLAs) Abbreviations uSLA usage Service Level Agreement VO Virtual Organization RM Resource Manager 1 Introduction GRUBER is an infrastructure for usage service level agreements (uSLAs [1]) specification, management and enforcement in any distributed environment in general; our implementation has been successfully J Grid Computing DOI 10.1007/s10723-006-9060-6 DO09060; No of Pages This work was carried out for CoreGrid IST project n°004265, funded by the European Commission. C. L. Dumitrescu (*) Mathematics and Computer Science, Delft University of Technology, Mekelweg 4, Delft, 2628 CD Delft, The Netherlands e-mail: [email protected]I. Raicu Computer Science Department, The University of Chicago, 5801 S. Ellis Ave., Chicago, IL 60637, USA e-mail: [email protected]I. Foster Mathematics and Computer Science Division, Argonne National Laboratory, 9700 S. Cass Ave., MCS/221, Argonne, IL 60439, USA e-mail: [email protected]
Transcript
The Design, Usage, and Performance of GRUBER:A Grid Usage Service Level Agreement based BrokERingInfrastructure
Catalin L. Dumitrescu & Ioan Raicu & Ian Foster
Received: 9 March 2006 /Accepted: 4 November 2006# Springer Science + Business Media B.V. 2007
Abstract WepresentGRUBER,aGrid Resource Usageservice level agreement (uSLA) based BrokERinginfrastructure, aimed at addressing the challengingissues that can arise within virtual organizations(VOs) that integrate participants and resources span-ning multiple physical administrative domains. In suchenvironments, participants delegate to one or moreVOs the right to use certain resources subject to localpolicy and service level agreements; each VO then usesthose resources subject to VO policy. GRUBERsupports the explicit representation, enforcement, andmanagement of service level agreements (SLAs)
concerning resource usage (uSLAs) that can serve asan objective organizing principle for controlled re-source sharing in distributed systems. uSLAs expresshow resources must be used over various time intervalsand represent a novelty for the Grid domain. This paperprovides a detailed overview of the GRUBER infra-structure, the evolution of its design to improvescalability, specifically the distribution of the resourcebrokering service, and the extended support fordynamic environments. We also present various resultsachieved over time that demonstrate both the utility andperformance of GRUBER under various applicationworkloads and scenarios.
Key words controlled resource sharing .
Grid computing . resource managementresource and service brokering .
usage service level agreements (uSLAs)
Abbreviations
uSLA usage Service Level AgreementVO Virtual OrganizationRM Resource Manager
1 Introduction
GRUBER is an infrastructure for usage service levelagreements (uSLAs [1]) specification, managementand enforcement in any distributed environment ingeneral; our implementation has been successfully
J Grid ComputingDOI 10.1007/s10723-006-9060-6
DO09060; No of Pages
This work was carried out for CoreGrid IST project n°004265,funded by the European Commission.
C. L. Dumitrescu (*)Mathematics and Computer Science,Delft University of Technology,Mekelweg 4, Delft,2628 CD Delft, The Netherlandse-mail: [email protected]
I. RaicuComputer Science Department,The University of Chicago,5801 S. Ellis Ave.,Chicago, IL 60637, USAe-mail: [email protected]
I. FosterMathematics and Computer Science Division,Argonne National Laboratory,9700 S. Cass Ave., MCS/221,Argonne, IL 60439, USAe-mail: [email protected]
deployed and used in Grid environments in particular.In these environments, each resource owner assigns apercentage of the owned resources to several con-sumers. The aggregated virtual resources assigned toa consumer can be used for the consumer immediatebenefits, may be shared among various consumer’sentities or provided further to other third parties. Inthe third scenario, the first consumer acts as amiddleman for resource aggregation and provisioning.The total resource amount each consumer can usedepends on the specified uSLAs at each level in theallocation chain.
The novelty of GRUBER consists in its capabilityto provide a means for enforcement of uSLAs andsupport for automated agents to select availableresources in a Grid-like environment. It focuses oncomputing resources such as computers, storage, andnetworks and higher-level services as well as anyGrid services. A VO is a group of participants whoseek to share resources for some common purpose.From the perspective of a single site in an environ-ment such as Grid3 [2, 69], a VO corresponds toeither one or several users, depending on localuSLAs. However, the problem is more complex thana cluster fair-share allocation problem, because eachVO has different allocations under different schedul-ing policies at different sites and, in parallel, each VOmight have different task assignment policies. Thisheterogeneity makes the analogy untenable whenthere are many sites and VOs.
The rest of this paper is organized as follows.Section 2 introduces the controlled resource sharingproblem, presents the common scenarios for uSLA-based sharing and related work. Section 3 presentsGRUBER’s design. Next, Sections 5 and 6 coverGRUBER’s performance in terms of request schedulingperformance and, respectively, its infrastructure perfor-mance for various scenarios. The paper ends withacquired lessons during this work and our conclusions.
2 Background Information and Related Work
The thread shared by most Grid environments iscooperative computing [3, 49]. The goal of thesesystems is to provide large-scale, flexible and secureresource sharing among dynamic collections of indi-viduals, institutions, and resources, also referred asvirtual organizations (VOs) [4]. In such settings, users
from multiple administrative domains pool availableresources to harness their aggregate power, to benefitfrom the increased computing power and the diversityof these resources, especially when their applicationsare customized for a specific computing configuration.
2.1 The Controlled Resource Sharing Problem
Resource sharing within large distributed systems (i.e.,P2P and Grid environments [13, 68]) that integrateparticipants and resources spanning multiple physicalinstitutions raises challenging issues [2]. Physicalinstitutions may wish to delegate to one or moreparticipants the right to use certain resources subject tolocal preferences and various agreements; each partic-ipant then wishes to enable those resources subject totheir own policy.
2.1.1 Usage Service Level Agreements (uSLAs)
We distinguish in our work between “resource accesspolicies” and “resource usage service level agree-ments” (uSLAs). Resource access policies typicallyenforce authorization rules. They specify the priv-ileges of a specific user to access a specific resourceor resource class, such as submitting a job to aspecific site, running a particular application, oraccessing a specific file. Resource access policiesare typically binary: they either grant or deny access.In contrast, resource uSLAs govern the sharing ofspecific resources among multiple groups of users.Once a user is permitted to access a resource via anaccess policy, then the resource uSLA steps in togovern how much of the resource the user is permittedto consume.
uSLAs [1, 5–8, 41, 47, 50] represent a novelty forGrids. The term uSLA was introduced to denote thisnew type of specific resource sharing for multi-typeresources. In the networking domain, usage policies,the equivalent of uSLAs, are used to address theproblem of bandwidth allocation based on specificrules. Such policies are specified by network admin-istrators and contain the rules for handling differenttypes of traffic. In this domain, a simple usage policyexample is “Email traffic is only allowed from outsidethe company’s servers only from a special mailgateway” [9–12, 48, 56].
C.L. Dumitrescu, et al.
2.1.2 Motivating Scenarios
In the following sub-sections we describe twoscenarios that motivate controlled resource sharing.These scenarios are based on the real OSG/Grid3environment and an external provider example thatoutsources service. In each scenario, common uSLAexamples are provided.
2.1.2.1 OSG/Grid3 Scenario OSG/Grid3 comprisestens of institutions and hundreds to thousands ofindividual investigators that collectively controlthousands of computers and associated storage systems[2, 14]. Each individual investigator and institutionparticipates in, and contributes resources to multiplecollaborative projects that vary in scale and formality.Figure1 depicts a graphical representation of the OSG/Grid3 sites.
In this environment, several VOs exist that arecomposed of users with various common interests andapplications. The most common ones are the USAT-LAS [15], Sloan Digital Sky Survey (SDSS) [16] andiVDGL [17] VOs. USATLAS VO users simulate thecollisions of protons on protons at 14 TeV for theLHC experiment; applications are composed of hun-dreds of embarrassingly parallel programs with largeinput/output files. The SDSS VO users measure thedistance to, and the masses of, clusters of galaxies inthe SDSS data set; applications are composed again ofmany components, but in this case they have input/output dependencies [18] that can be represented usingdirect acyclic graphs (DAGs). The iVDGL VO per-forms protein sequence comparisons at increasinglylarger scales (various size workflows in which asingle BLAST job has an execution time of about anhour – the exact duration depends on the CPU, reads
about 10–33 KB of input, and generates about 0.7–1.5 MB of output). OSG/Grid3 sites are sponsoredeither by different VOs or directly by the hostinginstitutions. Each site has uSLAs (expressed sometimesas “provide 30% of resources to USATLAS”) that areenforced by means of a local resource manager (RM).
For the OSG/Grid3 scenario, some of the require-ments include the provisioning of uSLAs capable ofexpressing situations both with and without conten-tion. Usually, resource providers (universities andlaboratories) and resource consumers (scientists fromdifferent domains) want access to these resourcespooled together according to various needs. Forexample, before important conferences we haveobserved that Grid utilization increases and job highercontention occurs, while during holidays most of thetime resources are free for long time intervals [19].These observations presented by Iosup et al. [19]motivate our introduction of a uSLA that ensures“whenever there is no contention users can use acertain amount of resources, while when contentionoccurs, the resources are allocated according to pre-defined rules that provide the incentives for Gridparticipation”. The following sharing example iswidely accepted by each individual site (or withdifferent variations in terms of the amount ofresources provided) [20]: “there are three types ofincoming jobs to balance: one from SDSS, one fromATLAS, and one from iVDGL. We call them SDSS-Prod, USATLAS-Prod, and IVDGL. We want SDSS-Prod and USATLAS-Prod to get an equal share ofavailable CPUs, but IVDGL should get a small frac-tion, of the resources, if there is contention (a 4th ofwhat the others get)”. Figure 2 shows this controlledresource sharing scenario for the University ofChicago resources.
2.1.2.2 Outsourcing Scenario In the second scenario,we envisage that a community outsources one ormore services to reduce deployment and operationalcosts. Community members acquire resources andservices from independent utility providers thatspecialize in providing those services (see Fig. 3).Service examples include scheduling predictionservices, monitoring services (MonALISA [22]), orcommunity authorization services (e.g., DOE certificateauthority [23]).
In this scenario, the service provider requires thatuSLAs express the amount of resources or services aconsumer is entitled to: “I provide 1000 requests forservice A from 7:00 to 16:00 for 1 month for anyremote user from Grid3” or “I accept 1,000 requestsfor service A from 7:00 to 16:00 on date X for anyremote user from Grid3” [24].
2.2 Related Work
Current solutions for controlling resource access inlarge scale distributed systems focus mainly on accesscontrol [25, 26, 40], but other groups have startedpursuing various paths for controlled resource sharing
[5, 6, 27–32]. Finer access control mechanisms focuson enabling resource providers in expressing addi-tional conditions about access and in delegatingpartial control to other entities.
For example, a Community Authorization Service(CAS) for access control policy management allowsresource providers to maintain ultimate authority overtheir resources, but spars them from day-to-day policyadministration tasks (e.g. adding and deleting users,modifying user privileges) [25]. However, accesscontrol dictates the operations an entity is entitled toperform on certain resource without any furtherrestrictions once access is granted. A policy basedscheduling framework for Grid-enabled resourceallocations is under development at the University ofFlorida [32]. This framework provides schedulingstrategies that (a) control the request assignment toGrid resources by adjusting resource usage accountsor request priorities; (b) manage efficiently resourcesassigning usage quotas to intended users; and (c)supports reservation based Grid resource allocation.
Other methods focus on economic models or match-making for controlled resource provisioning. In suchcases, mini-markets are built for resource brokering
and provisioning [27]. The Grid Service Broker, a partof the GridBus Project, mediates access to distributedresources by (a) discovering suitable data sources for agiven analysis scenario, (b) suitable computationalresources, (c) optimally mapping analysis jobs toresources, (d) deploying and monitoring job executionon selected resources, (e) accessing data from local orremote data source during job execution, and (f)collating and presenting results. The broker supportsa declarative and dynamic parametric programmingmodel for creating Grid applications [33].
Service level agreements [5, 21] focus on estab-lishing consumer-provider relationships concerninghow resources must be consumed. Such relationshipscan be designed by bi-lateral rules that are driven byinternal policies that govern any institution. Cremonais a such project developed at IBM as a part of theETTK framework [31]. It is an implementation of theWS-Agreement specification and its architectureseparates multiple layers of agreement management,
orthogonal to the agreement management functions:the Agreement Protocol Role Management, theAgreement Service Role Management, and theStrategic Agreement Management. Cremona focuseson advance reservations, automated SLA negotiationand verification, as well as advanced agreementmanagement.
GARA [29] represents modular and extensiblesystem architecture for resource reservations tosupport end-to-end applications QoS. It offers a singleinterface for reserving different types of resources(network, CPU, disk), and focuses on provisioninggeneric solutions and algorithms for different typesRMs (the heart of GARA). Reservations (and QoS)are specified through a specialized C-API, composedof client and Globus gatekeeper modules (admissioncontrol). GARA has a modular design, based on threelevels: low-level QoS RMs, a QoS component forinterfacing with the low-level RMs and provisioningthe common interface, and high-level libraries (at the
Fig. 3 Service outsourcing scenario
The design, usage, and performance of GRUBER
user level) for leveraging reservation synchronizationsfor user-level applications. The prototype supportsonly finite reservations, with three main classes ofelements: reservations, resources, and QoS elements.All communications client-agreement provider aredone through a specific API, and the underlyinglanguage for messages is RSL, with only one QoSquantitative parameter per reservation request.
SNAP [6] tries to overcome previous resource man-agements by providing a generic framework instead ofconsidering specialized classes of resources. Thegeneralized framework maps resource interactions ontoa well defined set of platform independent service levelagreements. SNAP represents the instantiation of thisgeneralized framework, which provides a managementinfrastructure for such SLAs. However, the entireapproach is generic enough and can be used beyondthe Grid domain and Globus Toolkit in particular. TheSLAs are categorized as: task service level agreements,resource level agreements, and binding service levelagreements. A minimal number of scenarios are alsointroduced, to provide a basic understanding how SNAPshould be used in practice: file transfer service, jobstaging with transfer service, resource virtualization.Also, for agreements realization, a supporting protocolis introduced that supports the SLA management.
2.3 The Motivation and Challenges for Our Work
None of the above systems fully address the problemof controlled resource management in distributedenvironments where many independent resourceproviders share their resources according to somelocal preferences and uSLAs. Mechanisms for sup-porting controlled resource sharing that allow coop-erative systems to provision resources based on localpreferences become a necessity in such cooperativeenvironments.
Important challenges for uSLA-based resourcemanagement can arise in practice from: the lack ofautomated mechanisms for uSLA discovery, publica-tion, or interpretation [34] to the complexity of theuSLA operations to be performed (to satisfy therequirements [35, 36]) of many resources and users.
Additionally, the increased scale of such distribut-ed systems calls for minimizing the need for humansupervision and for automating as many managementtasks as possible. In a systemwith over 1,000 providersand consumers, new uSLAs will occur thousands of
times more often than on a single system. In the sametime, the complexity of necessary services for uSLAdiscovery, management and enforcement will increasewith the scale of the system.
Increasing scale in large cooperative computingalso makes performance and reliability challengingand centralized systems are unlikely to be suitable forthese challenges. A single unified managementapproach over hundreds to thousands of consumersand providers can become a bottleneck in terms ofboth reliability and performance. Additionally, in awide area network, where short and transient failuresoften occur, the centralized approach can becomeinaccessible for varying time periods.
3 The GRUBER Infrastructure
GRUBER was specifically developed to address theproblem of uSLA discovery, management and en-forcement in large Grid environments. GRUBERaddresses issues regarding how uSLAs can be stored,retrieved, and disseminated efficiently in these typesof distributed environments and has been imple-mented by means of both the Grid Services (OGSI[37]) and Web Services (WS [38]) versions of theGlobus Toolkit (GT3, respectively, GT4). The mainelements of GRUBER are:
(a) The GRUBER specification language representsthe medium for uSLA specification. It supportswell-defined semantics and syntax for allocationspecification, as described next.
(b) The GRUBER engine implements various algo-rithms for detecting available resources andmaintains a generic view of resource utilizationin the Grid. Our implementation is a Grid servicecapable of serving multiple requests and basedon all the features provided by the GT3 or GT4container (authentication, authorization, state orstate-less interaction, etc).
(c) The GRUBER site monitor is one of the dataproviders for the GRUBER engine. It is com-posed of a set of UNIX and Globus tools forcollecting Grid status elements.
(d) The GRUBER site selectors are tools thatcommunicate with the GRUBER engine andprovide answers to the question: “Which is thebest site at which I can run this job?” Site
C.L. Dumitrescu, et al.
selectors can implement various task assignmentpolicies, such as round robin, least used, or lastrecently used task assignment policies.
(e) The GRUBER queue manager is a complexGRUBER client that must reside on a submittinghost. It monitors VO policies and decides how
many jobs to start and when. The GRUBERarchitecture is presented in Fig. 4.
Planners, work-runners, or application infrastructuresinvoke GRUBER site selectors to get site recommen-dation, while the GRUBER queue manager is respon-
Gruber engine
ES
Submit HostQueue
VOJobs
GRUBER QMand V-PEP
GRUBER Site Sel.
StateGrid Site
R
R
P
H
H
ES = External Scheduler (e.g., Condor G)
Queue State: Hold (H), Pending (P)
GRUBER Monitor
and S-PEP
3
4
6
5
71
2
GRUBER Messages:
1. & 2. Get site states and uSLAs
3. How many nodes are available?
4. Hold or release jobs
5. Select specific site
6. Where can this job run?
7. Run the job
Grid Site
1GRUBER Monitor
and S-PEP
1
2
Questions / Answers
Data / Job Submission
Gruber engine
ES
Submit HostQueue
VOJobs
GRUBER Site Sel.
State
R
R
P
H
H
ES = External Scheduler (e.g., Condor G)
Queue State: Hold (H), Pending (P)
3
4
6
5
7
1,2
GRUBER Messages: 1. & 2. Get site states and uSLAs 3. How many nodes are available? 4. Hold or release jobs5. Select specific site6. What can this job run?7. Run the job
8. Submit reservations
9. What sites can serve this request?
10. Submit request
WSRFClient
8
10
9
GRUBER QMand V-PEP
Questions / Answers
Data / Job Submission / Service Invocation
a
b
Reservations
Fig. 4 GRUBER architec-ture: low-level resource(left – A) vs. service(right – B) brokering
The design, usage, and performance of GRUBER
sible for controlling job starting time. If the queuemanager is not enabled, GRUBER becomes only a siterecommender, without the capacity to enforce any usageSLA expressed at the VO level. The site level usageSLA is still enforced by limiting the choices a job canhave and by means of removing a site for an alreadyover-quota VO user from the list of available sites.
Without a well-defined specification language foruSLAs, their interpretation may become misleading.We focus in the rest of this section on the GRUBER’slanguage semantic and syntax and describe the foursemantics for specifying resource usage constraintsand give examples for the proposed schemas.
3.1.1 GRUBER uSLA Semantics
The uSLA semantics capture how controlled resourcesharing is performed in the scenarios described inSection 2.1. They are generic enough to be alsoapplied for any distributed environment. The pro-posed semantics are named after their goals: no-limit,fixed-limit, extensible-limit, and commitment-limit[43–46].
& The no-limit uSLA is a statement that specifies nolimit in acquiring resources. Resources are ac-quired on a first come first executed basis.
& The fixed-limit uSLA specifies a hard upper limiton the fraction of resources Ri available to a VOi.A request to run a job is granted if this limit is notexceeded, and rejected otherwise. More precisely,a job requiring J resources is admitted if and onlyif Ci+J≤Ri, where Ci denotes the resourcescurrently consumed by VOi at the site. Note thatan admitted job will always be able to runimmediately, unless the resource owner oversub-scribes resources, i.e., ΣiRi >1.
& The extensible-limit uSLA also specifies an upperlimit, but this limit is enforced only under con-tention. Thus, under this SLA a job requiring Jresources is admitted if Ci+J≤Ri or ≤ Cfree, whereCi and Ri have the same meaning as before, andCfree denotes the site’s current unused resources.Note that because this policy allows VOs toconsume more than their allocated resources,
whether or not an admitted job can run immedi-ately may depend on the site’s preemption policy.While the fixed or extensible-limit uSLA are
sufficiently expressive for the OSG/Grid3 scenarios,their limitations become obvious when moving to thesecond scenario.
& The uSLA, the commitment-limit SLA, supportsthese more complex queries. It specifies two upperlimits, an epoch limit Repoch and a burst limitRburst, and specifies for each an associatedinterval, Tepoch and Tburst, respectively. A job isadmitted if and only if (a) the average resourceutilization for its VO is less than the corre-sponding Repoch over the preceding Tepoch, or (b)there are idle nodes and the average resourceutilization for the VO is less than Rburst over thepreceding Tburst. Both periods are modeled hereas recurring within fixed time slots. A providermay grant requests above the epochal allocation ifsufficient resources are available, but theseresources can be preempted if other parties withappropriate allocations request those resources ata later stage. More precisely, any job accepted bythe following algorithm is admitted, with thefollowing definitions:
1. if EAi > Repoch then2. return reject# Case 2: sub-allocated site3. else if Σk(BAk) + J < TOTAL and
BAi + J < Rburst then4. return accept# Case 3: over-allocated site
C.L. Dumitrescu, et al.
5. else if Σk(BAk) = TOTAL andBAi + J < Repoch then
6. return accept7. else8. return reject
This uSLA can be extended further by introducingan unlimited number of sharing intervals, whichmakes it generic enough to express any requirementsin practice. I note that the fixed limit and extensible-limit uSLA can be expressed as particular cases of thecommitment-limit uSLA.
Based on the UNIX quota system, the same fouruSLAs can be implemented with success in the case ofdisk space. However, in this case once a file is saved at asite and the allocation is higher than the uSLA limitallows (extensible-limit and commitment limit cases),the space cannot be preempted without evicting theviolating files to other sites. Our approach builds on theUNIX quota system, which prevents one user on a staticbasis from using more than his hard limit (but it stillconsiders soft and hard limits similarly to the commit-ment limit). More precisely, for scheduling decisions alist of site candidates that are available for use by a VOi
for a job with disk requirements J is built by executingthe following logic, with the following definitions:
s Site Setk index for any VO != VOi at site sIPi Epoch uSLA for VOi at site sISPi Instantaneous uSLA for VOi at
site sIAi Instantaneous Resource Usage
for VOi at site sTOTAL Upper limit allocation on the
1. if IAi IPi for VOi at site s2. return reject# Case 2: over soft-limit site by VOi
3. if IAi > ISPi and time < grace periodfor VOi at site s
4. if Σk(IAk) < s.TOTAL-J && IAi + J < IPi
then6. return reject# Case 3: un-allocated site
7. if Σk(IAk) < s.TOTAL-J&&IAi + J < IPi
then8. return accept
The final type of resources we consider is a Gridservice (a high-level resource). Because Grid servicesare difficult to quantify in term of their utilization (aweather service and a matrix multiplication serviceare difficult to compare in terms of resource con-sumption without a through service performancemodel analysis), we maintain the CPU semanticsunchanged for the uSLAs, but with different utiliza-tion metrics: instead of CPU utilization, the number ofrequests a client can perform on a certain service isconsidered for the uSLA algorithms. While thisapproach may seem an over-simplification, the endresult is similar: a service provider states by means ofthe uSLAs how many requests a certain consumer canperform on its resources.
Based on this approach, the algorithm to supportscontrolled Grid service sharing is identical with theone for controlled CPU sharing. We present avariation of the previous algorithm that allowsadvance service reservations (a request can be sentin well in advance of its starting time). It applies apre-specified uSLA on each “future” sub-intervalsresulted from a variation in terms of either servicerequests or allocations. The algorithm for acceptingnew advance reservations is introduced next, where:
R resource requestAj allocated resourcesRj requested resourcesAllocations set of accepted allocationsRequests set of already accepted
1. response = accept# Stores availability on the considered
interval2. S = empty
# Identify all requests overlappingcurrent request
# (save their start/end times andrequested quantities)
3. foreach Rj in Requests do4. if Rj time overlaps R time then
The design, usage, and performance of GRUBER
5. save S, R j.start, + Rj.attributes6. save S, Rj.stop, − Ri.attributes7. fi8. done
# Identify all allocations overlappingcurrent request
# (update accordingly previous valueswith these allocations)
9. foreach Aj in Allocations do10. foreach Tj in S do11. if Aj overlaps Tj then12. save S, Tj, + Aj.attributes13. fi14. done15. done# Check constraints (available resources)16. compute Request = R.attributes17. foreach Availability in S do# Apply the according uSLA algorithm
(hard limit example here)18. if (Request > Availability)
response = reject19. done20. if response == accept then21. update accordingly one of the
overlapping uSLAs22. add R to Requests23. fi24. return response
3.1.2 GRUBER uSLA Syntax
We have considered two syntaxes for uSLAs: a simplerone based on allocations and one based on WSLA.
3.1.2.1 Tuple-based Syntax Our starting point for thefirst approach is the Maui [51] syntax for specifyingallocations. Maui supports three types of fair sharelimits: “at least limit”, “average limit” and “at mostspecification”. In the first case, when the utilizationfor a group goes below the limit, the mechanismincreases the priority of the jobs from that group(expressed as a real number preceded by a “+”). In thesecond case, whenever the utilization for a group isdifferent from the limit, the fair share mechanismeither increases the priority (for under-utilization) ordecreases the priority (for over-utilization) of jobsfrom that group. In the final case, when the utilization
for a group goes above the limit, the fair sharemechanism decreases the priority of jobs from thatgroup (expressed as a real number preceded by a “−”).Our first syntactic form is represented as a set ofallocations of the form:
ANY ::= matches any name* ::= means instantaneous- ::= means not specified
However, this syntax has its limitations forexpressing sharing rules about resources of differenttypes. First, this syntax does not provide a mechanismfor specifying monitoring requirements. Second, itdoes not support the specification of complex con-ditions (AND, OR, etc.).
3.1.2.2 Schema-based Syntax We considered as asecond approach, a uSLA syntax based on the WS-Agreement specification, to take advantage of itshigh-level structure SLA specification and ofavailable parsers. The objective of a WS-Agreementspecification is to provide standard means forestablishing and monitoring service agreements. Thespecification draft comprises three major elements: adescription format for agreement templates, a basicprotocol for establishing agreements, and an interfacefor monitoring agreements at runtime.
For this syntax, we use a schema that includesfrom the WS-Agreement specification support forresource monitoring metrics and goal specifications[31, 43, 52]. The resource monitoring metricsdescribe how various utilizations must be measuredor how these quantities should be collected froman underlying monitoring system. A goal specifi-cation provides support for describing the targeted
C.L. Dumitrescu, et al.
allocations in a form that can be parsed by automatedagents. The other elements (i.e., obligations and han-dlings violation) were considered beyond the scopeand capacity of current site and VO RMs.
The schema of this grammar is described next.First, the monitoring metric element defines how acertain resource metric required for a guaranteeshould be measured.
The next element of the grammar, MonitoredType,describes the entire list of monitored metrics requiredin enabling the considered uSLA. This list can havezero or more of required metrics that must bemonitored.
<!-– Monitored Type -–><xsd:complexType name="MonitoredType">
The goal element describes the conditions underwhich a resource is provided. It uses the constraintelement for defining conditions (with LessEqual,Equal and GreaterEqual corresponding to the seman-tics introduced earlier for – , <space>, + signs, whileRange has a special meaning for time specifications):
Similarly to a MonitoringType element, a Guaran-teeElement contains a list of all guarantees that aresource provider agrees to when providing theresources.
<!–- Guarantee Type -–><xsd:complexType name="GuaranteeType">
The OSG/Grid3 example is represented using thissyntax as follows. Three metrics are monitored (bymeans of MonALISA, for example, and these valuesare retrieved from a certain URL): CPUBurst-Met-USATLAS, CPUBurst-Met-SDSSS and CPUBurst-Met-iVDGL.
The GRUBER engine represents the main componentof the brokering infrastructure. We often call through-out this paper an engine instance a decision point(DP). Usually, all the other elements in the brokeringinfrastructure rely on communicating with one DP toperform their operations. It maintains a view of theavailable resources at each Grid site and invokesvarious algorithms for deciding the resource avail-abilities. Two main resource types are considered for
C.L. Dumitrescu, et al.
brokering: low-level resources (e.g., CPU, disk, andnetwork) and services (e.g., any Grid-enabled ser-vice). The GRUBER algorithms address differentlythese two resource types, conditioned mainly by thelocal site managers in each case (Condor [53], PBS[54], etc. for low-level resources vs. ARESRAN [24],SAML [55], etc for services).
3.2.1 CPU Brokering
GRUBER decides which consumers are best for arequest from a CPU availability point of view byimplementing the following logic:
& If there are fewer waiting jobs at a site thanavailable CPUs or an extensible-limit uSLA is inplace, then GRUBER assumes the job will startright away if an extensible-limit uSLA is in place.
& If there are more waiting jobs than available CPUsor if an extensible-limit uSLA is not in place, thenGRUBER determines the VO’s allocation, thenumber of jobs already scheduled, and the RMtype. Based on this information, if the VO is:
– under its allocation, GRUBER assumes that anew job can be started (in a time that dependson the local RM type: for example, in theCondor RM case a 15 min delay period mightbe enforced before a newly free machine canbe acquired)
– over its allocation, GRUBER assumes that anew job cannot be started (the running time isunknown for the jobs already running)
More precisely, for any job placement CPU-baseddecision a list of available sites is built and providedunder the following algorithm to find those sites inthe site set G that are available for use for job J (Jkeeps place for job characteristics as well in thefollowing algorithm) from the VO number I (with thefollowing definitions):
G Grid Site Set ;S Matching Site Set;
algorithm get-avail-sites_cpu inputs(sites G, VOi, job J) returns S
1. S = empty2. for each site s in G do
# Apply one of the algorithmsintroduced in Section 4.1
3. if commitment-uSLA_CPU(J, VOi, s) == accept then
4. add (s, S)5. else6. next8. return S
3.2.2 Disk-space Brokering
Disk space brokering introduces additional complexi-ties in comparison to CPUs, if job files cannot befetched from some generic replication service. If anentitled-to-resources job becomes available, it isusually possible to delay scheduling other jobs, or topreempt them if they are already running. In contrast, afile that has been staged to a site cannot be “delayed,” itcan only be deleted. Yet deleting a file that has beenstaged for a job can result in livelock, if a job’s files arerepeatedly deleted before the job runs. So far, we haveconsidered a UNIX quota-like approach. Usually,quotas just prevent one user on a static basis fromusing more than his limit. There is no adaptation tomake efficient use of disk in the way a site CPU RMadapts to make efficient use of CPU (by implementingmore advanced disk space management techniques).The set of disk-available site candidates is combinedwith the set of CPU-available site candidates and theintersection of the two sets is used for furtherscheduling decisions. More precisely, for schedulingdecisions a list of site candidates that are available foruse by a VOi for a job with disk requirements J, interms of provided disk space, is built by executing thefollowing logic, with the following definitions:
G Grid Site Set ;S Matching Site Set ;F File requirements for Job J
algorithm get-avail-sites_disk inputs(F, VOi, G) returns S
1. S = empty2. for each site s in G do# Apply one of the algorithms
introduced in Section 4.13. if commitment-uSLA_disk
(F, VOi, s) == accept then
The design, usage, and performance of GRUBER
4. add (s, S)5. else6. next7. return S
3.2.3 Service (Higher-level Resource) Brokering
For service brokering, GRUBER uses an internalrepresentation based on a variable time-slot represen-tation, as introduced by Wolf et al. [57], and eachuSLA has assigned its own time intervals. Thisstructure allows for an unlimited number of uSLAsand unlimited or unknown periods of time intervals.The processing logic we propose and evaluate isbased on the following algorithms, where:
G Grid Site Set ;S Matching Site Set ;R Request ;
algorithm get-avail-sites_serviceinputs (R, VOi, G) returns S
1. S = empty2. for each site s in G do# Apply one of the algorithms
In addition to the engine, the GRUBER infrastructurerelies on various site monitors and queue managers.We describe in this section the current solutionsalready implemented and available for integrationwhenever GRUBER must be deployed.
3.3.1 uSLA Enforcers (PEPs) and Observers (POPs)
We describe here three solutions for site uSLAmanagement and enforcement as implemented forGRUBER. The first solution considers the case of
simple RMs unable to handle any type of arbitrationamong concurrent requests for resources. The secondsolution instead takes in consideration advanced sitemanagers capable of enforcing complex uSLAs thatmust be discovered and published mainly at the Gridlevel. The final solution targets instead Grid servicemanagement.
Solution 1 (Stand-alone S-PEP) Our first solutiondoes not require a usage policy-cognizant cluster RM.It works with any primitive batch system that has atleast the following capabilities: provide accurateusage and state information about all scheduled jobs,job start/stop/held/remove capabilities, and runningjob increase/decrease priority capabilities. The S-PEPsits at the level of the local scheduler(s), checkscontinuously the status of jobs in all queues andinvokes management operations on the cluster RMwhen required to enforce policy. In more detail, the S-PEP gathers site uSLAs, collects monitoring informa-tion from the local schedulers about cluster usage,computes CPU-usage parameters, and sends com-mands to schedulers to start, stop, restart, hold, andprioritize jobs. Our approach provides priority-basedenforcements. The processing logic of our prototypeS-PEP is based on the algorithms presented below,with the following definitions:
EPi Epoch allocation policyfor VOi
BPi Burst allocation policyfor VOi
Qi Set of queues with jobsfrom VOi
BAi Burst Resource Allocationfor VOi
EAi Epoch Resource Allocationfor VOi
TOTAL possible allocation on thesite
Over-quotajob
job of VOj
procedure s-pep1. while (true) do2. sleep N # (seconds)3. foreach VOi with EPi
# Case 1: available and BAi < BPi
4. if Σ(BAk)<TOTAL & BAi<BPi &Qi has jobs then
C.L. Dumitrescu, et al.
5. release job J from some Qi#(e.g., FIFO scheduling policy)
# Case 2: res. contention: fill EPi
6. else if Σ(BAk) == TOTAL & BAi < EPi
& Qi has jobs then7. if j exists & BAj >= EPj then8. suspend an over-quota job Qj
9. release job J from some Qi#(e.g.,FIFO scheduling policy)
10. foreach VOi with EPi
11. if EAi > EPi then12. suspend jobs for VOi
from all Qi
As a further clarification, BA or EA represents theshare actually utilized by a VO. BP or EP representsupper values for these utilized shares. When BP or EPincreases for example, the VO is entitled to moreshares starting with the moment of the change. Animportant novelty of this S-PEP over a cluster RM isits capability to keep track of jobs under several RMsand to allow the specification of more complex usagepolicies without the need to change the actual clusterRM implementation.
Solution 2 (Policy-aware Scheduler) The secondsolution was developed and implemented with suc-cess in the context of the Grid3 environment. Wedecoupled the functionalities of the S-PEP in twomajor components and mapped to existing solutions:a standalone site policy observation point (S-POP)and the cluster RMs for resource allocation control. Inthis case, we assume that the cluster RM is able toenforce by itself the desired usage policies, which areprovided by means of our S-POP module. Examplesof such cluster RMs are Condor [58], Portable BatchSystem [59], and Load Sharing Facility [60], widelyused on Grid3 [2]. The S-POP’s main functions are tooptionally provide and to translate to/from the RMlocal usage policies, and to monitor the actualresource utilization. An advantage of this approachis that the local administrators do not have to use anadditional Grid component in managing their clusters.
At the RM level (sites), owners state how theirresources must be allocated and used by differentconsumers. This statement represents the high-levelGOAL an owner (MP) wants to achieve. Siteadministrators map MPs to different software RMs’semantics/syntax. The end product is a set of RM
configuration files, named the local POLICY orsystem configuration (SC ). For automated consump-tion, site policies are translated from SCs by auto-mated tools into an abstract usage policy (AP ) set,i.e., the uSLA syntax/semantic described above. SCdescriptions are collected from the site RM config-urations, filtered and, after translation, publishedthrough a specific monitoring system, e.g., VO-Ganglia [61], MonALISA [22, 34] or GRUBER-SiteMonitor [43, 62].
We have proposed three levels of description forthe statement “site X gives ATLAS 30% over amonth:”
& MP: a description of a site manager’s policy forthe site, e.g., MP (VOs) = “give ATLAS 30% overa month”. I assume that simple English statementsdescribe the MP set (site level);
& SC: an RM configuration: SC (VO) = <number ofnodes, scheduler-type, scheduler-config>, whichis written by the site administrators during the RMconfiguration process (site level);
& AP: An SC (VO) translated into a uSLA repre-sentation: AP (VO, Site) expresses SC (VO) in ascheduler-independent format and is publishedthrough a monitoring tool for resource brokeringor other automated tools (Grid level).
The key point is determining how an SC maps toan AP. We have achieved this part by providingspecialized SC-translators for each type of SCsupported by a specific RM. The translator parsesthe configuration files or queries the resource providerRM, and outputs the resulting configuration directlyinto AP form. The information flow for this process iscaptured in a graphical way in Fig. 5.
Solution 3 (Stand-alone Service S-PEP) For Gridservices, the solution adopted for uSLA managementis the ARESRAN prototype. ARESRAN is a GT4service for uSLA and reservation management,specification and enforcement at the level of a singlesite based on the Globus technology [38]. ARESRANprototype is based on the authorization schemesimplemented by the GT4, the so called PolicyDecision Point (PDP). GT4 allows a chain of PDPsto be configured internally, with each PDP evaluatingto an independent decision [63]. The authorizationengine of the framework then uses a policy combina-tion algorithm to combine the decisions returned by
The design, usage, and performance of GRUBER
each PDP to render a final decision. We havedeveloped two specific ARESRAN PDPs – one formanaging service reservations and one for lower levelresource reservations such as compute nodes that aremanaged by the WS-GRAM service.
The overall ARESRAN architecture is described inFig. 6. Whenever a service request comes in for aservice managed through ARESRAN, its PDP stepsin and verifies if the request is acceptable or not.Now, at the Grid level, GRUBER collects uSLAsfrom all individual ARESRAN services and providesbrokering services for the consumers.
The main components here are the ARESRANService, the ARESRAN PDPs and the ARESRANReservation Database. These three components inter-
communicate in order to ensure that requestedresources or services are used accordingly. Thespecific details of these components are:
& Service: represents the reservation and uSLAengine of our prototype. Every time a newreservation is requested, the engine is invoked toverify if the new reservation can be honored. Theverification procedure takes into account variousinformation from the ARESRAN database andreturns either ACCEPT, DENY or PROBABLY. Ifthe reservation request is accepted, it is also savedto the ARESRAN database
& Database: stores reservations, uSLAs as well asinformation about the requests in progress. In thismanner, ARESRAN has a complete view aboutthe utilization of the services and resources that itmanages. So far, the database is implemented onlyin memory, but future enhancements target theusage of a specific database system for this part ofARESRAN. Every time a reservation is served,various statistics are also saved, such as: requesttime, running time, remote client, etc. Based on suchinformation, we believe the ARESRAN enginecould later perform probabilistic reservations
& PDPs: authorize based on the rules stored indatabase about various services and resources.Each PDP returns either REJECT or ACCEPT.When a request is accepted (associated with areservation), the database is also updated to reflectthe current state of the managed resourcesFig. 6 ARESRAN architecture
Fig. 5 Correlations MP,SC, and AP
C.L. Dumitrescu, et al.
3.3.2 Queue Managers and VO-level uSLAEnforcement
GRUBER queue managers reside at the submissionhosts and are responsible for determining how manyjobs per VO or VO group can be scheduled at acertain moment in time and when to release them.Usually, a VO planner is composed of a job queue, ascheduler, and job assignment and enforcementcomponents. Here, the last two components are partof GRUBER and have multiple functionalities. Queuemanager components answer: “How many jobsshould group Gm of VOn V be allowed to run?”and “When to start these jobs?” The queue manager isimportant for uSLA enforcement at the VO level andbeyond. This mechanism also avoids site and resourceoverloading due to un-controlled submissions. TheGRUBER queue manager implements the followingalgorithm (with the assumption that all jobs are heldinitially at the submission host), with the followingdefinitions:
J = Job Id ;Q = Job Queue ;S = Site Set ;G = All Site Set ;VO = Mapping Function jobId –> VO
procedure v-pep1. while (true) do2. sleep N # (seconds)3. if Q != empty then4. get J from Q5. else6. next7. S = get-avail-sites(G, Vo(J), J)8. if S != empty then9. release J from Q
3.3.3 Site Selectors
While GRUBER provides mainly brokering servicesin distributed environments, we have extended it toprovide also primitive scheduling services. By intro-ducing site selector tools, the GRUBER infrastructureis able to select from the set of available sites the bestsite for running a job according to various schedulingpolicies. The site selector components answer thequestion: “Where is best to run next?” These siteselectors are invoked by any automated tool requiring
a list of available sites for scheduling (e.g., Euryale,KOALA or WMS).
The four main policies implemented by GRUB-ER’s site selectors are: random assignment (G-RA),round-robin assignment (G-RR), least-recently-usedassignment (G-LRU), and last-used assignment (G-LU). Each of these implements the scheduling policydescribed by their name. “GRUBER observant” (G-Obs) is a custom site selector that associates a furtherjob to a site each time a job previously submitted tothat site has started, but without bypassing the uSLAenforced at the site. In effect, this fifth site selectorfills up what a site provides by associating jobs untilsite’s limit is reached.
3.4 GRUBER Extensions
Several extensions were performed over time forGRUBER. The most notable ones are described inthis section: high-level resource brokering, distributeduSLA management, client scheduling among thedecision points and human interfacing for easy systemchecking.
3.4.1 DI-GRUBER (DIstributed GRUBER)
Managing uSLAs within environments that integrateparticipants and resources spanning many physicalinstitutions can become a challenging problem inpractice. A single unified uSLA management decisionpoint providing brokering decisions over hundreds tothousands of jobs and sites can become a bottleneckin terms of reliability as well as performance. DI-GRUBER, an extension of the GRUBER infrastruc-ture, was developed as a distributed Grid uSLA-basedresource broker that allows multiple decision points tocoexist and cooperate in real-time. DI-GRUBERtargets to provide a scalable management service withthe same functionalities as GRUBER but in adistributed approach. [64] It is a two layer resourcebrokering service (see Fig. 7), capable of workingover large Grids, extending GRUBER with support formultiple scheduling decision points that cooperate byperiodically exchanging various status information.
3.4.2 WS-Index Service Support
As already described, the ability to bring online adecision point is important in a large and dynamic
The design, usage, and performance of GRUBER
Grid. Our previous work on GRUBER and DI-GRUBER [64] assumed a static environment, andhence did not offer flexibility for dynamic environ-ments. The DI-GRUBER implementation evolved tosupport dynamic environments through the use of theWS-Index Service [39] provided with GT4; oursolution uses the functionalities offered by the WS-Index Service for service registering and querying. Inour implementation, each DI-GRUBER decisionpoint registers with a predefined WS-Index Serviceat startup, while it is automatically deleted when itvanishes. Further, all decision points and clients can
use this registry to find information about the existinginfrastructure and select the most appropriate point ofcontact. Whenever we use the term “most appropri-ate”, we refer to metrics such as load and number ofclients already connected. A decision point schedul-ing policy can easily be incorporated by each client,the default one being least-used policy (in terms ofnumber of clients). In Fig. 8 is presented a viewachieved by means of the GRUBER graphical console.Now, whenever a new client boots (at a submissionpoint), it can easily find which decision point is mostappropriate. Also, whenever a decision point stops
responding to a client, this client automatically queriesthe registry and selects a different point of contact.
We consider that this approach is less error-pronethan the static solution, and, additionally, it offers thesupport for dynamically bootstrapping new decisionpoints whenever new ones register with the WS-IndexService. While we have not implemented this facility100% yet, a human operator can easily perform suchan operation (starting a new GT4 container with a DI-GRUBER decision point web service deployed).Additionally, if a pool of decision points are main-tained in background and forced to register with theWS-Index Service only when needed, the operationwould be 100% automated.
While dynamic DI-GRUBER decision point boot-strapping might be difficult to automate in a genericenvironment, the solution we have devised for suchenvironments is semi-automatic. Every time a clientfails to communicate or to connect with a decisionpoint, it registers with the WS-Index Service a requestfault. These faults are then used by a human operatorin order to bring up new DI-GRUBER instances andstabilize the brokering infrastructure whenever re-quired. As future work, we envisage to fully automatesuch operations by means of Grid technologies wherepossible. Such faults can be consumed by a specialized
entity that can dynamically start new decision pointsby means of WS-GRAM [38]. For example, in theOSG/Grid3 scenarios considered here, whenever thecondition for bringing up a new decision occurs, aspecial job is submitted to a site and a new containeris started. In a more specialized context, a dedicatedpool of nodes can be used for bringing up suchdecision points and really used only when necessary.For the remaining time, the dedicated pool might beused for other Grid specific operations.
3.4.3 Human Interfacing
Firstly, accurate monitoring is important if we are tounderstand how the framework actually performs indifferent situations (the verifier concept introduced byDumitrescu et al. [1]). As a first step towards thisgoal, we have developed mechanisms for measuringhow resources are used by each VO and by the Grid,overall. The monitoring tool built for GRUBER is agraphical interface able to present the current alloca-tions and uSLAs at each decision point and over inthe managed Grid infrastructure. This interface con-nects to a decision point, collects the local or genericview and presents it in an easy to visualize mode (seeFig. 9).
Fig. 9 Resource allocationscenario
The design, usage, and performance of GRUBER
In order to avoid gathering large amount ofinformation, we also introduced various summationoperations for different metrics. Practically from ahuman verifier point of view, this interface answersthe question “Are uSLAs adequately enforced by eachdecision point?” and “What are the utilizations andallocations of different resource in the Grid?”
Also, the same graphical interface provides supportfor uSLA specification at group, VOs and site levels.The uSLAs can be entered and associated either witha site, a VO or a group. In another approach, variousWS-Agreement like rules can be specified that areparsed when required to perform various job steeringoperations. Further, all uSLAs specified at a certaindecision point are distributed to all other decisionpoints if not marked as private. While this solutionseems not very scalable (when going towardshundreds of decision points), we assume that for aGrid one hundred times larger than today Grid3, it issufficient. As an additional note, uSLAs are associat-ed with the decision point that distributed them andthey can be deleted only by the same point ofdecision.
4 The Usage of GRUBER
GRUBER was tested with success in the OSG/Grid3[2] environment for raw resource brokering forvarious workloads. Also, we tested its performancefor higher-level (Grid) service brokering on an ad-hocGrid testbed deployed on PlanetLab [70].
4.1 Testing Environment
The first testing environment, OSG/Grid3, uses Eur-yale [65] as a system to run jobs over Grid3/OSG [2].Euryale uses Condor-G [30] (and Globus’ WS-GRAM [38]) to submit and to monitor jobs at sites.It takes a late binding approach in assigning jobs tosites, meaning that site placement decisions are madeimmediately prior to running the job, rather than in anearlier planning phase. Euryale also implements asimple fault tolerance mechanism by means of job re-planning whenever a failure is discovered. We use theEuryale planner as our job submission tool while alsointerfacing with the GRUBER infrastructure. A toolcalled DagMan executes the Euryale prescript andpostscript. The prescript calls out to the external
GRUBER site-selector to identify the site on whichthe job should run, rewrites the job submit file tospecify that site, transfers necessary input files to thatsite, registers transferred files with the replica mech-anism, and deals with re-planning. The postscript filetransfers output files to the collection area, registersproduced files, checks on successful job execution,and updates file popularity.
DAS-2 environment [66], a wide-area distributedcomputer of 200 Dual Pentium-III computer nodes,represents a second example where GRUBER can beused with success. The environment is built out ofclusters of workstations, which are interconnected bySurfNet, the Dutch university Internet backbone forwide-area communication, whereas Myrinet, a popu-lar multi-Gigabit LAN, is used for local communica-tion. The Grid scheduling technologies used in thisenvironment are the Sun Grid Engine (SGE) [67] andKOALA [66]. KOALA has been designed andimplemented by the PDS group in Delft in the contextof the Virtual Lab for e-Science (VL-e) researchproject. The main feature of KOALA is its support forco-allocation, that is, the simultaneous allocation ofresources in multiple clusters to single applicationswhich consist of multiple components. Applicationsare executed using KOALA runners, which must beinstalled on all submission hosts and which handlesthe reservations of resources and the submission ofapplications. KOALA provides different kinds ofrunners and each runner is specialized for a differentapplication type. These runners can be directlyinterfaced with GRUBER to take advantage of uSLAsfor scheduling decisions. In this approach KOALA’srunners invoke the GRUBER site selectors for adetailed list of available sites and their specificresources.
4.1.1 Usage Performance Metrics
We have used five metrics to evaluate the effective-ness of GRUBER raw-resource brokering perfor-mance over OSG/Grid3. We note that these metricsare independent, in the sense that a smaller totalexecution time does not imply a higher speedup.
& Comp: the percentage of jobs that completesuccessfully.
Comp ¼ Completed Jobð Þ�#jobs � 100:00
C.L. Dumitrescu, et al.
& Util: average resource utilization, the ratio of theper-job CPU resources consumed (ET i) to thetotal CPU resources available as a percentage:
Util ¼X
i¼1::NETi
�#cpus � $ t� � � 100:00
& Delay: average time per job (DT i) that elapsesfrom when the job arrives in a resource providerqueue until it starts:
Delay ¼X
i¼1::NDTi
�#jobs
& Time: the total execution time for the workload.& Speedup: the serial execution time to the Grid
execution time for a workload. We note thedefinition implies a comparison between therunning times on similar resources – practically,the first time in sequence and the second time inparallel.
For the service brokering tests, we used twoadditional metrics: # of requests and response time(for various operations as presented in Table 3).
4.1.2 Environment Characteristics
We performed all raw-resource brokering experimentson Grid3 (December 2004), which comprises around30 sites across the U.S., of which we used 15. Eachsite is autonomous and managed by different localresource managers, such as Condor, PBS, and LSF.Each site enforced different uSLAs which werecollected through GRUBER-SiteMonitor and used inscheduling workloads. We submit all jobs within theiVDGL VO, under a VO uSLA that allowed amaximum of 600 CPUs in parallel (fixed-limit uSLAat the VO level). Furthermore, we used also a fixed-limit uSLA at the VO group level where eachindividual workload corresponded to separate iVDGLgroup; the uSLA enforced was that any group can notget more than 25% of iVDGL CPUs, i.e., 150.
For the service brokering measurements, wedeployed 10 ARESRAN managed WSRF serviceson PlanetLab. Each instance was providing Gridservice for consumption while GRUBER was usedfor service brokering for the ad-hoc Grid. Eachservice ran inside a GT4 container on a PlanetLabnode, except the GRUBER and WS-Index Servicethat ran inside the same container on a node at theUniversity of Chicago. PlanetLab nodes were Linux-
based PCs connected to the PlanetLab overlaynetwork with worldwide distribution. All nodes areconnected via 10 Mb/s network links (with 100 Mb/son several nodes), have processor speeds exceeding1.0 GHz IA32 PIII class processor, and at least512 MB RAM. The clients were situated in the samenetwork as the GRUBER node (UofC LAN), and ranon a node having an Intel Pentium 2.0 GHz processor,1 GB of memory, 100 MBit/s network connectivity,and running the Linux-SuSe9.1 OS. We must alsonote that the PlanetLab configuration between the twoexperiment cases was different (and also a fewmonths difference in time) – as an explanation forthe difference in the overall performance.
4.1.3 Workloads and Settings
We used a single job type in all our raw-resourcebrokering experiments, the sequence analysis programBLAST. A single BLAST job has an execution timeof about an hour (the exact duration depends on theCPU), reads about 10–33 KB of input, and generatesabout 0.7–1.5 MB of output: i.e., an insignificantamount of I/O. We used this BLAST job in twoworkload different configurations. In 1×1 K, we havea single workload of 1,000 independent BLAST jobs,with no inter-job dependencies. This workload issubmitted once. Finally, in the 4×1 K case, the 1×1 Kworkload is run in parallel from four different hostsand under different VO groups. Also, each job can bere-planed at most four times through the submissioninfrastructure.
For service brokering tests, we opted for a simpleservice that is provided by default with the GT4container, namely the SecureCounterService. TheARESRAN’s PDP was enabled on all the containersproviding services, while each instance of ARESRANwas also registering with a pre-defined WS-IndexService in order to support such a dynamic environ-ment [64]. The GRUBER framework was aware onlyof the services registered with this WS-Index Serviceand performed brokering and reservations function-alities only for those. The uSLAs enforced at each sitewere fixed-limit uSLAs, with the following values(allowed number of requests over one hour): 10, 16,27, 46, 77, 128, 214, 357, 595 and 992 for the #1 caseand 0, 0, 0, 0, 0, 0, 0, 0, 0, 1000 for #2 case. Eachexperiment ran for 1 h.
In Table 1 are captured the results achieved by run-ning a BLAST workload composed of 1×500 jobs[71] over Grid3. Here, in the ideal case, the valuesare: Comp = 100, Util = 25.00, Delay = 3,600,Time = 3,000, and Speedup = 150, while the metricswere defined by Dumitrescu et al. [71]. In this caseGRUBER besides enforcing the required uSLAs alsoprovides a 65 value for the Speedup metric in average.Taking in account that not all jobs in Grid3 werescheduled using GRUBER, we consider these valuesas encouraging from a user perspective that otherwisemight have to wait three times longer when using auSLA-unaware scheduling strategy (see Table 2).
We also compared GRUBER-based schedulingperformance with an un-aware of any site uSLAsscheduling approach. In the first of the two methods,we use G-LU and G-Obs. The third alternative, S-RA,associates each job to a site selected at random.Table 2 shows the results obtained on Grid3 for the1×1 K workload. We found out that a GRUBER siteselector achieves a three times better performancethan the one of S-RA site selector (resource utilizationdrops a few order of magnitude while the totalexecution time is almost three times higher).
4.3 Service Brokering Example on an Ad-hoc GridDeployed on PlanetLab
Next, we present our experiments with the integratedGRUBER-ARESRAN infrastructure. Firstly, we con-sider the case when all service requests are donethrough GRUBER, secondly by means of a randomscheduling strategy and thirdly by means of a roundrobin strategy. Case #1 shows the worst caseperformance of GRUBER as all provided servicesare 100% available; this case will favor simplescheduling approaches, such as the random or roundrobin assignment as the scheduling decision is notvery important because of the abundance of availableresources; furthermore, this case also shows theoverhead incurred by the GRUBER assignment inrelation to the trivial random or round robin assign-ments. Case #2 shows the best scenario for theGRUBER engine as a large portion of all providedservices are not available; this case will favor ascheduling approach that can make good and in-formed decisions regarding the scheduling decisionsin order to utilized the little available resources.Table 3 depicts the results for the client situated onthe same network as the GRUBER engine for thethree scenarios under the two different cases men-tioned above.
For case #1, we can observe that the total numberof request completed in one hour differs greatlybetween the third approaches (177 vs. 321 and,respectively, 312). These results show that thebrokering request has taken an important ratio of thetotal execution time. However, in case #1, allprovided services were initially 100% available, sothe brokering decisions did not provide any real helpin the beginning. Once the allocations started to fillup, the importance of GRUBER decisions haveincreased, but are not very apparent in the case #1experiments.
For case #2, we see the potential benefits from theGRUBER assignments as we were able to obtainmany more assignments than with either the randomor the round robin assignment (150 vs. 59 and 52).We conclude that the utility of the GRUBER servicebrokering engine occurs only when the amount ofservices is large, custom advance reservations areperformed in the system or the number of the requestsis large and can potentially exhaust the availableallocations.
Table 1 Results of four GRUBER scheduling policies for1×500 workloads
Table 2 G-LU, G-Obs and S-RA strategies: performance for1×1,000 workloads
G-LU G-Obs. S-RA
Comp (%) 99.3 97.3 60.2Util (%) 14.56 12.59 0.57Delay (s) 50.50 62.01 121.0Time (s) 33,300 40,320 80,280
C.L. Dumitrescu, et al.
5 The Performance of GRUBER
In this section we focus on the experimental resultsthat capture some of the GRUBER capabilities interms of both scalability and accuracy. Of course,these results are captured for the distributed version ofGRUBER, because they make more sense in thisscenario.
5.1 Testing Environment
For the GRUBER infrastructure performance meas-urements, we used a simulated Grid environment 10times larger than the current OSG/Grid3 environment.The decision points were deployed on PlanetLab,providing brokering services while no real submissionwas actually performed.
5.1.1 Infrastructure Performance Metrics
We use three metrics to evaluate the effectiveness ofDI-GRUBER: Average Response Time (Response),Average Throughput (Throughput), and AverageScheduling Accuracy (Accuracy).
We define Response as follows, with RTi being theindividual job time response and N being the numberof jobs processed during the execution period:
Response ¼X
i¼1::NRTi=N
Throughput is defined as the number of requestscompleted successfully by the service per unit time.Finally, we define the scheduling accuracy for aspecific job (SAi) as the ratio of free resources at theselected site to the total free resources over the entire
Grid. Accuracy is then the aggregated value of allscheduling accuracies measured for each individualjob:
Accuracy ¼X
i¼1::NSAið Þ=N
5.1.2 Environment Characteristics and Workloads
For the service brokering measurements of thissection, we used similar settings as in the previoussection for service brokering: 10 ARESRAN man-aged WSRF services on PlanetLab, while eachinstance GRUBER was used for service brokeringfor the ad-hoc Grid. Therefore, we skip the testbeddescription. Using DiPerF, we performed several testswhere the decision points were exchanging statusinformation under various settings: predefined bro-kering mesh connectivity (all, one half and, respec-tively, one quarter), predefined time intervals (1, 3, 10and 30 min) and predefined number of decision points(1, 3 and 10). The workloads were generated bymeans of DiPerF, with jobs submission simulationfrom 120 submission hosts every 1 s.
5.2 Scalability Test Results and Comparisonwith a Peer-to-Peer Service
Figure 10 reports the experiments performed whenusing 1, 3 and 10 GRUBER decision points onPlanetLab. As can be easily observed, the resultsshow improvement in terms of Throughput andResponse Time when moving from 1 decision pointto 10 decision points. The Throughput metric’s valueincreases practically linearly with the number ofdecision points, reaching a constant value of 5 queriesper seconds for 3 decision points, while going us up
Table 3 Service brokering performance results (Metrics: number of request, GRUBER infrastructure response time, tested serviceresponse time and tested service reject time)
as 16 queries per second for 10 decision points(Throughput is defined as the number of requestscompleted successfully by the service per time unit).The results are convincing in our view that moving toa distributed infrastructure provides real advantagesfrom a performance point of view.
For convincing the reader that even though DI-GRUBER’s transaction throughput seems low com-
pared to ‘other transaction processing systems,’ wehave performed further performance studies by meansof DiPerF [72] on PlanetLab for a pretty well knowdistributed lookup service. The service chosen fortesting was the PAST application, built on top of thePASTRY substrate [42].
The chosen setup was very similar to the one usedfor DI-GRUBER: the same PlanetLab nodes (around
Fig. 10 DI-GRUBERthroughput and response for1, 3 and 10 decision points
C.L. Dumitrescu, et al.
120). This time we used five machines for runningpermanent PAST nodes, while the rest ones werebrought up dynamically, joining and leaving thenetwork in a controlled manner. Again, we used onlyone of the five nodes as the main contact point (anode situated at the University of Chicago). The restones were maintained as backup and to mimic the DI-GRUBER network. The length of the experiment wasagain one hour, while each joining node requested alookup and an insert operation every second (or, if theprevious operation took more than one second, atsoon as the previous operation ended).
Our performance results are presented in Fig. 11.The measurements show that for insert and lookupoperations, the PAST’s response time is around 2.5 swith a higher variance in the beginning (the stabili-zation of the P2P network), while the throughput goesas up as 27 transaction per second in average. Also,the message lost rate for this ad-hoc network waspretty high compared with the one of DI-GRUBER.However, the network stabilization delay is higher forthe P2P system (first 18% of the experimental time)compared with DI-GRUBER clients’ instantaneous
network join operation. Our last note is that alloperations were performed and measured on the localnodes (insertion followed by lookup); each node wasresponsible to propagate the results further (thus thehigher response time and lower throughput than in thecase of employing the continuation).
5.3 GRUBER Accuracy Performance Results
We consider three main directions of analysis: meshconnectivity, synchronization among decision pointsand the total number of decision points of theinfrastructure. For each dimension, our results andconsiderations (where available) are captured in thefollowing sections.
5.3.1 Accuracy with Mesh Connectivity
First, we measure Accuracy of the brokering infra-structure function of the decision points’ averageconnectivity. We consider practically three cases: fullconnectivity (all DPs see each other), half connectivity(DP collects information only from half of all theothers), and one-fourth connectivity (DP collectsinformation only from a quarter of all the others).The results were achieved by means of the DI-GRUBER infrastructure in all three above configu-rations and are captured in Table 4.
We can observe that the performance of thebrokering infrastructure drops substantially with con-nectivity degree of each individual decision point; in a
Table 4 DI-GRUBE accuracy function of the infrastructuremesh connectivity
Connectivity (N=10) Accuracy (%)
N-1 75N/2 62N/4 55
Fig. 11 PAST network response time (left axis) and throughput(right axis) for a variable load (left axis*10) on 120 PlanetLabnodes
Table 5 DI-GRUBER accuracy function of the exchange timeinterval for three decision points
Exchange interval (min) Accuracy (%)
1 893 8710 8630 83
Table 6 DI-GRUBER accuracy function of the number ofdecision points
Number of decision points Accuracy (%)
1 983 8910 75
The design, usage, and performance of GRUBER
nutshell, Accuracy drops almost linearly with clients’connectivity degree.
5.3.2 Accuracy with Time Exchange Intervals
The results in Table 5 show that, for a three decisionpoint infrastructure, a three to ten minutes exchangeinterval is sufficient for achieving almost 90%Accuracy. However, this accuracy value depends alsoon the number of the jobs scheduled by the decisionpoints.
5.3.3 Accuracy with the Number of Decision Points
Next, we analyze the performance of DI-GRUBERand its strategies for providing accurate schedulingdecisions function of the number of decision points inthe infrastructure. Table 6 depicts the accuracyperformance. We note that the accuracy drops to75% in the 10 decision points case.
6 Summary and Conclusions
We have presented a Grid resource broker, GRUBER,for representing and managing resource allocationpolicies in a multi-site, multi-VO environment.GRUBER is an infrastructure for uSLA specificationand enforcement in large and dynamic distributedenvironments. It is our result of a uSLA-based Gridmanagement infrastructure as described by Dumi-trescu et al. [1, 21]. We note that GRUBER is acomplex service: a query to a decision point mayinclude multiple message exchanges between thesubmitting client and the decision point, and multiplemessage exchanges between the decision points andthe job manager in the Grid environment. In a WANenvironment with message latencies in the 100 s ofmilliseconds, a single query can easily take multipleof seconds to serve. We expect that performance willbe significantly better in LAN environments. Howev-er, one of GRUBER’s design goals was to offerresource brokering in a WAN environment such asmulti-site, multi-VO Grids.
Managing uSLAs within large virtual organizationsthat integrate participants and resources spanningmultiple physical institutions is a challenging prob-
lem. Maintaining a single unified decision point foruSLA management is a problem that arises whenmany users and sites need to be managed. We providea solution, namely the GRUBER infrastructure withthe distributed variation, to address the question onhow uSLAs can be stored, retrieved and disseminatedefficiently in a large distributed environment. Webelieve that all these features presented in this papermake GRUBER capable working not only in largeGrid environments, but also in dynamic and heavily-loaded environments where automatic recovery canbe a problem. The novelty of this paper consists inintroducing and evaluating a model and architecturefor generic Grid resource and service brokering,uSLA-based provisioning and advance reservation inlarge distributed and dynamic environments.
Acknowledgments Authors would like to thank Anne Rogersfor her feedback on the material in this paper. Also, we thankJens-S Vöckler and Michael Wilde for their support and help inintegrating Euryale and GRUBER. Some of the experimentalresults reported in this paper were supported by the NSFInformation Technology Research GriPhyN project, undercontract ITR-0086044.
References
1. Dumitrescu, C., Wilde, M., Foster, I.: A model for usagepolicy-based resource allocation in Grids. In: Proceedings ofthe 6th IEEE International Workshop on Policies for Dis-tributed Systems and Networks (POLICY 2005), Stockholm,Sweden, pp. 191–200 (2005) (ISSN: 0-7695-2265-3)
2. Foster, I., et al.: The Grid 2003 production Grid: principlesand practice. In: Proceedings of the 13th InternationalSymposium on High Performance Distributed Computing(HPDC), pp. 236–245 (2004) (ISSN: 1082–8907)
3. Foster, I.: Grid computing. In: Proceedings of the Ad-vanced Computing and Analysis Techniques in PhysicsResearch (ACAT). AIP Conference Proceedings, Chicago,IL, vol. 583, pp. 51–56 (2000)
5. Dan, A., Davis, D., Kearney, R., Keller, A., King, R.,Kuebler, D., Ludwig, H., Polan, M., Spreitzer, M.,Youssef, A.: Web services on demand: WSLA-drivenautomated management. IBM Syst. J. 43, 136 (2004)
6. Czajkowski, K., Foster, I., Kesselman, C., Sander, V.,Tuecke, S.: SNAP: a protocol for negotiating service levelagreements and coordinating resource management indistributed systems. In: Proceedings of the 8th Workshopon Job Scheduling Strategies for Parallel Processing,Edinburgh, Scotland (2002)
C.L. Dumitrescu, et al.
7. Czajkowski, K., Dan, A., Rofrano, J., Tuecke, S., Xu, M.:WS-agreement: agreement-based Grid service management(OGSI-Agreement), Version 0. [Online: http://forge.gridforum.org/projects/graap-wg/document/Draft_OGSI-agreement_Specification/en/1/Draft_OGSI-Agreement_Specification.doc]
8. Gimpel, H., Ludwig, H., Dan, A., Kearney, R.: PANDA:specifying policies for automated negotiations of servicecontracts. In: Proceedings of the 1st International Confer-ence on Service Oriented Computing, pp. 287–302. Trento,Italy (2003)
9. Verma, D.C.: Policy Based Networking, Architecture andAlgorithm. New Riders, Indianapolis, IN (2000 November)
10. Dumitrescu, C.: INTCTD: a peer-to-peer approach forintrusion detection. In: Proceedings of the 6th IEEE Inter-national Symposium on Cluster Computing and the Grid(CCGrid’06), Singapore (2006) (ISBN: 0-7695-2585-7)
11. Verma, D.C.: Simplifying Network Administration usingPolicy based Management. IBM, UK (2004)
12. Lamanna, D., Skene, J., Emmerich, W.: SLang: a languagefor defining service level agreements. In: Proceedings ofthe 9th IEEE Workshop on Future Trends in DistributedComputing Systems, Puerto Rico, pp. 100–106. IEEE-CSPress (2003 May)
13. LHC Computing Project (2004)14. Ranganathan, K., Foster, I.: Decoupling computation and
data scheduling in distributed data-intensive applications.In: Proceedings of the 11th IEEE International Symposiumon High Performance Distributed Computing, pp. 352.Edinburgh, Scotland (2002) (ISSN: 1082-8907)
15. Mambelli M.: Capone and VDS: The University of Chicagoand Argonne National Laboratory: Chicago [Online: http://griddev.uchicago.edu/swhome/atgce/] (2005)
16. Annis, J., Kent, S., Szalay, A.: The SDSS-GriPhyNChallenge Problems: Cluster Finding, Correlation Func-tions and Weak Lensing. FermiLab, Batavia, IL (2001)
17. Maltsev, N., Sulakhe, D., D’Souza, M.J., Glass, E.,Rodriguez, A., Syed, M., Zhang, Y.: GNARE: GenomeAnalysis Research Environment. 2005, Argonne NationalLaboratory/Chicago [Online: http://compbio.mcs.anl.gov/gnare/gnare_home.cgi] (2006)
18. Foster, I., Voeckler, J., Wilde, M., Zhao, Y.: Chimera: avirtual data system for representing, querying, and auto-mating data derivation. In: Proceedings of the Global andPeer-to-Peer Computing on Large Scale Distributed Sys-tems Workshop. IEEE Computer Society, Washington, DC(1995 May)
19. Iosup, A., Dumitrescu, C., Epema, D., Liu, H., Wolters, L.:An analysis of four long-term Grid traces. TechnicalUniversity of Delft: Delft, Netherlands [Online: http://pds.twi.tudelft.nl/reports/2006/PDS-2006-003/PDS-2006-003.pdf] (2006)
20. Open ScienceGrid (OSG) [Online: http://www.opensciencegrid.org/] (2004)
21. Dan, A., Dumitrescu, C., Ripeanu, M.: Connecting clientobjectives with resource capabilities: an essential compo-nent for Grid service management infrastructures. In:Proceedings of the 2nd ACM International Conference onService Oriented Computing (ICSOC’04), pp. 57–64.New York, NY (2004) (ISSN 1-58113-871-7)
22. Legrand, I., Newman, H., Galvez, P., Voicu, E.,Cirstoiu, C.: MonALISA: a distributed monitoring servicearchitecture in computing. In: Proceedings of the HighEnergy Physics (HEP), La Jolla, CA (2003)
23. DOE Science Grid PKI Certificate Policy and CertificationPractice Statement (2002)
24. Dumitrescu, C.: ARESRAN: A WSRF-based resourcereservation service for Grid service. [Online: http://peopellcs.uchicago.edu/~cldumitr/ARESRAN] (2005)
25. Pearlman, L., Welch, V., Foster, I., Kesselman, C.,Tuecke, S.: A community authorization service for groupcollaboration. In: Proceedings of the IEEE 3rd InternationalWorkshop on Policies for Distributed Systems and Net-works, pp 55–59. Monterey, CA (2002) (ISBN: 0-7695-1611-4)
26. The Globus Project Team: CAS – community authorizationservice. [Online: http://www.nsf-middleware.org/Lists/Products/DispForm.aspx?ID=47] (2006)
27. Zhao, T., Karamcheti, V.: Expressing and enforcing distrib-uted resource agreements. In: Proceedings of High Perfor-mance Networking and Computing Conference (SC’2000),pp. 62. Dallas, Texas (2000) (ISSN 0-7803-9802-5)
28. Raman, R.: Matchmaking Frameworks for DistributedResource Management. PhD Thesis, University ofWisconsin (2000)
29. Foster, I., Roy, A., Sander, V., Winkler, L.: End-to-EndQuality of Service for High-end Applications. ComputerCommunications 27(14). Kluwer, Norwell, MA (2004)(ISBN: 1375–1388)
30. Thain, D., Tannenbaum, T., Livny, M.: Condor and the Grid.In: Berman, F., Hey, A.J.G., Fox , G. (eds.) Grid Computing:Making The Global Infrastructure a Reality. Wiley, NewYork, NY (2003) (ISBN: 0-470-85319-0)
31. Ludwig, H., Dan, A., Kearney, B.: Cremona: an architec-ture and library for creation and monitoring WS-Agree-ments. In: Proceedings of the ACM InternationalConference on Service Oriented Computing (ICSOC’04),New York, NY (2004)
32. In, J., Avery, P., Cavanaugh, R., Ranka, S.: Policy basedscheduling for simple quality of service in Grid comput-ing. In: Proceedings of the International Parallel & Dis-tributed Processing Symposium (IPDPS), p. 23. Santa Fe,New Mexico (2004) (ISBN: 0-7695-2132-0)
33. Buyya, R.: GridBus: A Economy-based Grid ResourceBroker. The University of Melbourne, Melbourne,Australia (2004)
34. Foster, I.: The Grid: a new infrastructure for 21st centuryscience. Phys. Today 55(2), 42–47 (2002)
35. Mueller, E.T., Moore, J.D., Popek, G.J.: A nested transac-tion mechanism for LOCUS. In: Proceedings of the 9thACM Symposium on Operating System Principles (SOSP),Bretton Woods, New Hampshire (1983)
36. Stonebraker, M., et al.: Mariposa: a wide-area distributeddatabase system. VLDB J. 5(1), 48–63 (1996)
37. Foster, I., Kesselman, C., Nick, J., Tuecke, S.: GridServices for Distributed Systems Integration. IEEE Com-puter 35(6), 37–46 (2002)
38. Humphrey, M., Wasson, G., Jackson, K., Boverhof, J.,Rodriguez, M., Bester, J., Gawor, J., Lang, S., Foster, I.,Meder, S., Pickles, S., McKeown, M.: State and events for
web services: a comparison of five WS-resource frameworkand WS-notification implementations. In: Proceedings ofthe 4th IEEE International Symposium on High Perfor-mance Distributed Computing (HPDC-14), Research Tri-angle Park, NC, 24–27 July 2005
39. Czajkowski, K., et al.: Grid information services fordistributed resource sharing. In: Proceedings of the 10thIEEE International Symposium on High PerformanceDistributed Computing. San Francisco, IEEE ComputerSociety, Los Alamitos, CA (2001)
40. Thompson, M.R., Essiari, A., Mudumbai, S.: Certificate-based authorization policy in a PKI environment. ACMTrans. Inf. Syst. Secur. 6(4), 566–588 (2003)
41. Lupu, E.: A role-based framework for distributed systemsmanagement, in Department of Computing. PhD thesis,University of London, London (1998)
42. Rowstron, A., Druschel, P.: Pastry: scalable, distributedobject location and routing for large-scale peer-to-peersystems. Lect. Notes Comput. Sci. 2218, 329–350 (2001)
43. Dumitrescu, C., Foster, I.: GRUBER: a Grid resourceSLA broker. In: Proceedings of the 11th InternationalEuro-Par Conference, pp. 465. Portugal (2005) (ISBN:3-540-28700-0)
44. The University of Wisconsin: UWMadisonCMS OpenScience Grid Site Policy Page. University of Wisconsin,Madison, WI (2006)
45. FNAL: FNAL: GPFARM Site Policy for OSG. FNAL(2006)
actions for experimental science. In: Proceedings of the10th International Euro-Par Conference, p. 399. Italy(2004) (ISBN: 3-540-22924-8)
48. Kay, J., Lauder, P.: A Fair Share Scheduler. University ofSydney, AT&T Bell Labs (1998)
49. Epema, D.H.J., Livny, M., van Dantzig, R., Evers, X.,Pruyne, J.: A worldwide flock of condors: load sharingamong workstation clusters. Future Gener. Comput. Systs.12, 53–65 (1996) (ISSN: 0167-739X)
50. Wolf, L.C., Steinmetz, R.: Concepts for reservation inAdvance. Multimed. Tools Appl. 4(3), 255–278 (1997)(ISSN 1380–7501) (Kluwer)
51. Maui Team: Maui Scheduler. Center for HPC ClusterResource Management and Scheduling. [Online: http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php]
52. Keller, A., Ludwig, H.: The WSLA framework: specifyingand monitoring service level agreements for web services.J. Netw. Syst. Manag. 11(1), 57–81 (2003)(Plenum)
53. Litzkow, M.J., Livny, M., Mutka, M.W.: Condor – a hunterof idle workstations. In: Proceedings of the 8th Interna-tional Conference on Distributed Computing Systems,pp. 104–111. San Jose, CA (1998) (ISBN: 0-8186-0865-X)
55. Foster, I., Kesselman, C.: Globus: a toolkit-based Gridarchitecture. In: The Grid: Blueprint for a Future Comput-ing Infrastructure, pp. 259–278. Morgan Kaufmann, SanMateo, CA (1998)
56. Dumitrescu, C., Foster, I.: Usage policy-based CPU sharingin virtual organizations. In: Proceedings of the 5th
International Workshop on Grid Computing, pp.53–60.Pittsburgh, PA (2004) (ISSN: 1550-5510)
57. Wolf, L.C., Steinmetz, R.: Concepts for resource reserva-tion in advance. Multimed. Tools Appl. 4(3): 255–278
58. Tannenbaum, T., Wright, D., Miller, K., Livny, M.: Condor– a distributed job scheduler. In: Berman, F., Hey, A.J.G.,Fox, G. (eds.) Grid Computing: Making the GlobalInfrastructure a Reality. Wiley, New York, NY (2003)(ISBN: 0-470-85319-0)
59. Henderson, R., Tweten, D.: Portable batch system: externalreference specification. Technical report, NASA, AmesResearch Center (1996)
61. Dumitrescu, C.: Policy Research for iVDGL. 2004, TheUniversity of Chicago/GriPhyN Project NSF Review 2004.Chicago, USA (2004) [Online: http://poeple.cs.uchicago.edu/~cldumitr/]
62. Dumitrescu, C., Wilde, M., Foster, I.: Usage policies at thesite level in Grid. iVDGL/GriPhyN Project: The Universityof Chicago (2006) [Online: http://poeple.cs.uchicago.edu/~cldumitr/]
63. Constandache, I.: Policy based dynamic negotiation forGrid services authorization. In: L3S Research Center.University of Hannover, Hannover, Germany (2005)
64. Dumitrescu, C., Raicu, I., Foster, I.: DI-GRUBER: adistributed approach for Grid resource brokering. In:Proceedings of the Proceedings of the 2005 ACM/IEEEConference on Supercomputing (SC’2005), p. 38. Seattle,WA (2005) (ISBN: 1-59593-061-2)
65. Vöckler, J.-S., Wilde, M., Foster, I.: The GriPhyN VirtualData System. GriPhyN Technical Report, The University ofChicago (2002) [Online: http://www.griphyn.org/]
66. Mohamed, H.H., Epema, D.H.J.: Experiences with theKOALA co-allocating scheduler in multiclusters. In:Proceedings of the 5th IEEE/ACM Int’l Symp. on ClusterComputing and the GRID (CCGrid2005), Cardiff, UK(2005 May)
67. SUN: Sun Grid Engine. (2004) [Online: http://www.sun.com]
68. LCG: LHC – The Large Hadron Collider Project [Online:http://lcg.web.cern.ch/LCG/] (2006)
69. Chervenak, A., Foster, I., Kesselman, C., Salisbury, C.,Tuecke, S.: The Data Grid: towards an architecture for thedistributed management and analysis of large scientific datasets. J. Netw. Comput. Appl. 23, 187–200 (2001) [Online:http://www.globus.org/]
70. Chun, B., Culler, D., Roscoe, T., Bavier, A., Peterson, L.,Wawrzoniak, M., Bowman, M.: PlanetLab: an overlay test-bed for broad-coverage services. ACM SIGCOMM Com-put. Commun. Rev. 33(3), 3–12 (2003) (ISSN: 0146-4833)
71. Dumitrescu, C., Raicu, I., Foster, I.: Experiences in runningworkloads over Grid3. In: Proceedings of the Grid andCooperative Computing (GCC2005), pp. 274–286. Beijing,China (2005) (ISBN: 3-540-30510-6)
72. Dumitrescu, C., Raicu, I., Ripeanu, M., Foster, I.: DiPerF:automated distributed performance testing framework. In:Proceedings of the 5th IEEE/ACM International Workshopin Grid Computing (Grid’04), pp. 289–296. IEEE Com-puter Society, Los Alamitos, CA (2004) (ISBN: 0-7695-2256-4)
