1
CENTER FORPARALLEL
COMPUTERS
DEPARTMENT OF COMPUTER SCIENCE
DEPARTMENT OFCOMPUTING SCIENCE
2nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
An OGSA-Based Accounting System for Allocation Enforcement across HPC Centers
TS10 – Service Applications
Thomas Sandholm [email protected] MulmoPeter GardfjällErik ElmrothLennart Johnsson
22nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Key Question
?How do we share national Grid compute resources in a fair, secure, open, and
scalable way
32nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
42nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Fairness vs. Utilization
• Trade-off: Fair resource distribution and optimal resource utilization
• Soft real-time quota enforcement• User preferences• Resource policies• Allocation authority policies
Fair Distribution
Maximum Utilization
SecureOperation
Scalable Efficiency
52nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Security vs. Scalability
• Integrity & Privacy• Single Sign-on/Impersonation• DoS/Replay Attack prevention• Privilege Delegation• Message Level vs. Transport Level• Policy Driven Authorization: PDP, PAP, PIP, PEPScale:• National Grid • No single point of failure but
coordinated allocationenforcement
Fair Distribution
Maximum Utilization
SecureOperation
Scalable Efficiency
62nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Openness & Interoperability
• Systems Integration Platform• Scheduler/Workload Manager Agnostic• Programming Language/Model Agnostic• Portable (100% pure Java)• XML Based Standards: XPath, XQuery, XSLT, GGF-
UR, XML-Signature, XML-Encryption, XACML• Web/Grid Services Standards: SOAP, WSDL, WS-
Security, OGSA, GGF-UR, GSI, GSSAPI, OGSI/WSRF
72nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
82nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Open Grid Services Architecture• Global Grid Forum Standardization Initiative• Architecture extending SOA and WSA to dynamically
share stateful resources across organizational boundaries (=realizing the Grid vision)
• “… defining, within a service-oriented architecture, a set of core capabilities and behaviors that address key concerns in Grid systems.” OGSAv1
• Assumes state modeling according to OGSI/WSRF Core Infrastructure offering Inspection,
Discovery, Lifetime Management, Notifications, Fault Handling
• WS-Resource = stateful resource and associated Web service. Provide context for message exchange
• Addresses Grid security requirements such as Delegation and Single Sign-On
92nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Role of Accounting in OGSA
Accounting foundational service to: • Job Execution
Make sure that only jobs with sufficient quota can be executed on the compute resource
Decide queue priority based on available funds and usage history
• QoS/SLA Management Negotiate pricing based on resource usage Optimizing Utilization SLA Attainment/Policing
• Security Auditing Access Control PEP/PIP
102nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities • Open Grid Services Architecture • SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
112nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
SweGrid
• SweGrid connects 600 compute nodes (Intel P4) across 6 Swedish HPC centers interconnected by 10Gbs GigaSunet network
• 400 HPC users at all centers (some overlapping)
• Inaugurated March 2004• ~50 currently active researchers • Up to 10k jobs per month per site
122nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
SweGrid Continued
• Resource quotas allocated by Swedish National Allocations Committee (SNAC) after peer-review of promising research projects with high computational demands (c.f. NRAC)
• Initially homogeneous hardware but heterogeneous scheduling, security, and accounting environment (policies, tools, data, processes, etc)
• Wanted: Uniform resource quota use & allocation
132nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
142nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
SweGrid Accounting System (SGAS) Key Design Points
1. Decentralized accounting solution based on standard, open protocols in compliance with the proposed OGSA
2. 3-party (user, resource, allocation authority) policy customization
3. Non-intrusive to local site accounting systems4. All components governed by a scalable cross-
organizational authorization framework
152nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
SGAS Component Overview
PAT
Bank LUTS
Resource
Scheduler
Resource Manager
BrokerUser
WSDL WSDL
WSDL WSDL
JARM
Policy Administration ToolLogging and Usage Tracking ServiceJob Account Reservation Manager
SubmitJob
Reserve/Release PublishUR
QueryAddUser
162nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
SGAS Security Design
PAT
Bank
Resource
Scheduler
Resource Manager
BrokerUser
JARMPEP
PDP
PAP
Site Policy Manager
PIP
PDP
LUTSPIP
PAP
External Authorization Service
PDP
Membership/CommunityService
PIP
Policy Administration PointPolicy Decision PointPolicy Information PointPolicy Enforcement Point
Credential Delegation
WS-SecureConversation
XML-SignatureXML-Encryption
PKI
Kerberos
172nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
182nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Overdraft XACML Policy
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97
175%
150%
125%0
20000
40000
60000
80000
100000
120000
140000
Execution Time (ms)
Jobs
Overdraft Limit
<Condition FunctionId= "urn:oasis:names:tc:xacml:1.0:function:integer-less-than-or-equal"> <Apply FunctionId= "urn:oasis:names:tc:xacml:1.0:function:integer-one-and-only"> <EnvironmentAttributeDesignator AttributeId= "sgas:overdraw:percent:requested" DataType= "http://www.w3.org/2001/XMLSchema#integer"/> </Apply> <AttributeValue DataType= "http://www.w3.org/2001/XMLSchema#integer"> 175 </AttributeValue></Condition>
192nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Overdraft Fuzzy Logic Policy
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97
Fair
Fuzzy
Unfair0
20000
40000
60000
80000
100000
120000
140000
Execution Time (ms)
Jobs
Submission Flow
R1: overdraft is low allocation left is much allow reservationR2: overdraft is high allocation left is little disallow reservationR3: allocation proximity is soon overdraft is high allocation left is much allow reservationR4: allocation proximity is soon overdraft is low allocation left is little allow reservation
202nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Super Computing 2004 Demonstration
212nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A
222nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Conclusions
• Document centric communication in conjunction with semi-structured native XML databases is a very flexible combination
• Batch charging and eager prepare reservation needed for scalability
• Timestamp based allocations distributed in a staggered monthly flow result in the best trade-off between fairness and utilization
• Generic PEP/PDP/PIP/PAP model useful for encapsulating and evolving authorization code
• OGSI/WSRF state management ideal for controlling fine grained service state such as account quotas, reservations and policies in a standard way
232nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Future Work
• With large-scale flexibility and configurability comes complexity and it becomes hard to optimize high-level goals and to realize detailed user QoS requirements – development of an SLA Management framework and user/resource goal driven optimizing agents (WS-Agreement, ContractNet)
• Initial focus has been on scientific community resource sharing - support economic brokering and for-profit banks
• Multi jobs may overload the bank - SAML assertions (c.f. cheques) as a multi-allocation payment and reservation method
242nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Learn more…
http://www.sgas.se
http://www.swegrid.se
252nd International Conference on Service Oriented Computing, New York, NY, 15-18 November, 2004
Outline
• Requirements on Software Qualities• Open Grid Services Architecture• SweGrid National Grid Testbed• SweGrid Accounting System• Results• Lessons Learned• Q & A