Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | tyrone-horton |
View: | 214 times |
Download: | 0 times |
Web: www.omii.ac.uk Email: [email protected]
Towards Grid Interoperability
Richard Boardman, Stephen Crouch, Hugo Mills, Steven Newhouse, Juri Papay
and the OMII-UK Team
All Hands Meeting11/09/07
Web: www.omii.ac.uk Email: [email protected]
Contents
• Introduction & background• Interoperability and standards
o Job Submissiono Implementations
• Build and test• Job brokering using job submission
standards
Web: www.omii.ac.uk Email: [email protected]
Introduction: What is the Grid?
• The grid – many definitions!
“Grid computing offers a model for solving massive computational problems by making use of the unused CPU cycles of large numbers of disparate, often desktop, computers treated as a virtual cluster embedded in a distributed telecommunications infrastructure” – Wikipedia
An infrastructure for “coordinated resource sharing and problem solving in dynamic, multiinstitutional virtual organizations” – Foster, Kesselman, Tuecke.
“A service for sharing computer power and data storage capacity over the Internet.“ – CERN (European Organisation for Nuclear Research)
• Differing perceptions of the Grid:o Particle physics community: massive, loosely coupled, distributed
computing environment with computing capability, bandwidth and storage.
o Bio-informaticians: global virtual federated database of experimental data, research papers and laboratory records.
• Suggested common view:o The Grid provides secure virtualisation of resources and enables
collaboration and establishment of virtual organisations.
Web: www.omii.ac.uk Email: [email protected]
Background
• Many Grid infrastructures have been developed, but, traditionally, with little interoperability:
o Policies governing access/use of distributed resourceso Lack of adherence to emerging common standards
• Interoperability offers huge benefits for categories of users within user community:
o e-Infrastructure providers: easier deployment/management of software distributions
o e-Science users: freedom to choose services deployed in different Grids; based on functionality, not deployed on a particular Grid
o e-Science application developers: portability of apps across multiple Grids to increase uptake
Web: www.omii.ac.uk Email: [email protected]
Standards: the key to Interoperability• Adoption of common standards strongly
supported and implemented by OMII-Europe and GIN (Grid Interoperability Now) for:
o Job Submission, Accounting, Virtual Organisation Management…
o Standards from: OGF, OASIS, W3C, DMTF…o Across platforms: EGEE, Globus, UNICORE &
others
• Will focus on Job Submission (OGSA-BES, JSDL) standards
Web: www.omii.ac.uk Email: [email protected]
Emerging Job Submission Standards• Two key standards for two key elements:
o The Basic Execution Service interface (OGSA-BES)
• Simplified version of OGSA-EMS (Execution Management Service)
• Handles basic job lifecycle management• Defines simple (but extendable) job state model
– Pending, running, cancelled, failed or finished
o The Job Submission Description Language (JSDL)
• Specify job executable, data staging and resource requirements
Web: www.omii.ac.uk Email: [email protected]
Summary of OGSA-BES
BES-Management Port-typeStopAcceptingNewActivities Request that the BES stop accepting new activities
StartAcceptingNewActivities Request that the BES start accepting new activities
BES-Factory Port-typeCreateActivity Request the creation of a new activity
GetActivityStatuses Request the status of a set of activities
TerminateActivities Request that a set of activities be terminated
GetActivityDocuments Request the JSDL documents for a set of activities
GetFactoryAttributesDocument Request XML document containing BES properties
BES-Activity Port-type (optional)GetStatus Request the status of an activity
Terminate Request that an activity be terminated
GetDocument Request the JSDL document for an activity
GetActivityAttributesDocument Request XML document containing activity properties
Web: www.omii.ac.uk Email: [email protected]
Interface/Interaction Standards not Enough• BES & JSDL alone not enough for real interoperability
o JSDL is extensibleo Differing security models across Grid infrastructures
• HPC-Profile proposes Grid interoperability through:o Restricted OGF Job Submission Description Language (JSDL)o OGF OGSA Basic Execution Service (BES)o WS-I Basic Profile
• In addition, an agreed security framework between participants
o Via HTTPS transport (server offers certificate) & username/password (client) for user authentication
• OMII-UK has implemented the HPC-Profile within:o GridSAM – funded by OMII-UK, first to adopt BESo CROWN – with OMII-Europe, in collaboration with Beihang
University, China
Web: www.omii.ac.uk Email: [email protected]
Build and Test
• To gain confidence of compliance with interoperability standard, test against other standards-compliant infrastructures
o Test multiple BES/JSDL clients against multiple service endpoints
o Mechanistic process; automation advantageous• ETICS (CERN) provides test automation framework
o e-Infrastructure for Testing, Integration and Configuration of S/W
o Leverages Metronome (formerly NMI Build & Test) across Condor cluster
o Controls management of software builds and testingo Create project configurations, maintain historical records
• ETICS deployed to enable automated compliance testing
Web: www.omii.ac.uk Email: [email protected]
Build and Test
• BES method sequence for testing (core functionality):
o Get attributes documento Create jobo Query job statuso Show job outputo Get job’s JSDL documento Terminate job
• Scenarios implemented in ETICS to test interoperability
o Testing across HPC-Profile endpoints (with simple config)o Test OMII-UK client component against service
Web: www.omii.ac.uk Email: [email protected]
Job Brokering using the CROWN Scheduler
• CROWN Grid developed by Beihang University
• With OMII-Europe as part of Component Exchange activity with OMII China:
o Identified CROWN Meta Schedulero Integrated into OMII release (interoperation)
• Usage of OMII-UK Grimoires service registry
o Coordinated implementation of BES interface to the Scheduler
Web: www.omii.ac.uk Email: [email protected]
CROWN BES Scheduler
BESClient
Registrysubmit
monitor
subm
itm
onito
r
submitmonitor
CROWN Scheduler
Supported Platform• 2 jobs, different requirements• Delegate BES resource selection by
submitting both jobs to Scheduler• Monitor each until completion
BES
BES
1. requestresources
3. monitor
Grimoires
2. submit
2. submit
3. monitor
• 2 jobs, different requirements• Identify 2 appropriate BES instances• Submit to both
Web: www.omii.ac.uk Email: [email protected]
Conclusions
• First phase of projects dealt with infrastructures, test beds and application software
o Led to greater understanding of key issues• Next phase concentrates on interoperability
and providing solutions-based approaches• Interoperability offers key benefits to the
community:o Ease of managemento Choiceo Simplicity of implementation
• Have illustrated a process of standards adoption, build and compliance testing, and usage within a scheduling application