Post on 20-Jan-2016
transcript
1
Globus Grid Middleware: Basics, Components, and
Services
Source: The Globus ProjectArgonne National Laboratory &
University of Southern California http:/www.globus.org
Compiled/Compacted for 433-678: R. Buyya and S. Venugopal
2
Agenda
The evolution of Globus Security Services
GSI (Grid Security Infrastructure) Execution Management Services
GRAM(Grid Resource Allocation and Management)
Information Services MDS (Monitoring and Discovery System)
Data Services GridFTP RFT (Reliable File Transfer)
Summary/Conclusion
3
The Globus Project
Basic research in grid-related technologies Resource & data management, security, QoS, policy,
communication, adaptation, etc. Focused on creating a core Grid middleware
Core services for grid-enabled tools & apps Developed Globus Toolkit (1.0 in 1998, 2.0 in 2001, 3.0 in 2004 based
on web services/WSRF, 4.0 in 2008, and 5.0 in 2011) Construction of production grids & testbeds
Multiple deployments to distributed organizations for production & prototyping
Impacts The Southern California Earthquake Center uses Globus
software to visualize earthquake simulation data. Scientists in the Earth System Grid (ESG) are producing,
archiving, and providing access to climate data that advances our understanding of global climate change.
Find more on http://www-unix.globus.org/alliance/impact/
Grid and Web Services:Convergence
Grid
Web
The definition of WSRF(Web Services Resource Framework) means that Grid and Web communities can move forward on a common base
WSRF
Started far apart in apps & tech
OGSI
GT2
GT1
HTTPWSDL,
WS-*
WSDL 2
Have beenconverging
Ack: Prof. Ian Foster, Globus Project
Open Grid Services Architecture
Web Services Messaging, Security, Etc.
Open Grid Services Infrastructure
Domain-Specific Services
Core Services
ProgramExecution Data Services
Sta
nd
ard
izati
on
WS-Resource Framework
Ack: Prof. Ian Foster, Globus Project
6
Globus Toolkit 4
7
Globus Toolkit Components
Grid Resources and Local Services
Grid Resource Management (GRAM, GASS)
GSI Security Layer
Third Party User-Level Middleware
Grid Information Services
(MDS)
Grid Data Management
(GridFTP, ReplicaCatalog)
Applications
Globus
Grid Resources and Local Services
Grid Resource Management
(GRAM)
GSI Security Layer
Third Party User-Level Middleware
Grid Information Services
(MDS)
Grid Data Management
(GridFTP, ReplicaCatalog)
Applications
Globus
8
Security Services
Review of Security Services
9
Grid Security Infrastructure (GSI)
Based on Public key cryptography X.509 certificate
Certificate/Proxy - Single Sign-on Gives information about the proxy
$ grid-proxy-info Create proxy at /tmp/x509up_u<USER ID>
Destroy proxy certificate $ grid-proxy-destroy
10
PKI: Creating Private & Public Keys
To request a certificate a user starts by generating a key pair.
The private key is stored encrypted with a pass phrase the user gives
The public key is put into a certificate request
CertificateRequest
Public Key
EncryptedOn local
disk
USER
11
PKI: Certificate Issuance
The user then takes the certificate to the CA
The CA usually includes a Registration Authority (RA) which verifies the request:
The name is unique with respect to the CA
It is the real name of the user
Etc.
CertificateRequest
Public KeyID
12
PKI: Certificate Issuance
The CA then signs the certificate request and issues a certificate for the user.
CertificateRequest
Public Key
NameIssuerPublic KeySignature
Sign
Rajkumar Buyya111, Barry StreetCarlton
BD 01-04-1971 Male 165cms, 65KgB&W Eyes
State ofVictoria
Seal
13
Mutual Authentication (between A and B)
A and B are two parties: Both need to trust each others’ CA.
A B (A establishes connection to B and gives his certificate (name,pub. Key) to B).
B makes sure that it can trust CA of A. B generates random message A and asks it
encrypt it. A encrypts it and send to B B decrypts using A’s public key. If the msg. is
same as what B has sent, then A is who it is claiming to be.
14
Example: Secure Remote Startup in Globus
key
cert
gatekeeperclient
1. Exchange certificates, authenticate, delegate
2. Check gridmap file3. Lookup service4. Run service program
(e.g. jobmanager)
jobmanager
key
cert
1.
2.
map
4.
services3.
15
Execution Management Services
Grid Resource Allocation and Management (GRAM)
16
What is GRAM?
Grid resource access & job execution services: Handles placement, provisioning and
lifetime management of jobs GRAM is a unifying remote interface
for remote job submission and resource management
GRAM is for stateful job control Reliable operation Asynchronous monitoring and control
17
Grid Job Management Goals
Provide a service to securely: Create an environment for a job Stage files to/from environment Cause execution of jobs Monitor execution of jobs Signal important state changes to client Enable client access to output files
18
GRAM Command Line Tools
Pre-WS GRAM (GT2) Command Line Tools Interactive job submission
globus-job-run host:port/jobmanager-pbs command
Batch job submission globus-job-submit command
Checking job status when done output globus-job-status jobID globus-job-output –out|err jobID
Kill a job globus-job-cancel jobID
19
Grid Information Service
MDS (Monitoring and Discovery Service)
20
The Need for Information
System information is critical to operation of the grid and construction of applications How does an application determine what
resources are available? What is the “state” of the computational grid? How can we optimize an application based on
configuration of the underlying system? We need a general information
infrastructure to answer these questions
21
Using Information forResource Brokering
“10 GFlops, EOS data,20 Mb/sec -- for 20 mins”
MetacomputingDirectoryService
GRAMGRAMGRAM
ResourceBroker
Info service:location + selection
Globus ResourceAllocation Managers
GRAM
ForkLSFEASYLLCondoretc.
“What computers?”“What speed?”“When available?”
“50 processors + storage from 10:20 to 10:40 pm”
“20 Mb/sec”
22
Examples of Useful Information
Characteristics of a compute resource IP address, software available, system
administrator, networks connected to, OS version, load
Characteristics of a network Bandwidth and latency, protocols, logical
topology Characteristics of the Globus
infrastructure Hosts, resource managers(PBS, Condor etc)
23
Data Services
Overview of Data Services
24
Data Services
Handle movement, access, replication and updates of data
Example services in Globus Toolkit GridFTP Reliable File Transfer (RFT)
25
Data Services - GridFTP
Basic tool provided with Globus Toolkit globus-url-copy [-tcp-bs TCPBufSizeBytes] [-p
parallelism] sourceURL destURL URL format = protocol://[host]:[port]/path
Supported protocols gsiftp://, file://(on a local machine only), ftp://,
http://, https:// It is legal to specify a user name and
password in the the URL as follows gsiftp://myname:
[mypassword]@myhost.mydomain.com/file.dat
26
Data Services - GridFTP
local -> local $ globus-url-copy gsiftp://localhost/etc/hosts
file:///tmp/hosts_copy remote -> local
$ globus-url-copy gsiftp://ng2.vpac.org/etc/hosts file:///tmp/hosts_copy
local -> remote $ globus-url-copy file:///tmp/hosts_copy
gsiftp://ng2.vpac.org/tmp
27
Data Services - Reliable File Transfer (RFT)
Manages a set of third-party GridFTP transfers
Uses a database to checkpoint transfer state
Recovers from Source/destination server failures Network failures Container failures
Transfers retried with exponential backoff Resumes transfers where they left off
28
Summary
Globus as a core Grid middleware solution Four major services
Security Service GSI
Execution Management Service GRAM(Grid Resource Allocation and Management )
Information Service MDS (Monitoring and Discovery Service)
Data Service GridFTP Reliable File Transfer
Very prominent (during 1998-2005) – used in several collaborative scientific computing and national grid projects
29
Some Issues with Globus
Very complex system, need a strong administration skills to install/maintain
Deep learning curve for developers who need to work with APIs
More on Globus, visit: http://www.globus.org/