Introduction to the current EDG Testbed
SoftwareKraków, December 2002
Steve Fisher [email protected] – RAL
on behalf of
The European DataGrid Project Team
http://www.edg.org/
The EDG Testbed Software - 2
The European DataGrid
Funded by the European Union Jan 1, 2001 - Dec 31, 2003
Develop, implement and exploit a large-scale data and CPU-oriented computational GRID.
Develop middleware, in collaboration with some of the leading centres of competence in GRID technology.
Complement, and help to coordinate at a European level, several on-going national GRID projects.
The EDG Testbed Software - 3
The EDG Main Partners
CERN – International (Switzerland/France)
CNRS - France
ESA/ESRIN – International (Italy)
INFN - Italy
NIKHEF – The Netherlands
PPARC - UK
The EDG Testbed Software - 4
Research and Academic Institutes• CESNET (Czech Republic)• Commissariat à l'énergie atomique (CEA) – France• Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)• Consiglio Nazionale delle Ricerche (Italy)• Helsinki Institute of Physics – Finland• Institut de Fisica d'Altes Energies (IFAE) - Spain• Istituto Trentino di Cultura (IRST) – Italy• Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany• Royal Netherlands Meteorological Institute (KNMI)• Ruprecht-Karls-Universität Heidelberg - Germany• Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands• Swedish Research Council - Sweden
EDG Assistant Partners
Industrial Partners• Datamat (Italy)• IBM-UK (UK)• CS-SI (France)
The EDG Testbed Software - 5
}} ApplicationsApplications
WP1: Work Load Management System
WP2: Data Management
WP3: Information and Monitoring
WP4: Fabric Management
WP5: Storage Element
WP6: Testbed and demonstrators
WP7: Network Monitoring
WP8: High Energy Physics
WP9: Earth Observation
WP10: Biology
WP11: Dissemination
WP12: Management
EDG structure: work packages
The EDG Testbed Software - 6
Current EDG Testbed
CERNLyon
RAL
Manchester NIKHEF
Reference site: CERN
Testbed1 EDG sites
NorduGrid
Italy:• Bologna • Cagliari• Catania• Milano• Napoli• Padova• Parma• Pisa• Roma• Torino
NorduGrid:• Bergen• Copenhagen• Helsinki• Lund• Oslo• Stockholm• Uppsala
Karlsruhe
BarcelonaMadrid
Lisboa
The EDG Testbed Software - 7
Security: Authentication/Authorization
Authentication Who you are
users identified by certificates signed by a CA
Authorization What you are allowed to do
based on membership of Virtual Organizations (VO).
The EDG Testbed Software - 8
Certificate Request
VO
user
cert-request
grid-cert-request
once in every two-three years
The EDG Testbed Software - 9
Requesting a Certificate
grid-cert-request
A certificate request and private key is being created.
[...]
Using configuration from /usr/local/grid/globus/etc/globus-user-ssleay.conf
Generating a 1024 bit RSA private key
[...]
A private key and a certificate request has been generated with the subject:
/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner
[...]
Your private key is stored in .../.globus/userkey.pem
Your request is stored in .../.globus/usercert_request.pem
Please e-mail the certificate request to the CERN CA
cat .../.globus/usercert_request.pem | mail [email protected]
Your certificate will be mailed to you within two working days.
The EDG Testbed Software - 10
Certificate Signing
CA
user
cert-request
grid-cert-request
certificate
cert signing
The EDG Testbed Software - 11
Registration/Authorization
User registration in an EDG Virtual Organisation
convert your certificate: openssl pkcs12 –export –in ~/.globus/usercert.pem –inkey
~/.globus/userkey.pem –out user.p12 –name ’Joe Smith’
import your certificate in your browser
sign the usage guidelines: https://marianne.in2p3.fr/cgi-bin/datagrid/register/account.pl
ask an account from your VO administrator by email
-> You are registered in the VO server and have a user account.
The EDG Testbed Software - 12
Registration
VO
user
registrationcert.pkcs12
convertcertificate
Usage guidelines
Account Registration
once for the lifetime of the VO – you may
change the certificate keys!
The EDG Testbed Software - 13
Starting a Session
user
proxy-certgrid-proxy-init
cert.pkcs12
certificate
every 12/24 hours
The EDG Testbed Software - 14
Usage
You must have a valid certificate from a trusted CA!
“login”: grid-proxy-init
short lifetime certificate: 24 hours
Enter PEM pass phrase:
...........................+++++
....................................+++++
checking the proxy: grid-proxy-info -subject
/O=Grid/O=CERN/OU=cern.ch/CN=Akos Frohner/CN=proxy
-> use the grid services
“logout”: grid-proxy-destroy
The EDG Testbed Software - 15
Configuration on the Server
CA
service
host-cert
cert signing
host-request
grid-cert-request
ca-certificate
crl
cert/crl update
crl automatically updated
periodically
The EDG Testbed Software - 16
Authorization Information
VO-server
service
host-cert
gridmapmkgridmap
ca-certificates
crls
automatically updated
periodically
The EDG Testbed Software - 17
Using a Service
user service
proxy-certgrid-proxy-init
cert.pkcs12
certificate
host/proxy certs exchanged
host-cert
gridmap
ca-certificates
crls
The EDG Testbed Software - 18
EDG Logical Machine Types1. User Interface (UI)
2. Resource Broker (RB)
3. Information Service (IS)
4. Computing Element (CE) Gatekeeper
(Front-end Node)
Worker Nodes (WN)
5. Storage Element (SE)
6. Replica Catalog (RC)
The EDG Testbed Software - 19
Information Systems overview
The aim of the Information and Monitoring Service is to deliver a flexible infrastructure that provides information on
the EU DataGrid itself grid applications
EDG info systems are based upon Globus MDS (Metacomputing Directory Service or Monitoring and Discovery Service as it is now called)
Based on OpenLDAP, a hierarchical database
The information system is currently used mainly by the middleware.
You can use it to find out what is going on
The EDG Testbed Software - 20
LDAP attributes
A schema describes the attributes and the types of the attributes associated with data objects
Example - some attributes of SiteInfo: siteName: RALDEV
sysAdminContact: [email protected]
userSupportContact: [email protected]
siteSecurityContact: [email protected]
dataGridVersion: 1.2
InstallationDate: 20020704142800Z
The EDG Testbed Software - 21
LDAP hierarchy
Lightweight Directory Assess Protocol (LDAP) offers a hierarchical view of information
The objects are arranged in a Directory Information Tree (DIT)
One or more attributes represent the Relative Distinguished Name (RDN)
An object is identified by its Distinguished name This is its RDN with the Distinguished name of its parent
The EDG Testbed Software - 22
RDNs and DNs
RDN
SE seId=dev02.hepgrid.clrc.ac.u
k
Protocols seProtocol=gridftp seProtocol=rfio seProtocol=file
DN Site
Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
SE seId=dev02.hepgrid.clrc.ac.uk,M
ds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
Protocols seProtocol=gridftp,
seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
seProtocol=rfio, seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
seProtocol=file, seId=dev02.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
supportedprotocols
SE
site
The EDG Testbed Software - 23
MDS GRISs & GIISs
Information providers are scripts which when invoked by the LDAP server make available the desired information
Information is cached by the server to improve performance
Within MDS the EDG information providers are invoked by a local LDAP server, the Grid Resource Information Server (GRIS)
“Aggregate directories”, Grid Information Index Servers (GIIS), are used to group resources
The GRISs use soft state registration to register with one or more GIISs
The GIIS can then act as a single point of contact for a number of resources
A GIIS may represent a site, country, virtual organization, etc.
In turn a GIIS may register with another GIIS
The EDG Testbed Software - 24
EDG Information Providers & the Directory Information Tree
computing element
storage elements that are close (not necessarily at the same site)
status supportedprotocols
file statistics
network information between this and other sites
storage element
site information
site
The EDG Testbed Software - 25
EDG GRIS/GIIS Hierarchy
Information providers publish information to a local LDAP server known as a Grid Resource Information Server (GRIS)
Each country has a GIIS to which all of the site GIISs register
There is a top level datagrid GIIS to which all of the country GIISs register
Each Site has a Grid Information Index Server (GIIS) which acts as a single point of contact for all of the sites resources. The GRISs register with their site GIISsiteA siteDsiteCsiteB
countryA countryB
datagrid
information providers
information providers
information providers
information providers
The EDG Testbed Software - 26
EDG Information Providers
The EDG have produced information providers: Site information
The Computing Element
The Storage Element
Network Monitoring
All of the EDG data objects are dynamic, they have a time stamp and a time to live (used by the cache mechanism) associated with them
The EDG Testbed Software - 27
Siteinfo
in=siteinfo,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
objectClass: SiteInfo
objectClass: DataGridTop
objectClass: DynamicObject
siteName: RALDEV
sysAdminContact: [email protected]
userSupportContact: [email protected]
siteSecurityContact: [email protected]
dataGridVersion: 1.2
installationDate: 20020704142800Z
The EDG Testbed Software - 28
Computing Element
ceId=dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs-M,hn=dev01.hepgrid.clrc.ac.uk,Mds-Vo-name=ral-dev,Mds-Vo-name=uk,o=Grid
objectClass: DataGridTopobjectClass: ComputingElement CEId:
dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs-M
GlobusResourceContactString:dev01.hepgrid.clrc.ac.uk:2119/jobmanager-pbs:/O=Grid/O=UKHEP/CN=dev01.hepgrid.clrc.ac.uk
GRAMVersion: ?Architecture: intelOpSys: RH 6.2MinPhysicalMemory: 258MinLocalDiskSpace: 2048TotalCPUs: 1FreeCPUs: 1NumSMPs: 0MinSPUProcessors: 0MaxSPUProcessors: 0TotalJobs: 0RunningJobs: 0IdleJobs: 0
MaxTotalJobs: 1 MaxRunningJobs: 1 WorstTraversalTime: 108000 EstimatedTraversalTime: 0 Active: TRUE Priority: 20 MaxCPUTime: 108000 MaxWallClockTime: 432000AverageSI00: 300 MinSI00: 300 MaxSI00: 300 AuthorizedUser:/O=Grid/O=UKHEP/
OU=hepgrid.clrc.ac.uk/CN=Tim EvesAuthorizedUser:/O=Grid/O=UKHEP/
OU=hepgrid.clrc.ac.uk/CN=Tim Folkes RunTimeEnvironment: RALDEV AFSAvailable: FALSE OutboundIP: TRUE InboundIP: FALSE QueueName: MLRMSType: PBS LRMSVersion: OpenPBS_2.3
The EDG Testbed Software - 29
Queries can be posed to the current Information and Monitoring Service using LDAP search commands
An LDAP search consists of the following components
Querying the Information & Monitoring Service
$ldapsearch\ -x\ -H ldap://lxshare0225.cern.ch:2135\ -b 'Mds-Vo-name=datagrid,o=grid\ 'objectclass=ComputingElment‘\ CEId FreeCPUs \ -s base|one|sub
“simple” authenticationuniform resource identifierbase distinguished name for searchfilterattributes to be returnedscope of the search specifying just
the base object, one-level or the complete subtree
The EDG Testbed Software - 30
Querying the GRIS/GIIS Hierarchy
Mds-Vo-name=siteB,o=grid This will look at all the data from siteB
Mds-Vo-name =siteA
Mds-Vo-name =siteD
Mds-Vo-name =siteC
Mds-Vo-name =siteB
Mds-Vo-name =countryA
Mds-Vo-name =countryB
Mds-Vo-name =datagrid
Mds-Vo-name=countryA,o=grid This will look at all the data from
countryA
Mds-Vo-name=datagrid,o=grid This will look at all the data
Mds-Vo-name=siteB,Mds-Vo-name=countryA,o=grid
This will look at all the data from siteB
Mds-Vo-name=siteB, Mds-Vo-name=countryA,Mds-Vo-name=datagrid,o=grid
This will look at all the data from siteB
The EDG Testbed Software - 31
The EDG WMS
The user interacts with GRID via a Workload Management System
The Goal of WMS is the distributed scheduling and resource management in a GRID environment.
What does it allow GRID users to do?
To submit their jobs
To execute them
To get information about their status
To retrieve their output
The WMS tries to optimize the usage of resources
The EDG Testbed Software - 32
WMS Components
WMS is currently composed of the following parts:
1. User Interface (UI) : access point for the user to the GRID
2. Resource Broker (RB) : the broker of GRID resources, performing the match-making
3. Job Submission System (JSS) : provides a reliable submission system
4. Information Index (II) : a specialized Globus GIIS (LDAP server) used by the Resource Broker as a filter to the information service (IS) to select resources
5. Logging and Bookkeeping services (LB) : store Job Info available for users to query
The EDG Testbed Software - 33
WMS UI Commands dg-job-submit
submits a job
dg-job-list-matchlists resources matching a job description
dg-job-cancelcancels a given job
dg-job-statusdisplays the status of the job (submitted, waiting, ready, scheduled, running, chkpt,
done, outputready, aborted, cleared)
dg-job-get-outputreturns the job-output to the user
dg-job-get-logging-infodisplays logging information about submitted jobs
dg-job-id-infois a utility for the user to display job info in a formatted style
The EDG Testbed Software - 34
Example of UI Command Options
dg-job-submit –r <res_id> –n <user e-mail address> -c <config file> -o <output file> <job.jdl>
-r the job is submitted by the RB directly to the computing element identified by <res_id>
-n an e-mail message containing basic information regarding the job (status and identification) is sent to the specified <e-mail address> when the job enters one of the following status:
DONE or ABORTED
READY
RUNNING
-c the configuration file <config file> is pointed by the UI instead of the standard configuration file
-o the generated dg_jobId is written in the <output file>
dg-job-status –i <input file> (or dg_jobId)
-i the bookkeeping information about dg_jobId contained in the <input file> are displayed
The EDG Testbed Software - 35
Job Description Language (JDL) Mandatory for every single JDL file:
• Executable (contains the command name)
• Other attributes:• InputSandbox
• OutputSandbox
Mandatory for JDL file dealing with Data Management:• ReplicaCatalog (contains the Replica Catalog Identifier)
• DataAccessProtocol (contains the protocol or the list of protocols which the application is able to speak with for accessing InputData on a given SE)
If InputData contains at least one PFN and no LFNs, only DataAccessProtocol is mandatory.
If InputData contains at least one LFN, both ReplicaCatalog and DataAccessProtocol are mandatory.
The EDG Testbed Software - 36
Example JDL File
Executable = “gridTest”;
InputData = “LF:testbed0-00019”;
ReplicaCatalog = “ldap://sunlab2g.cnaf.infn.it:2010/ \ rc=WP2 INFN Test, dc=infn, dc=it”;
DataAccessProtocol = “gridftp”;
StdError = “stderr.log”;
StdOutput = “stdout.log”;
OutputSandbox = {“stderr.log”, “stdout.log”};
InputSandbox = {“home/joda/test/gridTest”};
Rank = “other.MaxCpuTime”;
Requirements = other.Architecture==“INTEL” && \ other.OpSys==“LINUX” && other.FreeCpus >=4;
The EDG Testbed Software - 37
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement CE)Element CE)
Information Service (IS)
ReplicaCatalogue(RC)
The EDG Testbed Software - 38
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job SubmitEvent
Input Sandbox
Job Status
submitted
The EDG Testbed Software - 39
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job Status
submitted
waiting
The EDG Testbed Software - 40
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job Status
submitted
waiting
ready
The EDG Testbed Software - 41
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService(JSS)
StorageElement (SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job Status
submitted
waiting
ready
BrokerInfo
scheduled
The EDG Testbed Software - 42
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job Status
submitted
waiting
ready
scheduledInput Sandbox
running
The EDG Testbed Software - 43
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JSS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Job Status
submitted
waiting
ready
scheduled
Job Status
running
The EDG Testbed Software - 44
A Job Submission Example
UIJDL
Logging &Book-keeping
ResourceBroker
Job SubmissionService
StorageElement
ComputeComputeElementElement
Information Service
ReplicaCatalogue
submitted
waiting
ready
scheduled
running
Job Status
done
Job Status
The EDG Testbed Software - 45
A Job Submission Example
UIJDL
Logging &Book-keeping
ResourceBroker
Job SubmissionService
StorageElement
ComputeComputeElementElement
Information Service
ReplicaCatalogue
submitted
waiting
ready
scheduled
running
done
Job Status
Job Status
outputready
Output Sandbox
The EDG Testbed Software - 46
A Job Submission Example
UIJDL
Logging &Book-keeping(LB)
ResourceBroker (RB)
Job SubmissionService (JS)
StorageElement(SE)
ComputeComputeElement (CE)Element (CE)
Information Service (IS)
ReplicaCatalogue(RC)
Output Sandbox
cleared
submitted
waiting
ready
scheduled
running
done
Job Status
outputready
The EDG Testbed Software - 47
EDG Data Management Tools
Tools for Locating data
Copying data
Managing and replicating data
Meta Data management
On EDG Testbed you have EDG Replica Catalog
globus-url-copy (GridFTP)
EDG Replica Manager
Grid Data Mirroring Package (GDMP)
The EDG Testbed Software - 48
EDG Replica Catalog
Based upon the Globus LDAP Replica Catalog (will be replaced by RLS)
Stores LFN/PFN mappings and additional information (e.g. filesize): Physical File Name (PFN): host + full path & and file name
Logical File Name (LFN): logical name that may be resolved to PFNs
LFN : PFN = 1 : n
Only files on storage elements may be registered
Each VO has a specific storage dir on an SE
Example PFN: lxshare0222.cern.ch/flatfiles/SE1/iteam/file1.dat host storage dir
LFN must be full path of file starting from storage dirLFN of above PFN: file1.dat
The EDG Testbed Software - 49
EDG Replica Catalog
API and command line tools addLogicalFileName
getLogicalFileName
deleteLogicalFileName
getPhysicalFileName
addPhysicalFileName
deletePhysicalFileName
addLogicalFileAttribute
getLogicalFileAttribute
deleteLogicalFileAttribute
http://cmsdoc.cern.ch/cms/grid/userguide/gdmp-3-0/node85.html
The EDG Testbed Software - 50
globus-url-copy
Low level tool for secure copying
globus-url-copy <protocol>://<source file> \ <protocol>://<destination file>
Main Protocols: gsiftp – for secure transfer, only available on SE and CE
file – for accessing files stored on the local file system on e.g. UI, WN
globus-url-copy file://`pwd`/file1.dat \ gsiftp://lxshare0222.cern.ch/ \ flatfiles/SE1/EDGTutorial/file1.dat
The EDG Testbed Software - 51
The EDG Replica Manager
Extends the Globus replica manager
Client side tool
Allows replication (copy) and registering of files in RC
Keeps RC consistent with stored data.
The EDG Testbed Software - 52
The Replica Manager APIs
(un)registerEntry(LogicalFileName lfn,
FileName source)
Replica Catalogue operations only - no file transfer
copyFile(FileName source,
FileName destination,
String protocol)
allows for third-party transfer
transfer between: two StorageElements or ComputingElement and Storage Element Space management policies under development
The EDG Testbed Software - 53
copyAndRegisterFile(LogicalFileName lfn,
FileName source,
FileName destination,
String protocol)
third-party transfer but :
files can only be registered in Replica Catalogue if destination PFN contains a valid SE
replicateFile(LogicalFileName lfn,
FileName source,
FileName destination,
String protocol)
deleteFile(LogicalFileName lfn,
FileName source)
The Replica Manager APIs
The EDG Testbed Software - 54
based on CMS requirements for replicating Objectivity files for High Level Trigger studies
production prototype project for evaluating Grid technologies (especially Globus)
http://cern.ch/GDMP
The EDG Testbed Software - 55
Overview of Components
EDG Replica Catalogue
Site1 Site3Site2
GDMP client
The EDG Testbed Software - 56
Subscription Model
All the sites that subscribe to a particular site get notified whenever there is an update in its catalog.
Site 1
Site 3
Site 2
Subscriberlist
Subscriberlist
subscribe subscribe
The EDG Testbed Software - 57
Export / Import Catalogue
Export Catalog information about the new files
produced . is published
Import Catalog information about the files which
have been published by other sites but not yet transferred locally
As soon as the file is transferred locally, it is removed from the import catalogue.
Possible to pull the information about new files into your import catalogue.
Site 1
Site 3
exportcatalog
importcatalog
Site 2
exportcatalog
1)register, publish new files
2) transfer files2) transfer files
1) get info aboutnew files
3) delete files
The EDG Testbed Software - 58
Usage gdmp_ping
Ping a GDMP server and get its status
gdmp_host_subscribe first thing to be done by a site
gdmp_register_local_file Registers a file in local file catalogue but NOT in Replica Catalogue (RC)
gdmp_publish_catalogue send information of newly created files to subscribed hosts (no real data transfer) – update RC
gdmp_replicate_get - gdmp_replicate_put get/put all the files from the import catalogue – update RC
gdmp_remove_local_file Delete a local file and update RC
gdmp_get_catalogue Get remote catalogue contents – for error recovery
The EDG Testbed Software - 59
GDMP vs. EDG Replica Manager
GDMP Replicates sets of files
Replication between SEs
Mass storage interface
File size as logical attribute
Subscription model
Event notification
CRC file size check
Support for Objectivity
Replica Manager Replicates single files
Replication between SEs, CEs to SE.