CERN-INTAS project No. 03-52-4297
Subtask 6.2: OGSA/Globus evaluation for data intensive
applications by A. Demichev (SINP-MSU)
Objectives & executed work
The subtask objectives : An evaluation of new OGSA/WS(RF)-based Grid technology considered as
a candidate for further stages of LCG Tested/evaluated
Toolkits & MW components potentially useful for gLite OMII Globus Toolkit 4 (GT4)
gLite components The subtask participants:
M. Lamanna (CERN IT) V. Pose (JINR) A. Demichev (SINP-MSU) L. Shamardin (SINP-MSU)
Objectives & executed work (2)
In collaboration with ARDA team (CERN) KIAM RAS (Moscow) JINR (Dubna) JRA1 team (CERN) OMII & GT4 teams
Russian participants visited CERN for works related to the subtask (~ 6 PM)
OMII testing/evaluation (while GT4 was being prepared for release...)
Testing/evaluation of OMII (www.omii.ac.ru) MW was interesting for JRA1 from the point of view of potential use of some its components for EGEE MW
The goal of the evaluation was to realize: how well the components of the OMII MW meets
the needs of a large grid environment with ten thousands of users, tens of big virtual organizations, hundreds of resource centres, ten thousands of jobs, Peta bytes of data etc.
OMII testing/evaluation (2)
Description of the functionality provided by the MW Installation and configuration of the MW on different platforms Testing of performance, concurrency, reliability, usability and
management of the services Job Services Data Services Security
Adding an application Adding a new service
Performance and concurrency of dummy services Interoperability with gLite Workload Management System
(WMS) Quality of OMII Support and Documentation
OMII: general features
OMII has several interesting features Web services architecture Account management Resource consumption accounting Conversation mechanism and authorization
module PBAC Process Based Access Control Easy and compact installation (but restricted to
the certain OS) The provided documentation clearly covers the main
topics, but at some points (e.g. budget management, service creation) it was not clear and comprehensive enough
OMII: Some results on performance, concurrency,
reliability and usability Job Service
remained stable with up to 20 concurrent clients a maximal job submission rate of about 6 jobs/min for a 2.4GHz P4 CPU server
node stability: in case of sequential submission of multiple jobs by a user an error
occurred near the 180-th job submission Data Service
reliability: all 1000 upload + download cycles of 10MB big files were successful concurrency: all upload + download cycles of 1..100KB big files were successful
with up to 5 concurrent clients Dummy Services
response time for a non-PBAC dummy service - about 0.1sec with PBAC about 0.7sec
Concurrency - 100 parallel clients executed successfully Adding simple applications is easy A simple new service has been easily deployed
OMII: Evaluation results
OMII is oriented towards web services rather than Grid architecture
There are no Workload Management System & Resource Broker, Information System, Logging and Bookkeeping, Replica Catalogs and Replica Management System
OMII does not support GSI and proxy credentials Virtual Organizations management
OMII: Evaluation results (2)
No interoperability with Globus Toolkit and the gLite WMS
Management and administrative techniques are intended to servicing individual servers and small projects but not large VOs or extended resource infrastructure
More details on OMII evaluation: http://lit.jinr.ru/Reports/annual-report05/omni-55.pdf http://theory.sinp.msu.ru/~demichev/omii/OMII_evaluation-
EGEE3.ppt
OMII's plans...
OMII Technical Roadmap (mid. 2005):“We will collaborate with EGEE (in particular CERN) to ensure that uses of the EGEE software (in particular gLite) will be able to have solutions that interoperate with OMII hosted components.”
Globus Toolkit 4 (GT4) Testing
GT4.0.0 was installed and properly configured on a
number of platforms (in total on 7 machines): Fedora Core 3 – using the Fedora Core 3 Binary Installer
Redhat 9 – using the RedHat 9 Binary Installer + Patches from
http://www.globus.org/toolkit/downloads/4.0.0/
SLC3 (Scientific Linux Cern Release 3.0.4) - using the
Source Instsaller
SL4 (Scientific Linux Release 4.0) - using the RPMs file
from ftp://ftp.linux-ink.ru/pub/SL/40/projects/GRID+HEP
GT4 studying, testing and evaluation
WSRF implementations comparison: GT4-Java (Java WS Core of GT4);
GT4-C (C WS Core of GT4); pyGridWare (Python WS Core); the Perl-based WSRF::Lite; Functional comparison:
Significant commonalities are seen with regard to
dispatching and SOAP processing techniques, though in
each implementation they are different
GT4 studying, testing and evaluation (2)
All the GT4 WSRF implementations Security: support three security protocols:
TLS/SSL transport-level security protocol; Secure Conversation SecureMessage.
WSRF::Lite supports only transport-level security. Persist WS-Resources in memory by default and come
with modules that allow resources to be saved to disk,
providing the ability to survive server failure at the cost of
some performance. Lifetime Management includes resource creation and
resource destruction.
GT4 studying, testing and evaluation (3)
WS-Notification. GT4-Java and pyGridWare do not implement WS-Brokered
Notification and have some other limitations. GT4-C does not implement producer-side notification. WSRF::Lite does not support any Notification specifications.
Authorization. GT4-Java and pyGridWare allows the service developer to provide
custom authorization. GT4-C implements three built-in mechanisms. WSRF::Lite client passes security information to the WS-Resource
through environment variables, leaving the WS-Resource
implementation to implement its own authorization.
GT4 studying, testing and evaluation (4)
Interoperability. Being consistent with the WS-Interoperability, the five
implementations achieve a base level of interoperability with
regard to XML, HTTP, SOAP, and WSDL. But the implementations, nevertheless, are not fully
interoperable. Namespace incompatibilities are the key
concern, as well as several technical details (HTTP headers,
application-specific portions of messages) which lies outside
the WSRF/WSN specifications). By now: OASIS has accepted the WSRF 1.2 (WS-Resource, WS-
ResourceProperties, WS-ResourceLifetime, WS-ServiceGroup, WS-
BaseFaults) as an OASIS standard.
GT4 Testing
Estimation of the overhead expenses of GT4 Java container overhead expenses of GT4 Java container (i.e. first part which is
common for invocation of all services) and client application. Estimation of average times for common WSRF operations
average times for common WSRF operations (in GT4 Java container) on server side and client side and investigation the influence of the number of created resources on the request time.
Estimation of GT4 Java container scalability and robustness GT4 Java container scalability = maximum possible number of
parallel invocations before server failure). GridFTP and RFT testing Further details and numerical results of the testing are available at
http://theory.sinp.msu.ru/~demichev/GT4evaluation.doc
gLite – GT4 WSS compatipility
Studying of the gLite – GT4 WorkSpace Service (WSS) compatibility was carried out by special request from JRA1.
WSS should allow to enforce VO politics in a way which does not contradict to local sites politics.
The goal was to verify compatibility of WSS with gLite and evaluate the service
Succeeded in combining gLite+WSS At the time of the study (Fall of 2005) WSS
proved to be too immature and unstable to be considered as real candidate for gLite component.
Comparative studying: GT4 - gLite
In each main area gLite has services, which are absent in
GT4, but are substantial: Security. Virtual organization support (VOMS gLite); Execution Management - Workload Management System; Data Management - support of storage resource internal
organization: SE, SRM, file catalogues R-GMA (gLite) vs. MDS4: essentially different
approaches: MDS4 - hierarchical R-GMA - relational
Glite Testing
The subtask team participated in testing gLite components: Fireman (file + replica) and Metadata catalogs
http://lcg.web.cern.ch/lcg/PEB/arda/public_docs/CHEP04/CaseStudies/Fireman1.doc
IO-server http://lcg.web.cern.ch/lcg/PEB/arda/public_docs/CHEP04/CaseStudies/gLiteIO
%20-%20testing-v2.pdf
Workload Management System
http://lcg.web.cern.ch/lcg/activities/arda/public_docs/2005/Q1/ARDAatTaiwan-apr05.ppt
AMGA Metadata Service (V.Pose's report at GRID'2006 on 28.06)
The work was carried out in ARDA framework (partially supported by the
INTAS-CERN project) and in close collaboration with JRA1 Team