Abstract Distributed Computing Infrastructures havededicated mechanisms to provide user communitieswith computational environments. While in the lastdecade the Grid has demonstrated to be a powerfulparadigm in supporting scientific research, the com-plexity of the user experience still limits its adoptionby unskilled user communities. Command line inter-faces, X.509 certificates, template customization forjob submission and data access tools require end-usersto dedicate significant learning effort and thus repre-sent a barrier to access Grid computing facilities. Inthis paper, we present a Web portal that solves theaforementioned limitations by providing simplified
Valerio Venturi deceased 25 December 2013
R. AlfieriINFN Parma and Parma University,Viale G.P. Usberti 7/A, Parma, Italy
M. Bencivenni · A. Ceccanti · D. Cesini · E. Fattibene ·D. Michelotto · G. Misurelli · P. Veronesi · V. VenturiIGI and INFN-CNAF, Via Ranzani 13/2, Bologna, Italy
R. Brunetti · L. Gaido (�)IGI and INFN Torino, Via P. Giuria 1, Torino, Italye-mail: [email protected]
A. CostantiniIGI and INFN-Perugia, Via A. Pascoli, Perugia, Italy
E. Ronchieri · D. Salomoni · M. C. VistoliINFN-CNAF, Viale Berti Pichat 6/2, Bologna, Italy
access to Grid and Cloud computing services. The por-tal provides a set of interfaces that support federatedauthentication mechanisms, storage discovery and jobdescription templates, enabling user communities torun specific use cases. We developed the portal frame-work within the Italian Grid Infrastructure where themajor national user representatives drove its design,the implemented solutions and its validation by testingsome specific use cases.
In the last decade, distributed computing infrastruc-tures (DCIs) have supported experimental research forheterogeneous communities (e.g., high energy physicsand bioinformatics) by mostly exploiting the Gridparadigm [1]. The Worldwide LHC Computing Grid(WLCG) [2], one of the largest DCIs, has providedscientists with access to virtually unlimited computingpower and storage, increasing the quality and through-put of their statistical data analysis and leading to thetimely discovery of the Higgs Boson [3]. While theGrid is helping these communities in fulfilling theirobjectives, it still requires from the end-users signifi-cant effort and low-level technical knowledge to fullyleverage its services. Our experience shows that it isoften hard for untrained user communities to port their
applications to the Grid environment due to the inher-ent complexity of its authentication mechanisms andjob submission and data access interfaces.
As a consequence, several Web portals and sciencegateways [4–9] are being developed to hide the afore-mentioned complexities and simplify access to Gridservices. Some existing solutions [6, 10] are domain-specific and typically rely on ROBOT certificates foruser authentication [11]. The main drawback of thisapproach is that robot certificates hide whole usercommunities behind a small set of credentials, thuslimiting the granularity and effectiveness of the Gridmiddleware tracing and authorization mechanisms.Furthermore, Grid security policies restrict the use ofrobot certificates to the submission of computationaljobs that run only pre-determined applications. Hence,this approach is unsuitable for portals that need tosupport heterogeneous user communities. Other solu-tions, such as the Web Services Parallel Grid Runtimeand Developer Environment (WS-PGRADE) [12], arenot tied to a single community and allow users toaccess computing and storage services through a Webinterface.
This paper describes the Italian Grid Infrastructure(IGI) [13] Web Portal, designed and developed to pro-vide simplified access to Grid and Cloud resourcesto wide-ranging scientific communities. The portalprovides customized user interfaces and support forfederated authentication mechanisms, storage discov-ery and job description templates; it is designed tobe usable in the context of different DCIs. Noneof the existing Web portals and Science Gatewaysavailable at the time of this writing satisfied all theabove requirements or were adaptable to the needsof our user communities. In order to reduce thedevelopment effort and provide a robust solution,the portal glues together several existing componentssuch as as MyProxy [14], WS-PGRADE, DistributedInfrastructure with Remote Control (DIRAC) [15],and Grid User Support Environment (gUSE) [16].The portal Grid components are already in produc-tion, whereas the Cloud ones are in a prototypicalevolving form. Since May 2013, the portal has sup-ported 116 users, distributed in 12 VOs, who havesubmitted 17.8k jobs via DIRAC and 3.4k jobs viaWS-PGRADE for a total amount of about 380kCPU-hours.
The paper is organized as follows. Section 2presents related work which is classified according to
the requirements of our user communities. Section 3provides a high-level overview of the portal architec-ture, highlighting the authentication and authorizationcomponents in Section 4 and focusing on the com-puting and storage services in Section 5. Section 6presents a set of use cases selected to validate the por-tal. Finally, Section 7 concludes the paper and presentsdirections for future work.
2 Related Works
Grid-enabled Web portals and science gateways haveproved to be effective mechanisms for exposing com-puting resources and distributed systems to user com-munities without forcing them to deal with the com-plexities of the underlying systems. For this reason,several science collaborations and projects have devel-oped their own Web application [4–9]. Even if theseWeb applications and portals have requirements sim-ilar to most known customer-oriented portals (typ-ically services that include support for login, jobsubmission and file management operations), theybecame domain-specific oriented applications adopt-ing solutions that are difficult to be exported to otherdisciplines.
In light of these limitations, we developed the IGIportal by adopting and integrating specific technologi-cal solutions able to provide general and customizableWeb interfaces. The portal aims to support smartusers, who belong to different scientific domains, dur-ing the definition of their jobs, the submission oftheir applications and the exploitation of tools fortheir needs. This is what distinguishes our solutionsfrom others that, like science gateways, tend to sup-ply customized solutions by offering a limited setof operations. Although the literature reports manyprojects for accessing distributed resources throughWeb portals and gateways [12, 17–19], we devoted ourattention to those supporting three elements: federatedauthentication, storage discovery, and job template.They cover most of the Grid activity scenarios suchas granting federated access, customizing templatesfor job submission and locating storage endpointsfor data replication. We focused on each element byselecting a subset of solutions, assessing their weak-nesses and strengths, and also considering the require-ments from our supported communities (described inTable 1).
Accessing Grid and Cloud Services Through a Scientific Web Portal 161
Table 1 The requirements collected from our usercommunities
Services Requirements
Authentication Single-sign-on based authentication
and Authorization Personal certificate handling
Certificate provisioning on demand
Workload Management Managing workflows
Detailing Job Description Language
(JDL) customization
Getting customized Web interface
applications
Data Management Simplified access to data storage
Simplified data moving
Concerning the federated authentication based ononline Certificate Authority (CA), we considered thefollowing solutions:
GridCertLib [20] uses a short lived credential servicethat issues certificates lasting maximum 11 days,hence the job lifetime must comply with this limi-tation. This is one of the reasons why we decidedto discard GridCertLib in favour of solutions thatsupport long-lived credentials (i.e. a 13-monthsvalidity) to address the lifetime constraint.
The CILogon [21] solution provides two main typesof long-lived certificates each one with a specificlevel of assurance [22]: openID – the lowest, silver– the highest. While the openID relies on an authen-tication mechanism (e.g. Google account) unac-ceptable for the organisation coordinating the trustfabric for e-Science in Europe (EUGridPMA [23]),the silver, although accepted by EUGridPMA, onlycovers members of the InCommon federation [24],therefore excluding our referential research com-munities.
The Terena Certificate Service (TCS) [25] allows usersto obtain their X.509 credentials by using a spe-cific procedure: users must authenticate themselvesto the Terena portal by using their federated iden-tities, and obtain their X.509 certificates registeredin the adopted browsers. Then users need to requesta certificate and manage a key pair to upload theircredentials into the IGI portal. TCS herein requiresusers to handle their certificates.
Due to the limitations of all the three solutions,we started designing our Web portal on top of an
Enterprise Java Based PKI Certificate Authority(EJBCA) [26] software that provides an EJBCA Webservice and programmatic interfaces for different cer-tificates signing request formats.
Referring to the storage discovery, we focused onthe existing solutions to manage files that match spe-cific file features, such as searching, drag and drop,and sharing:
Pydio [27] is based on a modular architecture allow-ing new portals to integrate external services andsupport further protocols by just developing plug-ins. In our experience, it has proven to be scalablein complex scenarios and supports multi-data andmulti-users handling with suitable performances.
The ownCloud [28] framework could be a valid alter-native to Pydio, but we discovered some weak-nesses about scalability and performance.
The elFinder [29] solution provides PHP and Pythonconnectors for portals binding, but the Java oneis under development determining an importantlimitation for the Java-based portlets.
Consequently, we chose Pydio as file manager toolwithin the IGI Web portal. Since most recent browsersare compatible with HTML-5 and handle chunkingoperation through HTML-5 directly, we decided toadopt jQuery File Upload [30] for the file uploading.
The job submission was another important task totackle. In particular, we had to consider that our pri-mary research community – high energy physics –uses its own middleware. For that reason we tookDIRAC [15] for the following main reasons: it pro-vides standard components with a well-defined set ofextensions provided by the LHCb experiment [31]; itis also effective in sustaining multiple small groupsaccess and supports code refurbishing, too. The IGIWeb portal leverages DIRAC to offer general jobsubmission services to a variety of scientific commu-nities. Furthermore, it provides users with templatesfor customizing job description.
3 Architecture
We designed the portal architecture with the aimto reuse existing and maintained open solutions forall the features mentioned in Section 1. In addition,we adopted Liferay [32] – a popular open sourceportal and collaboration software made of functional
162 M. Bencivenni et al.
units called portlets – to ensure a modular Web por-tal structure. The portlets are components that builddynamic Web contents and use the JSR168 [33] andJSR286 [34] standards to simplify portlets migration.We implemented portlets for each specific service(e.g. user authentication, job submission, and datamanagement). This combination makes it easy toenrich the portal with new functionalities.
The IGI Web portal architecture, presented inFig. 1, can be conceptually divided into five main lay-ers. Each one contains components internally devel-oped for the portal as well as provided by third-parties.Let us start describing the Fig. 1 content.
At the highest level, the Portal AuthN (Authen-tication) and AuthZ (Authorization) Layer verifiesall the mandatory credentials provided by the users:X.509 certificate, Virtual Organization (VO) mem-bership and Identity Provider (IdP) trusted by theportal. The certificate univocally identifies users andensures a secure communication across the distributedresources. The AuthN portlet interfaces with single-sign-on solutions such as IdPs federations and theportal IdP. The AuthZ portlet collects users’ creden-tials during the first access to the portal. For these
two portlets, we added a MySQL-based DataBase(DB) [35], called portal DB, to the default Liferay DBto collect users’ information. Not only does it storeusers’ personal information (such as name, surnameand mail), but also data concerning personal certificateand VOs membership.
At the second level, the External AuthZ and AuthNServices Layer supports the upper level in a set ofoperations:
– Users’ credentials vetting leverages the IdPs fed-erations and VOMSes (VO Membership Ser-vice) [36] components invoking single-sign-on ormembership-credential requests.
– Providing missing credential exploits the portalIdP, online CA, and VOMS components acting asfallback and supplying an alternative credentialchain.
– Storing delegated credentials (proxies) makes useof the MyProxy [14] components that manageshort and long proxies.
In addition, the CA-bridge component is paramountfor the interconnection among the AuthZ portletand the other components involved in this layer,
Fig. 1 The IGI Web portalarchitecture: rectangularshapes identify internallydeveloped components,rounded rectangular shapesshow third-partycomponents with localconfiguration, and finallyoval shapes representthird-party componentsused as they are
Accessing Grid and Cloud Services Through a Scientific Web Portal 163
performing all the necessary steps to validate usersidentity, to provide a X509 certificate, to generate theGrid credentials and to archive user data. The CA-bridge, MyProxy server and online CA componentsimplement a certificate provisioning service integratedin the portal framework. For the reasons detailed inSection 2 the online CA supplies MICS (Member Inte-grated Credential Services) certificates [37] with a13-months validity.
At the middle level, the Portal Services Layercontains all the portlets implementing data manage-ment, job submission, application and workflow sub-missions, and cloud provisioning. While a job is asequence of key pairs (attribute, value) based on theJDL, a workflow is a sequence of connected jobswhere the execution of one or more steps dependon the results of the previous ones. WS-PGRADEconsists of various portlets (following the Liferay sup-ported standards) providing different functionalitiesranging from user registration to data managementup to workflow handling; however, we only used theworkflow feature. In order to hide the complexityof workflow configuration and execution to specificcommunities, we developed different portlets based onthe Application Specific Management (ASM) APIs,available in the WS-PGRADE framework. Theseportlets provide application-specific interfaces and usethe workflows defined in WS-PGRADE. Both WS-PGRADE and ASM interact with the Grid and Clouduser support environment (gUSE). The Job portletallows the portal to submit jobs over computing sys-tems, while the Cloud portlet enables an interfacetowards user-specified virtual machines in an Infras-tructure as a Service scenario. At last the Data Man-agement portlet interfaces with the storage systems.This level allows to add custom Web interfaces justby creating new portlets to support new applications,providing a set of scenarios suitable for users’ needs.
At the second to last level, the External Data andComputing Layer provides the tools to handle datarequired by the portlets in the Portal Services Layer.The Data Mover component allows users to transferdata among Grid resources, hiding all the complex-ity of accessing data so that users can spend no effortto learn Grid data tools. Then, the DIRAC componentintegrates computing resources providing solutions forrouting jobs to the proper environment on the basisof the JDL requirements and the user VO permis-sions and roles. Finally, the gUSE and DCI Bridge
components assist the workflow submission: the for-mer provides a set of Web services that interpret andhandle a workflow to consume computing resources;the latter, a web application, represents a compo-nent between gUSE and the DCIs creating a standardaccess to Grids, clusters and Clouds.
At the lowest level, the Middleware ResourcesLayer consists of the Grid and Cloudmiddleware com-ponents to provide physical and virtual resources. Theinteraction with the Grid services is already in produc-tion, whereas the Cloud one is in a prototype versionand still in pre-production.
In the following sections we detail the Portal andExternal AuthZ and AuthN layers and the Portal andExternal Data/Computing Services layers.
4 AuthZ and AuthN Details
The two AuthZ and AuthN Layers (described inSection 3 and shown in Fig. 1) determine the firstaccess to the portal where users through registrationmust provide the mandatory qualifications to accessGrid or Cloud services.
During the authorization phase users must providethe following information: a X.509 certificate issuedby a trusted EUGridPMA-member CA, the affilia-tion to a recognized VO and a trusted IdP. If a useronly has a subset of these credentials, the portal coulduse them to provide the missing ones as described insubsections 4.1 and 4.3. Table 2 shows the creden-tials combinations required to successfully concludethe registration phase: the x symbol specifies a miss-ing information; the o symbol states an availableinformation.
The request for a personal certificate and its man-agement (such as storing and renewal) often representtedious operations that many users wish to avoid. Weaddressed this issue: by interfacing the portal with anonline CA, which provides X.509 certificates to usersauthenticated by a federated identity management
Table 2 The required credentials for the registration phase
Case IDP member X.509 certificate VO
1 x o o
2 o x x
3 o o o
164 M. Bencivenni et al.
system; and by implementing a service to managethese certificates on behalf of the users. Depending ontheir credentials, users can select whether to uploadtheir certificate or ask for a new one through the por-tal. The AuthZ portlet is able to manage each step ofthe registration phase and to perform all the actionsaccording to users’ different choices.
During the authentication phase, the portalleverages a federated authentication mechanism –based on the Security Assertion Markup Language(SAML) [38] – to offload the portal from managingusers’ credentials and to exploit a single-sign-on solu-tion. Indeed, the portal trusts all the IdPs belongingto the EDUgain federation [39] that interconnectsdistributed identity federations around the world;therefore all members of these IdPs can access theportal components by using the credentials con-tained in their own organization IdP. Since Liferaynatively supports the Central Authentication Ser-vice (CAS) [40] but not SAML, the portal uses theCasshib [41] software; this enables the CAS server toact as a Shibboleth service provider [42].
The following subsections detail the registrationsteps for all the cases described in Table 2 and theauthentication mechanism. The registration phase col-lects users’ information authorizing them to accessGrid and Cloud resources.
4.1 Registration to the Portal
Users holding a valid X.509 personal certificate anda VO membership can access the portal: if they areregistered in a trusted IdP (case 3 in Table 2), the por-tal retrieves their personal information from the IdPitself; otherwise (case 1 in Table 2), the portal regis-ters users without IdP in an IdP internal to the portalby using the information got back from their personalcertificate. For case 3, the registration phase consistsof four steps as shown in Fig. 2: retrieving personaldata from the IdP, uploading certificate, declaring theVO membership, and encrypting credentials.
Step 1. The registration page redirects users towardstheir IdPs. Once validation has succeeded, the por-tal retrieves personal information, such as first andlast name, email address and institute, and storesthem in the portal DB.
Step 2. Users have to upload their certificate andprovide the passphrase to decrypt it. Hence, the
portal gets the passphrase to create two proxycertificates compliant to GSI (Grid Security infras-tructure) [43] and RFC 3820 [44] with the samelifetime as the personal certificate. The portalalso encrypts the proxies by using a randompassphrase, saves the encrypted proxies in a ded-icated MyProxy server and finally deletes thepersonal certificate following the EUGridPMAguidelines on private key protection [45]. Ulti-mately, the portal inserts the certificate data intothe DB.
Step 3. Users have to declare their VOs member-ship. By the VOMS interfaces the portal assessesthe claimed membership and then retrieves theirVO information (such as membership, roles andgroups) through a secure connection. Users can settheir roles and groups for each VO they belong toand set a default VO. Then, the portal inserts theVO information into the DB.
Step 4. Users have to choose a personal passphraseto encrypt the proxy certificates. The portalreplaces the old proxy passphrase, randomly gener-ated during the second step, with this new one thatis valid until users upload a new certificate. Then,the portal discards the new passphrase and registersthe users with the “active” role in the DB.
4.2 Authentication Mechanism
The authentication process consists of two steps asshown in Fig. 3: handling single-sign-on validation,and retrieving Grid credentials.
Step 1. The portal through the authentication portletredirects users to their IdP login page. Then the por-tal redirects them to the main page where they haveaccess to all the tasks and services made available;otherwise, the portal exposes the registration pageby default.
Step 2. Users select a VO from a list containing allthe VOs they declared to be part of, and insert thepassphrase set during the registration phase (seesteps 3 and 4 in subsection 4.1). The portal retrievesa proxy by querying the MyProxy server (long)and saves it setting its lifetime to N days (userscan choose N in the interval 1–7). The portal thenmoves the proxy to the MyProxy server (short) andcontacts the appropriate VOMS to add extension tothis short-lived proxy.
Accessing Grid and Cloud Services Through a Scientific Web Portal 165
Fig. 2 The registrationsteps via Grid credentials:the open arrows describeactions performed by users
4.3 Authorization Via Online CA
Users without a valid X.509 personal certificate butmember of a trusted IdP (case 2 in Table 2) can auto-matically get a personal certificate and obtain a proxy
by the portal, which combines the functionalities of anonline CA with those of a MyProxy server. In case 2,the CA relies on pre-existing identity data maintainedby users’ home institutions, which act in all respectsas a trusted registration authority.
Fig. 3 The authenticationmechanism: the openarrows describe actionsperformed by users
166 M. Bencivenni et al.
Concerning the online CA, we created a proof-of-concept to deal with the security level imposed by theEUGridPMA policy. Figure 4 sketches the steps tofulfil the authorization chain from login up to the cre-ation of proxy credentials. The authorization processconsists of the following two steps:
Step 1. Retrieving personal data from IdPs is thesame as step 1 in Section 4.1. In addi-tion, the portal generates a first token andappends it to the redirected URL towards theCA-bridge endpoint.
Step 2. CA-bridge requires a further authenticationbecause the Shibboleth delegation is stillin an experimental phase. This procedureis completely transparent to the users dueto the single-sign-on mechanism. The CA-bridge makes use of the retrieved informa-tion and performs the following operations:it generates a second token; it compares thisnew token with the one created by the portal;and it verifies the matching. If and onlyif the two tokens are equal, the CA-bridgetriggers the following actions: it asks users
to provide a password; it generates a pri-vate key and a certificate signing request onbehalf of the users; and it sends the requestto the CA to have it signed. On receivingthe certificate, the CA-bridge stores the longterm proxy credential in the MyProxy server(long), destroys the certificate private key,and adds users to a catch-all VOmanaged bya dedicated VOMS server. Ultimately, theCA-bridge stores certificate and VO pecu-liar information (e.g. VO name, VO group,and certificate distinguished name) into theportal DB.
By taking all the required steps (retrieving personaldata from IdPs and triggering CA-bridge interactions),users get a 13-months lifetime certificate and belongto a catch-all VO without performing any other cre-dential handling.
5 Data and Computing Services Details
The Portal and External Data/Computing ServicesLayers (described in Section 3 and shown in Fig. 1)
Fig. 4 The registration steps via Online CA: the open arrows describe actions performed by users
Accessing Grid and Cloud Services Through a Scientific Web Portal 167
bridge users towards Grid and Cloud resources cov-ering all the requirements described in Table 1. Thecomputing services scenarios comprise the submis-sion of: simple Grid jobs with binaries and input data;workflows with complex use cases; specific applica-tions oriented to custom interfaces; and Cloud provi-sioning with IaaS (Infrastructure as a Service) alloca-tion. The data service scenario abstracts moving dataasynchronously among Grid Storage Elements (SEs)in a drag and drop way. The following subsectionsdetail the foregoing outlines.
5.1 Simple Grid Jobs
Users submit jobs by uploading their executables andinput files and by adopting JDL to specify the exe-cutable parameters. The portal implements this sce-nario by interfacing the Job portlet with a multi-VOsconfigured DIRAC server. Despite the DIRAC serverprovides its own Web interface, at the design levelwe chose to disable it exploiting the Job portlet thatuses the DIRAC command line interface to handle thecommunication between the portal and the DIRACserver. Consequently, we can both have a single accessgate (i.e. the portal) and avoid duplication of fea-tures. In addition, the Job portlet allows users to: buildtheir JDL by selecting the correct values from a listof attributes and settings; save JDLs as templates forsharing and reusing purposes; show the list of submit-ted jobs; monitor the job state during its execution;retrieve the output; resubmit an ended job; log files inthe end.
By the DIRAC server, users submit their jobs byexploiting the pilot job mechanism [46] – suitable formassive job submissions. Then the server sends jobsto a central task queue, and later the Grid resourcesget them asynchronously by executing pilot jobs.The mechanism decouples resource allocation and jobmanagement. The DIRAC server also instantiates vir-tual resources for the job executions discerning amongGrid and Cloud resources. It automatically performsthis operation according to the job requirements.
5.2 Workflows
The workflow submission is a step-by-step procedurefor performing complex computations on differentresources optimizing the overall task. Each step rep-resents a specific portion of the entire calculation –
identifying what we call a job. It can follow condi-tional constraints according to the evolution of thecomputation. The adopted acyclic workflow structurecan assume a simple or complex form in relation to theaddressed problem. Using a subset of WS-PGRADEportlets (workflows creation, submission and manage-ment) and wrapping them through the gUSE compo-nent the portal allows the support to both types ofworkflows.
5.3 Specific Applications
Some communities may need to submit ad-hoc appli-cations exploiting job and workflow submissionsaccording to VO-specific requirements. In this case,we have adopted the following procedure to adaptexisting applications to the Web portal: first, we anal-ysed the portability of the application on the Grid;then, we checked hardware and software requirements(such as RAM, number of CPU cores, and specificlibraries) as well as required input and output files, andapplication parameters needed for retrieving log andoutput files at runtime. In addition, we created a scriptthat takes care of getting users’ inputs, parametersand files, executing the application and retrieving theoutput produced during the entire computation. Thisutility script translates the application requirements inthe proper execution environment, enabling featuressuch as monitoring the simulation and retrieving run-time logs. If the application required a job submission,we defined an ad-hoc JDL template, included in thejob portlet and available to the users for the submis-sion of the job; if the application required a workflowsubmission, we created a custom workflow in the WS-PGRADE framework and we implemented a portletthat exploits such workflow through the ASM APIsprovided by WS-PGRADE.
There are several advantages in using a workflowwith a custom interface to handle a specific appli-cation. The job submission only requires filling aWeb form – a simplified access to the Grid. It isstraightforward to alter the lifetime of long runningGrid jobs by considering the queue maximum wall-clock time, and applying a stop-and-go mechanismon the various resources. If a job fails, it is possi-ble to either rely on standard Grid recovery mecha-nisms or provide the application with a proper toolhandling the problem and adopting the appropriatesolution.
168 M. Bencivenni et al.
Inspecting produced files and monitoring theapplication are paramount items for long runningapplications. The job perusal solution (provided byWMS [47]) supports the first item but works only withsmall-size output files to avoid network overload. Toovertake these limitations and to integrate applicationmonitoring into the IGI portal, we developed a newmechanism, called application progress monitoring,exploiting Grid SEs and Storage Resource Manager(SRM) command line interfaces (CLIs) to make tem-porary and partial output files available to be inspectedat runtime. The adopted SRM CLIs mechanism copiesselected files from the computing resource where thejob is physically running to a Grid SE that stores thefiles. Then users can directly access these SE files viathe job or ASM portlet.
At last to simplify to the user the choice ofinput files and output destinations, these application-specific portlets interact with the Grid data manage-ment services through their specific utilities.
5.4 Cloud Provisioning
Some communities may need to run their applicationsexploiting the Cloud paradigm. The portal implementsthis scenario by interfacing a Cloud portlet with aset of services that supply resources according to theIaaS provisioning model. The services, fed with apre-built configuration file, can interact with variousIaaS Cloud providers exploiting the benefits offeredby existing Cloud platforms such as WNoDeS [48],OpenStack [49] and Opennebula [50]. The devel-oped portlet exposes a Web interface for each servicesimplifying users’ tasks to create and manage newinstances.
To instantiate new virtual machines (VMs), usershave to upload their own SSH public key [51], or gen-erate a SSH private and public key pair through theportal. This is necessary to allow logging into a VMwith root privileges without requiring any password.The keys generated by the portal can be retrieved byusers in a secure way. Users can then create new VMinstances choosing from a list of images preloaded ina repository. For each image it is possible to selectthe size of the cloud environment (defined accordingto the number of cores, memory and disk size) andhow many instances have to be created at a time; eachimage has a range size that depends on the Cloudplatform it belongs to. As soon as users create new
instances, a list of their VMs is displayed together withinformation such as architecture, size and status. Byselecting the instance name, users are automaticallylogged into the VMwith root privileges through aWebterminal [52] that is part of the portal. The portal letsusers select VM images offered by different Cloudproviders via a repository, which access depends onthe resource providers internal policies.
The portal Cloud feature is in a prototype ver-sion but close to a production-quality solution. Wepresented at the EGI Community Forum 2013 [53]its successful validation by the World Wide e-Infrastructure for NMR (WeNMR) [54] virtualresearch community to address its CING use case(fully detailed in [55]). The prototype uses the WorkerNodes on Demand Service (WNoDeS) Cloud com-mand line utilities – developed in the context ofthe EGI Federated Cloud task force [56] to simplifythe handling of further Cloud needs and site pecu-liarities – leveraging the strength of different Cloudplatforms such as OpenStack and WNoDeS. Further-more, it interacts with the Stratuslab Marketplace-based repository [57] to get preloaded images infor-mation. Figure 5 shows the Cloud portlet interface forchoosing a custom WeNMR image.
5.5 Data Management
In a standard Grid environment users may benefit froma set of command line tools to perform data man-agement tasks such as copying files on a Grid SE,registering files in a LCG File Catalogue (LFC) [58],and replicating files on other Grid SEs. To savelearning effort, we designed and implemented a datamanagement service [59] sketched in Fig. 6. The ser-vice includes several elements intercommunicating ina secure way.
The Data Mover component controls and managesevery step of the file transfer operations: upload-ing and downloading files. It controls data transfersthrough an external storage service – composed ofa set of Storage Resource Manager (StoRM)-basedportal SEs [60] acting as a cache memory for thefiles – until they are transferred to or downloadedfrom a Grid SE. The Data Mover is a Web Pydio-based data management service that implements andextends the functionalities of the Grid data manage-ment command line tools, exposing a Web interfacethat manages either Grid files or some other types of
Accessing Grid and Cloud Services Through a Scientific Web Portal 169
Fig. 5 The Cloud portletinterface for choosing acustom WeNMR imagepreloaded from theStratuslab Marketplacerepository
files. We developed a plug-in for handling data that bythe IGI portal allows users to browse the content of theVO file catalogue and to perform operations on eitherthe logical data (affecting the information containedin the catalogue) and/or the physical files (involvingthe files physically stored in the SEs). The LFC con-tains the mapping of a generic file name, the LogicalFile Name (LFN), to one or more physical locationsof a file on the Grid. Each file is univocally identifiedby a Global Unique Identifier (GUID), a non-human-readable fixed-format string that identifies an item ofdata. Each file can be replicated in different locationsand each replica is identified by a Site URL (SURL) orPhysical File Name (PFN). Table 3 details the possibleoperations performed on data.
To upload a local file to a SE, users can select theSE from a list of possible destinations dynamically
produced according to the available storage space andusers’ VO.When users select a specific SE, the systemtries to upload the file to that destination; on failure,it randomly tries other SEs from the same list. If usersneed to download one or more files from the Grid, theycan choose to copy it/them either to a local destinationor to an external server. To avoid transferring big filesto users’ local space, we set a limit of five Gbytes tothe maximum file size to be downloaded locally. Thesupported protocols for the data transfer to externalservers are SFTP, FTP(S), and HTTP(S). To performthe operations shown in Table 3 on logical and physi-cal data, the Data Mover directly communicates withthe standard Grid services, such as LFC and SEs.Therefore the data management service architectureallows a complete decoupling of theWeb interface, thecontrol service (Data Mover) and the physical storage
Fig. 6 The IGI Portal datamanagement servicearchitecture
170 M. Bencivenni et al.
Table 3 The possible operations performed on data
Data Types Operations
Logical Data
Creating a new folder;
Deleting an empty folder;
Renaming a folder or file (changing the LFN);
Moving a folder or file (changing the LFN);
Getting detailed information about a file
(LFN, GUID - Global Unique Identifier,
list of replicas, owner, ACL);
Sharing a file with other portal users.
Physical DataReplicating files on different storage elements;
Downloading files.
Logical and Deleting files;
Physical Data Uploading files.
(portal SEs). In addition, the number of portal SEscan be extended, if necessary, to provide final userswith a reliable and fast data transfer in case of a portalbottleneck (e.g. many users and many transfers).
6 Use Cases
The Italian and European Grids were initially createdto satisfy the computational and storage requirementsof the high energy physics communities applicationsthat had driven the shaping of the infrastructure. In thelast decade, as the result of the participation to vari-ous Italian and European projects (i.e. the EGEE [61]series, Grid.it, INFNGrid and EGI-InSPIRE [62]), thesame infrastructure [61] has increasingly supportednew user communities belonging to various scientificdomains such as computational chemistry, bioinfor-matics, astronomy and astrophysics, earth science,mathematics, and engineering. These new communi-ties differ in computational and storage requirements,composition, computational experience and size. Port-ing their applications and creating Grid-based comput-ing models (that fit their needs) are sometimes hard torealize. Dedicated support, in fact, assists the devel-opment of high level Web interfaces addressing theircomputing models to hide the Grid complexity. Wealso designed the IGI Web portal to create application-specific Web interfaces enabling users to access thestandard functionalities of the Grid infrastructure andto run their computational applications by hiding the
inner complexity. For this reason, we developed a setof specialized portlets that run specific use cases basedon commonly adopted applications. Table 4 lists theapplication portlets currently hosted by the IGI portal,where the computational requirements are related tothe specific use cases. It also shows the data require-ments per single job (only for the most demandinguse cases) and the exploited IGI portal features. It isimportant to point out that all the implemented solu-tions are the result of a collaboration between theIGI unit dealing with user support and the user repre-sentatives who drove their design and validated theirimplementations.
The following subsections describe two of the inter-faces developed so far: the Fluka portlet [63] that isan example of a pure high throughput use case (i.e.,many independent, single core jobs); and the portletfor Theoretical Physics applications that is an exam-ple of a use case that needs long parallel jobs andcheckpointing.
6.1 Fluka MonteCarlo Simulation
Porting an application to the Grid environment meansexecuting code on multiple Grid resources with dif-ferent input files simultaneously. As, in general, thesesimulations can run independently from each other,they can take advantage of their distribution on awide computing infrastructure. This is the case of theFluka code resulting from a collaboration betweenINFN and CERN – a general Monte Carlo tool forthe calculations of particle transport and interactionwith matter. It has a wide range of applications infields such as cosmic ray physics, particle physics,and neutrino physics. We compiled Fluka as a staticbinary by using the gfortran compiler and other opensource libraries. The static compilation of the packageensures that the program is binary compatible with theGrid computing resources, preventing the incompati-bility errors associated with the usage of dynamicallyloaded libraries.
The sequential Fluka program follows the so-calledparameter study (or “parameter sweep”) approach touse the Grid. In our case, the various Fluka jobs workwith different seeds for a different set of initial con-ditions. For the above mentioned reasons the Flukaapplication portlet makes use of the DIRAC frame-work to perform a massive job submission. In suchway a simple and intuitive interface assists users in
Accessing Grid and Cloud Services Through a Scientific Web Portal 171
Table4
The
applicationportletsforspecificapplications
createdin
theIG
IPo
rtal
Application
Scientificdomain
Com
putatio
nalrequirements
Mainportalfeatureexploited
portlet
oftheaddressedusecases
Ansys
Engineering
simulationsoftware
Lim
itednumberof
multicorejobs
Input/O
utputd
atamanagem
entembedded
intheportlet
License
handlin
gCheckpointin
gDatamanagem
entcapability
tomonito
rtheexecution
forlong
run
Workflowhandlin
gthroughWS-PG
RADE
Fluka
PhysicsMonteCarlo
simulation
Embarassingparalleljobs
DIRACpilotjob
subm
ission
fram
ework
package
Input/O
utputd
atamanagem
ent
Input/O
utputd
atadone
with
thegenerald
atamanagem
entp
ortlet
Crystal
Abinitioquantum
chem
istry
Longparallelm
ultin
odejobs
(MPI)
MPI
Jobsubm
ission
viaWS-PG
RADEworkflow
forcalculations
oncrystals
Checkpointin
gim
plem
ented
Outputretrieved
with
thestandard
datamanagem
ent
Quantum
Espresso
Electronicstructurecalculations
Fewlong
parallelm
ultin
odejobs
(MPI)
MPI
Jobsubm
ission
viaWS-PG
RADEworkflow
andmaterialsmodeling
10GBof
output
dataperrun
Checkpointin
gim
plem
entedas
aworkflow
Checkpointin
gneeded
Input/O
utputd
atadone
with
thegenerald
atamanagem
entp
ortlet
Nem
oOceanographicmodeling
Fewlong
paralleljobs(M
PI)
MPI
Jobsubm
ission
viaWS-PG
RADEworkflow
Input/O
utputd
atamanagem
entembedded
inthedatamanagem
entp
ortlet
Blast
Alig
nmentsearchtool
for
Highnumberof
multicore
DIRACpilotjob
subm
ission
fram
ework
genomesequences
independentjobs
Input/O
utputd
atadone
with
thegenerald
atamanagem
entp
ortlet
Venus
Classicaltrajectories
simulations
Embarassingparalleljobs
DIRACpilotjob
subm
ission
fram
ework
Input/O
utputh
andled
with
thestandard
datamanagem
entp
ortlet
EinsteinTo
olkit
TheoreticalPh
ysics
Longparalleljobs
MPI
Jobsubm
ission
viaWS-PG
RADEworkflow
10GBof
output
dataperrun
Input/O
utputd
atamanagem
entembedded
intheportlet
checkpointingneeded
DMRG
Electronicstructuremodeling
Highnumberof
singlecore
independentjobs
DIRACpilotjob
subm
ission
fram
ework
Input/O
utputd
atadone
with
thegenerald
atamanagem
entp
ortlet
172 M. Bencivenni et al.
the management of a multitude of sequential jobsfrom submission up to output retrieving. We tunedthe portlet for a Fluka-based use case provided bythe SPES [64] experiment community, which aimed atevaluating the low-energy neutrons treatment and theirrelated gamma emission inside the experimental appa-ratus. Within this particular use case, users submitteda set of 1000 jobs and, after about 32 hours, collectedthe results from 847 successful job submissions for agrand total of about 10.1k CPU-hours. The remain-ing jobs failed due to hardware and/or software relatedproblems that need to be investigated. The average runtime of successful jobs was about 12 hours.
Using the Fluka portlet (see Fig. 7), users can setthe needed fields such as the name of the submittedjobs to distinguish each set of submission; the inputfile already stored in the SE and selectable from theuser by using a dropdown menu; the output folder inthe LFC directory where the output will be stored; andthe number of parameter sweep jobs to be run in asingle submission.
6.2 Theoretical Physics Applications
In the theoretical physics field, applications requireboth serial jobs-based computation and High Per-formance Computing (HPC) resources ranging fromlarge computing clusters up to specialized super-computers, according to the problem size. Amongthe latter, we can mention lattice quantum chromo-dynamics, fluid dynamics and numerical relativity.
Building on resources available on IGI, the INFNtheoretical physics community (Theophys VO) hasrecently started a project [65] aiming at fulfilling therequirements of medium-sized HPC applications. Thisproject is achieving important results for the Theo-phys community, such as the deployment of a commoninfrastructure for parallel applications that shares thesame platform with the other Grid resources availablefor serial jobs. On the other hand, there are aspects thatstill have to be improved such as the usability of theinfrastructure. For this reason the Theophys commu-nity undertook a collaboration with the IGI Web portal
Fig. 7 The Flukaapplication portlet
Accessing Grid and Cloud Services Through a Scientific Web Portal 173
developers, aiming at supplying a customized portletsee Fig. 8) conceived to ease the parallel job life cyclein all its stages:
– Creation/submission: The data management forthese applications is based on two Grid directoriesused to upload respectively the executables andthe input files. Three dedicated drop-down menussupport users in the executables uploading, filesuploading and resources selection (defined by acommunity administrator). To create a new job,users only have to select files and resources fromthe proper menus and to provide execution details(such as executable arguments, output filename,and HPC computational environment)
– Monitor: the portlet makes available standard out-put and standard error files; since they are period-ically updated, users can check the job state at anytime;
– Output retrieval: after the job completion, thejob packs the whole execution directory into atar archive and copies it back in the previouslyselected Grid directory. This feature is particularlyuseful for checkpointing and debugging purposes.
Among the different parallel applications usedby the Theophys community, we selected the Ein-stein Toolkit [66] – an open software providing thecore computational tools needed by relativistic astro-physics, i.e. to solve the Einstein’s equations coupledto matter and magnetic fields. The toolkit needs paral-lelization to speed up the computation and to distributethe memory allocation of the evolved variables amongthe available resources. The code uses an hybrid
MPI/OpenMP [67, 68] environment, so it requiresentire computational nodes which number depends onthe lattice size. Another important requirement of thisapplication is the need for checkpointing that is oftenused for writing large data-output (of the order of 10GBytes). The Theophys user community performeda set of production submissions (requiring 32 wholenodes, long execution time and checkpoints) to test thebehaviour of the portlet. The runtime monitoring ofstandard output and standard error proved to be usefulin different stages of the job life-cycle, while the datamanagement portlet performed the retrieving and themanagement of checkpoints among different sites.
7 Conclusions and Future Activities
This paper describes a Web portal targeted at scien-tific communities making use of distributed resources.Through the portal, indeed, users can access Grid andCloud resources, submit jobs or complex workflowswith dependency-related applications, move data toand from the Grid resources, and finally access spe-cific applications through custom interfaces. Further-more, the portal modular architecture and the use ofwidely adopted frameworks make the integration ofnew services easy by developing appropriate portlets.
This paper also shows the benefits offered by theIGI portal by describing some relevant use cases. Theimplemented use cases show that various advantagescan be obtained from the heterogeneous nature of theEGI Grid production infrastructure when porting sci-entific applications on distributed platforms. These
Fig. 8 The Theophysapplication portlet
174 M. Bencivenni et al.
include the possibility of partitioning large calcula-tions into smaller ones and selectively distributingsegments on different machines to gain efficiency aswell as feasibility. This makes the work performed inthe application porting phase (see Section 5.3) of moregeneral use by providing a reusable example for otherdomain-specific applications.
In the future work, we are planning to integrateadditional scientific applications into the portal, alsotargeted to attract new user communities, on differ-ent operating system flavours as well as hardwareplatforms. The current portal implementation mainlyfocuses on access to resources belonging to the Gridworld. However, we have planned to make a deeperintegration with Cloud resources: extending the Cloudhigh-level interface for the direct provisioning ofSaaS (Software as a Service) solutions; implement-ing a tighter integration with Cloud frameworks suchas OpenStack or OpenNebula; supporting data trans-fer across Grid and Cloud resources; enhancing thereporting for status errors linked to storage provision-ing; improving support for scalable and distributedGrid and Cloud storage systems; extending the Cloudinterface to support non-Linux systems.
Acknowledgements The research leading to the results pre-sented in this paper has been possible thanks to the Gridresources and services provided by the European Grid Infras-tructure (EGI) and the Italian Grid Infrastructure (IGI). Weexpress our appreciation to the late Dr. Valerio Venturi.
overview6. Ardizzone, V., Barbera, R., Calanducci, A., Fargetta, M.,
Ingra, E., Porro, I., La Rocca, G., Monforte, S., Ricceri,R., Rotondo, R., Scardaci, D., Schenone, A.: The DECIDEscience gateway. J. G. Comp. 10(4), 689–707 (2012)
7. What is a science gateway? http://sciencegateways.org/what-is-a-science-gateway
10. Barbera, G., Donvito, A., Falzone, J.J., Keijser, G., LaRocca, L., Milanesi, G.P., Maggi, R., Andronico, G.,Vicario, S.: A grid portal with robot certificates forbioinformatics phylogenetic analyses. Con. Comp: Prac.Expe. 23(3), 246–255 (2011)
11. EUGridPMA guideline on approved robots. http://www.euGridpma.org/guidelines/robot/
12. Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann, G.,Balasko, A., Karoczkai, K., Istvan, M.: WS-PGRADE/gUSE Generic DCI gateway framework for a large varietyof user communities. J. G. Comp. 9(4), 479–499 (2012)
13. IGI website. www.italianGrid.it/about14. Basney, J., Humphrey, M., Welch, V.: The MyProxy online
15. Tsaregorodtsev, A., Bargiotti, M., Brook, N., Ramo, A.C.,Castellani, G., Charpentier, P., Cioffi, C., Closier, J.,Graciani, R., Kuznetsov, G., Li, Y.Y., Nandakumar, R.,Paterson, S., Santinelli, R., Smith, A.C., Miguelez, M.S.,Gomez, S.: DIRAC a community grid solution. J. Phys.Conf. Ser. 119(062048), 2008 (2008)
16. gUSE website. guse.hu17. Thomas, M., Mock, S., Dahan, M., Mueller, K., Sutton, D.,
Boisseau, J.R.: The GridPort Toolkit: a system for build-ing Grid portals, pp. 216–227. IEEE Xplore Digital Library,San Francisco (2001)
18. Calzolari, F., Licari, D.: Proxy dynamic delegation in gridgateway. Proceedings of Science (PoS), (PoSISGC 2011 &OGF 31027) (2011)
19. Gannon, D., Alameda, J., Chipara, O., Christie, M., Dukle,V., Fang, L., Farrellee, M., Kandaswamy, G., Kodeboyina,D., Krishnan, S., Moad, C., Pierce, M., Plale, B., Rossi,A., Simmhan, Y., Sarangi, A., Slominski, A., Shirasuna, S.,Thomas, T.: Building grid portal applications from a webservice component architecture, vol. 93, pp. 551–563. EEEXplore Digital Library (2005)
20. Murri, R., Kunszt, P.Z., Maffioletti, S., Valery T.: Grid-CertLib a single sign-on solution for grid web applicationsand portals. J. G. Comp. 9(4), 441–453 (2011)
Accessing Grid and Cloud Services Through a Scientific Web Portal 175
37. TAGPMA, profile for member integrated X.509credential services with secured infrastructure. http://www.eugridpma.org/guidelines/MICS/IGTF-AP-MICS-1.2-clean.pdf
www.eugridpma.org/guidelines/pkp/46. Sfiligoi, I., Tiradani, A., Holzman, B., Bradley, D.C.: The
Glidein WMS approach to the ownership of system imagesin the cloud world. In: Leymann, F., Ivanov, I., Sinderen,M.v., Shan, T. (eds.): CLOSER, pp. 443–447 (2012). SciTePress
47. Cecchi, M., Fabio, C., Dorigo, A., Ghiselli, A., Giacomini,F., Maraschini, A., Marzolla, M., Monforte, S., Pacini, F.,Petronzio, L., Prelz, F.: The glite workload managementsystem. In: GPC of lecture notes in computer science,vol. 5529, pp. 256–268. Springer (2009)
48. Salomoni, D., Italiano, A., Ronchieri, E.: WNoDeS, a toolfor integrated grid and cloud access and computing farmvirtualization. In: Journal physics: Conference Series 3315(Computing Fabrics and Networking Technologies) (2011)
secure shell the definitive guide O’Reilly Media (2005)52. GateOne website. http://liftoffsoftware.com/Products/
GateOne
53. EGI Community Forum 2013 website. cf2013.egi.eu.54. WeNMR website. www.wenmr.eu55. Ronchieri, E., Verlato, M., Salomoni, D., Torre, G.,
Italiano, A., Ciaschini, V., Andreotti, D., Pra, S.D., Touw,G.V.W.G., Vuister, G.W.: Accessing Scientific Applicationsthrough the WNoDeS Cloud Virtualization Framework. In:PoSISGC 2013029 (2013)
56. EGI Federated Clouds Task Force website. https://wiki.egi.eu/wiki/Fedcloud-tf:FederatedCloudsTaskForce
57. Konstantinou, I., Floros, E., costs, N.K.: Public vs privatecloud usage the StratusLab case. In: The 2nd InternationalWorkshop on Cloud Computing Platforms CloudCP’12.ACM, Bern (2012)
59. Bencivenni, M., Brunetti, R., Caltroni, A., Ceccanti, A.,Cesini, D., Di Benedetto, M., Fattibene, E., Gaido, L.,Michelotto, D., Misurelli, G., Venturi, V., Veronesi, P.,Zappi, R.: A web-based utility for Grid data management.In: PoS(ISGC 2012004) (2013)
60. Magnoni, L., Zappi, R., Ghiselli, A.: StoRM: a FlexibleSolution for Storage Resource Manager in Grid. In: TheIEEE 2008 Nuclear Science Symposium (NSS-MIC 2008),pp. 19–25. IEEE Computer Society., Dresden (2008)
inspire63. FLUKA website. http://www.fluka.org/fluka.php (2014)64. Spes website. http://web.infn.it/spes65. Alfieri, R., Arezzini, S., Ciampa, A., De Pietri, R., Mazzoni,
E.: HPC on the Grid the Theophys experience. J. GridCompu. 11, 260–265 (2013)
66. Loffler, F., Faber, J., Bentivegna, E., Bode, T., Diener, P.,Haas, R., Hinder, I., Mundim, B., Ott, C., Schnetter, E.,Allen, G., Campanelli, M., Laguna, P.: The Einstein toolkit:a community computational infrastructure for relativisticastrophysics. Class. Quant. Grav. 29 (2012)
