Abstract The Grid Virtual Organization (VO)“Theophys”, associated to the INFN (IstitutoNazionale di Fisica Nucleare), is a theoreticalphysics community with various computationaldemands, spreading from serial, SMP, MPI andhybrid jobs. That has led, in the past 20 years, to-wards the use of the Grid infrastructure for serialjobs, while the execution of multi-threaded, MPIand hybrid jobs has been performed in severalsmall-medium size clusters installed in differentsites, with access through standard local submis-sion methods. This work analyzes the support forparallel jobs in the scientific Grid middlewares,then describes how the community unified themanagement of most of its computational need(serial and parallel ones) using the Grid throughthe development of a specific project which in-tegrates serial e parallel resources in a commonGrid based framework. A centralized nationalcluster is deployed inside this framework, pro-viding “Wholenodes” reservations, CPU affinity,and other new features supporting our High Per-
R. Alfieri (B) · R. De PietriINFN Parma and Parma University,Viale G.P. Usberti 7/A, Parma, Italye-mail: [email protected]
S. Arezzini · A. Ciampa · E. MazzoniINFN Pisa, Polo Fibonacci Largo B. Pontecorvo 3,Pisa, Italy
formance Computing (HPC) applications in theGrid environment. Examples of the cluster per-formance for relevant parallel applications in the-oretical physics are reported, focusing on thedifferent kinds of parallel jobs that can be servedby the new features introduced in the Grid.
Keywords HPC · Grid · Theoretical physics
1 Introduction
Since early 2000s, Grid [1] has been the emergingparadigm for the setting up of distributed com-putational infrastructures and services. This par-adigm has been mainly used for serving the needsof High Throughput Computing (HTC), but it wasscarcely used for parallel applications. The INFNhas been always at the forefront on the develop-ment and deployment of Grid enabled computingfarms and storage systems, but it was relaying onsmall local clusters or dedicated HPC facilities forall of its needs for parallel resources. Recently,the general availability of multi-core architectureshave pushed the demand for parallel and multi-threaded resources, not only in communities likethe Theophys one, involved in compute-intensivesimulations.
This trend has brought the Theophys commu-nity to consider the opportunity to undertake aproject towards a common framework, able to
R. Alfieri et al.
integrate all its resources in the EGEE distributedinfrastructure [2], which was widely used by thecommunity for the management of serial jobs. Theproject would have carried out important (and ob-vious) benefits in terms of resources management,exploitation and usability. The need to installnew resources for the whole Theophys communitybrought to the deployment of a national clusterdesigned to support heterogeneous applications(serial, parallel and multi-thread) and to be ac-cessed as a Grid service.
Various scientific communities have developedproposals or projects addressed to provide a uni-form interface for users, or to enable the sub-mission of applications requiring multi-scale re-sources. These objectives have been investigatedusing very different approaches.
A significant example of this trend is given bythe effort made by the Computational Chemistryinitiative [3], led by the University of Kentuckyand involving other Universities and supercom-puter centers in USA, with the aim to providea common infrastructure for the chemistry com-munity. The approach has been the developmentof a dedicated Grid infrastructure, CCG (Com-putational Chemistry Grid), consisting of basicGrid middleware components, such as the GlobusToolkit (see Section 3) and CGI scripts, providingcore functionalities to the client applications.
Other communities, like ours, are pursuing theidea of performing multi-threaded or HPC com-putation into standard Grid infrastructure, likethe example in the field of Cosmological simula-tions [4]. Although the work demonstrated thatthe EGEE infrastructure can be successfully usedto run numerical simulations, the authors claimthat the possibility to select WNs on the basis ofthe memory, number of cores and local disk spacewould simplify the porting and the managementof their parallel jobs.
A similar approach has been pursued in thefield of the Earth Science [5], where has been donean analysis of the requirements for the portingof MPI applications in the EGEE environment.Despite the improvements in the MPI support,thanks to the work of specific MPI WorkingGroups [6] promoted by EGEE, the infrastructurewas not considered a production tool suitable fortheir applications.
Last related work analyzed here is the Share-Grid Peer-to-Peer Desktop Grid [7] that is aGrid infrastructure aiming at the federation ofresources provided by a set of small research lab-oratories. The infrastructure is designed to havegood scalability and performance for differentkind of jobs, including serial, parallel and SMPones. What is worth of mention in this work is acomplementary technique adopted for the multi-core support, based on the use of multiple virtualmachines per physical machine. With this solution,the users can agree with the site administratorson a specific hardware and software environment,which can be deployed on clusters composed byheterogeneous hosts. Disadvantages are the over-head introduced in the resources usage and therequirement of relevant configuration efforts.
This paper is organized as follow. In Section 2we give a short description of the users com-munity, its computational needs and the projectobjectives. In Section 3 we analyze the actual HPCsupport in the most common Grids. In Section 4,that is the main part of the present work, we dis-cuss problems encountered and solutions adoptedin the design and deployment of the project. InSection 5 there is a description of two applicationsin theoretical physics executed in the Grid envi-ronment, that we used to test the new featuresintroduced by the project. The conclusions of ourwork are drawn in Section 6.
2 The INFN Theoretical Physics Community,Computational Needs and Project Motivations
The INFN theoretical physics community consistsof more than one thousand researchers distrib-uted over 28 sites and involved in more than60 research projects. In this community, besideseveral activity that require computation basedon serial jobs, there are research fields, suchas Lattice Quantum Chromo-Dynamics (QCD),Fluid Dynamics and Numerical Relativity, whichrelies on the execution of massive parallel applica-tions. These applications require HPC resourcesranging from large PC clusters up to specializedsupercomputers.
HPC on the Grid: The Theophys Experience
In the past these computational needs havebeen fulfilled through very different solutions.
The EGEE Grid infrastructure has been widelyused for serial calculation. During the year 2010Theophys VO has been the sixth major CPU con-sumer, with about 1 Million of executed jobs (seeFig. 1).
In theoretical physics there is the request ofnumerical simulations, such as Lattice QCD, re-quiring the computational power provided only bymassively parallel supercomputers. Since early 80sINFN has developed a own supercomputers fam-ily, named APE [9], dedicated to such problems.QCD simulation jobs have been executed also atCineca and in general on PRACE facilities (seeSection 3).
For smaller parallel jobs (“medium-sized”) sev-eral independent small clusters have been in-stalled and accessed using standard methodsand local policies. These resources are used fordifferent levels of parallelism, ranging from ap-plications which require the exclusive access toa single whole-node for multi-threaded applica-tions, or single process applications requiring theexclusive usage of the whole memory of the node,up to complex hybrid MPI-openMP applications(see Section 5) requiring hundreds of cores.
Often research groups have multi-scale appli-cations requiring access to different kind of re-sources (see Fig. 2). Consequently, users haveto interact with different resource providers andto manage different access credentials and meth-
Fig. 1 Theophys Grid usage. This chart, derived from theEGI Accounting Portal [8] shows the usage in 2010 of theGrid infrastructure by the Theophys VO in comparisonwith the other EGI/EGEE VOs, in terms of NormalizedCPU time
Fig. 2 Theophys resources. In 2010 there were 3 typesof resources: EGEE infrastructure for serial jobs, smallparallel clusters dedicated to medium-sized jobs and super-computers for LQCD simulations
ods. The use of independent resources leads toa waste of computational power and administra-tion efforts. The need of a single shared platformwas considered increasingly important and Gridappeared to be the natural solution since it hasbeen successfully used by the community for serialapplications.
For these reasons, in early 2010, we decided toevaluate the reliability of the sites, the flexibilityof the support and in general whether the middle-ware was suitable for our “medium-sized” applica-tions. The evaluation has been realized through aseries of probe jobs submitted to all sites support-ing MPI and the Theophys VO. These probe jobsexecute a shell script that collects environmentalinformation from WNs (LRMS type, flavor andversion of the MPI implementations) and esti-mates network and CPU performance through theexecution of basic bench-marking programs. Theresults, presented in Table 1, show that there wasa general lack of opportunity to use the Grid toperform even small scale parallel computation onthe already active resources.
An important step towards a better supportfor HPC has been the release in June 2010 ofa document proposed by the EGEE MPI Work-ing Group [6]. The document recommended amethod for deploying MPI support that shouldwork for both users and site administrators and itproposed new middleware features to support theupcoming multi-core architectures.
R. Alfieri et al.
Table 1 The table presents the situation of the MPI sup-port among sites supporting Theophys VO in early 2010,prior to the start of the project
MPI/LRMS Status Network CPU(μs-MBytes/s) (GFlops)
MPICH/PBS OK 114-54 0.7MPICH/PBS OK 109-56 0.7MPICH/PBS OK 114-54 0.7MPICH/LSF OK 98-108 1.2MPICH/LSF OK 94-101 1.3MPICH/PBS OK 1287-10 0.4MPICH/LSF mpich misconf. – 1.3MPICH/LSF mpich misconf. – 1.5MPICH/LSF mpich misconf. – 1.5MPICH/LSF mpich misconf. – 1.5MPICH/LSF ssh misconf. – 0.8MPICH/LSF ssh misconf. – 0.7MPICH/LSF ssh misconf. – 0.8MPICH/PBS aborted – –MPICH/PBS aborted – –
The numbers presented in the Network column specifyLatency and Bandwidth measured among 2 nodes of thecluster. Almost half of the resources were working forserial jobs, but not for parallel ones, due to software (sshor MPICH) misconfigurations
This recent evolution has convinced the INFNTheoretical Physics community to start a projectwith different objectives and different time-scales:
– to provide to the community a single largeresource able to satisfy all the small andmedium-sized computational needs describedin Section 2. This objective has clear advan-tages in terms of simplification of the user in-terface, administrator tasks, resource exploita-tion, and transparency on the user’s activitythrough a detailed web accounting service.Of course, this result can be achieved with atraditional cluster too.
– to integrate the medium sized activity in theexisting Grid infrastructure implemented forserial jobs, as mentioned earlier. Here theneed for a Grid-enabled HPC cluster has ev-ident benefits for both users and administra-tors point of view: architectural componentsand functionalities (such as AAA services),job life-cycle management, resources integra-
Fig. 3 The desired Theophys infrastructure
tion and management can be reused for thenew resources.
– to realize the first step toward a single, globalinfrastructure for the whole community, basedon Grid services (see Fig. 3 for a view ofthe envisioned resources architecture). Theinfrastructure should be able to integrate oldand new parallel clusters, so the design of thefirst parallel cluster, we are describing here,must be scalable and reliable. Moreover, aswe will see later on (see Section 3), PRACE,that is actually an important supercomputingprovider for the Theoretical physics commu-nity, could be able in a near future to extend itsservices to Grid users. This integration can beendorsed by the scientific communities, espe-cially if every other computational activity ofthe communities is already performed insidethe infrastructure.
3 HPC Support in the Grids
A widely used middleware is the Globus Toolkit(GT) [10]. It includes software modules for secu-rity (GSI), data services (GridFTP), Informationservices (MDS), Execution Management Services(GRAM) and communication, which are oftentaken by other projects to build new middlewares.
For non-trivial jobs, the user has to write aGlobus job script, which is written using the Re-source Specification Language (RSL). In case ofparallel jobs the RSL parameter needed to specifythe CPUs number is “count”. The following is an
HPC on the Grid: The Theophys Experience
Fig. 4 Comparisonbetween gLite andUNICORE architecturesfrom the job generationprocess point of view
example of a RSL file (GT4 syntax) requiring 8CPUs:
Some middlewares, such as gLite [11] andARC [12] (Advanced Resource Connector), pro-vide users with high level services for schedulingand running computational jobs, accessing andmoving data, and obtaining information on theGrid infrastructure. They still use selected GlobusToolkit libraries, chiefly in the security context(GSI). Other middlewares have been developedto support specialized services such as dCache[13] which is designed for data management, andUNICORE (Uniform Interface to Computing Re-sources) [14] which is specialized in supercomput-ing applications.
In 2010 gLite, ARC, UNICORE and dCachejoined forces to develop and unify their middle-wares in a common project called EMI (EuropeanMiddleware Initiative) [15].
Considering the purpose of this work, we ana-lyze here a comparison between the parallel sup-port in gLite, which was the middleware in useby Theophys for serial jobs at the moment of theproject start up (see Section 2), and in UNICORE,which was specifically created for HPC applica-tions [16] (Fig. 4).
UNICORE supports resource specificationsthrough a Job description file, written in JSDL
(OGF standard) [17] with the addition of exten-sions for parallel applications and the supportfor the Execution Environments. The ExecutionEnvironments, specified by the administrator bymeans of the IDB file, provides to the users a highlevel of abstraction, hiding resource details andproviding simple execution environments to theuser [18].
The user can specify resources and environ-ment requirements through a command line scriptor a Java web portal (Rich client). The mainresource requirements supported by UNICOREare the total number of CPUs, number of nodes,CPUs per node and RAM per node. Through theExecution Environment it is possible to specifythe software environment and other options, suchas the MPI flavor and the number of requestedprocesses (see Fig. 5 for an example).
The parallel support in gLite is derived fromthe recommendations of specific MPI Work-ing Groups promoted by EGEE. It is achievedthrough the Job Description Language (JDL) at-tribute CPUNumber, which specifies the numberof Job Slots that will be requested to the ResourceManager (see Fig. 6 for an example).
Fig. 5 UNICORE Command-line client example
R. Alfieri et al.
Fig. 6 MPI job in gLite: JDL script example
MPI-Start [19] is an abstraction layer, locatedbetween the middleware and the underlying LRMSand MPI flavors, that offers a unique interface tostart parallel jobs with different implementationsof the execution environment. The architecture isorganized in three main frameworks: The sched-uler framework provides support for different Lo-cal Resource Management Systems (LRMS). Itdetects the availability of a given scheduler andgenerates a list of machines that will be used toexecute the application. The Execution frame-work prepares the environment needed to startthe particular MPI implementation. The user hasonly to specify which MPI flavor will be used toexecute the application. The Hooks framework isresponsible for the files distribution in case of filesystem not shared between WNs and it managescustomizable pre-run and post-run hooks.
The example reported in Fig. 6 shows theJDL file of a job demanding 8 processors andthe OpenMPI flavor. The Requirements attributemakes the brokering service to select sites publish-ing that they support MPI-Start and OpenMPI.
A step towards a better support for HPC ingLite has been the release in June 2010 of a docu-ment proposed by the second EGEE MPI Work-ing Group [6] which proposed the introduction ofspecific JDL Tags. The new features are designedto allow users to specify how cores should bedistributed over the cluster, whether full nodesshould be used or not and how many nodes shouldbe involved in the allocation itself.
In the comparison between the MPI supportedmethods in UNICORE and gLite we can distin-guish two separate stages: the resource selectionand the job environment setup.
The resource selection in gLite has a higherabstraction level thanks to the Resource Broker,which permits the integration of large sets of se-rial and parallel resources. In UNICORE usershave a higher level of flexibility, since they canspecify a more detailed hardware environment,such as CPUs per node and RAM per node, thatare needed to support multi-thread and hybridapplications. This feature was specified in gLiteas a recommendation document, but it was notimplemented.
The job environment setting is an advancedfeature that allows users to configure the way jobsare executed in a more detailed and user-friendlyfashion. A common scenario is the configurationof an environment for parallel jobs, such as MPI.UNICORE explicitly supports this feature in theUNICORE/X Execution Environment. The samefeature is provided in gLite by MPI-Start.
Since the major mandate of EMI is the unifi-cation of the different middlewares, works are on-going towards a common solution for the execu-tion task. EMI-ES (EMI Execution Service) [20]is a specification for the common Execution Ser-vice, developed by the EMI project, that intendsto define a joint web service interface to ARC(ARC-CE), gLite (CREAM) and UNICOREcompute services, by incorporating good ideasfrom all three. The HPC capabilities supportedby EMI-ES are NumberOfSlots, SlotsPerHosts,ExclusiveExecution (whether a host should beallocated for exclusive use by the job), ProcessPer-Host and ThreadsPerProcesses. The deploymentof EMI-ES (foreseen in EMI-2) will make possi-ble the introduction of common clients and theinteroperability among different Computing Ele-ments.
3.1 Grid Infrastructures
Grid technologies have been introduced in Europesince early 2000s pushed by big scientific experi-ments and projects, starting from LHC at CERN[21]. In these years the European Union (EU)has funded several projects aiming at the setting-up of a continental infrastructure able to providecomputational resources to the whole scientificcommunity.
HPC on the Grid: The Theophys Experience
The European infrastructure in use by the The-oretical Physics community at the time of theproject start was EGEE, based on the gLite mid-dleware.
EGEE ended in 2010 and was substituted bythe EGI project (European Grid Infrastructure)[22]; EGI is now based on the EMI middlewareand is composed of a central coordinating body(called the EGI.eu) and more than 40 NationalGrid Initiatives (NGIs), representing the resourceinfrastructure providers. The users are organizedin Virtual Organizations and Theophys is the Vir-tual Organization representing the Italian The-oretical Physics community, supported by IGI,which is the Italian NGI.
Grid technologies are now in use also in themost important supercomputing consortia in USand Europe:
XSEDE [23] is a US project supporting 16 su-percomputers and data analysis resources acrossthe country, replacing and expanding the formerTeraGrid initiative [23]. The Grid interconnectionis provided by OSG (Open Science Grid) [24]which is a Grid infrastructure based on GlobusToolkit with additional modules for security, stor-age and data management, work-flow and otherhigher level services.
The analogue European initiative is PRACE(Partnership for Advanced Computing in Eu-rope) [25] which is a EU funded project estab-lished to create a permanent pan-European HighPerformance Computing service for research.The project incorporate the previous initiative,DEISA (Distributed European Infrastructure forSupercomputing Applications) [26], which wascompleted in April 2011. PRACE is composedby more than 20 partners, whose HPC facilitiesare organized in a hierarchical model. The higherlevel, Tier-0, includes 6 supercomputers locatedin Germany, France, Spain and Italy. The Tier-1 machines are national supercomputers madeavailable from Finland, France, Germany, Ire-land, Italy, Poland, Sweden, The Netherlands,Serbia, Switzerland, Turkey, and United King-dom. PRACE has deployed a common servicesinfrastructure, Grid based, to integrate and op-erate the Tier-0 and Tier-1 systems as a singledistributed Research Infrastructure and to pro-vide a transparent access to users. From the Grid
perspective, an important feature is the support ofUNICORE as a core service for both Tier-0 andTier-1 facilities [25].
Recent scientific projects are growing in Eu-rope relying on resources provided by both super-computer centers, like PRACE, and HTC Gridinfrastructures, like EGI. These projects share thecommon interest in multi-disciplinary multi-scalemodels and they require large scale or even ex-treme scale computing capabilities.
Among others, we mention GEMS [27] andMAPPER [28]. GEMS is an application softwaredeveloped inside CompChem VO, that distributesa large quantity of independent tasks on a HTCplatform (EGI). Their results are passed as a setof multiple inputs for a a final elaboration im-plemented on a HPC platform hosted at Cineca(the Italian PRACE member). The computationalsolution pursued in this project is the definition ofa work-flow engine that combines together HTCand HPC elements. It will be based on EMI-ES,as soon as it will be available.
MAPPER is an EU-funded project aimingat the integration of heterogeneous infrastruc-tures for programming and execution of multi-scale simulations from different user communi-ties: physiology, computational biology, fusion,hydrology and nano-material science. The newservices and interoperability tools will be intro-duced by joint task-force between EGI, MAPPERand PRACE.
The scientific interest towards a convergencebetween HTC and HPC has been one of the mainreason behind the development of our project andit follows the line of integrating, into the EMImiddleware, the Grid components in use in Super-computing Centers.
4 The Project
Although, even if the middlewares comparison (seeSection 3) has highlighted that UNICORE was moresuitable for HPC support, we decided to adopt thegLite middleware for the following reasons:
– GLite is a stable and robust middleware forHTC and is actually the middleware in usein the Italian NGI (IGI). This means that
R. Alfieri et al.
the Theophys community of users and admin-istrators has a deep experience with it and,moreover, a consolidated infrastructure basedon gLite is in place, providing services that canbe used for HPC resources too.
– Although the support and the usage of paral-lel computing in gLite was quite inadequate,the Recommendation document elaboratedby the EGEE MPI-WG has accelerated theprocess towards an improvement of the sup-port. As a consequence, an additional objec-tive of our work has been the contributiontowards a real start up of the usage of parallelcomputing in gLite, through a close collabora-tion with the middleware developers.
The key points we faced in the design of a re-source able to support not only serial and MPI butalso SMP and hybrid applications, as describedearlier, have been the following:
– how to implement “WholeNodes support”,which was one of the main targets of theproject, since, when we decided to start, thesupport wasn’t implemented yet. This pointcover two distinct aspects: how to get thewhole nodes allocated and how to distributeMPI ranks and/or threads on the reserved JobSlots.
– how to deal with serial and parallel jobs or,in other words, how to keep serial jobs out ofparallel resources. Indeed serial jobs withoutany requirement may land on any resources,including parallel ones.
– An issue faced in this project, concerning anyHPC resources (Grid and traditional ones), isthe optimization in the cores utilization, bymaximizing cores exploitation and minimizingparallel jobs starvation.
– often jobs in theoretical physics are HTC (run-ning for several days) and data-intensive (seefor example the Einstein Toolkit in Section5.2), so a special point in this work is thedesign of a storage architecture optimized forsuch job types.
4.1 Cluster Characteristics
The Cluster had to be integrated into the exist-ing Grid infrastructure and the deployment pro-
cedure active within our organization (INFN).They are standardized on the basis of the Sci-entific Linux (SL) distribution and the Yaimconfiguration tool. These requirements ruled outthe possibility of consider any other cluster man-agement solutions. Moreover, the choice of theLocal Resource Management System (LRMS), inour case LSF, was a consequence of the exist-ing infrastructure present at the PISA computingcenter and we did not examined other possiblesystems, in terms of performances or capabilities.
The main problem is that there are differentsoftware requisites that need to be harmonized.The requisites given by the local infrastructuremanagement, the middleware and the User Com-munities are in general different and, in someaspect, conflicting with each other. We made thestrategic choice of decoupling the handling of therequisites in the following way. On the WNs,we installed an Enterprise class operating system,that guarantees the hardware and software sup-port. Inside that system, the middleware and userapplications are run using the “chroot” mecha-nism.
In our case the version of the running operat-ing system is the “Suse Linux Enterprise Server(SLES)”, version 10sp2: this system guaranteedthe support of the hardware characteristics, for in-stance the used HCA IB and the GPFS software.The middleware, instead, is installed on a specific“chroot” inside which runs a specific version ofScientific Linux along with all the libraries nec-essary for the correct middleware and user appli-cations work. Hence, from the user point of view,the environment is Scientific Linux 5.5, while fromthe operative core system point of view it is SLES10sp2. As an example of the benefits allowed bythis kind of installation we can report a couple ofcases:
– A bug in the AMD processors of the Cluster(AMD erratum 298) caused the effect to halfthe performances of the WN. The solution hasbeen identified in the creation of an ad-hockernel, operation that has been possible andrelatively simple thanks to the decoupling.
– Another case is the updating of the middle-ware and user libraries. With the use of thisdecoupling mechanism we can rollback and
HPC on the Grid: The Theophys Experience
extend to all the WN of the Cluster simply un-packing a tar.gz. As far as the performances isconcerned, the “chroot” mechanism does notintroduce any overhead, while this happens inthe case of real virtualization solutions. Weproved it running SPEC benchmark inside andoutside the “chroot” environment obtainingvery similar results.
The fabric of the Cluster has been realized us-ing hardware based on Cisco components (switchSFS 7012, HCA SFS HCA320 A1 with Mellanoxtechnology MT25204 chipset) managed by a soft-ware layer based on Open Fabric version OFED-1.5.1 that has a better and more continuous devel-opment.
4.2 Storage
An important objective of this project is to easethe data management [29]. The choice of the Gridstorage services made available to the Clusterhas been inherited from the existing Data Cen-ter infrastructure, which is based on the StoRM[30] Storage Element. The back-end of StoRM iscomposed by a couple of Enterprise Class storagesystem Data Direct Network S2A9900, for about1PB of RAW disk space, with GPFS file system.
The file system is accessible from the clusterWNs via POSIX and from the Storage Elementusing the Storage Resource Manager (SRM) pro-tocol (see Fig. 7).
The cluster leverages on the IB connection toincrease the file system performance using the IPover IB protocol.
Parallel jobs have typically a long execution time(several days), while elaborating large amount ofdata-sets (>1 GB). In the standard Grid architec-ture such jobs need to spend considerable amountof time uploading the input data-set from theuser’s storage area to the WNs and, at the end ofthe calculation, from the WNs back to the user’sstorage area. The data transfer involves typically asingle process, while other allocated cores remainunused. Moreover, parallel jobs often require aCPU time larger than the maximum time allowedby the queue, so the user, in principle, has tomove checkpoint data-sets forward and backwardseveral times. To ease up this problem we haveexploited the possibility that, on parallel clusters,the same physical storage can be shared betweenWNs and Storage Elements. On the Cluster a stor-age space (“chkPoint”) is shared for reading andwriting between WNs and the Storage Element.
The job work-flow on the Cluster is the follow-ing:
– Input data-sets are uploaded from the user’sSE to the resource SE via GridFTP or, incase of small amount of data, they are shippedalong with the InputSandbox.
– The running job writes outputs (standard out-put, standard error and output data) to the“ChkPoint” Area.
– The user can monitor the status of the runningjob by perusing the output files via SRM re-quests.
– At the job completion the user can retrieve theoutput files via SRM, or continue the calcu-lation on the same data-set through a subse-quent submission performed on the cluster.
Fig. 7 Cluster storagearchitecture
R. Alfieri et al.
4.3 Queues Organization
The Cluster is integrated in a wider Grid site thatmust provide resources also for serial jobs. Theresult is the fact that the biggest part of servedjobs is of the serial type, for which there is nogranularity issue: 1 job = 1 CPU (in reality 1Core) and it is not important where the specificcore is allocated. For parallel jobs this is not thecase. For this reason in the CE, at the middlewarelevel, a set of parameters have been introducedto allow users to specify their requests in terms ofdistribution of the used cores. These modificationsto the middleware were included in the experi-mental version of the middleware we used for theCluster. Now these modifications are a regularpart of the normal middleware distribution (seenext section).
This issue has two faces and can be viewedfrom two different point of view. From the userperspective is useful/necessary to know the gran-ularity of the available systems and to be ableto specify requests on the allocation of the JobSlots: we know that some types of applications canbenefit from having “near” Job Slots (to use nearcore) and in other cases this is less important. Sothe user is happy for the opportunity to submitspecific requests to the middleware.
From the site administrator point of view thiscan lead to inefficiencies: for instance it can hap-pen that cores remain free due to the reserva-tion by a job waiting the availability of otherrequested cores, impeding to maintain the clusterfully loaded. The experience matured in the lastyears with our cluster showed evidence of theseissues:
– The coexistence of serial and parallel jobs onshared resources is not an easy issue for thedimension of our cluster. The best way to fillthe holes of free cores would be the “shortqueue” but generally it doesn’t work properlybecause it requests the collaboration by theusers that, in our experience, not need thiskind of resources.
– The second best choice to fill the unallocatedcores is to allow the execution of serial jobs
submitted by communities different from themain one. But in this case the jobs, not hav-ing specific requests for their collocation, caneasily saturate the available resources makingdifficult the start of parallel jobs.
– The best solution would be the possibility tomanage the “packing” of the serial jobs onthe same Worker Node. This option is notpossible in LSF (at least is not available inthe version we are using), so that the onlyway to obtain this result is the migration ofrunning jobs. We are exploring this solutionbut there are problems in the interaction withthe middleware.
– Mechanisms like the Job Slot reservation andthe back fill are used. They allow to mitigatethe problems and to increase the exploitationof the available resources but they have theside effects of leading to long wait times forthe execution of parallel jobs and/or to re-source starvation.
4.4 Granularity and Multicore Support
The cluster model supported by the gLite mid-dleware didn’t consider the different levels ofcommunication types among cores in the modernmulti-core systems. Figure 8 displays the archi-tecture of a modern cluster with multicore bi-processor nodes. In this example the inter-processcommunication performance (latency and band-width) depends on the position of the processeswith respect to the cores. We measured the com-munication round-trip time (see Fig. 8, bottomleft) and bandwidth (see Fig. 8, bottom right) us-ing the NetPipe [31] utility. The bandwidth perfor-mance makes evidence of the Level 2 cache effectat a packet size of about 8 KBytes and the Level 3cache effect for packet sizes up to 2 MBytes.
Grid middlewares should support multi-threaded and hybrid OpenMP/MPI parallelism.They should also support the CPU and memoryaffinity control for the exploitation of cacheeffects and the regulation of the DDR access inorder to limit the bottleneck introduced by theinter-processor links (QPI or HT).
HPC on the Grid: The Theophys Experience
Fig. 8 Communicationtypes patterns inmulticore systems (top).Different communicationpatterns correspond todifferent performances.The actual performances,as measured using theNetpipe tool for thedifferent communicationpatterns, are shown in thebottom-left box (roundtrip time) andbottom-right box(Bandwidth). Continuousred line for intra-socketcommunications (1),black-dashed line forintra-nodecommunications (2) andblue-dotted line forextra-node infinibandcommunications (3)
3.63 µs
0.77 µs
0.64 µs
64k 2M
1 µs
10 µs
100 µs
1 ms
Packet Size (Bytes) Packet Size (Bytes)
Rou
ndtr
ip ti
me
NetPipe
8k64k 2M0
2
4
6
8
10
12
14
Ban
dwid
th (
Gbp
s)
NetPipe
The MPI-WG recommendation document, pre-viously mentioned, introduces three new JDLattributes for the Granularity support (Fig. 9).These attributes allow users to specify:
– SMPGranularity: This value determines theminimum number of cores that should be al-located on any host.
– WholeNodes: Whether whole nodes (all thecore present on a single node) should be ex-clusively allocated for the job.
– HostNumber: How many nodes should be used.
Fig. 9 Grid cluster model with granularity support
The use of these new attributes will allowusers to better specify the execution environmentneeded by the application. In Fig. 10 are shownsome specific usage examples.
These attributes were not implemented yet inEGEE so we started a collaboration with thegLite middleware developers aiming at the de-velopment and testing of a provisional patch forCREAM CE [32], including a preliminary supportfor the new attributes.
Fig. 10 Examples of use of the new Granularity attributes.Top example: request for a multi-threaded job requiring awhole node with a minimum of 8 cores. Middle example:request of 4 exclusive nodes with at least 8 cores per node.Bottom example: request for an MPI job spread over asmany nodes as possible
R. Alfieri et al.
This patch is based on the ability of theCREAM CE to forward specific requirements(CeRequirements) [32], written by the user inthe JDL, directly to the CE and there processedto properly instruct the local batch system.CeRequirements made possible the introductionof the attributes (with a different syntax, but withan equivalent semantics) on the production en-vironment in a transparent way, preserving thestandard functionality of the service. In Fig. 11 arereported the examples shown in Fig. 10 using theprovisional syntax.
Unlike the JDL Requirements, CeRequire-ments don’t involve the match-making process,so the attributes can be used only by means ofdirect submissions to the resource. This impliesthat the end-point must be explicitly specified inthe Requirements. Of course this strategy doesn’tscale but, in this first phase of the project whereit exists only a single HPC resource, it is a reason-able choice.
4.5 MPI-Start
In many cases parallel jobs require to allocate JobSlots on the minor possible number of nodes tominimize the communication overhead. The ideais to bind each MPI process (and its threads) toa specific subset of processing resources (cores,sockets, etc.) in order to inhibit excessive processmovement away from its data (L3 cache, physicalmemory bank, ...) and to improve inter-processcommunications efficiency.
Fig. 11 Multicore support in gLite with the provisionalgranularity patch. Here are reported the same examples ofFig. 10 using the provisional syntax
At the start of the project, there was no stan-dardized way of performing these optimizationsand users needed to implement process bindingon the starting shell script, in a way dependent onthe target execution cluster. This approach is notsuited to the whole idea of the Grid and neededto be overcome. For this reason, in collaborationwith the MPI-Start developers, an experimentalversion of the tool has been released and installedon the Theophys cluster, providing the support foropenMP programming and CPU affinity. By usingthis experimental release of MPI-Start, it is pos-sible to start MPI processes at CPU socket level,preserving an affinity lock between processes andmemory.
The importance of this tuning can be seen onthe testing results shown in Table 2: performancedecreases when increasing the number of threadsper node (‘-psocket’ and ‘-pnode’) from 4 to 8,due to the fact that, in the latter case, memoryaffinity between processes and data is not ensured.The last two lines show that, if MPI processesare not forced to completely fill the WNs (“wholenodes” is not set), the computation results havelarge variation (we reported the minimum andmaximum measured time) and they are alwaysgreater than in the other case (first line markedwith ‘-pcore’).
The MPI-Start tool allows a flexible manage-ment of the execution environment for mixedMPI/openMP jobs, through the following com-mand line options: (i) -pcore, one MPI processper CPU core; (ii) -psocket, one MPI processper CPU socket; (iii) -pnode, one MPI processper node. In Fig. 12 is reported the example of ahybrid MPI/openMP job that launches one MPIprocess per CPU socket and one openMP threadper core.
4.6 Authentication and Authorization
The gLite security architecture is based on wellestablished work in the Grid community. On theauthentication side a credential storage ensuresproper security of credentials while proxy X.509certificates enable single sign-on. The authoriza-tion functionality is obtained with the help ofthe VOMS [33] service, which provides support
HPC on the Grid: The Theophys Experience
Fig. 12 This is anexample of how to luncha hybrid OpenMP/MPIprogram using MPI-Startwith four MPI instances(each with 4 threads) ontwo computing nodes
mpi-start -psocket -- my-mpi-prog
for Roles and Groups membership. In a Gridinfrastructure such as EGEE/EGI, where serialand parallel jobs coexist in the same environment,we need a way to avoid standard serial jobs beingdispatched to parallel queues by the WMS matchmaking process.
For this purpose we introduced the “Role=Parallel” which has to be assigned to HPC users.According to this method, the command tostart a parallel session on the Theophys clusteris obtained by adding the option “-voms theo-phys:/theophys/Role=parallel”, to the “voms-proxy-init” command.
To submit a serial job, the parallel role mustnot be specified, i.e., one should give the option“-voms theophys:/theophys”.
This method is not user friendly since usershave to generate different kind of VOMS prox-ies depending on the kind of jobs they need toexecute, but the use of VOMS extensions (Roleor Group) was the simplest way to implementthis feature in the current Grid architecture. Theproblem of keeping generic jobs out of a givenresource is common to other special resources,such as nodes equipped with GP-GPU cards orlarge amount of memory. This issue should andwill be investigated by middleware designers inorder to elaborate a common solution.
5 Parallel Applications
We report here the functionality test and the over-all performance of the system, using two freelyavailable scientific codes used by some of thegroups that use the system.
The two production codes are the “Chroma”[34] code, which performs Monte Carlo simulationof lattice QCD, and the Einstein Toolkit [35–37], which performs the time evolution of mat-ter coupled to the Einstein’s equations (GeneralRelativistic Hydrodynamics). The first one, the“Chroma” code, needs parallelization for reduc-ing computation time on a small data-set. Thesecond one, the “Einstein Toolkit”, needs paral-lelization in order to distribute, among nodes, thememory allocation of evolved variables. While thefirst one is a pure-MPI application, the second onehas a mixed MPI/OpenMP parallelization.
5.1 The Chroma Library
The first test is based on the freely availableUSQCD collaboration “Chroma” library. It refersto Hybrid Monte Carlo simulations of a PureSU(3) Lattice Gauge Theory, using two differentlattice Grid sizes (32 × 32 × 32 × 4 and 16 × 16 ×16 × 16) and performing 2000 sweeps. The totalallocated memory is very small: 36 MBytes, for aGrid size of 32 × 32 × 32 × 4, or 16 MBytes, whenthe Grid size is 16 × 16 × 16 × 16.
The parallelization is executed by distributingthe data Grid on N-processes that communicateto each other using the MPI library. The timinghas been measured by executing the simulation on1, 2, 4, 8, 16, 32 nodes, i.e. on 8, 16, 32, 64, 128, 256MPI processes, respectively. All the runs are ex-ecuted with MPI processes bound to their core,by imposing memory affinity. The results are re-ported in Table 2, and in Fig. 13 is reported a plotof the efficiency against the number of nodes.
R. Alfieri et al.
Table 2 Total execution time for two data Grid size
The reported efficiency is computed by multiplying theexecution time by the number of execution nodes, andscaled with respect to the execution on a single node.#CPUs is the number of MPI processes
5.2 The Einstein Toolkit
The test is performed by simulating the evolutionof a stable general relativistic TOV-Star modelusing the Einstein Toolkit consortium code. Weevolve the system on a cubic multi-Grid meshwith five levels of refinement (each of local size40 × 40 × 40) and we perform 800 time evolutionsteps. The total allocated memory for a scalarevolution is of the order of 1 GBytes. Since thisapplication supports hybrid parallelization, it hasgiven us the possibility to test the behavior ofour system under different distributions of thecomputation between MPI processes and openMP
Grid 16x16x16x16Data 1 MByte x Socket
Grid 32x32x32x4Data 1 MByte x Socket
0 5 10 15 20 25 300.8
0.9
1.0
1.1
1.2
1.3
1.4
1.5
nodes
Spe
edup
node
s
CHROMA Lattice QCD
Fig. 13 Plot of the efficiency of the parallel execution as afunction of the # s of nodes for the Hybrid Mote Carlo sim-ulation of the pure gauge SU(3) Gauge theory performedusing the Chroma library. The required simulation timesare reported in Table 2. Please note that the maximumefficiency is reached when the data size assigned to a groupof 4 MPI processes, which are executed on the same CPUsocket, is approximately equal to the size of the L3 cache
Table 3 Total execution time (in minutes) for a test evo-lution of a Relativistic Star on the same Grid, as a functionof the number of computing cores used
Times in the same column correspond to the same numberof CPUs allocated for each job. “-pcore”, “-psocket”, “-pnode”, correspond to the allocation of 1 MPI process(with 1 thread) for each core, 1 MPI process (with 4threads) for each CPU socket and 1 MPI process (with 8threads) per node, respectively. The last two lines refer tothe minimum and maximum execution time obtained whenthe whole-node allocation is relaxed
threads. The results are reported in Table 3. InFig. 14 is reported the total simulation time.
6 Conclusions and Future Works
Even though the project is based on provisionalcomponents, the Cluster is fully operative andit is producing scientific results. The project
Time = 31.5759Np
0.079 Np
0.0 0.2 0.4 0.6 0.8 1.0
40
60
80
100
120
1/( nodes)
Tim
e (m
in)
Einstein Toolkit
Fig. 14 Total execution time (in minutes) for the evolutionof a Relativistic star for pure MPI parallelization as afunction of the number of nodes on which the executiontake place. Each MPI process is constrained on a fixedCPU core and all the cores on the same node are allocatedfor the execution. The time values are reported in Table 3in the line labeled by “-pcore”. The dotted blue line showsthe fitted scaling law
HPC on the Grid: The Theophys Experience
achieved two important results for the Theophyscommunity:
– the deployment of a centralized resource forparallel applications that shares the same plat-form with the other Grid resources availablefor serial jobs
– the design of a global infrastructure that willbe able to fulfill all the computational needsof the community.
Besides these two primary objectives that wereachieved for the Theophys VO, this work has pro-duced other contributions to the whole computa-tional community. A step towards a better supportin gLite/EMI for jobs needing to exploit multi-socket and multi-core architectures. To show that,even if users generally consider (see Section 1 and[6]) the Grid not ready for parallel computations,the adoption of the new features described hereenables a flexible and easy porting of HPC appli-cations on the Grid.
After the experimental activity described here,the Granularity attributes have been included inthe EMI middleware, which is now under de-ployment inside the EGI infrastructure. In thesecond phase of the project we intend to deploy aHTC/HPC infrastructure based on the new EMImiddleware by involving a set of selected sitesand users from different scientific communities.Beside the tests to verify resilience and scalabilityof the platform, which will be performed in closecollaboration between users and sites administra-tors, a special attention will be dedicated to theusability, through the introduction of a Web portalwith HPC support.
Acknowledgements We would like to thank: M.Sgaravatto, S. Monforte and A. Gianelle (INFN, Italy)for their efforts in the development of the CE CREAMpatch supporting multicore architectures; E. Fernandez(IFCA, España) for the development of the support ofCPU affinity and openMP in MPI-Start; and A. Feo for thetesting activity with the Lattice QCD program “Chroma”.
2. Ferrari, T., Luciano Gaido, L.: Resources and servicesof the EGEE production infrastructure. J. Grid Com-puting 9(2), 119–133 (2011)
3. Dooley, R., Milfeld, K., Guiang, C., PamidighantamS., Allen, G.: From proposal to production: lessonslearned developing the computational chemistry Gridcyberinfrastructure. J. Grid Computing 4(2), 195–208(2006)
4. Becciani, U., Antonuccio-Delogu, V., Costa, A., Petta,C.: Cosmological simulations and data exploration: atestcase on the usage of Grid infrastructure. J. GridComputing 10(2), 265–277 (2012)
5. Vilotte, J.P., Moguilny, G.: Earth science: require-ments and experiences with use of MPI in EGEE.In: EGEE09 Conference, Barcelona, 21–25 September2009
6. Engelberts, J.: Towards a robust and userfriendly MPIfunctionality on the EGEE Grid. In: EGEE UserForum 2010, Uppsala, 12–15 April 2010. See also:http://www.Grid.ie/mpi/wiki/WorkingGroup
7. Anglano, C., Canonico, M., Guazzone, M.: The Share-Grid peer-to-peer desktop Grid: infrastructure, appli-cations and performance evaluation. J. Grid Comput-ing 8(4), 543–570 (2010)
Calvayrac, F., Della Morte, M., Del Re, A., De Pietri,et al.: APE computers—past, present and future. Com-put. Phys. Commun. 147(1–2), 402–409 (2002)
10. Foster, I.: Globus toolkit version 4: software forservice-oriented systems. In: IFIP International Con-ference on Network and Parallel Computing, LNCS3779, pp. 2–13. Springer, Berlin (2005)
11. Laure, E., Fisher, S.M., Frohner, A., Grandi, C.,Kunszt, P., Krenek, A., Mulmo, O., Pacini, F., Prelz,F., White, J., Barroso, M., Buncic, P., Hemmer, F., DiMeglio, A., Edlund, A.: Programming the Grid withgLite. Comput. Methods Sci. Technol. 12(1), 33–45(2006)
12. Ellert, M., et al.: Advanced resource connector mid-dleware for lightweight computational Grids. FutureGener. Comput. Syst. 23, 219–240 (2007)
13. Ernst, M., Fuhrmann, P., Mkrtchyan, T., Bakken,J., Fisk, I., Perelmutov, T., Petravick, D.: Manageddata storage and data access services for data Grids.In: Computing in High Energy Physics and Nu-clear Physics 2004 (CHEP04), Interlaken, Switzerland,p. 665, 27 Sept–1 Oct 2004
14. Streit, A., Bala, P., Beck-Ratzka, A., Benedyczak, K.,Bergmann, S., Breu, R., Daivandy, J. M. , Demuth, B.,Eifer, A. Giesler, A., Hagemeier, B., Holl, S., Huber,V., Lamla, N., Mallmann, D., Memon, A.S., Memon,M.S., Rambadt, M., Riedel, M., Romberg, M., Schuller,B., Schlauch, T., Schreiber, A., Soddemann, T., Ziegler,W.: UNICORE 6—recent and future advancements.Ann. Télécommun. 65(11–12), 757–762 (2010)
15. Fuhrmann, P.: EMI, the introduction. In: Proceeding ofCHEP 2010, Taipei, 18 October 2010
16. Fernández, E.: A unified user experience for MPI jobsin EMI. In: EGI User Forum 2011, Vilnius, 11 April2011
17. Anjomshoaa, A., Drescher, M., Fellows, D., Ly, A.,McGough, S., Pulsipher, D., Savva, A.: Job Submis-sion Description Language (JSDL) specification, ver-sion 1.0. White Paper, 7 November 2005
18. Schuller, B.: MPI in UNICORE. In: EGI TechnicalForum 2010, Amsterdam, 15 September 2010
19. Dichev, K., Stork, S., Keller R., Fernández, E.: MPIsupport on the Grid. Comput. Inform. 27(2), 213–222(2008)
20. Schuller, B., Konya, B., Konstantinov, A., Sgaravatto,M., Zangrando, L.: EMI-ES, a common interface toARC, gLite and UNICORE computing elements. In:EGI User Forum, Vilnius, 11 April 2011
21. Carminati, F., Templon, J., et al.: Common use casesfor a HEP common application layer. White Paper,LHC-SC2-20-2002
22. European Grid Infrastructure: An integrated sustain-able Pan-European infrastrucute for researchers inEurope (EGI-InSPIRE). White Paper, 18 April 2011.Document Link: https://documents.egi.eu/document/201
23. XSEDE Production Baseline: Service provider soft-ware and services. White paper, released on 22 Febru-ary 2012
24. Altunay, M., Avery, P., Blackburn, K., Bockelman,B., Ernst, M., et al.: A science driven productioncyberinfrastructure—the open science Grid. J. GridComputing 9(2), 201–218 (2011)
25. Berg, A.: PRACE distributed infrastructure servicesand evolution. In: EGI Community Forum 2012,Garching, 28 March 2012
26. Gentzsch, W., Denis Girou, D., Kennedy, A., Lederer,H., Reetz, J., et al.: DEISA-distributed European in-frastructure for supercomputing applications. J. GridComputing 9(2), 259–277 (2011)
27. Laganá, A., Costantini, A., Gervasi, O., Faginas Lago,N., Manuali, C., et al.: Compchem: progress to-wards GEMS a Grid empowered molecular simula-tor and beyond. J. Grid Computing 8(4), 571–586(2010)
28. Borgdorff, J., Falcone, J., Lorenz, E., Chopard, B.,Hoekstra, A.: A principled approach to distributedmultiscale computing, from formalization to execution.In: Proceedings of The Seventh IEEE InternationalConference on e-Science Workshops, Stockholm,
Sweden, 5–8 December 2011, pp. 97–104. IEEE Com-puter Society, Washington, DC (2011)
29. Frohner, A., Baud, J.P., Garcia Rioja, R.M.,Grosdidier, G., Mollon, R., Smith D., Tedesco, P.:Data management in EGEE. In: Journal of Physics,Conference Series 219 (2010)
30. Zappi, R., Magnoni, L., Donno, F., Ghiselli, A.:StoRM: Grid middleware for disk resource manage-ment. In: Proceedings of Computing in High-EnergyPhysics, 27 Sept–1 Oct 2004 Interlaken, Switzerland,(CHEP04), pp. 1238–1241 (2005)
31. Turner, D., Oline, A., Chen, X., Benjegerdes, T.: In-tegrating new capabilities into NetPIPE. Lect. NotesComput. Sci. 2840, 37–44 (2003)
32. Aiftimiei, C., Andreetto, P., Bertocco, S., Dalla Fina,S., Alvise Dorigo, A., Frizziero, E., Gianelle, A.,Marzolla, M., Mazzucato, M., Sgaravatto, M., TraldiS., Zangrando, L.: Design and implementation of thegLite CREAM job management service. Future Gener.Comput. Syst. 26(4), 654–667 (2010)
33. Alfieri, R., Cecchini, R., Ciaschini, V., dell’Agnello, L.,Frohner, A., Lorentey, K., Spataro F.: From Gridmap-file to VOMS: managing authorization in a Grid envi-ronment. Future Gener. Comput. Syst. 21(4), 549–558(2005)
34. Edwards, R.G., (LHPC Collaboration), Joó, B.,(UKQCD Collaboration): The chroma software sys-tem for lattice QCD. arXiv:hep-lat/0409003. In: Pro-ceedings of the 22nd International Symposium forLattice Field Theory (Lattice2004), Nucl. Phys. B140(Proc. Suppl), 832 (2005). See also: http://usqcd.jlab.org/usqcd-docs/chroma/
35. Goodale, T., et al.: The cactus framework and toolkit:design and applications. In: Vector and ParallelProcessing—VECPAR’2002, 5th International Confer-ence, Lecture Notes in Computer Science. Springer,Berlin (2003).
36. Schnetter, E., Hawley, S.H., Hawke, I.: Evolutions in3-D numerical relativity using fixed mesh refinement.Class. Quantum Grav. 21, 1465–1488 (2004)
37. Baiotti, L., Hawke, I., Montero, P.J., Löffler, F.,Rezzolla, L., Stergioulas, N., Font, J.A., Seidel, E.:Three-dimensional relativistic simulations of rotatingneutron star collapse to a Kerr black hole. Phys. Rev. D71, 024035 (2005). See also: http://einsteintoolkit.org/
