Grid workflow authoring tools are typically specific toparticular workflow engines built into Grid middleware, orare application specific and are designed to interact withspecific software implementations. g-Eclipse is a middle-ware independent Grid workbench that aims to provide aunified abstraction of the Grid and includes a Grid work-flow builder to allow users to author and deploy workflowsto the Grid. This paper describes the g-Eclipse WorkflowBuilder and its implementations for two Grid middlewares,gLite and GRIA, and a case study utilizing the WorkflowBuilder in a Grid user’s scientific workflow deployment.
1. Introduction
Workflow composition is an important part of reusingexisting scientific analysis methods. Many of these scien-tific methods require high-performance computing (HPC)resources, and these are commonly provided by large clus-ters and supercomputers, or through using distributed com-modity hardware forming computational grids. Tools forconnecting to and manipulating such resources are usuallymiddleware specific with no universal standard for HPC orGrid computing. Likewise, tools for workflow authoringalready exist, but they are not designed to cater for multi-ple middlewares. In this paper we describe the WorkflowBuilder plugin of the g-Eclipse [1, 2] workbench, a uni-
versal graphical user interface for accessing existing Gridinfrastructures, and its exemplary support for the author-ing and submitting of workflows to two different middle-wares, gLite and GRIA. We provide an overview of the dif-ferent functionality that is available to a Grid User actor andhow the Workflow Builder integrates with the job and datamanagement tools in g-Eclipse. Finally we report on howg-Eclipse was used to demonstrate scientific and industrialapplications with the Workflow Builder being a core com-ponent of a Grid User’s activities.
2 Background
As the need for processing, analyzing and simulatinglarge sets of data grows, so does the need for larger com-puting and data capacity. The field of Grid computing ad-dresses this by providing methods for multiple collaborat-ing entities to share distributed compute resources under anumbrella of a virtual organization [3]. For example, one ofthe aims of the EGEE project is to provide the worldwidescientific research community with a universally accessi-ble Grid infrastructure. This infrastructure is divided into anumber of virtual organizations segmenting groups of users,resources and institutions that map onto logical adminis-trative domains. Grid middleware is software that is usedas the ’glue’ that ’sticks’ together distributed computingand storage resources, presenting a standardized interfaceto the collective resource, where from a user-perspective,the underlying disparity of resources is hidden. The EGEE
project developed the gLite middleware and included com-ponents taken from a number of Grid projects [4] includingDatagrid (EDG), DataTag (EDT), DataGrid (EDG), INFN-GRID, Globus and Condor. Although the gLite middlewaregoes some distance to simplify the underlying organizationand management of Grid computing resources, interfacingwith gLite-based virtual organizations is typically carriedout through command-line sessions and interacting with thedifferent subsystem and services individually. Some efforthas been made to create graphical clients to access EGEE’sGrid infrastructure, however these approaches targeted spe-cific functionality such as job submission or data manage-ment, or produced domain-specific user interfaces. Therewas no single graphical client software that encapsulatesGrid user functionality available for interfacing with EGEEGrids, and this is atypical of many other Grid and high-performance computing (HPC) infrastructures.
The European Commission Sixth Framework funded g-Eclipse project built an integrated workbench framework toaccess and operate existing Grid infrastructures and is de-scribed in detail by Gjermundrød et al [5]. Based on topof the open-source Eclipse framework, the g-Eclipse work-bench provides tools to customize Grid users’ applications,to manage Grid resources, and to support the developmentcycle of new Grid applications. The project aimed at pro-viding generic Grid workbench tools that can be extendedfor many different Grid middlewares, such as gLite, UNI-CORE [6], and Globus [7], and it comes with exemplarysupport for the gLite and GRIA [8, 9] middlewares. Anadapter to Amazon’s Web services, Simple Storage Service(S3) and Elastic Compute Cloud (EC2) [10], was also devel-oped which introduced Cloud Computing support to whatwas initially a project intended for just the Grid.
An integral part of the g-Eclipse workbench is the Work-flow Builder plugin that allows users to compose work-flows made up of Grid job descriptions. These job descrip-tions describe the semantics of a Grid job including exe-cutable definitions and locations of input and output data.The Workflow Builder enables users to take these descrip-tions and stitch them together into a workflow. These work-flows can then be executed or deployed using the standardjob submission mechanism provided by the g-Eclipse work-bench (which is also used for executing single jobs) whereconversion to middleware specific workflow descriptions isdone on-the-fly.
3 The g-Eclipse Workflow Builder
The problem of expressing Grid workflows has been ad-dressed by many different projects, however there has notbeen any single adopted standard for expressing and per-sisting workflows. While the g-Eclipse Workflow Builderattempts to cater for different kinds of Grid middleware sys-
tems that support workflows, it must be noted that the modelis not intended to be a solution to enable interoperability be-tween workflow systems. The approach taken is to providea workflow builder that fits into g-Eclipse’s own generalizedGrid model abstractions whilst being able to cater for othermiddlewares.
The g-Eclipse Grid model abstracts a compute job tobe executed as a Job Description, and a job that has beensubmitted to execute on the Grid as a Job. The workflowmodel only links into the Job Description abstractions, asthe Workflow Builder plugin does not yet link into the jobmonitoring functionality that is present in the workbench.Job Descriptions provide a standard structure in which todefine the parameters of a Grid Job and typically includesthe location of an executable, locations of input data andlocations on the Grid to write output data to. In g-Eclipse,Job Descriptions are expressed in an Open Grid Forum’sstandard, the Job Submission Definition Language (JSDL)[11], and Job Descriptions are saved in the workbench filesystem as JSDL files. When creating a new Job Descrip-tion, a wizard provides a step-by-step interface for users topopulate the most important fields in a new JSDL file andg-Eclipse includes a multipage JSDL editor (see figure 1) inwhich users can edit a variety of parameters defined by thestandard.
Figure 1. The g-Eclipse workbench with theJSDL Editor displayed in the main view.
Workflows can be thought of as process-oriented sys-tems, where an overall process is broken down into sub-processes [12]. These sub-processes can then be organizedin such a way as to optimize their execution, for exampleexecuting processes that are not dependent on each otherin parallel to reduce total execution time. A workflow de-scribes semantic relationships between sub-processes. InGrid workflow systems these relationships are almost al-ways data flow dependencies, where the input of a givensub-process depends on one or more outputs of another.
87
To maintain consistency with the rest of the Grid model,workflows are modeled in g-Eclipse as a set of Job Descrip-tions (as JSDLs) and the data dependencies between them.This allows users to reuse existing single Job Descriptionsto build more complex processes.
The Workflow Builder presents users with a graphicaldiagramming canvas on which to build their workflow. Apalette allows users to pick and drop four kinds of elementon to elements of the diagram. The following list of ele-ments are illustrated in figure 2:
• Workflow Job - A container for Job Descriptions illus-trated as a rectangle on the diagram canvas. By default,adding a Workflow Job to a diagram logically adds anempty description. As such, a context menu action ordouble-clicking on an empty Workflow Job loads upthe JSDL wizard so that the user can define a new JobDescription and associate it with the Workflow Job.
• Input Port - A representation of a single location ofdata intended as input for a Workflow Job and illus-trated as a small square with a downward facing whitearrowhead placed on a Workflow Job. Any number ofInput Ports can be added to Workflow Jobs. Double-clicking on a Input Port allows a user to provide a URIlocation of some associated data.
• Output Port - Similar to an Input Port, however rep-resents output data locations but is decorated with anupward facing white arrowhead. Adding and remov-ing ports updates the associated Workflow Job’s JSDLfile’s corresponding data-staging properties.
• Link - Representation of the actual flow between Work-flow Jobs illustrated as an arrow between Input Portsand Output Ports. Links can only flow out of an OutputPort and in to an Input Port.
Figure 2. Three jobs with ports and connectedby links.
Although a workflow can be built from scratch using thepalette, a degree of automation has been implemented into
the g-Eclipse Workflow Builder. Users can take their ex-isting JSDLs that they may have authored using the JSDLEditor, or imported from elsewhere, and can quickly usethem to build a new workflow. JSDL files that are presentwithin a g-Eclipse Grid project explorer can be drag-and-dropped directly onto a workflow diagram canvas. Whena JSDL file is dropped on the canvas, a new Workflow Jobis created, and corresponding Input Ports and Output Portscreated according to the job description. A context menu ac-tion has also been implemented to attempt to automate thecreation of links between ports. When the ”auto-connect”action is executed, the Workflow Builder attempts to de-termine the semantic links between Input Ports and OutputPorts by searching for what URI locations match each other.For example, where an Output Port’s URI matches an InputPort’s, a link is automatically added.
A Grid workflow in g-Eclipse is saved as a combinationof files. When JSDLs are added to a workflow diagram, acopy of the original JSDL is created and uniquely associatedwith the diagram. This was done to ensure that JSDLs couldnot be modified and moved outside of the context of theworkflow once they have been associated with a WorkflowJob. The actual description of the graphical diagram and itssemantics is saved in a file in XML Metadata Interchange(XMI) format [13], which is an XML-based serialization ofthe graphical model defined by the Eclipse Graphical Mod-eling Framework (GMF) 1. When inspecting a workflow di-agram project, a single diagram is seen as a file-system di-rectory containing the XMI description and the associatedJSDL files.
4 Supported Middlewares
Workflows are handled by different middlewares in dif-ferent ways, and the two exemplary middleware implemen-tations for g-Eclipse highlight this. In g-Eclipse, a user canuse the context menu on a workflow in the Grid Project ex-plorer to execute middleware-specific functionality that isprovided by the corresponding middleware plugins whereappropriate. One must remember that g-Eclipse does notprovide any form of workflow orchestration or enactmentbut acts only as an authoring tool and delegates workflowdescriptions to the Grid middleware to handle. At the timeof writing, the g-Eclipse Workflow Builder provides im-plementations for two major Grid middlewares, gLite andGRIA.
1Eclipse GMF was used as an aid to develop the graphical model thatthe Workflow Builder is based on. More details on GMF can be found athttp://www.eclipse.org/gmf
88
4.1 gLite
The gLite middleware provides a user-centric workflowengine that allows any Grid user to try and submit a work-flow to the Grid. Bascially, gLite handles workflow jobsin the same way as it does single jobs. gLite does not ac-cept JSDL files, so the g-Eclipse job submission mechanismtranslates the JSDL descriptions on-the-fly into JDL (JobDescription Language, developed for gLite specifically),where the process is transparent to the user. JDL supports itsown description of workflows in a special case of JDL file.Regular JDL files describe a single anonymous job. Work-flow JDL files contain multiple jobs which are labelled, andan extra attribute that describes the dependencies as a setof label tuples is included. In gLite the dependencies mustdescribe a direct acyclic graph (i.e. the dependency graphmust have no cyclic dependencies), and such workflow jobsare referred to as DAGs.
For gLite workflows, it must be noted that the intendeduse-cases for workflows make several assumptions. Firstly,each defined job in a workflow must be valid in that anexecutable application is available, and the input and out-put data locations are valid and reachable. Second, the re-quired resources are available to the user. Finally, the indi-vidual jobs have been configured to deal with long delayson the Grid. In our experience, we found that when testingworkflows against the EGEE infrastructure using gLite, re-sources are not always immediately available. As a result,the sub-jobs within a workflow can easily timeout or reachtheir maximum retry count before the workflow completes.Although this is a difficulty we experienced in our develop-ment tests, once the workflows are submitted to gLite fromg-Eclipse, the responsibility for its execution lies with themiddleware.
4.2 GRIA
The GRIA middleware differs from gLite in that it onlyallows users to execute applications that have already beeninstalled on a GRIA Job Service, by a service operator. Thismeans that typical Grid users are restricted to specific pa-rameters afforded by those applications, and g-Eclipse han-dles this by auto-populating a JSDL when a user selects aparticular application returned by the GRIA Job Service inthe JSDL Wizard. The workflow functionality in GRIA isalso not treated in the same manner as in gLite. GRIA work-flows are not generally intended to be user-centric, but forGrid operators to use to provide encapsulated complex ser-vices for users.
However, GRIA provides application service packagesto support either workflows published on a GRIA Job Ser-vice (and consumed by end users as GRIA applications) oran additional package to add support for the user submit-
ting a workflow to be published automatically by the GRIAservice. The GRIA Workflow Application [9] provides acommand-line tool which takes a workflow description ex-pressed in XML Simple Conceptual Unified Flow Language(XScufl) [14], and a few other parameters, and creates a newapplication on the GRIA Job Service that exposes the work-flow to users as a single encapsulated process. The GRIAWorkflow Deployer Application can also be installed on aGRIA Job Service (as a GRIA application); this wraps thecommand-line workflow deployment tool, taking in an XS-cufl workflow as input, as provided by the end user. The re-sult is a newly deployed GRIA application that implementsthe workflow.
g-Eclipse authored workflows can be translated into XS-cufl descriptions using a context menu action. After trans-lation, the GRIA Workflow Deployer Application can beaccessed by a user via the standard g-Eclipse job submis-sion mechanism (by creating a new JSDL to use the GRIAWorkflow Deployer Application) to upload and install thenew workflow to the Job Service site, whereby a new appli-cation becomes available to the user that encapsulates thenew workflow’s functionality. The user can then execute theworkflow by selecting the new application using the JSDLWizard and submitting the JSDL to the GRIA Job Service.
5 Case Study: A Pharmaceutical Application
To illustrate the use of the g-Eclipse Workflow Buildera number of exemplary case studies were developed thatdemonstrate several aspects of the g-Eclipse workbench, in-cluding the Workflow Builder’s functionality. At a genericlevel the case studies followed a number of steps includ-ing: Creating a Grid project; Creating GRIA job descrip-tions as JSDLs; Constructing a workflow from the JSDLfiles; Translating the workflow into XScufl; Deploying theXScufl to the GRIA Job Service using the GRIA WorkflowDeployer Application; Creating a Job Description (JSDL)for the newly deployed workflow application; Executing thenew workflow JSDL; Visualizing the final output using theg-Eclipse visualization facilities.
One of the case studies carried out was a pharmaceuticalapplication that implements several steps in the drug dis-covery process of analysing organic molecules. The Gridservices used in this case study were derived from the SIM-DAT project [21] that aimed to introduce Grid technologyto various industry sectors, including the pharmaceutical in-dustry. The workflow that was to be realized included threemain sub-processes that were deployed on a GRIA site asindividual applications:
• BLAST sequence analysis - The BLAST (Basic Lo-cal Alignment Search Tool) , described by Altschul in[15], is an application used to perform alignment anal-
89
ysis on biological sequences to discover similar struc-tures in existing sequences databases.
• ANTIGENIC - This application analyzes a protein se-quence to find potential antigenic regions.
• SRS3D [16] - As a final step, an application is usedto transform the output of the sequence analyses intoa data format that can be visualized by the g-Eclipsevisualization plugins.
Figure 3. The pharmaceutical workflow beingauthored in the g-Eclipse Workflow Builder.
Each application step can take outputs directly from pre-vious steps in the workflow, and it was possible to con-struct a workflow using the g-Eclipse Workflow Builderfrom these applications, as shown in figure 3. The workflowwas simply the sequence of applications linked together inserial, each taking the output from the previous step. Figure4 shows the output visualized in g-Eclipse from the con-structed workflow.
Once the workflow had been constructed and deployed,users could then simply provide an intial input sequence,and as an output receive the analyzed and transformed dataready for visualization without any intervention in the in-termediary steps as was before. Simplifying repetitive setsof processes such as the BLAST-ANTIGENIC-SRS3D caseinto a single application that hides the underlying workflowcan greatly reduce the effort required in not just pharma-ceutical analyses, but with any other scientific analysis pro-cesses that require repetitive and complex sub-activities.
6 Related work
There are two major scientific workflow design tools thatare more mature and established than the g-Eclipse Work-flow Builder. However it should be noted that g-Eclipseaims to be a generic Grid workbench, where the WorkflowBuilder is just one component of the workbench.
Taverna is a workflow composition and enactment toolwhich is a product of the myGrid UK e-Science project [17].
Figure 4. The output of the workflow dis-played in the g-Eclipse Visualization Plugin.
Taverna aimed to enable bioinformatics analyses by allow-ing users to build workflows to access Web services that ex-pose information repositories and compute resources. TheScufl workflow language (a precursor to XScufl) was de-veloped by the project as there was no suitable standardfor composing scientific workflows at the time. Like g-Eclipse, Taverna provides a graphical workbench for au-thoring workflows, however it also includes a workflow en-actor called Freefluo. The g-Eclipse Workflow Builder doesnot provide any facility for workflow enactment and re-lies on the middleware implementations to deal with work-flow execution. GRIA supports deployment of Taverna au-thored XScufl workflows via a GRIA plugin to the Tavernaworkbench. Taverna has been widely adopted by the UKeScience community and the details of this are discussed atlength by Oinn et al in [18].
Kepler is a scientific workflow creation and executiontool that builds on the Ptolemy II concurrent componentcomputation system [19]. Kepler allows users to build com-plex data-centric workflows in contrast to Taverna that isbased on a singular-dataflow paradigm of workflows. AsKepler is builds on the Ptolemy system, it adopts Ptolemy’sModeling Markup Language (MoML) to persist workflows.Again, like g-Eclipse, Kepler provides a graphical work-flow authoring tool, and like Taverna, also supports work-flow execution. Kepler supports both a Web service andGrid resource workflow composition and has been adoptedby a range of scientific fields such as biology, astrophysicsand chemistry. The system is described in Altintas et al in[20].
7 Conclusions
In this paper we described the Workflow Builder tool thatis part of the g-Eclipse Grid workbench. Authoring and de-ploying workflows is an essential part of developing scien-tific analyses techniques that require access to large data and
90
computing resources, and the g-Eclipse Workflow Buildergoes some way to enabling users to author workflows anddeploy them to different middlewares. Currently two Gridmiddlewares are supported by g-Eclipse, and the workbenchframework is extensible enough to introduce support forother systems in the future. g-Eclipse is available as opensource and can be downloaded from http://www.geclipse.euor http://www.eclipse.org/geclipse
Acknowledgments
This work was supported in part by the EU under project(#FP6-2005-IST-034327), and the Eclipse Foundation. Theauthors would also like to thank the participating institu-tions of the g-Eclipse consortium and its project members.
References
[1] H. Kornmayer et al, ”g-Eclipse Project,” in Proc. 2ndAustrian Grid Symp., Innsbruck, Austria, 2006.
[2] ”g-Eclipse - Access the power of the Grid,” 2009. [On-line]. Available: http://www.geclipse.eu [Accessed: Oct.14, 2009].
[3] I. Foster et al, ”The Anatomy of the Grid: EnablingScalable Virtual Organizations,” in Int. J. High Perfor-mance Computing Applicat., vol. 15, no. 3, pp. 200-222,Aug. 2001.
[4] Sciaba et al, Ed. (2009, Apr. 28), gLite3.1 User Guide (v1.2) [Online]. Available:http://glite.web.cern.ch/glite/documentation/ [Accessed:Oct. 14, 2009].
[5] H. Gjermundrød et al, ”g-Eclipse - An IntegratedFramework to Access and Maintain Grid Resources,”in Proc. 9th IEEE/ACM Int. Conf. Grid Computing,Tsukuba, Japan, 2008, pp. 57-64.
[6] D. Erwin, Ed. (2003) UNICOREPlus Final Report [Online]. Available:http://www.unicore.eu/documentation/files/erwin-2003-UPF.pdf [Accessed: Oct. 14, 2009].
[7] I. Foster, ”Globus Toolkit Version 4: Software forService-Oriented Systems,” in J. Comput. Sci. and Tech-nology, vol. 21, no. 4, pp. 513-520, Jul. 2006.
[8] M. Surridge et al, ”Experiences with GRIA - Indus-trial Applications on a Web Services Grid,” in Proc. 1stInt. Conf. e-Science, Melbourne, Australia, 2005, pp. 98-105.
[9] ”GRIA - Service Oriented Collaborations for In-dustry and Commerce,” 2009. [Online]. Available:http://www.gria.org [Accessed: Oct. 14, 2009].
[15] S. Altschul, ”Basic Local Alignment Search Tool,” inJ. Molecular Biology, vol. 215, no. 3. pp. 403-410, Oct.1990.
[16] S.I. O’Donoghue et al, ”The SRS 3D module: inte-grating structures, sequences and features,” in Bioinfor-matics, vol. 20, no. 15, pp. 2476-2478, Apr. 2004.
[18] T. Oinn et al, ”Taverna: a tool for the composition andenactment of bioinformatics workflows,” in Bioinformat-ics, vol. 20, no. 17, pp. 3045-3054, Jun. 2004.
[20] I. Altintas et al, ”Kepler: An Extensible System forDesign and Execution of Scientific Workflows,” in Proc.16th Conf. Scientific and Statistical Database Manage.,Santorini, Greece, 2004, pp. 423-424.
![Netex learningMaker | Multipage Template v2.5.3 [En]](https://static.documents.pub/doc/80x56/541608928d7f728a6c8b4939/netex-learningmaker-multipage-template-v253-en.jpg)
![M:JRA1.9 Implementation of JSDL into OMII-Europe ... · The Job Submission Description Language (JSDL) [JSDL] is an XML-based notation for describing the requirements of computational](https://static.documents.pub/doc/80x56/5ed9033c6714ca7f4768fbd9/mjra19-implementation-of-jsdl-into-omii-europe-the-job-submission-description.jpg)
