23
What is a multi-workflow simulation?
• A simulation workflow where nodes of the simulation are themselves workflows potentially based on different workflow languages
• A practical example: LINGA application (outGRID project)– Combining several workflows (2 CIVETs+FreeSurfer+STAT)
– Heterogeneous workflow systems (LONI/MOTEUR)
• It also should enable the execution in different DCIs. LINGA application:– LONI Cluster (USA)– gLite-based neuGRID infrastructure (Europe)– CBRAIN HPC infrastructure (Canada)
2
Experiment setup
3
CIVET @ CBRAIN
LONI Pipeline151 input data items
CIVET @ neuGRID
LONI Pipeline146 input data items
FreeSurfer @ CRANIUM
LONI Pipeline
STATS @ EGI
MOTEUROutputs of both CIVETs
SHIWA solution for LINGA
Sub-WorkflowsManagement
4
Multi-Workflow
5
Start date: 01/07/2010
Duration: 27 months
Total budget: 2,101,980 €
Funding from the EC: 1,800,000 €
Total funded effort in person-months: 231
Web site: www.shiwa-workflow.eu
Coordinator: Prof. Peter Kacsuk, email: [email protected]
SHIWA (SHaring Interoperable Workflows for Large-Scale Scientific Simulations on Available DCIs) project
6
Motivations 1• In many cases large simulations are organized
as scientific workflows that run on DCIs• However, there are too many different
• WF formalism• WF languages• WF engines
• If a community selected a WF system it is locked into this system:• They can not share their WFs with other communities
(even in the same scientific field)• They can not utilize WFs developed by other
communities
WF Ecosystem
7
Who are the members of an e-science community from WF applications point of view?
End-users (e-scientists) (5000-50000)• Execute the published WF applications with custom
input parameters by creating application instances using the published WF applications as templates
WF Application Developers (500-1000)• Develop WF applications
• Publish the completed WF applications for end-users
WF System Developers (50-100)• Develop WF systems
•Writes technical, user and installation manuals
accessing a large set of various DCIs to make these WF
applications run
Clouds
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
E-science infrastructure
What does a WF developer need?WF App.
Repository
Access to a large set of ready-to-run
scientific WF applications
Portal
Using a portal/desktop to parameterize and run these applications, and to further
develop them
accessing a single DCI to make these WF applications run
Cluster based service grids (SG)
(e.g. ARC)
Grid system
In the past: WF developers worked in an isolated way, on a single DCI
Portal/desktop
Using a portal/desktop to develop WF applications
As a result if a community selected a WF system it is locked into this DCI• Porting the WF to another
DCI required large effort• Parallel execution of the
same WF in several DCIs is usually not possible
After SHIWA: Collaboration between WF application developers
SHIWA App.
Repository
SSP Portal
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
Application developers
• Publish WF applications in the repository to be continued by other appl. developers
•Application developers use the portal/desktop to develop complex
applications (executable on various DCIs) for various end-user communities
Project objectives• Enable user communities to share their WFs
– Publish the developed WFs– Access and re-use the published WFs– Build multi-workflows from the published WFs
• Toolset:– SHIWA Simulation Platform
• WF Repository (production)• SHIWA Portal (production)• SHIWA Desktop (prototype)
12
Coarse-grained interoperability (CGI)
• CGI = Nesting of different workflow systems to achieve interoperability of WF execution frameworks
Multi-workflow
Export to IWIR
Import from IWIR
WFBWFA
Interoperable Workflow Intermediate Representation IWIR
Fine-grained interoperability (FGI)
Tools for CGI SHIWA services
• SHIWA repository to:– Describe workflows– Share workflows
• SHIWA portal to:– Access and enact registered workflows– Compose and enact multi-workflows – Monitor workflows and multi-workflows
execution in various DCIs– Retrieve results of the execution
SHIWA Repository facilitates publishing and sharing workflows
Supports:• Abstract workflows with multiple implementations of over 10 workflow systems• Storing execution specific data
Available:• from the SHIWA Portal• standalone service at: repo.shiwa-workflow.eu
EGI Community Forum, Munich, March 28, 2012
Scenario: Find and test WFs• SHIWA Repository: Analyze description, inputs and outputs of published WFs
• SHIWA Portal: Instantiate WF from repo, execute with given sample data (inside WS-PGRADE workflow used as the Master WF system)
17
18
Title: Work Package
SA1...Author:.G
Terstyanszky..v.:1.0
18
SHIWA Portal: Workflow Editor
19
Title: Work Package
SA1...Author:.G
Terstyanszky..v.:1.0
19
SHIWA Portal: Configuring Workflow
20
Title: Work Package
SA1...Author:.G
Terstyanszky..v.:1.0
20
SHIWA Portal: Executing Workflow
21
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WE + WF
WF1 WFm
GEMLCA with GIB
WF list
WS-PGRADEWorkflow
engine
WS-PGRADE Workflow
editor
edit WF
s2
search WF
s1
s5
s4
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
pre-deployed-WEs
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
ASKALON WE
researcher
invoke WEs6
CGI User Scenario with WS-PGRADE as master
SHIWA Proxy Server
Proxy Server
s3
s6
submit WE
Advantages for the various types of user communities using SHIWA
• WF system developers– Better visibility: much more WF developers can access and use their
WF system than before (through the applications stored in the SHIWA repo)
– The joint impact is much bigger than the individual WF systems can achieve
• WF developers– They can collaborate: share and re-use existing WF applications
– WF application development can be accelerated
– More complex WFs can be created in shorter time
– They will access many different DCIs (their WF will be more popular)
• End-users– much bigger set of usable and more sophisticated WF applications
– These applications can run on various DCIs22
Conclusions
• SHIWA brings advantage for all the 3 kinds of user communities:– WF system developers– WF developers– End-users
• With relatively little effort– WF systems can join the SSP – WF system developers can adapt SHIWA technology
• Further information: www.shiwa-workflow.eu
23
