Date post: | 20-Jan-2016 |
Category: |
Documents |
Upload: | alexia-chandler |
View: | 214 times |
Download: | 0 times |
June 30 2006 Amsterdam
A Workflow Bus for e-Science Applications
Dr Zhiming ZhaoFaculty of Science, University of AmsterdamVL-e SP 2.5
Outline
Introduction A workflow bus and generic e-Science
framework Prototype and experiment results Discussion Conclusions Future work
Scientific workflow in e-Science
…
Grid infrastructure, E-Science and Scientific workflow
Step1: designing an experiment
Step
2: p
erformin
g th
e experim
ent
Step3: analyzing the experiment
results
Discovery Grid
Scientific Workflows in e-ScienceExperiment processes
Abstract workflows
Executable (concrete workflows)
wo
rkflow
s for ad
min
istration
, e.g.,
AA
A, an
d o
ther issu
es.
A SWMS is able to:
Automate experiment routines
Rapid prototype experimental computing systems
Hide integration details between resources
Manage experiment lifecycle
Insight a Scientific Workflow Management System
In our view, a SWMS at least implements:
A model for describing workflows;
An engine for executing/managing workflows;
Different levels of support for a user to compose, execute and control a workflow.
Workflow (based on certain model)
Engine
User su
pp
ort
resources
Composition
Engine level control
Resource level control
A SWMS
Diversity in SWMSTaverna:
-Web services based language: Scufl;
-FreeFluo: engine
-Graphical viz of workflow
Kepler:
-Actor,director
-MoML
-Execution models
Triana:
-Components
-Task graph
-Data/control flow
DAGMan:
-Computing tasks
-DAG
Pegasus:
-Based on DAGMan
-VDL
-DAG
…
Research context
Virtual LabGrid Layer
Application Layer
Different levels of abstraction
Workflow services: Short term Long term: a
generic and effective workflow management service
Mission
Effectively reuse existing workflow managements systems, and provide a generic e-Science framework for different application domains.
A generic framework can Improve the reuse of workflow components and the
workflows for different experiments Reduce the learning cost for different systems Allow application users to work on a consistent
environment when underlying infrastructure changed
Abstract approach
Extend approach
Aggregate approach
Possible options
SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMS1
SWMS2
SWMS3
SWMS G
Why we choose an aggregation approach? Abstract approach
Build a perfect system Difficult to find a set of systems cover all the required generic
functionality; it requires re-implementation of existing things Extend approach
Incrementally development The solution depends on a specific system
Aggregate approach Maximize the reuse of the existing workflow systems Has to handle interoperability issues; provide customized
interface existing workflow system
A workflow bus paradigm
Workflow bus
Taverna KeplerTriana
Sub workflow 1
Sub workflow 2
Sub workflow 3
Workflow
A workflow bus is a special workflow system for executing meta workflows, in which sub workflows will be executed by different engines.
Architecture
Terminology: The execution of a workflow is one study, and the execution of a
sub-workflow is called a sub-study, or a scenario Basic idea
Study manager schedules sub workflows Scenario managers interface third party workflow engines and
reacts to the Study manager A user interface for composition and execution control.
Network
Scenario MngerScenario MngerScenario Mnger Study Mnger
Taverna
Engine
Triana
Engine
User interface
Requirements
A distributed framework for study and scenario managers
Data input/output of a sub-workflow, description of the workflow can be described and recognized by study and scenario managers
Handle the user interactions which are needed in scenarios
The engine can be decoupled from a SWMS Be fault tolerant
Considerations
From integration point view: study and scenario managers can be coupled by: Web services Object oriented middleware (CORBA, HLA, etc.) Agent based middleware Or an existing workflow system (Kepler, Taverna,
Triana or others) The description of meta workflow The execution model of the meta workflow
A JADE/Ptolemy based prototype
Director
Actor
Actor
Actor
JADE agent framework
Scenario MngerScenario MngerScenario Mnger Study Mnger
Taverna
Engine
Triana
Engine
Ptolemy
User interface
How it works
In user front end: a user defines meta workflow, each actor represents a sub workflow
At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager
A scenario manager controls an engine and execute the sub-workflow
Prototype
Experiment results
Message delayMessage delay
0.001
0.01
0.1
1
10
100
0.001 0.01 0.1 1 10 100 1000 10000 100000
Size of message (KB, 1KB=1024 bytes)
Tim
e (s
eco
nd
s)
Scenario managers
SOAP
Cont.
Overhead
584.6
767.4
1636.1
1776.2
0
200
400
600
800
1000
1200
1400
1600
1800
time in ms
2048x1536 4096x3072Image
Triana workflow execution
Agent
Triana GUI
10~20% performance improvement.
10~20% performance improvement.
584.6
767.4
1636.11776.2
0
200
400
600
800
1000
1200
1400
1600
1800
Milliseconds
2048x1536 4096x3072Image
Triana workflow execution AgentTriana GUI
Discussion
Challenges in supporting scientific workflows Requirements on domain specific experiments Generic workflow support and domain specific
applications Existing workflow management systems are
diverse in functionality, design and user support Related work
Interoperability among workflow systems (sister Link project)
Resource level: e.g., Kepler invokes Taverna’s resources
Applications of workflow bus
Use case 1: A user has workflow in Taverna Some functionality is missing in Taverna but can be
provided by Triana He can develop the workflow in two systems, and run
it via the workflow bus
Use case 2: A user wants to execute a Taverna or Triana workflow
in multiple instances with different input data
Conclusions
A workflow bus is a feasible approach to realize generic e-Science framework
Multi agent technology provides a distributed environment for decomposing and encapsulating control intelligence
Ptolemy II provides different computing paradigms which give user freedom to execute workflows
Future work
Working on developing a scenario manager for Kepler engine.
Synchronized data flow is currently used; more computing modes will be evaluated.
Data provenance for workflow bus.
Referneces
Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp. 339-345 . IEEE Computer Society Press, Shanghai, China, September 2005.
Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp. 357-364. IEEE Computer Society Press, Melbourne, Australia , September 19th-21st 2005.
Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp. 480-485. Springer, November 2005. ISBN 3-540-30510-6. (DOI: 10.1007/11590354_61)
Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp. 19-23. IEEE Computer Society Press, Hongkong, China, November 14th-16th 2005.
Acknowledgement Suresh Booms All the members in VL-e SP2.5