Flexible Scientific Workflows Using Dynamic Embedding

Ngu, Texas StatePtolemy Miniconference, February 13, 2007

Flexible Scientific Workflows Using Dynamic Embedding

Anne H.H. Ngu, Nicholas HaaschTerence Critchlow

Shawn Bowers, Timothy McPhilips, Bertram Ludaescher


Outline

Scientific Workflow Problems with static scientific workflow Frame actor Dynamic Embedding Implementation of Dynamic Embedding TSI case study Conclusion


Scientific Workflows

A model of the way a scientist works with their data and tools Mentally coordinate data export, import, analysis and visualization via various

software tools

Goals: Design Automation Component reuse

… make data analysis and

management tasks easier

for the scientist!


SPA/Kepler Scientific Workflow System

Scientific Process Automation (SPA) Modeling Workflows using actor-

oriented framework Executing and Monitoring

Workflows Built on top of Ptolemy II

(Berkeley) Graphical User Interface

Similar to a charting program (Visio)

• Drag-and-Drop Components• Connect components

Execute workflows Monitor Execution Ptolemy II

SPA


Actor-Oriented Modeling

Actors single component or task well-defined interface (signature) given input data, produces output data



Parameter Input that is configured statically

• User changes at design Produces data



Sub-Workflows (aka Composite Actors) composite actors “wrap” sub-workflows like actors, have signatures (i/o ports of sub-workflow) hierarchical workflows (arbitrary nesting levels) for abstraction versus ‘Atomic Actors'



Directors define the execution semantics of workflow graphs schedule and execute workflow graphs sub-workflows may be governed by different directors Examples: Synchronous Data-Flow (SDF), Process Networks (PN),

Discrete Event (DE), Finite State Machine (FSM)


Problems in current SPA/Kepler modeling framework

Workflow is static and must be completely specified before orchestration

Specific tools used in the workflow must be picked at design time (It

may not be picking the best tool) All alternative tools

must be enumerated exhaustively (resulted in

complex workflow)


Problems in current SPA/Kepler modeling framework (cont.)

Our work aims to provide an abstraction of a tool, a resource, or an algorithm

Specific tool or resource resource is selected during execution


Terascale Supernova Initiative (TSI) Workflow

Submit JobTransfer Files

Process Files

Y-A

xis

X-Axis

Visualize


Different TSI Workflows

Each containing a Submit Job Actor

A

B C


TSI Submit Job Actors…when we look at the sub-workflows

Each workflow does job submission in a different way

Each containing a remote execution

AC

B


Goals

There exists actors (atomic and composite) that perform a similar task Submit Job Transfer Files Process Files RemoteExection

• SSH2Exec

• SSHWrapper

• InvokeSubmitJob Can we provide an abstraction for encapsulating different implementations of a

specific task that can be reused across different workflows? Can we execute the workflow flexibly?

Choosing a specific implementation depending on runtime condition


Requirements

Select and execute actor based on run time conditions Execute with data process networks

• Built in capability for streaming data• Built in concurrency

Selected actors instantiated on demand Reusable actor can be nested Actors are usable by scientist


Frames

Actors are concrete Correspond to particular

implementations Frames are abstract

Placeholder for actor / composite actor

input, output, and parameter ports An embedding occurs when a actor

is placed in a frame A refinement is an actor that can be

embedded in a frame

F

a

Embedding F[a]

F

a

RemoteExecutionFrame F

SSH Actor a

a


SSH2Exec

WebService

WebService

SSH2Exec

Static Embedding Frames

A refinement to a Frame is embedded during design Frames become concrete and cannot be reconfigured during

workflow execution

F

Design Time

WebService

SSH2Exec

Run Time


Static Embedding Frames (cont.)

Execute with data process networks Refinement instantiated as needed Can be nested Actors are usable by scientist Select refinements based on run time conditions

SSH2Exec

F

Run Time

WebService

F


Dynamic Embedding Frames

Refinement to frames are embedded during execution Frames are not concrete during workflow execution

SSH2Exec

WebService

WebService

SSH2Exec

F

WebService

SSH2Exec

Run Time


Why Dynamic Embedding?

Select refinements based on run time conditions Execute with data process networks Refinement instantiated as needed Can be nested Actors are usable by scientist


Implementation Of Dynamic Embedding

Construct a new workflow to execute the actor

Generates

Remote Execution FrameGenerated Workflow


Dynamic Embedding Process

1. Wait for inputs to arrive to the Frame.

2. Select a refinement.

3. Transfer of input tokens from Frame to the refinement.

4. Select mappings Input Port Output Port Parameter

5. Constructs internal workflow.

6. Run internal workflow.

7. Transfer of output tokens from the refinement to the Frame.

Model Generated By Dynamic Frame

Remote Execution Frame


ModelReference Actor

A higher order actor that can execute a given model (workflow) through its input port.

It fits most of the requirement for dynamic embedding except: User must pre-construct the given model Output tokens from the given model are transferred only after

completion of the internal workflow.

Our implementation of dynamic embedding leverage the capability of ModelReference actor with two major improvements: The given model is constructed automatically Output tokens are transferred synchronously

Frame is thus implemented as a subclass of ModelReference actor with four additional components: SelectActor(), FrameSourceActor(); FrameSinkActor() and PortWiring()


Implementing a Type of Frame

Subclass Frame and Implement Selection Process

• selectActor()

Configure Ports and Parameters • getIntputMappings()• getOutputMappings() • getParameterMappings()


Selection Process: selectActor()

Implements selection policy Returns a refinement Refinement that is returned

gets automatically embedded

Remote ExecutionString selectedActor

selectActor(){ If(testWebService()) selectedActor = “webservice” return getWebServiceActor() else selectedActor = “ssh2exec” return getSSH2ExecActor()}


Port Wiring

Transfer Token Frame Input PortActor Input Port Actor Output PortFrame Output Port

Expressed as list containing pairs of strings {“hostname”,”hostname”}

{“command”,”cmd”}

SSH2Exec

Ferrors

stdouterrors

outhostname

cmd

hostname

command

Remote Execution

String selectedActor

getInputPortMapping(){ if(selectedActor==“SSH2Exec”) return {{“hostname”,”hostname”} {“command”,”cmd”}} else if(selectedActor==“webservice”) return {{“hostname”,”url”} {“command”,”method”}}}


TSI-A Workflow

SubmitJobFrame

TSI-B Workflow

TSI Case Study


Remote Execution Frame

SubmitJobFrameTSI-B subworkflow

TSI-A subworkflow


Benefits

Select refinements based on run time conditions Execute with data process networks Refinement instantiated as needed Can be nested Actors are usable by scientist


Limitations

Limitations Unable to type check internal workflow before execution There is overhead of creating an additional workflow to execute a

refinement Change in selection process requires recoding/recompiling Can not monitor internal workflow (Useful for debugging)


Future Work

Semantic binding of Ports and Parameters Configurable selection criteria

Intelligent brokering Ptolemy Expression Language Python Perl …

Simplified refinement creation Caching of generated workflows Design time type checking of internal workflow


Work performed under the auspices of the U. S. Department of Energy by Lawrence Livermore National Laboratory under Contract W-7405-Eng-48

UCRL-ABS-226047

Date post:	31-Dec-2015
Category:	Documents
Upload:	iris-park
View:	33 times
Download:	4 times

Flexible Scientific Workflows Using Dynamic Embedding

Documents