Post on 27-Mar-2015
transcript
Workflow + Globus Toolkit Update
September 8, 2004
Allan Doyle, NASAAllan Doyle, NASA
adoyle@intl-interfaces.comadoyle@intl-interfaces.com
Sept 8, 2004Workflow + Globus Update
2
AgendaAgenda
• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)
Sept 8, 2004Workflow + Globus Update
3
AgendaAgenda
• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)
Sept 8, 2004Workflow + Globus Update
4
Workflow ActivityWorkflow Activity
• Began in late 2003• Goal was to be able to chain
processing among participants in a single workflow
• Held a series of teleconferences• http://grid-tech.ceos.org/gridwiki/WorkflowTige
rTeam
(Ask for username/password)
Sept 8, 2004Workflow + Globus Update
5
Workflow activity - 1Workflow activity - 1
• First, we learned about workflow and about CEOS Grid Participants’ workflow tools…
Sept 8, 2004Workflow + Globus Update
6
Workflow overviewWorkflow overview
• The next slides are taken from Tech Team members’ presentations.
• Annotations that were added for this presentation look like this
Annotation box…
Sept 8, 2004Workflow + Globus Update
7
What is a Workflow?What is a Workflow?
Dutch Space has renamed their
workflow tools to GridAssist…
Sept 8, 2004Workflow + Globus Update
8
Workflow LevelsWorkflow Levels
Application Domain
Abstract Workflow
Concrete Workflow
Execution Environment
Application Component Selection
Resource SelectionData Replica Selection
Transformation Instance Selection
= Data Transfer
FFT
FFT filea
host2
host1Retry
Pick DifferentResources
Specify a DifferentWorkflow
copy host1://home/filea to host2://u1/file1
/usr/local/bin/fft /u1/file1
From Pegasus presentation by Ewa Deelman, ISI
This material was redrawn to enhance visibility, it’s
from Ewa Deelman’s workflow presentation…
Sept 8, 2004Workflow + Globus Update
9
Typical Workflow EnvironmentTypical Workflow Environment
Sept 8, 2004Workflow + Globus Update
10
Workflow exampleWorkflow example
This is a domain (or application level)
workflow…
Sept 8, 2004Workflow + Globus Update
11
Workflow detailed exampleWorkflow detailed example
This is an abstract workflow…
Sept 8, 2004Workflow + Globus Update
12
GMU Virtual Product ConceptGMU Virtual Product Concept
Virtual data concept requires workflow tools in
order to be built…
Sept 8, 2004Workflow + Globus Update
13
Workflow progressionWorkflow progression
Different levels of workflow…
Domain… Abstract… Concrete…
Sept 8, 2004Workflow + Globus Update
14
Workflow toolsWorkflow tools
• DutchSpace - Grid Assist– Developed in-house, workflow engine and
client tools. (Runs on Globus Toolkit 2.x)
• NCSA (UAH partner) - Open GCE Runtime Engine– Based on Apache Ant (not Globus dependent)
• ISI - Pegasus– Developed as part of Earth Systems Grid work
w/Argonne (Runs on Globus 2.x, 3.x?)
• GMU - GeoTree– Based on BPEL (Business Process Execution
Language) (Runs on Globus 3.2.x)
Sept 8, 2004Workflow + Globus Update
15
Workflow Activity - 2Workflow Activity - 2
• Then we investigated workflow interoperability.
• The plan was to1. Ask participants for use cases2. Decompose use cases into workflow
patterns
Sept 8, 2004Workflow + Globus Update
16
Use case descriptionsUse case descriptions
• Identifier• Description• Actors• Goal• Preconditions• Assumptions• Frequency
• Basic Course• Alternate Courses• Included Use
Cases• Extended Use
Cases• Issues• Decisions
Workflows can be derived from these descriptions of actions
Sept 8, 2004Workflow + Globus Update
17
Workflow PatternsWorkflow Patterns
• http://tmitwww.tm.tue.nl/research/patterns/
• Provides a good explanation and definition of 21 workflow patterns.
• Next slide shows some of the simple ones as examples.
Sept 8, 2004Workflow + Globus Update
18
Basic workflow patternsBasic workflow patterns
A B CSequence: B is started after A completes. C is started after B completes.
A ANDB
C
Parallel Split: B and C are begun any time after A has completed.
CANDA
B
Synchronization: C is not begun until A and B have both completed.
A XORB
C
Exclusive Choice: Following A, a choice is made to pursue B or C but not both.
CXORA
B
Simple Merge: C activated once for every completion of A or B
Sept 8, 2004Workflow + Globus Update
19
Use case analysisUse case analysis
• Use cases were received from– USGS– UAH/NOMADS– Dutch Space
• USGS and UAH/NOMADS use cases were refined via teleconference + email
• All are available on Workflow web page
Sept 8, 2004Workflow + Globus Update
20
Initial FindingsInitial Findings
• Workflow tools are a necessary component!– Grid applications are partioned into work units
that run on many machines.– Runtime error handling is a vital part of
workflow engine capability
• Grid workflows + tools are complex– Tools take a long time to learn– Grid workflows support complex processing
tasks
• Many (~30) different Grid workflow systems exist– Interoperability among workflow tools is
currently nonexistent
Sept 8, 2004Workflow + Globus Update
21
Use case findings - 1Based on the USGS and NOMADS/UAH Use Cases
Use case findings - 1Based on the USGS and NOMADS/UAH Use Cases
• Found that use cases tended to have many “hands-on” steps where scientists would make choices– This is one of the key areas the Grid workflow
community has identified as needing work.
• Use of “formal” workflow tools may be more work than it’s worth for the use cases– Application projects are using the Grid mainly
for data transfer, not computing– Workflows include some automation areas
(reprojection, reformatting, subsetting).
Sept 8, 2004Workflow + Globus Update
22
Use case findings - 2Based on the Dutch Space Application description
Use case findings - 2Based on the Dutch Space Application description
• Dutch Space application is well suited for formal tool
• Dutch Space has implemented their own tool in response to their need and the (then) lack of existing tools.
• OMI application has long processing chain (c.f. earlier slides)
Sept 8, 2004Workflow + Globus Update
23
Workflow ConclusionsWorkflow Conclusions
• Workflow community is starting to talk about interoperable workflows– http://www.extreme.indiana.edu/groc/
ggf10-ww/index.html
• Perhaps cross-workflow interoperability is not an immediate need– A single workflow tool can be used to control
remote job execution without needing to interoperate with other workflow tools.
– CEOS Grid activity should wait until Application Projects have a clear need for cross-tool workflow interoperability before doing further work.
Sept 8, 2004Workflow + Globus Update
24
AgendaAgenda
• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)
Sept 8, 2004Workflow + Globus Update
25
Globus Toolkit (GT) UpdateGlobus Toolkit (GT) Update
• Current GT version is 3.2– Most CEOS Globus users are still using 2.4
• GT version 4.0 release is scheduled for January 31, 2005– http://www-unix.globus.org/toolkit/GT4Facts/
• GT4 will be backwards compatible with GT3.2
• Next 5 slides taken from the presentation given in Tromso, also available at
http://grid-tech.ceos.org/gridwiki/GlobusVersions
Sept 8, 2004Workflow + Globus Update
26
Sept 8, 2004Workflow + Globus Update
27
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Sept 8, 2004Workflow + Globus Update
28
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Sept 8, 2004Workflow + Globus Update
29
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Sept 8, 2004Workflow + Globus Update
30
Sept 8, 2004Workflow + Globus Update
31
GT 4.0 schedule and contentGT 4.0 schedule and content
• Next two slides have material from http://www-unix.globus.org/toolkit/GT4Facts/
Sept 8, 2004Workflow + Globus Update
32
GT4 Release ScheduleGT4 Release Schedule
Sept 8, 2004Workflow + Globus Update
33
GT4 ComponentsGT4 Components
Sept 8, 2004Workflow + Globus Update
34
Globus Toolkit RecommendationsGlobus Toolkit Recommendations
• If you are using– GT 2.4
• Migrate to 4.0• Don’t go to 3.2 as intermediate step
– GT 3.2• Migrate to 4.0• 4.0 will be backwards compatible
Sept 8, 2004Workflow + Globus Update
35
AgendaAgenda
• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)
Sept 8, 2004Workflow + Globus Update
36
CA ActivityCA Activity
• Following earlier work on Certificate Authorities (CA), NASA IPG (Information Power Grid) volunteered to provide CA services for CEOS Grid.
• NASA IPG funding has been cut– CEOS Grid cannot rely on IPG CA
anymore
• We decided to establish a CEOS CA on a NASA machine at GSFC
Sept 8, 2004Workflow + Globus Update
37
CA Activity - cont.CA Activity - cont.
• Ananth Rao has installed a test CA and has produced a Certificat Request tool similar to the one provided by IPG
• This tool will undergo further testing after WGISS-18
• Once we are satisfied with it, we will provide it to the CEOS Grid