Load-Workflow in WS-VLAM

Third Provenance ChallengeAmsterdam, The Netherlands June 2009

http://twiki.ipaw.info/bin/view/Challenge/ThirdProvenanceChallengehttp://www.science.uva.nl/~gvlam/wsvlam

PC3 workflow

PC3 workflow components (wsvlam modules)

• M1: ReadCSVFileReady– Parameter: JobID [String]– Outputs: Stream objects [CSV_File_ENTRY]

• M2: MatchCSVFileTable– Inputs– Outputs: DB_ENTRY

• M3:ReadCVSFileColumnName– Inputs: Stream objects [CSV_File_ENTRY]– Outputs: Stream objects [CVS_FILE_ENTRY_ColumnName]

• M4-M8:– Inputs: DB_ENTRY, Stream objects [CVS_FILE_ColumnName]– Outputs: Stream [true, false]

• M9: compact DB– Inputs: DB_ENTRY, Stream Strings [true/false]– Outputs:

M1: ReadCSVFileReady

M1: ReadCSVFileReady

M2: MatchCSVFileTable

M2: MatchCSVFileTable

M3:ReadCVSFileColumnName

M3:ReadCVSFileColumnName

M4-M8M4-M8

M9:CompoactDB

M9:CompoactDB

Stream- CSVFileEntry

DB-ENTRY Stream ofTrue, false

DB-ENTRY

PC3 workflow (Streaming mode)

•M1 –M9 are running concurrently

•The workflow components becomes active when data is available at their inputport(s)

•The workflow terminates when M1 has finished processing all the data

CP3 workflow farming mode (1)

M1: ReadCSVFileReady

M1: ReadCSVFileReady

M2: MatchCSVFileTable

M2: MatchCSVFileTable

M3:ReadCVSFileColumnName

M3:ReadCVSFileColumnName M4-M8M4-M8

M9:CompoactDB

M9:CompoactDB

List- CSVFileEntry

DB-ENTRY

CSVFileEntry-columnnames

Boolean

FileWriterFileWriter

FileWriterFileWriter

FileWriterFileWriter

FileReaderFileReader FileReaderFileReader

FileReaderFileReader

FileReaderFileReader

Pre-Load workflowPre-Load workflow

Load workflow (Farmed)Load workflow (Farmed)

Post-Load workflowPost-Load workflow • The 3 phases of the workflow are executed sequentially/independently

• intermediate data is saved into files

CP3 workflow farming mode (2)

Pre-Load workflowPre-Load workflow

Load workflowLoad workflow

Post-Load workflowPost-Load workflow

Continue(run-workflow)

Continue(run-workflow)

Workflow description

Continue(run-workflow)

Continue(run-workflow)

Workflow description

Data locations

• The 3 phases of the workflow are executed independently

• CSV_FILE_ENTRY are processed in parallel

PC3 workflow sequential Mode

Pre-Load workflowPre-Load workflow

Load workflowLoad workflow

Post-Load workflowPost-Load workflow

Continue(run-workflow)

Continue(run-workflow)

Workflow description

Continue(run-workflow)

Continue(run-workflow)

Workflow description

Data locations

Next - CSVFileEntry Next - CSVFileEntry

CSV_FILE_ENTRY

1/0

• The 3 phases of the workflow are executed sequentially

• CSV_FILE_ENTRY are processed sequentially

Computing Nodes• Workflow components • Grid Middleware GT4

WSRF Services - WS-VLAM engine - workflow component repository

VLe Studio• WS-VLAM composer• VBrowser• Semantic tools SAW: Semantic Annotation for Workflow CLAMP: Connecting LAnguage for Modules & Programs HAMMER: Hybrid-bAsed MatchMaker for e-Science Resources

DAS-3

Matrix-Sara

pc-vlab19.uva

PoC environment

RP environment

Testbed

SRB

Testbed

User start VL-e Studio Search for services for the workflow Upload in the workflow composer Compose the workflow Search for input data set in the SRB Set the workflow parameters Execute the workflow Monitor the execution

DAS-3

Matrix-Sara

pc-vlab19.uva

SRB

PoC environment

RP environment

http://www.vl-e.nl/