PegasusA Framework for Workflow

Planning on the Grid

Ewa Deelman

USC Information Sciences Institute

Pegasus Acknowledgments:Carl Kesselman, Gaurang Mehta, Mei-Hui Su, Gurmeet Singh, Karan Vahi

pegasus.isi.edu Ewa Deelman

Pegasus

Flexible framework, maps abstract workflows onto the Grid

Possess well-defined APIs and clients for:– Information gathering

> Resource information> Replica query mechanism> Transformation catalog query mechanism

– Resource selection> Compute site selection> Replica selection

– Data transfer mechanism Can support a variety of workflow executors

pegasus.isi.edu Ewa Deelman

Pegasus May reduce the workflow based on

available data products Augments the workflow with data stage-in

and data stage-out Augments the workflow with data

registration

Job e

Job g

Job h

Job d

Job aJob c

Job f

Job i

Job b

KEYThe original nodePull transfer nodeRegistration nodePush transfer nodeInter-pool transfer node

Job e

Job g

Job h

Job d

Job a

Job c

Job f

Job i

Job b

pegasus.isi.edu Ewa Deelman

Pegasus Components

PEGASUS ENGINE

CPlanner (gencdag)

Rls-client Tc-clientGenpoolconfig

client

Data Transfer Mechanism

Gridlabtransfer

Transfer2

Multiple Transfer

Globus-url-copy

Stork

Transformation Catalog

Mechanism(TC)

DatabaseFile

Resource Information

Catalog

MDS File

Submit Writer

CondorStork Writer

GridLab GRMS

Pegasus command line clients

RoundRobin

Site Selector

Min-Min

Max-MinProphesy

Random

Grasp

RLS

Replica Query and Registration

Mechanism

Replica Selection

Existing Interfaces

Research Implementations

Production Implementations

Interfaces in development

RLS

pegasus.isi.edu Ewa Deelman

Original Pegasus configuration

Original Abstract Workflow

Original Pegasus Configuration

Pegasus(Abstract Workflow)

DAGMan(CW))

Co

ncre

te W

orfklo

w

Workflow Execution

Simple scheduling: random or round robin using well-defined scheduling interfaces.

pegasus.isi.edu Ewa Deelman

Deferred Planning through Partitioning

PW A

PW B

PW C

A Particular PartitioningNew Abstract

Workflow

A variety of planning algorithms can be implemented

pegasus.isi.edu Ewa Deelman

Mega DAG is created by Pegasus and then submitted to DAGMan

DAGMan(Su(A))

Pegasus(A) = Su(A)

Pegasus(X): Pegasus generated the concrete workflow and the submit

files for Partition X -- Su(X)

DAGMan(Su(X): DAGMan executes the concrete

workflow for X DAGMan(Su(B))

Pegasus(B) = Su(B)

DAGMan(Su(C))

Pegasus(C) = Su(C)

pegasus.isi.edu Ewa Deelman

Re-planning capabilities

DAGMan(Su(A))

Pegasus(A) = Su(A)

Pegasus(X): Pegasus generated the concrete

workflow and the submit files for Partition X --- Su(X)

DAGMan(Su(X): DAGMan executes the concrete workflow for Partition X DAGMan(Su(B))

Pegasus(B) = Su(B)

DAGMan(Su(C))

Pegasus(C) = Su(C)

Retry Y times

Pegasus’ Log files record sites considered

Retry Y times

Retry Y times

pegasus.isi.edu Ewa Deelman

Complex Replanning for Free (almost)

DAGMan(Su(A))

Pegasus(A) = Su(A)

Retry Y times

A

CB

D

f1

f2 f3

f4

f1

Original abstract workflow partition

Move f2 to R1

Move f3 to R1

Move f4 to

Output location

Execute D at R1

Pegasus mapping, f2 and f3 were found in

a replica catalog

Workflow submitted to DAGMan

Move f2 to R1

Move f3 to R1

Move f4 to

Output location

Execute D at R1

failure

Pegasus is called again with original

partition

A

CB

D

f1

f2

f4

New mapping, here assuming R1 was

picked again

Move f1 to R2

Move f3 to R1

Move f4 to

Output location

Execute D at R1

Execute C at R2

f1

f2 f3

pegasus.isi.edu Ewa Deelman

Optimizations

If the workflow being refined by Pegasus consists of only 1 node– Create a condor submit node rather than a

dagman node

– This optimization can leverage Euryale’s super-node writing component

pegasus.isi.edu Ewa Deelman

Planning & Scheduling Granularity Partitioning

– Allows to set the granularity of planning ahead Node aggregation

– Allows to combine nodes in the workflow and schedule them as one unit (minimizes the scheduling overheads)

– May reduce the overheads of making scheduling and planning decisions

Related but separate concepts– Small jobs

> High-level of node aggregation

> Large partitions

– Very dynamic system > Small partitions

pegasus.isi.edu Ewa Deelman

Montage Montage (NASA and NVO)

– Deliver science-grade custom mosaics on demand

– Produce mosaics from a wide range of data sources (possibly in different spectra)

– User-specified parameters of projection, coordinates, size, rotation and spatial sampling.

Bruce Berriman, John Good, Anastasia Laity, Caltech/IPAC

Joseph C. Jacob, Daniel S. Katz, JPL

Doing large: 6 and 10 degree dags (for the m16 cluster).

The 6 degree runs had about 13,000 compute jobs and the 10 degree run had about 40,000 compute jobs

Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.

pegasus.isi.edu Ewa Deelman

Montage Workflow

1 2 311 22 33

mProject 1mProject 1 mProject 2mProject 2 mProject 3mProject 3

mDiff 1 2mDiff 1 2 mDiff 2 3mDiff 2 3

D12D23

mFitplane D12mFitplane D12 mFitplane D23mFitplane D23

mBgModelmBgModel

ax + by + c = 0 dx + ey + f = 0

a1 x + b1 y + c1 = 0

a2 x + b2 y + c2 = 0

a3 x + b3 y + c3 = 0

mBackground 1mBackground 1 mBackground 2mBackground 2 mBackground 3mBackground 3

11 22 33

mAddmAdd

Final MosaicFinal Mosaic

1 2 311 22 33

Data Stage in nodes

Montage compute nodes

Data stage out nodes

Inter pool transfer nodes

pegasus.isi.edu Ewa Deelman

Future work

Staging in executables on demand Expanding the scheduling plug-ins Investigating various partitioning

approaches Investigating reliability across partitions

pegasus.isi.edu Ewa Deelman

Non-GriPhyN applications using Pegasus Galaxy Morphology

(National Virtual Observatory)– Investigates the dynamical

state of galaxy clusters– Explores galaxy evolution

inside the context of large-scale structure.

– Uses galaxy morphologies as a probe of the star formation and stellar distribution history of the galaxies inside the clusters.

– Data intensive computations involving hundreds of galaxies in a cluster

The x-ray emission is shown in blue, and the optical mission is in red. The colored dots are located at the positions of the galaxies within the cluster; the dot color represents the value of the asymmetry index. Blue dots represent the most asymmetric galaxies and are scattered throughout the image, while orange are the most symmetric, indicative of elliptical galaxies, are concentrated more toward the center.

pegasus.isi.edu Ewa Deelman

BLAST: set of sequence comparison algorithms that are used to

search sequence databases for optimal local alignments to a query

Lead by Veronika Nefedova (ANL) as part of the PACI Data Quest Expedition program

2 major runs were performed using Chimera and Pegasus:

1) 60 genomes (4,000 sequences each), In 24 hours processed Genomes selected

from DOE-sponsored sequencing projects67 CPU-days of processing time

delivered~ 10,000 Grid jobs>200,000 BLAST executions50 GB of data generated

2) 450 genomes processed

Speedup of 5-20 times were achieved because the compute nodes we used efficiently by keeping the submission of the jobs to the compute cluster constant.

pegasus.isi.edu Ewa Deelman

Biology Applications (cont’d) Tomography (NIH-funded project) Derivation of 3D structure from a

series of 2D electron microscopic projection images,

Reconstruction and detailed structural analysis– complex structures like synapses– large structures like dendritic

spines. Acquisition and generation of huge

amounts of data Large amount of state-of-the-art

image processing required to segment structures from extraneous background. Dendrite structure to be rendered by

Tomography

Work performed by Mei Hui-Su with Mark Ellisman, Steve Peltier, Abel Lin, Thomas Molina (SDSC)

pegasus.isi.edu Ewa Deelman

Southern California Earthquake Center

Grid-BasedData Selector

CompositionalAnalysis Tool

(CAT)

DAXGenerator

Pegasus

CondorDAGMAN

PathwayComposition

Tool

GRID

host1host2

Data

Data

CAT KnowledgeBase

SCEC DatatypeDB

MetadataCatalog Service

ReplicaLocationService

Dax

Dag

Rsl

HAZARD MAP

The SCEC/IT project, funded by (NSF), is developing a new framework for physics-based simulations for seismic hazard analysis building on several information technology areas, including knowledge representation and reasoning, knowledge acquisition, grid computing, and digital libraries.

People involved: Vipin Gupta, Phil Maechling (USC)