Towards Jungle Computing with

Ibis/Constellation

Jason Maassen, Niels Drost

Henri Bal, Frank Seinstra

Department of Computer Science

VU University, Amsterdam, The Netherlands

Introduction

● HPC is entering many domains

● Not just: physics / chemistry / climate modelling

● Also: semantic web / medical / multimedia analysis /

neuroinformatics / remote sensing / astronomy / ...

● HPC is becoming more complex

● Not just large SMP or clusters, instead:

● Clusters of SMPs / Grids / Clouds / Supers / ...

● Heterogenous machines using GPU / Cell / FPGA

● “It‟s a jungle out there“

3DAPAS Workshop 2011 2

Example DomainComputational Astrophysics (amusecode.org)

Jungle Computing

● Worst case computing ... as required by users

● Arbitrary combination of distributed, hierarchical,

and heterogenous computing

3DAPAS Workshop 2011 4

Many Task ComputingAccording to Raicu, Foster, et al [SC‟08]

“High-performance computations comprising multiple

distinct activities, coupled via file system operations or

message passing. Tasks may be small or large,

uni-processor or multi-processor, compute-intensive or

data-intensive. The set of tasks may be static or

dynamic, homogeneous or heterogeneous, loosely

coupled or tightly coupled. The aggregate number of

tasks, quantity of computing, and volumes of data may

be extremely large.”

3DAPAS Workshop 2011 5

● Applications are dynamic and heterogeneous

workflows / DAGs of activities

MTC in the Jungle

● MTC has advantages for Jungle Computing

● Many distinct activities

● Can be implemented independently using the tools

and targeted to the HPC architecture, that best suit

them

● Reduced programming complexity

● Complete applications are constructed using

sequences and combinations of activities

3DAPAS Workshop 2011 6

Constellation

● MTC system for

Jungle Computing

● Model based on:

activities (tasks)

executors (resources)

contexts (matchmaking)

events (communication)

3DAPAS Workshop 2011 7

Constellation ModelApplication

● Application: set of activities

● Distinct tasks

● Size and complexity may vary

● Targeted at specific HPC platform

● (Loosly) Coupled using events

● Often wrapper around existing code

● Similar to workflow or DAG of tasks

● Dynamic and unlimited in size

3DAPAS Workshop 2011 8

Constellation ModelHardware

● Hardware: set of executors

● Capable of running activities

● May represent anything from a single

core to an entire cluster, a GPU, etc.

● May be application specific

● Provides an application specific

heterogeneous resource pool

3DAPAS Workshop 2011 9

Constellation ModelContext

● Both activities and executors

are tagged with a context

● Application defined label (+ rank)

● Used to defines relationship between

activites and executors, e.g.:

● Data dependencies, hardware

requirements, ...

● May combine contexts

● Executors may have preference

for label or rank

3DAPAS WorkShop 2011 10

Constellation ModelMatchmaking

● RTS performs load-balancing

and match-making

● Ensures activities are forwarded to

a suitable executor

● Tries to keep all executors busy

● Uses context-aware work-stealing

● RTS also performs event routing

● Based on unique activity identifier

ComplexHPC Spring School 2011 11

Constellation API

3DAPAS Workshop 2011 12

Constellation API

3DAPAS Workshop 2011 13

DACH 2008Data Challenge in conjunction with IEEE Cluster/Grid 2008

● Supernova detection

● Analyse 1052 image pairs on 11 clusters (Intrigger)

● „Sequential‟ executable provided

3DAPAS Workshop 2011 14

DACH 2008Problem

● Main problems:

● Data distribution

● Heterogeneity of

work and hardware

● Load balancing

3DAPAS Workshop 2011 15

DACH 2008Workflow

● Winning approach in 2008:

● Parallelize workflow to improve hardware utilization

● Create hierarchical master worker framework

● Scheduling heuristics using data location and size

3DAPAS Workshop 2011 16

Constellation VersionOption 1: Monolythic

● Wrap entire application in

a single activity

● One activity per image pair

● Wrap each machine in

one executor

● Multiple cores per executor

● Use context to influence

order and placement of

each of activities

3DAPAS Workshop 2011 17

Evaluation

● Intrigger not available

● Instead we use DAS3+DAS4

● 5+6 clusters in the Netherlands

● Mix of 2/4/8/12/48 core machines

● Various types of GPUs

● Three Scenarios

● Data locality

● (Executor granularity)

● Heterogeneous processing

3DAPAS Workshop 2011 18

Scenario 1Data Locality

● Data distributed over 4 clusters of DAS3 + DAS4

● Use context to express data locality and preferred

processing order

● Adapt context to tune application

● No change in application

3DAPAS Workshop 2011 19

Scenario 1Results

3DAPAS Workshop 2011 20

Activity Executor Effect

“any” “any” Random

order

“any”,50 “any”,

biggest

Sorted

by size

“VU3”,”VU4”,50 “VU3”,

biggest

Local only

Sorted

by size

“VU3”,”VU4”,

”any”,50

“VU3”,

“any”,

biggest

Preference

for local

Fallback

to any,

Sorted

by size

Constellation VersionOption 2: Workflow

● Wrap each stage in activity

● Wrap each core executor

● Use context to influence

order and placement of

each of the jobs

3DAPAS Workshop 2011 21

Scenario 3:Heterogeneous System

● 18 node GPU cluster● 8 cores + 1 GPU per node

● Activity: single task

● Executor:

1 core (top)

1 core or GPU (bottom)

● Replaced activity 7.2

with GPU version. ● Label activities and

executors accordingly

● Significant performance

gain.

ComplexHPC Spring School 2011 22

Conclusions

● We think Jungle Computing is a neccesity for

some application areas.

● Constellation offers a suitable model (MTC) to

create such applications.

● Initial experiments show that Constellation works

well for a wide range of hardware configurations

● Easy to reconfigure applications to match resources

● Allows integration of specialized accellerator codes

● Suitable basis for a Jungle Computing model

3DAPAS Workshop 2011 23

Future Work

● Application development

● AMUSE

● Remote Sensing

● Climate modelling

● Platform improvements

● Easier integration of existing codes

● Smart/automatic deployment/tuning of executors

● Improve data handling

● Better monitoring

3DAPAS Workshop 2011 24

Questions ?

jason@cs.vu.nl

www.cs.vu.nl/ibis

3DAPAS Workshop 2011 25

Scenario 2Executor Granularity

● 30 largest images only

● Single 48 core machine● Activity: entire application (a-c)

single task (d)

● Executor: [n]-cores

● No change in application

for experiment (a-c)● Only change executor config.

● Completely ported

application in (d)● Significant performance gain!

3DAPAS Workshop 2011 26