+ All Categories
Home > Documents > Funding and Support (2003 - IIT-Computer...

Funding and Support (2003 - IIT-Computer...

Date post: 25-Mar-2018
Category:
Upload: lykiet
View: 213 times
Download: 1 times
Share this document with a friend
49
Transcript
Page 1: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU
Page 2: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Funding and Support (2003 – 2010)

– CRA/NSF CIFellows Program

– University of Chicago

• Computer Science

• Computational Institute

– Argonne National Laboratory

• Math and Computer Science Division

• Argonne Leadership Computing Facility

– NASA: Ames Research Center, GSRP Program

• Over 60 Collaborators

– Ian Foster (UC/ANL), Rick Stevens (UC/ANL), Alex Szalay (JHU),

Jim Gray (MSR), Pete Beckman (ANL), Jerry Yan (NASA ARC),

Mike Wilde (UC/ANL), Douglas Thain (ND), Amitabh Chaudhary (ND),

Yong Zhao (MS), Zhao Zhang (UC), Catalin Dumitrescu (FNAL),

Matei Ripeanu (UBC), Alok Choudhary (NU), and many more…

Ian FosterAlok Choudhary

2Many-Task Computing on Grids, Clouds, and Supercomputers

Page 3: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

3Many-Task Computing on Grids, Clouds, and Supercomputers

Page 4: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

SDSC

TACC

UC/ANL

NCSA

ORNL

PU

IU

PSC

NCAR

2007

(504

TF)

2008

(~1PF

)Tennesse

eLONI/

LSU

• Clouds are a large-scale distributed computing paradigm driven by: 1. economies of scale

2. virtualization

3. dynamically-scalable resources

4. delivered on demand over the Internet

• Grids tend to be composed of multiple clusters, and are typically

loosely coupled, heterogeneous, and geographically dispersed

• Supercomputers are highly-tuned computer clusters using

commodity processors combined with custom network interconnects

and customized operating system

• Computer clusters using commodity processors, network

interconnects, and operating systems.

4Many-Task Computing on Grids, Clouds, and Supercomputers

Page 5: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• HPC: High-Performance Computing

– Synonymous with supercomputing

– Tightly-coupled applications

– Implemented using Message Passing Interface (MPI), needs low latency networks

– Measured in FLOPS

• HTC: High-Throughput Computing

– Typically applied in clusters and grids

– Loosely-coupled applications with sequential jobs

– Measured in operations per month or years

• MTC: Many-Task Computing

– Bridge the gap between HPC and HTC

– Applied in clusters, grids, and supercomputers

– Loosely coupled apps with HPC orientations

– Many activities coupled by file system ops

– Many resources over short time periods

Number of Tasks

Input Data Size

Hi

Med

Low

1 1K 1M

HPC(Heroic

MPI Tasks)

HTC/MTC(Many Loosely Coupled Tasks)

MapReduce/MTC(Data Analysis,

Mining)

MTC(Big Data and Many Tasks)

[MTAGS08] “Many-Task Computing for Grids and Supercomputers”

5Many-Task Computing on Grids, Clouds, and Supercomputers

Page 6: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

6Many-Task Computing on Grids, Clouds, and Supercomputers

Page 7: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0

50

100

150

200

250

300

2004 2006 2008 2010 2012 2014 2016 2018

Nu

mb

er

of

Co

res

0

10

20

30

40

50

60

70

80

90

100

Man

ufa

ctu

rin

g P

rocess

Number of Cores

Processing

Pat Helland, Microsoft, The Irresistible Forces Meet the Movable

Objects, November 9th, 2007

Top500 Projected Development,

http://www.top500.org/lists/2008/11/performance_development

Page 8: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0.1

1

10

100

1000

2002 2009

MB

/s p

er P

roce

sso

r C

ore

Local Disk

Parallel File System

-99X -438X

• 2002:

– Local disk

• ANL/UC TG Site (70GB

SCSI)

– Parallel File System

• 2002: IBM Blue Gene/L

(GPFS, 1GB/s)

• 2009:

– Local disk

• PADS (RAID-0, 6 drives

750GB SATA)

– Parallel File System

• IBM Blue Gene/P (GPFS,

65GB/s)

8Many-Task Computing on Grids, Clouds, and Supercomputers

Page 9: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Segregated storage and compute

– NFS, GPFS, PVFS, Lustre

– Batch-scheduled systems: Clusters, Grids, and

Supercomputers

– Programming paradigm: HPC, MTC, and HTC

• Co-located storage and compute

– HDFS, GFS

– Data centers at Google, Yahoo, and others

– Programming paradigm: MapReduce

– Others from academia: Sector, MosaStore, Chirp

Network Link(s)

NAS

Network

Fabric

Compute

Resources

9Many-Task Computing on Grids, Clouds, and Supercomputers

Page 10: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Segregated storage and compute

– NFS, GPFS, PVFS, Lustre

– Batch-scheduled systems: Clusters, Grids, and

Supercomputers

– Programming paradigm: HPC, MTC, and HTC

• Co-located storage and compute

– HDFS, GFS

– Data centers at Google, Yahoo, and others

– Programming paradigm: MapReduce

– Others from academia: Sector, MosaStore, Chirp

Compute & Storage

Resources

Network

Fabric

10Many-Task Computing on Grids, Clouds, and Supercomputers

Page 11: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

What if we could combine the

scientific community’s existing

programming paradigms, but yet

still exploit the data locality that

naturally occurs in scientific

workloads?

Network Link(s)

NAS

Network

Fabric

Compute & Storage

Resources

11Many-Task Computing on Grids, Clouds, and Supercomputers

Page 12: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

12Many-Task Computing on Grids, Clouds, and Supercomputers

Page 13: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Streamlined task dispatching

• Dynamic resource provisioning

– Multi-level scheduling

– Resources are acquired/released in response to demand

• Data diffusion

– Data diffuses from archival storage to transient resources

– Resource “caching” allows faster responses to subsequent requests

– Co-locate data and computations to optimize performance

[HPDC09] “The Quest for Scalable Support of Data Intensive Workloads in Distributed Systems”

[DIDC09] “Towards Data Intensive Many-Task Computing”

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”

[UC07] “Harnessing Grid Resources with Data-Centric Task Farms”

[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”

[TG07] “Dynamic Resource Provisioning in Grid Environments”

13

Page 14: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Abstract model– Models the efficiency and speedup of entire workloads

– Captures techniques to support MTC• Streamlined task dispatching, dynamic resource provisioning, data

diffusion

– Lead to proof of O(NM) competitive caching

• Middleware to support MTC– Falkon: a fast a light-weight execution framework

• Reference Implementation of the abstract model

– Swift: A Parallel Programming System

[TPDS10] “Middleware Support for Many-Task Computing”, under preparation

[DIDC09] “Towards Data Intensive Many-Task Computing”

[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”14

Page 15: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Goal: enable the rapid and efficient execution of many independent jobs on large compute clusters

• Combines three components:– a streamlined task dispatcher

– resource provisioning through multi-level scheduling techniques

– data diffusion and data-aware scheduling to leverage the co-located computational and storage resources

• Integration into Swift to leverage many applications– Applications cover many domains: astronomy, astro-physics,

medicine, chemistry, economics, climate modeling, etc[SciDAC09] “Extreme-scale scripting: Opportunities for large task-parallel applications on petascale computers”

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

[Globus07] “Falkon: A Proposal for Project Globus Incubation”

[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”

[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”

Task Dispatcher

Data-Aware SchedulerPersistent Storage

Available Resources

(GRAM4)

Provisioned Resources

text

Executor

1

Wait Queue

Executor

i

Executor

n

Dynamic

Resource

Provisioning

User

15

Page 16: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Falkon is a real system

– Late 2005: Initial prototype, AstroPortal

– January 2007: Falkon v0

– November 2007: Globus incubator project v0.1

• http://dev.globus.org/wiki/Incubator/Falkon

– February 2009: Globus incubator project v0.9

• Implemented in Java (~20K lines of code) and C

(~1K lines of code)

– Open source: svn co https://svn.globus.org/repos/falkon

• Source code contributors (beside myself)

– Yong Zhao, Zhao Zhang, Ben Clifford, Mihael Hategan[Globus07] “Falkon: A Proposal for Project Globus Incubation”

[CLUSTER10] “Middleware Support for Many-Task Computing”

• Workload

• 160K CPUs

• 1M tasks

• 60 sec per task

• 2 CPU years in 453 sec

• Throughput: 2312 tasks/sec

• 85% efficiency

16Many-Task Computing on Grids, Clouds, and Supercomputers

Page 17: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

[TPDS09] “Middleware Support for Many-Task Computing”, under preparation

17Many-Task Computing on Grids, Clouds, and Supercomputers

Page 18: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

Provisioner

Dispatcher

1

Executor

1

Cobalt

ClientExecutor

256

Dispatcher

N

Executor

1

Executor

256

Login Nodes

(x10)

I/O Nodes

(x640)

Compute Nodes

(x40K)

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

18Many-Task Computing on Grids, Clouds, and Supercomputers

Page 19: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

High-speed local disk

Falkon

Slower distributed

storage

ZeptOS

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

19Many-Task Computing on Grids, Clouds, and Supercomputers

Page 20: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

text

Task Dispatcher

Data-Aware SchedulerPersistent Storage

Shared File System

Idle Resources

Provisioned Resources

[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”

• Resource acquired in response to demand

• Data diffuse from archival storage to newly acquired transient resources

• Resource “caching” allows faster responses to subsequent requests

• Resources are released when demand drops

• Optimizes performance by co-scheduling data and computations

• Decrease dependency of a shared/parallel file systems

• Critical to support data intensive MTC20Many-Task Computing on Grids, Clouds, and Supercomputers

Page 21: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• FA: first-available

– simple load balancing

• MCH: max-cache-hit

– maximize cache hits

• MCU: max-compute-util

– maximize processor utilization

• GCC: good-cache-compute

– maximize both cache hit and processor utilization at the same time

[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”

21Many-Task Computing on Grids, Clouds, and Supercomputers

Page 22: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

22Many-Task Computing on Grids, Clouds, and Supercomputers

Page 23: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

ANL/UC, Java 200 CPUs 1 service

ANL/UC, C 200 CPUs1 service

SiCortex, C 5760 CPUs

1 service

BlueGene/P, C 4096 CPUs

1 service

BlueGene/P, C163840 CPUs 640 services

604

2534

3186

1758

3071

Th

rou

gh

pu

t (t

asks

/sec)

Executor Implementation and Various Systems

System CommentsThroughput

(tasks/sec)

Condor (v6.7.2) - Production Dual Xeon 2.4GHz, 4GB 0.49

PBS (v2.1.8) - Production Dual Xeon 2.4GHz, 4GB 0.45

Condor (v6.7.2) - Production Quad Xeon 3 GHz, 4GB 2

Condor (v6.8.2) - Production 0.42

Condor (v6.9.3) - Development 11

Condor-J2 - Experimental Quad Xeon 3 GHz, 4GB 22[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

23

Page 24: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

256 1024 4096 16384 65536 163840

Eff

icie

nc

y

Number of Processors

256 seconds128 seconds64 seconds32 seconds16 seconds8 seconds4 seconds2 seconds1 second

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Eff

icie

nc

y

Number of Processors

32 seconds16 seconds8 seconds4 seconds2 seconds1 second

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

24Many-Task Computing on Grids, Clouds, and Supercomputers

Page 25: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0

1

2

3

4

5

first-

available

without I/O

first-

available

with I/O

max-

compute-util

max-cache-

hit

good-

cache-

compute

CP

U T

ime p

er

Task (

ms

)

0

1000

2000

3000

4000

5000

Th

rou

gh

pu

t (t

asks

/sec)

Task SubmitNotification for Task AvailabilityTask Dispatch (data-aware scheduler)Task Results (data-aware scheduler)Notification for Task ResultsWS CommunicationThroughput (tasks/sec)

[DIDC09] “Towards Data Intensive Many-Task Computing”, under review

• 3GHz dual CPUs

• ANL/UC TG with 128 processors

• Scheduling window 2500 tasks

• Dataset

• 100K files

• 1 byte each

• Tasks

• Read 1 file

• Write 1 file

25Many-Task Computing on Grids, Clouds, and Supercomputers

Page 26: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Monotonically Increasing Workload

– Emphasizes increasing loads

• Sine-Wave Workload

– Emphasizes varying loads

• All-Pairs Workload

– Compare to best case model of active storage

• Image Stacking Workload (Astronomy)

– Evaluate data diffusion on a real large-scale data-intensive application from astronomy domain

[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”

[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”

[DIDC09] “Towards Data Intensive Many-Task Computing”26

Page 27: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

10

20

30

40

50

60

70

80

90

100

Ca

ch

e H

it/M

iss

%

No

des A

llo

ca

ted

Th

rou

gh

pu

t (G

b/s

)Q

ue

ue L

en

gth

(x1

K)

Time (sec)Cache Miss % Cache Hit Global % Cache Hit Local %Throughput (Gb/s) Demand (Gb/s) Wait Queue LengthNumber of Nodes

[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”

[DIDC09] “Towards Data Intensive Many-Task Computing”

27

Page 28: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

Throughput:

– Average: 14Gb/s vs 4Gb/s

– Peak: 81Gb/s vs. 6Gb/s

Response Time

– 3 sec vs 1569 sec 506X

80

6

12

7381 81

21

46

0

2

4

6

8

10

12

14

16

18

20

Ideal FA GCC 1GB

GCC 1.5GB

GCC 2GB

GCC 4GB

MCH 4GB

MCU 4GB

Th

rou

gh

pu

t (G

b/s

)

Local Worker Caches (Gb/s)

Remote Worker Caches (Gb/s)

GPFS Throughput (Gb/s)

1569

1084

1143.4 3.1

230287

0

200

400

600

800

1000

1200

1400

1600

1800

FA GCC 1GB

GCC 1.5GB

GCC 2GB

GCC 4GB

MCH 4GB

MCU 4GB

Avera

ge R

esp

on

se T

ime (sec)

[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”

[DIDC09] “Towards Data Intensive Many-Task Computing”

28

Page 29: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• GPFS 5.7 hrs, ~8Gb/s, 1138 CPU hrs

• GCC+SRP 1.8 hrs, ~25Gb/s, 361 CPU hrs

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

10

20

30

40

50

60

70

80

90

100

Cach

e H

it/M

iss

No

de

s A

llo

ca

ted

Th

rou

gh

pu

t (G

b/s

)Q

ue

ue

Le

ng

th (x

1K

)

Time (sec)

Cache Hit Local % Cache Hit Global % Cache Miss %Demand (Gb/s) Throughput (Gb/s) Wait Queue LengthNumber of Nodes

j

[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”, under review

[DIDC09] “Towards Data Intensive Many-Task Computing”, under review

29

Page 30: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Pull vs. Push

– Data Diffusion

• Pulls task working set

• Incremental spanning

forest

– Active Storage:

• Pushes workload

working set to all nodes

• Static spanning tree

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

500x500200 CPUs

1 sec

500x500200 CPUs

0.1 sec

1000x10004096 CPUs

4 sec

1000x10005832 CPUs

4 sec

Eff

icie

nc

y

Experiment

Best Case (active storage)Falkon (data diffusion)Best Case (parallel file system)

[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”, under review

[DIDC09] “Towards Data Intensive Many-Task Computing”, under review

Experiment Approach

Local

Disk/Memory

(GB)

Network

(node-to-node)

(GB)

Shared

File

System

(GB)

Best Case

(active storage)6000 1536 12

Falkon

(data diffusion)6000 1698 34

Best Case

(active storage)6000 1536 12

Falkon

(data diffusion)6000 1528 62

Best Case

(active storage)24000 12288 24

Falkon

(data diffusion)24000 4676 384

Best Case

(active storage)24000 12288 24

Falkon

(data diffusion)24000 3867 906

500x500

200 CPUs

1 sec

500x500

200 CPUs

0.1 sec

1000x1000

4096 CPUs

4 sec

1000x1000

5832 CPUs

4 sec

Christopher Moretti, Douglas Thain,

University of Notre Dame

30

Page 31: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

31Many-Task Computing on Grids, Clouds, and Supercomputers

Page 32: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Wide range of analyses

– Testing, interactive analysis,

production runs

– Data mining

– Parameter studies[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”

[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”

Improvement:

up to 90% lower end-to-end run time

32

Page 33: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

B. Berriman, J. Good (Caltech)J. Jacob, D. Katz (JPL)

[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”

[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”

Improvement:

up to 57% lower end-to-end run time

Within 4% of MPI

33

Page 34: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Determination of free

energies in aqueous solution

– Antechamber – coordinates

– Charmm – solution

– Charmm - free energy

[NOVA08] “Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments”

Improvement:

up to 88% lower end-to-end run time

5X more scalable

34Many-Task Computing on Grids, Clouds, and Supercomputers

Page 35: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Classic benchmarks for MapReduce

– Word Count

– Sort

• Swift and Falkon performs similar or better than

Hadoop (on 32 processors)Sort

42

85

733

25

83

512

1

10

100

1000

10000

10MB 100MB 1000MB

Data Size

Tim

e (

sec

)Swift+Falkon

Hadoop

Word Count

221

11431795

863

46887860

1

10

100

1000

10000

75MB 350MB 703MB

Data Size

Tim

e (

se

c)

Swift+PBS

Hadoop

35Many-Task Computing on Grids, Clouds, and Supercomputers

Page 36: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0

200000

400000

600000

800000

1000000

Th

rou

gh

pu

t (t

as

ks/s

ec

)

Ta

sk

s C

om

ple

ted

Nu

mb

er

of

Pro

ce

ss

ors

Time (sec)

ProcessorsActive TasksTasks CompletedThroughput (tasks/sec)

• CPU Cores: 130816

• Tasks: 1048576

• Elapsed time: 2483 secs

• CPU Years: 9.3

Speedup: 115168X (ideal 130816)

Efficiency: 88%

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

36Many-Task Computing on Grids, Clouds, and Supercomputers

Page 37: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

start

report

DOCK6

Receptor

(1 per protein:

defines pocket

to bind to)

ZINC3-D

structures

ligands complexes

NAB script

parameters

(defines flexible

residues,

#MDsteps)

Amber Score:

1. AmberizeLigand

3. AmberizeComplex

5. RunNABScript

end

BuildNABScript

NAB

Script

NAB

Script

Template

Amber prep:

2. AmberizeReceptor

4. perl: gen nabscript

FRED

Receptor

(1 per protein:

defines pocket

to bind to)

Manually prep

DOCK6 rec file

Manually prep

FRED rec file

1 protein(1MB)

6 GB2M

structures(6 GB)

DOCK6FRED~4M x 60s x 1 cpu

~60K cpu-hrs

Amber~10K x 20m x 1 cpu

~3K cpu-hrs

Select best ~500

~500 x 10hr x 100 cpu

~500K cpu-hrsGCMC

PDBprotein

descriptions

Select best ~5KSelect best ~5K

For 1 target:

4 million tasks

500,000 cpu-hrs

(50 cpu-years)[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

37Many-Task Computing on Grids, Clouds, and Supercomputers

Page 38: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

CPU cores: 118784

Tasks: 934803

Elapsed time: 2.01 hours

Compute time: 21.43 CPU years

Average task time: 667 sec

Relative Efficiency: 99.7%

(from 16 to 32 racks)

Utilization:

• Sustained: 99.6%

• Overall: 78.3%

[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”

38Many-Task Computing on Grids, Clouds, and Supercomputers

Page 39: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Purpose

– On-demand “stacks” of

random locations within

~10TB dataset

• Challenge

– Processing Costs:

• O(100ms) per object

– Data Intensive:

• 40MB:1sec

– Rapid access to 10-10K

“random” files

– Time-varying load

AP SloanData

+

+

+

+

+

+

=

+

Locality Number of Objects Number of Files

1 111700 111700

1.38 154345 111699

2 97999 49000

3 88857 29620

4 76575 19145

5 60590 12120

10 46480 4650

20 40460 2025

30 23695 790[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”

[TG06] “AstroPortal: A Science Gateway for Large-scale Astronomy Data Analysis”

39

Page 40: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• AstroPortal

– Makes it really easy for astronomers to create

stackings of objects from the Sloan Digital Sky

Servey (SDSS) dataset• Throughput

– 10X higher than GPFS

• Reduced load– 1/10 of the original GPFS load

• Increased scalability• 8X

40Many-Task Computing on Grids, Clouds, and Supercomputers

Page 41: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

41Many-Task Computing on Grids, Clouds, and Supercomputers

Page 42: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• There is more to HPC than tightly coupled MPI, and more to HTC than embarrassingly parallel long jobs

– MTC: Many-Task Computing

– Addressed real challenges in resource management in large scale distributed systems to enable MTC

– Covered many domains (via Swift and Falkon): astronomy, medicine, chemistry, molecular dynamics, economic modelling, and data analytics

• Identified that data locality is critical at large-scale data diffusion

– Integrated streamlined task dispatching with data aware scheduling

– Heuristics to maximize real world performance

– Suitable for varying, data-intensive workloads

– Proof of O(NM) Competitive Caching

42Many-Task Computing on Grids, Clouds, and Supercomputers

Page 43: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Publications

– 51 articles and proposals

– 66 formal presentations

– 854 citations H-Index 14

• Activities for broader community engagement

– ScienceCloud: ACM Workshop on Scientific Cloud Computing, 2010

– TPDS: IEEE Transactions on Parallel and Distributed Systems, Special Issue on

Many-Task Computing, 2010

– MTAGS: ACM Workshop on Many-Task Computing on Grids and

Supercomputers, 2009

– MTAGS: IEEE Workshop on Many-Task Computing on Grids and

Supercomputers, 2008

– BegaJob: Bird of Feather Session – “How to Run One Million Jobs”, at

IEEE/ACM SC08, 2008

– 53 more activities as a program committee member and/or reviewer43Many-Task Computing on Grids, Clouds, and Supercomputers

Page 44: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Introduction to Programming (Spring 2010)– Northwestern University, Instructor

• Hot Topics in Distributed Systems: Data-Intensive Computing (Winter

2010)– Northwestern University (EECS495), Instructor

– http://www.eecs.northwestern.edu/~iraicu/teaching/EECS495-DIC/index.html

• Networks and Distributed Systems (2006)– University of Chicago (CMSC 33300), Lead TA

– http://dsl.cs.uchicago.edu/Courses/CMSC33300/index.html

• Grid Computing (2005)– University of Chicago (CMSC 33340), TA

– http://www.mcs.anl.gov/~itf/CMSC23340/

• Introduction to Networking (2003)– Purdue University, Lab TA

• Data Structures and Algorithm Analysis in C++ (2002)– University of Michigan, Adjunct Assistant Professor

• 9 more courses at University of Chicago and

Wayne State University– Advanced Network Design, Introduction to

Programming for the World Wide Web I, Introduction to

Computer Science 1 & 2, Introduction to Computer

Systems , Fundamentals of Computer Programming I

in Scheme, Problem Solving & Programming in C++,

Data Structures & Abstraction in C++

44Many-Task Computing on Grids, Clouds, and Supercomputers

Page 45: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Embarrassingly Happily parallel apps are trivial to run

– Logistical problems can be tremendous

• Loosely coupled apps do not require “supercomputers”

– Total computational requirements can be enormous

– Individual tasks may be tightly coupled

– Workloads frequently involve large amounts of I/O

– Make use of idle resources from “supercomputers” via backfilling

– Costs to run “supercomputers” per FLOP is among the best

• Loosely coupled apps do not require specialized system software

– Their requirements on the job submission and storage systems can be extremely large

• Shared/parallel file systems are good for all applications

– They don’t scale proportionally with the compute resources

– Data intensive applications don’t perform and scale well

– Growing compute/storage gap

“Impossible only means that you

haven't found the solution yet.”Anonymous

45Many-Task Computing on Grids, Clouds, and Supercomputers

Page 46: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Defining Many-Task Computing

• Motivation: Scalability Challenges

• Novel Resource Management Techniques

• Performance Evaluation: Micro-benchmarks

• Performance Evaluation: Applications

• Contributions

• Future Work

46Many-Task Computing on Grids, Clouds, and Supercomputers

Page 47: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Many-Task Computing and Data Intensive Computing

– Develop theoretical and practical aspects of building efficient and

scalable support for MTC

– Build a new distributed data-aware execution fabric that will support

HPC, MTC, and HTC

– Support interactive HPC applications

• Cloud Computing

– Enable scientific computing on clouds (e.g. Magellan project)

– Enable HPC through novel networking

• Many-Core Computing

– Apply data-aware scheduling of jobs from distributed systems

OS scheduling of threads

– Parallel programming models

automatic parallelization using DAG-based data-flow techniques47Many-Task Computing on Grids, Clouds, and Supercomputers

Page 48: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• Maintain and expand collaborations– Government Labs: ANL, FNAL, NASA, LBNL,

– Academia: UC, CI, UIC, DePaul, NU, ND, Purdue, UIUC, IU, UWM,

WSU, LSU, UNC, USF, UBC, DU, MU

– Industry: Accenture, IBM, MS, Google, Yahoo, Amazon, Intel

• Interdisciplinary research– Bring HEC to the science domain

– Many applicable domains: astronomy, astrophysics, economic modeling,

pharmaceuticals, chemistry, bioinformatics, neuroscience, data analytics,

data mining, biometrics, molecular docking, structural equation modeling,

posttranslational protein modification, climate modeling

• Combining Education with Research– NSF CDI, NSF REU, NSF CPATH

• Apply for funding from NSF, DOE, NIH, and industry– DOE INCITE, NSF PetaApps, NSF HECURA, NIH R01

48

Page 49: Funding and Support (2003 - IIT-Computer Scienceiraicu/research/presentations/2010_MTC_IIT_1-22...• Funding and Support (2003 –2010) ... (504 TF) 2008 (~1PF Tennesse) LONI/ e LSU

• More information: http://www.eecs.northwestern.edu/~iraicu/

• Related Projects: – Falkon: http://dev.globus.org/wiki/Incubator/Falkon

– Swift: http://www.ci.uchicago.edu/swift/index.php

• People contributing ideas, slides, source code, applications, results, etc– Ian Foster, Alex Szalay, Rick Stevens, Mike Wilde, Jim Gray, Catalin Dumitrescu,

Yong Zhao, Zhao Zhang, Gabriela Turcu, Ben Clifford, Mihael Hategan, Allan Espinosa, Kamil Iskra, Pete Beckman, Philip Little, Christopher Moretti, AmitabhChaudhary, Douglas Thain, Quan Pham, Atilla Balkir, Jing Tie, VeronikaNefedova, Sarah Kenny, Gregor von Laszewski, Tiberiu Stef-Praun, Julian Bunn, Andrew Binkowski , Glen Hocky, Donald Hanson, Matthew Cohoon, FangfangXia, Mike Kubal, Alok Choudhary…

• Funding:– NASA: Ames Research Center, Graduate Student Research Program

– DOE: Office of Advanced Scientific Computing Research

– NSF: TeragGrid and Computing Research Innovation Fellow Program

49Many-Task Computing on Grids, Clouds, and Supercomputers


Recommended