• Funding and Support (2003 – 2010)
– CRA/NSF CIFellows Program
– University of Chicago
• Computer Science
• Computational Institute
– Argonne National Laboratory
• Math and Computer Science Division
• Argonne Leadership Computing Facility
– NASA: Ames Research Center, GSRP Program
• Over 60 Collaborators
– Ian Foster (UC/ANL), Rick Stevens (UC/ANL), Alex Szalay (JHU),
Jim Gray (MSR), Pete Beckman (ANL), Jerry Yan (NASA ARC),
Mike Wilde (UC/ANL), Douglas Thain (ND), Amitabh Chaudhary (ND),
Yong Zhao (MS), Zhao Zhang (UC), Catalin Dumitrescu (FNAL),
Matei Ripeanu (UBC), Alok Choudhary (NU), and many more…
Ian FosterAlok Choudhary
2Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
3Many-Task Computing on Grids, Clouds, and Supercomputers
SDSC
TACC
UC/ANL
NCSA
ORNL
PU
IU
PSC
NCAR
2007
(504
TF)
2008
(~1PF
)Tennesse
eLONI/
LSU
• Clouds are a large-scale distributed computing paradigm driven by: 1. economies of scale
2. virtualization
3. dynamically-scalable resources
4. delivered on demand over the Internet
• Grids tend to be composed of multiple clusters, and are typically
loosely coupled, heterogeneous, and geographically dispersed
• Supercomputers are highly-tuned computer clusters using
commodity processors combined with custom network interconnects
and customized operating system
• Computer clusters using commodity processors, network
interconnects, and operating systems.
4Many-Task Computing on Grids, Clouds, and Supercomputers
• HPC: High-Performance Computing
– Synonymous with supercomputing
– Tightly-coupled applications
– Implemented using Message Passing Interface (MPI), needs low latency networks
– Measured in FLOPS
• HTC: High-Throughput Computing
– Typically applied in clusters and grids
– Loosely-coupled applications with sequential jobs
– Measured in operations per month or years
• MTC: Many-Task Computing
– Bridge the gap between HPC and HTC
– Applied in clusters, grids, and supercomputers
– Loosely coupled apps with HPC orientations
– Many activities coupled by file system ops
– Many resources over short time periods
Number of Tasks
Input Data Size
Hi
Med
Low
1 1K 1M
HPC(Heroic
MPI Tasks)
HTC/MTC(Many Loosely Coupled Tasks)
MapReduce/MTC(Data Analysis,
Mining)
MTC(Big Data and Many Tasks)
[MTAGS08] “Many-Task Computing for Grids and Supercomputers”
5Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
6Many-Task Computing on Grids, Clouds, and Supercomputers
0
50
100
150
200
250
300
2004 2006 2008 2010 2012 2014 2016 2018
Nu
mb
er
of
Co
res
0
10
20
30
40
50
60
70
80
90
100
Man
ufa
ctu
rin
g P
rocess
Number of Cores
Processing
Pat Helland, Microsoft, The Irresistible Forces Meet the Movable
Objects, November 9th, 2007
Top500 Projected Development,
http://www.top500.org/lists/2008/11/performance_development
0.1
1
10
100
1000
2002 2009
MB
/s p
er P
roce
sso
r C
ore
Local Disk
Parallel File System
-99X -438X
• 2002:
– Local disk
• ANL/UC TG Site (70GB
SCSI)
– Parallel File System
• 2002: IBM Blue Gene/L
(GPFS, 1GB/s)
• 2009:
– Local disk
• PADS (RAID-0, 6 drives
750GB SATA)
– Parallel File System
• IBM Blue Gene/P (GPFS,
65GB/s)
8Many-Task Computing on Grids, Clouds, and Supercomputers
• Segregated storage and compute
– NFS, GPFS, PVFS, Lustre
– Batch-scheduled systems: Clusters, Grids, and
Supercomputers
– Programming paradigm: HPC, MTC, and HTC
• Co-located storage and compute
– HDFS, GFS
– Data centers at Google, Yahoo, and others
– Programming paradigm: MapReduce
– Others from academia: Sector, MosaStore, Chirp
Network Link(s)
NAS
Network
Fabric
Compute
Resources
9Many-Task Computing on Grids, Clouds, and Supercomputers
• Segregated storage and compute
– NFS, GPFS, PVFS, Lustre
– Batch-scheduled systems: Clusters, Grids, and
Supercomputers
– Programming paradigm: HPC, MTC, and HTC
• Co-located storage and compute
– HDFS, GFS
– Data centers at Google, Yahoo, and others
– Programming paradigm: MapReduce
– Others from academia: Sector, MosaStore, Chirp
Compute & Storage
Resources
Network
Fabric
10Many-Task Computing on Grids, Clouds, and Supercomputers
What if we could combine the
scientific community’s existing
programming paradigms, but yet
still exploit the data locality that
naturally occurs in scientific
workloads?
Network Link(s)
NAS
Network
Fabric
Compute & Storage
Resources
11Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
12Many-Task Computing on Grids, Clouds, and Supercomputers
• Streamlined task dispatching
• Dynamic resource provisioning
– Multi-level scheduling
– Resources are acquired/released in response to demand
• Data diffusion
– Data diffuses from archival storage to transient resources
– Resource “caching” allows faster responses to subsequent requests
– Co-locate data and computations to optimize performance
[HPDC09] “The Quest for Scalable Support of Data Intensive Workloads in Distributed Systems”
[DIDC09] “Towards Data Intensive Many-Task Computing”
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”
[UC07] “Harnessing Grid Resources with Data-Centric Task Farms”
[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”
[TG07] “Dynamic Resource Provisioning in Grid Environments”
13
• Abstract model– Models the efficiency and speedup of entire workloads
– Captures techniques to support MTC• Streamlined task dispatching, dynamic resource provisioning, data
diffusion
– Lead to proof of O(NM) competitive caching
• Middleware to support MTC– Falkon: a fast a light-weight execution framework
• Reference Implementation of the abstract model
– Swift: A Parallel Programming System
[TPDS10] “Middleware Support for Many-Task Computing”, under preparation
[DIDC09] “Towards Data Intensive Many-Task Computing”
[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”14
• Goal: enable the rapid and efficient execution of many independent jobs on large compute clusters
• Combines three components:– a streamlined task dispatcher
– resource provisioning through multi-level scheduling techniques
– data diffusion and data-aware scheduling to leverage the co-located computational and storage resources
• Integration into Swift to leverage many applications– Applications cover many domains: astronomy, astro-physics,
medicine, chemistry, economics, climate modeling, etc[SciDAC09] “Extreme-scale scripting: Opportunities for large task-parallel applications on petascale computers”
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
[Globus07] “Falkon: A Proposal for Project Globus Incubation”
[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”
[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”
Task Dispatcher
Data-Aware SchedulerPersistent Storage
Available Resources
(GRAM4)
Provisioned Resources
text
Executor
1
Wait Queue
Executor
i
Executor
n
Dynamic
Resource
Provisioning
User
15
• Falkon is a real system
– Late 2005: Initial prototype, AstroPortal
– January 2007: Falkon v0
– November 2007: Globus incubator project v0.1
• http://dev.globus.org/wiki/Incubator/Falkon
– February 2009: Globus incubator project v0.9
• Implemented in Java (~20K lines of code) and C
(~1K lines of code)
– Open source: svn co https://svn.globus.org/repos/falkon
• Source code contributors (beside myself)
– Yong Zhao, Zhao Zhang, Ben Clifford, Mihael Hategan[Globus07] “Falkon: A Proposal for Project Globus Incubation”
[CLUSTER10] “Middleware Support for Many-Task Computing”
• Workload
• 160K CPUs
• 1M tasks
• 60 sec per task
• 2 CPU years in 453 sec
• Throughput: 2312 tasks/sec
• 85% efficiency
16Many-Task Computing on Grids, Clouds, and Supercomputers
[TPDS09] “Middleware Support for Many-Task Computing”, under preparation
17Many-Task Computing on Grids, Clouds, and Supercomputers
Provisioner
Dispatcher
1
Executor
1
Cobalt
ClientExecutor
256
Dispatcher
N
Executor
1
Executor
256
Login Nodes
(x10)
I/O Nodes
(x640)
Compute Nodes
(x40K)
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
18Many-Task Computing on Grids, Clouds, and Supercomputers
High-speed local disk
Falkon
Slower distributed
storage
ZeptOS
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
19Many-Task Computing on Grids, Clouds, and Supercomputers
text
Task Dispatcher
Data-Aware SchedulerPersistent Storage
Shared File System
Idle Resources
Provisioned Resources
[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”
• Resource acquired in response to demand
• Data diffuse from archival storage to newly acquired transient resources
• Resource “caching” allows faster responses to subsequent requests
• Resources are released when demand drops
• Optimizes performance by co-scheduling data and computations
• Decrease dependency of a shared/parallel file systems
• Critical to support data intensive MTC20Many-Task Computing on Grids, Clouds, and Supercomputers
• FA: first-available
– simple load balancing
• MCH: max-cache-hit
– maximize cache hits
• MCU: max-compute-util
– maximize processor utilization
• GCC: good-cache-compute
– maximize both cache hit and processor utilization at the same time
[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”
21Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
22Many-Task Computing on Grids, Clouds, and Supercomputers
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
ANL/UC, Java 200 CPUs 1 service
ANL/UC, C 200 CPUs1 service
SiCortex, C 5760 CPUs
1 service
BlueGene/P, C 4096 CPUs
1 service
BlueGene/P, C163840 CPUs 640 services
604
2534
3186
1758
3071
Th
rou
gh
pu
t (t
asks
/sec)
Executor Implementation and Various Systems
System CommentsThroughput
(tasks/sec)
Condor (v6.7.2) - Production Dual Xeon 2.4GHz, 4GB 0.49
PBS (v2.1.8) - Production Dual Xeon 2.4GHz, 4GB 0.45
Condor (v6.7.2) - Production Quad Xeon 3 GHz, 4GB 2
Condor (v6.8.2) - Production 0.42
Condor (v6.9.3) - Development 11
Condor-J2 - Experimental Quad Xeon 3 GHz, 4GB 22[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
23
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
256 1024 4096 16384 65536 163840
Eff
icie
nc
y
Number of Processors
256 seconds128 seconds64 seconds32 seconds16 seconds8 seconds4 seconds2 seconds1 second
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Eff
icie
nc
y
Number of Processors
32 seconds16 seconds8 seconds4 seconds2 seconds1 second
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
24Many-Task Computing on Grids, Clouds, and Supercomputers
0
1
2
3
4
5
first-
available
without I/O
first-
available
with I/O
max-
compute-util
max-cache-
hit
good-
cache-
compute
CP
U T
ime p
er
Task (
ms
)
0
1000
2000
3000
4000
5000
Th
rou
gh
pu
t (t
asks
/sec)
Task SubmitNotification for Task AvailabilityTask Dispatch (data-aware scheduler)Task Results (data-aware scheduler)Notification for Task ResultsWS CommunicationThroughput (tasks/sec)
[DIDC09] “Towards Data Intensive Many-Task Computing”, under review
• 3GHz dual CPUs
• ANL/UC TG with 128 processors
• Scheduling window 2500 tasks
• Dataset
• 100K files
• 1 byte each
• Tasks
• Read 1 file
• Write 1 file
25Many-Task Computing on Grids, Clouds, and Supercomputers
• Monotonically Increasing Workload
– Emphasizes increasing loads
• Sine-Wave Workload
– Emphasizes varying loads
• All-Pairs Workload
– Compare to best case model of active storage
• Image Stacking Workload (Astronomy)
– Evaluate data diffusion on a real large-scale data-intensive application from astronomy domain
[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”
[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”
[DIDC09] “Towards Data Intensive Many-Task Computing”26
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
10
20
30
40
50
60
70
80
90
100
Ca
ch
e H
it/M
iss
%
No
des A
llo
ca
ted
Th
rou
gh
pu
t (G
b/s
)Q
ue
ue L
en
gth
(x1
K)
Time (sec)Cache Miss % Cache Hit Global % Cache Hit Local %Throughput (Gb/s) Demand (Gb/s) Wait Queue LengthNumber of Nodes
[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”
[DIDC09] “Towards Data Intensive Many-Task Computing”
27
Throughput:
– Average: 14Gb/s vs 4Gb/s
– Peak: 81Gb/s vs. 6Gb/s
Response Time
– 3 sec vs 1569 sec 506X
80
6
12
7381 81
21
46
0
2
4
6
8
10
12
14
16
18
20
Ideal FA GCC 1GB
GCC 1.5GB
GCC 2GB
GCC 4GB
MCH 4GB
MCU 4GB
Th
rou
gh
pu
t (G
b/s
)
Local Worker Caches (Gb/s)
Remote Worker Caches (Gb/s)
GPFS Throughput (Gb/s)
1569
1084
1143.4 3.1
230287
0
200
400
600
800
1000
1200
1400
1600
1800
FA GCC 1GB
GCC 1.5GB
GCC 2GB
GCC 4GB
MCH 4GB
MCU 4GB
Avera
ge R
esp
on
se T
ime (sec)
[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”
[DIDC09] “Towards Data Intensive Many-Task Computing”
28
• GPFS 5.7 hrs, ~8Gb/s, 1138 CPU hrs
• GCC+SRP 1.8 hrs, ~25Gb/s, 361 CPU hrs
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
10
20
30
40
50
60
70
80
90
100
Cach
e H
it/M
iss
No
de
s A
llo
ca
ted
Th
rou
gh
pu
t (G
b/s
)Q
ue
ue
Le
ng
th (x
1K
)
Time (sec)
Cache Hit Local % Cache Hit Global % Cache Miss %Demand (Gb/s) Throughput (Gb/s) Wait Queue LengthNumber of Nodes
j
[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”, under review
[DIDC09] “Towards Data Intensive Many-Task Computing”, under review
29
• Pull vs. Push
– Data Diffusion
• Pulls task working set
• Incremental spanning
forest
– Active Storage:
• Pushes workload
working set to all nodes
• Static spanning tree
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
500x500200 CPUs
1 sec
500x500200 CPUs
0.1 sec
1000x10004096 CPUs
4 sec
1000x10005832 CPUs
4 sec
Eff
icie
nc
y
Experiment
Best Case (active storage)Falkon (data diffusion)Best Case (parallel file system)
[HPDC09] “The Quest for Scalable Support of Data Intensive Applications in Distributed Systems”, under review
[DIDC09] “Towards Data Intensive Many-Task Computing”, under review
Experiment Approach
Local
Disk/Memory
(GB)
Network
(node-to-node)
(GB)
Shared
File
System
(GB)
Best Case
(active storage)6000 1536 12
Falkon
(data diffusion)6000 1698 34
Best Case
(active storage)6000 1536 12
Falkon
(data diffusion)6000 1528 62
Best Case
(active storage)24000 12288 24
Falkon
(data diffusion)24000 4676 384
Best Case
(active storage)24000 12288 24
Falkon
(data diffusion)24000 3867 906
500x500
200 CPUs
1 sec
500x500
200 CPUs
0.1 sec
1000x1000
4096 CPUs
4 sec
1000x1000
5832 CPUs
4 sec
Christopher Moretti, Douglas Thain,
University of Notre Dame
30
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
31Many-Task Computing on Grids, Clouds, and Supercomputers
• Wide range of analyses
– Testing, interactive analysis,
production runs
– Data mining
– Parameter studies[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”
[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”
Improvement:
up to 90% lower end-to-end run time
32
B. Berriman, J. Good (Caltech)J. Jacob, D. Katz (JPL)
[SC07] “Falkon: a Fast and Light-weight tasK executiON framework”
[SWF07] “Swift: Fast, Reliable, Loosely Coupled Parallel Computation”
Improvement:
up to 57% lower end-to-end run time
Within 4% of MPI
33
• Determination of free
energies in aqueous solution
– Antechamber – coordinates
– Charmm – solution
– Charmm - free energy
[NOVA08] “Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments”
Improvement:
up to 88% lower end-to-end run time
5X more scalable
34Many-Task Computing on Grids, Clouds, and Supercomputers
• Classic benchmarks for MapReduce
– Word Count
– Sort
• Swift and Falkon performs similar or better than
Hadoop (on 32 processors)Sort
42
85
733
25
83
512
1
10
100
1000
10000
10MB 100MB 1000MB
Data Size
Tim
e (
sec
)Swift+Falkon
Hadoop
Word Count
221
11431795
863
46887860
1
10
100
1000
10000
75MB 350MB 703MB
Data Size
Tim
e (
se
c)
Swift+PBS
Hadoop
35Many-Task Computing on Grids, Clouds, and Supercomputers
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
0
200000
400000
600000
800000
1000000
Th
rou
gh
pu
t (t
as
ks/s
ec
)
Ta
sk
s C
om
ple
ted
Nu
mb
er
of
Pro
ce
ss
ors
Time (sec)
ProcessorsActive TasksTasks CompletedThroughput (tasks/sec)
• CPU Cores: 130816
• Tasks: 1048576
• Elapsed time: 2483 secs
• CPU Years: 9.3
Speedup: 115168X (ideal 130816)
Efficiency: 88%
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
36Many-Task Computing on Grids, Clouds, and Supercomputers
start
report
DOCK6
Receptor
(1 per protein:
defines pocket
to bind to)
ZINC3-D
structures
ligands complexes
NAB script
parameters
(defines flexible
residues,
#MDsteps)
Amber Score:
1. AmberizeLigand
3. AmberizeComplex
5. RunNABScript
end
BuildNABScript
NAB
Script
NAB
Script
Template
Amber prep:
2. AmberizeReceptor
4. perl: gen nabscript
FRED
Receptor
(1 per protein:
defines pocket
to bind to)
Manually prep
DOCK6 rec file
Manually prep
FRED rec file
1 protein(1MB)
6 GB2M
structures(6 GB)
DOCK6FRED~4M x 60s x 1 cpu
~60K cpu-hrs
Amber~10K x 20m x 1 cpu
~3K cpu-hrs
Select best ~500
~500 x 10hr x 100 cpu
~500K cpu-hrsGCMC
PDBprotein
descriptions
Select best ~5KSelect best ~5K
For 1 target:
4 million tasks
500,000 cpu-hrs
(50 cpu-years)[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
37Many-Task Computing on Grids, Clouds, and Supercomputers
CPU cores: 118784
Tasks: 934803
Elapsed time: 2.01 hours
Compute time: 21.43 CPU years
Average task time: 667 sec
Relative Efficiency: 99.7%
(from 16 to 32 racks)
Utilization:
• Sustained: 99.6%
• Overall: 78.3%
[SC08] “Towards Loosely-Coupled Programming on Petascale Systems”
38Many-Task Computing on Grids, Clouds, and Supercomputers
• Purpose
– On-demand “stacks” of
random locations within
~10TB dataset
• Challenge
– Processing Costs:
• O(100ms) per object
– Data Intensive:
• 40MB:1sec
– Rapid access to 10-10K
“random” files
– Time-varying load
AP SloanData
+
+
+
+
+
+
=
+
Locality Number of Objects Number of Files
1 111700 111700
1.38 154345 111699
2 97999 49000
3 88857 29620
4 76575 19145
5 60590 12120
10 46480 4650
20 40460 2025
30 23695 790[DADC08] “Accelerating Large-scale Data Exploration through Data Diffusion”
[TG06] “AstroPortal: A Science Gateway for Large-scale Astronomy Data Analysis”
39
• AstroPortal
– Makes it really easy for astronomers to create
stackings of objects from the Sloan Digital Sky
Servey (SDSS) dataset• Throughput
– 10X higher than GPFS
• Reduced load– 1/10 of the original GPFS load
• Increased scalability• 8X
40Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
41Many-Task Computing on Grids, Clouds, and Supercomputers
• There is more to HPC than tightly coupled MPI, and more to HTC than embarrassingly parallel long jobs
– MTC: Many-Task Computing
– Addressed real challenges in resource management in large scale distributed systems to enable MTC
– Covered many domains (via Swift and Falkon): astronomy, medicine, chemistry, molecular dynamics, economic modelling, and data analytics
• Identified that data locality is critical at large-scale data diffusion
– Integrated streamlined task dispatching with data aware scheduling
– Heuristics to maximize real world performance
– Suitable for varying, data-intensive workloads
– Proof of O(NM) Competitive Caching
42Many-Task Computing on Grids, Clouds, and Supercomputers
• Publications
– 51 articles and proposals
– 66 formal presentations
– 854 citations H-Index 14
• Activities for broader community engagement
– ScienceCloud: ACM Workshop on Scientific Cloud Computing, 2010
– TPDS: IEEE Transactions on Parallel and Distributed Systems, Special Issue on
Many-Task Computing, 2010
– MTAGS: ACM Workshop on Many-Task Computing on Grids and
Supercomputers, 2009
– MTAGS: IEEE Workshop on Many-Task Computing on Grids and
Supercomputers, 2008
– BegaJob: Bird of Feather Session – “How to Run One Million Jobs”, at
IEEE/ACM SC08, 2008
– 53 more activities as a program committee member and/or reviewer43Many-Task Computing on Grids, Clouds, and Supercomputers
• Introduction to Programming (Spring 2010)– Northwestern University, Instructor
• Hot Topics in Distributed Systems: Data-Intensive Computing (Winter
2010)– Northwestern University (EECS495), Instructor
– http://www.eecs.northwestern.edu/~iraicu/teaching/EECS495-DIC/index.html
• Networks and Distributed Systems (2006)– University of Chicago (CMSC 33300), Lead TA
– http://dsl.cs.uchicago.edu/Courses/CMSC33300/index.html
• Grid Computing (2005)– University of Chicago (CMSC 33340), TA
– http://www.mcs.anl.gov/~itf/CMSC23340/
• Introduction to Networking (2003)– Purdue University, Lab TA
• Data Structures and Algorithm Analysis in C++ (2002)– University of Michigan, Adjunct Assistant Professor
• 9 more courses at University of Chicago and
Wayne State University– Advanced Network Design, Introduction to
Programming for the World Wide Web I, Introduction to
Computer Science 1 & 2, Introduction to Computer
Systems , Fundamentals of Computer Programming I
in Scheme, Problem Solving & Programming in C++,
Data Structures & Abstraction in C++
44Many-Task Computing on Grids, Clouds, and Supercomputers
• Embarrassingly Happily parallel apps are trivial to run
– Logistical problems can be tremendous
• Loosely coupled apps do not require “supercomputers”
– Total computational requirements can be enormous
– Individual tasks may be tightly coupled
– Workloads frequently involve large amounts of I/O
– Make use of idle resources from “supercomputers” via backfilling
– Costs to run “supercomputers” per FLOP is among the best
• Loosely coupled apps do not require specialized system software
– Their requirements on the job submission and storage systems can be extremely large
• Shared/parallel file systems are good for all applications
– They don’t scale proportionally with the compute resources
– Data intensive applications don’t perform and scale well
– Growing compute/storage gap
“Impossible only means that you
haven't found the solution yet.”Anonymous
45Many-Task Computing on Grids, Clouds, and Supercomputers
• Defining Many-Task Computing
• Motivation: Scalability Challenges
• Novel Resource Management Techniques
• Performance Evaluation: Micro-benchmarks
• Performance Evaluation: Applications
• Contributions
• Future Work
46Many-Task Computing on Grids, Clouds, and Supercomputers
• Many-Task Computing and Data Intensive Computing
– Develop theoretical and practical aspects of building efficient and
scalable support for MTC
– Build a new distributed data-aware execution fabric that will support
HPC, MTC, and HTC
– Support interactive HPC applications
• Cloud Computing
– Enable scientific computing on clouds (e.g. Magellan project)
– Enable HPC through novel networking
• Many-Core Computing
– Apply data-aware scheduling of jobs from distributed systems
OS scheduling of threads
– Parallel programming models
automatic parallelization using DAG-based data-flow techniques47Many-Task Computing on Grids, Clouds, and Supercomputers
• Maintain and expand collaborations– Government Labs: ANL, FNAL, NASA, LBNL,
– Academia: UC, CI, UIC, DePaul, NU, ND, Purdue, UIUC, IU, UWM,
WSU, LSU, UNC, USF, UBC, DU, MU
– Industry: Accenture, IBM, MS, Google, Yahoo, Amazon, Intel
• Interdisciplinary research– Bring HEC to the science domain
– Many applicable domains: astronomy, astrophysics, economic modeling,
pharmaceuticals, chemistry, bioinformatics, neuroscience, data analytics,
data mining, biometrics, molecular docking, structural equation modeling,
posttranslational protein modification, climate modeling
• Combining Education with Research– NSF CDI, NSF REU, NSF CPATH
• Apply for funding from NSF, DOE, NIH, and industry– DOE INCITE, NSF PetaApps, NSF HECURA, NIH R01
48
• More information: http://www.eecs.northwestern.edu/~iraicu/
• Related Projects: – Falkon: http://dev.globus.org/wiki/Incubator/Falkon
– Swift: http://www.ci.uchicago.edu/swift/index.php
• People contributing ideas, slides, source code, applications, results, etc– Ian Foster, Alex Szalay, Rick Stevens, Mike Wilde, Jim Gray, Catalin Dumitrescu,
Yong Zhao, Zhao Zhang, Gabriela Turcu, Ben Clifford, Mihael Hategan, Allan Espinosa, Kamil Iskra, Pete Beckman, Philip Little, Christopher Moretti, AmitabhChaudhary, Douglas Thain, Quan Pham, Atilla Balkir, Jing Tie, VeronikaNefedova, Sarah Kenny, Gregor von Laszewski, Tiberiu Stef-Praun, Julian Bunn, Andrew Binkowski , Glen Hocky, Donald Hanson, Matthew Cohoon, FangfangXia, Mike Kubal, Alok Choudhary…
• Funding:– NASA: Ames Research Center, Graduate Student Research Program
– DOE: Office of Advanced Scientific Computing Research
– NSF: TeragGrid and Computing Research Innovation Fellow Program
49Many-Task Computing on Grids, Clouds, and Supercomputers