Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | irma-cooley |
View: | 48 times |
Download: | 1 times |
Introduction to Scalable Programming using Makeflow and
Work Queue
Dinesh Rajan and Mike AlbrechtUniversity of Notre Dame
October 24 and November 7, 2012
3
I have a standard, debugged, trusted application that runs on my laptop. One simulation runs in an hour.I have to run 100.Then I have to analyze the results, tweak the simulation, and run 100 more.
Can I get a single result faster?Can I get more results in the same time?
Last year,I heard aboutthis grid thing.
This year,I heard about
this cloud thing.
What do I do next?
Should I port my program to MPI or Hadoop?Learn C / JavaLearn MPI / HadoopRe-architectRe-writeRe-testRe-debugRe-certify
I can get as many machineson the cloud as I want!
How do I organize my applicationto run on those machines?
9
An Old Idea: Makefiles
part1 part2 part3: input.data split.py ./split.py input.data
out1: part1 mysim.exe ./mysim.exe part1 >out1
out2: part2 mysim.exe ./mysim.exe part2 >out2
out3: part3 mysim.exe ./mysim.exe part3 >out3
result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result
10
Makeflow = Make + Workflow
• Provides portability across batch systems.• Enable parallelism (but not too much!)• Fault tolerance at multiple scales.• Data and resource management.
Makeflow
Local Condor SGE WorkQueue
http://www.nd.edu/~ccl/software/makeflow
Makeflow Language - Rules
• Each rule specifies:– a set of target files to
create;– a set of source
files needed to create them;
– a command that generates the target files from the source files.
part1 part2 part3: input.data split.py ./split.py input.data
out1: part1 mysim.exe ./mysim.exe part1 >out1
out2: part2 mysim.exe ./mysim.exe part2 >out2
out3: part3 mysim.exe ./mysim.exe part3 >out3
result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result
out1 : part1 mysim.exemysim.exe part1 > out1
PrivateCluster
CampusCondor
Pool
PublicCloud
Provider
CRCSGE
Cluster
Makefile
Makeflow
Local Files and Programs
Makeflow + Batch System
makeflow –T sge
makeflow –T condor
Work Queue
Work Queue
How to run a Makeflow
• Run a workflow locally (multicore?)– makeflow -T local sims.mf
• Clean up the workflow outputs:– makeflow –c sims.mf
• Run the workflow on SGE:– makeflow –T sge sims.mf
Hands Onhttp://nd.edu/~ccl/software/tutorials/ndtut12/mf-tutorial.php
Practice Problems
http://nd.edu/~ccl/software/tutorials/ndtut12/mf-hw.php
1. Construct a makeflow to render a short movie featuring a Rubik’s cube
2. Launch the makeflow on both your laptop and SGE
3. Consider ways you might use Makeflow for your research