Date post: | 14-Jul-2015 |
Category: |
Technology |
Upload: | daniel-s-katz |
View: | 302 times |
Download: | 1 times |
www.ci.anl.govwww.ci.uchicago.edu
1www.ci.anl.govwww.ci.uchicago.edu
Multi-component Modeling with Swift at Extreme Scale
Daniel S. Katz, Justin Wozniak, Michael Wilde, and Ian Foster
Computation Institute, University of Chicago & Argonne National Laboratory
http://swift-lang.org/
Supercomputing Frontiers 2015
March 17, 2015, Singapore
http://slideshare.com/danielskatz
www.ci.anl.govwww.ci.uchicago.edu
2
Swift Domain of Interest
“Time”
“Co
mp
lexi
ty”
Increasing capabilities in computational science
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
3
Extreme-scale application space
Single large problem; e.g. PDE solution over single domain
Problem composed of large number of sub-problems; e.g. parameter sweep
Problem composed of small number of sub-problems; e.g. components, ensemble
Extreme-scale application space
Application space Swift addresses
Multi-component Modeling with Swift at Extreme Scale
Today’s hero problems
Yesterday’s hero problems
www.ci.anl.govwww.ci.uchicago.edu
4
Climate models
• Built from components
Multi-component Modeling with Swift at Extreme Scale
Ocean
Atmosphere
Land
Sea ice
www.ci.anl.govwww.ci.uchicago.edu
5
Methods of coupling
• Coupling library – add to each component– E.g. Model Coupling Toolkit
(http://www.mcs.anl.gov/research/projects/mct/)
• Coupling component – add new component– E.g. CSM Flux Coupler
(http://www.cesm.ucar.edu/models/cpl/)
• Using data store (in RAM) or file system (on disk)– E.g., DataSpaces
(http://personal.cac.rutgers.edu/TASSL/projects/data/index.html) and ADIOS (https://www.olcf.ornl.gov/center-projects/adios/)
• Traditional usage: climate, fusion, REMD• Newer usage: in situ analysis
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
6
Climate component models
Multi-component Modeling with Swift at Extreme Scale
Credit: Kaitlin Alexander & Steve Easterbrook, “The Software Architecture of Global Climate Models,” U. Toronto
www.ci.anl.govwww.ci.uchicago.edu
7
One climate model
Multi-component Modeling with Swift at Extreme Scale
Credit: Kaitlin Alexander & Steve Easterbrook, “The Software Architecture of Global Climate Models,” U. Toronto
www.ci.anl.govwww.ci.uchicago.edu
8
Multicomponent models for multiscale
Finite Element (microstructure, solve for k)
Phase Field (evolve microstructure)
),,( 11 iii tTrk
TTrk ),(0
QTkt
TCP
Finite Element (fuel element model, solve for T)
),( ii trT
newoldold tkT ,,
ii kk 1
ii TT 1
newnewnew tkT ,,
newkYes
Yes
No
No
newT
Multi-component Modeling with Swift at Extreme Scale
Credits:
FEM: B. Mihaila, et al., J. Nucl. Mater., 394 (2009) 182-189
PF: S.Y. Hu et al., J. Nucl. Mater. 392 (2009) 292–300
Nuclear fuel rod workflow: Marius Stan, Argonne
www.ci.anl.govwww.ci.uchicago.edu
9
Types of coupling
• Loose (through files, sequential order to components)
• Tighter (through messages over WAN, distributed simultaneous components)
• Tight (through [MPI] messages over LAN/local IC, parallel simultaneous components)
Model1(MS A) File
Model2(MS A|B)File
Multi-component Modeling with Swift at Extreme Scale
Model1(MS A)
Model2(MS B)
Msg
Model1(MS A)
Model2(MS A)
Msg
www.ci.anl.govwww.ci.uchicago.edu
10
Component state
• Do components have state?– No
o Init, run, stop
o Communication with other components before and after run, communication can be outside component lifecycle
– Yeso Init, run, run, run, run, stop
o Communication with other components before and after run, communication must be inside component lifecycle
• Can this be distinct from component execution?– “Stateless” component can remain in memory, if
each “run” depends fully on external inputs
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
11
Swift http://swift-lang.org/
• Swift is a parallel scripting system for grids, clouds and clusters
• F() and G() are computed in parallel– Can be Swift functions, or leaf tasks (executables or scripts in
shell, python, R, Octave, MATLAB, ...)
• r computed when they are done
• This parallelism is automatic
• Works recursively throughout the program’s call graph
Multi-component Modeling with Swift at Extreme Scale
(int r) myproc (int i)
{
int f = F(i);
int g = G(i);
r = f + g;
}
www.ci.anl.govwww.ci.uchicago.edu
12
Swift Variables, tasks, files, concurrency
• Variables are single assignment futures– Unassigned variables are open
• Variables can represent files– When a file doesn’t exist, the variable is open
– When a file exists, the variable is closed
• All tasks found at runtime
• Tasks with satisfied dependencies (all inputs are closed variables) are run on whatever resources are available
• These runs create files/variables that allow more tasks to run
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
13
Advanced Photon Source (APS)
• Moves electrons at electrons at >99.999999% of the speed of light.• Magnets bend electron trajectories, producing x-rays, highly focused
onto a small area• X-rays strike targets in 35 different laboratories – each a lead-lined,
radiation-proof experiment station• Beam time allocated in multi-day chunks; very valuable!
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
14
High Energy Diffraction Microscopy (HEDM)
Used to non-destructively find crystal structure of metals at Argonne Advanced Photon Source (APS)
Near Field (NF)-HEDM grain identification for cross section of gold wire.L: colors are grains, R: colors are confidence
Confidence obtained by larger amount of analysis
Far field (FF)-HEDM grain identification for cross section of
experimental material.Colors are grain centers
Credit: Hemant Sharma, Jonathan Almer, Materials Physics and Engineering Group, X-Ray Science Division, ANL. Justin Wozniak, MCS and CI, ANL
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
15
NF-HEDM Using Mira via SwiftSingle integrated cross-system script – 4 GB processed every 4-10 mins, during beam time
320-node X86 cluster
Up to 8192 nodes used
Credit: Hemant Sharma, Jonathan Almer, Materials Physics and Engineering Group, X-Ray Science Division, ANL. Justin Wozniak, MCS and CI, ANL
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
16
Parallel tasks in Swift
Multi-component Modeling with Swift at Extreme Scale
• Swift expression: z = @par=8 f(x,y);
• When x, y are stored, Turbine releases task f with parallelism=8
• Performs ADLB_Put(f, parallelism=8)
• Each worker performs ADLB_Get(&task, &comm)
• ADLB server finds 8 available workers
• Workers receive ranks from ADLB server, which performsMPI_Comm_create_group()
• ADLB_Get() returns:task=f, size(comm)=8
• Workers perform user task– Communicate on comm
• comm is freed by Turbine
• Task data can be– in-memory objects stored in ADLB
– file stored on disk, file vars in ADLB
• Files can be stored in shared disk, or locally using MosaStore, Hercules, etc.
www.ci.anl.govwww.ci.uchicago.edu
17
Simple climate model in Swift
• Swift yesterday (pseudocode)
• atm_extract and ocean_extract can run in parallel, e.g. each on n/2 cores
• atm_step and ocean_step can run in parallel, each on n/2 cores (or load balanced), or in sequence, each on n cores
• May use ADIOS or similar to handle MxN I/O
Multi-component Modeling with Swift at Extreme Scale
file: atm_state[], ocean_state[], wind[], sst[];
for t ...
{
wind[t] = atm_extract(atm_state[t]);
sst[t] = ocean_extract(ocean_state[t]);
atm_state[t+1] = atm_step(atm_state[t], stt[t]);
ocean_state[t+1] = ocean_step(ocean_state[t],
wind[t]);
}
www.ci.anl.govwww.ci.uchicago.edu
18
More complex climate model in Swift
• Swift today (pseudocode)
• Functions still need to init, run, exit each step
• Functions can be compiled into single executable, rather than re-launched
Multi-component Modeling with Swift at Extreme Scale
file: atm_state0[x,y,z], ocean_state0[x,y,z];
float: atm_state[x,y,z][], ocean_state[x,y,z][], wind[x,y][],
sst[x,y][];
atm_state[][0] = read_atm(atm_state0[]);
ocean_state[][0] = read_ocean(ocean_state0[]);
for t ...
{
wind[][t] = atm_extract(atm_state[][t]);
sst[][t] = ocean_extract(ocean_state[][t]);
atm_state[][t+1] = atm_step(atm_state[][t], stt[][t]);
ocean_state[][t+1] = ocean_step(ocean_state[][t], wind[][t]);
}
www.ci.anl.govwww.ci.uchicago.edu
19
More complex climate model in Swift
• Swift tomorrow – work on language and runtime needed (pseudocode)
• Function state & variables cached in node memory
• Maybe make use of local FLASH/storage
Multi-component Modeling with Swift at Extreme Scale
file: atm_state0[x,y,z], ocean_state0[x,y,z];
float: atm_state[x,y,z][], ocean_state[x,y,z][], wind[x,y][],
sst[x,y][];
atm_state[][0] = read_atm(atm_state0[]);
ocean_state[][0] = read_ocean(ocean_state0[]);
for t ...
{
wind[][t] = atm_extract(atm_state[][t]);
sst[][t] = ocean_extract(ocean_state[][t]);
atm_state[][t+1] = atm_step(atm_state[][t], stt[][t]);
ocean_state[][t+1] = ocean_step(ocean_state[][t], wind[][t]);
}
www.ci.anl.govwww.ci.uchicago.edu
20
NAMD/VMD and Swift/T
Multi-component Modeling with Swift at Extreme Scale
Typical Swift/T Structure
MD1.cMD1.c MD2.cppMD2.cpp viz.cppviz.cpp
SWIG-generated Tcl wrappersSWIG-generated Tcl wrappers
Swift/T runtimeSwift/T runtimeMPI
Top-level dataflow scriptexchange.swift
Top-level dataflow scriptexchange.swift
NAMD/VMD Structure
Swift/T runtimeSwift/T runtime
NAMD (C++)NAMD (C++)
Tcl Evaluation (uplevel-eval)Tcl Evaluation (uplevel-eval)
Top-level dataflow scriptexchange.swift
Top-level dataflow scriptexchange.swift
Credit: Phillips et al., Petascale Tcl with NAMD, VMD, and Swift/T, HPTCDL 2014
• NAMD and VMD embed Tcl, and thus can run Swift dataflow programs
• Swift has been used to demonstrate n:m multiplexing of n replicas across a smaller arbitrary number m of NAMD processes
• This is very complex to do with normal NAMD scripting but can be expressed naturally in under 100 lines of Swift/T code
www.ci.anl.govwww.ci.uchicago.edu
21
Conclusions
• Multicomponent models increasingly important for science
• Swift: effective means of programming and running many task computing (MTC) applications– MTC model traditionally based on read (file), run
(executable), write (file) model
– Now extended to many parallel tasks, read & write to data store, run functions
• Swift demonstrated for multicomponentmodeling in MD, material science
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
22
Future work
• Abstractions– For hierarchically complex, concurrent application components– For stateful tasks across functional programs
o Build on Swift's growing location and topology awareness
– For events and exceptions
• Language– How to express/hint that a function and/or its state should remain
local?– Relearn lessons from CCA, Babel, etc.– Expose topology features with dynamic programming constructs
• Runtime– How to implement the above?
o Cache/swap process and data in memory, FLASH, local disk?
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
23
Further research directions
• Orchestrating in-situ analytics for extreme-scale simulations
In-situ analytics of a climate simulation
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
24
Acknowledgements
• Swift (http://swift-lang.org/) is supported in part by NSF grant OCI-1148443
• Extreme scaling research on Swift (ExM project) was supported by the DOE Office of Science, ASCR Division, X-Stack program
• Additional support is provided by the Argonne Leadership Computing Facility and the Argonne Director’s Grand Challenge Project “Discovery Engines for Big Data”
• The Swift team (mostly UC/ANL):– Tim Armstrong, Yadu Nand Babuji, Ian Foster, Mihael
Hategan, Dan Katz, David Kelly, Ketan Maheshwari, Ioan Raicu(IIT), Michael Wilde, Justin Wozniak, Sophia Yang
• Special thanks to all Swift users and collaborators
Multi-component Modeling with Swift at Extreme Scale
www.ci.anl.govwww.ci.uchicago.edu
25
Thank you!
Questions?
Multi-component Modeling with Swift at Extreme Scale