Computer Science and Mathematics
Division
Jack C. Wells
Computer Science and Mathematics Division &
Center for Nanophase Materials Science
Oak Ridge National Laboratory
RAMS Workshop
December, 2005
Oak Ridge
2
Fusion Simulation
Focused on grand challengescientific applications
Astrophysics Genomesto Life
Center for NanophaseMaterials
ClimateModeling
ComputationalChemistry
3
Complex
Systems
J. Barhen P. Boyd
R. Bennink3
Y. Y. Braiman R. J. Carter C. W. GloverW. P. GriceT. S. Humble5
N. ImamD. L. JungV. Kireev5
S. M. LenhartH. K. LiuY. LiuL. E. ParkerN. S. V. RaoD. B. Reister7
D. R. Tufano
Q. Wu5
M. Zhu
Network and
Cluster Computing
G. A. Geist
C. Sonewald
P. K. AgarwalN. E. Baldwin5
R. Baumann5
D. E. BernholdtM. L. ChenJ. J. DongarraW. R. ElwasifC. Engelmann
J. Gao5
G. H. Kora5a
J. A. KohlR. Krishnamurthy5a
A. Longo2
X. Ma2
J. L. Mugler5a
T. J. Naughton5a
H. H. Ong
B.-H. Park5
N. F. SamatovaS. L. ScottJ. SchwidderC. T. Symons5a
K. Uhlemann5
G. R. Vallee5
S. VazhkudaiW. R. WingM. Wolf2
S. Yoginath5a
B. Zhang5
Statistics and
Data Science
J. A. Nichols (acting)
L. E. Thurston
R. W. CountsJ. D. Evans1
E. L. FromeB. L. JacksonG. OstrouchovL. C. PouchardD. D. Schmoyer
D. A. Wolf1Student/1aFaculty2Joint Faculty3Wigner Fellow4Householder Fellow5Postdoc/5aPostmaster6Dual Assignment7Part-time
Climate Dynamics
J. B. Drake C. Sonewald
M. L. BranstetterD. J. EricksonX. Guo1a
M. W. HamF. HoffmanJ. L. HernandezG. MahinthakumarP. H. Worley
Computational
Mathematics
E. F. D’Azevedo L. E. Thurston
V. AlexiadesR. K. Archibald4
V. K. Chakravarthy G. I. FannL. J. GrayR. Hartman-Baker5
A. K. KhamaysehS. N. Fata5
S. Pannala
Computational
Materials Science
T. C. Schulthess L. C. Holbrook
N. D. Arnold5
J. A. Alford5
G. Alvarez3
G. A. AramayoR. M. Day5a
Y. Gao2
A. A. GorinI. Jouline2
T. KaplanP. R. Kent5
J. Lu5
T. A. Maier3
D. M. NicholsonP. K. NukalaY. OsetskiyL. Petit5
B. RadhakrishnanG. B. SarmaM. Shabrov5
S. SimunovicX. Tao5
J. C. WellsX-G. ZhangJ. Zhong C. Zhou5
Computational
Chemical
Sciences
R. J. Harrison L. E. Thurston
W. Alvaro2a
J. BernholcA. Beste5
B. Carlen1
S. Dag5
M. L. Drummond5
T. Fridman5
Z. Gan5
B. C. HathornD. Jiang5
A. Kalemos5
V. Meunier
M. B. Nardelli2
D. W. NoidW. A. SheltonS. Sugiki5
B. G. SumpterN. Vence1
Y. Xu5
J. Cobb, Lead,Teragrid
Future
Technologie
s
J. S. Vetter T. S.Darland
S. R. Alam5
R. F. BarrettN. Bhatia5a
J. A. KuehnC. B.
McCurdy5a
O. StoraasliK. J. RocheP. C. Roth
Operations Council
Finance
U. F. Henderson
Human Resources
M. J. Palermo
OrganizationalManagement
N. Y. Wright6
Recruiter
J. K. Johnson6
Technical Informationand Communications
B. A. Riley6
A. D. Harris
Facility 5600/5700
B. A. Riley
Computer Security
T. K. Jones6
ESH/Safety Officer
Ross Toedte6
5600 Labs Manager
N. S. Rao6
CSB Computing CenterManager
M. W. Dobbs6
Quality Assurance
R. W. Counts6
ADVISORY COMMITTEEDavid Keyes – Visiting Distinguished ScientistJack Dongarra – Distinguished ProfessorThom Dunning – Distinguished ProfessorJim Hack – NCAR ScientistThomas Sterling – Visiting DistinguishedScientist
Computer Science and MathematicsJ. A. Nichols, Director
L. M. Wolfe, Division SecretaryN. Fletcher1
B. Hagen1
11/15/05
4
Our Motivation:Opportunities for Breakthrough Science
Two recent examples:
High-TC superconducting materials:
First solution of 2D Hubbard Model
(T. Maier, PRL, accepted 10/2005)
Fusion plasma simulation:
Largest simulation of plasma behavior
in a tokamak
(F. Jaeger, APS-DPP invited
presentation, 10/2005)
5
Advancing scientific discoverythrough computing
• Practical nanotube devices demand airstable, controllable, large-scale doping
• Recent experiments suggest organicdoping is viable
• Simulations confirm experiment andenable detailed understanding & design
• Large-scale quantum conductancecalculations on Cray-X1 and SGI-Altix
• Meunier and Sumpter, Journal ofPhysical Chemistry (2005, accepted)
A New Second Variational BindingEnergy Model
Design of ceramic materials
• Interprets observed properties in termsof chemical bonding
• Control of the microstructureand mechanical properties
• Correctly predicts interface positionsof sintering additives
• Simulations led experiment!
• N. Shibata et al., Nature 428,730–733 (2004)
Design and Control of Nanoscale andMolecular Electronic Devices
Computational Nanoscience
6
Prediction and discovery of giant tunnelingmagnetoresistance (TMR) for spin valves
Magnetoresistanceapplications today:
Magnetic RandomAccess MemoryRecording head
Typical TMR foramorphousaluminumoxide barrier:
Computationalprediction: TMR of1000% is possiblefor crystalline MgObarrier, if interfacesare good enough
By 2004,MgO-basedheterostructureswith >300% TMRdiscoveredexperimentally
•Recording headin computerhard discs
•Magnetic randomaccess memory
• 1995: ~10%(discovery @ MIT)
• 2005: ~70%(after a bigexperimental effort)
•Butler, Zhang,Schulthess, andMacLaren (ORNL),Phys. Rev. B (2001)
• Parkin et al.,Nature Materials(2004)
• Yasa et al., NatureMaterials (2004)
7
2004 2005 2006 200
7
2008 2009
100 TF
1000 TF
250 TF
25 TF
5TF
Cray X1E
IBM BG
NLCF plan for the next 5 years:
Cray XT3
TBD
Cray X1E
IBM Blue Gene
Vector Arch
Global memory
Powerful CPU
Cluster Arch
Low latency
High bandwidth
Scalability
100K CPU
MB/CPU
Cray XT3
18 TF
8
Observations
• Memory-processor performancegap leading to low sustainedperformance on importantapplications
• ITRS predicts that Moore’s Lawwill be coming to an end in2012-2015
• Practical considerations (power,cooling, floor space) are forcingarchitects to change theirdesigns now
Power DensityPower Density
Will Get Even WorseWill Get Even Worse
Need to Keep the Junctions CoolNeed to Keep the Junctions Cool
•• Performance (Higher Frequency)Performance (Higher Frequency)
•• Lower leakage (Exponential) Lower leakage (Exponential)
•• Better reliability (Exponential)Better reliability (Exponential)
Hot PlateHot Plate
Nuclear ReactorNuclear Reactor
Rocket NozzleRocket Nozzle
Sun’s SurfaceSun’s Surface
4004400480088008
8080808080858085
80868086
286286386386
486486
PentiumPentium®®
processorsprocessors
11
1010
100100
1,0001,000
10,00010,000
’70’70 ’80’80 ’90’90 ’00’00 ’10’10
Power DensityPower Density
(W/cm2)(W/cm2)
© 2001 IEEE International Solid-State Circuits Conference © 2001 IEEE© 2001 IEEE International Solid-State Circuits Conference © 2001 IEEE
Source: Pat Gelsinger, CTO, Intel
October 15, 2004
9
How Big Is Big?
• Every 10X brings new challenges
64 processors was once considered large
It hasn’t been “large” for quite a while.
1024 processors is today’s “medium” size
2048-16192 processors is today’s “large”
We are struggling even here.
• 100K processor systems
are being designed/deployed
have fundamental challenges …
… and no integrated research programs
• Petascale data archives
the “personal petabyte” very near
• See recent PITAC report
www.nitrd.gov
0
5
10
15
20
25
30
128 256 512 1024 2048 4096 8192 16384 More
Count
Top 500 Size
Distribution
Preparing for Big:Math and CS challenges
• Theoretical Models (existing)
May not perform well on petascale computers
May not have needed fidelity
May be inadequate to describe new phenomena revealed by experiment or
simulation
• Scientific Modeling and Simulation Codes (existing)
Do not take advantage of new architectures (5%-10% of peak)
New computing capabilities lead to new simulation possibilities and, thus, new
applications codes
• Systems Software
Vendor operating systems do not provide needed functionality
Systems software for petascale applications non-existent
• Software to manage and visualize massive (petabyte) data sets
• Software to accelerate development and use of petascale scientific applications
• Techniques for porting software to the next generation inadequate
Few mathematical algorithms scale to thousand-million processors
11
People and partnerships
Thomas Maier Wigner Fellow
Jennifer RyanHouseholder Fellow
Gonzalo AlvarezWigner Fellow
We have methodically hired122 research staff in 5 yearsin support of computing
Elbio Dagotto Distinguished Scientist
Jeff VetterFuture Technologies
43464134Students
30564058Postdocs
23312115S&T
2005200420032002
• Strong partnerships with sister labs(ANL, LBNL, PNNL) and other centers
• Interagency partnerships with NSF, NSA, NNSA, NASA, DHS
– Key resource for Intergovernmental Panel on Climate Change (IPCC)
• Joint Institute for Computational Sciences/core universities
– Graduate program in computational Sciences
– Distinguished Scientists/ Joint faculty appointments
• Cray Center of Excellence
12
World leaderin scientific computing
Intellectual center incomputational science
Transform scientificdiscovery through
advanced computing
“User facility providingleadership-class
computing capability toscientists and engineersnationwide independent
of their institutionalaffiliation or source
of funding”
Create an interdisciplinaryenvironment where
science and technologyleaders converge
to offer solutions totomorrow’s challenges
“Deliver major researchbreakthroughs,
significant technologicalinnovations, medicaland health advances,enhanced economic
competitiveness, andimproved quality of life
for the American people”
Our Aspirations