Overview of the Exascale Additive Manufacturing Project (ExaAM) One of 15 Applications in the US DOE Exascale Computing Project
John A. Turner Oak Ridge National Laboratory Group Leader: Computational Engineering and Energy Sciences Chief Computational Scientist: Consortium for Advanced Simulation of Light Water Reactors (CASL) Principle Investigator: Transforming Additive Manufacturing Through Exascale Simulation (ExaAM)
Numerous others on the ExaAM team (incomplete list): Jim Belak (co-PI, LLNL), Andy Anderson (LLNL), Suresh Babu (UTK), Mark Berrill (ORNL), Curt Bronkhorst (LANL), Neil Carlson (LANL), Ondrej Certik (LLNL), Jean-Luc Fattebert (LLNL), Neil Hodge (LLNL), Wayne King (LLNL), Lyle Levine (NIST), Chris Newman (LANL), B. Radhakrishnan (ORNL), Adrian Sabau (ORNL), Srdjan Simunovic (ORNL)
HPC User Forum Santa Fe, NM 17-19 Apr 2017
www.ExascaleProject.org
2 Exascale Computing Project
Outline
• Additive Manufacturing • Exascale Computing Program • ExaAM Project
3 Exascale Computing Project
Slide from my presentation at the April 2014 HPC User Forum Meeting
(also in Santa Fe)
4 Exascale Computing Project
Slide from my presentation at the April 2014 HPC User Forum Meeting
(also in Santa Fe)
• Computer-Aided Engineering for Batteries Program (DOE / EERE / VTO)
• Battery Crashworthiness (DOT / NHTSA)
A lot has happened in the last three
years.
5 Exascale Computing Project
I assume most are aware of additive manufacturing, a.k.a. 3D printing, and that it is being used for metal as well as polymers
21.1 g 12.1 g 14.4 g
6 Exascale Computing Project
Test Stand at NASA Marshall Space Flight Center (Huntsville, AL)
7 Exascale Computing Project Powder Bed Technologies
Design
Material Feedstock
In-situ Process Control
Material µm-nm
Structure
Static and Dynamic
Mechanical Properties
Plasma (wire)
E-beam (wire)
Laser (wire)
Large Melt Pool Technologies
Laser (powder)
Direct Metal Deposition
Laser (powder)
E-beam (powder)
There are multiple metal additive manufacturing technologies Physical processes are similar • Energy Deposition • Melting & Powder Addition • Evaporation & Condensation • Heat & Mass Transfer • Solidification • Solid-State Phase Transformation • Repeated Heating and Cooling • Complex Geometries
8 Exascale Computing Project
Multiple computational challenges must be addressed for AM
• 1 m3 ~ 1012 particles ~ 109 m of “weld” line (assuming 50µm particles) and build times of hours
• Large temperature gradients, rapid heating and cooling – necessary / sufficient coupling between thermomechanics and melt/solidification
• Heterogeneous and multi-scale – resolution of energy sources and effective properties of powder for continuum simulations
• Path optimization • Large number of parameters and incomplete understanding
– key uncertainties and propagation of those uncertainties
• Validation is difficult as characterization is limited
9 Exascale Computing Project
Overview of electron beam Additive Manufacturing (Arcam®)
http://www.arcam.com/technology/electron-beam-melting/hardware/
3D CAD Model
Thin 2D Layers
To Machine
Nth Layer
Preheating Melting (N+1)th layer
Final Part
Conventional raster melt sequence
Microstructure manipulation of IN718 via additive
manufacturing is not well understood. Always results in
columnar grains oriented along the build direction (001)
• Microstructure plays significant role in determining mechanical properties of final part
• Directional vs. Isotropic properties • Feasibility of site specific microstructure control?
10 Exascale Computing Project
Mechanical anisotropy and poor properties are observed in z-direction
Kobryn and Semiatin (2001)
• Anisotropy is a function of material thermal path. • Thermal path of deposit material is non-uniform • HIP is not feasible for all additive deposits • This poses a challenge in part qualification
lacking fundamental understanding of process-structure-property-performance relationships
trial and error optimization is incredibly inefficient P. A. Kobryn and S. L. Semiatin, “The laser additive manufacture of Ti-6Al-4V,” JOM, vol. 53, no. 9, pp. 40–42, Sep. 2001. doi:10.1007/s11837-001-0068-x.
11 Exascale Computing Project
A more complete understanding of the linkage between process, structure, properties, and performance is needed
Courtesy of Wayne King, Director of the Accelerated Certification of Additively Manufactured Metals Initiative at LLNL
12 Exascale Computing Project
What is the Exascale Computing Project (ECP)? • Created in support of President Obama’s National Strategic
Computing initiative (NSCI) • A collaborative effort of two US Dept of Energy (DOE) offices:
– Office of Science (DOE-SC) – National Nuclear Security Administration (NNSA)
• A 10-year project to accelerate the development of a capable exascale ecosystem – 50x the performance of today’s 20 PF/s systems – Operates in a power envelope of 20–30 MW – Is sufficiently resilient (average fault rate: ≤1/week) – Includes a software stack that meets the needs of a broad
spectrum of applications and workloads – Led by DOE laboratories – Executed in collaboration with academia and industry
A capable exascale computing system will have a well-balanced ecosystem (software,
hardware, applications)
13 Exascale Computing Project
Application Development Software Technology
Hardware Technology
Exascale Systems
Scalable software stack
Science and mission applications
Hardware technology elements
Integrated exascale supercomputers
ECP has formulated a holistic approach that uses co-design and integration to achieve capable exascale
Correctness Visualization Data Analysis
Applications Co-Design
Programming models, development environment, and
runtimes Tools Math libraries and
Frameworks
System Software, resource management threading,
scheduling, monitoring, and control
Memory and Burst buffer
Data management I/O and file
system Node OS, runtimes
Res
ilienc
e
Wor
kflo
ws
Hardware interface
14 Exascale Computing Project
ExaAM is one of 15 initial ECP application development projects
Advanced Manufacturing Gaps and Opportunities • Improve quality, reliability, and application breadth of
additive manufacturing (AM) • Accelerate innovation in clean energy manufacturing
institutes (NNMIs) • Capture emerging manufacturing markets Simulation Challenge Problems • Continuum level predictions of non-uniform
microstructure and its relationship to process parameters
• Predictive mesoscale models for dendritic solidification scale-bridged to continuum
Prospective Outcomes and Impact • Routine qualification of AM parts via process-aware
design specs and reproducibility through process control • Fabrication of metal parts with unique properties such
as light weight strength and failure-proof joints and welds
15 Exascale Computing Project
Models and Code(s) • Physical Models: fluid flow, heat transfer,
phase change (melting/solidification and solid-solid), nucleation, microstructure formation and evolution, residual stress
• Codes: • Continuum: ALE3D, Diablo, Truchas • Mesoscale: AMPE, MEUMAPPS, Tusas
• Motifs: Sparse Linear Algebra, Dense Linear Algebra, Spectral Methods, Unstructured Grids, Dynamical Programs, Particles
Transforming Additive Manufacturing through Exascale Simulation (ExaAM)
PI: John Turner (ORNL), co-PI: Jim Belak (LLNL)
Goal and Approach • Accelerate the widespread adoption of
additive manufacturing (AM) by enabling fabrication of qualifiable metal parts with minimal trial-and-error iteration and realization of location-specific properties • Coupling of high-fidelity sub-grid simulations
within a continuum process simulation to determine microstructure and properties at each time-step using local conditions
Software and Numerical Library Dependencies • C++, Fortran • MPI, OpenMP, OpenACC, CUDA • Kokkos, Raja, Charm++ • Hypre, Trilinos, P3DFFT,
SAMRAI, Sundials, Boost • DTK, netCDF, HDF5, ADIOS,
Metis, Silo • GitHub, GitLab, CMake, CDash,
Jira, Eclipse ICE
Critical Needs Currently Outside the Scope of ExaAM • modeling of powder properties and spreading • shape and topology optimization • post-build processing, e.g. hot isostatic
pressing (HIP) • data analytics and machine learning of
process / build data • reduced-order models
16 Exascale Computing Project
Additive Manufacturing Physics / Process Workflow
17 Exascale Computing Project
Quick survey of selected ExaAM application codes (components)
18 Exascale Computing Project
ExaAM codes and attributes
Code(s) Area(s) Physical Models Computational Motifs Prog. Lang.
(Model)
Numerical Library
Depenencies
Proxy App
Diablo Process (part scale), Performance
solid mechanics, heat & mass trans, contact, implicit time integration
Lagrangian FEM, nonlinear physics, staggered & monolithic solvers, adaptive h-refinement
Fortran (MPI)
Hypre, HDF, Metis, Silo TBD
Truchas Process (melt pool to part scale)
free-surface flow, heat transfer, phase change, species diffusion
FVM, unstructured mesh, implicit mimetic finite difference, linear & nonlinear solvers
Fortran (MPI)
Hypre, HDF5, netCDF Pececillo
ALE3D Process (melt pool scale), Properties
implicit and explicit hydro, heat trans, phase change
FEM, unstructured mesh, advection, linear & nonlinear solvers
C++ (MPI) Hypre LULESH
MEUMAPPS Microstructure phase-field Fourier spectral method Fortran (MPI) P3DFFT N/A
AMPE Microstructure phase-field implicit FVM, linear & nonlinear
solvers, AMR C++ (MPI) Hypre, SAMRAI, Sundials
AMG2013
Tusas Microstructure phase-field implicit FEM, preconditioned
JFNK, unstructured 2D and 3D C++, (MPI, OpenMP)
Trilinos, netCDF, HDF5, Boost
N/A
Cont
inuu
m sc
ale
Mes
osca
le
19 Exascale Computing Project
ALE3D (LLNL) has been used to study details of the beam-powder interaction and melt pool dynamics
Khairallah, S.A., Anderson, A., 2014. Mesoscopic Simulation Model of Selective Laser Melting of Stainless Steel Powder. Journal of Materials Processing Technology 214, 2627-2636 DOI:10.1016/j.jmatprotec.2014.06.001.
Laser Thin Powder Layer
Thick Powder Layer
Bridge area
a
Yadroitsev, I., Gusarov, A., Yadroitsava, I., Smurov, I., 2010. Single track formation in selective laser melting of metal powders. Journal of Materials Processing Technology 210, 1624-1631.
Movie replaced by still image for distribution
20 Exascale Computing Project
Diablo (LLNL) simulation of residual stress during build
Hodge, N.E., Ferencz, R.M., Vignes, R.M., 2016. Experimental Comparison of Residual Stresses for a Thermomechanical Model for the Simulation of Selective Laser Melting. Additive Manufacturing DOI. http://dx.doi.org/10.1016/j.addma.2016.05.011.
Movie replaced by still image for distribution
21 Exascale Computing Project
• J. D. Hunt, “Steady state columnar and equiaxed growth of dendrites and eutectic,” Mater. Sci. Eng., vol. 65, no. 1, pp. 75–83, 1984. • M. Gäumann, C. Bezençon, P. Canalis, and W. Kurz, “Single-crystal laser deposition of superalloys: Processing-microstructure maps,” Acta
Mater., vol. 49, no. 6, pp. 1051–1062, 2001.
Simulation helped enable local control of grain structure in AM parts
• Given G and R, can calculate volume fraction of equiaxed grains at any location • G is temperature gradient, • R is velocity of liquid-solid interface, • No is nucleation density, • Φ is volume fraction of equiaxed grains
(probability of stray grain formation) • n and a are alloy constants
Lee, Y., Nordin, M., Babu, S. S., & Farson, D. F. (2014). Effect of Fluid Convection on Dendrite Arm Spacing in Laser Deposition. Metallurgical and Materials Transactions B, 45(4), 1520-1529.
• Columnar-to-Equiaxed Transition (CET) in rapid solidification processes primarily controlled by: – Thermal gradient at the liquid solid interface (G) – Velocity or growth rate of liquid-solid interface (R)
• Difficult to measure experimentally – Spatial resolution (microns) – Temporal resolution required (milliseconds) – Thermal imaging camera cannot capture 3D data
22 Exascale Computing Project
Truchas provides: • Thermal gradient at the liquid solid interface
• Velocity of liquid-solid interface
Truchas metal casting code (LANL) can determine G and R
Conventional Raster Pattern
Spot Melt Pattern along the contour “DOE”
Temperature gradient and melt pool isotherm
• Dehoff, R. R., Kirka, M. M., Sames, W. J., Bilheux, H., Tremsin, A. S., Lowe, L. E., & Babu, S. S. (2015). Site specific control of crystallographic grain orientation through electron beam additive manufacturing. Materials Science and Technology, 31(8), 931-938.
• N. Raghavan, R. Dehoff, S. Pannala, S. Simunovic, M. Kirka, J. Turner, N. Carlson, and S. S. Babu, “Numerical modeling of heat-transfer and the influence of process parameters on tailoring the grain morphology of IN718 in electron beam additive manufacturing,” Acta Materialia, vol. 112, pp. 303–314, Jun. 2016. doi:10.1016/j.actamat.2016.03.063.
Movie replaced by still image for distribution
23 Exascale Computing Project
Using MEUMAPPS (ORNL phase field code), nucleation rate has been identified as the main factor in formation of colony grain structure
Crucial Findings • Low nucleation rate promotes colony when a new nucleus
sees well developed strain field from a nearby variant • High nucleation rate promotes basket weave when all nuclei
see complex strain field due to multiple, evolving nuclei
N=0.5 s-1
Colony structure
N=5.0 s-1
Basket weave structure
950K
1000K B. Radhakrishnan, S. Gorti, and S. S. Babu, “Phase Field Simulations of Autocatalytic Formation of Alpha Lamellar Colonies in Ti-6Al-4V,” Metallurgical and Materials Transactions A, vol. 47, no. 12, pp. 6577–6592, Dec. 2016. doi:10.1007/s11661-016-3746-6.
Parametric studies performed using phase field simulations • Two levels of thermodynamic driving force: low
(1000K) and high: 950K • Two levels of nucleation rate: low (0.5 s-1) and
high (5 s-1) Auto-catalytic colony nucleation
Movie replaced by still image for distribution
24 Exascale Computing Project
Objective: Utilize exascale concurrency and locality to dynamically bridge continuum and mesoscale physics • Task-based embedded Scale-Bridging
escapes the traditional synchronous SPMD paradigm and exploits the heterogeneity expected in exascale hardware.
• To achieve this, we are developing a UQ-driven adaptive physics refinement approach.
• Coarse-scale simulations dynamically spawn tightly coupled and self-consistent fine-scale simulations as needed.
• This task-based approach naturally maps to exascale heterogeneity, concurrency, and resiliency issues.
24
25 Exascale Computing Project
Ultimately, ExaAM will deliver and deploy a new integrated simulation environment for AM
26 Exascale Computing Project
Questions? e-mail: [email protected] The research and activities described in this presentation were performed using the resources at Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC0500OR22725.
This research was supported by the Exascale Computing Project (http://www.exascaleproject.org), a joint U.S. Department of Energy and National Nuclear Security Administration project responsible for delivering a capable exascale ecosystem, including software, applications, hardware, and early testbed platforms, to support the nation’s exascale computing imperative.