UNCLASSIFIED UNCLASSIFIED Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA LA-UR-15-21345 | Page 1 of 18 This presentation is approved for unlimited, public release. Date: February-23-2015 Reference: LA-UR-15-21345 Number of pages: 18 A BRIEF DISCUSSION OF SIMULATION UNCERTAINTY François Hemez Technical Staff Member, Los Alamos National Laboratory Adjunct Professor, University of California San Diego [email protected]
Transcript
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 1 of 18
This presentation is approved for unlimited, public release.Date: February-23-2015Reference: LA-UR-15-21345Number of pages: 18
A BRIEF DISCUSSION OF SIMULATION UNCERTAINTY
François HemezTechnical Staff Member, Los Alamos National Laboratory
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 2 of 18
AbstractA BRIEF DISCUSSION OF SIMULATION UNCERTAINTY
This presentation offers a high-level discussion of uncertainty in science-based numerical simulations, such as those carried out to analyze the performance of engineered systems (aircraft, nuclear power plant, building, etc.). Three broad classes of uncertainty in simulations are defined: variability and “randomness”, numerical uncertainty, and model-form assumption-making. Each category is briefly defined as well as uncertainty management techniques that can be applied to deterministic models and simulations. The statistical methods available to propagate and quantify variability are well-developed and mature. Even though meshes used for industrial applications are usually not as “resolved” as they should be, methods also exist to estimate the bounds of numerical uncertainty that originates from truncation effects. When it comes to the third class, model-form uncertainty, the essential question is to ask its effect on numerical predictions and the decisions that they support. The presentation concludes by briefly addressing the concept of “confidence” and suggesting that it derives from establishing robustness of simulation predictions to potentially erroneous modeling assumptions. (Abstract and presentation approved for public release, LA-UR-15-21345.)
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 3 of 18
• In many industries, simulation is usually restricted to analyzing a single, deterministic model.
• “Quality” of a model is usually limited to its ability to reproduce a few past observations.
This topic is of great importance because mediocre uncertainty management practices in numerical simulation yield bad decisions.
• What if future scenarios do not reproduce the past?
• What is, in your opinion, a “confident” decision?
Hurricane Irene (August 2011)
“The model […] is based on certain assumptions where we do not have clear scientific evidence.”
Iceland Ash Cloud (April 2010)
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 4 of 18
≈ 2000’s≈ 1980’sCar Crash Simulations
1.0o-grid Resolution
0.1o-grid Resolution
Global Circulation Models
Computing power, and the accompanying evolution of analysis software, has brought unprecedented fidelity to the simulations.
• Better mechanics, better physics
• Better coupling of phenomena
• More resolution of calculations
• More accurate numerical solvers
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 5 of 18
Climate Modeling
Asteroid Impact
Molecular Dynamics
The potential for introducing uncertainty in numerical simulations, however, is endless …• Environmental variability
• Manufacturing variability
• Material property uncertainty
• Mesh-induced truncation effects
• Many other numerical errors …
• Unknown source functions
• Unknown energy dissipation
• Approximate coupling of physics
• Assumptions of sub-scale behavior
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 6 of 18
The treatment of uncertainty is usually restricted to probabilistic variability caused by “randomness”.
• Uncertainty in numerical simulations is overwhelmingly treated as variability.
• Successes with probability and statistics have been demonstrated for many disciplines.
• Because probability theory is so powerful, analysts tend to apply it to every source of uncertainty. Is it the right thing to do?
100 microns
Unit-to-unit Repeatability
Micro-structure Variability
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 7 of 18
Code “A”(University of Chicago)
Code “B”(Los Alamos)
Credit: LA-UR-07-2575. AMR fluid dynamics simulations starting from the same grid with 640-by-1,280 cells; density contours shown at the same time.
Not everything, however, is variability …The differences between these predictions of a rising bubble, for example, are not random.
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 8 of 18
Can Your Code Simulate This?
“Assumption-making≠
Randomness”
Likewise, formulating modeling assumptions reflects a special kind of uncertainty that is fundamentally different from “randomness”.
• Eulerian or Lagrangian frame?
• Single pressure or one pressure per material within a zone?
• How is the interface between different materials tracked?
• How are operators, such as the product ∇P ×∇ρ that produces the vorticity term, discretized?
• Does the code conserve internal energy or total energy?
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 9 of 18
Modal Approximation
Fully Coupled Navier-Stokes
LinearAero-elasticity
Scientists and engineers are confronted to three broad classes of uncertainty in model development and numerical simulations.
• Variability and randomness
• Numerical uncertainty
• Model-form uncertainty
Credit: D. Higdon, CCS-6, Los Alamos; K.K. Choi, University of Iowa.
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 10 of 18
Reactor Core Flow Simulation
Contaminated Water Simulation
What are things that this community should be thinking about?• Should the design standards
be kept deterministic?
• Which is the role played by calibration to measurements?
• What sources of uncertainty are considered in an analysis?
• How is “worst case” defined?
• How to establish the credibility of numerical predictions if some of the modeling assumptions are incorrect?
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 11 of 18
Inference of Uncertainty,Validation Metrics
Upper Bounds of Truncation Error
Robustness to Lack-of-knowledge
What kinds of uncertainty management techniques can be “wrapped” around deterministic models and simulations?• Variability and randomness
• Numerical uncertainty
• Model-form uncertainty
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 12 of 18
Flow-chart of Bayesian Inference
Technology developed to deal with “randomness” using probability and statistics is well-developed and mature.• Design of computer experiments
• Variance-based effect screening
• Fast-running statistical emulators
• Sampling methods
• Fast probability integration
• Inference of uncertainty
Credit: D. Higdon, CCS-6, Los Alamos.
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 13 of 18
Metrics, such as Langenbrunner’s DN, are available for quantitative comparisons that account for the prediction/data uncertainty.
Credit: J. Langenbrunner, XCP-8, Los Alamos, LA-UR-15-23453.
Quantitative Comparison2
1
( ; )1( )
TestNi i
i iN
y y ps
D N kθ
=
−
=− ∑
Predictions(+ Uncertainty)
Measurements(+ Uncertainty)
Numerical Simulation
PhysicalExperiment
Application of the DN Metric to the Simulation of Structural Vibrations
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 14 of 18
U(Δx)
DiscreteSolutions, y(Δx)
Numerical uncertainty, that originates from the discretization of continuous equations on a computational mesh, is rarely quantified.
Credit: LA-UR-07-4221.
• There is strong evidence that typical meshes used in industrial applications are significantly under-resolved.
• A prediction reported without bounds of numerical uncertainty is meaningless.
UNCLASSIFIED
UNCLASSIFIED
Operated by Los Alamos National Security, L.L.C. for the U.S. Department of Energy's NNSA
LA-UR-15-21345 | Page 15 of 18
Performance, RM
odel
ing
Erro
r, α
RequirementCompliant
FailureDomain
Managing model-form uncertainty is a matter of understanding the effect that assumptions exercise on model-informed decisions.• Shift the focus away from
“quantification” and, instead, towards decision-making.
• An intuitive and powerful communication tool is the robustness function.
• How “wrong” can the assumptions be while still guaranteeing that the performance requirement is met?
![Networking in the 21st Century - [short highlevel virtual view]](https://static.documents.pub/doc/80x56/58d1704b1a28abed798b4ad1/networking-in-the-21st-century-short-highlevel-virtual-view.jpg)
