Model-Based Rigorous UncertaintyQuantification in Complex Systems
Michael OrtizMichael OrtizCalifornia Institute of Technology
Applied Mathematics SeminarWarwick Mathematics Institute, June 23, 2010
Work done in collaboration with: Marc Adams, Addis Kadani, Bo Li, Mike McKerns, Ali Lashgari, Jon Mihaly, Houman
PSAAP: Predictive Science Academic Alliance Program
o , e c e s, as ga , Jo a y, ou aOwhadi, G. Ravichandran, Ares J. Rosakis, Tim Sullivan
Michael OrtizWarwick 06/23/10- 1
ASC/PSAAP Centers
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizPSAAP Peer Review, 10/23/2008- 2
Quantification of margins and uncertainties (QMU)( )
• Aim: Predict mean performance and uncertainty in the behavior of complex physical/engineered systems
• Example: Short-term weather prediction,– Old: Prediction that tomorrow will rain in Warwick…
New: Guarantee same with 99% confidence– New: Guarantee same with 99% confidence…
• QMU is important for achieving confidence in high-consequence decisions, designs
• Paradigm shift in experimental science, modeling and simulation, scientific computing (predictive science):
Deterministic Non deterministic systems– Deterministic → Non-deterministic systems– Mean performance → Mean performance + uncertainties– Tight integration of experiments, theory and simulation
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 3
– Robust design: Design systems to minimize uncertainty– Resource allocation: Eliminate main uncertainty sources
Certification view of QMU
system inputs
performance measures
responsefunctioninputs measures
• Random variables
• Known or
• Observables• Subject to
performance• Known or unknown pdfs
• Controllable,
performance specs
• Random due
S t bl k b
uncontrollable, unknown-unknowns
to randomnessof inputs or of system
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 4
System as black boxunknowns system
Hypervelocity impact as an example of a complex systemp y
Challenge: Predict hypervelocity impact phenomena (10Km/s ) with quantified margins and uncertainties
NASA Ames Research Center E fl h f h l it t tEnergy flash from hypervelocity test
at 7.9 Km/s
PSAAP: Predictive Science Academic Alliance Program
Hypervelocity impact test bumper shield(Ernst-Mach Institut, Freiburg Germany)
Michael OrtizWarwick 06/23/10- 5
Hypervelocity impact as an example of a complex systemp y
• Hypervelocity impact is of interest to a broad scientific community: Micrometeorite shields, geological impact cratering…
Hypervelocity impact test of The International Space Station uses
PSAAP: Predictive Science Academic Alliance Program
multi-layer micrometeorite shield 200 different types of shield to protect it from impacts
Michael OrtizWarwick 06/23/10- 6
Hypervelocity impact at Caltech
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 7
Caltech’s Small Particle Hypervelocity Impact Range facility(A.J. Rosakis, Director)
Hypervelocity impact at Caltech
BREECH PUMP TUBE
LAUNCH TUBE
FLIGHT TUBE
TARGET TANK
Front Spall Cloud
Debris Cloud
Witness Plate
STAGE 1 STAGE 2α
Impactor
TargetWitness Plate
aluminum witness plates replaced by capture media
Target Materials•SteelAluminum Ø 71 mil (1x10-3 in)
by capture media
• Impact Speeds: 2 to 10 km/sI t Obli iti 0 t 80 d
•Aluminum•Tantalum
Impactor Materials•Steel
Ø 71 mil (1x10 3 in) launch tube bore
PSAAP: Predictive Science Academic Alliance Program 8
• Impact Obliquities: 0 to 80 degrees• Impactor Mass: 1 to 50 mg
Steel•Nylon
Michael OrtizWarwick 06/23/10- 8
Hypervelocity impact as system
System Outputs (Y)System inputs (X)
Diagnostics
Conoscope
MetricsProfilometry
Projectile velocity
Projectile mass Conoscope
CGS
yPerforation area
Real-time, full-field back-surface deformation
Projectile mass
Number of target plates
VISAR
S t
deformation
Real-time back-surface velocimetry
I t fl h d b i d
plates
Plate thicknesses
Spectro-photometer
Capture media
Impact flash, debris and spall clouds, spectra over IR to UV range
Debris & spall clouds, Particle consistency
Plate obliquities
Projectile/plate
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz
Capture media Particle consistency, size & velocity vector
j pmaterials
Warwick 06/23/10- 9
Certification view of QMU
• Certification = Rigorous guarantee that complex system will perform safely and according to specifications
• Certification criterion: Probability of failure must be below tolerance,
• Alternative (conservative)Alternative (conservative) certification criterion: Rigorous upper bound of probability of failure must be below tolerancemust be below tolerance,
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 11
• Challenge: Rigorous, measurable/computable upper bounds on the probability of failure of systems
Concentration of measure (CoM)
• CoM phenomenon (Levy, 1951): Functions over1951): Functions over high-dimensional spaces with small local oscillations in each variable are almost constant
• CoM gives rise to a classCoM gives rise to a class of probability-of-failure inequalities that can be
d f iused for rigorous certification of complex systems
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 12
Paul Pierre Levy (1886-1971)
The diameter of a function
• Oscillation of a function of one variable:
• Function subdiameters:
• Function diameter:evaluation requires
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 13
qglobal optimization!
McDiarmid’s inequality
McDiarmid C (1989) “On the method of bounded differences” In J Simmons (ed ) Surveys in
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 14
McDiarmid, C. (1989) On the method of bounded differences . In J. Simmons (ed.), Surveys inCombinatorics: London Math. Soc. Lecture Note Series 141. Cambridge University Press.
McDiarmid’s inequality
• Bound does not require distribution of inputs• Bound depends on two numbers: Function
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 15
pmean and function diameter!
McDiarmid’s inequality and QMU
Probability of failure Upper bound Failure tolerance
• Equivalent statement (confidence factor CF):
• Rigorous definition of margin (M)
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 16
• Rigorous definition of uncertainty (U)
Extension to empirical mean
• Equivalent statement (confidence factor CF):
• Rigorous definition of margin (margin hit!)
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 17
Rigorous definition of margin (margin hit!)• Rigorous definition of uncertainty (U = DG)
Extension to multiple performance measures
• Equivalent statement (confidence factor CF):Equivalent statement (confidence factor CF):
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 18
Multiple performance measures and unknown mean performancep
E i l t t t t ( fid f t CF)• Equivalent statement (confidence factor CF):
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 19
McDiarmid’s inequality and QMU
• Direct evaluation of McDiarmid’s upper bound requires:– Determination of mean performance (e.g., by sampling)– Determination of system diameter by solving a sequence of
global optimization problems
• Viable approach for systems that can be tested cheaplyViable approach for systems that can be tested cheaply• Prohibitively expensive or unfeasible in many cases!
– Tests too costly, time-consuming– Operating conditions are not observable– Political/environmental constraints…
• Alternative: Model-based certification!Alternative: Model based certification!• Challenge: How can we use physics-based models to
achieve rigorous certification with a minimum of testing?
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 25
Model-based QMU – The model
System inputs
performance measures
Modelinputs measures
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 26
Model-based QMU – McDiarmid
• Two functions that describe the system:– Experiment: G(X) F(X) G(X) M d li f tiExperiment: G(X)– Model: F(X)
• Linearity:
F(X)-G(X) ≡ Modeling-error function
• Corollary: A conservative certification criterion is:
y• Triangular inequality:
Corollary: A conservative certification criterion is:
• E[F]: Model mean; E[F-G]: Model mean error• DF: Model diameter (variability of model)
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 27
DF: Model diameter (variability of model)• DF-G: Modeling error (badness of model)
Model-based QMU – McDiarmid
• Working assumptions:– F-G far more regular than F
or G aloneor G alone– Global optimization for DF-G
converges fast– Evaluation of D requires
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 28
– Evaluation of DF-G requires few experiments
Model-based QMU – McDiarmid
• Calculation of DF requires exercising model only
• Evaluation of DF-G requires (few) experiments
• Uncertainty Quantification burden mostly shifted to modeling and simulation!
• Rigorous certification not achievable by modeling and simulation alone!
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 29
g
Model-based QMU – Implementation
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 30
Model-based QMU – Implementation
UQ
Modeling d
ExperimentalS i
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 31
andSimulation
Science
Sample UQ Analysis – Ballistic range
• Target/projectile materials:– Target: Al 6061-T6 plates (6”x 6”)– Projectile: S2 Tool steel balls (5/16”)
• Performance measure (output): Perforation area
Target and projectile
e o at o a ea• Admissible operation range:
Perforation area > 0!M d l t (i t ) • Model parameters (inputs): – Plate thickness (0.032’’-0.063’’)– Impact velocity (100-400 m/s)
• Optimal Transportation Meshfree (OTM) solver (sequential)
• Modifier adaption BFGS; in-
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz
• Modifier adaption, BFGS; inhouse UQ pipeline (Mystic)
04/23/10- 32
OTM simulation
Sample UQ Analysis – Ballistic range
BarrelPressure
Barrel Gun
Light detector(velocity )
D
front view
ZY X
Data recorder LeCroy
OptimetMiniConoscan
3000
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz04/23/10 - 33
rear view
Sample UQ Analysis – Ballistic range
400
300
350
400
y (m
/s)
40
50
60
mm
2)
200
250not perforated
perforated
ctile
Velocity
20
30
40
rated area
(m
32 milli‐inch
40 milli‐inch
50 milli‐inch
100
150
30 40 50 60 70
Projec
0
10
0 100 200 300 400
Perfor 63 milli‐inch
30 40 50 60 70Plate thickness (milli‐inch)Impact Velocity (m/s)
Perforation area vs. impact velocity(note small data scatter!)
Perforation/non-perforationbo ndar
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz04/23/10 - 34
(note small data scatter!) boundary
Sample UQ Analysis – Ballistic range
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz04/23/10 - 35
Computed vs. measured perforation area
Sample UQ Analysis – Ballistic range
OTM simulations
/s)
experimental
OTM simulations
a (m
m2 )
velo
city
(m perforation
ratio
n ar
ea
vno perforation
Per
for
OTM simulationsexperimental
thickness (thousands of an inch)
p
velocity (m/s)
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz04/23/10 - 36
Measured vs. computed perforation area
Sample UQ Analysis – Ballistic range
Modelthickness 4.33 mm2
velocity 4.49 mm2operating range(180 400 m/s)
50
60
m2)
diameter DF
ytotal 6.24 mm2
Modelingthickness 4.96 mm2
velocity 2 16 mm2
(180-400 m/s)
20
30
40
ed area (m
m
32 milli‐inch40 milli‐i h
Modeling error DF-G
velocity 2.16 mm2
total 5.41 mm2
Uncertainty DF + DF-G11.65 mm2
0
10
20
Perforate inch
50 milli‐inch
y F F G
Empirical mean <G> 47.77 mm2
Margin hit α (ε’=0.1%) 4.17 mm2
0 100 200 300 400
Impact Velocity (m/s)Confidence factor M/U 3.74
• Perforation can be certified with ~ 1-10-12 confidence!
PSAAP: Predictive Science Academic Alliance Program
Michael Ortiz04/23/10 - 37
Perforation can be certified with 1 10 confidence!• Total number of tests ~ 50 → Approach feasible!
Beyond McDiarmid - Extensions
• A number of extensions of McDiarmid may be required in practice:– Some input parameters cannot be controlled– There are unknown input parameters (unknown unknowns)– There is experimental scatter (G defined in probability)There is experimental scatter (G defined in probability)– McDiarmid may not be tight enough (convergence?)– Model itself may be uncertain (epistemic uncertainty)
D t t b il bl ‘ d d’ (l d t )– Data may not be available ‘on demand’ (legacy data)
• Extensions of McDiarmid that address these challenges include:– Martingale inequalities (unknown unknowns, scatter…)– Partitioned McDiarmid inequality (convergent upper bounds)
Optimal Uncertainty Quantification (OUQ)
PSAAP: Predictive Science Academic Alliance Program
– Optimal Uncertainty Quantification (OUQ)– Optimal models (least epistemic uncertainty
Michael OrtizWarwick 06/23/10- 38
Beyond McDiarmid – Scatter
• Added challenges:– Experimental
!10
11
12
Plot assumes average of obliquities (a) for all targets tested of a given thickness (h)
Vbl = Ho*( h/Dp)^n / (cos a)^sA = if V<Vbl, 0, K*Dp^2 * {(h/Dp)^p} * {[cos(a)]^u} * {tanh[(V/Vbl)-1]}^m
A = perforation area (mm^2)) scatter!– Impact velocity
uncontrollable!7
8
9
ea (m
m)
A = perforation area (mm 2)h = target thickness (mm)a = imapct obliquity (rad)V = impact speed (km/s)Dp = imactor diameter (mm)Ho, K, n, s, p, u, m are curve fit parametersre
a (m
m2 )
Distribution of Speeds Obtained
0 8
0.9
1
peed
Fit Errors: Uniform = 0.32 Gaussian = 0.08
GaussianMean = 2.49StDev = 0.25
Measured speed distribution
bilit
y
4
5
6
Perfo
ratio
n A
re
1.9mm Data
2.3mm Data
2 6mm Datarfora
tion
a
0.4
0.5
0.6
0.7
0.8
roba
bilit
y of
Low
er S
pat
ive
prob
ab
1
2
32.6mm Data
1.9mm Model
2.3mm Model
2.6mm Model
Impactor area
Per
0
0.1
0.2
0.3
0.4
Cum
ulat
ive
PrC
umul
a
01.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3
Impact Speed (km/s)Impact speed (km/s)Experimental ballistic curves (SPHIR)
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 39
1.6 1.8 2 2.2 2.4 2.6 2.8 3
Impact Speed (km/sImpact speed (km/s)Experimental ballistic curves (SPHIR)
440 C Steel spherical projectiles304 Stainless Steel plate targets
Beyond McDiarmid – Scatter
known controllable performance Response
inputs performance
measuresfunction
uncontrollableinputs
& unknown unknown unknowns
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 40
Beyond McDiarmid – Scatter
• Let denote averaging with respect to uncontrollable variables and unknown unknowns
• Let be the fluctuation
• Theorem [Lashgari Owhadi MO] A conservative• Theorem [Lashgari, Owhadi, MO] A conservative certification criterion is:
f i t l tt !• Simulations and experiments must be averaged
wrt uncontrolled variables and unknown unknowns
measure of experimental scatter!
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 41
wrt uncontrolled variables and unknown unknowns• Data scatter contributes to uncertainty!
Beyond McDiarmid – Scatter
Model D‹F›
thickness 1.82 mm2
obliquity 2.41 mm2
Di t
‹F› q ytotal 3.02 mm2
Modeling errorthickness 1.80 mm2
obliquity 4 50 mm2Diameters Modeling error D‹F-G›
obliquity 4.50 mm2
total 4.85 mm2
Experimentaltt D
total 7.78 mm2
scatter DG’
Mean values
Model E[F] total 3.30 mm2
Modeling error total 0.32 mm2
Steel-on-steel, 2.6 km/sPerforation and impactor
E[F-G]
• Perforation cannot be certified with any reasonablefid !
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 42
confidence!
Beyond McDiarmid - Partitioning
1 1
cliff!operating
range
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 43
Beyond McDiarmid - Partitioning
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 44
Beyond McDiarmid – Optimal UQ
• What is the least probability of failure upper bound given what is known about the system?
• Best probability of failure upper bound given that probability μ of inputs and response function G are in a set A:set A:
• Can be reduced, to finite-dimensional optimization (Choquet theory, representation of linear functionals by measures on extreme points moment problems )measures on extreme points, moment problems…)
• Example: Mean performance and diameter known• Explicit solutions for finite-dimensional inputs (Owhadi et
PSAAP: Predictive Science Academic Alliance Program
Michael OrtizWarwick 06/23/10- 45
Explicit solutions for finite dimensional inputs (Owhadi et al.), optimal McDiarmid-type inequalities!
Concluding remarks…
• QMU represents a paradigm shift in predictive science:– Emphasis on predictions with quantified uncertaintiesp p q– Unprecedented integration between simulation and experiment
• QMU supplies a powerful organizational principle in predicti e science Theorems r n entire centers!predictive science: Theorems run entire centers!
• QMU raises theoretical and practical challenges:– Tight and measureable/computable probability-of-failure upper g / p p y pp
bounds (need theorems!)– Efficient global optimization methods for highly non-convex,
high-dimensionality, noisy functionsg y, y– Effective use of massively parallel computational platforms,
heterogeneous and exascale computing– High-fidelity models (multiscale effective behavior )
PSAAP: Predictive Science Academic Alliance Program
High fidelity models (multiscale, effective behavior…)– Experimental science for UQ (diagnostics, rapid-fire testing…)…
Michael OrtizWarwick 06/23/10- 46