21
Calibration of Computational Models with Categorical Parameters and Correlated Outputs via Bayesian Smoothing Spline ANOVA Curtis Storlie Los Alamos National Laboratory June 7, 2013 TM
Calibration of Computational Models with Categorical Parameters and Correlated Outputs via Bayesian Smoothing Spline ANOVA
Curtis Storlie Los Alamos National Laboratory June 7, 2013
TM
2
TM
For Accelerating Technology Development
National Labs Academia Industry
Identify promising concepts
Reduce the time for design &
troubleshooting
Quantify the technical risk, to enable reaching larger
scales, earlier
Stabilize the cost during commercial deployment
What is Computer Model Calibration?
! Find a plausible set of modelparameter values (!) thatbest produce the reality ofexperimental (or field) data.
! In the Bayesian paradigm,this entails putting a priordistribution on ! andconditioning on theexpirimental data to refinethis prior distribution.
! There can also be a modelform discrepancy functionwhich admits the possibilityof model bias.
Bayesian Calibration
!"#"$%&%#'(
!#)*#'+),&#)-.&)*/,
!"#"$%&%#'! '+),0#%1"/02
341%#)$%/&"5'+"&"
!"#"$%&%#'(
!*,&%#)*#'+),&#)-.&)*/,
!"#"$%&%#'! '+),0#%1"/02
TM
“Traditional” (Kennedy & O’Hagan 2001) Calibration! Represent the output of the physical system producing the
experimental data as
yn = !(xn,!) + "(xn) + #n, n = 1, . . . ,N.
(i) !(xn, t) is a simulator of the physical system.
(ii) "(x) is a discrepancy function to alow for model bias.
(iii) #niid! N (0,$2) are observational measurement errors.
(iv) t = [t1, . . . , tQ ] is a vector of model parameters. If fixed at anappropriate (unknown) value of t = !, then !(x,!) will bestapproximate the physical system.
! Typically it is assumed that " is a Gaussian Process (GP)
! If simulator runs are expensive, then a sample (e.g., LHS) of runs isobtained and ! is modeled as a GP as well.
! Estimation of !, !, and " is done within a Bayesian framework(Higdon, Kennedy, Cavendish, Cafeo & Ryne 2004).
TM
Bubbling Fluidized Bed Example
Goal: Calibration of a computational fluid dynamics (CFD) model as afirst step toward upscaling to a large CO2 capture system.
Experimental Setup CFD simulation Setup
!!"#$%!&'
(!&'
%(!&'
(!&' (!&'
)!&'
*+,-.+/0-1.!234-5
67!&'
87!&'
9.55/14.:
2.1/5 2.1/5
!;
";
#
!"#!
!$%!#!
!"#!
&"#!&$%!
#!
!$%!
!"#!
&"#!&$%!
#!
!$%!
! ! ! (a) (b)
TM
Bubbling Fluidized Bed Example
! Experimental Outputs y:y1 : Bubble Frequency (measured in Hertz)
observed at angles {!90,!45, 0, 45, 90}! and velocities{5.5, 7.0, 11.0, 12.6} cm/sec.
y2 : Phase Fraction (proportion of time a bubble is present)observed at angles {!90,!45, 0, 45, 90}! and velocity 12.6 cm/sec).
! Experimental Inputs x:x1 : Gas Velocity, [5.5, 16.1]x2 : Angular Location on Tube, ["90, 90]
! Model Parameters t:t1 : Coe!cient of restitution, particle-particle # [0.8, 0.997]t2 : Coe!cient of restitution, particle-wall # [0.8, 0.997]t3 : Friction angle, particle-particle # [25.0, 45.0]t4 : Friction angle, particle-wall # [25.0, 45.0]t5 : Packed bed void fraction # [0.3, 0.4]t6 : Drag model # {Syamlal-OBrien, Wen-Yu, Gidaspow}
TM
Bubbling Fluidized Bed Example
! Latin Hypercube Sample (LHS) of 90 runs was used to make CFDmodel runs.
! Each run produced (after post-processing) the y1 and y2 values atangles x2 = {"90.0,"67.5,"45.0,"22.5, 0.0, 22.5, 45.0, 67.5, 90.0}.
! Seven free “parameters” to choose values for are then(x1, t1, t2, . . . , t6).
! x1 was restricted to values where there were experimental datax1 # {5.5, 7.0, 11.0, 12.6} cm/sec.
! So all in all there are:– experimental observations for y1 at 4 velocities (each at 5 angles)– experimental observations for y2 at 1 velocity (each at 5 angles)– 90 CFD runs total covering four distinct velocites (each run provides
output at 9 distinct angles)
TM
Bubbling Fluidized Bed Example
!
!!
!
!!
! ! !
!
!!
!
!!
! ! !
!
!!
!
! !! ! !
!
!!
!
!!
! ! !
!
!!
!
!
!
! ! !
!!!
!
!
!
! ! !
−50 0 50
01
23
45
6
Gas Velocity = 5.5
Angle
Bubb
le F
requ
ency
x x xx x
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!!
!
!
!
! ! ! !
!!
!
!
!
!
! ! !
−50 0 50
01
23
45
6
Gas Velocity = 7
Angle
Bubb
le F
requ
ency
x x xx x
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!
!!
!
!
!
!! !
!
!!
!
!
!
! ! !
!
!
!
!
!
!
!
! !
!
!!
!
!
!
!! !
−50 0 50
01
23
45
6
Gas Velocity = 11
Angle
Bubb
le F
requ
ency
x xx
x x
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!
!!!
!
!
!!
!
!
!
!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!
!
!
!
!
!!
!
!!!
!
!
!
!
! !
!!
!!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!
!
!
!
!
!! !
!
!!
!
!
!
!!
!
!!!
!
!
!
!!
!
!
!!
!
!
!
!!
!
!
!!!
!
!
! !
!
!!
!
!
!
!
!
! !
!
!!
!
!
!
!!
!
!
!!
!
!
!
! !!
!
!!
!
!
!
!!
!
−50 0 50
01
23
45
6
Gas Velocity = 12.6
Angle
Bubb
le F
requ
ency
x xx
x x
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
TM
Bubbling Fluidized Bed Example
!!!!! ! ! ! !
!!!
!! ! ! ! !
!!!
!! ! ! ! !
!!!
!! ! ! ! !
!!!
!! ! ! ! !
!!!!! ! ! ! !
−50 0 50
0.0
0.1
0.2
0.3
0.4
0.5
Gas Velocity = 5.5
Angle
Phas
e Fr
actio
n
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!
!!!
!! ! ! !
!
!!
!!
!! ! !
−50 0 50
0.0
0.1
0.2
0.3
0.4
0.5
Gas Velocity = 7
Angle
Phas
e Fr
actio
n
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!!
!
!
!
!
! ! !
!
!!
!
!
!! ! !
!!!
!
!
!
!! !
!
!!
!
!
!
! ! !
−50 0 50
0.0
0.1
0.2
0.3
0.4
0.5
Gas Velocity = 11
Angle
Phas
e Fr
actio
n
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
!
!!
!
!
!
!! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
!! !
!
!!
!
!
!!
!!
!
!!
!
!
!
! ! !
!
!!
!
!
!
!! !
!!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
!! !
!!!
!!
!
! ! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
! ! !
!
!!
!
!
!
!! !
!
!!
!
!
!
!! !
!
!!
!
!
!! ! !
!
!!
!
!
!
!!
!
−50 0 50
0.0
0.1
0.2
0.3
0.4
0.5
Gas Velocity = 12.6
Angle
Phas
e Fr
actio
n xx
x
xx
!
Drag Model−−−−−−−−−−−−−−−GidaspowSyamlalWen
TM
Common Complications in Computer Model Calibration
1. There are multiple correlated outputs (e.g., Bubble Frequency andVoid Fraction) so the observations (yn) are really vectors.
2. There are categorical model parameters (e.g., which Drag model isused inside the CFD model).
3. There may be multiple possible models ! (e.g., a coarseapproximation that runs much faster than a more accurate highresolution model. Not in the bubbling bed example, however.)
4. There may be some missing experimental observations for some ofthe outputs. (e.g., not all outputs were measured in all trials, ordata is combined from multiple sources).
TM
Calibration w/ Multiple Outputs & Categorical Parameters
A multivariate output of the physical system y = [y1, . . . , yM ]T is now avector of simulator outputs plus a multivariate discrepancy function ",plus the measurement error vector, ", i.e.,
yn = !(xn,!) + "(xn) + "n, n = 1, . . . ,N.
! e.g., !(xn,!) = [!1(xn,!), . . . , !M(xn,!)]T
! !, " and "n need a multivariate representation to appropriatelyaccount for correlation among the multiple outputs.
! The emulator also needs to account for categorical parameters (e.g.,which drag model to use).
! These will be accomplished within the Bayesian Smoothing Spline(BSS-)ANOVA GP (Reich, Storlie & Bondell 2009, Storlie, Fugate,Higdon, Huzurbazar, Francois & McHugh 2012).
TM
BSS-ANOVA Model
! We assume the emulator for ! (and the discrepancy " with obviouschanges) is a GP with the BSS-ANOVA covariance function.
! This GP can be conveniently written as a sum of main e"ects plusinteraction components, i.e.,
!(x) = %0 +J!
j=1
!j(xj) +J!
j<j "
!j ,j "(xj , xj ") + · · · (1)
! Each functional component in (1) can be further written as anorthogonal basis expansion via Karhunen-Loeve, e.g.,
!j(xj) =P!
p=1
%p,j&p(xj), with %p,jiid! N (0, '2j ). (2)
TM
BSS-ANOVA Model: Basis Functions! The &p get increasingly higher frequency and have decreasingly less
magnitude, so the expansion can be truncated at some value P .
0.0 0.2 0.4 0.6 0.8 1.0
−0.4
−0.2
0.0
0.2
0.4
Phi_1(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
50.
050.
100.
15
Phi_2(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
3−0
.01
0.01
0.03
Phi_3(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
3−0
.01
0.01
0.03
Phi_4(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
050.
000
0.00
5
Phi_5(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
050.
000
0.00
5
Phi_6(x)
0.0 0.2 0.4 0.6 0.8 1.0−0
.004
0.00
00.
002
0.00
4 Phi_7(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
040.
000
0.00
20.
004 Phi_8(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
020.
000
0.00
2
Phi_9(x)
0.0 0.2 0.4 0.6 0.8 1.0
−0.0
020.
000
0.00
2
Phi_10(x)
0.0 0.2 0.4 0.6 0.8 1.0−0.0
015
−0.0
005
0.00
050.
0015 Phi_11(x)
0.0 0.2 0.4 0.6 0.8 1.0−0.0
015
−0.0
005
0.00
050.
0015 Phi_12(x)
TM
BSS-ANOVA Model Advantages
! This is just a linear model in the %’s! Just need to estimate the %’sand the discrepancy function is analytically specified by (1) and (2).
! Categorical parameters can be easily treated (Storlie, Reich, Helton,Swiler & Sallaberry 2013). Multiple models can be treated as levelsof a categorical parameter.
! O(J2(N +M)) computational e!ciency for the MCMC algorithm asopposed to O((N +M)3) for the traditional squared exponentialcovariance GP, where N +M is the total number of experimentalobservations plus simulator runs.
! Analytic forms are also nice for portability from one problem to thenext (calibration $ uncertainty propogation, or upscaling,...).
! Conjugate priors (i.e., inverse Wishart) for the variance terms ('j)leads to Gibbs sampling for all parameters in the model, with theexception that MH updates are needed for the elements of !.
TM
Bubbling Fluidized Bed: Theta Trace Plots
TM
Bubbling Fluidized Bed: Theta PosteriorRes−PP
dens
ity
0.80 0.85 0.90 0.95 1.00
02
46
810
Res−PW
dens
ity
0.80 0.85 0.90 0.95 1.00
02
46
8
FricAng−PP
dens
ity
25 30 35 40 45
0.0
0.2
0.4
0.6
0.8
FricAng−PW
dens
ity
25 30 35 40 45
0.00
0.02
0.04
0.06
0.08
0.10
PBVF
dens
ity
0.30 0.32 0.34 0.36 0.38 0.40
05
1015
20
Gidaspow Syamlal Wen−Yu
DragMod
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
TM
Bubble Frequency Fitted Plots
!50 0 50
01
23
45
Simulator+Discrepancy, Velocity=5.5
Angle
Bubb
le F
requ
ency
XX
Emulator MeanEmulator RealsEmu + Dis MeanEmu + Dis RealsObserved DataMissing Data
x xx
x x!50 0 50
01
23
45
Simulator+Discrepancy, Velocity=7.0
Angle
Bubb
le F
requ
ency
XX
Emulator MeanEmulator RealsEmu + Dis MeanEmu + Dis RealsObserved DataMissing Data
x xx
x x
!50 0 50
01
23
45
Simulator+Discrepancy, Velocity=11.0
Angle
Bubb
le F
requ
ency
XX
Emulator MeanEmulator RealsEmu + Dis MeanEmu + Dis RealsObserved DataMissing Data
x xx
x x!50 0 50
01
23
45
Simulator+Discrepancy, Velocity=12.6
Angle
Bubb
le F
requ
ency
XX
Emulator MeanEmulator RealsEmu + Dis MeanEmu + Dis RealsObserved DataMissing Data
x xx
x x
TM
Bubbling Fluidized Bed: Discrepancy Plots
6 8 10 12
−1.0
−0.5
0.0
Delta.1.1 by x.1
Velocity
Bubb
le F
requ
ency
Dis
crep
ancy
Posterior Mean95% BandsZero LinePosterior Realizations
−50 0 50
−2.5
−1.5
−0.5
0.5
Delta.2.1 by x.2
Angle
Bubb
le F
requ
ency
Dis
crep
ancy
Posterior Mean95% BandsZero LinePosterior Realizations
6 8 10 12
0.00
0.10
0.20
0.30
Delta.1.2 by x.1
Velocity
Phas
e Fr
actio
n D
iscr
epan
cy
Posterior Mean95% BandsZero LinePosterior Realizations
−50 0 50
−0.2
0.0
0.1
0.2
0.3
0.4
Delta.2.2 by x.2
Angle
Phas
e Fr
actio
n D
iscr
epan
cy
Posterior Mean95% BandsZero LinePosterior Realizations
TM
Bubbling Fluidized Bed: Cross Validation Plots
−50 0 50
01
23
45
67
Simulator+Discrepancy, Velocity=5.5
Angle
Bubb
le F
requ
ency
X
Emulator MeanEmulator 95% BandsSim + Dis MeanSim + Dis 95% BandsData
x xx
x x−50 0 50
01
23
45
6
Simulator+Discrepancy, Velocity=7.0
Angle
Bubb
le F
requ
ency
X
Emulator MeanEmulator 95% BandsSim + Dis MeanSim + Dis 95% BandsData
x xx
x x
−50 0 50
01
23
45
Simulator+Discrepancy, Velocity=11.0
Angle
Bubb
le F
requ
ency
X
Emulator MeanEmulator 95% BandsSim + Dis MeanSim + Dis 95% BandsData
x x
x
x x−50 0 50
01
23
45
Simulator+Discrepancy, Velocity=12.6
Angle
Bubb
le F
requ
ency
X
Emulator MeanEmulator 95% BandsSim + Dis MeanSim + Dis 95% BandsData
x xx
x x
TM
ReferencesHigdon, D., Kennedy, M., Cavendish, J., Cafeo, J. & Ryne, R. (2004),
‘Combining field data and computer simulations for calibration andprediction’, SIAM Journal on Scientific Computing 26, 448–466.
Kennedy, M. & O’Hagan, A. (2001), ‘Bayesian calibration of computermodels (with discussion)’, Journal of the Royal Statistical Society B63, 425–464.
Reich, B., Storlie, C. & Bondell, H. (2009), ‘Variable selection inBayesian smoothing spline ANOVA models: Application todeterministic computer codes’, Technometrics 51, 110–120.
Storlie, C. B., Reich, B., Helton, J., Swiler, L. & Sallaberry, C. (2013),‘Analysis of computationally demanding models with continuous andcategorical inputs’, Reliability Engineering and System Safety (inpress) 113, 30–41.
Storlie, C., Fugate, M., Higdon, D., Huzurbazar, A., Francois, E. &McHugh, D. (2012), ‘Methods for characterizing and comparingshock wave curves’, Technometrics (in press) .
TM
3
TM
!"#$%&'($()*+,-)%.+$%&'(&+'(/%+$%+)%+00-1)*%-2%.-'3%$&-)$-'(/%45%+)%+6()05%-2%*"(%7)#*(/%8*+*($%9-:(');()*<%=(#*"('%*"(%7)#*(/%8*+*($%9-:(');()*%)-'%+)5%+6()05%*"('(-2>%)-'%+)5%-2%*"(#'%(;&?-5(($>%;+3($%+)5%.+''+)*5>%(@&'($$%-'%#;&?#(/>%-'%+$$1;($%+)5%?(6+?%?#+4#?#*5%-'%'($&-)$#4#?#*5%2-'%*"(%+001'+05>%0-;&?(*()($$>%-'%1$(21?)($$%-2%+)5%#)2-';+,-)>%+&&+'+*1$>%&'-/10*>%-'%&'-0($$%/#$0?-$(/>%-'%'(&'($()*$%*"+*%#*$%1$(%.-1?/%)-*%#)2'#)6(%&'#:+*(?5%-.)(/%'#6"*$<%A(2('()0(%"('(#)%*-%+)5%$&(0#B0%0-;;('0#+?%&'-/10*>%&'-0($$>%-'%$(':#0(%45%*'+/(%)+;(>%*'+/(;+'3>%;+)12+0*1'('>%-'%-*"('.#$(%/-($%)-*%)(0($$+'#?5%0-)$,*1*(%-'%#;&?5%#*$%()/-'$(;()*>%'(0-;;()/+,-)>%-'%2+:-'#)6%45%*"(%7)#*(/%8*+*($%9-:(');()*%-'%+)5%+6()05%*"('(-2<%!"(%:#(.$%+)/%-&#)#-)$%-2%+1*"-'$%(@&'($$(/%"('(#)%/-%)-*%)(0($$+'#?5%$*+*(%-'%'(C(0*%*"-$(%-2%*"(%7)#*(/%8*+*($%9-:(');()*%-'%+)5%+6()05%*"('(-2<%
Thank you!
Disclaimer
