+ All Categories
Home > Documents > BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference &...

BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference &...

Date post: 16-Oct-2020
Category:
Upload: others
View: 6 times
Download: 0 times
Share this document with a friend
119
BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction Running the GPM/SA Code Dave Higdon, Brian Williams & Jim Gattiker Statistical Sciences Group, LANL h = 3.5 cm h = 3.8 cm h = 4 cm h = 4 cm h = 4.1 cm h = 4.4 cm 1
Transcript
Page 1: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

BAYESIAN MODELING AND CALIBRATION

OF COMPUTER MODELS

Bayesian inference & Markov chain Monte CarloGaussian processes,Computer model calibration and predictionRunning the GPM/SA CodeDave Higdon, Brian Williams & Jim Gattiker Statistical SciencesGroup, LANL

h = 3.5 cm h = 3.8 cm h = 4 cm h = 4 cm h = 4.1 cm h = 4.4 cm

1

Page 2: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

BRIEF INTRO TO BAYES

2

Page 3: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bayesian inference - iid N (µ, 1) example

Observed data y are a noisy versions of µ

y(si) = µ + ǫi with ǫiiid∼ N(0, 1), k = 1, . . . , n

0 2 4 6 8

−5

0

5

i

y i

sampling model prior for µ

L(y|µ) ∝ ∏ni=1 exp{−1

2(yi − µ)2} π(µ) ∝ N(0, 1/λµ), λµ small

posterior density for µ

π(µ|y) ∝ L(y|µ) × π(µ)

∝ n∏

i=1exp{−1

2(yi − µ)2} × exp{−1

2λµµ

2}

∝ exp

1

2

[n(µ − yn)

2 + λµ(µ − 0)2] × f(y)

⇒ µ|y ∼ N

ynn + 0 · λµ

n + λµ,

1

n + λµ

λµ→0→ Nyn,

1

n

3

Page 4: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bayesian inference - iid N (µ, λ−1y ) example

Observed data y are a noisy versions of µ

y(si) = µ+ǫi with ǫiiid∼ N(0, λ−1

y ), k = 1, . . . , n

0 2 4 6 8

−5

0

5

i

y i

sampling model prior for µ, λy

L(y|µ) ∝ ∏ni=1 λ

12y exp{−1

2λy(yi − µ)2} π(µ, λy) = π(µ) × π(λy)π(µ) ∝ 1, π(λy) ∝ λay−1

y e−byλy

posterior density for (µ, λy)

π(µ, λy|y) ∝ L(y|µ) × π(µ) × π(λy)

∝ λn2y exp{−1

2λy

n∑

i=1(yi − µ)2} × λay−1

y exp{−byλy}

π(µ, λ|y) is not so easy recognize.Can explore π(µ, λ|y) numerically or via Monte Carlo.

4

Page 5: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Full conditional distributions for π(µ, λ|y)

π(µ, λy|y) ∝ λn2y exp{−1

2λy

n∑

i=1(yi − µ)2} × λay−1

y exp{−byλy}

Though π(µ, λ|y) is not of a simple form, its conditional distributions are:

π(µ|λy, y) ∝ exp{−1

2λy

n∑

i=1(yi − µ)2}

⇒ µ|λy, y ∼ N

yn,

1

nλy

π(λy|µ, y) ∝ λay+n2−1

y exp

by +

1

2

n∑

i=1(yi − µ)2

⇒ λy|µ, y ∼ Γay +

n

2, by +

1

2

n∑

i=1(yi − µ)2

.

5

Page 6: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Markov Chain Monte Carlo – Gibbs sampling

Given full conditionals for π(µ, λy|y), one can use Markov chain Monte Carlo(MCMC) to obtain draws from the posterior

The Gibbs sampler is a MCMC scheme which iteratively replaces each parame-ter,in turn, by a draw from its full conditional:

initialize parameters at (µ, λy)0

for t = 1, . . . , niter {

set µ = a draw from N

yn,

1

nλy

set λy = a draw from Γa +

n

2, b +

1

2

n∑

i=1(yi − µ)2

} (Be sure to use newly updated µ when updating λy)

Draws (µ, λy)1, . . . , (µ, λy)

niter are a dependent sample from π(µ, λy|y).

In practice, initial portion of the sample is discarded to remove effect of initial-ization values (µ0, λ0

y).

6

Page 7: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gibbs sampling for π(µ, λy|y)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−1.5

−1

−0.5

0

0.5

1

1.5

2

λy

µ7

Page 8: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

posterior summary for π(µ, λy|y)

0 2 4 6 8

−5

0

5

i

y i

0 1 2 3 4 5−1

−0.5

0

0.5

1

1.5

2

λy

µ

0 200 400 600 800 1000−2

0

2

4

6

8

iteration

µ

λ y

0 2 4 6 8

−5

0

5

i

µ

8

Page 9: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gibbs sampler: intuition

Gibbs sampler for a bivariate normal density

π(z) = π(z1, z2) ∝∣∣∣∣∣∣∣∣

1 ρρ 1

∣∣∣∣∣∣∣∣

−12

exp

−1

2

( z1 z2 )

1 ρρ 1

−1 z1

z2

Full conditionals of π(z):

z1|z2 ∼ N(ρz2, 1 − ρ2)

z2|z1 ∼ N(ρz1, 1 − ρ2)

• initialize chain with

z0 ∼ N

00

,

1 ρρ 1

• draw z11 ∼ N(ρz0

2, 1 − ρ2)

now (z11, z

02)

T ∼ π(z)

u

6

u-

u

z0 = (z01, z

02) (z1

1, z02)

(z11, z

12) = z1

-

6

z1

z2

9

Page 10: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gibbs sampler: intuition

Gibbs sampler gives z0, z2, . . . , zT which can be treated as dependent drawsfrom π(z).

If z0 is not a draw from π(z), then the initial realizations will not have thecorrect distribution. In practice, the first 100?, 1000? realizations are discarded.

The draws can be used to make inferenceabout π(z):

• Posterior mean of z is estimated by:µ1

µ2

=

1

T

T∑

k=1

zk

1

zk2

• Posterior probabilities:

P (z1 > 1) =1

T

T∑

k=1I [zk

1 > 1]

P (z1 > z2) =1

T

T∑

k=1I [zk

1 > zk2 ]

• 90% interval: (z[5%]1 , z

[95%]1 ).

-

6

z1

z2

10

Page 11: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Sampling of π(µ, λ|y) via MetropolisInitialize parameters at some setting (µ, λy)

0.

For t = 1, . . . , T {update µ|λy, y {

• generate proposal µ∗ ∼ U [µ − rµ, µ + rµ].

• compute acceptance probability

α = min

1,

π(µ∗, λy|y)

π(µ, λy|y)

• update µ to new value:

µnew =

µ∗ with probability αµ with probability 1 − α

}update λy|µ, y analagously

}Here we ran for T = 1000 scans, giving realizations (µ, λy)

1, . . . , (µ, λy)T from

the posterior. Discarded the first 100 for burn in.

Note: proposal width rµ tuned so that µ∗ is accepted about half the time;proposal width rλy tuned so that λ∗

y is accepted about half the time.

11

Page 12: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Metropolis sampling for π(µ, λy|y)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−1.5

−1

−0.5

0

0.5

1

1.5

2

λy

µ12

Page 13: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

posterior summary for π(µ, λy|y)

0 2 4 6 8

−5

0

5

i

y i

0 1 2 3 4 5−1.5

−1

−0.5

0

0.5

1

1.5

2

λy

µ

0 200 400 600 800 1000−1

0

1

2

3

4

5

6

iteration

µ

λ y

0 2 4 6 8

−5

0

5

i

µ

13

Page 14: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Sampling from non-standard multivariate distributionsNick Metropolis – Computing pioneer at Los AlamosNational Laboratory

− inventor of the Monte Carlo method

− inventor of Markov chain Monte Carlo:

Equation of State Calculations by Fast ComputingMachines (1953) by N. Metropolis, A. Rosenbluth,M. Rosenbluth, A. Teller and E. Teller, Journal of

Chemical Physics.

Originally implemented on the MANIAC1 computer atLANL

Algorithm constructs a Markov chain whose realiza-tions are draws from the target (posterior) distribu-tion.

Constructs steps that maintain detailed balance.

14

Page 15: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gibbs Sampling and Metropolis for a bivariate normal density

π(z1, z2) ∝∣∣∣∣∣∣∣∣

1 ρρ 1

∣∣∣∣∣∣∣∣

−12

exp

−1

2

( z1 z2 )

1 ρρ 1

−1 z1

z2

sampling from the full conditionals

z1|z2 ∼ N(ρz2, 1 − ρ2)

z2|z1 ∼ N(ρz1, 1 − ρ2)

also called heat bath

Metropolis updating:generate z∗1 ∼ U [z1 − r, z1 + r]

calculate α = min{1, π(z∗1 ,z2)π(z1,z2)

= π(z∗1 |z2)π(z1|z2)

}

set znew

1 =

z∗1 with probability αz1 with probability 1 − α

15

Page 16: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Kernel basis representation for spatial processes z(s)

Define m basis functions k1(s), . . . , km(s).

−2 0 2 4 6 8 10 12

0.0

0.4

0.8

s

basi

s

Here kj(s) is normal density cetered at spatial location ωj:

kj(s) =1√2π

exp{−1

2(s − ωj)

2}

set z(s) =m∑

j=1kj(s)xj where x ∼ N(0, Im).

Can represent z = (z(s1), . . . , z(sn))T as z = Kx where

Kij = kj(si)

16

Page 17: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

x and k(s) determine spatial processes z(s)

kj(s)xj z(s)

−2 0 2 4 6 8 10 12

−0.

50.

51.

5

s

basi

s

−2 0 2 4 6 8 10 12

−0.

50.

51.

5

s

z(s)

Continuous representation:

z(s) =m∑

j=1kj(s)xj where x ∼ N(0, Im).

Discrete representation: For z = (z(s1), . . . , z(sn))T , z = Kx where Kij =

kj(si)

17

Page 18: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Formulation for the 1-d exampleData y = (y(s1), . . . , y(sn))T observed at locations s1, . . . , sn. Once knotlocations ωj, j = 1, . . . ,m and kernel choice k(s) are specified, the remainingmodel formulation is trivial:

Likelihood:

L(y|x, λy) ∝ λn2y exp

1

2λy(y − Kx)T (y − Kx)

where Kij = k(ωj − si).

Priors:

π(x|λx) ∝ λm2x exp

1

2λxx

Tx

π(λx) ∝ λax−1x exp{−bxλx}

π(λy) ∝ λay−1y exp{−byλy}

Posterior:

π(x, λx, λy|y) ∝ λay+n2−1

y exp{−λy[by + .5(y − Kx)T (y − Kx)]

λax+m2 −1

x exp{−λx[bx + .5xTx]

}

18

Page 19: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior and full conditionals

Posterior:

π(x, λx, λy|y) ∝ λay+n2−1

y exp{−λy[by + .5(y − Kx)T (y − Kx)]

λax+m2 −1

x exp{−λx[bx + .5xTx]

}

Full conditionals:

π(x| · · ·) ∝ exp{−1

2[λyx

TKTKx − 2λyxTKTy + λxx

Tx]}

π(λx| · · ·) ∝ λax+m2 −1

x exp{−λx[bx + .5xTx]

}

π(λy| · · ·) ∝ λay+n2−1

y exp{−λy[by + .5(y − Kx)T (y − Kx)]

}

Gibbs sampler implementation

x| · · · ∼ N((λyKTK + λxIm)−1λyK

Ty, (λyKTK + λxIm)−1)

λx| · · · ∼ Γ(ax +m

2, bx + .5xTx)

λy| · · · ∼ Γ(ay +n

2, by + .5(y − Kx)T (y − Kx))

19

Page 20: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

1-d example

m = 6 knots evenly spaced between −.3 and 1.2.n = 5 data points at s = .05, .25, .52, .65, .91.k(s) is N(0, sd = .3)ay = 10, by = 10 · (.252) ⇒ strong prior at λy = 1/.252; ax = 1, bx = .001

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

s

y(s)

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

s

k j(s)

basis functions

0 200 400 600 800 1000−4

−2

0

2

4

6

8

iteration

x j

0 200 400 600 800 10000

10

20

30

40

50

iteration

λ x, λy

20

Page 21: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

1-d example

From posterior realizations of knot weights x, one can construct posterior real-izations of the smooth fitted function z(s) = ∑m

j=1 kj(s)xj.

Note strong prior on λy required since n is small.

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

s

z(s)

, y(s

)

posterior realizations of z(s)

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

sz(

s), y

(s)

mean & pointwise 90% bounds

21

Page 22: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bayesian analysis of an inverse problem

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior uncertainty

• A simple example...x experimental conditionsθ model calibration parametersζ(x) true physical system response given inputs xη(x, θ) forward simulator response at x and θ.y(x) experimental observation of the physical systeme(x) observation error of the experimental data

Assume:y(x) = ζ(x) + e(x)

= η(x, θ) + e(x)θ unknown.

22

Page 23: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bayesian formulation

Sampling model:

yi = η(xi, θ) + ei, where eiiid∼ N(0, 1/λy)

which gives likelihood:

L(y|θ, λy) ∝ λn2y exp

1

2 · .252λy

n∑

i=1(yi − η(xi, θ))2

Priors

π(θ) ∝ I [0 ≤ θ ≤ 1]

π(λy) ∝ λay−1y exp{−byλy}, ay = 5, by = 5

π(θ, λy|y) ∝ L(y|η(x, θ), λy) × π(θ) × π(λy)

∝ λn2y exp

1

2 · .252λy

n∑

i=1(yi − η(xi, θ))2

× I [0 ≤ θ ≤ 1] ×

λay−1y exp{−byλy}

23

Page 24: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Use MCMC to sample from π(θ, λy|y)

π(θ, λy|y) ∝ L(y|η(x, θ), λy) × π(θ) × π(λy)

∝ λn2y exp

1

2 · .252λy

n∑

i=1(yi − η(xi, θ))2

× I [0 ≤ θ ≤ 1] ×

λay−1y exp{−byλy}

• Metropolis updates for θ

π(θ|λy, y) ∝ exp

1

2 · .252λy

n∑

i=1(yi − η(xi, θ))2

× I [0 ≤ θ ≤ 1]

• Gibbs updates for λy

λy|θ, y ∼ Γay +

n

2, by +

1

2

n∑

i=1(yi − η(xi, θ))2

Such an approach may require many evaluations or η(xi, θ)!

24

Page 25: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

MCMC output from π(θ, λy|y)

0 1000 2000 3000 40000

0.5

1

iteration

θ

0.2 0.4 0.6 0.8 10

500

1000

1500

θ

0 1000 2000 3000 40000

2

4

iteration

λ

0 1 2 3 40

500

1000

1500

λ

0 1 2 3 40.2

0.4

0.6

0.8

1

λ

θ

25

Page 26: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior for η(x, θ)

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior uncertainty

26

Page 27: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Using Importance Sampling (IS) to construct π(θ|y)

Importance sampling:

• draw θ1, . . . , θT ∼ π(θ)

• compute IS weights wt = L(y|θt), t = 1, . . . , T .

• estimate π(θ|y) by the empirical (pdf)value θ1 · · · θT

prob w1/w+ · · · wT/w+

Straightforward to estimated predictive pdf for η(x′, θ)|yvalue η(x′, θ1) · · · η(x′, θT )prob w1/w+ · · · wT/w+

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior uncertainty

27

Page 28: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

An Inverse Problem in Hydrology

Time

Wel

l.NW

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.N

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.NE

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.W

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

0 10 20 30 40 50 60

010

2030

4050

60

Time

Wel

l.E

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.SW

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.S

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

Time

Wel

l.SE

0 10 20 30 40 50 60

0.0

0.10

0.20

0.30

L(y|η(z)) ∝ |Σ|−12 exp

1

2(y − η(z))TΣ−1(y − η(z))

π(z|λz) ∝ λm2z exp

1

2zTWzz

π(λz) ∝ λaz−1z exp {bzλz}

π(z, λz|y) ∝ L(y|η(z)) × π(z|λz) × π(λz)

28

Page 29: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior realizations of z under MRF and moving average priors

Well Data

0.42

0.34 0.93 0.52

0.17

P

P

P

I

I

I

I

MRF Realization

0 10 20 30 400

1020

30

MRF Realization

0 10 20 30 40

010

2030

MRF Posterior Mean

0 10 20 30 40

010

2030

GP Realization

0 10 20 30 40

010

2030

GP Realization

0 10 20 30 40

010

2030

GP Posterior Mean

0 10 20 30 40

010

2030

29

Page 30: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

References

• A. Gelman, J. B. Carlin, H. S. Stern and D. B. Rubin (1995) Bayesian Data

Analysis, Chapman & Hall.

• Besag, J., P. J. Green, D. Higdon, and K. Mengersen (1995), Bayesian com-putation and stochastic systems (with Discussion), Statistical Science, 10,3-66.

• D. Higdon (2002) Space and space-time modeling using process convolu-tions, in Quantitative Methods for Current Environmental Issues (C.Anderson and V. Barnett and P. C. Chatwin and A. H. El-Shaarawi,eds),37–56.

• D. Higdon, H. Lee and C. Holloman (2003) Markov chain Monte Carloapproaches for inference in computationally intensive inverse problems, inBayesian Statistics 7, Proceedings of the Seventh Valencia Interna-

tional Meeting (Bernardo, Bayarri, Berger, Dawid, Heckerman, Smith andWest, eds).

30

Page 31: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

GAUSSIAN PROCESSES 1

31

Page 32: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models for spatial phenomena

0 1 2 3 4 5 6 7−2

−1

0

1

2

s

z(s)

An example of z(s) of a Gaussian process model on s1, . . . , sn

z =

z(s1)...

z(sn)

∼ N

0...0

,

Σ

, with Σij = exp{−||si − sj||2},

where ||si − sj|| denotes the distance between locations si and sj.

z has density π(z) = (2π)−n2 |Σ|−1

2 exp{−12z

TΣ−1z}.

32

Page 33: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Realizations from π(z) = (2π)−n2 |Σ|−1

2 exp{−12z

TΣ−1z}

0 1 2 3 4 5 6 7−2

−1

0

1

2

z(s)

0 1 2 3 4 5 6 7−2

−1

0

1

2

z(s)

0 1 2 3 4 5 6 7−2

−1

0

1

2

s

z(s)

model for z(s) can be extended to continuous s

33

Page 34: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Generating multivariate normal realizations

Independent normals are standard for any computer package

u ∼ N(0, In)

Well known property of normals:

if u ∼ N(µ, Σ), then z = Ku ∼ N(Kµ, KΣKT )

Use this to construct correlated realizations from iid ones.

Want z ∼ N(0, Σ)

1. compute square root matrix L such that LLT = Σ;

2. generate u ∼ N(0, In);

3. Set z = Lu ∼ N(0, LInLT = Σ)

• Any square root matrix L will do here.

• Columns of L are basis functions for representing realizations z.

• L need not be square – see over or under specified bases.

34

Page 35: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Standard Cholesky decomposition

z = N(0, Σ), Σ = LLT , z = Lu where u ∼ N(0, In), L lower triangular

Σij = exp{−||si − sj||2}, s1, . . . , s20 equally spaced between 0 and 10 :

2015

105

5 10 15 20

row

s

columns

0 2 4 6 8 10

−1.

0−

0.5

0.0

0.5

1.0

s

basi

s

35

Page 36: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Cholesky decomposition with pivoting

z = N(0, Σ), Σ = LLT , z = Lu where u ∼ N(0, In), L permuted lower triangular

Σij = exp{−||si − sj||2}, s1, . . . , s20 equally spaced between 0 and 10 :

2015

105

5 10 15 20

row

s

columns

0 2 4 6 8 10

−1.

0−

0.5

0.0

0.5

1.0

s

basi

s

36

Page 37: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Singular value decomposition

z = N(0, Σ), Σ = UΛUT = LLT , z = Lu where u ∼ N(0, In)

Σij = exp{−||si − sj||2}, s1, . . . , s20 equally spaced between 0 and 10 :

2015

105

5 10 15 20

row

s

columns

0 2 4 6 8 10

−1.

0−

0.5

0.0

0.5

1.0

s

basi

s

37

Page 38: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Conditioning on some observations of z(s)

0 1 2 3 4 5 6 7−2

−1

0

1

2

z(s)

We observe z(s2) and z(s5) – what do we now know about{z(s1), z(s3), z(s4), z(s6), z(s7), z(s8)}?

z(s2)z(s5)z(s1)z(s3)z(s4)z(s6)z(s7)z(s8)

∼ N

00000000

,

1 .0001.0001 1

∣∣∣∣∣∣∣

.3679 · · · 00 · · · .0001

.3679 0. . . . . .0 .0001

∣∣∣∣∣∣∣∣∣∣∣

1 · · · 0... . . . ...0 · · · 1

38

Page 39: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Conditioning on some observations of z(s)

z1

z2

∼ N

00

,

Σ11 Σ12

Σ21 Σ22

, z2|z1 ∼ N(Σ21Σ

−111 z1, Σ22 − Σ21Σ

−111 Σ12)

0 1 2 3 4 5 6 7−2

−1

0

1

2z(

s)conditional mean

0 1 2 3 4 5 6 7

−2

−1

0

1

2

z(s)

contitional realizations

s

39

Page 40: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

More examples with various covariance functions and spatial scalesΣij = exp{−(||si − sj ||/scale)2} Σij = exp{−(||si − sj||/scale)1}

x

z

0 5 10 15

-2-1

01

2

•• •

Gaussian C(r), scale = 2

x

z

0 5 10 15

-2-1

01

2

•• •

Gaussian C(r), scale = 3

x

z

0 5 10 15

-2-1

01

2

•• •

Gaussian C(r), scale = 5

x

z

0 5 10 15

-2-1

01

2

•• •

Exponential C(r), scale = 1

x

z

0 5 10 15

-2-1

01

2

•• •

Exponential C(r), scale = 10

x

z

0 5 10 15

-2-1

01

2

•• •

Exponential C(r), scale = 20

x

z

0 5 10 15

-2-1

01

2

•• •

Brownian motion C(r), p = 1.5 scale = 1

x

z

0 5 10 15

-2-1

01

2

•• •

Brownian motion C(r), p = 1.5 scale = 3

x

z

0 5 10 15-2

-10

12

•• •

Brownian motion C(r), p = 1.5 scale = 5

40

Page 41: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

More examples with various covariance functions and spatial scalesΣij = exp{−(||si − sj ||/scale)2} Σij = exp{−(||si − sj||/scale)1}

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Gaussian C(r), scale = 2

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Gaussian C(r), scale = 3

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Gaussian C(r), scale = 5

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Exponential C(r), scale = 1

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Exponential C(r), scale = 10

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Exponential C(r), scale = 20

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Brownian motion C(r), p = 1.5 scale = 1

x

z

-5 0 5 10 15 20

-3-2

-10

12

3

•• •

Brownian motion C(r), p = 1.5 scale = 3

x

z

-5 0 5 10 15 20-3

-2-1

01

23

•• •

Brownian motion C(r), p = 1.5 scale = 5

41

Page 42: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

A 2-d example, conditioning on the edgeΣij = exp{−(||si − sj ||/5)2}

510

1520

X5

10

15

20

Y

-2-1

01

23

4Z

a realization

510

1520

X5

10

15

20

Y

-2-1

01

23

4Z

mean conditional on Y=1 points

510

1520

X5

10

15

20

Y

-2-1

01

23

4Z

realization conditional on Y=1 points

510

1520

X5

10

15

20

Y

-2-1

01

23

4Z

realization conditional on Y=1 points

42

Page 43: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Soft Conditioning (Bayes Rule)

0 1 2 3 4 5 6 7

−2

−1

0

1

2

z(s)

s

Observed data y are a noisy version of z

y(si) = z(si) + ǫ(si) with ǫ(sk)iid∼ N(0, σ2

y), k = 1, . . . , n

Data spatial process prior for z(s)y Σy = σ2

yIn

y1...yn

σ2y 0 0

0 . . . 00 0 σ2

y

µz Σz

0...0

Σz

L(y|z) ∝ |Σy|−12 exp{−1

2(y − z)TΣ−1

y (y − z)} π(z) ∝ |Σz|−12 exp{−1

2zTΣ−1

z z}

43

Page 44: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Soft Conditioning (Bayes Rule) ... continued

sampling model spatial prior

L(y|z) ∝ |Σy|−12 exp{−1

2(y − z)TΣ−1y (y − z)} π(z) ∝ |Σz|−

12 exp{−1

2zTΣ−1

z z}⇒ π(z|y) ∝ L(y|z) × π(z)

⇒ π(z|y) ∝ exp{−12[z

T (Σ−1y + Σ−1

z )z + zTΣ−1y y + f(y)]}

⇒ z|y ∼ N(V Σ−1y y, V ), where V = (Σ−1

y + Σ−1z )−1

0 1 2 3 4 5 6 7

−2

−1

0

1

2

z(s)

conditional realizations

π(z|y) describes the updated uncertainty about z given the observations.

44

Page 45: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Updated predictions for unobserved z(s)’s

0 1 2 3 4 5 6 7

−2

−1

0

1

2

z(s)

conditional realizations

data locations yd = (y(s1), . . . , y(sn))T zd = (z(s1), . . . , z(sn))T

prediction locations y∗ = (y(s∗1), . . . , y(s∗m))T z∗ = (z(s∗1), . . . , z(s∗m))T

define y = (yd; y∗) z = (zd; z∗)

Data spatial process prior for z(s)

y =yd

y∗

=

yd

0m

Σy =

σ2

yIn 00 ∞Im

µz =

0n

0m

Σz =

cov rule applied

to (s, s∗)

define Σ−y =

1σ2

yIn 0

0 0

Now the posterior distribution for z = (zd, z∗) is

z|y ∼ N(V Σ−y y, V ), where V = (Σ−

y + Σ−1z )−1

45

Page 46: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Example: Dioxin concentration at Piazza Road Superfund Site

x

y

0 20 40 60 80 100

050

100

150

200

-1 1 2 3 4

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

0 20 40 60 80 100

050

100

150

200

0 20 40 60 80 100

050

100

150

200

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.4

0.4

0.4

0.4

0.40.4

0.4

0.4

0.6

0.6

0.6

0.60.8 0.8

data Posterior mean of z∗ pointwise posterior sd

46

Page 47: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bonus topic: constructing simultaneous intervals

• generate a large sample of m-vectors z∗ from π(z∗|y).

• compute the m-vector z∗ that is the mean of the generated z∗s

• compute the m-vector σ that is the pointwise sd of the generated z∗s

• find the constant a such that 80% of the generated z∗s are completely

contained within z∗ ± aσ

0 1 2 3 4 5 6 7

−2

−1

0

1

2z(

s)posterior realizations and mean

0 1 2 3 4 5 6 7

−2

−1

0

1

2

+/−

sd[

z(s)

]

pointwise estimated sd

0 1 2 3 4 5 6 7

−2

−1

0

1

2

z(s)

simultaneous 80% credible interval

47

Page 48: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

References

• Ripley, B. (1989) Spatial Statistics, Wiley.

• Cressie, N. (1992) Statistics for Spatial Data, Wiley.

• Stein, M. (1999) Interpolation of Spatial Data: Some Theory for Krig-

ing, Springer.

48

Page 49: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

GAUSSIAN PROCESSES 2

49

Page 50: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models revisited

Application: finding in a rod of material

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−2

−1.5

−1

−0.5

0

0.5

1

1.5

scaled frequency

scal

ed a

ccel

erat

ion

• for various driving frequencies, accelerationof rod recorded

• the true frequency-acceleration curve issmooth.

• we have noisy measurements of accelera-tion.

• estimate resonance frequency.

• use GP model for frequency-accel curve.

• smoothness of GP model important here.

50

Page 51: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models formulation

Take response y to be acceleration and spatial value s to be frequency.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−2

−1.5

−1

−0.5

0

0.5

1

1.5

scaled frequency

scal

ed a

ccel

erat

ion

data: y = (y1, . . . , yn)T at spatial locationss1, . . . , sn.

z(s) is a mean 0 Gaussian process with covariancefunction

Cov(z(s), z(s′)) =1

λzexp{−β(s − s′)2}

β controls strength of dependence.

Take z = (z(s1), . . . , z(sn))T to be z(s) restricted to the data observations.

Model the data as:

y = z + ǫ, where ǫ ∼ N(0,1

λyIn)

We want to find the posterior distribution for the frequency s⋆ where z(s) ismaximal.

51

Page 52: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Reparameterizing the spatial dependence parameter β

It is convenient to reparameterize β as:

ρ = exp{−β(1/2)2} ⇔ β = −4 log(ρ)

So ρ is the correlation between two points on z(s) separated by 12.

Hence z has spatial prior

z|ρ, λz ∼ N(0,1

λzR(ρ; s))

where R(ρ; s) is the correlation matrix with ij elements

Rij = ρ4(si−sj)2

Prior specification for z(s) is completed by specfying priors for λz and ρ.

π(λz) ∝ λaz−1z exp{−bzλz} if y is standardized, encourage λz to be close to 1 –

eg.az = bz = 5.

π(ρ) ∝ (1 − ρ)−.5 encourages ρ to be large if possible

52

Page 53: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Bayesian model formulation

LikelihoodL(y|z, λy) ∝ λ

n2y exp{−1

2λy(y − z)T (y − z)}Priors

π(z|λz, ρ) ∝ λn2z |R(ρ; s)|−1

2 exp{−12λzz

TR(ρ; s)−1z}π(λy) ∝ λay−1

y e−byλy, uninformative here – ay = 1, by = .005

π(λz) ∝ λaz−1z e−bzλz, fairly informative – az = 5, bz = 5

π(ρ) ∝ (1 − ρ)−.5

Marginal likelihood (integrating out z)

L(y|λǫ, λz, ρ) ∝ |Λ|12 exp{−12yTΛy}

where Λ−1 = 1λy

In + 1λz

R(ρ; s)

Posterior

π(λy, λz, ρ|y) ∝ |Λ|12 exp{−12y

TΛy} × λay−1y e−byλy × λaz−1

z e−bzλz × (1 − ρ)−.5

53

Page 54: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior Simulation

Use Metropolis to simulate from the posterior

π(λy, λz, ρ|y) ∝ |Λ|12 exp{−12y

TΛy} × λay−1y e−byλy × λaz−1

z e−bzλz × (1 − ρ)−.5

giving (after burn-in) (λy, λz, ρ)1, . . . , (λy, λz, ρ)T

For any given realization (λy, λz, ρ)t, one can generate z∗ = (z(s∗1), . . . , z(s∗m))T

for any set of prediction locations s∗1, . . . , s∗m.

From previous GP stuff, we know

zz∗

| · · · ∼ N

V Σ−

y

y0m

, V

where

Σ−y =

λǫIn 0

0 0

and V −1 = Σ−

y + λzR(ρ, (s, s∗))−1

Hence, one can generate corresponding z∗’s for each posterior realization at afine grid around the apparent resonance frequency z⋆.

54

Page 55: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

MCMC output for (λy, λz, ρ)

0 500 1000 1500 2000 2500 30000

0.5

1

iteration

rho

0 500 1000 1500 2000 2500 30000

0.5

1

1.5

2

iteration

lam

z

500 1000 1500 2000 2500 3000

50

100

150

200

250

iteration

lam

y

55

Page 56: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior realizations for z(s) near z⋆

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

scaled frequency

scal

ed a

ccel

erat

ion

56

Page 57: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior for resonance frequency z⋆

0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5 0.51 0.520

10

20

30

40

50

60

70

80

90posterior distribution for scaled resonance frequency

scaled resonance frequency

57

Page 58: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian Processes for modeling complex computer simulators

data input settings (spatial locations)

y =

y1...yn

S =

s1...sn

=

s11 s12 · · · s1p... ... ... ...

sn1 sn2 · · · snp

Model responses y as a (stochastic) function of s

y(s) = z(s) + ǫ(s)

Vector form – restricting to the n data points

y = z + ǫ

58

Page 59: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Model response as a Gaussian processes

y(s) = z(s) + ǫ

LikelihoodL(y|z, λǫ) ∝ λ

n2ǫ exp{−1

2λǫ(y − z)T (y − z)}Priors

π(z|λz, β) ∝ λn2z |R(β)|−1

2 exp{−12λzz

TR(β)−1z}π(λǫ) ∝ λaǫ−1

ǫ e−bǫλǫ, perhaps quite informative

π(λz) ∝ λaz−1z e−bzλz, fairly informative if data have been standardized

π(ρ) ∝p∏

k=1(1 − ρk)

−.5

Marginal likelihood (integrating out z)

L(y|λǫ, λz, β) ∝ |Λ|12 exp{−12y

TΛy}

where Λ−1 = 1λǫ

In + 1λz

R(β)

59

Page 60: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

GASP Covariance model for z(s)

Cov(z(si), z(sj)) =1

λz

p∏

k=1exp{−βk(sik − sjk)

α}

• Typically α = 2 ⇒ z(s) is smooth.

• Separable covariance – a product of componentwise covariances.

• Can handle large number of covariates/inputs p.

• Can allow for multiway interactions.

• βk = 0 ⇒ input k is “inactive” ⇒ variable selection

• reparameterize: ρk = exp{−βkdα0} – typically d0 is a halfwidth.

60

Page 61: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior Distribution and MCMC

π(λǫ, λz, ρ|y) ∝ |Λλ,ρ|12 exp{−1

2yTΛλ,ρy} × λaǫ−1

ǫ e−bǫλǫ ×λaz−1

z e−bzλz ×p∏

k=1(1 − ρk)

−.5

• MCMC implementation requires Metropolis updates.

• Realizations of z(s)|λ, ρ, y can be obtained post-hoc:

− define z∗ = (z(s∗1, . . . , z(s∗m))T to be predictions at locations s∗1, . . . , s∗m,

then

zz∗

| · · · ∼ N

V Σ−

y

y0m

, V

where

Σ−y =

λǫIn 0

0 0

and V −1 = Σ−

y + λzR(ρ, (s, s∗))−1

61

Page 62: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Example: Solar collector Code (Schonlau, Hamada and Welch, 1995)• n = 98 model runs, varying 6 independent variables.

• Response is the increase in heat exchange effectiveness.

• A latin hypercube (LHC) design was used with 2-d space filling.

wind.vel

0.010 0.020 0.030 0 100 200 300 2 4 6 8 10

0.02

00.

035

0.05

0

0.01

00.

020

0.03

0

slot.width

Rey.num

5070

90

010

030

0

admittance

plate.thickness

0.01

0.03

0.05

0.07

0.020 0.030 0.040 0.050

24

68

10

50 60 70 80 90 100 0.01 0.03 0.05 0.07

Nusselt.num

62

Page 63: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Example: Solar collector Code• Fit of GASP model and predictions of 10 holdout points

• Two most active covariates are shown here.

0

0.5

1

0

0.5

1−4

−2

0

2

x4

data

x5

y

0 500 1000 1500 20000

5

10

15

20

iteration

beta

(1:p

)

0 500 1000 1500 20000

1

2

iteration

lam

z 0 500 1000 1500 20000

2

4x 10

iteration

lam

e

0

0.5

1

0

0.5

1−4

−2

0

2

x4

posterior mean

x5 −3 −2.5 −2 −1.5 −1−3

−2.5

−2

−1.5

−1

63

Page 64: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Example: Solar collector Code• Visualizing a 6-d response surface is difficult

• 1-d marginal effects shown here.

0 0.5 1−3

−2

−1

0

1

2

x1

y

x1

0 0.5 1−3

−2

−1

0

1

2

x2

y

x2

0 0.5 1−3

−2

−1

0

1

2

x3

y

x3

0 0.5 1−3

−2

−1

0

1

2

x4

y

x4

0 0.5 1−3

−2

−1

0

1

2

x5

y

x5

0 0.5 1−3

−2

−1

0

1

2

x6

y

x6

1−D Marginal Effects

64

Page 65: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

References

• J. Sacks, W. J. Welch, T. J. Mitchell and H. P. Wynn (1989) Design andanalysis of comuter experiments Statistical Science, 4:409–435.

65

Page 66: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

COMPUTER MODEL CALIBRATION 1

66

Page 67: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Inference combining a physics model with experimental data

2 4 6 8 10

1

2

3

4

5

6

7

dro

ptim

e

drop height (floor)

2 4 6 8 10

1

2

3

4

5

6

7

dro

ptim

e

drop height (floor)

2 4 6 8 10

1

2

3

4

5

6

7

dro

ptim

e

drop height (floor)

Data generated from model:d2zdt2

= −1 − .3dzdt + ǫ

simulation model:d2zdt2

= −1

statistical model:y(z) = η(z) + δ(z) + ǫ

Improved physics model:d2zdt2

= −1 − θdzdt + ǫ

statistical model:y(z) = η(z, θ) + δ(z) + ǫ

67

Page 68: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Accounting for limited simulator runs

0 0.5 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

data & simulations

00.5

1

0

0.5

1

−2

0

2

x

model runs

θ

η(x,

θ)

• Borrows from Kennedy and O’Hagan (2001).

x model or system inputsθ calibration parametersζ(x) true physical system response given inputs xη(x, θ) simulator response at x and θ.

simulator run at limited input settingsη = (η(x∗

1, θ∗1), . . . , η(x∗

m, θ∗m))T

treat η(·, ·) as a random functionuse GP prior for η(·, ·)

y(x) experimental observation of the physical systeme(x) observation error of the experimental data

y(x) = ζ(x) + e(x)

y(x) = η(x, θ) + e(x)

68

Page 69: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

OA designs for simulator runs

Example: N = 16, 3 factors each at 4 levelsOA(16, 43) design 2-d projections

0

.5

1

0

.5

1

0

.5

1

x2

x1

x 3

x_1

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

x_2

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

x_3

OA design ensures importance measures R2 can be accurately estimated for lowdimensions

Can spread out design for building a response surface emulator of η(x)

69

Page 70: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian Process models for combining field data and complexcomputer simulators

field data input settings (spatial locations)

y =

y(x1)...

y(xn)

x11 x12 · · · x1px... ... ... ...

xn1 xn2 · · · xnpx

sim data input settings x; params θ∗

η =

η(x∗1, θ

∗1)

...η(x∗

m, θ∗m)

x∗11 · · · x∗

1pxθ∗11 · · · θ∗1pθ... ... ... ... ... ...

x∗m1 · · · x∗

mpxθ∗m1 · · · θ∗mpθ

Model sim response η(x, θ) as a Gaussian process

y(x) = η(x, θ) + ǫ

η(x, θ) ∼ GP (0, Cη(x, θ))

ǫ ∼ iidN(0, 1/λǫ)

Cη(x, θ) depends on px + pθ-vector ρη and λη

70

Page 71: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Vector form – restricting to n field obs and m simulation runs

y = η + ǫ

η ∼ Nm(0m, Cη(ρη, λη))

⇒yη

∼ Nn+m

0n

0m

, Cyη = Cη +

1/λǫIn 0

0 1/λsIm

where

Cη = 1/ληRη

xx∗

,

1θθ∗

; ρη

and the correlation matrix Rη is given by

Rη((x, θ), (x′, θ′); ρη) =px∏

k=1ρ

4(xk−x′k)2

ηk ×pθ∏

k=1ρ

4(θk−θ′k)2

η(k+px)

λs is typically set to something large like 106 to stabalize matrix computationsand allow for numerical fluctuation in η(x, θ).

note: the covariance matrix Cη depends on θ through its “distance”-basedcorrelation function Rη((x, θ), (x′, θ′); ρη).

We use a 0 mean for η(x, θ); an alternative is to use a linear regression meanmodel.

71

Page 72: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

LikelihoodL(y, η|λǫ, ρη, λη, λs, θ) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

Priorsπ(λǫ) ∝ λaǫ−1

ǫ e−bǫλǫ perhaps well known from observation process

π(ρηk) ∝px+pθ∏

k=1(1 − ρηk)

−.5, where ρηk = e−.52βηk correlation at dist = .5 ∼ β(1, .5).

π(λη) ∝ λaη−1η e−bηλη

π(λs) ∝ λas−1s e−bsλs

π(θ) ∝ I [θ ∈ C]

• could fix ρη, λη from prior GASP run on model output.• Many prefer to reparameterize ρ as β = − log(ρ)/.52 in the likelihood term

72

Page 73: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior Density

π(λǫ, ρη, λη, λs, θ|y, η) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

×

px+pθ∏

k=1(1 − ρηk)

−.5 × λaη−1η e−bηλη × λas−1

s e−bsλs ×λaǫ−1

ǫ e−bǫλǫ × I [θ ∈ C]

If ρη, λη, and λs are fixed from a previous analysis ofthe simulator data, then

π(λǫ, θ|y, η, ρη, λη, λs) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

×

λaǫ−1ǫ e−bǫλǫ × I [θ ∈ C]

73

Page 74: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Accounting for limited simulation runs

0 0.5 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

00.5

1

0

0.5

1

−2

0

2

θ*

posterior realizations of η(x,θ*)

x

η(x,

θ* )

0 0.5 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior uncertainty

Again, standard Bayesian estimation gives:

π(θ, η(·, ·), λǫ, ρη, λη|y(x)) ∝ L(y(x)|η(x, θ), λǫ) ×π(θ) × π(η(·, ·)|λη, ρη)

π(λǫ) × π(ρη) × π(λη)

• Posterior means and quantiles shown.

• Uncertainty in θ, η(·, ·), nuisance parameters are incorporated into the forecast.

• Gaussian process models for η(·, ·).

74

Page 75: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Predicting a new outcome: ζ = ζ(x′) = η(x′, θ)

Given a MCMC realization (θ, λǫ, ρη, λη), a realization for ζ(x′) can be producedusing Bayes rule.

Data GP prior for η(x, θ)(s)

v =

yηζ

Σ−v =

λǫIn 0 00 λsIm 00 0 0

µz =

0n

0m

0

Cη = λ−1η Rη

xx∗

x′

,

1θθ∗

θ

; ρη

Now the posterior distribution for v = (y, η, ζ)T is

v|y, η ∼ N(µv|yη = V Σ−v v, V ), where V = (Σ−

v + C−1η )−1

Restricting to ζ we have

ζ|y, η ∼ N(µv|yηm+n+1, Vn+m+1,n+m+1)

Alternatively, one can apply the conditional normal formula to

yηζ

∼ N

000

,

λ−1ǫ In 0 00 λ−1

s Im 00 0 0

+ Cη

so that

ζ|y, η ∼ NΣ21Σ

−111

, Σ22 − Σ21Σ

−111 Σ12

75

Page 76: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Accounting for model discrepancy

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty• Borrows from Kennedy and O’Hagan (2001).

x model or system inputsθ model or system inputsζ(x) true physical system response given inputs xη(x, θ) simulator response at x and θ.y(x) experimental observation of the physical systemδ(x) discrepancy between ζ(x) and η(x, θ)

may be decomposed into numerical error and biase(x) observation error of the experimental data

y(x) = ζ(x) + e(x)

y(x) = η(x, θ) + δ(x) + e(x)

y(x) = η(x, θ) + δn(x) + δb(x) + e(x)

76

Page 77: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Accounting for model discrepancy

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior model uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

δ(x)

posterior model discrepancy

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, ζ(x

)

calibrated forecast

Again, standard Bayesian estimation gives:

π(θ, η, δ|y(x)) ∝ L(y(x)|η(x, θ), δ(x)) ×π(θ) × π(η) × π(δ)

• Posterior means and 90% CI’s shown.

• Posterior prediction for ζ(x) is obtainedby computing the posterior distribution forη(x, θ) + δ(x)

• Uncertainty in θ, η(x, t), and δ(x) are in-corporated into the forecast.

• Gaussian process models for η(x, t) andδ(x)

77

Page 78: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian Process models for combining field data and complexcomputer simulators

field data input settings (spatial locations)

y =

y(x1)...

y(xn)

x11 x12 · · · x1px... ... ... ...

xn1 xn2 · · · xnpx

sim data input settings x; params θ∗

η =

η(x∗1, θ

∗1)

...η(x∗

m, θ∗m)

x∗11 · · · x∗

1pxθ∗11 · · · θ∗1pθ... ... ... ... ... ...

x∗m1 · · · x∗

mpxθ∗m1 · · · θ∗mpθ

Model sim response η(x, θ) as a Gaussian process

y(x) = η(x, θ) + δ(x) + ǫ

η(x, θ) ∼ GP (0, Cη(x, θ))

δ(x) ∼ GP(

0, Cδ(x))

ǫ ∼ iidN(0, 1/λǫ)

Cη(x, θ) depends on px + pθ-vector ρη and λη

Cδ(x) depends on px-vector ρδ and λδ

78

Page 79: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Vector form – restricting to n field obs and m simulation runs

y = η + δ + ǫ

η ∼ Nm(0m, Cη(ρη, λη))yη

∼ Nn+m

0n

0m

, Cyη = Cη +

Cδ 00 0

where

Cη = 1/ληRη

xx∗

,

1θθ∗

; ρη

+ 1/λsIm+n

Cδ = 1/λδRδ(x; ρδ) + 1/λǫIn

and the correlation matricies Rη and Rδ are given by

Rη((x, θ), (x′, θ′); ρη) =px∏

k=1ρ

4(xk−x′k)2

ηk ×pθ∏

k=1ρ

4(θk−θ′k)2

η(k+px)

Rδ(x, x′; ρδ) =px∏

k=1ρ

4(xk−x′k)2

δk

λs is typically set to something large like 106 to stabalize matrix computationsand allow for numerical fluctuation in η(x, θ).

We use a 0 mean for η(x, θ); an alternative is to use a linear regression meanmodel.

79

Page 80: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

LikelihoodL(y, η|λǫ, ρη, λη, λs, ρδ, λδ, θ) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

Priorsπ(λǫ) ∝ λaǫ−1

ǫ e−bǫλǫ perhaps well known from observation process

π(ρηk) ∝px+pθ∏

k=1(1 − ρηk)

−.5, where ρηk = e−.52βηk correlation at dist = .5 ∼ β(1, .5).

π(λη) ∝ λaη−1η e−bηλη

π(λs) ∝ λas−1s e−bsλs

π(ρδk) ∝px∏

k=1(1 − ρδk)

−.5, where ρδk = e−.52βδk

π(λδ) ∝ λaδ−1δ e−bδλδ,

π(θ) ∝ I [θ ∈ C]

• could fix ρη, λη from prior GASP run on model output.• Again, many choose to reparameterize correlation parameters: β = − log(ρ)/.52

in the likelihood term

80

Page 81: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior Density

π(λǫ, ρη, λη, λs, ρδ, λδ, θ|y, η) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

×

px+pθ∏

k=1(1 − ρηk)

−.5 × λaη−1η e−bηλη × λas−1

s e−bsλs ×px∏

k=1(1 − ρδk)

−.5 × λaδ−1δ e−bδλδ × λaǫ−1

ǫ e−bǫλǫ × I [θ ∈ C]

If ρη, λη, and λs are fixed from a previous analysis ofthe simulator data, then

π(λǫ, ρδ, λδ, θ|y, η, ρη, λη, λs) ∝

|Cyη|−12 exp

−1

2

T

C−1yη

×

px∏

k=1(1 − ρδk)

−.5 × λaδ−1δ e−bδλδ × λaǫ−1

ǫ e−bǫλǫ × I [θ ∈ C]

81

Page 82: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Predicting a new outcome: ζ = ζ(x′) = η(x′, θ) + δ(x′)

y = η(x, θ) + δ(x) + ǫ(x)

η = η(x∗, θ∗) + ǫs, ǫs small or 0

ζ = η(x′, θ) + δ(x′), x′ univariate or multivariate

yηζ

∼ Nn+m+1

0n

0m

0

,

λ−1ǫ In 0 00 λ−1

s Im 00 0 0

+ Cη + Cδ

(1)

where

Cη = 1/ληRη

xx∗

x′

,

1θθ∗

θ

; ρη

Cδ = 1/λδRδ

xx′

; ρδ

, on indicies 1, . . . , n, n + m + 1; zeros elsewhere

Given a MCMC realization (θ, λǫ, ρη, λη, ρδ, λδ), a realization for ζ(x′) can beproduced using (1) and the conditional normal formula:

ζ|y, η ∼ NΣ21Σ

−111

, Σ22 − Σ21Σ

−111 Σ12

82

Page 83: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Accounting for model discrepancy

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

prior uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, η(

x,θ)

θ0 .5 1

posterior model uncertainty

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

δ(x)

posterior model discrepancy

0 0.2 0.4 0.6 0.8 1−3

−2

−1

0

1

2

3

x

y(x)

, ζ(x

)

calibrated forecast

Again, standard Bayesian estimation gives:

π(θ, ηn, δ|y(x)) ∝ L(y(x)|η(x, θ), δ(x)) ×π(θ) × π(η) × π(δ)

• Posterior means and 90% CI’s shown.

• Posterior prediction for ζ(x) is obtainedby computing the posterior distribution forη(x, θ) + δ(x)

• Uncertainty in θ, η(x, t), and δ(x) are in-corporated into the forecast.

• Gaussian process models for η(x, t) andδ(x)

83

Page 84: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

References

• T. Santner, B. J. Williams and W. I. Notz (2003) The Design and Analysis

of Computer Experiments, Springer.

• M. Kennedy and A. O’Hagan (2001) Bayesian Calibration of Computer Mod-els (with Discussion), Journal of the Royal Statistical Society B, 63, 425–464.

• J. Sacks, W. J. Welch, T. J. Mitchell and H. P. Wynn (1989). Design andAnalysis of computer experiments (with discussion). Statistical Science, 4,409–423.

• Higdon, D., Kennedy, M., Cavendish, J., Cafeo, J. and Ryne R. D. (2004)Combining field observations and simulations for calibration and prediction.SIAM Journal of Scientific Computing, 26, 448–466.

84

Page 85: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

COMPUTER MODEL CALIBRATION 2

DEALING WITH MULTIVARIATE OUTPUT

85

Page 86: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Carry out simulated implosions using Neddermeyer’s model

Sequence of runs carried at m input settings (x∗, θ∗1, θ∗2) = (me/m, s, u0) varying

over predefined ranges using an OA(32, 43)-based LH design.

x∗1 θ∗11 θ∗12... ... ...

x∗m θ∗m1 θ∗m2

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10−5

0

0.5

1

1.5

2

2.5

time (s)

inne

r ra

dius

(cm

)

0

1

2

3

4

5x 10

−5

0

pi

2pi0

0.5

1

1.5

2

2.5

angle (radians)time (s)

inne

r ra

dius

(cm

)

radius by time radius by time and angle φ.

Each simulation produces a nη = 22 · 26 vector of radii for 22 times × 26angles.

86

Page 87: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Application: implosions of steel cylinders – Neddermeyer ’43

• Initial work on implosion for fat man.

• Use high explosive (HE) to crush steel cylindrical shells

• Investigate the feasability of a controlled implosion

87

Page 88: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Some HistoryEarly work on cylinders called “beer can experi-ments.”

• Early work not encouraging:“...I question Dr. Neddermeyer’s serious-ness...” – Deke Parsons.“It stinks.” – R. FeynmanTeller and VonNeumann were quite sup-portive of the implosion idea

Data on collapsing cylinder from high speed pho-tography.

Symmetrical implosion eventually accomplishedusing HE lenses by Kistiakowsky.

Implosion played a key role in early computer ex-periments.

Feynman worked on implosion calculations withIBM accounting machines.

Eventually first computer with addressable mem-ory was developed (MANIAC 1).

88

Page 89: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

The Experiments

89

Page 90: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Neddermeyer’s Model

−2

0

2

cm

0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s

−2

0

2cm

0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s

Energy from HE imparts an initial inward velocity to the cylinder

v0 =me

m

√2u0

1 + me/m

mass ratio me/m of HE to steel; u0 energy per unit mass from HE.

Energy converts to work done on the cylinder:

work per unit mass = w =s

2ρ(1 − λ)

{r2i log r2

i − r2o log r2

o + λ2 log λ2}

ri = scaled inner radius; ro = scaled outer radius; λ = initial ri/ro; s = steelyielding stress; ρ = density of steel.

90

Page 91: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Neddermeyer’s Model

−2

0

2

cm

0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s

−2

0

2cm

0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s 0 1 2 3 4 X 10−5 s

ODE:dr

dt=

[1

R21f(r)

{v2

0 −s

ρg(r)

}]12

where

r = inner radius of cylinder – varies with time

R1 = initial outer radius of cylinder

f(r) =r2

1 − λ2ln

(r2 + 1 − λ2

r2

)

g(r) = (1 − λ2)−1[r2 ln r2 − (r2 + 1 − λ2) ln(r2 + 1 − λ2) − λ2 ln λ2]

λ = initial ratio of cylinder r(t = 0)/R1

constant volume condition: r2outer

− r2 = 1 − λ2

91

Page 92: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Goal: use experimental data to calibrate s and u0; obtainprediction uncertainty for new experiment

−2

0

2

expt 1

cm

−2

0

2

cm

−2 0 2

−2

0

2

cm

−2 0 2

expt 2

expt 3

−2 0 2

t = 10 µs

t = 25 µs

t = 45 µs

me/m ≈ .32 me/m ≈ .17 me/m ≈ .36Hypothetical data obtained from photos at different times during the 3 exper-imental implosions. All cylinders had a 1.5in outer and a 1.0in inner radius.(λ = 2

3).

92

Page 93: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Carry out simulated implosions using Neddermeyer’s model

Sequence of runs carried at m input settings (x∗, θ∗1, θ∗2) = (me/m, s, u0) varying

over predefined ranges using an OA(32, 43)-based LH design.

x∗1 θ∗11 θ∗12... ... ...

x∗m θ∗m1 θ∗m2

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10−5

0

0.5

1

1.5

2

2.5

time (s)

inne

r ra

dius

(cm

)

0

1

2

3

4

5x 10

−5

0

pi

2pi0

0.5

1

1.5

2

2.5

angle (radians)time (s)

inne

r ra

dius

(cm

)

radius by time radius by time and angle φ.

Each simulation produces a nη = 22 · 26 vector of radii for 22 times × 26angles.

93

Page 94: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

A 1-d implementation of the cylinder application

0

0.5

1

0

0.5

1

0

1

2

t

model runs

x

η(x,

t)

0 0.5 1

−0.5

0

0.5

1

1.5

2

2.5

x (scaled time)

y(x)

, η(

x,t)

θ0 .5 1

data & prior uncertainty

0

0.5

1

0

0.5

1

0

1

2

t

posterior mean for η(x,t)

x

η(x,

t)

0 0.5 1

−0.5

0

0.5

1

1.5

2

2.5

x (scaled time)

y(x)

, η(

x,θ)

θ0 .5 1

calibrated simulator prediction

0 0.5 1

−0.5

0

0.5

1

1.5

2

2.5

x (scaled time)

δ(x)

posterior model discrepancy

0 0.5 1

−0.5

0

0.5

1

1.5

2

2.5

x (scaled time)y(

x), ζ

(x)

calibrated prediction

experimental data are collapsed radially

94

Page 95: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Features of this basic formulation

• Scales well with the input dimension, dim(x, θ).

• Treats simulation model as “black box” – no need to get inside simulator.

• Can model complicated and indirect observation processes.

Limitations of this basic formulation

• Does not easily deal with highly multivariate data.

• Inneficient use of multivariate simulation output.

• Can miss important features in the physical process.

Need extension of basic approach to handle multivariate experimental observa-tions and simulation output.

95

Page 96: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Carry out simulated implosions using Neddermeyer’s model

Sequence of runs carried at m input settings (x∗, θ∗1, θ∗2) = (me/m, s, u0) varying

over predefined ranges using an OA(32, 43)-based LH design.

x∗1 θ∗11 θ∗12... ... ...

x∗m θ∗m1 θ∗m2

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10−5

0

0.5

1

1.5

2

2.5

time (s)

inne

r ra

dius

(cm

)

0

1

2

3

4

5x 10

−5

0

pi

2pi0

0.5

1

1.5

2

2.5

angle (radians)time (s)

inne

r ra

dius

(cm

)

radius by time radius by time and angle φ.

Each simulation produces a nη = 22 · 26 vector of radii for 22 times × 26angles.

96

Page 97: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Basis representation of simulation output

η(x, θ) =

pη∑

i=1

ki(t, φ)wi(x, θ)

Here we construct bases ki(t, φ) via principal components (EOFs):

05

0−1.5

−1−0.5

0

angle

PC 1 (98.9% of variation)

time

r

05

0−1.5

−1−0.5

0

angle

PC 2 (0.9% of variation)

time

r

05

0−1.5

−1−0.5

0

angle

PC 3 (0.1% of variation)

time

r

basis elements do not change with φ – from symmetry of Neddermeyer’s model.

Model untried settings with a GP model on weights:

wi(x, θ1, θ2) ∼ GP(0, λ−1wi R((x, θ), (x′, θ′); ρwi))

00.5

1

00.5

1−5

0

5

x

PC 1

θ1

w1(x

,θ)

00.5

1

00.5

1−5

0

5

x

PC 2

θ1

w2(x

,θ)

00.5

1

00.5

1−5

0

5

x

PC 3

θ1

w3(x

,θ)

97

Page 98: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

PC representation of simulation output

Ξ = [η1; · · · ; ηm] – a nη × m matrix that holds output of m simulations

SVD decomposition: Ξ = UDV T

Kη is 1st pη columns of [ 1√mUD] – columns of [

√mV T ] have variance 1

Cylinder example:

05

0−1.5

−1−0.5

0

angle

PC 1 (98.9% of variation)

time

r

05

0−1.5

−1−0.5

0

angle

PC 2 (0.9% of variation)

timer

05

0−1.5

−1−0.5

0

angle

PC 3 (0.1% of variation)

time

r

pη = 3 PC’s: Kη = [k1; k2; k3] – each vector ki holds trace of PC i.

ki’s do not change with φ – from symmetry of Neddermeyer’s model.

Simulated trace η(x∗i , θ

∗i1, θ

∗i2) = Kηw(x∗

i , θ∗i1, θ

∗i2)+ǫi, ǫi’s

iid∼ N(0, λ−1η ), for any

set of tried simulation inputs (x∗i , θ

∗i1, θ

∗i2).

98

Page 99: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models for PC weights

Want to evaluate η(x, θ1, θ2) at arbitrary input setting (x, θ1, θ2).

Also want analysis to account for uncertainty here.

Approach: model each PC weight as a Gaussian process:

wi(x, θ1, θ2) ∼ GP(0, λ−1wi R((x, θ), (x′, θ′); ρwi))

where

R((x, θ), (x′, θ′); ρwi) =

px∏

k=1

ρ4(xk−x′k)2

wik ×pθ∏

k=1

ρ4(θk−θ′k)

2

wi(k+px) (1)

Restricting to the design settings

x∗1 θ∗11 θ∗12... ... ...

x∗m θ∗m1 θ∗m2

and specifying

wi = (wi(x∗1, θ

∗11, θ

∗12), . . . , wi(x

∗m, θ∗m1, θ

∗m2))

T

gives

wiiid∼ N

(0, λ−1

wi R((x∗, θ∗); ρwi)), i = 1, . . . , pη

where R((x∗, θ∗); ρwi)m×m is given by (??).

*note: additional nugget term wiiid∼ N

(0, λ−1

wi R((x∗, θ∗); ρwi) + λ−1

ǫi Im

), i = 1, . . . , pη, may be useful.

99

Page 100: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models for PC weightsAt the m simulation input settings the mpη-vector w has prior disribution

w =

w1...

wpη

∼ N

0...0

,

λ−1w1R((x∗, θ∗); ρw1) 0 0

0 . . . 00 0 λ−1

wpηR((x∗, θ∗); ρwpη)

⇒ w ∼ N(0, Σw);

note Σw = Ipη ⊗ λ−1w R((x∗, θ∗); ρw) can break down.

Emulator likelihood: η = vec([η(x∗1, θ

∗11, θ

∗12); · · · ; η(x∗

m, θ∗m1, θ∗m2)])

L(η|w,λη) ∝ λmnη

2η exp

{− 1

2λη(η − Kw)T (η − Kw)

}, λη ∼ Γ(aη, bη)

where nη is the number of observations in a simulated trace and

K = [Im ⊗ k1; · · · ; Im ⊗ kpη].Equivalently

L(η|w,λη) ∝ λmpη

2η exp

{− 1

2λη(w − w)T (KTK)(w − w)

λm(nη−pη)

2η exp

{− 1

2ληη

T (I − K(KTK)−1KT )η}

∝ λmpη

2η exp

{− 1

2λη(w − w)T (KTK)(w − w)

}, λη ∼ Γ(a′η, b

′η)

a′η = aη +m(nη−pη)

2, b′η = bη + 1

2ηT (I −K(KTK)−1KT )η, w = (KTK)−1KTη.

100

Page 101: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Gaussian process models for PC weights

Resulting posterior can then be based on computed PC weights w:

w|w,λη ∼ N(w, (ληKTK)−1)

w|λw, ρw ∼ N(0, Σw)

⇒ w|λη, λw, ρw ∼ N(0, (ληKTK)−1 + Σw)

Resulting posterior is then:

π(λη, λw, ρw|w) ∝∣∣(ληK

TK)−1 + Σw

∣∣−12 exp{− 1

2wT ([ληK

TK]−1 + Σw)−1w} ×

λa′η−1η e−b′ηλη ×

pη∏

i=1

λaw−1wi e−bwλwi ×

pη∏

i=1

px∏

j=1

(1 − ρwij)bρ−1

pθ∏

j=1

(1 − ρwi(j+px))bρ−1

MCMC via Metropolis works fine here.

Bounded range of ρwij’s facilitates MCMC.

101

Page 102: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior distribution of ρw

1 2 30

0.5

1

PC

1

[x θ]

1 2 30

0.5

1P

C2

[x θ]

1 2 30

0.5

1

PC

3

[x θ]

Separate models by PC

More opportunity to take advantage of effect sparsity

102

Page 103: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Predicting simulator output at untried (x⋆, θ⋆1, θ

⋆2)

Want η(x⋆, θ⋆1, θ

⋆2) = Kw(x⋆, θ⋆

1, θ⋆2)

For a given draw (λη, λw, ρw) a draw of w⋆ can be produced:(

ww⋆

)∼ N

((00

),

[((ληK

TK)−1 00 0

)+ Σw,w⋆(λw, ρw)

])

Define

V =

(V11 V12

V21 V22

)=

[((ληK

TK)−1 00 0

)+ Σw,w⋆(λw, ρw)

]

Thenw⋆|w ∼ N(V21V

−111 w, V22 − V21V

−111 V12)

Realizations can be generated from sample of MCMC output.

Lots of info (data?) makes conditioning on point estimate (λη, λw, ρw) a goodapproximation to the posterior.

Posterior mean or median work well for (λη, λw, ρw)

103

Page 104: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Comparing emulator predictions to holdout simulations

emulator 90% prediction bands and actual (holdout) simulations

0 1 2 3 4 5 6

x 10−5

0.8

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

time µs

inne

r ra

dius

(cm

)

o holdout simulation

− 90% emulator bounds

104

Page 105: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Exploring sensitivity of simulator output to model inputs

Simulator predictions varing 1 input, holding others at nominal

105

Page 106: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Basic formulation – borrows from Kennedy and O’Hagan (2001)

05

02

4x 10−5

0

1

2

φ

Experiment 1

time

r, η

05

02

4x 10−5

−1

0

1

φtime

δ

05

02

4x 10−5

0

1

2

φtime

r, η

δ

x = me/m ≈ .32θ1 = s ≈ ?θ2 = u0 ≈ ?

(t, φ) simulation output spacex experimental conditionsθ calibration parametersζ(x) true physical system response given conditions xη(x, θ) simulator response at x and θ.y(x) experimental observation of the physical systemδ(x) discrepancy between ζ(x) and η(x, θ)

may be decomposed into numerical error and biase(x) observation error of the experimental data

y(x) = ζ(x) + e(x)

y(x) = η(x, θ) + δ(x) + e(x)

106

Page 107: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Kernel basis representation for spatial processes δ(s)Define pδ basis functions d1(s), . . . , dpδ

(s).

−2 0 2 4 6 8 10 12

0.0

0.4

0.8

s

basi

s

Here dj(s) is normal density cetered at spatial location ωj:

dj(s) =1√2π

exp{−1

2(s − ωj)

2}

set δ(s) =

pδ∑

j=1

dj(s)vj where v ∼ N(0, λ−1v Ipδ

).

Can represent δ = (δ(s1), . . . , δ(sn))T as δ = Dv where

Dij = dj(si)

107

Page 108: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

v and d(s) determine spatial processes δ(s)

dj(s)vj δ(s)

−2 0 2 4 6 8 10 12

−0.

50.

51.

5

s

basi

s

−2 0 2 4 6 8 10 12

−0.

50.

51.

5

s

z(s)

Continuous representation:

δ(s) =

pδ∑

j=1

dj(s)vj where v ∼ N(0, λ−1v Ipδ

).

Discrete representation: For δ = (δ(s1), . . . , δ(sn))T , δ = Dv where Dij =

dj(si)

108

Page 109: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Basis representation of discrepancy

time

angle φ

Represent discrepancy δ(x) using basis functions and weights

pδ = 24 basis functions over (t, φ); D = [d1; · · · ; dpδ]; dk’s hold basis.

δ(x) = Dv(x) where v(x) ∼ GP(0, λ−1

v Ipδ⊗ R(x, x′; ρv)

)

with

R(x, x′; ρv) =

px∏

k=1

ρ4(xk−x′k)

2

vk (2)

109

Page 110: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Integrated model formulation

Data y(x1), . . . , y(xn) collected for n experiments at input conditions x1, . . . , xn.

Each y(xi) is a collection of nyimeasurements over points indexed by (t, φ).

y(xi) = η(xi, θ) + δ(xi) + ei

= Kiw(xi, θ) + Div(xi) + ei

y(xi)|w(xi, θ), v(xi), λy ∼ N

([Di; Ki]

(v(xi)

w(xi, θ)

), (λyWi)

−1

)

Since support of each y(xi) varies and doesn’t match that of sims, the basisvectors in Ki must be interpolated from Kη; similary, Di must be computedfrom the support of y(xi):

0

pi

2pi

0

5x 10−5

−1

−0.5

0

angletime

r

0

pi

2pi 0 2 4x 10

−5

−0.05

0

0.05

timeangle

r

0

pi

2pi

05 x 10

−5

−10

−5

0

5

x 10

timeangle

r*note: cubic spline interpolation over (time, φ) used here.

110

Page 111: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Integrated model formulation

Define

ny = ny1 + · · · + nyn, the total number of experimental data points,

y to be the ny-vector from concatination of the y(xi)’s,

v = vec([v(x1); · · · ; v(xn)]T ) and

u(θ) = vec([w(x1, θ1, θ2); · · · ; w(xn, θ1, θ2)]T )

y|v, u(θ), λy ∼ N

(B

(v

u(θ)

), (λyWy)

−1

), λy ∼ Γ(ay, by) (3)

where

Wy = diag(W1, . . . ,Wn) and

B = diag(D1, . . . , Dn,K1, . . . ,Kn)

(P T

D 00 P T

K

)

PD and PK are permutation matricies whose rows are given by:

PD(j + n(i − 1); ·) = eT(j−1)pδ+i, i = 1, . . . , pδ; j = 1, . . . , n

PK(j + n(i − 1); ·) = eT(j−1)pη+i, i = 1, . . . , pη; j = 1, . . . , n

111

Page 112: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Integrated model formulation (continued)

Equivalently (??) can be represented(

vu

) ∣∣∣∣(

vu(θ)

), λy ∼ N

((v

u(θ)

), (λyB

TWyB)−1

), λy ∼ Γ(a′y, b

′y)

with

ny = ny1 + · · · + nyn, the total number of experimental data points(vu

)= (BTWyB)−1BTWyy

a′y = ay + 1

2[ny − n(pδ + pη)]

b′y = by +1

2

[(y − B

(vu

))T

Wy

(y − B

(vu

))]

dimension reduction

model simulator data and discrep

standard nη · m ny

basis pη · m n · (pδ + pη)

Basis approach particularly efficient when nη and ny are large.

112

Page 113: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Marginal likelihood

The (marginal) likelihood L(v, u, w|λη, λw, ρw, λy, λv, ρv, θ) has the form

vuw

∼ N

000

,

Λ−1

y00

0 0 Λ−1η

+

Σv 0 000

Σuw

where

Λy = λyBTWyB

Λη = ληKTK

Σv = λ−1v Ipη ⊗ R(x, x; ρv)

R(x, x; ρv) = n × n correlation matrix from applying (??) to the conditionsx1, . . . , xn corresponding the the n experiments.

Σuw =

λ−1

w1R((x, θ), (x, θ); ρw1) 0 0 λ−1

w1R((x, θ), (x∗, θ∗); ρw1) 0 0

0 . . . 0 0 . . . 0

0 0 λ−1

wpηR((x, θ), (x, θ); ρwpη

) 0 0 λ−1

wpηR((x, θ), (x∗, θ∗); ρwpη

)

λ−1

w1R((x∗, θ∗), (x, θ); ρw1) 0 0 λ−1

w1R((x∗, θ∗), (x∗, θ∗); ρw1) 0 0

0 . . . 0 0 . . . 00 0 λ−1

wpηR((x∗, θ∗), (x, θ); ρwpη

) 0 0 λ−1

wpηR((x∗, θ∗), (x∗, θ∗); ρwpη

)

Permutation of Σuw is block diagonal ⇒ can speed up computations.

Only off diagonal blocks of Σuw depend on θ.

113

Page 114: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior distribution

Likelihood: L(v, u, w|λη, λw, ρw, λy, λv, ρv, θ)

Prior: π(λη, λw, ρw, λy, λv, ρv, θ)

⇒ Posterior:

π(λη, λw, ρw, λy, λv, ρv, θ|v, u, w) ∝ L(v, u, w|λη, λw, ρw, λy, λv, ρv, θ) ×π(λη, λw, ρw, λy, λv, ρv, θ)

Posterior exploration via MCMC

Can take advantage of structure and sparcity to speed up sampling.

A useful approximation to speed up posterior evaluation:

π(λη, λw, ρw, λy, λv, ρv, θ|v, u, w)

∝ L(w|λη, λw, ρw) × π(λη, λw, ρw) ×L(v, u|λη, λw, ρw, λy, λv, ρv, θ) × π(λy, λv, ρv, θ)

In this approximation, experimental data is not used to inform about parametersλη, λw, ρw which govern the simulator process η(x, θ).

114

Page 115: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior distribution of model parameters (θ1, θ2)

115

Page 116: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior mean decomposition for each experiment

05

02

4x 10−5

0

1

2

φ

Experiment 1

time

r, η

05

02

4x 10−5

−1

0

1

φtime

δ

05

02

4x 10−5

0

1

2

φtime

r, η

05

02

4x 10−5

0

1

2

φ

Experiment 2

time

r, η

05

02

4x 10−5

−1

0

1

φtime

δ

05

02

4x 10−5

0

1

2

φtime

r, η

05

02

4x 10−5

0

1

2

φ

Experiment 3

time

r, η

05

02

4x 10−5

−1

0

1

φtime

δ0

5

02

4x 10−5

0

1

2

φtimer,

η+

δ

116

Page 117: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

Posterior prediction for implosion in each experiment

0

5

0

5x 10−5

0.51

1.52

2.5

φ

Experiment 1

time

r

0

5

0

5x 10−5

0.51

1.52

2.5

φ

Experiment 2

time

r

0

5

0

5x 10−5

0.51

1.52

2.5

φ

Experiment 3

time

r

0

5

0

5x 10−5

0.51

1.52

2.5

φ

Experiment 1

time

r r

117

Page 118: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

90% prediction intervals for implosions at exposure times

−2 0 2

−2

−1

0

1

2

Tim

e 15

µs

−2 0 2

−2

−1

0

1

2

Tim

e 27

µs

−2 0 2

−2

−1

0

1

2

Tim

e 45

µs

−2 0 2

−2

−1

0

1

2

Tim

e 45

µs

−2 0 2

−2

−1

0

1

2

Tim

e 25

µs

−2 0 2

−2

−1

0

1

2

Tim

e 45

µs

Experiment 1 Experiment 2 Experiment 3

Predictions from separate analyses which hold data from the experiment being predicted.

118

Page 119: BAYESIAN MODELING AND CALIBRATION OF COMPUTER MODELS · OF COMPUTER MODELS Bayesian inference & Markov chain Monte Carlo Gaussian processes, Computer model calibration and prediction

References

• Heitmann, K., Higdon, D., Nakhleh, C. and Habib, S. (2006). Cosmic Cali-bration. To appear in Astrophysical Journal Letters.

• Williams, B., Higdon, D., Moore, L., McKay, M. and Keller-McNulty S.(2006). Combining Experimental Data and Computer Simulations, with anApplication to Flyer Plate Experiments, to appear in Bayesian Analysis.

• D. Higdon, J. Gattiker and B. Williams (2005). Computer Model Calibrationusing High Dimensional Output. LA-UR-05-6410.

• Bayarri, Berger, Garcia-Donato, Liu, Palomo, Paulo, Sacks, Walsh, Cafeo,and Parthasarathy (2006). Computer Model Validation with Functional Out-put. To appear in Annals of Statistics.

119


Recommended