Engineering subprogramme, 7 November 2006

Tony O’Hagan

Outline

Three parts:

• Turbofan engine vibration model• Reification• Predictors and validation

Part 1: The new model

Turbofan vibration model

Rolls-Royce, Derby, UKMaker of civil aeroplane engines

Simulator of a fan assembly

Our example has 24 blades

Primary concern is with vibration

If amplitude is too high on any one blade it may breakIn effect this will destroy the engine

Rolls-Royce Trent 500 engine

Model details

24 inputs are vibration resonant frequency of each blade

24 outputs are amplitude of vibration for each blade

Other factorsAmount of damping – more results in more complex behaviour and longer model run times

Model resolution – it’s possible to run the solver on higher or lower resolution grids

Could also vary e.g. number of blades, operating rpm and temperature

Parameter uncertainty

It’s not possible to manufacture and assemble blades to be all identical and perfectly oriented

Variation in resonant frequencies of blades creates complex variations in their vibration amplitude

Uncertainty distribution on each model input is the distribution achieved within manufacturing tolerances

Question:Given an assembly of bladessampled from this distribution,what is the risk of high amplitudevibrations resulting?

Emulation

Strategy:Emulate single output = blade 1 amplitude

24 inputs = frequencies of blades 1 to 24Because of rotational symmetry, each model run gives up to 24 design points

Simulate random blade assemblies

ResultsOutput depends most strongly on blade 1 inputAlso on neighbouring inputs, 2 and 24, etcBut high-order dependencies on all inputsSo far we’ve failed to emulate accurately even with very many design points

Challenges

What’s going on here?

Can we find a way to achieve the original strategy?

Should we try instead to emulate max amplitude?

This may also be badly behaved!

Part 2: Reification

Reification – background

Kennedy & O’Hagan (2001), “Bayesian calibration of computer models”

KO’H henceforth

Goldstein & Rougier (2006), “Reified Bayesian modelling and inference for physical systems”

GR henceforth

GR discuss two problems with KO’H1. Meaning of calibration parameters is unclear

2. Assuming stationary model discrepancy, independent of code, is inconsistent if better models are possible

Reification is their solution

Meaning of calibration parameters

The model is wrong

We need prior distributions for calibration parameters

Some may just be tuning parameters with no physical meaning

How can we assign priors to these?

Even for those that have physical meanings, the model may fit observational data better with wrong values

What does a prior mean for a parameter in a wrong model?

Example: some kind of machine

Simulator says output is proportional to inputEnergy in gives work out

Proportionality parameter has physical meaning

Observations with errorWithout model discrepancy, this is a simple linear model

LS estimate of slope is 0.568

But true parameter valueis 0.65

X Y

1.0 0.559

1.2 0.693

1.4 0.868

1.6 0.913

1.8 1.028

2.0 1.075

Model discrepancy

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Red line is LS fit

Black line is simulator with true parameter 0.65

Model is wrongIn reality there are energy losses

0

0.5

1

1.5

2

2.5

3

1 2 3 4 5x

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Case 1

Suppose we haveNo model discrepancy term

Weak prior on slope

Then we’ll getCalibration close to LS value, 0.568

Quite good predictive performance in [0, 2+]

Poor estimation of physical parameter

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Case 2

Suppose we haveNo model discrepancy term

Informative prior on slope based on knowledge of physical parameter

Centred around 0.65

Then we’ll getCalibration between LS and prior values

Not so good predictive performance

Poor estimation of physical parameter

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Without model discrepancy

Calibration is just nonlinear regressiony = f(x, θ) + e

Where f is the computer code

Quite good predictive performance can be achieved if there is a θ for which the model gets close to reality

Prior information based on physical meaning of θ can be misleading

Poor calibration

Poor prediction

Case 3

Suppose we haveGP model KO’H discrepancy term with constant mean

Weak prior on mean

Weak prior on slope

Then we’ll getCalibration close to LS value for regression with non-zero intercept

The GP takes the intercept

Slope estimate is now even further from the true physical parameter value, 0.518, albeit more uncertain

Discrepancy estimate ‘corrects’ generally upwards

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Case 4

Suppose we haveGP model KO’H discrepancy term with constant mean

Weak prior on mean

Informative prior on slope based on knowledge of physical parameter

Centred around 0.65

Then we’ll getSomething like linear regression with informative prior on the slope

Slope estimate is a compromise and loses physical meaning

Predictive accuracy weakened

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Adding simple discrepancy

Although the GP discrepancy of KO’H is in principle flexible and nonparametric, it still fits primarily on its mean function

Prediction looks like the result of fitting the regression model with nonlinear f plus the discrepancy mean

This process does not give physical meaning to the calibrated parameters

Even with informative priors

The augmented regression model is also wrong

Reification

GR introduce a new entity, the ‘reified’ modelTo reify is to attribute the status of reality

Thus, a reified simulator is one that we can treat as real, and in which the calibration parameters should take their physical values

Hence prior distributions on them can be meaningfully specified and should not distort the analysis

GR’s reified model is a kind of thought experimentIt is conceptually a model that corrects such (scientific and computational) deficiencies as we can identify in f

The GR reified model is not regarded as perfectIt still has simple additive model discrepancy as in KO’H

The discrepancy in the model is now made up of two parts

Difference between f and the reified modelFor which there is substantive prior information

Discrepancy of the reified modelIndependent of both models

Reification doubts

Can the reified model’s parameters be regarded as having physical meaning?

Allowing for model discrepancy between the reified model and reality makes this questionable

Do we need the reified model?Broadly speaking, the decomposition of the original model’s discrepancy is sensible

But it amounts to no more than thinking carefully about model discrepancy and modelling it as informatively as possible

Case 5

Suppose we haveGP model discrepancy term with mean function that reflects the acknowledged deficiency of the model in ignoring losses to friction

Informative prior on slope based on knowledge of physical parameter

Then we’ll getSomething more like the original intention of bringing in the model discrepancy!

Slope parameter not too distorted, model correction having physical meaning, good predictive performance

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1 1.2 1.4 1.6 1.8 2x

Moral

There is no substitute for thinking

Model discrepancy should be modelled as informatively as possible

Inevitably, though, the discrepancy function will to a greater or lesser extent correct for unpredicted deficiencies

Then the physical interpretations of calibration parameters can be compromised

If this is not recognised in their priors, those priors can distort the analysis

Final comments

There is much more in GR than I have dealt with here

Definitely repays careful reading

E.g. relationships between different simulators of the same reality

Their paper will appear in JSPI with discussionThis presentation is a pilot for my discussion!

Part 3: Validation

Simulators, emulators, predictors

A simulator is a model, representing some real world process

An emulator is a statistical description of a simulator

Not just a fast surrogate

Full probabilistic specification of beliefs

A predictor is a statistical description of realityFull probabilistic specification of beliefs

Emulator + representation of relationship between simulator and reality

Validation

What can be meaningfully called validation?Validation should have the sense of demonstrating that something is right

The simulator is inevitably wrong There is no meaningful sense in which we can validate it

What about the emulator?It makes statements like, “We give probability 0.9 to the output f(x) lying in the range [a, b] if the model is run with inputs x.”This can be right in the sense that (at least) 90% of such intervals turn out to contain the true output

Validating the emulator

Strictly, we can’t demonstrate that the emulator actually is valid in that sense

The best we can do is to check that the truth on a number of new runs lies appropriately within probability boundsAnd apply as many such checks as we feel we need to give reasonable confidence in the emulator’s validity

In practice, check it against as many (well-chosen) new runs as possible

Do Q-Q plots of standardised residuals and other diagnostic checks

Validating a predictor

The predictor is also a stochastic entity

We can validate it in the same wayAlthough getting enough observations of reality may be difficult

We may have to settle for the predictor not being yet shown to be invalid!

Validity, quality, adequacy

So, a predictor/emulator is valid if the truth lies appropriately within probability bounds

Could be conservativeNeed severe testing tools for verification

The quality of a predictor is determined by how tight those bounds are

Refinement versus calibration

A predictor is adequate for purpose if the bounds are tight enough

If we are satisfied the predictor is valid over the relevant range we can determine adequacy

Conclusion – terminology

I would like to introduce the word ‘predictor’, alongside the already accepted ‘emulator’ and ‘simulator’

I would like the word ‘validate’ to be used in the sense I have done above

Not in the sense that Bayarri, Berger, et al have applied it, which has more to do with fitness for purpose

And hence involves not just validity but quality

Models can have many purposes, but validity can be assessed independently of purpose