-
6th International Symposium on Imprecise Probability: Theories
and Applications, Durham, United Kingdom, 2009
Bayes Linear Analysis of Imprecision in Computer Models,
withApplication to Understanding Galaxy Formation
Ian VernonDepartment of Mathematical Sciences
Durham [email protected]
Michael GoldsteinDepartment of Mathematical Sciences
Durham [email protected]
Abstract
Imprecision arises naturally in the context of com-puter models
and their relation to reality. An im-precise treatment of general
computer models is pre-sented, illustrated with an analysis of a
complexgalaxy formation simulation known as Galform. Theanalysis
involves several different types of uncertainty,one of which (the
Model Discrepancy) comes di-rectly from expert elicitation
regarding the deficien-cies of the model. The Model Discrepancy is
thereforetreated within an Imprecise framework to reflect
moreaccurately the beliefs of the expert concerning the
dis-crepancy between the model and reality. Due to theconceptual
complexity and computationally intensivenature of such a Bayesian
imprecise uncertainty anal-ysis, Bayes Linear Methodology is
employed whichrequires consideration of only expectations and
vari-ances of all uncertain quantities. Therefore incorpo-rating an
Imprecise treatment within a Bayes Lin-ear analysis is shown to be
relatively straightforward.The impact of an imprecise assessment on
the inputspace of the model is determined through the use ofan
Implausibility measure.
Keywords. Bayesian Inference, Computer models,Calibration,
Imprecise model discrepancy, Implausi-bility, Galaxy Formation,
Graphical Representationof Model Imprecision.
1 Introduction
Computer models make imprecise statements aboutphysical systems.
This arises because of compromisesmade in the physical theory and
in approximationsto solutions of very complex systems of
equations.Therefore any statement about a physical system,
forexample climate change, which is derived from theanalysis of
computer models will be necessarily imper-fect, as it will usually
be very difficult to put a precisequantification on the discrepancy
between the modelanalysis and the physical system [1]. A full
probabilis-
tic representation of the imprecision arising from suchmodel
discrepancy will typically be very complex anddifficult to analyse.
However, there is an alternativeway to express such imprecision,
based on viewingexpectations rather than probability as the
naturalprimitive for expressing uncertainty statements.
Thisformulation allows us to focus directly on ‘high level’summary
expressions of imprecision. This approachis termed Bayes Linear
Analysis; for a detailed treat-ment see [2].
In this paper we show how the Bayes Linear approachmay be used
to capture the most important featuresof the imprecision arising
from the use of complexphysical models. We illustrate our approach
withthe galaxy formation model known as Galform. Gal-form simulates
the formation and evolution of approx-imately 1 million galaxies
from the beginning of theUniverse until the current day (a period
of approxi-mately 13 billion years). It gives outputs
representingvarious physical features of each of the galaxies
whichcan be compared with observational data [3].
This paper is structured as follows: in section 2 wediscuss the
Galform model in more detail, in section3 the theory of computer
models and the incorpora-tion of the imprecise model discrepancy is
described,and in section 4 we develop appropriate graphical
dis-plays for such imprecise analyses and demonstrate
theapplication of these methods to the Galform model.
2 Cosmology and Galaxy Formation
2.1 Understanding the Universe
Over the last 100 years, major advances have beenmade in
understanding the large scale structure ofthe Universe. Current
theories of cosmology suggestthat the Universe began in a hot,
dense state approx-imately 13 billion years ago, and that it has
been ex-panding rapidly ever since. However, there exists amajor
problem: observations of galaxies imply that
-
there must exist far more matter in the Universe thanthe visible
matter that makes up stars, planets and us.This is referred to as
‘Dark Matter’ and understand-ing its nature and how it has affected
the evolutionof galaxies within our Universe is one of the most
im-portant problems in modern cosmology.
In order to study many of the effects of Dark Mat-ter,
cosmologists try to model Galaxy formation usingcomplex computer
models. In this paper, we developthe Bayesian treatment of
imprecision for computermodels, and illustrate our analysis using
one suchmodel, known as Galform (developed by the Galformgroup at
the Institute for Computational Cosmology,Durham University).
2.2 Galform: a Galaxy FormationSimulation
Simulating the formation of large numbers of galax-ies from the
beginning of the Universe until the cur-rent day is a difficult
task and so the process is splitinto two parts. First a Dark Matter
simulation isperformed to determine the behaviour of fluctuationsof
mass in the early Universe, and their subsequentgrowth into
millions of galaxy sized lumps in the fol-lowing 13 billion years.
Second, the results of theDark Matter simulation are used by a more
detailedmodel called Galform which models the far more com-plicated
interactions of normal matter including: gascloud formation,
radiative cooling, star formation andthe effects of central black
holes.
The first simulation is run on a volume of space ofsize (1.63
billion light-years)3. This volume is splitinto 512 sub-volumes
which are independently simu-lated using the second model Galform,
which is thesubject of the Imprecise Uncertainty Analysis in
thispaper (see figure 1). Each run of Galform takes 20-30minutes
per subvolume per processor.
2.3 Galform Inputs and Outputs
The Galform simulation provides many outputs re-lated to
approximately 1 million simulated galaxies.We consider the two most
important types of out-put: the bj and K band luminosity functions.
Thebj band luminosity function gives the number of blue(i.e. young)
galaxies of a certain luminosity per unitvolume, while the K band
luminosity function de-scribes the number of red (i.e. old)
galaxies (see Fig-ure 1). The colour of a galaxy comes from the
starsit contains, stars which on average burn bluer earlyin their
lifecycle and redder as they age. These out-puts can be compared to
observational data gatheredby the 2dFGRS galaxy survey (see [3] and
referencestherein).
Figure 1: Top 4 panels: the evolution of both the DarkMatter
Simulation and Galform over a 13 billion yearperiod. Darker areas
show higher concentrations ofDark Matter, leading to the formation
of bright galax-ies (the white dots). Bottom 2 panels: the bj and
Kluminosity functions. The grey lines are from 60 runsof the
Galform simulation. The black points are ob-served data from the
2dFGRS survey with associatedmeasurement errors.
-
Galform has 17 input parameters that the cosmolo-gists were
interested in varying. Due to expert judge-ments regarding the
impact of these inputs on theluminosity functions we attempted to
calibrate Gal-form over only 8 of the input parameters (while
takinginto account the possible effects of the remaining 9).These
input parameters and their initial ranges are:
vhotdisk: 100 - 550aReheat: 0.2 - 1.2alphacool: 0.2 -
1.2vhotburst: 100 - 550epsilonStar: 0.001 - 0.1stabledisk: 0.65 -
0.95alphahot: 2 - 3.7yield: 0.02 - 0.05
The other 9 parameters are: VCUT, ZCUT, alphas-tar, tau0mrg,
fellip, fburst, FSMBH, epsilonSMB-HEddington and tdisk.
2.4 Galaxy Formation: Main Issues
The main physical questions that the cosmologistsare interested
in are: do we understand how galax-ies form, and could the galaxies
we observe have beenformed in the presence of large amounts of dark
mat-ter? In order to answer these questions it is vital tocorrectly
analyse all relevant sources of uncertaintywithin this situation.
Many of the sources of uncer-tainty derive from aspects of the
problem for which wehave a good physical understanding, for
example, thevarious types of measurement error associated withthe
observational data (which mainly come from op-tical deficiencies of
telescopes).
However, by far the most important uncertaintiesarise from the
fact that we are uncertain about thediscrepancy between the Galform
model and the realsystem, and we are also uncertain about which
choiceof input should be made when running the model.
3 Bayes Linear Analysis forComputer Simulators
To understand and describe all the sources of uncer-tainty in
the Galform simulator we apply computermodel emulation techniques.
Although here we willonly discuss the Galform simulator, these
techniquesare very general and can be applied to any com-plex model
of a physical system. Indeed they havebeen successfully applied to
a wide variety of physi-cal models (see [5] for a Bayes Linear
approach, [4]for a fully Bayesian approach, and for an overviewof
computer experiments in general see [6] or theManaging Uncertainty
in Complex Models website
http://mucm.group.shef.ac.uk/index.html).
3.1 Main Objectives
A common aim of computer experiment analysis is touse observed
data to reduce uncertainty about possi-ble choices of the input
parameters x (see [5] and [4]).In many problems the major interest
lies in whetherthere is any choice of x that would lead to an
ac-ceptable match between model outputs and observeddata. The
larger the assessed discrepancy betweenmodel and system, the weaker
the constraints the ob-servations will impose on this choice. In
this workwe treat this discrepancy as imprecise. Therefore oneof
the most important aspects of the analysis of themodel lies in
identifying and quantifying the impactof such imprecision on the
choice of possible inputvalues.
3.2 Computer simulators
The simulator (Galform) is represented as a function,which maps
the input parameters x to the outputsf(x). We use the “Best Input
Approach”, where weassume there exists a value x∗ independent of
thefunction f such that the value f∗ = f(x∗) summarisesall the
information the simulator conveys about thesystem. In order to make
meaningful statementsabout the system, denoted y, in relation to
the model,we link the simulator to the system using the
modeldiscrepancy denoted !md via the equation:
y = f∗ + !md, (1)
and assume that !md is independent of f and x∗, thatis,
independent in terms of our own beliefs.
The Model Discrepancy term !md links the real sys-tem y to the
best evaluation of the model representedby f∗. This is distinct
from other sources of uncer-tainty in our analysis and comes
directly from expertopinion regarding the ‘accuracy’ of the model.
Un-derstanding the nature of !md is a non-trivial taskas there are
various other sources of uncertainty thatare present that interfere
with any assessment of !md.For example, we can never measure the
real systemy directly. Instead we have measurements z observedwith
experimental error !obs which are linked to thesystem by:
z = y + !obs. (2)
Another important source of uncertainty is due to lackof
knowledge about the form of the function f(x). Asthe model takes a
significant time to run and has ahigh dimensional input space we
only have limitedknowledge about its behavior. Further, there is
un-certainty regarding the best input value of x∗ thatfeatures in
the definition of !md (equation (1)).
-
These other types of uncertainty make understanding!md
difficult, which is a significant problem as often!md is the most
important source of uncertainty dueto its size and nature. Due to
these difficulties, the ex-pert will often be imprecise over the
assessment of themodel discrepancy, and even more imprecision
couldoccur when we consider the opinions of a group of ex-perts. It
is therefore reasonable to analyse !md withinan imprecise
framework, while treating other less sig-nificant (and more
understood) sources of uncertaintyas precise.
We need to understand the behavior of the Galformsimulation
f(x): this is done by representing our be-liefs about f(x) as a
statistical function known asan Emulator, described in the next
section. We ad-dress the calibration problem (that of finding
inputsx that give rise to good matches between the outputsof f(x)
and the observed data z) by use of a tech-nique known as History
Matching [5]. This involvesdiscarding regions of the input
parameter space thatwe are reasonably sure will give bad fits to
the ob-served data, and we do this using an Implausibilitymeasure.
Analysing the effect on this measure of hav-ing an imprecise Model
Discrepancy !md (and the cor-responding effect on the History
Match) is the maingoal of this work.
3.3 Representing beliefs about f usingemulators
An emulator is a stochastic belief specification for
adeterministic function. This would be constructed af-ter
performing a large, space filling set of runs of themodel [6]. Our
emulator for component i of f is givenby:
fi(x) =∑
j
βij gij(x) + ui(x)
where B = {βij} are unknown scalars, gij are knowndeterministic
functions of x, and u(x) is a weakly sta-tionary stochastic
process. A simple specification isto suppose, for each x, that
ui(x) has zero mean withconstant variance and Corr(ui(x), ui(x′))
is a func-tion of ‖x− x′‖. From the emulator, we may extractthe
mean, variance and covariance for the function,at each input value
x.
µi(x) = E[fi(x)], κi(x, x′) = Cov(fi(x), fi(x′))
Often, because of the mode of construction, the ex-pectation of
the emulator interpolates known runs ofthe model, while the
variance represents uncertaintyof the function at x inputs that
have not been run. Akey feature of an emulator is that it is (in
most cases)several orders of magnitude faster to evaluate than
themodel itself. This is important as we will be exploring
high dimensional input spaces that necessitate largenumbers of
evaluations. Emulator techniques are vitalin the analysis of any
model that has a moderate/longrun time and a high dimensional input
space.
3.4 Bayes Linear approach
For large scale problems involving computer models,a full Bayes
analysis is hard for the following reasons.Firstly, it is very
difficult to give a meaningful fullprior probability specification
over high dimensionalinput spaces. Secondly, the computations for
learningfrom both observed data and runs of the model, andchoosing
informative runs, may be technically verychallenging. Thirdly, in
such computer model prob-lems, often the likelihood surface is
extremely compli-cated, and therefore any full Bayes calculation
maybe extremely non-robust. However, the idea of theBayesian
approach, namely capturing our expert priorjudgements in stochastic
form and modifying them byappropriate rules given observations, is
conceptuallyappropriate.
The Bayes Linear approach is (relatively) simple interms of
belief specification and analysis, as it is basedonly on the mean,
variance and covariance specifica-tion which, following de Finetti,
we take as primitive.It also allows a relatively straightforward
descriptionof imprecision which is vital for this work.
We replace Bayes Theorem (which deals with proba-bility
distributions) by the Bayes Linear adjustmentwhich is the
appropriate updating rule for expecta-tions and variances. The
Bayes Linear adjustment ofthe mean and the variance of y given z
is:
Ez[y] = E[y] + Cov(y, z)Var(z)−1(z − E[z]),Varz[y] = Var(y)−
Cov(y, z)Var(z)−1Cov(z, y)
Ez[y], Varz[y] are the expectation and variance for yadjusted by
z.
The Bayes linear adjustment may be viewed as an ap-proximation
to a full Bayes analysis, or more funda-mentally as the
“appropriate” analysis given a partialspecification based on
expectation (with methodologyfor modelling, interpretation and
diagnostic analysis).For more details see [2].
3.5 History Matching using ImplausibilityMeasures.
We can now use the emulator, the model discrepancyand the
measurement errors to calculate a UnivariateImplausibility Measure,
at any input parameter pointx, for each component i of the computer
model f(x).
-
This is given by:
I2(i)(x) = |E[fi(x)]− zi|2/Var(fi(x)− zi) (3)
which now becomes:
I2(i)(x) = |E[fi(x)]− zi|2/(Var(fi(x)) + IMD + OE)(4)
where E[fi(x)] and Var(fi(x)) are the emulator ex-pectation and
variance, zi are the observed data andIMD = Var(!md) and OE are the
(univariate) Impre-cise Model Discrepancy variance and
ObservationalError variance.
When I(i)(x) is large this implies that, even given allthe
uncertainties present in the problem, we wouldbe unlikely to obtain
a good match between modeloutput and observed data were we to run
the modelat input x. This means that we can cut down theinput space
by imposing suitable cutoffs on the im-plausibility function (a
process referred to as HistoryMatching). Regarding the size of
I(i)(x), if we as-sume that for fixed x the appropriate
distribution of(fi(x∗)− z) is unimodal, then we can use the 3σ
rulewhich implies that if x = x∗, then I(i)(x) < 3 witha
probability of approximately 0.95 (even if the dis-tribution is
asymmetric). Values higher than 3 wouldsuggest that the point x
should be discarded.
It should be noted that since the implausibility reliespurely on
means and variances (and therefore can beevaluated using Bayes
Linear methodology), it is bothtractable to calculate and simple to
specify and henceto use as a basis of imprecise analysis.
One way to combine these univariate implausibilitiesis by
maximizing over outputs:
IM (x) = maxi
I(i)(x) (5)
Using the above unimodal assumptions, values ofIM (x) of around
3.5 might suggest that x can be dis-carded, as is discussed in
section 4.2.
If we construct a multivariate model discrepancy, thenwe can
define a multivariate Implausibility measure:
I2(x) = (E[f(x)]− z)T Var(f(x)− z)−1(E[f(x)]− z),
which becomes:
(E[f(x)]−z)T (Var(f(x))+IMD+OE)−1(E[f(x)]−z).
Again, large values of I(x) imply that we would be un-likely to
obtain a good match between model outputand observed data were we
to run the model at inputx. Choosing a cutoff for I(x) is more
complicated. Asa simple heuristic, we might choose to compare
I(x)with the upper critical value of a χ2 distribution withdegrees
of freedom equal to the number of outputs.
4 Application to a Galaxy FormationSimulation
One of the long-term goals of the Galform project isto identify
the set of input parameters that give riseto acceptable matches
between outputs of the Gal-form model and observed data. We do this
using theHistory Matching ideas outlined above, the full de-tails
of which will be reported elsewhere. Before onecan embark on such a
process, the imprecise modeldiscrepancy must be constructed, and
its impact un-derstood, as we now describe.
We proceed to analyse the Galaxy Formation modelGalform using
the computer model techniques de-scribed above. We choose to
examine the mean ofthe first 40 sub-volumes (following the
cosmologists’own attempts to calibrate) and select 11 output
pointsfrom the bj and K luminosity graphs for use in ouranalysis,
as shown in figure 2.
First, 1000 evaluations of the model were made (alsoshown in
figure 2) using a space filling latin hypercubedesign across the
8-dimensional input space. Theseruns were used to construct an
emulator for Galformas discussed in section 3.3.
We now describe the imprecise model discrepancyused to capture
the cosmologist’s assessment of thediscrepancy between model and
reality, and then goon to examine the imprecise implausibility
measuresthis generates, and their impact on the judgement asto
which inputs x are deemed acceptable.
4.1 Imprecise Model Discrepancy
At this stage we need to assess the Model Discrep-ancy !md
related to all 11 outputs of interest. Thisis obtained from an
expert opinion regarding the dis-crepancy between the model and
reality, derived fromopinions about potential deficiencies of the
model. Asthis is a difficult assessment to make, an
imprecisequantification of the model discrepancy will often bethe
most realistic representation of such uncertainty.
As we are doing a Bayes Linear analysis we only needto consider
the assessment of E[!md] and Var(!md).This is a major benefit of
the Bayes Linear approachas we can represent any imprecision by
letting someof these quantities vary over specified ranges and
canthen explore the consequences in the rest of our anal-ysis. This
is straightforward in comparison to a fullyprobabilistic analysis
where such an imprecise specifi-cation would be extremely
difficult, and a subsequentexamination of the impact of such
imprecision wouldoften be intractable.
A leading expert stated that his beliefs regard-
-
Figure 2: The bj and K luminosity outputs from 1000runs of the
model. The vertical black lines show the11 outputs chosen for
emulation. The error bars nowincorporate the (univariate) model
discrepancy witha = a.
ing the model discrepancy were symmetric in thatE[!md] = 0.
Define IMD = Var(!md). Even for theunivariate case (i.e.
considering only one of the 11outputs) the individual expert was
unwilling to assessthe size of IMD precisely. However, the expert
waswilling to make an imprecise assessment by specifyinglower and
upper bounds IMD and IMD.
For the multivariate case, we needed to assessIMD = Var(!md)
which is now an 11x11 matrix. Thestructure of this matrix will come
from the expert’sopinion as to the deficiencies of the model. In
the caseof Galform there are two major physical defects thatcan be
identified. The first is the possibility that themodel has too much
(too little) mass in the simulateduniverse. This would lead to the
11 luminosity out-puts all being too high (or too low), and would
leadto positive correlation between all outputs in the MDmatrix.
The second possible defect is that the galaxiesmight age at the
wrong rate leading to more/less bluegalaxies and therefore
less/more red galaxies. Thiswould be represented as contributing a
smaller nega-tive correlation between the bj and K luminosity
out-puts. To respect the symmetries of these possible de-fects, the
multivariate Imprecise Model Discrepancy
(IMD) was parameterised in the following form:
IMD = a2
1 b .. c .. cb 1 .. c . c: : : : : :c .. c 1 b ..c .. c b 1 ..:
: : : : :
(6)
where now a, b and c are imprecise quantities,and we obtain the
following expert assessments:a = 3.76× 10−2, a = 7.52× 10−2, b =
0.4, b = 0.8,and c = 0.2, c = b.
It is possible to build in far more structure into IMDif
required. The more detailed the structure, the moredifficult
eliciting expert information becomes. How-ever, note the relative
ease of specifying useful high-level imprecise statements using
expectation as prim-itive, as compared to the corresponding effort
for afully probabilistic analysis. Exploring the effects ofthese
specifications is also an easier task, as we nowshow by examining
the effects of varying choices of a,b and c on the appropriate
implausibility measures.
4.2 Implausibility Measures
In section 3.5 we showed how to construct themaximised and
multivariate Implausibility measuresIM (x) and I(x). As these are
derived using the impre-cise model discrepancy we can write the
dependenceof these two implausibility measures on a, b and c
ex-plicitly. We can now explore the effects on IM (x, a)and I(x, a,
b, c) of varying a, b and c within the credalset C defined by:
a < a < a, b < b < b, c < c < b,
as is described in the next section. As the implausibil-ity
measures are now imprecise, in order for regions ofthe input space
x to be discarded as Implausible, theymust violate the
implausibility cutoff for all values ofa, b and c, that is:
I(x, a, b, c) > Icut ∀ a, b, c ∈ C, (7)
with a similar relation for IM (x, a):
IM (x, a) > IMcut ∀ a ∈ C. (8)
In section 4.3 we set Icut = 26.75 corresponding to acritical
value of 0.995 from a χ2 distribution with 11degrees of freedom
(and IMcut = 3.5) which were feltto be appropriate, conservative
choices for the cutoffs.Note that if an input x satisfies either
constraint (7) orconstraint (8) then it is deemed implausible and
willbe discarded. As can be seen from equations (6),(3)
-
and (5), IM (x, a) is a monotonically decreasing func-tion of a
and hence constraint (8) will be equivalentto:
mina∈C
IM (x, a) = IM (x, a) > IMcut (9)
The constraint for I(x, a, b, c) is more complex and ingeneral
no such monotonicity arguments can be used.In a full calibration
analysis we would, for fixed x,evaluate I(x, a, b, c) over a large
number of points inthe credal set C, and only discard the input x
if it doesnot satisfy the implausibility cutoffs for every one
ofthese points. However, here we are more interested
inunderstanding the impact of different choices of a, band c on the
input space, which we do in the nextsection.
4.3 Effect of the Imprecise ModelDiscrepancy on the Assessment
of theBest Input x∗
The most important effect of an imprecisely specifiedmodel
discrepancy is its impact upon the choice ofacceptable input
parameters x∗. Above we showedhow to construct the implausibility
measures and de-scribed their use in deciding which inputs would
bedeemed acceptable. Here we will explore the impactof the
imprecision on the multivariate measure itself,then on the
percentage of input space remaining, byanalysing the effects of
varying a, b and c. Note thatwhile we present all the pictures in
greyscale, thesedisplays are designed for presentation in
colour.
Figure 3 shows the multivariate implausibilityI(x, a, b, c) as a
function of a, b and c for two dif-ferent fixed values of x. In the
top (bottom) panel x7i.e. alphahot is set to its minimum (maximum)
valueof 2 (3.7). In both panels x1 i.e. vhotdisk is at itsmaximum
value of 550, and all other inputs are attheir midrange values. In
these and subsequent fig-ures we examine slightly larger ranges for
a, b and cthan are defined by the Credal Set: here they sat-isfy
0.5a < a < 2a, 0 < b < 0.95 and 0 < c < b.The top
panel shows that I(x, a, b, c) is minimisedfor large values of a, b
and c attaining a minimumof approximately I(x, a, b, c) = 14.2. In
the bot-tom panel however, the implausibility is minimisedfor low
values of b and c and only attains a minimumof I(x, a, b, c) =
38.3. This shows the dramaticallydifferent behaviour of the
implausibility measure as afunction of a, b and c for two different
parts of theinput space, and specifically that general
monotonic-ity arguments (such as used in equation (4.2)) can-not be
applied to the imprecise parameters b and c.Plots such as those
shown in figure 3 are very usefulin helping to understand the
impact of an impreciseassessment. However, one cannot examine such
plots
20
30
40
50
60
70
40
50
60
70
80
90
Figure 3: Both panels shows the multivariate implau-sibility
I(x, a, b, c) as a function of a, b and c for twodifferent fixed
values of x, with darker colours repre-senting lower
implausibility. Here a, b and c vary overthe ranges 0.5a < a
< 2a, 0 < b < 0.95 and 0 < c < b.Note that the scale
on the a-axis is in terms of mul-tiples of a. Top panel: vhotdisk =
550, alphahot =2, Bottom panel: vhotdisk = 550, alphahot = 3.7,
allother inputs set to their midrange values.
-
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Figure 4: Fraction of input space that survives the multivariate
implausibility cutoff given by equation (7) withIcut = 26.75 as a
function of a, b and c in the ranges 0.5a < a < 2a, 0 < b
< 0.95 and 0 < c < b. Note that thescale on the a-axis is
in terms of multiples of a.
for all points in the 8-dimensional input space. Wetherefore
look at other ways to summarise and visu-alise the analysis.
We can summarise the effect of the imprecise mul-tivariate
implausibility cutoff given by equation (7)on the whole of the
input space by looking at thefraction of space remaining once the
cutoff has beenimposed. Here we display the results correspondingto
Icut = 26.75, a value which was thought to be areasonably
conservative choice. Figure 4 shows thisfraction of space remaining
as a function of a, b andc in the ranges 0.5a < a < 2a, 0
< b < 0.95 and0 < c < b, with darker colours
representing higherfractions. Figure 5 shows the same 3D plot from
adifferent perspective. The 3D object has been cut in3 places to
allow one to see slices of the function atfixed values of a. This
shows that for large values ofa, the maximum space remaining would
occur for in-termediate values of b and c (approximately b = 0.7and
c = 0.6 for a = 2), however for smaller values ofa the space
remaining would be maximised by large band c (e.g. for a = 0.5a =
a, b = 0.95 and c = 0.95:see figure 5). These plots also suggest
that the spaceremaining is far less sensitive to variation in b and
cthan in a: it is useful for the expert to know thereforethat their
assessment for a is more significant than forb and c.
Figure 6 shows the fraction of space remaining as afunction of a
for fixed choices of b and c. The bound-aries of the Credal Set are
shown by dotted verticallines. Again one can see that to maximise
the space
remaining requires intermediate values of b and c forlarge a,
and large values of b and c for small a. Alsonote that as a tends
to small values, the fraction ofspace remaining varies only slowly:
in fact settinga = 0 (which is not shown in this figure) leads
to0.017 of the input space remaining: this is importantfor the
expert to know as it shows that some of the in-put space would
survive the cutoff even for zero modeldiscrepancy.
Examining the space remaining is useful in under-stand the
effects of the imprecise specification ofmodel discrepancy.
However, it is also vital to assessthe effect on the input space
directly i.e. to determinewhich inputs x would not be discarded due
to the im-precise specification. One way to analyze this is toask
what is the minimum value of a that is requiredto ensure that a
particular input point x satisfies theimplausibility cutoff. Figure
7 shows 3D plots of therequired value of a as a function of the
input parame-ters x1 and x7, and of b (with the other inputs at
theirmidrange values , with c = 0, and the key in terms ofmultiples
of a). The darkest areas are those that havea required a of less
than a and hence would survive thecutoff for the current
specification. These plots showthat while the value of b has
effects in some parts ofthe input space, the region defined by
required a < ais relatively independent of the value of b (a
similar re-sult is seen for plots with varying c and fixed b).
Thisdemonstrates that the required value of a is far moresensitive
to the value of x1 and x7 as opposed to thespecified range of the
imprecise quantity b, and givesmore evidence to suggest that the
experts assessment
-
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Figure 5: Alternative view of fraction of input spacethat
survives the multivariate implausibility cutoffgiven by equation
(7) with Icut = 26.75 as a functionof a, b and c in the ranges 0.5a
< a < 2a, 0 < b < 0.95and 0 < c < b. Note that
the scale on the a-axis is interms of multiples of a.
0.5 1.0 1.5 2.0
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Fraction of Space remaining against a
a
Frac
tion
of s
pace
rem
aini
ng
b = 0 , c = 0b = 0.4 , c = 0.2b = 0.7 , c = 0.6b = 0.95 , c =
0.95
Figure 6: Fraction of input space that survivesthe multivariate
implausibility cutoff given by equa-tion (7) with Icut = 26.75 as a
function of a for therange 0.25a < a < 2a, for various
choices of b and c.The scale on the a axis is in terms of multiples
of a.Note that a = 0.5a. It can be seen that more spacesurvives
when b = 0.7 and c = 0.6 for large a, however,for smaller a the
more extreme values b = 0.95 andc = 0.95 are preferred (which are
not in the CredalSet).
for a is far more significant than that for b and c.
We have seen the effects of the imprecise assess-ment on the
multivariate implausibility measureI(x, a, b, c), on the fraction
of space remaining afterthe cutoff is imposed, and on the set of
allowed valuesof x1 and x7. We showed that these effects are
non-trivial as the multivariate implausibility measure is
acomplicated function of x, a, b and c.
5 Conclusions
We have discussed how computer models make impre-cise statements
about physical systems. This impre-cision arises due to the immense
difficulty in givinga precise quantification on the discrepancy
betweenthe model analysis and the system. We have shownhow use of
Bayes Linear methods can provide a rela-tively straightforward
description of this imprecision,allowing a meaningful elicitation
of imprecise modeldiscrepancy while leading to a tractable analysis
of theissues involved in computer model calibration, whichwe
demonstrated in the context of the galaxy forma-tion simulation
Galform.
The mathematical tractability of treating expectationas
primitive also allows a detailed study of the effectsof such
imprecise assessments. In this case this in-
-
0
1
2
3
4
5
6
0
1
2
3
4
5
6
Figure 7: Plots showing the value of a that is requiredto ensure
a point in input space satisfies the multi-variate cutoff, as a
function of the input parametersvhotdisk and alphahot, and of the
imprecise quantityb (with c set to 0). The key is in terms of
multi-ples of a, and the darker areas represent low requireda. All
other input parameters have been set to theirmidrange values.
volved understanding the impact of the imprecision onthe
implausibility measures; measures that were usedto discard regions
on input parameter space thoughtto be very unlikely to give rise to
acceptable matchesbetween model output and observed data. In this
waywe were able to show the direct impact on parts of theinput
space of the expert’s imprecise judgements re-garding model
deficiency. The effects of the impreciseassessments were found to
be non-trivial and a varietyof methods were used to summarise the
data in orderto produce meaningful visual representations of
sucheffects.
Acknowledgements
This paper was produced with the support of the Ba-sic
Technology initiative as part of the Managing Un-certainty for
Complex Models project, and with EP-SRC funding through a mobility
fellowship. We wouldlike to thank Prof Richard Bower (Institute for
Com-putational Cosmology, Physics Department, DurhamUniversity) for
providing the expert assessments thatfeature in this work. We would
also like to thank ProfRichard Bower and the Galfrom group (also
based atthe Institute for Computational Cosmology,
PhysicsDepartment, Durham University) for access to theGalform
model and to their computer resources.
References
[1] M. Goldstein and J.C.Rougier (2008). ReifiedBayesian
modelling and inference for physical sys-tems (with discussion),
JSPI, to appear, .
[2] Goldstein, M.,Wooff, D. (2007). Bayes LinearStatistics:
Theory and Methods. Wiley
[3] Bower, R.G., Benson, A. J. et.al.(2006).The Broken hierarchy
of galaxy formation,Mon.Not.Roy.Astron.Soc. 370, 645-655
[4] Kennedy, M.C. and O’Hagan, A. (2001). Bayesiancalibration of
computer models (with discussion).Journal of the Royal Statistical
Society, B,63, 425-464
[5] P.S. Craig, M. Goldstein, A.H. Seheult, J.A.Smith (1997).
Pressure matching for hydocarbonreservoirs: a case study in the use
of Bayes linearstrategies for large computer experiments
(withdiscussion), in Case Studies in Bayesian Statistics,vol. III,
eds. C. Gastonis et al. 37-93. Springer-Verlag.
[6] Santner, T., Williams, B. and Notz, W. (2003).The Design and
Analysis of Computer Experi-ments. Springer Verlag: New York.
