1

Dipak K. DeyDipak K. DeyUniversity of Connecticut University of Connecticut

Some parts joint with: Some parts joint with: Junfeng LiuJunfeng LiuCase Western Reserve UniversityCase Western Reserve University

Prior Elicitation from Expert Opinion

2

Elicitation

Elicitation is the process of extracting expert knowledge about some unknown quantity

of interest, or the probability of some future event, which can then be used to supplement any numerical data that we may have.

If the expert in question does not have a statistical background, as is often the case, translating their beliefs into a statistical

form suitable for use in our analyses can be a challenging task.

3

Introduction

Prior elicitation is an important and yet under researched component of Bayesian statistics.

In any statistical analysis there will typically be some form of background knowledge available in addition to the data at hand.

For example, suppose we are investigating the average lifetime of a component. We can do tests on a sample of components to learn about their average lifetime, but the designer/ engineer of the component may have their own expectations about its performance.

4

Introduction

If we can represent the expert's uncertainty about the lifetime through a probability distribution, then this additional (prior) knowledge can be utilized within the Bayesian framework.

With a large quantity of data, prior knowledge tends to have less of an effect on final inferences. Given this fact, and the various techniques available for representing prior ignorance, practitioners of Bayesian statistics are frequently spared the effort of thinking about the available prior knowledge.

5

Introduction

It will not always be the case that we will have sufficient data to be able to ignore prior knowledge, and one example of this would be in the uncertainty in computer models application or modeling extreme events.

Uncertain model input parameters are often assigned probability distributions entirely on the basis of expert judgments. In addition, certain parameters in statistical models can be hard to make inferences about, even with a reasonable amount of data.

6

Introduction

The amount of research in eliciting prior knowledge is relatively low, and various proposed techniques are often targeted at specific applications. At the same time, recent advances in Bayesian computation have allowed far greater flexibility in modeling prior knowledge. In general, elicitation can be made difficult by the fact that we cannot expect the expert to provide probability distributions for quantities of interest directly.

7

Introduction

The challenge is then to find appropriate questions to ask the expert in order to extract their knowledge, and then to determine a suitable probabilistic description of the variables we are interested in based on the information we have learned from them.

8

MotivationMotivation

Three approaches: [1] Direct Prior Elicitation: Berger (1985) Relative frequency, and quantile based

elicitation.

[2] Predictive prior probability space, which requires simple priors and may be burdened with additional uncertainties arising from the response model. (Kadane, et al, 1980; Garthwaite and Dickey, 1988, Al-Awadhi and Garthwaite,

1998, etc.).

[3] Nonparametric Elicitation: (Oakley and O’Hagan, 2002)

9

Symmetric Prior Elicitation

Double bisection method: Expert provides q(.25), q(.5) and q(.75), the three quantiles

IQR = q(.75)-q(.25) Normal prior: Z(q)= IQR of std. normal, then, prior

mean and std. dev. are, q(.5) and IQR/ Z(q) respectively.

10

Student’s t Prior

Three non redundant quantiles are required to estimate the df ν. Kadane et.al. (1980) suggested obtaining q(.5), q(.75) and q(.9375)

a(x) = (q(.9375)-q(.5))/(q(.75)-q(.5)) depends on df ν only

Df is determined from look up table of a(x) vs df ν.

11

Student’s t Prior

After elicitation of df obtain tν,0.75 Calculate S(q) = (q(.75)-q(.5)) 2/ t2

ν,0.75

for elicitation of scale parameter σ.

This idea can be applied to any general location-scale family.

12

Lognormal Prior

Garthwaite (1989) used split-normal distribution, O’Hagan (1998) used 1/6, 3/6 and 5/6 quantiles.

),(~ln 2NX

Proposition: If X has a log-normal distribution, i.e.,

)1()( 22250.0 rrqXD

50.0)( rqXE eq 50.0

qq

ZZ

qqrX ),2

)/(lnexp(, 225.75.

2

, then the varianceand the mean ,where is themedian of is the IQRfor standard normal distribution.

13

Direct Prior ElicitationDirect Prior Elicitation

(1) Simple and limited prior family with only location and scale parameters (normal, exponential, etc.)

(2) Location-scale-shape (µ--) parameter joint elicitation (gamma, skew-normal, Student’s t, etc.)

14

• Normal

• Student’s t

• Log-normal

• Skew-normal

• Normal-exponential

• Skew-Student’s t

Symmetric and Asymmetric PriorsSymmetric and Asymmetric Priors

Location-scale, symmetric

No location scale but shape, symmetric

Location-scale, asymmetric

Location-scale-shape, asymmetric

Location-scale-shape, asymmetric

1

2exp

x 2

2 2

v 1 /2 v v /2

1 x 2 /v v1

2

1

x 2exp

ln x 2

2 2

2

x

x

2

v 1 /2

v v /2 1

1

v

x

2

v1

2

T1,v1

v x

2

v 1

0.5

x

exp 2

2

exp x

x

Location-scale-shape, asymmetric

15

Shape Parameter ElicitationShape Parameter Elicitation

This is most challenging.This is most challenging. Presumably, the Interquantile-Range-ratio (IQRR= [q(.75)-q(.5)]/[q(.

5)-q(.25)] is a monotone function of shape parameter.

We have two cases: (1) Shape-parameter is in the non-sensitive region, absolute value

larger than 1. (2) Shape-parameter is in the sensitive region, absolute value sma

ller than 1.

16

Nonsensitive and sensitive regions (Skew-normal)Nonsensitive and sensitive regions (Skew-normal)

IQRR (interquantile range ratio) vs. shape parameter

Non-sensitive

Sensitive

17

Shape Parameter Sensitive Region: Gamma Case

18

Parameter Elicitation Guideline:Parameter Elicitation Guideline:

We prefer a moderate sensitivity index (SI):

Hyperparameter change / elicitation input change

SI=∂ (IQRR)/∂ (l)

We look for SI close to 1.

Sensitive region: shape parameter is small in magnitude.

The elicitation input is IQRR and the hyperparameter is the shape parameter.

19

Parameter Elicitation on Shape Parameter Elicitation on Shape Parameter Non-Sensitive RegionParameter Non-Sensitive Region(1) Elicit shape parameter from plot of

IQRR() vs.

(2) Scale parameter

= IQR/IQR()

where, IQR is the interquantile range from expert, IQR() is the standardized

IQR with elicited from (1), =1 and µ=0.

(3) The location parameter is

Q(0.75)- Q(0.75,)

where, Q(0.75) is .75 quantile from expert, comes from (2), and Q(0.75,) is

the standardized .75 quantile with elicited from (1), =1 and µ=0.

20

The sensitivity index in “IQR() vs. ” and “Q(0.75,) vs. ” is usually moderate.

NoteNote::

21

Approximate Scale Parameter Approximate Scale Parameter Elicitation from Taylor’s Expansion (1: Elicitation from Taylor’s Expansion (1: Basics) Basics)

[1] g(*) is the characteristic point of density f(x|µ,,), say mean, median,

mode, etc.

[2] g(*) = µ+g(), where g() is the standardized characteristic point.

[3] f(g(*)|µ,,) = (1/)f(g()|0,1,).

General approach for any location, scale and shapeFamily:

22

Approximate Scale Parameter Approximate Scale Parameter Elicitation from Taylor’s Expansion (2: Elicitation from Taylor’s Expansion (2: Method) Method)

Letting (1)-(2) and only keeping first 2 terms on the right hand side, we get

We get the approximate scale parameter without considering any consequences as

1

f,0,1(g())IQR

2IQR f,0,1(g())

23

Relative Error in Student’s t Prior Relative Error in Student’s t Prior Elicitation Elicitation (1: Values)(1: Values)From Taylor’s expansion, we have approximate

ˆ IQR

p

12

2

The exact

IQR

T0.75, T0.25,

Where,

[1] v is degrees of freedom

[2] IQR is interquantile range from expert

[3] p = 0.5

[4] is .75 quantile of Student’s t distribution with v degrees of freedom

T0.75,v

24

Approximate Scale Parameter Approximate Scale Parameter Elicitation from Taylor’s Expansion (3: Elicitation from Taylor’s Expansion (3: Relative Error) Relative Error)

Now

(1)-(2)

Denote

(Only related to )

The relative error is

21 2

25

Relative Error in Student’s t Prior Relative Error in Student’s t Prior Elicitation Elicitation (2: Plot)(2: Plot)

(1) ``approximate” represents Taylor expansion value:

ˆ IQR

p

1

2

2

(2) ``exact” represents Taylor expansion value:

IQR

T0.75, T0.25,

(3) ``normal” represents , with as interquantile range for standardized normal distribution.

IQR

Z p

Z p

Z p

(1) : (2) approaches 1.0763 as v goes to infinity.

26

An Important ObservationAn Important ObservationWhen shape parameter is highly sensitive to IQRR, the approximate scale parameter elicitation by Taylor’s expansion will be very stable in terms of relative error.

27

Elicitation of Shape Parameter on Sensitive Region

(Skew-normal, Iteration on characteristic points)Iteration based on Taylor’s expansion at median , mode or mean .

(1) Start with current l, from high-proportional- fidelity by Taylor expansion, we have

IQR[2(q0.50,)(q0.50,)]/(p2

Zq )

(2) The skew(shape) parameter can be obtained by plotting

q0.75, q0.50, ~

(3) Go to (1) until convergence (complete and )

(4) Location parameter

q0.25 q0.25,

q0.50,

E 2

1 2

M

28

Elicitation on Shape Parameter Sensitive Region

(Skew-normal, Iteration on IQRs)Iteration based on IQRs

(1) Start with current , we look up , then

(2) The skew (shape) parameter can be obtained by plot

q0.75, q0.50, ~

(3) Go to (1) until convergence (complete and )

(4) Location parameter

q0.25 q0.25,

q0.75, q0.25, ~

q0.75 q0.25

q0.75, q0.25,

Since

q0.75, q0.50, q0.75 q0.50

29

Graphical Comparison 1 (reference: IQR based

iteration)

30

Graphical Comparison 2 (reference: median based

iteration)

31

Graphical Comparison 3 (reference: mean based

iteration)

32

Graphical Comparison 4 (reference: mode based

iteration)

33

Another Important Observation

The IQR based iteration is close to mean based iteration for

skew-normal case, since mean is explicit , other than numerically solved.

E 2

1 2

34

35

36

Non-Parametric Prior Elicitation

To estimate prior density directly such that• ,

),(f

0)( f 1)(

df

Suppose, ,)|(|)( pugfE

wherep = parametric family of distributions,

= vector of hyper parametersu = underlying parameters in p

37

Non-Parametric Prior Elicitation

),,()|()|(|)(),( 2 cugugffCov

),( c =(correlation function) = 1 if

decreasing function of || ),( c ensures that prior variance covariance matrix

of any set of observation

otherwise.

)(f or functional of

)(f is positive semi-definite.

38

Choice of Covariance function

2

2

1exp),(

bc

:2:b

specifies the true density function.

controls smoothness of the density.

b large implies )(),( ffCorr is large.

),,( 2 bu

39

Hierarchical prior (Gaussian Process Prior)

),()|( mNug

bbb

c

*2

*,

2

1exp),(

),,,( *2 bm

Special Case :

then

Then

Prior: )(),,,( *21*2 bpbmp

40

Let D = elicited summaries relating to )(f

)())(,(,)(,)( 22 tfDCovADVHDE

&m

= {data}

• H is a function of

• A and )(t is a function of *&, bm

41

This implies,

MVNbmDf ~),,,,|( 2

),()()(|)( 1 HDAtgfE T

)()(),()|()|(|)(),( 12 tAtcugugffCov T

with

),,,( *2 bm

42

Posterior

)()(2

1exp

log2

1exp

1||)|,,,(

12

2**

)2(211*2

HDAHD

bb

ADbmp

T

n

n = # of elements in Duse MCMC to obtain samples from *,,,|)( bmDf

43

Other Choices of Centering

),(~)|( dmtg

SMNg ~)|(

),,(~)|( dmtSkewg

a)

b)

c)

~)|( gd) Gamma or Log-normal etc.

44

Side Conditions

Given Derivatives or quantiles D will be appropriately changed. In fact D can incorporate all the constraints specified in the prior, e.g., moments.

45

Psychological Perspective of Imprecise Subjective Probabilities

Numerical probabilty estimates (N) Ranges of numerical values (R) Verbal phrases (V) Objective: Translate the triplate (N,R,V) to a decision

maker’s model

46

Imprecisely Assessed Distributions

Qqwqwgwg ),|(.)|()1()|(*

)|(* wg

Contamination:

Class of Bi-modal distribution

),0(~,)()( 2* NAPAP

47

Future problems

Prior elicitation in Extreme Value Modeling Quantile and graphical approaches for GEV

model, Coles and Powel(1996) Prior elicitation for short and long tailed

distribution Spatial modeling High dimensional modeling

48

References 1. Daneshkhah, A. (2004). Psychological Aspects Influencing Elicitation of

Subjective Probability. BEEP working paper.

2. Dey, D.K. and Liu, J. (2007). A quantitative study of quantile based direct prior elicitation from expert opinion. Bayesian Analysis, 2, 137-166.

3. Garthwaite, P. H., Kadane, J. B., and O'Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100, 680-701.

4. Jenkinson, D. (2005). The Elicitation of Probabilities-A Review of the Statistical Literature. BEEP working paper.

5. Kadane, J.B.,Dickey,J.M., Winkler, R.L., Smith, W.S. and Peters, S.C.(1980). Interactive elicitation of opinion for a normal linear model. JASA, 75, 845-854.

49

6. Oakley, J., and O'Hagan, A. (2005). Uncertainty in prior elicitations: a non-parametric approach. Revised version of research report No. 521/02 Department of Probability and Statistics, University of Sheffield.

7. O'Hagan, A. (2005). Research in elicitation. Research Report No.557/05, Department of Probability and Statistics, University of Sheffield. Invited article for a volume entitled Bayesian Statistics and its Applications.

8. O' Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006). Uncertain Judgements: Eliciting Expert Probabilities. This book Will be published by John Wiley and Sons in July 2006.

50

THANK YOU