MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 1

Roland Preuss and Udo v. Toussaint

Comparison of

Numerical Methods for

Evidence Calculation

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 2

• Evidence computation crucial for model comparison

• Evidence is normalization constant in posterior computation

High-dimensional integral over parameter space is required

How?

Evidence Computation

IMDp

IMpIMDpIMDp

,|

,|,,|,,|

IMpIMDpIDMp |,|,|

IMpIMDpdIMDp ,|,,|,|

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 3

Variety of techniques:

-Analytical methods:

- Laplace approximation (saddle-point approximation)

- Variation principle

-Deterministic methods

- Numerical quadrature

-Probabilistic methods

Evidence Computation

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 4

Evidence Computation: Test cases 1&2

|xpxdITest cases:

2exp

2

1|

xx

xpT

N

03.0

3.0

2

1

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 5

Evidence Computation: Test cases 3&4

222

211 expexp

2

1|

dxdxdxdx

xpTT

N

03.0

3.0

2

1

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 6

Evidence Computation: Test cases 5&6

222

211

2expexp10

2210

1|

dxdxdxdx

xp

TT

NN

03.0

3.0

2

1

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 7

Evidence Computation: Test cases 5&6

Methods:

1. Laplace Approximation

2. Quadrature: Trapezoidal Rule

3. Importance Sampling

4. VEGAS Algorithm

5. Thermodynamic Integration

6. Nested Sampling

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 8

Evidence Computation: Laplace approximation

Laplace Approximation: |2

1exp| *

00 xpxxHxxAxp T

H

xpxdxpxdN

det

2|| *

Only for unimodal distributions!

Pay attention to the integration region (eg. >0): Improvements eg by von der Linden, McKay,…

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 9

Evidence Computation: Trapezoidal rule

Quadrature eg by trapezoidal rule:

Curse of dimension: Number of support points per dimension: N(N=15,150)

Number of dimensions: K

Total number of support points:

KN Dimension 1 2 4 8

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 10

Evidence Computation: Importance Sampling

Importance Sampling: xgxg

xpxdxpxdI

Samples are generated from simpler function g(x) easy to sample from. Usually a gaussian with widths from variances from previous MCMC runs (Metropolis algorithm).

Dimension 1 2 4 8 16

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 11

Evidence Computation: Vegas

VEGAS algorithm:

(Peter Lepage)

Samples are generated from a separable function g(x)

Dimension 1 2 4 8 16

• Algorithm freely available from Numerical Recipes

• Appropriate coordinate system important

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 12

Evidence Computation: Thermodynamic IntegrationThermodynamic Integration: Slowly introduce likelihood structure into prior:

(Stat. Physics, Neal 1993, Skilling 97)

Z(0)=1 and Z(1)=Evidence

xxxdZ

1

0

1

0

lnln

ln

xxxdd

ZdI

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 13

Evidence Computation: Thermodynamic IntegrationThermodynamic Integration: Slowly introduce likelihood structure into prior:

xxxdZ

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 14

Evidence Computation: Nested Sampling

Nested Sampling: Sample from prior within likelihood constraint:

Evidence=

Probabilistic measure of the >active< prior volume

kk k

kk Lz

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 15

Evidence Computation: Nested Sampling

Nested Sampling:

Some recent developements: arXiv.org/abs/0704.3704 (Farhan Feroz)

arXiv.org/astro-ph/0701867 (R. Shaw)

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 16

Evidence Computation: Conclusion

Conclusion: - None of the methods is a foolproof black box

Use more than one method!

Outlook: - More methods are available, eg. Perfect Tempering

- Still lack of extensive and systematic studies about properties

- Multimodal (nongaussian) analytical tracktable examples needed

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 17

Evidence Computation: Conclusion

Conclusion: - None of the methods is a foolproof black box

Use more than one method!

Outlook: - More methods are available, eg. Perfect Tempering

- Still lack of extensive and systematic studies about properties

- Multimodal (nongaussian) analytical tracktable examples needed

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 18

Fusion Research

Example: W7-X

Thomson scattering

ECE

Magnetics

Langmuir Probes

Interferometry

Bolometry

Video

H

Calorimetry

Neutral gas

Vis. spectroscopy

CXRS

SPRED

Soft-X ray

Thermography

Flux surface measurements

Diagnostic beam

Neutron detectors

Diagnostics Software(data analysis)

General purpose (stellarator) software

Integrated data analysis

Modelling

Infrastructure &management

Spectroscopy

Imaging

Tomography

DedicatedDiagnostics(misc.)

W7-X DiagnosticsSoftware

2-D diagnostics essential!

BES

Zeff

CO-Monitor

(by A. Dinklage)

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 19

Fusion Research: Tomography

Wide variety of tomographic configurations: experimental requirements, costs,…

Goal: best possible reconstruction of 2-D profile

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 20

Fusion Research: Tomography

•Tomographic reconstruction: underdetermined & ill-posed

• Prior information available

standard inversion techniques

mostly perform poorly

??

?

??

? ??

?

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 21

Thin-Plate Splines

-Transport along magnetic field lines very fast: smooth profiles usually favored:

curvature only when enforced by the data

- 1D- case: cubic splines

- 2D- case: Minimize curvature IB of f(x,y) (can be generalized )

222 2 yyxyxxB fffdydxI

Fundamental solution[1]:

22 ln rrrU

[1] G. Wahba, Spline models for Observational Data, SIAM, 1990

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 22

Thin-Plate Splines

Interpolating function z=f(x,y):

3,||,,||, 01

nyaxaayxyxUwyxfz yxii

n

ii

Coefficients wi, a are given by the solution of

00

z

a

w

P

PKT

with Kij=U(||(xi,yi)-(xj,yj)||), Pi=(1,xi,yi)

The bending energy is given by zrwwKwwI TB ,,

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 23

Model

Model: jj

iji gCm , Cij: response matrix

gj : emissivity on grid space

Likelihood: 2

2

2,,|

i

iiii

mdIgdp

with ),( yxfdxdygjA

j

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 24

Prior Distributions

• Number of support points n on grid with N places:

)|(,|),,|()|( InpInRpInRzpIMp

• Curvature : Testable information

)),(),(exp(1

,,| zRwKzRwZ

nRzp T

• Hyperparameter :

MaxMinIp ,/1)|(

• Model prior factorizes:

!)3(

!)!()|,(

NN

nnNInRp

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 25

IpIRzpdIRzDpzdIMp

IRzpIRzDpzdIMpIMDpIMpIDMp

|,,|,,||

,|,,||,||,|

Posterior

• Posterior for Model Mi :

• Evidence approximation:

• Simple sampling in N,R

• 1-D search in : *

2/1**** )det(,|,,,,|,| HIMzpIMzDpIMDp

)det(

)2(,,|,,,||

,,|,,||,,|

0

**

HIMzpIMzDpIp

IMzpIMDpzdIpIMDp

n

• Optimization preferred: Laplace Approximations…

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 26

Thin-Plate Splines: Results

• Reconstruction of challenging (but realistic) 2-d emission profile:

• Result of evidence weighted average

Mock profile: Reconstruction: Difference:

Key features, shape and absolut intensity recovered

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 27

Thin-Plate Splines: Results

Location of z-optimized support points:

Open issue: Support points outside of region of interest beneficial?

Reconstruction within the uncertainty

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 28

Thin-Plate Splines: Outlook

• Extension to f(x,y,z): Viewing cone

• Extension to f(x,y,(z),t):

time correlated data

Connection to morphing applications

Drawback: Huge amounts of data to be processed

• Online (monitoring) requirements:Use as input for Bayesian Neural Networks

• Check approximations with MCMC

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 29

Thin-Plate Splines: Conclusion

Thank you!

MaxEnt 2007, Udo v. Toussaint, July 2007, udo.v.toussaint@ipp.mpg.de 30

W7-X: Design

• 50+36SL-Coils

• Optimized shape

• stability without current-drive

• Size:

major: 5.5m

minor: 0.53m

• Magnetic field:

3.3T (2.5T reg)

12kA, 38MW/m2

• Heating:

10 MW ECR

4 MW ICR

5 MW NBI