Jan KamenickýMariánská 2008

2

We deal with medical images◦ Different viewpoints - multiview◦ Different times - multitemporal◦ Different sensors – multimodal

Area-based methods (no features)

Transformation model Cost function minimization

3

Transformation model◦ Displacement field u(x)

4

)()( xuxxT ( ) ( )R SI x I T x

Transformation model◦ Displacement field u(x)

Cost function◦ Similarity measure (external forces)◦ Smoothing (penalization) term (internal forces)◦ Additional constraints (landmarks, volume

preservation)

5

)()( xuxxT ( ) ( )R SI x I T x

( ; , ) ( ; , ) ( ) ( )softR S R SC T I I S T I I P T C T

Transformation model◦ Displacement field u(x)

Cost function◦ Similarity measure (external forces)◦ Smoothing (penalization) term (internal forces)◦ Additional constraints (landmarks, volume

preservation)

Minimization

6

ˆ ˆarg min ( ; , ) arg min ( ; , )R S R ST

T C T I I C I I

)()( xuxxT ( ) ( )R SI x I T x

( ; , ) ( ; , ) ( ) ( )softR S R SC T I I S T I I P T C T

Translation Rigid (Euler)

◦ Translation, rotation Similarity

◦ Translation, rotation, scaling Affine B-splines

◦ Control points - regular grid on reference image

8

3( ) ( )k x

k kx N

T x x p x x

9

Sum of Squared Differences Normalized Correlation Coefficients Mutual Information Normalized Gradient Field

10

Sum of Squared Differences (SSD)◦ Equal intensity distribution (same modality)

Normalized Correlation Coefficients Mutual Information Normalized Gradient Field

11

21( ; , ) ( ) ( )

i R

R S R i S ixR

SSD I I I x I T x

Sum of Squared Differences Normalized Correlation Coefficients (NCC)

◦ Linear relation between intensity values (but still same modality)

Mutual Information Normalized Gradient Field

12

2 2

( ) ( )

( ; , )( ) ( )

i R

i R i R

R i R S i Sx

R S

R i R S i Sx x

I x I I T x I

NCC I II x I I T x I

Sum of Squared Differences Normalized Correlation Coefficients Mutual Information

◦ Any statistical dependence

Normalized Gradient Field

13

2

( , ; )( ; , ) ( , ; ) log

( ) ( ; )S R

R Ss L r L R S

p r sMI I I p r s

p r p s

Mutual Information (MI)◦ From entropy

14

( , ) ( ) ( | )

( ) ( | )

( ) ( ) ( , )

MI X Y H X H X Y

H Y H Y X

H X H Y H X Y

2( ) ( ) log ( ), ( ) 1 x X x X

H X p x p x p x

( , )2 ( ) ( )( , ) ( , ) log

X Y

p x yp x p y

y Y x X

MI X Y p x y

Mutual Information (MI)◦ From Kullback-Leibler distance

15

( )( )( , ) ( ) log p iq i

i

KL p q p i

( , )2 ( ) ( )( , ) ( , ) log

X Y

p x yp x p y

y Y x X

MI X Y p x y

Mutual Information (MI)◦ For images

p(x) … normalized image histogram

◦ Normalized Mutual Information (NMI)

16

( , ; )2 ( ) ( ; )( ; , ) ( , ; ) log

R S

S R

p r sR S p r p s

s L r L

MI I I p r s

2 2

2

( ) log ( ) ( ; ) log ( ; )

( ; , )( , ; ) log ( , ; )

R S

S R

R R S Sr L s L

R S

s L r L

p r p r p s p m

NMI I Ip r s p r s

( ) ( )

( , )

H X H Y

H X Y

Mutual Information (MI)◦ Joint probability estimation

Using B-spline Parzen windows

and are defined by the histogram bins widths

17

( )( )1( , ; )

i R

S iR iR S

xR R S

s I T xr I xp r s w w

R S

Sum of Squared Differences Normalized Correlation Coefficients Mutual Information Normalized Gradient Field (NGF)

◦ Based on edges

18

2 2

2( , ) ( )en I x I I x e

2( ; , ) ( , ) ( ( ), )

i R

TR S e R i e S i

x

NGF I I n I x n I T x

Elastic◦ Elastic potential (motivated by material

properties)

Fluid◦ Viscous fluid model (based on Navier-Stokes)

Diffusion◦ Much faster

19

2 2

4 2,

[ ] divj k

elasx k x j

j k

P u u u u dx

212[ ]diff

ll

P u u dx

Curvature◦ Doesn’t penalize affine transformation

Bending energy (Thin plate splines)

20

2 2

, ,

[ ] ( )p

q r

u

x xp q r

P u x dx

212

1

[ ]d

curvl

l

P u u dx

21

curvature

diffusion

elastic

fluid

Landmarks (fiducial markers)◦ “Hard” constraint

◦ “Soft” constraint

Volume preservation

22

( ) , 1, 2,...,j j j jC r u r t j m

2

1

msoftjj

C C

log det ( )R

softC u x dx

Full Grid

◦ Used with multi-resolution Random

◦ Random subset of voxels is selected◦ Improved speed

23

25

1 ,

0,1,2,...k k k ka d

k

Gradient Descent (GD)◦ Linear rate of convergence

Quasi-Newton Nonlinear Conjugate Gradient Stochastic Gradient Descent Evolution Strategy

26

1 ( )k k k ka g

Gradient Descent Quasi-Newton (QN)

◦ Can be superlinearly convergent

Nonlinear Conjugate Gradient Stochastic Gradient Descent Evolution Strategy

27

11 ( ) ( )k k k kH g

Gradient Descent Quasi-Newton Nonlinear Conjugate Gradient (NCG)

◦ Superlinear rate of convergence can be achieved

Stochastic Gradient Descent Evolution Strategy

28

1( )k k k kd g d

Gradient Descent Quasi-Newton Nonlinear Conjugate Gradient Stochastic Gradient Descent (SGD)

◦ Similar to GD, but uses approximation of the gradient (Kiefer-Wolfowitz, Simultaneous Perturbation, Robbins-Monro)

Evolution Strategy

29

Gradient Descent Quasi-Newton Nonlinear Conjugate Gradient Stochastic Gradient Descent Evolution Strategy (ES)

◦ Covariance matrix adaptation◦ Tries several possible directions (randomly

according to the covariance matrix of the cost function), the best are chosen and their weighted average is used

30

Data complexity◦ Gaussian pyramid◦ Laplacian pyramid◦ Wavelet pyramid

Transformation complexity◦ Transformation superposition◦ Different B-spline grid density

31

Registration toolkit based on ITK Handles many methods

◦ Similarity measures (SSD, NCC, MI, NMI)◦ Transformations (rigid, affine, B-splines)◦ Optimizers (GD, SGD-RM)◦ Samplers, Interpolators, Multi-resolution, …

http://elastix.isi.uu.nl

32