Iterative Image Registration:

Lucas & Kanade Revisited

Kentaro Toyama

Vision Technology Group

Microsoft Research

Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future.

Jorge Luis Borges

History

• Lucas & Kanade (IUW 1981)

LK BAHH ST S BJ HB BL G SI CETSC

• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

• Shi & Tomasi (CVPR 1994)

• Szeliski & Coughlan (CVPR 1994)

• Szeliski (WACV 1994)

• Black & Jepson (ECCV 1996)

• Hager & Belhumeur (CVPR 1996)

• Bainbridge-Smith & Lane (IVC 1997)

• Gleicher (CVPR 1997)

• Sclaroff & Isidoro (ICCV 1998)

• Cootes, Edwards, & Taylor (ECCV 1998)

Image Registration

Applications

Applications

• Stereo

LK BAHH ST S BJ HB BL G SI CETSC

Applications

• Stereo

• Dense optic flow

LK BAHH ST S BJ HB BL G SI CETSC

Applications

• Stereo

• Dense optic flow

• Image mosaics

LK BAHH ST S BJ HB BL G SI CETSC

Applications

• Stereo

• Dense optic flow

• Image mosaics

• Tracking

LK BAHH ST S BJ HB BL G SI CETSC

Applications

• Stereo

• Dense optic flow

• Image mosaics

• Tracking

• Recognition

LK BAHH ST S BJ HB BL G SI CETSC

?

Lucas & Kanade

#1

Derivation

L&K Derivation 1

I0(x)

)('0 xI

h

xIhxIh

)()(lim 00

0

)('0 xI

L&K Derivation 1

)('0 xI

h

xIhxI )()( 00

h I0(x)

I0(x+h)

L&K Derivation 1

h I0(x)

)('0 xI

h

xIxI )()( 0

I(x)

L&K Derivation 1

h I0(x)

h)(

)()('0

0

xI

xIxI

I(x)

L&K Derivation 1

I0(x)

h

Rx xI

xIxI

R )(

)()(

||

1'0

0

RI(x)

L&K Derivation 1

I0(x)

h

RxxxI

xIxIxw

xw )(

)]()()[(

)(

1'0

0

I(x)

L&K Derivation 1

h0 I0(x)

0h

I(x)

RxxxI

xIxIxw

xw )(

)]()()[(

)(

1'0

0

L&K Derivation 1

1h

Rxx

hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1

0'0

000

I0(x+h0)

I(x)

L&K Derivation 1

2h

Rxx

hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1

1'0

101

I0(x+h1)

I(x)

L&K Derivation 1

1kh

Rx k

k

x

k hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1'0

0

I0(x+hk)

I(x)

L&K Derivation 1

1kh

Rx k

k

x

k hxI

hxIxIxw

xwh

)(

)]()()[(

)(

1'0

0

I0(x+hf)

I(x)

Lucas & KanadeDerivation

#2

L&K Derivation 2

• Sum-of-squared-difference (SSD) error

E(h) = [ I(x) - I0(x+h) ]2x R

E(h) [ I(x) - I0(x) - hI0’(x) ]2x R

L&K Derivation 2

2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x Rh

E

I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

= 0

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

x

x

h

w(x)[I(x) - I0(x)]

w(x)x

x I0’(x)

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

x

h

x

w(x)[I(x) - I0(x)]

w(x)x

x I0’(x)

Generalizations

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Dimension of image

h ) = x R

(E [I( x ) - (x ]2)+ h

1-dimensional

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 1a

• Dimension of image

h ) = x R

(E [I( x ) - (x ]2)+ h

y

xx2D:

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 1b

• Dimension of image

h ) = x R

(E [I( x ) - (x ]2)+ h

1

y

x

xHomogeneous 2D:

I

LK BAHH ST S BJ HB BL G SI CETSC

Problem A

LK BAHH ST S BJ HB BL G SI CETSC

Does the iteration converge?

Problem A

Local minima:

Problem A

Local minima:

Problem B

- I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

h is undefined if I0’(x)2 is zerox R

LK BAHH ST S BJ HB BL G SI CETSC

Zero gradient:

Problem B

Zero gradient:

?

Problem B’

- (x)(I(x) - I0(x))x R

hy 2

x R

y

I )(0 xy

I

)(0 x

Aperture problem:

LK BAHH ST S BJ HB BL G SI CETSC

Problem B’

No gradient along one direction:

?

Solutions to A & B

• Possible solutions:– Manual intervention

LK BAHH ST S BJ HB BL G SI CETSC

• Possible solutions:– Manual intervention– Zero motion default

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation– Higher-order terms

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to A & B

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Transformations/warping of image

h ) = x R

(E [I( x ) -I(x ]2)+ h

Translations:

y

x

h

LK BAHH ST S BJ HB BL G SI CETSC

Problem C

What about other types of motion?

Generalization 2a

• Transformations/warping of image

A, h) = x R

(E [I(Ax ) - (x ]2)+h

Affine:

dc

baA

y

x

h

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 2a

Affine:

dc

baA

y

x

h

Generalization 2b

• Transformations/warping of image

A ) = x R

(E [I( A x ) - (x ]2)

Planar perspective:

187

654

321

aa

aaa

aaa

A

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 2b

Planar perspective:

187

654

321

aa

aaa

aaa

A

Affine +

Generalization 2c

• Transformations/warping of image

h ) = x R

(E [I( f(x, h) ) - (x ]2)

Other parametrized transformations

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 2c

Other parametrized transformations

Problem B”

-(JTJ)-1 J (I(f(x,h)) - I0(x)) h ~

Generalized aperture problem:

LK BAHH ST S BJ HB BL G SI CETSC

- I0’(x)(I(x) - I0(x))x R h I0’(x)2

x R

Problem B”

?

Generalizedaperture problem:

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image type

h ) = x R

(E [I( x ) - (x ]2)+ h

Grayscale images

I

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 3

• Image type

h ) = x R

(E ||I( x ) -I(x ||2)+ h

Color images

LK BAHH ST S BJ HB BL G SI CETSC

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Constancy assumption

h ) = x R

(E [I( x ) -I(x ]2)+ h

Brightness constancy

LK BAHH ST S BJ HB BL G SI CETSC

Problem C

What if illumination changes?

Generalization 4a

• Constancy assumption

h, )=x R

(E [I( x ) - I(x ]2)++ h

Linear brightness constancy

LK BAHH ST S BJ HB BL G SI CETSC

Generalization 4a

Generalization 4b

• Constancy assumption

h,) = x R

(E [I( x ) - B(x]2)+ h

Illumination subspace constancy

LK BAHH ST S BJ HB BL G SI CETSC

Problem C’

What if the texture changes?

Generalization 4c

• Constancy assumption

h,) = x R

(E [I( x ) - ]2+ h

Texture subspace constancy

B(x)

LK BAHH ST S BJ HB BL G SI CETSC

Problem D

Convergence is slower as #parameters increases.

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc.

LK BAHH ST S BJ HB BL G SI CETSC

Solutions to D

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization

Solutions to D

LK BAHH ST S BJ HB BL G SI CETSC

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation

Solutions to D

LK BAHH ST S BJ HB BL G SI CETSC

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation

• Difference decomposition

LK BAHH ST S BJ HB G SI CETSC

Solutions to D

BL

Solutions to D

• Difference decomposition

Solutions to D

• Difference decomposition

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation

• Difference decomposition

– Improvements in gradient descent

LK BAHH ST S BJ HB G SI CETSC

Solutions to D

BL

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization– Offline precomputation

• Difference decomposition

– Improvements in gradient descent• Multiple estimates of spatial derivatives

LK BAHH ST S BJ HB G SI CETSC

Solutions to D

BL

Solutions to D

• Multiple estimates / state-space sampling

Generalizations

x R

[I( x ) - (x ]2)+ h I

Modifications made so far:

Original

• Error norm

h ) = x R

(E [I( x ) -I(x ]2)+ h

Squared difference:

LK BAHH ST S BJ HB BL G SI CETSC

Problem E

What about outliers?

Generalization 5a

• Error norm

h ) = x R

(E (I( x ) -I(x ))+ h

Robust error norm:

22

2

)(uk

uuρ

LK BAHH ST S BJ HB BL G SI CETSC

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image region / pixel weighting

h ) = x R

(E [I( x ) -I(x ]2)+ h

Rectangular:

LK BAHH ST S BJ HB BL G SI CETSC

Problem E’

What about background clutter?

Generalization 6a

• Image region / pixel weighting

h ) = x R

(E [I( x ) -I(x ]2)+ h

Irregular:

LK BAHH ST S BJ HB BL G SI CETSC

Problem E”

What about foreground occlusion?

Generalization 6b

• Image region / pixel weighting

h ) = x R

(E [I( x ) -I(x ]2)+ h

Weighted sum:

w(x)

LK BAHH ST S BJ HB BL G SI CETSC

Generalizations

x R

[I( x ) - (x ]2)+ h I

Modifications made so far:

Generalizations: Summary

= x R

(I( ) - w(x) (x ))h )(E f(x, h)

h ) = x R

(E [I( x ) - (x ]2)+ h I

Foresight

• Lucas & Kanade (IUW 1981)

• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

• Shi & Tomasi (CVPR 1994)

• Szeliski & Coughlan (CVPR 1994)

• Szeliski (WACV 1994)

• Black & Jepson (ECCV 1996)

• Hager & Belhumeur (CVPR 1996)

• Bainbridge-Smith & Lane (IVC 1997)

• Gleicher (CVPR 1997)

• Sclaroff & Isidoro (ICCV 1998)

• Cootes, Edwards, & Taylor (ECCV 1998)

LK BAHH ST S BJ HB BL G SI CETSC

Summary

• Generalizations– Dimension of image– Image transformations / motion models– Pixel type– Constancy assumption– Error norm– Image mask

L&K ?Y

Y

n

Y

n

Y

Summary

• Common problems:– Local minima– Aperture effect– Illumination changes– Convergence issues– Outliers and occlusions

L&K ?Y

maybe

Y

Y

n

• Mitigation of aperture effect:– Manual intervention– Zero motion default– Coefficient “dampening”– Elimination of poor textures– Temporal filtering– Spatial interpolation / hierarchical – Higher-order terms

Summary

L&K ?n

n

n

n

Y

Y

n

Summary

• Better convergence:– Coarse-to-fine, filtering, etc.– Selective parametrization – Offline precomputation

• Difference decomposition

– Improvements in gradient descent• Multiple estimates of spatial derivatives

L&K ?Y

nmaybe

maybe

maybe

maybe

Hindsight

• Lucas & Kanade (IUW 1981)

• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

• Shi & Tomasi (CVPR 1994)

• Szeliski & Coughlan (CVPR 1994)

• Szeliski (WACV 1994)

• Black & Jepson (ECCV 1996)

• Hager & Belhumeur (CVPR 1996)

• Bainbridge-Smith & Lane (IVC 1997)

• Gleicher (CVPR 1997)

• Sclaroff & Isidoro (ICCV 1998)

• Cootes, Edwards, & Taylor (ECCV 1998)