Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group...

transcript

Iterative Image Registration:

Lucas & Kanade Revisited

Kentaro Toyama

Vision Technology Group

Microsoft Research

Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future.

Jorge Luis Borges

History

• Lucas & Kanade (IUW 1981)

LK BAHH ST S BJ HB BL G SI CETSC

• Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

• Shi & Tomasi (CVPR 1994)

• Szeliski & Coughlan (CVPR 1994)

• Szeliski (WACV 1994)

• Black & Jepson (ECCV 1996)

• Hager & Belhumeur (CVPR 1996)

• Bainbridge-Smith & Lane (IVC 1997)

• Gleicher (CVPR 1997)

• Sclaroff & Isidoro (ICCV 1998)

• Cootes, Edwards, & Taylor (ECCV 1998)

Image Registration

Applications

• Stereo

Applications

• Stereo

• Dense optic flow

Applications

• Stereo

• Image mosaics

Applications

• Stereo

• Image mosaics

• Tracking

Applications

• Stereo

• Image mosaics

• Tracking

• Recognition

Lucas & Kanade

Derivation

L&K Derivation 1

)('0 xI

xIhxIh

)()(lim 00

)('0 xI

L&K Derivation 1

)('0 xI

xIhxI )()( 00

h I0(x)

I0(x+h)

L&K Derivation 1

h I0(x)

)('0 xI

xIxI )()( 0

L&K Derivation 1

h I0(x)

)()('0

L&K Derivation 1

xIxIxw

)]()()[(

L&K Derivation 1

h0 I0(x)

xIxIxw

)]()()[(

L&K Derivation 1

hxIxIxw

)]()()[(

I0(x+h0)

L&K Derivation 1

hxIxIxw

)]()()[(

I0(x+h1)

L&K Derivation 1

hxIxIxw

)]()()[(

I0(x+hk)

L&K Derivation 1

hxIxIxw

)]()()[(

I0(x+hf)

Lucas & KanadeDerivation

L&K Derivation 2

• Sum-of-squared-difference (SSD) error

E(h) = [ I(x) - I0(x+h) ]2x R

E(h) [ I(x) - I0(x) - hI0’(x) ]2x R

L&K Derivation 2

2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x Rh

I0’(x)(I(x) - I0(x))x R h I0’(x)2

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

w(x)[I(x) - I0(x)]

x I0’(x)

Comparison

I0’(x)[I(x) - I0(x)] h I0’(x)2

w(x)[I(x) - I0(x)]

x I0’(x)

Generalizations

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Dimension of image

h ) = x R

(E [I( x ) - (x ]2)+ h

1-dimensional

Generalization 1a

h ) = x R

(E [I( x ) - (x ]2)+ h

Generalization 1b

h ) = x R

(E [I( x ) - (x ]2)+ h

xHomogeneous 2D:

Problem A

Does the iteration converge?

Problem A

Local minima:

Problem A

Local minima:

Problem B

- I0’(x)(I(x) - I0(x))x R h I0’(x)2

h is undefined if I0’(x)2 is zerox R

Zero gradient:

Problem B

Zero gradient:

Problem B’

- (x)(I(x) - I0(x))x R

I )(0 xy

Aperture problem:

Problem B’

No gradient along one direction:

Solutions to A & B

• Possible solutions:– Manual intervention

• Possible solutions:– Manual intervention– Zero motion default

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation

Solutions to A & B

• Possible solutions:– Manual intervention– Zero motion default– Coefficient “dampening”– Reliance on good features– Temporal filtering– Spatial interpolation / hierarchical estimation– Higher-order terms

Solutions to A & B

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Transformations/warping of image

h ) = x R

(E [I( x ) -I(x ]2)+ h

Translations:

Problem C

What about other types of motion?

Generalization 2a

A, h) = x R

(E [I(Ax ) - (x ]2)+h

Affine:

Generalization 2a

Affine:

Generalization 2b

A ) = x R

(E [I( A x ) - (x ]2)

Planar perspective:

Generalization 2b

Planar perspective:

Affine +

Generalization 2c

h ) = x R

(E [I( f(x, h) ) - (x ]2)

Other parametrized transformations

Generalization 2c

Other parametrized transformations

Problem B”

-(JTJ)-1 J (I(f(x,h)) - I0(x)) h ~

Generalized aperture problem:

- I0’(x)(I(x) - I0(x))x R h I0’(x)2

Problem B”

Generalizedaperture problem:

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image type

h ) = x R

(E [I( x ) - (x ]2)+ h

Grayscale images

Generalization 3

• Image type

h ) = x R

(E ||I( x ) -I(x ||2)+ h

Color images

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Constancy assumption

h ) = x R

(E [I( x ) -I(x ]2)+ h

Brightness constancy

Problem C

What if illumination changes?

Generalization 4a

h, )=x R

(E [I( x ) - I(x ]2)++ h

Linear brightness constancy

Generalization 4a

Generalization 4b

h,) = x R

(E [I( x ) - B(x]2)+ h

Illumination subspace constancy

Problem C’

What if the texture changes?

Generalization 4c

h,) = x R

(E [I( x ) - ]2+ h

Texture subspace constancy

Problem D

Convergence is slower as #parameters increases.

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc.

Solutions to D

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization

Solutions to D

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization – Offline precomputation

Solutions to D

• Difference decomposition

LK BAHH ST S BJ HB G SI CETSC

Solutions to D

– Improvements in gradient descent

Solutions to D

• Faster convergence:– Coarse-to-fine, filtering, interpolation, etc. – Selective parametrization– Offline precomputation

– Improvements in gradient descent• Multiple estimates of spatial derivatives

Solutions to D

• Multiple estimates / state-space sampling

Generalizations

[I( x ) - (x ]2)+ h I

Modifications made so far:

Original

• Error norm

h ) = x R

(E [I( x ) -I(x ]2)+ h

Squared difference:

Problem E

What about outliers?

Generalization 5a

• Error norm

h ) = x R

(E (I( x ) -I(x ))+ h

Robust error norm:

Original

h ) = x R

(E [I( x ) - (x ]2)+ h I

Original

• Image region / pixel weighting

h ) = x R

(E [I( x ) -I(x ]2)+ h

Rectangular:

Problem E’

What about background clutter?

Generalization 6a

h ) = x R

(E [I( x ) -I(x ]2)+ h

Irregular:

Problem E”

What about foreground occlusion?

Generalization 6b

h ) = x R

(E [I( x ) -I(x ]2)+ h

Weighted sum:

Generalizations

[I( x ) - (x ]2)+ h I

Modifications made so far:

Generalizations: Summary

(I( ) - w(x) (x ))h )(E f(x, h)

h ) = x R

(E [I( x ) - (x ]2)+ h I

Foresight

Summary

• Generalizations– Dimension of image– Image transformations / motion models– Pixel type– Constancy assumption– Error norm– Image mask

L&K ?Y

Summary

• Common problems:– Local minima– Aperture effect– Illumination changes– Convergence issues– Outliers and occlusions

L&K ?Y

• Mitigation of aperture effect:– Manual intervention– Zero motion default– Coefficient “dampening”– Elimination of poor textures– Temporal filtering– Spatial interpolation / hierarchical – Higher-order terms

Summary

L&K ?n

Summary

• Better convergence:– Coarse-to-fine, filtering, etc.– Selective parametrization – Offline precomputation

– Improvements in gradient descent• Multiple estimates of spatial derivatives

L&K ?Y

nmaybe

Hindsight

Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group...

Documents