Optical Flow

10-24-2005

Problem

• Problems in motion estimation– Noise, – color (intensity) smoothness, – lighting (shadowing effects), – occlusion, – abrupt movements, etc

• Approaches:– Block matching,– Generalized block matching,– Optical flow (block-based, Horn-Schunck etc)– Bayesian, etc.

• Applications– Video coding and compression,– Segmentation– Object reconstruction (structure-from-motion)– Detection and tracking, etc.

Motion description• 2D motion:

p = [x(t),y(t)] p’= [x(t+ t0), y(t+t0)]

d(t) = [x(t+ t0)-x(t),y(t+t0)-y(t)]

• 3D motion:

Α = [ Χ1, Υ1, Ζ1 ]Τ Β = [ Χ2, Υ2, Ζ2 ]

Τ

= R + T

• Basic projection models: Orthographic

Perspective

x x(t) d1(t)

y y(t) d2(t)

X

Y

Z

2

2

2

X

Y

Z

1

1

1

YyXx

Z

Yy

Z

Xx

l

ll

l

Optical Flow

• Basic assumptions: – Image is smooth locally– Pixel intensity does not change over time (no lighting changes)

• Normal flow:

• Second order differential equation:

Block-based Optical Flow Estimation

• Optical flow estimation within a block (smoothness assumption): all pixels of the block have the same motion

• Error:

• Motion equation:

and

Horn-Schunck

• We want an optical flow field that satisfies the Optical Flow Equation with the minimum variance between the vectors (smoothness)

Gauss-Seidel

Derivative Estimation with Finite differences

Example 1

Example 2

Example 3: frame reconstruction

Reconstructed I2 (second) frame

Reconstructed I2 (second) frame

Application Examples