Optical Flow Methods

CISC 489/689Spring 2009

University of Delaware

Outline

• Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods

• Lucas-Kanade Meets Horn and Schunck• 3D Regularization• Techniques for solving optical flow• Confidence Measures in Optical Flow

Optical Flow Constraint

0

0

),,(),,(

),,(),,(

tyx fvfuftf

dtdy

yf

dtdx

xf

tyxftft

yfy

xfxtyxf

tyxftttvytuxf

0limit takingand by Dividing tt

),( tvytux

),( yx

),,( tyxf

),,( ttyxf

Interpretation

Constraint Line

tT

tyx

fvu

f

fvfuf

0

yx ff ,

v

u 0

1

vu

fff tyx

Lucas-Kanade Method

* = JK J

1*),( v

uJKvuELK

2

2

2

2

2

2

01

ttytx

tyyyx

txyxx

ttytx

tyyyx

txyxx

fffffffffffffff

J

vu

fffffffffffffff

Lucas-Kanade Method

• Local Method, window based• Cannot solve for optical flow everywhere• Robust to noise

5.7 15

Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

Dense optical Flow

?)(),( Minimize 2 tyx fvfufvuE

Lacks SmoothnessFigures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

Horn and Schunck MethoddxdyvufvfufvuE tyxHS )()(),( 222

Euler-Lagrange Equations

Horn and Schunck Method

510 610Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

• Global Method • Estimates flow everywhere• Sensitive to noise• Oversmooths the edges

Why combine them?

• Need dense flow estimate• Robust to noise• Preserve discontinuities

Combining the two…

1*),( v

uJKvuELK

dxdyvufvfufvuE tyxHS )()(),( 222

dxdyvuvu

JKvuECLG )(1

*),( 22

Combined Local Global Method

dxdyvuvu

JKvuECLG )(1

*),( 22

Euler-Lagrange Equations

AverageError

Standard Deviation

Lucas&Kanade 4.3(density 35%)

Horn&Schunk 9.8 16.2Combining local

and global 4.2 7.7

Table: Courtesy - Darya Frolova, Recent progress in optical flow

Preserving discontinuities• Gaussian Window does not preserve

discontinuities• Solutions

– Use bilateral filtering

– Add gradient constancy

dxdyvuvu

JKvuE bilbil )(1

*),( 22

dxdyffvuvu

JKvuE tgrad2

122 )()(

1*),(

Bilateral support window

Images: Courtesy, Darya Frolova, Recent progress in optical flow

Robust statistics – simple exampleFind “best” representative for the set of numbers

min2 i

ixxEL2: mini

ixxEL1:

xix

Influence of xi on E: xi → xi + ∆

)mean( ixx

Outliers influence the most

ixx proportional to

xxEE ioldnew 2

)median( ixx

Majority decides

equal for all xi

oldnew EE

Slide: Courtesy - Darya Frolova, Recent progress in optical flow

Robust statistics

many ordinary people a very rich man

Oligarchy

Votes proportional to the wealth

Democracy

One vote per person

wealth

like in L2 norm minimization like in L1 norm minimization

Slide: Courtesy - Darya Frolova, Recent progress in optical flow

Combination of two flow constraints

robust: L1

x

robust in presence of outliers– non-smooth: hard to analyze

usual: L2

easy to analyze and minimize– sensitive to outliers

2x

robust regularized

smooth: easy to analyze robust in presence of outliers

22 x

ε

video

warpedwarped IIII min

),,( ; )1,,( tyxIItvyuxIIwarped

[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performanceSlide: Courtesy - Darya Frolova, Recent progress in optical flow

Robust statistics

3D Regularization

• accounted for spatial regularization

• If velocities do not change suddenly with time, can we regularize in time as well?

)( 22 vu

3D Regularization

tv

yv

xvv

tu

yu

xuu

3

3

dxdydtvuvu

JKvuETXCLG )(

1*),( 2

32

3],0[3

Multiresolution estimation

21

image Iimage J

Gaussian pyramid of image 1 Gaussian pyramid of image 2

Image 2image 1

run iterative estimation

run iterative estimation

warp & upsample

.

.

.

Multi-resolution Lucas Kanade Algorithm

Compute Iterative LK at highest level•For Each Level i• Take flow u(i-1), v(i-1) from level i-1

• Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.

• Multiply u*(i), v*(i) by 2

• Compute It from a block displaced by u*(i), v*(i)

• Apply LK to get u’(i), v’(i) (the correction in flow)

• Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)

Comparison of errors

For Yosemite sequence with cloudsTable: Courtesy - Darya Frolova, Recent progress in optical flow

Solving the system

fAu

How to solve?

2 components of success: fast convergence

good initial guess

Start with some initial guess

and apply some iterative method

initialu

Relaxation schemes have smoothing property:

Only neighboring pixels are coupled in relaxation

scheme

It may take thousands of iterations to propagate

information to large distance

. . . . . . . . . . . .

Relaxation smoothes the error

Relaxation smoothes the error Examples

2D case:

1D case:

Error of initial guess Error after 5 relaxation Error after 15 relaxations

Idea: coarser grid

On a coarser grid low frequencies become higher

Hence, relaxations can be more effective

initial grid – fine grid

coarse grid – we take every second point

Multigrid 2-Level V-Cycle

1. Iterate ⇒ error becomes smooth

2. Transfer error equation to the coarse level ⇒ low frequencies become high

3. Solve for the error on the coarse level ⇒ good error estimation

4. Transfer error to the fine level

5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts

make iteration process faster (on the coarse grid we can effectively minimize the error)

obtain better initial guess (solve directly on the coarsest grid)

Coarse grid - advantages

initialu

fAu

Coarsening allows:

go to the coarsest grid

solve here the equation

to find initialu

interpolate to the

finer grid

initialu

Multigrid approach – Full scheme

Confidence Metric

• Intrinsic in Local Methods• How to evaluate for global methods?

– Edge strength?• Doesn’t work (Barron et al.,1994)

Confidence Metric

• Histogram of error contribution

Error

Number of pixels

Confidence Metric

More Results

More Results

Further Reading• Combining the advantages of local and global optic flow methods

(“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005

• High accuracy optical flow estimation based on a theory for warping

T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005• Towards ultimate motion estimation: Combining highest accuracy with

real-time performance. A. Bruhn, J. Weickert, 2005• Bilateral filtering-based optical flow estimation with occlusion

detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006