Optical Flow Estimation using Variational Techniques Darya Frolova.

transcript

Optical Flow Estimation Optical Flow Estimation using using

Variational Techniques Variational Techniques

Darya FrolovaDarya Frolova

Part IPart I

Mathematical Preliminaries

AgendaAgenda

Minimization problem: ? )( min

Investigate the existence of a solution with respect to reasonable assumptions on functional F and space A

- conditions for A and F,- main theorem: existence and uniqueness of minimum

Optical Flow - what is this? - different approaches - formulation of minimization problem for Optical Flow - existence of solution (using main theorem)

minimization problem: ? )( min

TheoremA is a compact set, function F is continuous on A.

Ax 0 Then such that )( )( 0 min xFxFAx

What happens when A is not a compact?

What happens when functional F is not continuous?

Does the minimum exist?

New TheoremNew Theorem

Theorem

A is a compact set

Ax 0 Then such that )( )( 0 min xFxFx

function F is continuous on A

A = BV (Ω) – space of functions of bounded variation

functional F is coercive and lower semicontinuous on A

Bounded Variation -1D caseBounded Variation -1D case

Rbaxf ],[:)(

f has bounded variation over ],[ ba if exists const M such that

|)()(| ... |)()(||)()(| 1121 nxfbfxfxfafxf

for all bxxxa n 121 ...

Consider

a x1 x2 xn -1 b……

)(af )( 1nxf )(bf

)( 2xf)( 1xf

M | )( )(| 1

iii xfxf

Definition

Bounded VariationBounded Variation

Definition

The space of functions of bounded variation on Ω is denoted BV (Ω)

Total variation of function f can be represented as follows:

sup|)(|φ

where - function φ is compactly supported (is zero outside of a compact set),

] 1 ,1[ -

If f is differentiable, then dx

Supremum for

sign(df/dx)

dfsign

Then variation of function f =

max loc min loc

)( )(x x

Function f does not need to be differentiable

Bounded Variation – ND caseBounded Variation – ND case

sup |)(| dxdivffD

i xdiv

1 || ,)( ), ..... ,( )(

bounded open subset, function NR )( 1 Lf

Variation of over f

Bounded Variation – exampleBounded Variation – example

xxxf 1sin)( 21(

xxxxf 1cos1sin2)(

|1cos1sin2||)(| dxxxxdxxf

functions under integral are bounded on [0,1]

xxxf 1sin)( 2(

xxxxf 11cos1sin)(

|11cos1sin||)(| dxxxxdxxf - problem at x = 0, 2nd term is not bounded

DefinitionsDefinitions

Definition Banach space – complete, normed linear space

Definition

Space X is said to be complete, when any Cauchy sequence xn from X converges

DefinitionConsider sequence xn from X.

If N 0 natural, such that for any m, n > N

|| || mn xx , then xn - Cauchy sequence

Space of functions of bounded variation BV(Ω) is a Banach space

Definition

Functional f is coercive if

)( lim||

DefinitionConsider sequence xn. Denote . inf n

knk xX

Then lower limit of sequence xn is a limit of sequence Xk:

xXx inflimlimlim

Dual SpaceDual Space

Let X denote a real Banach space. Dual space of X : X * – space of linear bounded operators

X * = f : X → R

Definition

Linear operator:

f (αx + βy) = α f (x) + β f (y)

Bounded operator:

bounded set converts to bounded set

|||| ||)(||

such that 0

Dual spaceDual space

Definition

X is called reflexive if ( X *) *

Reflexive Not reflexive

e.g. finite-dimensional (normed) spaces,

Hilbert spaces

e.g. the space of sequences ℓ∞

||max nxℓ∞ = xn :

( if bidual space of X is equal to X )

Topologies on XTopologies on X

Definition Sequence xn from X strongly converges to point x if

)(n 0 | | Xn xx

If sequence strongly converges, then it weakly converges

Definition Sequence xn from X weakly converges to point x if

)(n )( )( xfxf n

for every functional f from X*

( if converges sequence of real numbers f (xn) for any linear bounded f )

Topologies on XTopologies on X**Definition

Sequence fn from X* strongly converges to f if

)(n 0 | | * Xn ff

Definition Sequence fn from X weakly converges to f if

)(n )( )( fgfg n

for every g from (X*)* (bidual space of X)

Definition Sequence fn from X weakly* converges to f if

)(n ffn

for every x from X

Direct MethodDirect Method

Problemf : X → R , where X is a Banach space

inf f (x)xX

Does the solution exist?

The proof consists of three steps , which is called

Direct Method of Variational Calculus

TheoremTheorem

functional f is coercive and lower semicontinuous on X,

X is Banach and reflexive space

Thenfunctional f has its minimum on X :

)( )( such that min00 xfxfXxXx

Direct MethodDirect Method

Step A Construct minimizing sequence xn for functional f :

)( )( inflim xfxfXx

Step Ba) If f is coercive, then minimizing sequence is bounded

Step C If f is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min0 xfxf

b) If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence:

Xxxjn 0

weakly exists

Step A

There exists an infimum of the set f (x), x X R

and exists a sequence R , which converges to this infimum

f (x), x X R

Step B (a)If f is coercive, then minimizing sequence is bounded :

CxC n || such that 0const

Proof )( )( inflim xfxfXx

Consider minimizing sequence xn:

Assume that it is not bounded: CxnC n || such that 0 (equivalently, for any ball with radius C = 1,2… there

exists an element outside this ball)cnx

cnxIf we form a subsequence of these , it will satisfy: ) ( || cn nxc

f is coercive

)( lim||

But is subsequence of f (xn), that is why )(cnxf

)( lim||

So, unbounded xn cannot be minimizing sequence.

To prove

Step B (b)

we can conclude that exists x0 X and exists subsequence

such that:

0weakly xx

We proved that minimizing sequence is bounded.

If X is reflexive ( ( X* )* = X ) then using theorem about weak sequential compactness, which states that:

for any bounded sequence in reflexive Banach space there exists weakly convergent subsequence

If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence

To prove

Step C

If f is lower semicontinuous at x0 x0 is a minimum point of f

0weakly xx

jn We proved the existence of such that : jnx

)()( )( inflimlim xfxfxfXx

If there exists limit of f (xn) then there exists a limit of its subsequence

and these limits are equal:

)( jnxf

On the other hand, from the lower semicontinuity it follows that )( )( 0lim xfxfjn

Hence )( )( inf0 xfxfXx

)( )( min0 xfxfXx

This means that

To prove

Part IIPart II

Optical Flow

Image SequenceImage Sequence

Sequence of images contains information about the scene,We want to estimate motion (using variational formulation)

2D motion field2D motion field

Optical center

2D motion field

Projection on the image plane of the 3D velocity of the scene

3D motion field

Image intensity

Motion vector - ?

Optical FlowOptical FlowWhat we are able to perceive is just an apparent motion, called

Optical flow

(motion, observable only through intensity variations)

Intensity remains constant – no motion is perceived

No object motion, moving light source produces intensity variations

Optical flow-methodsOptical flow-methods

Correlation-based techniques - compare parts of the first image with parts of the second in terms of the similarity in brightness patterns in order to determine the motion vectors

Feature-based methods - compute and analyze Optical Flow at small number of well-defined image features

Gradient-based methods - use spatiotemporal partial derivatives to estimate flow at each point

Brightness constancyBrightness constancy

Intensity of a point keeps constant along its trajectory

(reasonable for small displacements)

),( xtI intensity of the pixel ),( 21 xxx at time t

Start from point x0 at time t0. Trajectory → ( t, x (t) )

txtItxtI allfor ),( ))( ,( 00) ,( ) )( ,( 0000 xttxt

0at 0 ttt

Differentiating →

)( )( 00 tdt

dxtv We will search the Optical Flow as the velocity field:

Brightness constancyBrightness constancy

Given sequence I (t, x) and time t0

Find the velocity v (x) such that:

We need to find the velocity field - 2 componentsWe have 1 scalar equation

Can find only “normal flow”

Component in the direction of gradient I

0 ),( ),( )( 000 xtIxtIxv t

Optical Flow constraint (OFC)

Aperture problemAperture problem

Solving the aperture problemSolving the aperture problem

Second order derivative constraint : conservation of along trajectories

0 ) ,(

Rigid deformations are not considered (object moves locally in one direction)

Sensitive to noise

Weighted least squaresWeighted least squares

Velocities are constant in small window (spatial neighborhood)

dxIIvxwrx

),(Ball

w (x) is a window function ( gives more influence to the constraint at the center of the neighborhood than at the periphery)

Too local, no global regularity

Regularizing the velocity fieldRegularizing the velocity field

)( )( inf vSvAv

smoothing term

dxvdxIIvj

A (v) S (v)

Velocity should change slowly in spatial domain (in image plain)

Optical Flow constraint

Horn and Schunck

DiscontinuitiesDiscontinuities

But smoothing term does not allow to save discontinuities

Discontinuities near edges are

Synthetic example(method of Horn and Schunck)

Discontinuity-preserving approachDiscontinuity-preserving approach

dxIIv tv

j) || (

Where function η permits noise removal and edge conservation

dxIIv tv

dxv div ) )( (

[Black et.al, Cohen, Kumar, Balas, Tannenbaum, Blanc-Feraud]

[Suter, Gupta and Prince, Guichard and Rudin]

Discontinuity-preserving approach Discontinuity-preserving approach summarysummary

Given: sequence I (t, x)

Find: velocity field v that minimizes the energy functional E

||)( )( | | )( vDIcDvIDIvvE h

Smoothing term, we need to find

conditions on η for saving discontinuities

Is related to homogeneous

regions

L1 norm of the Optical Flow

constraint

Smoothing termSmoothing term

)( )(2

Function η : R+ → R+ is strictly convex, nondecreasing

η (0) = 0 (without loss of generality)

)( lim ss

bassbas )( η (s)

Homogeneous termHomogeneous term

2||)( )( vDIcvH

Idea: if there is no texture (there is no gradient), then there is no possibility to correctly estimate the flow field

So, we may force it to be zero

0 )( and 1 )( limlim0

scscss

]1 ; [ )( such that const exists cc mxcm

Without loss of generality:

Existence of solutionExistence of solution

Theorem Under the following hypotheses:

]1 ; [ )( such that const exists cc mxcm

bassbas )(

)( lim ss

η : R+ → R+ is strictly convex, nondecreasing, η (0) = 0

||)( )( | | )( min vDIcDvIDIvvE h

The minimization problem

has a unique solution in BV(Ω)

Existence of a solutionExistence of a solution

ProofUsing direct method of Variational Calculus:

Step A Construct minimizing sequence vn for functional E :

)( )( inflim)(

vEvEBVv

Step Ba) If E is coercive, then minimizing sequence is bounded

b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence:

Step C If E is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min

)(0 vEvE

||)( )( | | )( vDIcDvIDIvvE h

)( lim ss

c is bounded

So, functional E is coercive

)( )( inflim)(

vEvEBVv

)(0 vEvE

Space of functions of bounded variation is not reflexive: ( BV *)* ≠ BV

But it has such a property that

every bounded sequence Ij from BV (Ω) has a subsequence

that weakly* converges to some element I from BV (Ω)

)( )( inflim)(

vEvEBVv

)(0 vEvE

The End

Optical Flow Estimation using Variational Techniques Darya Frolova.

Documents