Motion Estimation using Markov Random Fields
Hrvoje Bogunović
Image Processing Group
Faculty of Electrical Engineering and Computing
University of Zagreb
Summer School on Image Processing, Graz 2004
Overview
• Introduction
• Optical flow
• Markov Random Fields
• OF+MRF combined
• Energy minimization techniques
• Results
Introduction
• Input:– Sequence of images (Video)
• Problem– Extract information about motion
• Applications– Detection, Segmentation, Tracking, Coding
Spatio-temporal spectrum
φ
f
Motion – aliasing
φ
f
1/x
1/t
Large area flicker
Loss of spatialresolution
Large motions - temporal aliasing
φ
fTemporal aliasing
Great loss of spatial resolution
Temporal anti-aliasing
φ
f
• No more overlaping on the f axis. • filtering (anit-aliasing) is performed after sampling, hence the blurring
Motion – eye tracking
φ
f
Motion estimation
• Images are 2-D projections of the 3-D world.
• Problem is represented as a labeling one.– Assign vector to pixel
• Vector field field of movement• Low level vision
– No interpretation
Example Ideal
Problems
• Problem is inherently ill-posed– Solution is not unique
• Aperture problem– Specific to local methods
Optical flow
• Main assumption: Intensity of the object does not change as it moves– Often violated
• First solved by Horn & Schunk– Gradient approach
• Other approaches include– Frequency based– Using corresponding features
Image differencing
Gradient approach
• Local by nature. Aperture problem is significant.
• Image understanding is not required– Very low level
Horn & Schunk
• Intensity stays the same in the direction of movement. I(x,y,t)
• After derivation
Horn & Schunk
• Spatial gradients Ix,Iy
– e.g. Sobel operator
• Temporal gradient It
– Image subtraction
( , ) ( , ) 0x y t
t
I I u v I
I I v
Regularization
• Tikhonov regularization for ill-posed problems
• Add the smoothness term
• Energy function
Result
Problems of the H-S method
• Assumption: There are no discontinuities in the image– Optical flow is over-smoothed.
• Gradient method. Only the edges which are perpendicular to motion vector contribute
• Image regions which are uniform do not contribute.
• Difficulty with large motions (spatial filtering)
Optical flow enhancement
• Optical flow can be piecewise smooth
• Discontinuities can be incorporated
• Solution: use the spatial context
• Problem is posed as a solution of the Bayes classifier. Solution in optimization sense. Search for optimum
Bayes classifier
• Main equation
• Solution using MAP estimation
( , ) ( )( | )
( )
P hypothesis observation P hypothesisP hypothesis observation
P observation
Markov Random Fields
• Suitable: Problems posed as a visual labeling problemn with contextual constraints
• Useful to encode a priori knowledge– required for bayes classifier (smoothness prior)– equvalence to Gibbs random fields (gibbs
distribution, exponential like)• Neighbourhoods• Cliques
– pairs,triples of neighbourhood points)– build the energy function
MRF
• Define sites: rectangular lattice
• Define labels
• define neighbourhood: 4,8 point
• Field is MRF:– P(f)>0
– P(fi|f{S-i})=P(fi|Ni)
Coupled MRF
• Field F is an optical flow field• Field L is a field of discontinuities
– line process
• Position of the two fields.
Context
• neighbourhoods and cliques
Motion estimation equations
Energy for MAP estimation
Parameters are estimated ad hoc
Energy minimization
• Global minimum– Simulated annealing– Genetic Algorithms
• Local minimum– Iterated Conditional Modes (ICM) (steepest
decent)– Highest Confidence First (HCF)
• specific site visiting
Simulated annealing(1) Find the initial temperature of the system T.
(2) Assign initial values of the field to random
(3) For every pixel:
Assign random value to f(i,j)
Calculate the difference in energy before and after If the change is better (diff>0) keep it.
Else keep it with the probability exp(diff/T)
(4) Repeat (3) N1 times
(5) T = f(T) where f decreases monotono
(6) Repeat (3-5) N2 times
Results (Square)
Horn-Schunk OF OF+MRF
Taxi
Results (Taxi)
Line process result (Taxi)
Cube
Results (cube)
Line process result (cube)
Q & A