Motion Estimation

Multimedia Systems and StandardsS2 IF Telkom University

2

Overviewvideo compression schemesInter-frame vs. Intra-frame Intra-frame (e.g. JPEG, DCT, Wavelet)

more space Independenteasy for editing

Inter-frame (long group of picture/ motion estimation, e.g. MPEG (1, 2, 4), H.261, and H.263) there is relation between the frames,better for compression

3

Motion estimationExploiting similarities between the regions boundaries in successive video frames Predictive differential coding: to code the changes incurred on the boundaries of detected regions from one to another. Trade-off: coding efficiency vs algorithmic complexity.

minimal complexity: image segmentation could only be utilised for each video frame with no consideration given to temporal redundancies of shape and texture information.

4

Motion estimationMotion Estimation

Motion VectorMotion CompensationMotion ModelMotion EstimationEncoding and Decoding

Matching AlgorithmSearching AlgorithmEvaluation Matrices

5

Motion Vector [1]Definition :(H.264/MPEG-4 AVC) : A two-dimensional vector used for inter prediction that provides an offset from the coordinates in the decoded picture to the coordinates in a reference picture.

6

Motion vector [2]Motion representation

7

Motion CompensationApplying the motion vectors to a frame to synthesize the transformation to the next frameMany frames on the movie only have little different between frames (camera or object moving)

8

9

Motion ModelMotion in 3-D space corresponds to displacements in the image planeMotion compensation is conducted to provide a prediction signal for efficient video compressionEfficient motion compensation often uses side information to transmit the displacementsDisplacement must be efficiently represented for video compression

10

Motion Estimation [1]The process of determining motion vectors that describe the transformation from one 2D image to another, usually from adjacent frames in a video sequenceBlock-matching with previous frameSpace for saving bit motion vector is smaller than full blockEfficient motion estimation reduces the energy in the motion-compensated residual frame and can dramatically improve compression performance.Motion estimation can be very computationally intensive and so this compression performance may be at the expense of high computational complexity.

11

Motion Estimation [2]Key performance issues�� Coding performance Complexity Storage and/or delay

The motion estimation creates a model by modifying one or more reference frames to match the current frame as closely as possible.The current frame is motion compensated by subtracting the model from the frame to produce a motion-compensated residual frame.This is coded and transmitted, along with the information required for the decoder to recreate the model (typically a set of motion vectors).

12

Motion Estimation [3]At the same time, the encoded residual is decoded and added to the model to reconstruct a decoded copy of the current frame( which may not be identical to the original frame because of coding losses).This reconstructed frame is stored to be used as reference frame for further predictions.

13

Motion Estimation [4]2-D spatial search :

14

Motion Estimation [5]Residual Error

15

Motion Estimation [6]Choice of reference frames1. Forward Prediction

o Using an older encoded frame2. Backwards Prediction

o Using a future frameo Encoder to buffer coded

frameo Out of temporal order

3. Bidirectional PredictionNote : by using multiple reference frames will make encode and decode complexity and storage requirement increase

16

Encoding and Decoding [1]

17

Encoding and decoding [2]

Encoding1. Find a block in previous frame(reference frame).2. Choose a region that have smallest error3. Subtraction the matching region with current block to get

difference block.4. Encode and transmit difference block.5. Encode and transmit motion vector that indicated position

of matching region. Decoding

1. Decoding the difference between block and motion vector2. Sum the difference block with matching region in the

reference frame. A region that was pointed by motion vector.

Fact: 60% computational complexity at encoder side. Fast algorithms are necessary for real time applications.

Matching Algorithm

19

Matching Algorithm [1]1. Dividing current and previous frame into

non-overlapping block MxN2. For each block in current frame:

• Finding in the region that have size 2p+N at the previous frame.

• Finding the nearst block by using evaluation matrices

20

Matching Algorithm [2]1. Slow Motion Object (reduce number of

search point) Cross Search Algorithm (CSA) Full Search Algorithm (FSA) Spiral Search Three Step Search Two Dimensional Logarithmic Search (TDL) Binary Search (BS) Four Step Search (FSS) Orthogonal Search Algorithm (OSA)

2. Fast Motion Object Hierarchical Search Algorithm (HSA)

21

Matching Algorithm [3]Search

Strategies I : full search

All possible displacements within the search range are comparedComputationally expensive

22

Matching Algorithm [4]Search Strategies II: Three Step SearchAlgorithm

Step 1 Select an initial step size (s) equal or

slightly larger than half of the maximum search range

Calculate the error for the block at the center of search area and 8 square neighborhood point at the distance of s from center

Step 2 Move the center to the point with

minimum distortion Reduce step size by a factor of two If the step size is greater than one,

repeat the step1, otherwise go to step 3

Step 3 Final point with minimum distortion is

the result

23

Matching Algorithm [5] Search Strategies III : 2D

logarithmic searchAnil K. JainIt can be more accurate than TSSThe search window is large

AlgorithmStep1

Select an initial step size (s) Calculate the error for the block at

the center of search area and four point at x and y axis at distance of s from center

Step2 If the position of best match is at

center keep the center unchanged and reduce the step size by half, otherwise the best match becomes the center

Then step 1 is repeatedStep3

When the step size becomes 1 all the 8 neighbor blocks around the center will be checked for finding the best match

24

Matching Algorithm [6]Search Strategies IV : Diamond Search

25

Matching Algorithm [7]No

Algorithm Advantages Disadvantages

1 Three Step Search Low Complexity in terms of selected candidate locationsGood regularity in terms of motion vector generation

Complexity factor increases with search area sizeHigh data bandwidth

2 2-D Logarithmic Search

Suitable for fast motion The accuracy of algorithm is low when the motion vector is at anangle to x-y axis

3 Four Step Search Very low comp, More accurate than TTS, memory saving, for small motion

Risk of local minima if motion far away from center

4 Orthogonal Search Very fast for small motions Local minima may stop the search very soon

5 One at Time Search

Low computational Local Maxima

6 Cross Search Low computational Performance is worse than Three-step search

26

Evaluation Matrices [1]Mean squared error (MSE)

Mean absolute error (MAE)

where :f(m,n)= a pixel on coordinate (m,n) of NxN pixel blockg(m+i,n+j) = a pixel on coordinate (m+i,n+j) that corresponding with previous frame.

N

m

N

n

wjiwjnimgnmfN

jiM1 1

22 , ,),(),(1),(

N

m

N

n

wjiwjnimgnmfN

jiM1 1

2 , ,,(),(1),(

27

Evaluation Matrices [2]

Current block

reference area

MSE at (0,0) {(1-4)2 + (3-2)2 + (2-3)2 + (6-4)2 + (4-2)2 + (3-2)2 + (5-4)2 + (4-3)2 + (3-3)2}/9 = 2.44

28

Evaluation Matrices [3]Sum of absolute differences (SAD)• Matching the template into search image• Sum all of number in same matrix Example: 20, 25, and 17