Introduction to Motion Introduction to Motion Estimation for Video CodecEstimation for Video Codec

DISP Lab, GICE, NTU

Advisor:Jian-Jiun DingSpeaker:Jian-Hwu WangDate:05/06/2011

1

DISP Lab, GICE, NTU

• Introduction to Motion Estimation–Motion Vector

– Cost Function

• Video Codec

• The Methods for Motion Estimation

• Conclusion

Outline

2 /33

DISP Lab, GICE, NTU

• Using motion estimation to find the motion vectors

–Motion Vector• A 2D point that represents the consistent between the

current block and space of previous frame.

Introduction to Motion Estimation

MotionVector (x,y)

(x+u,y+v)Currentblock

Best matchinglocation

Search range

Motion vector(u,v)

3 /33

DISP Lab, GICE, NTU

• Motion Estimation– Cost function• Sum of Absolute Difference(SAD)

• Sum of Squared Difference(SSD)

Introduction to Motion Estimation

4 /33

DISP Lab, GICE, NTU

• Motion Estimation– Cost function• Mean Absolute Error(MAE)

• Mean Square Error(MSE)

Introduction to Motion Estimation

5 /33

DISP Lab, GICE, NTU

• Introduction to Motion Estimation

• Video Codec– Video encoder

– Video decoder

• The Methods for Motion Estimation

• Conclusion

Outline

6 /33

DISP Lab, GICE, NTU

• Video Encoder[1]– Intra-prediction

Video Codec

7 /33

DISP Lab, GICE, NTU

• Video Encoder[1]– Inter-prediction

Video Codec

8 /33

DISP Lab, GICE, NTU

• Video Decoder[1]

Video Codec

9 /33

DISP Lab, GICE, NTU

• Introduction to Motion Estimation

• Video Codec

• The Methods for Motion Estimation– Full motion search

– Fast motion search

– True motion search

• Conclusion

Outline

10 /33

DISP Lab, GICE, NTU

• Full motion search

The Methods for Motion Estimation

(A) Top-to-bottom scan (B) Spiral search[2]

11 /33

DISP Lab, GICE, NTU

• Fast motion search

The Methods for Motion Estimation

(A) Three step search[3] (B) Diamond search[4]

12 /33

DISP Lab, GICE, NTU

• Motion Analysis

– Traditional motion estimation• Its goal is found for high compression rate.

• There are wrong motion vectors while blocks in several situations.

– True motion estimation(TME)• The goal of TME is described the meaningful

information of moving object in video sequence.

The Methods for Motion Estimation

13 /33

DISP Lab, GICE, NTU

• The feature of motion vector field(MVF) of TME• The consistency in spatial domain

• The dependent in time domain

The Methods for Motion Estimation

14 /33

DISP Lab, GICE, NTU

• True Motion Estimation• Overlapped block-based motion estimation[5]

– Take a block that block size bigger than normal block size

– 8x8 -> 16x16

– Perform motion estimation after sampling 16x16 block to 8x8

– Post smoothness – Motion vector median filter

The Methods for Motion Estimation

15 /33

DISP Lab, GICE, NTU

• True Motion Estimation• Overlapped block bi-directional motion estimation[6][7]

– 8x8 -> 12x12

– Post smoothness – Motion vector median filter

The Methods for Motion Estimation

9

2

9

21 ),(),(),(

j iycxcnycxcnyx jmvyimvxfjmvyimvxfmvmvSAD

16 /33

DISP Lab, GICE, NTU

• True Motion Estimation• 3-D recursive search[8]• 1-D recursive search

– Small search region» Candidate motion vector

– Update motion vector by using recursive research» Update motion vector

– Stop condition is necessary

The Methods for Motion Estimation

17 /33

DISP Lab, GICE, NTU

• True Motion Estimation• 3-D recursive search

The Methods for Motion Estimation

max

max

( , ) , ,

2 , ,0

( , ) , ,

2 , ,0

a a a

a

b b b

b

XCS X t C CS C D X t US X t

Y

XD X t T

Y

XCS X t C CS C D X t US X t

Y

XD X t T

Y

C

Sa Sb

Tb Ta

0

3,

0

3,

0

1,

0

1,

2

0,

2

0,

1

0,

1

0,

0

0nUS

18 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

• λa = 0.25

• 12 λ≦ b 20≦

The Methods for Motion Estimation

19 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

– The types of motion vector• Unrelated MV

• Matched MV

The Methods for Motion Estimation

20 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

– The types of motion vector

• Unmatched MV

• Uncertain MV

The Methods for Motion Estimation

Frame 39 Frame 40

Repeative Pattern The MVF of Uncertain MV

21 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

– Introduction to reliable motion decision unit• The drawbacks of conventional motion estimation

– Can’t classify the reliabilities of MVs

– It would make the blocks of consecutive frames produce wrong MVs.

• Reliable/Unreliable Motion Vector

The Methods for Motion Estimation

Reliable MV Unreliable MV

Matched MV

Unrelated MVUnmatched MVUncertain MV

22 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

– Introduction to reliable motion decision unit• Reliable motion decision unit based on Early-stop search.

– Early-stop search and Early-stop point(ESP)

The Methods for Motion Estimation

(a)ESP distribution of reliable MV

(b)ESP distribution of unreliable MV

23 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

– ESP classification

The Methods for Motion Estimation

(c)The CDF of ESP of reliable MV

(d)The CDF of ESP of unreliable MV

24 /33

DISP Lab, GICE, NTU

True motion estimation based on reliable motion decision unitSmoothness constraint (SC)

•The comparison of smoothness constraint and post-smoothness(PS)

The Methods for Motion Estimation

52 2

1

( , ) ( ) ( )xi yii

SC m n m mv n mv

( , ) ( , ) ( , )Cost m n SAD m n SC m n

,( , ) arg min ( , )

M m n Mu v Cost m n

5

1 32

4 C

The MVF of SC The MVF of PS

25 /33

DISP Lab, GICE, NTU

• The comparison of method

– Downsample frame rate 30Hz to 15Hz or 10Hz

–Make Bi-directional Motion Compensation Interpolation (BMCI) frames by its before and after frames

– Compare BMCI frames with extracted frame

The Methods for Motion Estimation

26 /33

DISP Lab, GICE, NTU

• Experiment result : – Compare PSNR between BMCI frame and extracted frame.

– 30Hz -> 15Hz

The Methods for Motion Estimation

PSNR(dB) Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

daughter News Silent Stefan Average

SS38.88 26.21 22.46 28.48 21.78 28.52 31.9 22.11 32.84 31.93 32.89 24.44 28.54

3DRS[9]44.29 30.77 23.4 30.77 21.71 29.53 33.98 25.8 36.55 34.17 34.17 23.35 30.71

Zhai[7]41.13 30.05 24.48 29.24 22.26 30.1 34.44 24.75 34.17 33.82 34.05 25.78 30.36

SC(λ = 12)43.22 29.01 23.21 30.87 22.19 30.73 35.57 25.02 37.64 34.1 34.47 26.01 31.01

SC(λ = 16)43.24 29.12 23.09 30.87 22.17 30.76 35.58 25.03 37.68 34.15 34.42 26.01 31.02

SC(λ = 20)43.26 29.18

23.087

30.84 22.17 30.72 35.59 25.05 37.66 34.15 34.42 25.91 31.01

27 /33

DISP Lab, GICE, NTU

• Experiment result : – Compare PSNR between BMCI frame and extracted frame.

– 30Hz -> 10Hz

The Methods for Motion Estimation

PSNR(dB) Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

daughter News Silent Stefan Average

SS35.45 23.41 17.41 26.64 19.84 24.51 29.87 19.15 31.3 29.49 29.61 18.34 25.42

3DRS[9]37.58 24.05 17.45 27.7 19.91 24.86 30.92 19.99 33.26 30.3 29.78 18.56 26.2

Zhai[7]36.33 24.11 17.45 26.86 19.81 24.9 30.65 19.78 31.83 30.04 29.86 18.4 25.84

SC(λ = 12)37.01 23.99 17.53 27.85 19.91 25.1 31.21 19.9 33.58 30.21 30.12 18.52 26.24

SC(λ = 16)37.01 23.99 17.54 27.86 19.91 25.09 31.26 19.92 33.59 30.18 30.11 18.54 26.25

SC(λ = 20)37.02 23.99 17.54 27.87 19.92 25.1 31.24 19.93 33.59 30.2 30.09 18.55 26.25

28 /33

DISP Lab, GICE, NTU

• Elapsed time:– 8x8 block, there are 64 subtraction operators and 63 addition operators.

– 12x12 block, there are 144 subtraction operators and 143 addition operators.

The Methods for Motion Estimation

Method Maximum operational complexity of motion estimation in each frame

SS (64 +63)x(33x33)x((352x288)/64) = 219,071,952

3DRS[10] (64 +63)x5x30x((352x288)/64) = 30,175,200

Zhai[8] (144+143)x(33x33)x((352x288)/64)

+ ((352x288)/64)x(9x8x(2+1)) = 495,410,256

Proposed 2x(64+63)x(33x33)x((352x288)/64) = 438,143,904

29 /33

DISP Lab, GICE, NTU

• Elapsed time:

The Methods for Motion Estimation

(ms)/Frame Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

daughter News Silent Stefan Average

SS241 1159 916 725 1076 671 688 768 694 359 469 814 715

3DRS[9]243 449 384 432 418 401 448 390 437 310 416 417 395

Zhai[7]372 2510 1789 1427 2396 1285 1338 1257 1395 622 826 1496 1393

SC(λ = 12)446 771 1173 738 1289 766 762 897 626 520 583 957 794

SC(λ = 16)442 747 1166 739 1283 753 744 889 621 513 582 947 786

SC(λ = 20)443 746 1161 738 1284 754 743 889 620 511 581 950 785

30 /33

DISP Lab, GICE, NTU

• The motion estimation(ME) of video processing was a popular research in recently decade years.

• It must be taken other side-information by ME to apply another applications, such as : object detection, tracking, video stabilization and frame rate up conversion…

Conclusion

31 /33

DISP Lab, GICE, NTU

• [1] Iain E. G. Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next-Generation Multimedia” John Wiley & Sons Inc, 2003

• [2] C. Chok-Kwan, P. Lai-Man, “Normalized Partial Distortion Search Algorithm for Block Motion Estimation,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 10, No. 3, Apr 2000, pp.417-422.

• [3] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro, “Motion compensated interframe coding for video conferencing,” in Proc. NTC81, pp. C9.6.1-9.6.5, New Orleans, LA, Nov./Dec. 1981.

• [4] J. Y. Tham, S. Ranganath, M. Ranganath, and A. A. Kassim,   "A novel unrestricted center-biased diamond search algorithm for block motion estimation",  IEEE Trans. Circuits Syst. Video Technol.,  vol. 8,  pp.369 - 377 , 1998.

• [5] Taehyeun Ha, Seongjoo Lee and Jaeseok Kim, “Motion Compensated Frame Interpolation by new Block-based Motion Estimation Algorithm,” IEEE Transactions on Consumer Electronics, Volume 50, Issue 2, pp.752-759, May 2004.

Reference

32 /33

DISP Lab, GICE, NTU

• [6]J. Zhai, K. Yu, J. Li, and S. Li, “A low complexity motion compensated frame interpolation method,” in Proc.IEEE ISCAS, May 2005, pp. 23–26.

• [7]Ya-Ting Yang, Yi-Shin Tung, and Ja-LingWu, “Quality enhancement of frame rate up-converted video by adaptive frame skip and reliable motion extraction,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 12, pp.1700–1713, Dec. 2007.

• [8]G. de Haan, P.W.A.C. Biezen, H. Huijgen, and O.A. Ojo, “True-Motion Estimation with 3-D Recursive Search Block Matching”, IEEE Transactions on Circuits and Systems for Video Technology, VOL.3, NO.5, OCTOBER 1993.

Reference

33 /33