Outline

transcript

Introduction to Motion Introduction to Motion Estimation for Video CodecEstimation for Video Codec

DISP Lab, GICE, NTU

Advisor:Jian-Jiun DingSpeaker:Jian-Hwu WangDate:05/06/2011

DISP Lab, GICE, NTU

• Introduction to Motion Estimation–Motion Vector

– Cost Function

• Video Codec

• The Methods for Motion Estimation

• Conclusion

Outline

DISP Lab, GICE, NTU

• Using motion estimation to find the motion vectors

–Motion Vector• A 2D point that represents the consistent between the

current block and space of previous frame.

Introduction to Motion Estimation

MotionVector (x,y)

(x+u,y+v)Currentblock

Best matchinglocation

Search range

Motion vector(u,v)

DISP Lab, GICE, NTU

• Motion Estimation– Cost function• Sum of Absolute Difference(SAD)

• Sum of Squared Difference(SSD)

DISP Lab, GICE, NTU

• Motion Estimation– Cost function• Mean Absolute Error(MAE)

• Mean Square Error(MSE)

DISP Lab, GICE, NTU

• Introduction to Motion Estimation

• Video Codec– Video encoder

– Video decoder

• The Methods for Motion Estimation

• Conclusion

Outline

DISP Lab, GICE, NTU

• Video Encoder[1]– Intra-prediction

Video Codec

DISP Lab, GICE, NTU

• Video Encoder[1]– Inter-prediction

Video Codec

DISP Lab, GICE, NTU

• Video Decoder[1]

Video Codec

DISP Lab, GICE, NTU

• Introduction to Motion Estimation

• Video Codec

• The Methods for Motion Estimation– Full motion search

– Fast motion search

– True motion search

• Conclusion

Outline

10 /33

DISP Lab, GICE, NTU

• Full motion search

The Methods for Motion Estimation

(A) Top-to-bottom scan (B) Spiral search[2]

11 /33

DISP Lab, GICE, NTU

• Fast motion search

(A) Three step search[3] (B) Diamond search[4]

12 /33

DISP Lab, GICE, NTU

• Motion Analysis

– Traditional motion estimation• Its goal is found for high compression rate.

• There are wrong motion vectors while blocks in several situations.

– True motion estimation(TME)• The goal of TME is described the meaningful

information of moving object in video sequence.

13 /33

DISP Lab, GICE, NTU

• The feature of motion vector field(MVF) of TME• The consistency in spatial domain

• The dependent in time domain

14 /33

DISP Lab, GICE, NTU

• True Motion Estimation• Overlapped block-based motion estimation[5]

– Take a block that block size bigger than normal block size

– 8x8 -> 16x16

– Perform motion estimation after sampling 16x16 block to 8x8

– Post smoothness – Motion vector median filter

15 /33

DISP Lab, GICE, NTU

• True Motion Estimation• Overlapped block bi-directional motion estimation[6][7]

– 8x8 -> 12x12

– Post smoothness – Motion vector median filter

21 ),(),(),(

j iycxcnycxcnyx jmvyimvxfjmvyimvxfmvmvSAD

16 /33

DISP Lab, GICE, NTU

• True Motion Estimation• 3-D recursive search[8]• 1-D recursive search

– Small search region» Candidate motion vector

– Update motion vector by using recursive research» Update motion vector

– Stop condition is necessary

17 /33

DISP Lab, GICE, NTU

• True Motion Estimation• 3-D recursive search

( , ) , ,

2 , ,0

( , ) , ,

2 , ,0

XCS X t C CS C D X t US X t

XD X t T

XCS X t C CS C D X t US X t

XD X t T

18 /33

DISP Lab, GICE, NTU

• True motion estimation based on reliable motion decision unit

• λa = 0.25

• 12 λ≦ b 20≦

19 /33

DISP Lab, GICE, NTU

– The types of motion vector• Unrelated MV

• Matched MV

20 /33

DISP Lab, GICE, NTU

– The types of motion vector

• Unmatched MV

• Uncertain MV

Frame 39 Frame 40

Repeative Pattern The MVF of Uncertain MV

21 /33

DISP Lab, GICE, NTU

– Introduction to reliable motion decision unit• The drawbacks of conventional motion estimation

– Can’t classify the reliabilities of MVs

– It would make the blocks of consecutive frames produce wrong MVs.

• Reliable/Unreliable Motion Vector

Reliable MV Unreliable MV

Matched MV

Unrelated MVUnmatched MVUncertain MV

22 /33

DISP Lab, GICE, NTU

– Introduction to reliable motion decision unit• Reliable motion decision unit based on Early-stop search.

– Early-stop search and Early-stop point(ESP)

(a)ESP distribution of reliable MV

(b)ESP distribution of unreliable MV

23 /33

DISP Lab, GICE, NTU

– ESP classification

(c)The CDF of ESP of reliable MV

(d)The CDF of ESP of unreliable MV

24 /33

DISP Lab, GICE, NTU

True motion estimation based on reliable motion decision unitSmoothness constraint (SC)

•The comparison of smoothness constraint and post-smoothness(PS)

( , ) ( ) ( )xi yii

SC m n m mv n mv

( , ) ( , ) ( , )Cost m n SAD m n SC m n

,( , ) arg min ( , )

M m n Mu v Cost m n

The MVF of SC The MVF of PS

25 /33

DISP Lab, GICE, NTU

• The comparison of method

– Downsample frame rate 30Hz to 15Hz or 10Hz

–Make Bi-directional Motion Compensation Interpolation (BMCI) frames by its before and after frames

– Compare BMCI frames with extracted frame

26 /33

DISP Lab, GICE, NTU

• Experiment result : – Compare PSNR between BMCI frame and extracted frame.

– 30Hz -> 15Hz

PSNR(dB) Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

daughter News Silent Stefan Average

SS38.88 26.21 22.46 28.48 21.78 28.52 31.9 22.11 32.84 31.93 32.89 24.44 28.54

3DRS[9]44.29 30.77 23.4 30.77 21.71 29.53 33.98 25.8 36.55 34.17 34.17 23.35 30.71

Zhai[7]41.13 30.05 24.48 29.24 22.26 30.1 34.44 24.75 34.17 33.82 34.05 25.78 30.36

SC(λ = 12)43.22 29.01 23.21 30.87 22.19 30.73 35.57 25.02 37.64 34.1 34.47 26.01 31.01

SC(λ = 16)43.24 29.12 23.09 30.87 22.17 30.76 35.58 25.03 37.68 34.15 34.42 26.01 31.02

SC(λ = 20)43.26 29.18

23.087

30.84 22.17 30.72 35.59 25.05 37.66 34.15 34.42 25.91 31.01

27 /33

DISP Lab, GICE, NTU

• Experiment result : – Compare PSNR between BMCI frame and extracted frame.

– 30Hz -> 10Hz

PSNR(dB) Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

SS35.45 23.41 17.41 26.64 19.84 24.51 29.87 19.15 31.3 29.49 29.61 18.34 25.42

3DRS[9]37.58 24.05 17.45 27.7 19.91 24.86 30.92 19.99 33.26 30.3 29.78 18.56 26.2

Zhai[7]36.33 24.11 17.45 26.86 19.81 24.9 30.65 19.78 31.83 30.04 29.86 18.4 25.84

SC(λ = 12)37.01 23.99 17.53 27.85 19.91 25.1 31.21 19.9 33.58 30.21 30.12 18.52 26.24

SC(λ = 16)37.01 23.99 17.54 27.86 19.91 25.09 31.26 19.92 33.59 30.18 30.11 18.54 26.25

SC(λ = 20)37.02 23.99 17.54 27.87 19.92 25.1 31.24 19.93 33.59 30.2 30.09 18.55 26.25

28 /33

DISP Lab, GICE, NTU

• Elapsed time:– 8x8 block, there are 64 subtraction operators and 63 addition operators.

– 12x12 block, there are 144 subtraction operators and 143 addition operators.

Method Maximum operational complexity of motion estimation in each frame

SS (64 +63)x(33x33)x((352x288)/64) = 219,071,952

3DRS[10] (64 +63)x5x30x((352x288)/64) = 30,175,200

Zhai[8] (144+143)x(33x33)x((352x288)/64)

+ ((352x288)/64)x(9x8x(2+1)) = 495,410,256

Proposed 2x(64+63)x(33x33)x((352x288)/64) = 438,143,904

29 /33

DISP Lab, GICE, NTU

• Elapsed time:

(ms)/Frame Akiyo Bream Bus Carphone Football Foreman Hallmonitor

MobileMother

SS241 1159 916 725 1076 671 688 768 694 359 469 814 715

3DRS[9]243 449 384 432 418 401 448 390 437 310 416 417 395

Zhai[7]372 2510 1789 1427 2396 1285 1338 1257 1395 622 826 1496 1393

SC(λ = 12)446 771 1173 738 1289 766 762 897 626 520 583 957 794

SC(λ = 16)442 747 1166 739 1283 753 744 889 621 513 582 947 786

SC(λ = 20)443 746 1161 738 1284 754 743 889 620 511 581 950 785

30 /33

DISP Lab, GICE, NTU

• The motion estimation(ME) of video processing was a popular research in recently decade years.

• It must be taken other side-information by ME to apply another applications, such as : object detection, tracking, video stabilization and frame rate up conversion…

Conclusion

31 /33

DISP Lab, GICE, NTU

• [1] Iain E. G. Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next-Generation Multimedia” John Wiley & Sons Inc, 2003

• [2] C. Chok-Kwan, P. Lai-Man, “Normalized Partial Distortion Search Algorithm for Block Motion Estimation,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 10, No. 3, Apr 2000, pp.417-422.

• [3] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro, “Motion compensated interframe coding for video conferencing,” in Proc. NTC81, pp. C9.6.1-9.6.5, New Orleans, LA, Nov./Dec. 1981.

• [4] J. Y. Tham, S. Ranganath, M. Ranganath, and A. A. Kassim, "A novel unrestricted center-biased diamond search algorithm for block motion estimation", IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp.369 - 377 , 1998.

• [5] Taehyeun Ha, Seongjoo Lee and Jaeseok Kim, “Motion Compensated Frame Interpolation by new Block-based Motion Estimation Algorithm,” IEEE Transactions on Consumer Electronics, Volume 50, Issue 2, pp.752-759, May 2004.

Reference

32 /33

DISP Lab, GICE, NTU

• [6]J. Zhai, K. Yu, J. Li, and S. Li, “A low complexity motion compensated frame interpolation method,” in Proc.IEEE ISCAS, May 2005, pp. 23–26.

• [7]Ya-Ting Yang, Yi-Shin Tung, and Ja-LingWu, “Quality enhancement of frame rate up-converted video by adaptive frame skip and reliable motion extraction,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 12, pp.1700–1713, Dec. 2007.

• [8]G. de Haan, P.W.A.C. Biezen, H. Huijgen, and O.A. Ojo, “True-Motion Estimation with 3-D Recursive Search Block Matching”, IEEE Transactions on Circuits and Systems for Video Technology, VOL.3, NO.5, OCTOBER 1993.

Reference

33 /33

Outline

Documents