PROJECT PROPOSAL FOR MULTIMEDIA PROCESSING

PERFORMANCE COMPARISON OF HEVC MAIN STILL PICTURE PROFILE

SPRING 2015

MULTIMEDIA PROCESSING- EE 5359

02/19/2015

ADVISOR: DR. K. R. RAO

DEPARTMENT OF ELECTRICAL ENGINEERING

UNIVERSITY OF TEXAS, ARLINGTON

DEEPU SLEEBA PHILIP

1001038966

deepu.philip@mavs.uta.edu

1 | P a g e

TABLE OF CONTENTS

1. Acronyms And Abbreviations............................................................... 22. Objective Of The Project...................................................................... 33. Overview Of HEVC.............................................................................. 34. WebP........................................................................................................

...95. Performance Comparison Metrics................ ......................................146. Implementation........................................................................................

157. Test

Configuration..................................................................................158. References...............................................................................................

...16

2 | P a g e

1. Acronyms And Abbreviations AMVP: Advanced motion vector prediction  AVC: Advanced Video Coding BD-PSNR: Bjontegaard metric calculation CABAC: Context Adaptive Binary Arithmetic Coding CB: Coding Block CIF: Common Intermediate Format CU: Coding Unit CTB: Coding Tree Block CTU: Coding Tree Unit DCT: Discrete Cosine Transforms DST: Discrete Sine Transform EBCOT: Embedded block coding with optimized truncation

GIF: Graphics interchange format HEVC: High Efficiency Video Coding HD: High Definition JCT-VC: Joint Collaborative Team on Video Coding MC: Motion Compensation ME: Motion Estimation MPEG: Moving Picture Experts Group MSP: Main Still Picture Profile MV: Motion Vector NGOV: Next Generation Open Video PNG: Portable Network Graphics PSNR: Peak Signal To Noise Ratio PU: Prediction Unit QP: Quantization Parameter QCIF: Quarter Common Intermediate Format RD: Rate Distortion SAO: Sample Adaptive Offset

3 | P a g e

SAD: Sum of Absolute Differences SATD: Sum of Absolute Transformed Differences (SATD) SHVC: Scalable HEVC SSIM: Structural Similarity SVC: Scalable Video Coding TM: True Motion TU: Transform Unit URQ: Uniform Reconstruction Quantization VCEG: Visual Coding Experts Group

2.Objective:

• The aim of this project is to compare the rate-distortion performance analysis of the HEVC MSP profile with that of WebP

• The peak-signal-to-noise ratio (PSNR) and the average bit rate savings in terms of Bjøntegaard delta rate (BD) is considered for this comparison

3.Overview of HEVC:• The High Efficiency Video Coding (HEVC) standard is the most recent

joint video project of the ITU-T Visual Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) standardization organizations, working together in a partnership known as the Joint Collaborative Team on Video Coding (JCT-VC) [1].

• It is widely used for many applications, including broadcast of high definition (HD) TV signals over satellite, cable, and terrestrial transmission systems, video content acquisition and editing systems, camcorders, security applications, Internet and mobile network video, Blu-ray Discs, and real-time conversational applications such as video chat, video conferencing, and telepresence systems. However, an increasing diversity of services, the growing popularity of HD video, and the emergence of beyond- HD formats (e.g., 4k×2k or 8k×4k resolution) are creating even stronger needs for coding efficiency

4 | P a g e

superior to H.264/ MPEG-4 AVC’s capabilities. An increased desire for higher quality and resolutions is also arising in mobile applications [2].

Block Diagram of HEVC ENCODER and DECODER :

Figure 1: Block Diagram of HEVC Encoder (with decoder modelling elements shaded in light gray). [3]

5 | P a g e

Figure. 1a HEVC Decoder Block Diagram [4]

3.1 Macro block concept and Prediction block sizes:

The concept of macro-block in HEVC [3] is represented by the Coding Tree Unit (CTU). CTU size can be 16x16, 32x32 or 64x64, while AVC macro-block size is 16x16. Larger CTU size aims to improve the efficiency of block partitioning on high resolution video sequence. Larger blocks provoke the introduction of quad-tree partitioning of a CTU into smaller coding units (CUs). A coding unit is a bottom-level quad-tree syntax element of CTU splitting. The CU contains a prediction unit (PU) and a transform unit (TU).

The TU is a syntax element responsible for storing transform data. Allowed TU sizes are 32x32, 16x16, 8x8 and 4x4. The PU is a syntax element to store prediction data like the intra-prediction angle or inter-prediction motion vector. The CU can contain up to four prediction units. CU splitting on PUs can be 2Nx2N, 2NxN, Nx2N, NxN, 2NxnU, 2NxnD, nLx2N and nRx2N where 2N is a size of a CU being split. In the intra-prediction mode only 2Nx2N PU splitting is allowed. An NxN PU split is also possible for a bottom level CU that cannot be further split into sub CUs.

6 | P a g e

3.2 Coding units (CUs) and coding blocks (CBs):

Figure 2. 64*64 CTBs split into CBs [54]The quad tree syntax of the CTU specifies the size and positions of its luma and croma CBs. The root of the quadtree is associated with the CTU. Hence, the size of the luma CTB is the largest supported size for a luma CB. The splitting of a CTU into luma and croma CBs is signaled jointly. One luma CB and ordinarily two croma CBs, together with associated syntax, form a coding unit (CU) as shown in Figure.3.

Figure 3. CUs split into CBs [54]

3.3 Prediction Modes :

7 | P a g e

3.3.1 Intra Prediction Modes : There are a total of 35 intra-prediction modes in

HEVC: planar (mode 0), DC (mode 1) (fig.4) and 33 angular modes (modes 2-34 in Figure 4). DC intra-prediction is the simplest mode in HEVC. All PU pixels are set equal to the mean value of all available neighbouring pixels. Planar intra-prediction is the most computationally expensive. It is a two- dimensional linear interpolation. Angular intra-prediction modes 2-34 are linear interpolations of pixel values in the corresponding directions. Vertical intra-prediction (modes 18- 34) is an updown interpolation of neighbouring pixel values. Also, intra prediction can be done at different block sizes, ranging from 4 X 4 to 64 X 64.

Fig 4: Modes and directional orientations for intra picture prediction [9]

3.3.2 Inter Prediction : Each PU is predicted from image data in one or two

reference pictures (before or after the current picture in display order), using motion compensated prediction. Transform and Quantization Any residual data remaining after prediction is transformed using a block transform based on the integer Discrete Cosine Transform (DCT) [4]. Only for 4x4 intra luma, a transform based on Discrete Sine Transform (DST) is used. One or more block transforms of size 32x32, 16x16, 8x8 and 4x4 are applied to residual data in each CU. Then the transformed data is quantized.

8 | P a g e

3.3.3 Motion compensation: Quarter-sample precision is used for the MVs and 7-

tap or 8-tap filters are used for interpolation of fractional-sample positions (compared to six-tap filtering of half-sample positions followed by linear interpolation for quarter-sample positions in H.264/MPEG-4 AVC). Similar to H.264/MPEG-4 AVC, multiple reference pictures are used as shown in Figure.5. For each PB, either one or two motion vectors can be transmitted, resulting either in unipredictive or bipredictive coding, respectively. As in H.264/MPEG-4 AVC, a scaling and offset operation may be applied to the prediction signal(s) in a manner known as weighted prediction

figure5. Concept of multi-frame motion-compensated prediction [5]

3.3.4 Entropy coding: Context adaptive binary arithmetic coding (CABAC) is used

for entropy coding. This is similar to the CABAC scheme in H.264/MPEG-4 AVC, but has undergone several improvements to improve its throughput speed (especially for parallel-processing architectures) and its compression performance, and to reduce its context memory requirements.

3.3.5 In-loop deblocking filtering: A deblocking filter similar to the one used in

H.264/MPEG-4 AVC is operated within the inter picture prediction loop.

9 | P a g e

However, the design is simplified in regard to its decision-making and filtering processes, and is made more friendly to parallel processing.

3.3.6 Sample adaptive offset (SAO): A nonlinear amplitude mapping is introduced

within the inter picture prediction loop after the deblocking filter. Its goal is to better reconstruct the original signal amplitudes by using a look-up table that is described by a few additional parameters that can be determined by histogram analysis at the encoder side.

4. Overview of WebP:

10 | P a g e

figure 6: Block Diagram of WebP Encoder [21]

WebP is an image format employing both lossy and lossless compression[19]. It is currently developed by Google, based on technology acquired with the purchase of On2 Technologies. As a derivative of the VP8 video format, it is a sister project to the WebM multimedia container format. WebP-related software is released under a BSD license. 

The format was first announced in 2010 as a new open standard for lossily compressed true-color graphics on the web, producing smaller files of comparable image quality to the older JPEG scheme [19]. According to Google's measurements, a conversion from PNG to WebP results in a 45% reduction in file size when starting with PNGs found on the web, and a 28% reduction compared to PNGs that are recompressed with pngcrush and pngout. [19]

11 | P a g e

Google has proposed using WebP for animated images as an alternative to the popular GIF format, citing the advantages of 24-bit color with transparency, combining frames with lossy and lossless compression in the same animation, and as well as support for seeking to specific frames. Google reports a 64% reduction in file size for images converted from animated GIFs to lossy WebP, and a 19% reduction when converted to lossless WebP. [19]

4.1 Prediction Techniques

WebP's lossy compression uses the same methodology as VP8 for predicting (video) frames. VP8 is based on block prediction and like any block-based codec, VP8 divides the frame into smaller segments called macro-blocks. It has two prediction modes : Intra prediction uses data within a single video frame and Inter prediction uses data from previously encoded frame.

4.1.1 Intra Prediction: WebP has three types of blocks:

4x4 luma 16x16 luma 8x8 chroma

Four common intra prediction modes used by these blocks are:H_PRED (horizontal prediction): Fills each column of the block with a copy of the left column, L. V_PRED (vertical prediction) : Fills each row of the block with a copy of the above row, A.DC_PRED (DC prediction): Fills the block with a single value using the average of the pixels in the row above A and the column to the left of L[16].TM PRED (True Motion prediction): In addition to the row A and column L, TM_PRED uses the pixel C above and to the left of the block. Horizontal differences between pixels in A and vertical differences between pixels in L are propagated (starting from C) to form the prediction block.

12 | P a g e

For 4x4 luma blocks, there are six additional intra modes corresponding to predicting pixels in different directions. As mentioned above, the TM_PRED mode is unique to VP8. Fig. 7 uses a 4x4 block as example to illustrate how the TM_PRED mode works:

In Fig. 7, C, A and L represent reconstructed pixel values from previously coded blocks, and Xoo through X33 represent predicted values for the current block. TM_PRED uses the followingequation to calculate Xij = Li + Aj - C (i,j=0, 1, 2, 3). The TM PRED prediction mode for 8x8 and 16x16 blocks works in a similar fashion. Among all the intra prediction modes, TM PRED is one of the more frequently used modes in VP8. For natural video sequences, it is typically used by 20% to 45% of allintra coded blocks. Together, these intra prediction modes help VP8 to achieve high compression efficiency, especially for key frames, which can only use intra modes.

figure 7: Illustration of intra prediction mode TM_PRED [16]

4.1.2 Inter prediction Modes: Inter prediction modes are used on inter

frames (non-keyframes). For any VP8 inter frame, there are typically three previously coded reference frames that can be used for prediction.

13 | P a g e

A typical inter prediction block is constructed using a motion vector to copy a block from one of the three frames. The motion vector points to the location of a pixel block to be copied. In video compression schemes, a good portion of the bits is spent on encoding motion vectors; the portion can be especially large for video encoded at lower data rates. VP8 provides efficient motion vector coding by reusing motion vectors from neighboring macro-blocks. For example, the prediction modes "NEAREST' and "NEAR" make use of last and second-to-last, non-zero motion vectors from neighboring macro-blocks. These inter prediction modes can be used in combination with any of the three different reference frames.In addition, VP8 has a sophisticated, flexible inter prediction mode called SPLITMV. This mode was designed to enable flexible partitioning of a macro-block into sub-blocks to achieve better interprediction. SPLITMV is useful when objects within a macro-block have different motion characteristics. Within a macro-block coded using the SPLITMV mode, each sub-block can have its own motion vector. Similar to the strategy of reusing without transmitting motion vectors at the macro-block level, a sub-blockcan also use motion vectors from neighboring sub-blocks above or left of the current block without transmitting the motion vectors. This strategy is very flexible and can encode any shape of submacro-blockpartitioning. Fig. 8(a) shows an example of a macro-block with l6xI6 luma pixels that is partitioned to 16 4x4 blocks:In Fig. 8 (a), New represents a 4x4 bock coded with a new motion vector, and Left and Above represent a 4x4 block coded using the motion vector from the left and above, respectively. This example effectively partitions the 16xI6 macro-block into three different segments with three different motion vectors (represented by 1, 2and 3), as seen in Fig. 8 (b).

14 | P a g e

figure 8: Illustration of VP8 inter prediction mode SPLITMV [16]

4.2 Adaptive Loop Filtering:

Loop filtering is a process of removing blocking artifacts introduced by quantization of the DCT coefficients from block transforms. VPS brings several loop-filtering innovations that speed up decoding by not applying any loop filter at all in some situations. VPS also supports a method of implicit segmentation where different loop filter strengths can be applied for different parts of the image, according to the prediction modes or reference frames used to encode each macro-block. For example it would be possible to apply stronger filtering to intra-coded blocks and at the same time specify that inter coded blocks that use the Golden Frame as a reference and are coded using a (0,0) motion vectorshould use a weaker filter. The choice of loop filter strengths in a variety of situations is fully adjustable on a frame-by-frame basis, so the encoder can adapt the filtering strategy in order to get the best possible results. In addition, similar to the region-based adaptive quantization in section 3, VPS supports the adjustment of loop filter strength for each segment. Most symbol values in VPS are binarized into a series of Boolean values using a pseudo Huffman Tree scheme. In such a scheme, a binary tree is first created for a set of symbols similarly to how a Huffman coding tree is created, and any symbol in the set can be represented by a series of binary

15 | P a g e

values generated by traversing the binary tree from the root node to the corresponding leaf node. Each non-leaf node in the binary tree has a probability assigned based on the likelihood of taking the left (0) branch for traversing. Through such a binarization scheme and storing data in pseudo Huffman tree structures, the encoding/decoding style in VP8 is very consistent for all the values in the bitstream, such as macro-block coding modes, reference frame types, motion vectors, quantized coefficients, and so on. Such a scheme improves module reusability in both hardware and software implementations of VPS entropy encoder or decoder.

4.3 Entropy Coding and Data Partitioning: Except for very few header bits

that are coded directly as raw values, the majority of compressed VPS data values are coded using a boolean arithmetic coder. The boolean arithmetic coder encodes one boolean value (0/1) at a time. It is used to losslessly compress a sequence of bools for which the probability of their being 0 or 1 can be well-estimated.

5. Performance comparison metrics

MSE and PSNR for an NxM pixel image are defined in equations 1 and 2 where O is the original image and R is the reconstructed image. M and N are the width and height of an image and ‘L’ is the maximum pixel value in the NxM pixel image.

16 | P a g e

Bjøntegaard-Delta Bit-Rate Measurements

As rate-distortion (R-D) performance assessment [14], Bjøntegaard-Delta bit-rate (BD-BR) measurement method is used for calculating average bit-rate differences between R-D curves for the same objective quality (e.g., for the same PSNRYUV values), where negative BD-BR values indicate actual bit-rate savings. As part of this project BD-BR performance metric will be used to determine bit-rate savings.

6. Implementation

For comparison purpose, open-source implementations of the reviewed codecs will be used. HEVC compression efficiency will be measured with the HM Test Model [12]. WebP is downloaded from [19].The rate distortion is compared in HEVC and WebP for MSP is done by plotting graphs for PSNR and BD rate. The implementation complexity will be evaluated based on the encoding time.

7. Test Softwares/Configurations

1) HM16.5

2) libwebp-0.4.3

17 | P a g e

3) ImageMagick-6.9.1

4)Intel Core i5 processor,1 Tb hard disk, 8GB RAM

8. Reference

[1] Joint Collaborative Team On Video Coding Information website- http://www.itu.int/en/ITU-T/studygroups/2013-2016/16/Pages/video/jctvc.aspx

[2] H.261: Video Codec for Audiovisual Services at px64 kbit/s,” http://www.itu.int/rec/T-REC-H.261-199303-I/en”

[3] G. J. Sullivan et al, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Transactions Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1649-1668, Dec. 2012.

[4] N. Ahmed , T. Natarajan and K.R. Rao, “Discrete Cosine Transform”, IEEE Transactions on Computers, Vol. C-23, pp. 90-93, Jan. 1974.

[5] P.K Ranjan, D. Pacharla, B. Ravindran and D. Mani "Quality evaluation of HEVC Main Still Picture with limited coding tree depth and intra modes", Advances in Computing, Communications and Informatics, New Delhi.[6] S. Bultje and M. Frost ,Access website http://www.webmproject.org/vp9/ PPT on “WebM and the new Open Video Codec”.[7] M. Budagavi and V. Sze, http://www.rle.mit.edu/eems/wp-content/uploads/2014/06/H.265-HEVC-Tutorial-2014-ISCAS.pdf, " Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial)".[8] http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html,S.C

18 | P a g e

Kodpadi ,"Comparative study of Intra Frame Coding efficiency in HEVC and VP9" EE5359, UTA, spring 2014[9] J. Bankoski et al, “Towards a Next Generation Open source Video Codec” SPIE Vol. 8666 Page 2, Dec. 2013.[10] D. Grois et al, “Performance Comparison of H.265/ MPEG-HEVC, VP9, and H.264/MPEGAVC Encoders”, IEEE PCS 2013, pp 394-397, San José, CA, USA, Dec 8-11, 2013[11] M.P. Sharabayko et al, "Intra Compression Efficiency in VP9 and HEVC" Applied Mathematical Sciences, Vol. 7, no. 137, pp.6803 – 6824, Hikari Ltd, 2013[12] HM Reference Software- https://hevc.hhi.fraunhofer.de/HM-doc/[13]

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf [14] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves”, ITU-T Q.6/SG16 VCEG 13th Meeting, Document VCEG-M33, Austin, USA, Apr. 2001.[15]F. Liang, X. Peng and J. Xu, "A light weight HEVC Encoder for Image Coding" MSRA-MOE joint key lab, Univ. of Sci and Technology of China, Hefei China [16] J. Bankoski, P. Wilkins and Xu Yaowu "Technical Overview of VP8,an open source video codec for the web", International conference on Multimedia and Expo 2011,pages:1-6[17] T. Nguyen and D. Marpe, "Objective Performance Evaluation of the HEVC Main Still Picture Profile" IEEE Transactions on circuits and systems for video technology ,15 September 2014 page:1[18] “The WebM Project.” [Online]. Available: http://www.webmproject.org/[19] “WebP Google Developers.” [Online]. Available: http://code.google. com/speed/webp/

19 | P a g e

[20] “Kodak Lossless True Color Image Suite.” [Online]. Available: http://r0k.us/graphics/kodak/[21]P.K. Bansal, M.N. Shukla and A.S. Motra, "VP8 Encoder-Cost effective implementation", SoftCOM,2012 pages(1-6)[22] Z. Xiong et.al, “A comparative study of DCT- and wavelet-based image coding,” IEEE Transactions on Circuits and Systems for Video Tech., vol.9, pp. 692-695, Aug. 1999.[23] Visual studio download for students for free- www.dreamspark.com

[24] Tortoise SVN download- http://tortoisesvn.net/downloads.html [25]MPL Website-http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html[26] K.R. Rao, D.N. Kim and J.J. Hwang, “Video Coding Standards: AVS China, H.264/MPEG-4 Part 10, HEVC, VP6, DIRAC and VC-1”, Springer, 2014.[27] http://www.imagemagick.org/script/install-source.php : program for converting to YUV format.[28] V. Sze ,M. Budagavi and G. J. Sullivan (Editors), "High Efficiency Video Coding (HEVC): Algorithms and Architectures," Springer,2014 [29] Iain Richardson/Vcodex.com “HEVC An Introduction to High Efficiency Video Coding” 2013[30] Iain Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002.

20 | P a g e