COMPLEXITY REDUCTION OF H.264 USING PARALLEL PROGRAMMING
By Sudeep GangavatiDepartment of Electrical Engineering
University of Texas at Arlington
Supervisor : Dr.K.R.Rao
Outline Introduction to video compression Why H.264 Overview of H.264 Motivation Possible approaches Related work Theoretical estimation Proposed approach Parallel computing NVIDIA GPUs and CUDA Programming Model Complexity reduction using CUDA Results Conclusion and future work
Introduction to video compression Video codec: A software or a hardware device that
can compress and decompress Need for compression : Limited bandwidth and limited
storage space. Several codecs : H.264, VP8, AVS China, Dirac etc.
Figure 1 Forecast of mobile data usage
Why H.264 ?H.264/MPEG-4 part 10 or AVC (Advanced Video
Coding) standardized by ITU-T VCEG and MPEG in 2004.
Approximately 50% bit-rate reductions over MPEG-2.
Most widely used standard.Built on the concepts of earlier standards like
MPEG-2.Substantial compression efficiency.Network friendly data representation. Improved error resiliency toolsSupports various applications
Overview of H.264There are two parts:
◦ Encoder : Carries out prediction, transform, quantization and encoding processes to produce a H.264 bit-stream.
◦ Decoder: Carries out the decoding, inverse transform, inverse quantization to reconstruct the earlier encoded video.
H.264 encoder
H.264 decoder
Intra predictionExploit spatial redundancies9 directional modes for prediction for
4 x 4 luma blocks4 modes for 16 x 16 luma blocks4 modes for 8 x 8 chroma blocks
Intra prediction9 modes for 4 x 4 luma block
4 modes for 16 x 16 luma blocks
Inter predictionExploits temporal redundancyInvolves prediction from one or more
previous frames called reference frames
Motion estimation and compensationMotion estimation and compensation
is a process of finding matching blockMotion search is performed.Motion vectors are obtained that
provide the displacement in the block.
Transform, Quantization and EncodingPredicted values are then
transformed.H.264 employs integer transform,
basically rough approximation of DCTAfter transform, the values are
quantized for compressionEntropy encoding : CAVLC / CABAC
Motivation
90%
5%
2% 3%
Motion EstimationTransform and quantiza-tionIntra PredictionVLC Encoding and others
Performed a time profiling on H.264 and obtained :
Motion estimation takes more time than any other module in H.264
Need to reduce this time by efficient implementation without sacrificing video quality and bitrate.
With reduced motion estimation time, the total time for encoding is reduced.
Possible approachesEncoder optimization Levels :
◦ Algorithmic Level : Develop new algorithms similar to
Three step algorithm, fast mode decision algorithm etc.◦ Compiler Level : Efficient programming ◦ Implementation Level: Using parallel
programming using CUDA, OpenMP , utilize underlying hardware etc.
Related workAuthor Features Advantages Disadvantages1. Chan et.al [41] Considers pyramid
algorithm for the motion estimation
Consider motion vector predicted to calculate SAD.
1.Video quality degradation2.RD performance is not considered.
2.Lee et.al [40] Multi-pass motion estimation. Generates local and global SADs in the first and second passes. Fast ME Search algorithm is used.
6 times speed up achieved compared to standard implementation.
1.Focus only on speed, not on rate and distortion.2. Threads are invoked for pixels3.Video resolution limit the thread creation
3.Rodriguez et.al [42]
Considers tree structured motion estimation algorithm
Three sequential steps 1. SAD Calculation 2.Uses binary reduction algorithm 3. Cost reduction
1.Implementation results in higher bitrate. 2.RD performance is not shown.
4. Cheng et.al [44] Based on simplified unsymmetrical multi-hexagon search. Divide into tiles.
3x speed up. Thread created for each tile.
Penalty in video quality
5.NVIDIA Encoder Searching algorithm is not disclosed. No documentation on internal details.
Provides 4 times speed up. Very good visual quality.
Fixed search range,
Issues with previous workFocus only on achievable speed up.Does not consider the methods to
decrease the bitrateDoes not consider techniques to
maintain video qualityThread creation overhead and
limitations in some approaches.
Theoretical estimation by Amdahl`s Law [43] We use this law to find out maximum achievable speed up
Widely used in parallel computing to predict theoretical maximum speed up using multiple processors.
Amdahl`s law states that if P is the proportion of a program that can be made parallel and (1-P) is the proportion that cannot be parallelized, then maximum speedup that can be achieved by using N processors is
Using Amdahl`s LawApproximation of speed up achieved upon
parallelizing a portion of the code◦ P: parallelized portion◦ N: Number of processor cores
In the encoder code, motion estimation accounts to approximately 2/3rd of the code .
Applying the law the maximum speedup that can be achieved in our case is 2.2 times or 55% time reduction.
Proposed workWe propose the following to address the problem :
◦ Using CUDA for parallel implementation for faster calculation of SAD (sum of absolute differences) and use one thread per block instead of one thread per pixel to address the thread creation overhead and limitation.
◦ Use a better search algorithm to maintain the video quality
◦ Combine SAD cost values and save the bitrate
The above methods address all the issues mentioned earlier
Along with the above, we utilize shared and texture memory of the GPU that reduces the global memory references and provides faster memory access.
Parallel ComputingMulti-core and many-core processors
improve the efficiency by parallel processing
Parallel processing provides significant improvement
Techniques to program software on multiple core processors: ◦ Data Parallelism◦ Task parallelism
Parallel ComputingData Parallelism
◦Split the large data set into smaller parts and execute them in parallel. After the execution, the data are grouped
Parallel ComputingTask Parallelism
◦Distribute threads to different processors
◦Data could be common ◦May execute same or different code
NVIDIA GPU And CUDA Programming Model
NVIDIA pioneered the Graphics Processing Units (GPU) Technology. First GPU: GeForce256 in 1999, had 128 MB of graphics memory.
GPUs, consisting of many core processors, are used in applications requiring high amounts of computation.
CPU-GPU Heterogeneous Model
Host-Device Connection
Compute Unified Device Architecture (CUDA)
NVIDIA introduced CUDA in 2006. Programming model that make
programs run on GPU.The serial portions of our program
written in C/C++ functions.Parallel portions are written as GPU
kernels.C/C++ functions execute on CPU,
kernels sent to GPU for processing.
Problem decompositionSerial C functions run on CPU CUDA Kernels run on GPU
Hardware ArchitectureMain element : Stream multiprocessor (SM)GT550M series has 2 SMs
Each SM has 48 cores
Each SM is capable ofexecuting 1536 threads
Total of 3072 threads running in parallel
ThreadingThreads are grouped into blocks
Blocks are grouped into grids
All threads within a block execute on the same SM
Complexity reduction using CUDA
Motion estimation: Process of finding the best matching block.
Complexity reduction using CUDATo find best matching block, search is
done in the search window (or region).Search provides the best matching
block by computing the difference i.e. it obtains sum of absolute difference (SAD).
SAD (dx, dy) = Search through search range of 8,16 or maximum 32 Select the block with least SAD. Larger the block size, more the computations
Complexity reduction using CUDA
1 1
1 |),(),(|Nx
xm
Ny
ynkk dyndxmInmI
A 352 x 288 frame
Standard algorithmDivide the block into 16 x 16 Again, further divide it into
Subblock of 8 x 8 . Search through the search areaCompute SAD obtain MVs
Our approach• Main idea is to:
– Minimize memory references and Memory transfer– Make use of shared memory and texture memory– Use single thread to compute SAD for single block– Make thread block creation dependent on the frame size for scalability– large number of threads are invoked that run in parallel and each block of thread consists of 396 threads that compute SADs of 396 - 8 x 8 blocks
SAD mapping to threads
Blocks 352 x 288 : (352/8) * (288 /8) = 1584 blocks that are to be computed for SAD.Total thread blocks = 4. Each block with 396 threads.This makes the approach scalable. For a video with higher resolution, like 704 x 480 ( 4SIF) or 704 x 576 (4CIF), we can create 16 blocks each with 396. So the number of threads created is dependent on video resolution.
Performance enhancementsWe consider Rate-distortion (RD) criteria
and employ following techniques: ◦ To minimize bitrate:
Calculate the cost for smaller sub blocks of 8 x 8 and combine 4 of these and form a single cost for 16 x 16 block.
◦ To enhance video quality: Incorporate exhaustive full search algorithm that goes
on to calculate the matching block for the entire frame without skipping any blocks as opposed to other algorithms. Previous studies [30] show that, this algorithm provides the best performance. Though it is highly computational, this is used keeping video quality in mind.
Memory access Memory access from texture memory to shared memory
Memcpy API to move data into the Array we allocated:cudaMemcpyToArray(a_before_dilated, // array pointer0, // array offset width0, // array offset heighth_before_dilated, // source width*height*sizeof(uchar1), // size in bytescudaMemcpyHostToDevice); // type of memcpy
Texture Memory
Shared memory
Performance Metrics
Test Sequences
Results
Akiyo Carphone News Container Foreman0100200300400500600700
Comparison of average encoding time for QCIF sequences
Reference SoftwareOptimized SoftwareNVIDIA Encoder
QCIF Video Sequences
Time in se
conds
The CPU-GPU implemented encoder performs better than the CPU-only encoder. But falls short when compared to NVIDIA Encoder. This is due to the fact that NVIDIA Encoder is heavily optimized at all levels of H.264 and not just motion estimation. NVIDIA has not released the type of searching algorithm it is using as well. Use of appropriate algorithm for motion search significantly changes the performance of quality, bitrate and speed.The theoretical speed up was about 2.2-2.5 times. From results, we achieve approx. 2 times speed up. This can be attributed to the other factors like the time it takes for load and store operations for functions , transfer of control to the GPU, memory transfer and references for operations that we have not considered and also other H.264 calculations etc.
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PSNR
(dB)
32
34
36
38
40
42
44
Reference softwareOptimized softwareNVIDIA Encoder
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PSNR
(dB)
30
32
34
36
38
40
42
Reference softwareOptimized softwareNVIDIA Encoder
Results for QCIF video sequences
PSNR vs. Bitrate for Akiyo sequencePSNR vs. Bitrate for Carphone sequence
PSNR vs. Bitrate for Container sequence PSNR vs. Bitrate for Foreman
sequence
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PSNR
(dB)
32
34
36
38
40
42
44
46
48
Reference softwareOptimized softwareNVIDIA Encoder
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PSNR
(dB)
32
34
36
38
40
42
Reference softwareOptimized softwareNVIDIA Encoder
Results
Akiyo Carphone News Container Foreman0.82
0.84
0.86
0.88
0.9
0.92
0.94
0.96
Reference SoftwareOptimized softwareNVIDIA Encoder
QCIF Sequences
SSIM
SSIM provides the structural similarity between the input and output videos. Ranges from 0.0 to 1.0. 0 is the least quality video. 1 is the highest quality video
Results
Akiyo Carphone News Container Foreman0500
1000150020002500
Comparison of average encoding time for CIF sequences
Reference SoftwareOptimized softwareNVIDIA Encoder
CIF Video sequences
Time in se
conds
Similar behavior is observed in case of CIF video sequences.
Results
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PS
NR
(dB)
36
38
40
42
44
46
48
Reference software Optimized softwareNVIDIA Encoder
News
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PS
NR
(dB)
37
38
39
40
41
42
43
44
45
Reference SoftwareOptimized softwareNVIDIA Encoder
Carphone
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PS
NR
(dB)
38
40
42
44
46
48
50
Reference SoftwareOptimized softwareNVIDIA Encoder
Container
Bitrate
200 kbps 400 kbps 600 kbps 800 kbps 1000 kbps
PS
NR
(dB)
34
36
38
40
42
44
Reference softwareOptimized softwareNVIDIA Encoder
Results
SSIM values for our optimized software and NVIDIA encoder are very close.
Akiyo Carphone News Container Foreman0.910.920.930.940.950.960.97
Reference softwareOptimized softwareNVIDIA Encoder
CIF Sequences
SSIM
ConclusionsNearly 50% reduction in encoding time on various
sequences close to the theoretical estimation.Less degradation in video quality is observed.Less bitrate is obtained by uniquely combining
the SAD costs of sub blocks into cost of larger macroblock
SSIM, Bitrate, PSNR are close to the values obtained without optimizations
Achieved data parallelismWith little modification in the code, the approach
is actually scalable to better hardware and increased video resolution
LimitationsAs the threads work in parallel, in case
when the sum of SADs till kth row (k<8) exceeds the current SAD, then there is no need to compute further. But due to the concurrent processing, no best SAD is available until the thread is done calculating.
Search range cannot be modified while encoding is in progress.
Since this is a hardware implementation, the performance largely depends on the type of hardware used.
Future work Other operations in H.264 like filtering, entropy encoding can be
parallelized.
Block dependencies are not considered in this approach. This could be challenging but results in higher compression efficiency
Different profiles like High and Main profiles can be used for implementation
Different motion estimation algorithm can be implemented in parallel and later on incorporated into H.264
CUDA can be applied to HEVC, next generation video coding standard, successor to H.264. HEVC is known be more complex than H.264.
Thank You
References[1] I.E. Richardson, “The H.264 advanced video compression standard”, 2nd Edition, Wiley, 2010.[2] S. Kwon, A. Tamhankar, and K.R. Rao, “Overview of H.264/MPEG-4 part 10”, Journal of Visual Communication and Image Representation, vol. 17, no.2, pp. 186-216, April 2006.[3] Draft ITU-T Recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264/ISO/IEC 14 496-10 AVC), Mar. 2003. [4] G. Sullivan, “Overview of international video coding standards (preceding H.264/AVC)”, ITU-T VICA Workshop, July 2005.[5] T.Wiegand, et al “Overview of the H.264/AVC video coding standard”, IEEE Transactions on. Circuits and Sytems for Video Technology, vol.13, pp 560–576, July 2003. [6] M. Jafari and S. Kasaei, “Fast intra- and inter-prediction mode decision in H.264 advanced video coding”, International Journal of Computer Science and Network Security, vol.8, no.5, pp. 1-6, May 2008.[7] W. Chen and H. Hang, “H.264/AVC motion estimation implementation on Compute Unified Device Architecture (CUDA)”, 2008 IEEE International Conference on Multimedia and Expo, pp. 697-700, 26 April 2008.
References[8] Y. He, I. Ahmad and M. Liou, “ A software-based MPEG-4 video encoder using parallel processing”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 8, no.7, pp. 909-920, November 1998.[9] D. Marpe, T. Wiegand and G. J. Sullivan, “The H.264/MPEG-4 AVC standard and its applications”, IEEE Communications Magazine, vol. 44, pp. 134-143, Aug. 2006.[10] Z.Wang, et al, “ Image quality assessment : From error visibility to structural similarity”, IEEE Transactions on Image Processing, vol 13. Pp. 600-612, April 2004.[11] G. Sullivan, P. Topiwala, and A. Luthra, “The H.264/AVC advanced video coding standard:overview and introduction to the fidelity range extensions” SPIE Conference on Applications of Digital Image Processing XXVII, vol. 5558, pp. 53-74, 2004.[12] A. Puri, X. Chen and A. Luthra, “Video coding using the H.264/MPEG-4 AVC compression standard”, Signal Processing:Image Communication , vol.19 793–849, 2004.[13] K.R. Rao and P. Yip, Discrete cosine transform, Academic Press, 1990.[14] H. Yadav, “Optimization of the deblocking filter in H.264 codec for real time implementation” M.S. Thesis, E.E. Dept, UT Arlington, 2006.[15] https://computing.llnl.gov/tutorials/parallel_comp/, Introduction to parallel computing.
References[16] J. Kim, et al, “Complexity reduction algorithm for intra mode selection in H.264/AVC video coding” J. Blanc-Talon et al. (Eds.): ACIVS 2006, LNCS 4179, pp. 454 – 465, Springer-Verlag, Berlin, Heidelberg, 2006.[17] B.Jung, et al, “Adaptive slice-level parallelism for real-time H.264/AVC encoder with fast inter mode selection”, Multimedia Systems and Applications X, edited by S. Rahardja, J.W.Kim and J.Luo, Proc. of SPIE, vol 6777, 67770J, 2007.[18] S.Ge, X.tian and Y. - K. Chen, “Efficient multithreading implementation of H.264 encoder on Intel Hyper-threading architectures”, ICICS-PCM 2003.[19] T. Rauber and G.Runger, “Parallel programming for multicore and cluster systems”, 2nd Edition, Wiley, 2008[20] D.Ailawadi, M.K.Mohapatra and A.Mittal, “Frame-based parallelization of MPEG-4 on Compute Unified Device Arcitecture(CUDA)”, IEEE Conference on Advanced Computing , pp 267-272 , 2010.
References[21] M. A. F. Rodriguez, “CUDA: Speeding up parallel computing”, International Journal of Computer Science and Security, November 2010.[22] NVIDIA, NVIDIA CUDA Programming Guide, Version 3.2, NVIDIA, September 2010.[23] “http://drdobbs.com/high-performance-computing/206900471” Jonathan Erickson, GPU Computing Isn’t Just About Graphics Anymore, Online Article, February 2008. [24] J. Nickolls and W. J. Dally,” The GPU Computing Era” , IEEE Computer Society Micro-IEEE, vol 30, Issue 2, pp . 56 - 69, April 2010.[25] M.Abdellah, “High performance Fourier volume rendering on graphics processing units”, M.S. Thesis, Systems and Bio-Medical Engineering Department, Cairo University, 2012.[26] J. Sanders and E. Kandrot, “CUDA by example: an introduction to general-purpose GPU programming” Addison-Wesley Professional, 2010.[27] NVIDIA, NVIDIA’s Next Generation CUDA Compute Architecture:Fermi, White Paper, Version 1.1, NVIDIA 2009.[28] NVIDIA, Best Programming Practices, 2009.[30] P.Kuhn, “Algorithms, complexity analysis and VLSI architectures for MPEG-4 motion estimation”, Kluwer Academic, 1999.
References[31] K.Shen and E.J. Delp, “ A spatial-temporal parallel approach for real time MPEG video compression”, Proc. of 25th International conference on parallel processing, pp. 100-107, 1996.[32] JM 16.0 software – http://iphome.hhi.de/suehring/tml/[33] JM Reference Software Manual –http://iphome.hhi.de/suehring/tml/JM Reference Software Manual (JVT-AE010).pdf[34] D. Han, A. Kulkarni and K.R. Rao, “Fast inter-prediction mode decision algorithm for H.264 video encoder”, ECTICON 2012, Cha Am, Thailand, May 2012.[35] S. Sun, et al, “A highly efficient parallel algorithm for H.264 encoder based on macro-block region partition”, Springer-Verlag, Berlin, Heidelberg, pp. 577–585, 2007.[36] Test sequences - http://trace.eas.asu.edu/yuv/[37] D.Kirk and W.-M. Hwu, “Programming massively parallel processors: A hands-on approach(Applications of GPU Computing series)”, Morgan Kauffman, 2010[38] Flynn`s Taxonomy, http://www.phy.ornl.gov/csep/ca/node11.html [39] T. Saxena, “Reducing the encoding time of H.264 Baseline profile using parallel programming”, M.S. Thesis, E.E. Dept, UT Arlington, 2012.[40] Lee et.al., “ Multi-pass and frame parallel algorithms of motion estimation of H.264/AVC for generic GPU”, Proc of International conference on multimedia and expo, Beijing,China, 2007.[41] Chan et.al ,“Parallelizing H.264 Motion Estimation Algorithm using CUDA”, MIT IAP 2009[42] Rodriguez et.al., “ Accelerating H.264 Inter Prediction in a GPU by using CUDA”, Proc. of the 2010 ICCE, pp 463-464, 2010[43] “Amdahl`s Law”, http://www.futurechips.org/thoughts-for-researchers/parallel-programming-gene-amdahl-said.html[44] Cheung et.al “ Video coding on Multi-core graphics processors”, IEEE Signal Processing Magazine, March 2010.
Appendix
CUDA Memory Model
GPU Hardware Specs
SSIM The difference with respect to other techniques mentioned previously such as
MSE or PSNR, is that these approaches estimate perceived errors on the other hand SSIM considers image degradation as perceived change in structural information. Structural information is the idea that the pixels have strong inter-dependencies especially when they are spatially close. These dependencies carry important information about the structure of the objects in the visual scene.
The SSIM metric is calculated on various windows of an image. The measure between two windows and of common size N×N is:
the average of μx ; the average of μy; the variance of σx; the variance of σy ; the covariance of and σxy; C1 and C2, two variables to stabilize the division with weak denominator; In order to evaluate the image quality this formula is applied only on luma. The
resultant SSIM index is a decimal value between -1 and 1, and value 1 is only reachable in the case of two identical sets of data. Typically it is calculated on window sizes of 8×8. The window can be displaced pixel-by-pixel on the image but the authors propose to use only a subgroup of the possible windows to reduce the complexity of the calculation.