Abstract An unequal error protection (UEP) techniquebased on motion information of video bitstreams compressedby H.264/AVC standard is proposed. Motion activities ofmacroblocks in a frame are analyzed, and those having higheffects on the video performance are extracted. Suitable for-ward error-correcting codes with different rates are con-structed according to the importance of macroblocks andframes. Simulation results show that the proposed techniquesignificantly improves the video quality, while maintaininga similar overall code rate in comparison with other UEPtechniques.
Unequal error protection (UEP) is a common technique usedin wireless video transmission systems, which protects dif-ferent parts of the bitstream based on their importance. InH.264/AVC, which is introduced as an efficient video com-pression standard, UEP can be conducted in correspondencewith distortion of frames. A distortion model for consecutiveframes based on the transmission bit rate is applied to deter-mine the importance of frames [1]. Frames having higherdistortion than the average value are protected more thanothers. This technique requires a delay for calculating thetotal distortion of consecutive frames.
H. D. Pham (B) · S. VafiSchool of Engineering and Information Technology,Charles Darwin University, Darwin, Australiae-mail: [email protected]
In H.264/AVC standard, a frame can be divided into sev-eral slices. An UEP technique is proposed by determiningdistortion depending on the number of slices lost duringtransmission (SLICE_DIST) [2]. The distortion of a sliceis estimated as the difference between the original and pre-dicted versions of the lost one. Different protection levels areevaluated by predetermined thresholds of slices’ distortion.As video sequences have different textures and motion lev-els, predetermined threshold values cannot properly respondto the allocation of protection levels of the video bitstream.
Alternatively, slices can be separated into groups accord-ing to their estimated distortions [3]. Those groups havingthe lowest impact on the distortion are ignored. Instead,ignored slices are replaced by redundant ones in order toimprove the resilience for important groups. Different for-ward error-correcting (FEC) code rates can be applied toprotect prioritized slices based on the estimation of distor-tion [4]. At decoder, corrected ones are used to update refer-ence frames in order to stop error propagation in consecutiveframes. Other techniques propose rate-distortion optimiza-tion (RDO) models [5,6]. In these techniques, different opti-mization schemes are conducted in order to allocate suitablebit rates for slices according to their estimated distortion.Error protection levels of slices are also determined basedon their types, positions and size of headers applied in videoframes (SLICE_PRIO) [7]. Delay is not generated for trans-mission systems by mentioned UEP techniques. However,as video quality is improved by increasing the number ofslices in a frame, the amount of inserted headers for recog-nizing their beginning in a video bitstream will significantlyrise. This consequently degrades the bandwidth usage of thetransmission system.
Human eyes are sensitive to the motion regions of avideo frame [8]. Hence, any error occurred in the motionareas becomes very considerable. In addition, the extent of
123
SIViP
such an error can appear in many frames, which signifi-cantly degrades the video quality. There are several tech-niques, which mitigate errors of the motion information prop-agated between consecutive frames. In one technique, thoseslices, which are highly corrupted due to an error propa-gated in consecutive frames, are protected more than oth-ers (ERR_PROP) [9]. This method requires a delay equal tothe length of several frames involved in the calculation ofthe error propagation. The importance of slices can be eval-uated by the difference of their luminance. Mean squarederror (MSE) [10], sum of absolute difference (SAD) [11], andmean absolute difference (MAD) [12] are three of the mostwell-known methods for calculating this difference. Sliceswith high MSE, SAD or MAD values demand higher pro-tection than others. The importance based on motion activi-ties can also be evaluated by their motion vector magnitudes(MV_MAGNITUDE) [13] or differences between the actualand predicted motion vectors [14,15]. In this case, motionvectors can be extracted during encoding. As simple calcula-tions are required in determining protection levels, these tech-niques are generally suited for video applications with lowcomplexity.
Recently, an UEP technique was proposed based on datapartitioning tool [16]. As the motion information of a slice ismainly located on partition A, the motion activity of a sliceis evaluated by the length of this partition (MOTION_INFO_SIZE). Therefore, protection levels are formed accordingto the length of partitions A. This method provides a goodsolution for protecting slices having high motion activities inthe expense of increasing the number of overheads [17].
Previous works have introduced different UEP techniquesbased on the motion information of video frames. How-ever, the movement of different sub-macroblocks betweenframes and their sizes was not considered. We foundthat the existence of an error in sub- macroblocks withlarge movement, i.e., long motion vector magnitude, gen-erates high distortion. This is confirmed by a significantreduction in correlation between actual and concealed sub-macroblocks [18]. In general, motion vector magnitudes ofsub-macroblocks are not proportional to their sizes [19].This concludes that motion activity of a sub-macroblock isnot only set by the motion vector magnitude, but also itssize. Indeed, it is determined by the product of the afore-mentioned parameters (motion vector magnitude and size ofthe sub-macroblock), which defines the motion energy of asub-macroblock.
In this paper, an UEP technique is proposed based onmotion energy of macroblocks. For this purpose, two meth-ods for determining a threshold, which separates the impor-tance of macroblocks, are presented. In the first method, thethreshold is calculated as the average of motion energy ofall macroblocks inside a frame. In the second method, itis obtained from the average of motion energy of the clos-
est neighboring macroblocks. UEP technique can be appliedfor video bitstreams at macroblock or frame levels. At mac-roblock level, macroblocks of a frame are grouped into high-and low-importance slices. Then, grouped macroblocks areunequally protected by reliable FEC codes. At frame level,the importance of a frame is determined based on the major-ity of macroblocks having the same importance. Simula-tion results show that with similar channel code rates, theUEP formed on the basis of motion energy is more effectivethan other well-known motion activity estimation techniques,while a similar number of overheads is applied.
This paper is organized as follows. Section 2 explains UEPtechniques at macroblock and frame levels based on motionenergy definition. Section 3 provides simulation results ofthe proposed UEP technique. Section 4 concludes the paperand suggests further works in this research.
2 Determining the importance of macroblocks based onmotion energy
In the H.264/AVC standard, a macroblock is constituted byits number of sub-macroblocks [20]. The motion vector ofa sub-macroblock represents its displacement between twoconsecutive frames [17]. Figure 1 illustrates the motion vec-tor of the sub-macroblock k at the frame i , which is predictedby the sub-macroblock k′ of the frame (i − 1). There aremany conventional concealment methods for macroblocks.In one method, when the content of a sub-macroblock islost, its motion vector is predicted by the motion vectorsof its neighbors. If the motion information of neighboringsub-macroblocks is lost, the content of the decoded sub-macroblock will be replaced by the one, which is similarlypositioned in the previous frame [20]. In Fig. 1, the concealedversion of the sub-macroblock k of the frame i is the sub-macroblock k′ of the frame (i − 1). In this case, the distancebetween sub-macroblocks k and k′ (d) equals the motionvector magnitude of the sub-macroblock k.
In order to determine distortion of the predicted block, i.e.,block k′, the block’s correlation with block k is examined.The correlation between two blocks with the size of M × Nis given by [20]:
Fig. 1 Motion vector of a block
123
SIViP
Fig. 2 Correlations betweenblocks with different distances aForeman, b Stefan
ρ =∑M−1
x=0∑N−1
y=0 Ik(x, y).Ik′(x, y)∑M−1
x=0∑N−1
y=0 [Ik(x, y)]2(1)
where Ik(x, y) and Ik′(x, y) are luminance values of the pix-els located at the position (x, y) of blocks k and k′, respec-tively. An analysis is conducted to determine the relation-ship between the correlation of blocks and their distance.It is conducted for 300 frames of QCIF Foreman and CIFStefan video sequences. In each frame, the correlation of arandom 4 × 4 block with others is calculated. The distanced is set no longer than 10 pixels in both X and Y dimen-sions. Figure 2 shows the correlations of random blocksversus the frame number and distance. It is shown that thecorrelation between blocks is indirectly related to their dis-tance. This means that distortion of the predicted block isincreased based on the amount of motion vector magni-tude.
It is also recognized that not only are motion vectormagnitudes important to determine the motion activity ofsub-macroblocks, but also sizes of sub-macroblocks [19].Hence, the motion vector magnitude and the size of a sub-macroblock are applied to evaluate its importance. This formsmotion energy (ME), which is defined as follows:
MEk � SBLCKk .MVk (2)
Table 1 Assigned values for different block sizes in ME calculation
Block size (pel. × pel.) Assigned value (SBLC K )
4 × 4 1
4 × 8 or 8 × 4 2
8 × 8 4
8 × 16 or 16 × 8 8
16 × 16 16
where SBCLKk is an assigned value for the size of the kth sub-macroblock. Table 1 describes values of SBLCK for differentblock sizes. MVk represents the motion vector magnitude ofthe sub-macroblock k.
2.1 Motion-energy-based UEP technique at macroblocklevel (ME_MB)
The motion energy of a macroblock is calculated as the sumof motion energies of its sub-macroblocks. This is given by:
MEMB =no_block∑
k=1
MEk (3)
where no_block is the number of sub-macroblocks inside amacroblock. In order to determine protection levels for mac-
123
SIViP
Fig. 3 Neighboring and examined macroblocks in a video frame
Table 2 Importance of macroblocks of the 79th frame of CIF Ste f ansequence
MB position Constituents of MB(number of sub-MB)
ME TH Importance(NB_AVG)
17 16 × 8 (2) 160.79 146.74 High
32 16 × 16 (1) 113.13 158.92 Low
134 16 × 8 (2) 244.46 302.81 Low
162 8 × 8 (3), 8 × 4 (2) 172.4 196.25 Low
183 16 × 16 (1) 162.5 114.78 High
229 4 × 4 (4), 4 × 8 (6) 421.6 387.35 High
265 16 × 16 (1) 97.32 139.77 Low
319 8 × 8 (3), 8 × 4 (2) 689.3 498.54 High
368 16 × 16 (1) 342.7 215.93 High
roblocks, a threshold value is used to separate their impor-tance. Two methods for calculating the threshold value areproposed. In the first method, the threshold is computed asthe average motion energy for all macroblocks in a frame(FR_AVG). In the second method, it is formed by averagingmotion energies of the nearest neighboring macroblocks ofthe examined one (NB_AVG). Figure 3 shows positions ofneighboring macroblocks in a frame. The number of neigh-boring macroblocks can vary depending on the position ofthe considered macroblock. Hence, the threshold value for amacroblock is given as follows:
TH =∑no_M B
n=1 MEMBn
no_M B(4)
where MBn is the nth macroblock of a frame. This equationcan be applied for FR_AVG, when no_MB is set as theoverall number of macroblocks in a frame. Macroblocks areevaluated as high importance, when their motion energies arehigher than the calculated threshold value.
Table 2 shows the motion energy and the importance ofmacroblocks for the 79th frame of the CIF Stefan videosequence. It is observed that a large sub-macroblock can bedetermined as a low-importance one. This situation occurs at
the 32nd and 265th macroblocks. Despite having the largestsize of these sub-macroblocks (16×16), their motion vectormagnitudes are small.1 Consequently, motion energies rele-vant to these motion vectors are not higher than the thresh-old value and they are evaluated as low-importance sub-macroblocks. Indeed, they belong to the smooth regions ofthe frame.
Separation of high- and low-importance macroblocks intogroups utilized for UEP technique can be done by FlexibleMacroblock Ordering (FMO) tool with explicit mode. Withthis tool, macroblocks with the same importance are groupedinto a slice. Constituents of each slice are recognized by theMacroblock Allocation map (MBAmap), which is insertedinto the beginning of each video frame. Slices are unequallyprotected by a suitable FEC channel code with different rates.
This UEP technique can efficiently protect differentregions of a frame. However, the size of the overhead isincreased by inserting MBAmap for each frame. In orderto prevent this overhead, a motion-energy-based UEP tech-nique applied at the frame level is introduced in the followingsection.
2.2 Motion-energy-based UEP technique at frame level(ME_FR)
Protection levels of video frames are evaluated by motionenergies of macroblocks calculated by Eq. 3. A macroblock isevaluated as a high-importance one, when its motion energyis greater than the threshold value (TH) obtained by Eq. 4.Then, the number of high- and low-importance macroblocks(no_hi_imp and no_lo_imp) is calculated. The importanceof a frame is determined based on the majority of mac-roblocks having the same importance.2
An analysis is conducted to verify the effect of motionenergy applied at macroblock and frame levels. Table 3shows the percentage of high- and low-importance parts(macroblocks and frames) in 300 frames of different videosequences. It is recognized that the percentage of high-importance parts evaluated at frame level are larger than thoseat macroblock level. This means that more macroblocks arehighly protected at frame level. The video performance pro-vided by the ME_FR technique is improved according to theincrease in high-importance parts of video bitstreams.
3 Simulation results
The performance of the proposed techniques is verified andcompared with other conventional motion-based UEP tech-
1 MV32 = 7.07, MV265 = 6.08.2 If no_hi_imp > no_lo_imp, the frame has high importance and viceversa.
123
SIViP
Table 3 Percentage of high-and low-importance parts ofdifferent video sequences
Video sequence Percentage of MBs (ME_MB) Percentage of frames (ME_FR)
High imp. Low imp. High imp. Low imp.
CIF Stefan 47.68 52.32 51.3 48.7
1080p Pedestrian area 52.8 47.2 53.62 46.38
1080p rush_hour 59.14 40.86 65.48 34.52
1080p riverbed 54.25 45.75 56.41 43.59
1080p bluesky 50.68 49.32 58.93 41.07
Fig. 4 PSNR results of 1080p Pedestrian_area video sequence
niques including MSE, SAD, MOTION_IN-FO_SIZE andMV_MAGNITUDE. It is also compared with recent UEPtechniques including SLICE_D-IST [2], SLICE_PRIO [7]and ERR_PROP [9]. JM 18.0 Reference Software is appliedfor encoding and decoding of the H.264/AVC video bitstream[21]. Various video sequences with different frame sizes andmotion activities are encoded as I P P P... format with therate of 30 frames per second. Three hundred (300) frames ofeach video sequence are used in the loop of 200 simulationsin order to compute the average peak signal-to-noise ratio(PSNR) for different video sequences.
This approach is applied at the physical layer, where chan-nel coding is used for protecting video signals from noise[22]. The Additive white Gaussian noise (AWGN) model isimplemented in simulation of random noise for video bit-streams modulated by the binary phase-shift keying (BPSK)technique. At the physical layer, a slice is mapped into apacket recognized by its header and added to the NetworkAbstraction Layer (NAL) Unit [20]. The protection of differ-ent packets is accomplished by product codes constituted bytwo cyclic Euclidean Geometry Low Density Parity Check(EG-LDPC) (63,37) codes with the length L = 1369 [22].Puncturing is conducted on the full-rate product code, i.e.,rate of 0.334, to unequally protect low-importance parts of
Fig. 5 PSNR results of 1080p rush_hour video sequence
video bitstreams. Zero padding is applied for packets to con-struct blocks with the length required for the cyclic EG-LDPC codes. Iterative decoding of cyclic EG codes withmaximum 100 iterations is implemented with sum -productalgorithm (SPA) [22]. Two different code rates (0.36 and 0.4)are set for P frames based on the importance of macroblocksor frames. Since I frames are more important, they havehigher protection than P frames [20]. In this case, only onecode rate (0.334) is applied. For EEP technique, the code rateis set to the average rate utilized in UEPs.3
3.1 Simulations of UEP techniques at macroblock level(M E_M B)
Figures 4 and 5 show the video performance in terms ofthe bit rate for different video sequences. Simulation resultswere obtained, when the energy per bit to noise (Eb/N0) isset at 5.2 dB. In both videos, the motion energy with thethreshold predicted by neighboring macroblocks (NB_AVG)
has the best performance. For 1080p Pedestrian_area, videoquality is improved by 0.96 and 1.30 dB in compari-son with SLICE_DIST and MV_MAGNITUDE techniques,
3 For all simulations, the code rate of EEP is set to 0.385.
123
SIViP
Table 4 PSNR results withdifferent Eb/N0
(a) E E P , (b)MV _M AG N I T U DE , (c)SL I C E_DI ST , (d)M E_M B(F R_AV G), (e)M E_M B(N B_AV G)
Eb/N0 (dB) 4 5 6 7 8 9
CIF Ste f an (a) 31.14 33.42 35.07 35.91 36.68 37.44
respectively. In 1080p rush_hour sequence, NB_AVG alsoprovides the best PSNR. With the bit rate of 1,500 kbps,it shows 3.2, 1.23 and 0.64 dB better performance thanMSE, MV_MAGNITUDE and SLI-CE_DIST techniques,respectively. These also show the percentage of overheadof different techniques compared with the EEP technique(ov). It is confirmed that the proposed technique applies anoverhead similar to other UEP techniques.
Table 4 shows video performance results with differentEb/N0s (from 4 to 9 dB). N B_AV G outperforms other tech-niques in all video sequences. Simulation results confirm thatUEP based on motion energy extracts high-importance mac-roblocks better than other techniques.
3.2 Simulations of UEP techniques at frame level(M E_F R)
Figures 6 and 7 show video performance of 1080p pedes-tr ian_area and rush_hour sequences at different Eb/N0s.For low Eb/N0 (less than 6 dB), PSNRs of N B_AV Goutperform other UEP techniques. For 1080p pedestri-an_area sequence, their PSNRs provide 1.27, 1.4 and
Fig. 6 PSNR results of 1080p Pedestrian_area video sequence
1.8 dB better than SL I C E_DI ST, SL I C E_P RI O andE R R_P RO P at Eb/N0 = 5.2 dB. For Eb/N0 > 6 d B,PSNRs of different techniques are gradually close to eachother. In 1080p rush_hour , simulation results confirm thatM E_F R(N B_AV G) gives the best video quality. These
123
SIViP
Fig. 7 PSNR results of 1080p rush_hour video sequence
results were obtained, when similar overheads are appliedfor video bitstreams. Table 5 shows the performance of theproposed method at different bit rates. It is observed thatN B_AV G outperforms other UEP techniques in all videosequences. It is proved that the proposed technique providesthe best video performance at different Eb/N0s and bit rates.
3.3 Comparison of UEP techniques based on FMO andnon-FMO tool
Table 6 shows the PSNR difference of video sequencesfor various Eb/N0s based on the motion energy deter-mined by N B_AV G technique at macroblock and framelevels (M E_F R(N B_AV G) and M E_M B(N B_AV G)).In all video sequences, video performance obtained byM E_F R(N B_AV G) is better than M E_M B(N B_AV -G). Table 7 shows overhead differences (in percentage)between previous motion-based and M E_F R(N B_AV G)
techniques. It is observed that UEP with FMO tool (M E_M B(N B_AV G) and MV _M AG N I -T U DE) has overheadsmore than others.
In summary, the motion-energy-based UEP techniqueapplied at frame level outperforms the one at macroblocklevel, while it utilizes a low overhead for protecting the videobitstream.
4 Conclusions and future works
This paper proposed an efficient unequal error protectiontechnique for H.264/AVC video bitstreams. The importanceof macroblocks is determined by their motion energy, whichis calculated as the product of motion vector magnitudesand block sizes. This UEP technique can be applied at mac-roblock or frame levels. High-importance macroblocks andframes are protected more than others. Simulation resultsconfirmed that the proposed technique outperforms otherconventional ones, while a similar overhead is applied forthe video bitstream. In future work, motion energy will beutilized for multi-level error protection technique in order tooptimize the overall code rate and video performance.
References
1. Wang, S., Ma, S., Wang, S., Zhao, D., Gao, W.: Rate-GOP basedrate control for high efficiency video coding. Sel. Top. SignalProcess. IEEE J. 1(99), 1–11 (2013)
2. Wu, Y., Kumar, S., Hu, F., Zhu, Y., Matyjas, J.: Cross-layer for-ward error correction scheme using raptor and RCPC codes for pri-oritized video transmission over wireless channels. Circuits Syst.Video Technol. IEEE Trans. 1(99), 1–14 (2014)
3. Luo, Z., Song, L., Zheng, S., Ling, N.: Raptor codes based unequalprotection for compressed video according to packet priority. Mul-timed. IEEE Trans., 1–12 (2013)
5. Hsia, S.C., Hsu, W.C., Wu, S.R.: A fast rate-distortion optimizationalgorithm for H.264/AVC codec. J. Signal Image Video Process.7(5), 939–949 (2013)
6. He, J., Yang, F.: High-speed implementation of rate-distortion opti-mized quantization for H.264/AVC. J. Signal Image Video Process.,1–9 (2013)
8. Gibson, J.: The Mobile Communications Handbook, 2nd edn. IEEEPress (1999)
9. Lin, C., Tillo, T., Zhao, Y., Jeon, B.: Multiple description codingfor H.264/AVC with redundancy allocation at macro block level.Circuits Syst. Video Technol. IEEE Trans. 21(5), 589–600 (2011)
10. Qu, Q., Pei, Y., Modestino, J.: An adaptive motion-based unequalerror protection approach for real-time video transport over wire-less IP networks. Multimedia IEEE Trans. 8(5), 1033–1044 (2006)
11. Obando, M., Freitas, W., Cavalcanti, F.: Video motion activitytransmission with redundant antennas at the transmitter side. In:Comm., IEEE Latin-American Conference on, pp. 1–6 (2010)
12. Malewar, A., Bahadarpurkar, A., Gadre, V.: A linear rate controlmodel for better target buffer level tracking in H.264. J. SignalImage Video Process. 7(2), 275–286 (2013)
13. Zeng, H., Cai, C., Ma, K.K.: Fast mode decision for H.264/AVCbased on macroblock motion activity. Circuits Syst. Video Technol.IEEE Trans. 19(4), 491–499 (2009)
15. Shanableh, T., Assaleh, K.: H.264/AVC motion vector concealmentsolutions using online and offline polynomial regression. J. SignalImage Video Process. 1(1), 1–8 (2013)
16. Pham, H., Vafi, S.: Unequal error protection of H.264/AVC videobitstreams based on data partitioning and motion information ofslices. In: Signal Proc., Comm. and Computing, International Con-ference on, ICSPCC 2012. 4th IEEE, pp. 634–639 (2012)
17. Wenger, S.: H.264/AVC over IP. Circ. Syst. Video Technol. IEEETrans. 13(7), 645–656 (2003)
18. Haralick, R.: Statistical and structural approaches to texture. Proc.IEEE 67(5), 786–804 (1979). doi:10.1109/PROC.1979.11328
19. Haller, M., Krutz, A., Sikora, T.: Robust global motion estima-tion using motion vectors of variable size blocks and automaticmotion model selection. In: Image Processing (ICIP), 2010 17thIEEE International Conference on, pp. 737–740 (2010)
20. Advanced Video Coding for Generic Audio Visual services.Telecommunication Standardization Sector of ITU (2009)
