Multi-resolution Edge-guided Image Gap Restoration

Bahareh Langari, Saeed Vaseghi, Seyed Karnran Pedrarn

School of Engineering and Design, BruneI University, London, UK, UB8 3PH

Emails: (bahareh.langari.saeed.vaseghi.seyed. pedramrad)@brunel.ac.uk

Abstract- An image gap restoration method is presented based on a

combination of a multi-scale pyramid discrete cosine transform

(DCT) and edge-guided interpolation of the missing samples.

Through a process of pyramid DCT and down-sampling, the image is

progressively transformed into a series of reduced size layers until at

the apex of the pyramid the gap size is reduced to one sample. The

process is then reversed; at each stage. the missing samples are

estimated, up-sampled and combined with the available neighbouring

samples. The main contributions of this work are the incorporation,

within the DCT pyramid layers, of a directional interpolation method

using the local edges or textures and a global edge-guided

interpolation method based on the Sobel edge detector. Evaluations

over a range of images demonstrate that the proposed method results

is improved PSNR compared to a range of published works.

Keywords-Error concealment, multi-scale DCT pyramid, edge

detection, image gap recovery, packet loss concealment.

I. INTRODUCTION

Packet loss errors may occur in the transmISSIOn of image/video over internet protocol (IP) networks, due to network congestions or signal loss and the fact that IP networks are best-effort environments [1, 2]. The rapid growth in the use of relatively high bandwidth image/video streaming applications over IP networks motivates the need for packet loss recovery and concealment in order to provide more reliable network services and acceptable user experience [3].

There are three broad approaches for mitigating the loss of quality in received images due to packet loss: (a) request for retransmission of the lost packets, (b) error control via forward error correction (FEC) methods and (c) error concealment (EC) methods.

The fIrst method retransmits a copy of the damaged/lost packet; this method can be used on request from receiver in networks where there is an interaction between sender and receiver. This method involves an increase in the use of bandwidth and some increase in the delay [2].

The FEC methods employ error correction coding techniques such that when a packet is lost the corresponding image pixels can be recovered from the available received information. This generally implies that through coding and signal processing methods the pixel values in successive blocks of images would

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

be combined and/or spread over several successive packets. This method may involve and increase in bandwidth and delay.

The EC methods are receiver-based signal processing methods that aim to replace the lost packet with an estimate obtained from the received packets. EC methods utilise the observation, that images often contain high spatial correlations and recurring patterns, to recover lost packets from the neighbouring blocks [4-13].

Among the three solutions described above for packet loss recovery, an effective EC would be most beneficial as it does not require an increase in bandwidth, in contrast retransmission and FEC requires additional bandwidth and perhaps delay. Furthennore, retransmISSIOn and FEC techniques are not immune to errors. EC methods can be deployed in networks or used as embedded applications on the receiver handsets/tenninals, they do not require an ITU approved change in coding standard. Therefore, spatial EC is the solution explored in this paper.

Error concealment (EC) methods may be divided into two categories of spatial error concealment (SEC) and temporal error concealment (TEC). SEC utilise the correlations and patterns shared between close neighbouring samples in an image to interpolate the missing blocks [4], whereas TEC makes use of correlation across time and/or motion vectors for extrapolation/interpolation. Temporal EC methods find applications in time series such as in video. TEC recovers the missing parts of a video frame from a combination of interpolation and extrapolation by using the available infonnation from the current frame plus temporal correlations from previous frames.

In addition, methods have been proposed that combine the two types of concealment (SEC,TEC) and may switch between them by mode selection [7]. This is based on the both characteristic of the temporal and spatial concealment approach.

At their core, the EC methods often involve some form of extrapolation or interpolation methods such as non-directional, bilinear [4,7,9,13] and directional [5,8,10-12], but each of these methods has a drawback. Bilinear and non-directional techniques are able to recover the smooth area but fail to restore the visually important edge infonnation. On the other

000374

hand while directional interpolation methods do well in the recovery of edges, they suffer from leaving stripe-shaped artifacts in the smooth part of the image.

Adaptive methods are proposed in [14] in order to develop PLC methods that benefit from a combination of different methods. Two steps are involved in this technique, at the first stage the type of the error block (EB) is detected and classified into the uniform, texture or edge group. Then, the suitable EC method is applied to each category.

Basic spatial interpolation methods use a weighted average of the boundary pixels to recover the MB. Although in this way satisfactory result is achieved in the smooth areas, the performance around the edges can be blurred and unsatisfactory [9]. Edge-oriented directional interpolations for image restoration have been investigated in [10-12]. A further approach to SEC described in [13] which searches for the best similar MB in the image to replace the missing MB, using a technique called best neighbourhood matching (BNM).

All above mentioned methods shows that to obtain appropriate result, it is vital to consider edges along with bilinear and non­directional techniques. Many techniques have been proposed for using the edge related information [5,7-8,10-12]. A Hough transform-based technique is capable of systematically connecting the edges [8]. DCT pyramids are used with Bayesian estimation in [4].

The main contribution of this work is improvements in multi­scale image gap restoration obtained through incorporation of the local texture interpolation and global edge-guided interpolation, based on Sobel edge detector, within the pyramid layers.

The remainder of this paper organized as follows. In section II, the proposed method is introduced. The experimental results are presented in section III. And the conclusion is given in section IV.

II. THE PROPOSED IMAGE GAP CONCEALMENT

A. Multi-Scale Pyramid Discrete Cosine Transform

The multi-scale pyramid processing method decomposes image macro blocks (MB) into four spectral quadrants; LL, LH, HL, HH where L and H denote low and high frequency halves of spectrum respectively. After the first stage of decomposition, at each subsequent down-sampling and decomposition stage, the LL quadrant is further decomposed into four spectral quadrants until the macro block is reduced to a single pixel as shown in Fig 1.

The advantage of a multi-scale approach is the ability to transform the interpolation of relatively large size block to a series of subsampled blocks where ultimately at the lowest level the interpolation is a relatively simple task. Furthermore, the result of the interpolation at each reduced sampling level can be used as the basis for interpolation at the next higher sampling level.

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

For a MB of size 8 x 8, three stages of decomposition and down-sampling reduces the MB to one sample. This method is similar to wavelet filter structures, however, the DCT functions are used here as the basis function.

The 2D-DCT of an M x N matrix, A, is defined as:

M-l N-l

� � rr(2m - 1)p rr(2n - 1)q Bpq = apaq L., L., Amn cos 2M cos 2N

m=O n=O Where 0 ::; p ::; M - 1, 0::; q ::; N - 1 and

{llJM, p = 0 ap=

.J2/M, p =1= 0

{1/...JN, q = 0 and aq = rl7iV

,,2/N, q =1= 0

(1)

Note that, as shown in Fig. 1, the down-sampling by a factor of half is performed by simply retaining a quarter of the low-frequency index coefficients, the LL quadrant, and discarding the remaining three quarters, higher index, coefficients. Note that a single DCT is sufficient to produce a set of layered pyramid coefficients composed of the subsets of the DCT coefficients. However, at each sub-processing stage and in the recombination stages, separate DCT and IDCT may be required as shown in Fig. 2.

During the re-composition stages, starting from the apex of the multi-scale pyramid, image up-sampling by a factor of two is perfonned by a combination of a process of zero-padding of the 2D-DCT coefficients and the subsequent application of inverse 2D-DCT.

8x8 MB

Sub proc

Ii>k LH

HL HH

Sub proc

Sub proc Sub proc

Fig. 1. Above, block diagram of the three-stage DCT pyramid image decomposition and its application to Foreman image.

B. Image Gap Concealment Using Pyramid DCT

The proposed method for MB gap interpolation, illustrated in Fig. 2, is as follows:

1) Decompose image macro-blocks into a DCT pyramid

structure, with the apex of the pyramid representing

the last stage of down-sampling where each MB of

size 8 x 8 is reduced to one pixel only.

000375

2) Starting from the apex of the DCT pyramid

interpolate the decimated (down-sampled) gap using

the local edge information from neighbouring pixels.

3) Using an edge detector, track the global edges in the

interpolated images.

4) Enhance the interpolated gap estimates using the

global edge information.

5) Up-sample the enhanced interpolated image, via zero­

padded inverse 2D-DCT, and combine/merge with

the available received samples of the same layer of

up-sampling.

6) Go to step (1) and repeat the process for each

intermediate stage of up-sampling.

The details of these sub-processes are described next.

8x8 MB

Local·edge Interpolators

Global·edge Interpolators

Enhanced

Fig. 2. Three-stage DCT pyramid image decomposition, (D-(Tnterp=Directionallnterpolation).

Image with Replace missing missing MBs MBs

Directional Global edge interpolation detection

Edge·guided interpolation

Up sampled image ----------1' ---------

"

Fig. 3. Application of the proposed multi-scale method to restoration of corrupted image of Lena with 25% of 8x8 block loss.

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

C. Local Edge-Guided Enhancement

The proposed multi-scale gap restoration method strives to preserve the local edge or texture information at each scale of the reconstruction process. Note that the local edges may not show up at the later stage of detection of the global main edges after thresholding out the insignificant edges.

At each multi-scale level, spatial gap concealment interpolates the missing block by using the edge information obtained from the surrounding neighbours. Preserving the texture edges is important for successful error concealment. In this respect several observations are instructive:

1) Along the direction of an edge, the differences of pixel values are relatively small.

2) Across the direction of an edge the differences of pixel values are relatively large.

3) On either side of a gap, the differences of pixel values across an edge are consistent and of similar sign, with the possible exception of the gap coinciding with the end-points of an edge segment.

The estimate for the missing pixel at the final level of decomposition, i.e. at the pyramid apex, is an edge-weighted mean of the neighbouring pixels with consistent edges. At the

successive levels where an N x N block replaces a gap, directional edge-guided interpolation are used to fit the missing blocks with the edge patterns of the available neighbouring pixels.

As illustrated in Fig. 4, the directional interpolation strives to preserve the following three types of local edges:

1) Horizontal edges above and below the missing pixels, Fig. 4.a.

2) Vertical edges to the left and right of the missing pixels, Fig. 4.b.

3) Cross edges across four directions, Fig. 4.c.

(a) (b) (c)

Fig. 4. Directional interpolation for each missing pixel at the apex of multi­scale pyramid (in eight possible directions).

At the apex of the pyramid, where each sample gap is reduced to a single missing sample, the edge-enhanced estimation of the missing sample is given by the following expression.

Am.n = I I Wm+k,n+l(Am+k,n+l+edgem+k,n+l) (2) H,v,C k,l

Where edge(m + k, n + l) is a local estimation of the edge obtained separately in each of horizontal, vertical and cross directions depicted in Fig. 4, The edges along the directions

(m, n) � (m + k, n + l) are obtained from the average of all the available edges of the same direction in the immediate neighbourhood of the missing sample. For example, at the

000376

apex level, where the gap is reduced to one sample, for the

horizontal direction, edge(m - 1, n), may be obtained as:

{ 0 if edgem-l,n-l x edgem-l,n+l < 0 edgem-l n =

( ) , 0.5 edgem-l,n-l + edgem-l,n+l else

(3)

In order to make an estimate consistent with the most distinct neighbourhood edges, the edge combination weights can be expressed as a function of the intensity of the edges, as:

edgem+k,n+l Wm+k,n+l = (4)

Lk=-l:l,l=-l:l k,I�O edgem+k,n+l

Note that Lk=-l:l,l=-l:l k,I�O Wm+k,n+l = 1. After interpolation of the apex sample, at the subsequent

stages of interpolation, for blocks of size 2 x 2, 4 x 4, 8 x 8, a strategy similar to that described above is used. Staring from the outer boundaries of the MB, the missing pixels are progressively replaced towards the center with the edge­guided interpolation methodology that strives for an estimate consistent with the neighbouring edges in each of the horizontal, vertical and cross directions.

D. Global Edge-Guided Enhancement

From the above analysis and evaluations it is observed that preserving the local edges mitigates blurring distortions of textures and provides improved interpolation at a local texture level and in particular at the boundaries of the available and the missing samples. For further improvement where the missing blocks contain significant edges, the global edge information, not necessarily evident within the lost macro blocks, need to be utilised.

The global edges are used in a manner as to avoid blurred/smeared interpolation across the significant edges; the main cause of large interpolation errors and visible distortions. Hence with the availability of the boundary traces of the edges, it is possible to segment the pixels within and in the neighborhood of missing blocks and to confine the available samples used for interpolation of a missing sample to within a relatively homogeneous region on each side of the edge or onto the edge itself as required.

After edge-based segmentation, Equation 2 would be modified to include edge-segmented (ES) interpolation regions as:

Am,n = L L Wm+k,n+l (Am+k,n+l +edgem+k,n+l) (5) H,v,CEES k,l

Where now the interpolation and estimation of the edges, H, V, C E ES, are confmed to edge-segmented (ES) regions composed of relatively homogenous textures.

Note from Fig.2 that the global edge-guided interpolation is performed after local edge interpolation in order to mitigate the impact of the missing samples on the edge detection. For estimation of the main edges in the image, we investigated the applications of the popular edge detection method the Sobel detector.

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

D.l The Sobel Filter

The Sobel filter is a difference function applied to each pixel,

in the horizontal x -axis and the vertical y-axis directions, by using the following operators.

[-1 0 1] Sx = -2 0 2 , -1 0 1

[-1 -2 -1] Sy = S[ = 0 0 0 1 2 1 (6)

We experimented with alternative edge detection operators such as Laplacian of Gaussian (LoG), Mexican hat and Roberts filters [ ]. For our application the Sobel filter provides

satisfactory results. Filtering of the source image A with the

differential operators, Sx and Sy, yields two differentially­

enhanced images Gx and Gy as:

Cx = Sx * A and Cy = Sy * A ( 7)

Where the operator * denotes the 2D convolution or filtering

operation. The two differentially-enhanced images, Gx and Gy, can be combined to give the gradient magnitude operator defined as:

C = JC1 + C} (8)

Fig. 5 illustrates the application of Sobel filter to the Lena

image. Note that in Fig. 5 in Gx the horizontal edges are

enhanced while in Gy the vertical edges are enhanced. G, the

gradient magnitude of Gx and Gy, displays edges enhanced in

both the horizontal and the vertical dimensions.

A = LENA G

Fig. 5. Application of Sobel filter to Lena in x, y and the combined gradient directions.

Percentile-Rank Threshold Edge-Enhanced Image

In order to retain the most prominent edges only and suppress

the remainder, an edge threshold value, 8thresh, is derived

from the a-percentile statistics using the following algorithm.

G_sorted = sort(G, 'descend');

index = round(a * length(G_sorted));

8thresh= G_sorted (index);

Gthresh=sign(G,8thresh)

Where through a process of experimentation, the faction a is set to a value of 0.2 (i.e. 20 percentile statistic) and the function sign(G,8thresh) sets the values of GCi,}) less than

8thresh to zero. The application of threshold results to differentially processed Lena is shown in Fig. 6.

000377

G Gthresh

Fig. 6. Thresholding the output of Sobel filter, input G, output Gthresh.

III. EXPERIMENTAL EV ALUA nON RESULTS

For performance evaluation results the proposed algorithm has been tested on a number of standard test images namely; Lena, Peppers and Man. The image sizes are 512x512 pixels, with each grey-scale or one of the primary colours represented by 8 bits per pixel in unsigned integer format with a range of 0-255. The size of the missing blocks MB is set to 8x8 pixels. Two types of missing MBs are evaluated; regular missing MB at 25% loss rate (Fig. 7) and random missing MB with the loss rate set to 10% (Fig.8). The choice of the percentage loss is guided by our desire to compare our results with available results reported in the literature [4,15,16].

The performance measure criteria used for assessment of the quality of image recovery is the widely employed Peak­Signal-to-Noise-Ratio (PSNR) defined as:

MAXI PSN R = 20 iog10 RMSE dB (9)

Where MAXI = 255 for a pixel value represented in unsigned integer format and the root mean squared error (RMSE) function is defined as:

RMSE = 2 � Ldomain(A(m, n) - Ar(m, n))2 (10)

Where the domain over which the RMSE is calculated may

include only the missing samples or it may alternatively

include the entire image samples composed of the missing and

the available samples and N is the total number of samples

used in calculation of the RMSE.

A. Evaluation Case i-image with Regular Missing MBs

The proposed method is applied to images of Lena, peppers and Man as shown in Fig.7. The PSNR results are compared to a set of thirteen published work representing a number of methods that employ Bayesian and/or edge information for the recovery of regular lost macro blocks. The results are displayed in two different tables. Table 1 represents comparison with published results where the PSNR are averaged over the whole image including the available samples and table 2 represents comparison with published results where the PSNR are averaged over the missing pixels only.

Table 1 illustrates the performance of the several methods (values are taken from [16]). As displayed in Table 1, the proposed method performs better than the alternatives considered and there is an improvement of 0.43 dB to compare

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

with the best average performance when the PSNR are computed from whole image.

In addition, in table 2 (values are taken from [4,15]) the PSNR is calculated for the region of missing sample blocks only, and there is an increase of 2.66 dB. It can be observed that by using the Sobel filter more details are recovered and the image quality is improved.

Fig. 7. From left to right; the original images, the image with 25% missing MBs and the restored images for Lena, Pepper and Man.

Table 1. Performance comparisons for MB loss rate 25%, MB size= 8x8, PSNR I I d hI · L d

. h h S b I fil ca cu ate over w o e Image on ena: propose , WIt t e o e I ter.

Methods PSNR(dB)

Lena Man Pepper Average Ancis'S 28.68 25.47 27.92 27.35

Sun's 29.99 27.25 29.97 29.07

Shirani's 31.69 27.44 31.72 30.28

Hemami's 31.86 27.65 31.83 30.44

Alkachouh's 31.57 27.94 32.76 30.75

RIBMAP's 34.65 29.87 34.20 32.90

Proposed 34.18 30.58 35.23 33.33

Table 2. Performance comparisons for MB loss rate 25%, MB size= 8x8, PSNR calcul d

. f

. bl k d · h h Sobel filter. ate on regIon 0 mlssmg oc , propose =Wlt t e

Methods Image Lena PSNR (dB)

Zhai'S (1) 28.51

Agrafiotis'S 22.97

Parks'S 26.00

Zhai'S (2) 28.11

Zeng'S 27.43

Li's 28.25

Jung's 26.34

Proposed 31.17

000378

B. Evaluation Case 2 - Image with Random Missing MBs

In order to evaluate the performance result for the random block loss the application of the proposed method to Lena, Pepper and Man images, shown in Fig. 8, are compared to published methods [4,15].

Table 3 (values are taken from [4,15]) represents comparison with five published results where the PSNR are averaged over the region of missing sample blocks which is not including the available samples. As displayed in Table 3, the proposed method performs better than the rest and there is an improvement of 5.28 dB compared with the best performance among all results.

Fig. 8. From left to right; the original Lena, Pepper and Man images, the image with 10% random missing MBs and the restored image.

It can be observed that by using the Sobel filter more details are recovered and the quality of image is improved.

Table 3. Performance comparisons, MB loss rate random 10%, MB size= 8x8, PSNR calculated on region of missing block: proposed=with the Sobel

filter

Methods Image Lena PSNR(dB)

Zhai'S (1) 28.13

Zhai'S (2) 27.65

Zeng'S 26.60

Li's 27.38

SECOAI'S 28.25

Proposed 33.53

IV. CONCLUSION

In this paper we have evaluated an image gap restoration method with application to packet loss concealment in

978-1-4799-4796-6/13/$31.00 © 2013 IEEE

networks where image ME may be lost due to congestion or signal fading. The main contribution of the work is the inclusion of local and global edge enhancement strategies within a pyramid DCT image processing framework. The proposed algorithm includes combination of multi-resolution transforms, directional interpolation and edge-guided enhancement capable of restoring missing ME including the edges. The experimental results demonstrate that significant improvement in the quality and PSNR of the restored images are obtained by the proposed edge guided image restoration method. Further work includes the use of complex wavelets instead ofDCT for pyramid decomposition.

REFERENCES

[I] M. Yang, and N. G. Bourbakis, " An Efficient Packet Loss Recovery Methodology for Video Streaming Over IP Networks," IEEE Trans. Broadcasting, Vol. 55, No. 2, Jun. 2009.

[2] M. Hayasaka, M. Gamage and T. Miki, "An Efficient Loss Recovery Scheme for On-demand Video Streaming over the Internet," IEEE, Page(s): 1301 - 1306,2005.

[3] Polycom's Lost Packet Recovery (LPR) Capability, Wainhouse Research, 2008.

[4] G. Zhai, X Yang, W. Lin, W. Zhang, "Bayesian Error Concealment with DCT Pyramid for , Images," IEEE Trans. Circuits and Systems for video technology, Vol. 20, No. 9, Sept. 2010.

[5] H. Asheri, H. R. Rabiee, N. Pourdamghani, M. Ghanbari, "Multi­Directional Spatial Error Concealment Using Adaptive Edge Thresholding," IEEE Trans. Consumer Electronics, Vol. 58, No. 3, Aug. 2012.

[6] Y. Wang, and Q. F. Zhu, "Error control and concealment for video communication: A review," Proceedings of the IEEE, Vol. 86, No. 5, Page(s):974-997, 1998.

[7] D. Agrafiotis, D. R. Bull, and C. N. Canagarajah, "Enhanced Error Concealment With Mode Selection," IEEE Trans. Circuits and Systems for video technology, Vol. 16, No. 8, Aug. 2006.

[8] W. Kim, J. Koo, and J. Jeong, "Fine Directional Interpolation for Spatial Error Concealment," IEEE Trans. Consumer Electronics, Vol. 52, No. 3, Aug. 2006.

[9] Y.-K. Wang, M. M. Hannuksela, V. Varsa, A. Hourunranta, and M. Gabbouj, "The error concealment feature in the H.26L test model," in Proc. lnt. Conf. Image Processing (IClP), Page(s): 729-732,2002.

[10] S.-c. Hsia, "An edge-oriented spatial interpolation for consecutive block error concealment," IEEE Signal Process. Lett., Vol. II, No. 6, Page(s): 577-580, Jun. 2004.

[II] J.-W. Suh and Y.-S. Ho, "Error concealment based on directional interpolation," IEEE Trans. Consum. Electron., Vol. 43, No. 3, Page(s): 295-302,Aug. 1997.

[12] W. Kwok and H. Sun, "Multidirectional interpolation for spatial error concealment," IEEE Trans. Consum. Electron., Vol. 39, No. 3, Page(s): 455-460, Aug. 1993.

[13] Z. Wang, Y. Yu, and D. Zhang, "Best neighborhood matching: An information loss restoration technique for block-based image coding systems," IEEE Trans. Image Process., Vol. 7, No. 7, Page(s): 1056-1061, Jul. 1998.

[14] Z. Rongfu, Z. Yuanhua, and H. Xiaodong, "Content-adaptive spatial error concealment for video communication," IEEE Trans. Consum. Electron., Vol. 50, No. I, Page(s): 335-341, Jan. 2004.

[15] Xin Li, and Michael T. Orchard, "Novel Sequential Error-Concealment Techniques Using Orientation Adaptive Interpolation," IEEE Trans. Circuits Syst. Video Technol., Vol. 12, No. 10, Oct. 2002.

[16] J. Park, D. -C. Park, R. J. Marks, II, M. A. EI-Sharkawi, "Recovery of image blocks using the method of alternating projections," IEEE Trans. Image Process., Vol. 14, No. 4, Page(s): 461 - 474, Apr. 2005.

000379