Concealment of damaged blocks by Neighborhood Regions Partitioned Matching

William Y F Wong’, Angus K Y Cheng’ and Horace H S

Image Computing Group, Department of Computer Science, Centre for Innovative Applications of Internet and Multimedia Technologies (AIMtech Centre)

City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong

e-mail: cship@,cityu.edu.hk

1

2

ABSTRACT

This paper presents a new error concealment algorithm for the masking of damaged blocks due to data loss during transmission over packet network of DCT based images. The algorithm is called Neighborhood Regions Partitioned Matching (NRPM). In our approach, instead of repairing damaged block as a whole, each damaged block is divided into a number of sub-blocks. Each sub-block is recovered utilizing not only the surrounding pixels, but also remote pixels which share similar local image characteristics. The approach yields good quality repair and since smaller block size is used, it can significant reduce the computational time for the reconstruction process.

1. INTRODUCTION

For the transmission of digital image and video signals over network, image data is compressed in order to reduce the transmission bit rates. Among them, block-based techniques have been shown to be the most practical and are adopted by most existing image and video compression standards, such as JPEG[ 11, MPEG[2] and H.261 [3]. During the transmission over the Internet and ATM networks, these kinds of fast packet networks frequently introduce heavy fidelity problems, produced by packet or cell losses. Since the signal transmitted is highly compressed, a packet loss can produce catastrophic effects in the decoded data. To cope with such problem, error concealment techniques have been introduced [4- 141.

Error concealment is aimed at masking the missing blocks to create subjectively acceptable images. Basically, the techniques are based on a spatial interpolation or temporal interpolation and replenishment using a picture correlation property. Such methods consist of low-pass filtering operation, so they can recover low-frequency information, but tend to ignore high frequency components, e.g. edges [4-61. High frequency information

can be recovered by fuzzy logic theory [7] and Gerchberg- type iterative process [8], but they are complex in processing. Sketch-Based recovery [9], [ lo] and [12] need pre-processing of edge detection of image before restoration; Best Neighborhood Matching (BNM) [ 141 needs an extra flag image to detect the positions of damaged blocks, they increase the processing time and can be only applied on data loss on still images. Other methods [I 13 and [13] use the interpolation of local surrounding blocks only, and aims to establish a mathematical hnctions to represent the relations of the damaged block with surrounding pixels. Actually, the satisfactory results were obtained mainly when applied to low frequency components. Apart from these, all the methods described above tend to do the restoration in a block-based approach, and rarely consider sub-dividing a whole block into smaller pieces for restoration.

The difficulties for the restoration of damaged blocks are serious loss of data. It is may not be appropriate to use only surrounding pixels to restore the entire block at one time, or to establish some mathematical functions to explain the relation between damaged block and surrounding pixels. The key issues in solving this problem are restoration efficiency and good matching between the damage block and some image data within the same image. Our contention is that by sub-dividing the whole damaged block into a number of smaller sub-block, we can improve the restoration efficiency by reducing the search time for appropriate image data for restoration and, at the same time, not restricting our search within a local area of the damaged block. Neighborhood Regions Partitioned Matching (NRPM) presented here uses surrounding blocks to restore damaged block, but the selected blocks are not used for interpolation. A distinguished feature of this approach is that each damaged block is partitioned into a number of smaller sub-blocks and each sub-block is matched with other far away partitioned blocks and the damaged block is restored “pixel by pixel”., More importantly this technique does not rely on pre-defined

0-7803-6725-1/01/$10.00 02001 IEEE 45

constrains on the spectra or structures of the lost blocks and it is independent of block encoding approach. Besides, no pre-processing is needed, e.g. edge detection, segmentation of image or extra procedure to detect positions of damaged blocks. The combination of these features results in short processing time and can be applied to real-time applications with high quality of recovered images.

In Section 2, the algorithm of NRPM is presented. Section 3 is the experimental results. Section 4 is the parameters sensitivity analysis. The final section presents the conclusions.

2. NEIGHBORHOOD REGION PARTITIONED MATCHING

To detect the occurrence of corrupted data, after the transmission of an image, if the checksum between the blocks before and after transmission is not matched, the block after transmission is regarded as corrupted and its position is recorded. After checking all pairs of relevant blocks, the restoration algorithm is triggered. In our experiment, we exaggerate the severity of the damaged blocks to demonstrate the efficiency of this approach.

In the experiments, a11 the damaged images are 512 x 512 in pixels. They are gray-scale and have 10% of damage severity, in which the damaged blocks are located at random positions of size 8 x 8 in pixels. No two or more damaged blocks are contiguity, that means each damaged block are surrounded by eight good blocks.

During the restoration, a damaged block is divided into 64 pixels. Each pixel is restored one by one in specific sequence. The sequence is shown in Fig. 1. Referring to Fig. 2, and taking pixel 1 as an example, it is surrounded by good pixels of a, b and c. These 3 contiguity pixels are extracted, named partitioned piece 1 and compared with other target pieces. Target pieces are the combination of 3 good pixels near the damaged block within a range, and have the same configuration with the partitioned piece. First, pixels a, b and c of partitioned piece 1 are compared with pixels d, e and f of target piece 1, a Mean Square Error (MSE) as defined in (1 ) is calculated.

(a - d ) 2 + (b - e ) 2 + (d - f ) 2 MSE = -> 3

Then, partitioned piece 1 is compared with another target piece 2 of pixels h, i and j and a new MSE is calculated by using the above formula with different substituted variables. Two MSE are compared, and the one with lower value i s chosen, and the corresponding pixel is used to restore damaged pixel. In this example, if the MSE from target piece 1 is lower, pixel g will restore pixel 1. It is taken that pixel g is similar to pixel 1 if their

surrounding pixels are similar to each other in a close texture pattern. For real cases, the selection of the target pieces is done in a spiral-searching path, which starts from the surrounding of partitioned piece as center, and expands the searching radius for far away good pixels. In practice, the restoration of one pixel involved a number of comparisons under a pre-defined threshold value. More importantly, the searching is not only limited to local pixels near the boundaries, but also remote regions. For the pixel 6, pixels 1, 2 and 5 are used as the boundary, in which they are just been restored. Following the sequence to restore all the 64 damaged pixels , a whole damaged block is recovered. It can be seen that this approach allows different damaged pixels within a damaged block to be restored using different configuration of good pixels to form the final target piece. The relation between the damaged pixel and partitioned piece is expressed below:

where dn (x ,v ) means damaged pixel of restoration

sequence n at position (x,y); p o , p , and p z mean the 3 pixels of partitioned piece; for all 0 < x 5 5 12, 0 5 y 5 5 12. Actually, MSE between partitioned piece and target piece can be summarized below:

where p means a particular partitioned piece; t , means a

target piece of number n and O l n l t h r e s h o l d value of comparisons; MSE ( p , 2, ) means the Mean Square Error

between partitioned piece p and target piece t , .

Figure 1. Specific restoration

I --/\ I

Figure 2. Searching of Target

46

sequence for each damaged Piece under Spiral-Searching pixels Path

3. EXPERIMENTAL RESULTS

In this experiment, three 256 gray-scale images of “Barb”, “Lena” and “Baboon” are chosen for demonstration. Their image sizes are 512 pixels x 512 pixels. Moreover, the damaged blocks are isolated with damage seventy 10%.

In this paper, all the experiments were run under a fixed configuration of hardware and software. The hardware configurations were Intel CPU 200 MHz with 32 MB Ram. All test programs were written in the C Language and run under Window95 environment.

In order to evaluate the quality of reconstructed images,, peak signal-to-noise (PSNR) as defined in (7) is used tot give a* quantitative evaluation.

where x, y are pixels value at position (ij)) of original image and restored image respectively with dimension N x M pixels. In brief, higher PSNR means a better quality of restored image.

After applying NRPM on both images, their PSNR and processing time are 35.94(db), 5.66(s); 37.06(db), 5.75(s) and 28.29(db), 5.92(s) respectively. Fig. 3.land Fig. 3.2 show the damaged image and restored image of “Barb”.

size 25% - “Barb” size 25% - “Barb”

Table 1 shows the comparisons with other algorithms, including Best Neighborhood Matching (BNM)[ 141, Sketch-Based Recovery [9], Projections onto the Overcomplete Basis Approach [ 131 and Transform coded image reconstruction exploiting interblock correlation [6]. As the authors of BNM provide the source code of programs and hardware for reference and testing, the comparison with this algorithm was done in more details. Fig. 4.1 - Fig. 4.4 show some enlarged parts for illustration in 200% from image “Lena” recovered by BNM and NRPM respectively. The positions of the damaged blocks are located at visually complex region,

e.g. eye. Refemng to the images recovered by NRPM, it shows it can handle complex region more efficiently than other algorithms. In all case, NRPM can reduce the restoration time by over 20 folds. By sub-dividing a whole damaged block into smaller pieces and handles the small pieces separately, experiments show that this approach is very efficient as well as achieves good estimation for damaged pixel restoration.

Table 1. PSNR (dh) and Recovering Time (s) of different algorithms for an Intel 200 MHz CPU)

I Barb I Lena 1 Baboon

* “\”‘meanstno data provided:.

Figure 4.1 Original image of size 200% - “Lena”

Figure 4.2 Damaged image of size 200% - “Lena”

bigurc 4.3 Restored image Of Figure 4.4 Restored imase of

4. NRPM PARAMETERS SENSITIVITY ANALYSIS

Before dividing a damaged block into smaller pieces, two parameters are needed to be set: the number of surrounding pixels required around the damaged pixels and number of damaged pixels involved in one partitioned piece. Fig. 5 shows different combinations of surrounding pixels and damaged pixels for a partitioned piece. In order to analysis the performance of NRPM with respect to parameter setting, , we have tested all possible combinations of partitioned piece applying on the recovery for damaged images “Barb”, “Lena” and “Baboon”. There are totally 48 experiments. The PSNR and time required

47

are recorded and mean values are calculated. After experiments, we find ‘out that Fig. 5 (a), using 3 surrounding pixels with 1 damaged pixel as the partitioned piece is the best. Chart 1 shows the relation between these parameters with the quality of recovered images and processing time required. It can be seen that the approach does not degrade suddenly with small changes in parameter values and $how that best results can be obtained with a small partitioned size.

Chart 1. The relation between all types of partitioned piece

.I*.,_,

,. .. . . o...... I

.. .. ”* m ,I. 110 ,.I 1.0 ,.I Figure 5. Possible types of a

partitioned piece _.,.I

5. CONCLUSIONS

In this paper, we propose Neighborhood Regions Partitioned Matching algorithm (NRPM) for error concealment of block-coded images. The novelty of this algorithm is that it sub-tlivides a whole damaged block into smaller pieces and uses surrounding pixels and remote pixels for restoration. This approach allows different partitions of the damaged block to be restored using different configuration of good pixels within the same image and reduction in the search time. Experiments have shown that the recovered images have a higher PSNR than existing algorithms, reduces the restoration time substantially and with a low requirement of hardware configurations. The NRPM algorithm has been designed to be applied to restoration of still images. Due to its low computational requirements, with further optimization on the algorithm, the approach can potentially be applied to restoration of video images.

6. ACKNOWLEDGEMENT

The authors would like to thank Mr. Wang Zhou, who is the author of BNM [14] algorithm. He provides the source code of programs and hardware for reference and testing.

7. RElFERENCES

[I] Joint Photo,graphic Experts Group, ISO/IEC/JTCI/SC2/WG8, “JPEG technical specification, Revision 8,” JPEG-8-R8, Aug. 1990. “Coding of Moving Picture and Associated Audio for Digital Storage Media at up to about 1.5 Mbps, Part 2: Video,” ISO-IEC/JTCl SC29/WG11, Nov. 23, 1991. CCITT Draft Revision of Recommendation H.261, “Video Code for Audiovisual Services at px64 kbiffs,”

[2]

[3]

[41

CCITT SGXV, “Description of Ref. Model 8 (RM8),” Doc 525, June 1989. S. Yamasaki, “A reconstruction method of damaged two-dimensional signal blocks using error correction coding based on DFT”, Proceedings of IEEE Asia Pacific Conference on Circuits and Systems ‘96, November 18-2 1, 1996, Seoul, Korea. H.Sun, K.Challapali and J.Zdepslu, “Error concealment in digital simulcast AD-HDTV decoder”, IEEE Trans. on Consumer Electronics, vol. 38, no. 3,

S.S.Hemami and T.H.Y.Meng, “Transform coded image reconstruction exploiting interblock correlation”, IEEE Trans. on Image Processing, vol. 4, no. 7, pp. 1023-1027, July 1995. X.Lee, Y.Q.Zhang and A.L.Garcia, “Information Loss Recovery for Block-Based Image Coding Techniques - A Fuzzy Logic Approach”, IEEE Trans. Image Proc., Vo1.4, No.3, pp.259-273, March 1995. H.Sun and W.Kwok, “Concealment of damaged block transform coded images using projections onto convex sets”, IEEE Trans. on Image Processing, vol. 4, no. 4, pp. 470-477, Apr. 1995. Atzori.L, De Natale, and F.G.B., “Concealment of Visual Effects of Image Transmission Errors by a Sketch-Based Recovery Approach”, Proceedings. 1998 Intemationa! Conference on Image Processing, 1998.

H.Sun and W.Kwok, “Error concealment with directional filtering for block transform coding”, Global Telecommunications Conference, 1993, including a Communications Theory Mini-Conference. Technical Program Conference Record, IEEE in Houston. GLOBECOM ’93., IEEE, 1993, pp. 1304- 1308, vol. 2. W.P.Jong, S.K.Dong and U.L.Sang, “On the error concealment technique for DCT based image coding”, IEEE International Conference on Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., vol. iii,

M.Ancis and D.D.Giusto, “Reconstruction of missing blocks in JPEG picture transmission”, IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, 1999, pp. 288-291. J.H.Chang, J.K.Kim and C.W.Lee, “A projections onto the overcomplete basis approach for block loss recovery”, Proceedings., International Conference on Image Processing, 1997., pp. 93-96, vol.:!. Z.Wang, Y. Yu and D.Zhang, “Best neighborhood matching: an information loss restoration technique for block-based image coding systems”, IEEE Transactions on Image Processing, vol. ’77, July 1998,

pp. 108-118, Aug. 1992.

ICIP 98, pp. 482-486.

1994, pp. 293-296.

pp. 1056-1061.

48