Second-generation error concealment for video transport...

WIRELESS COMMUNICATIONS AND MOBILE COMPUTINGWirel. Commun. Mob. Comput. 2002; 2:607–624 (DOI: 10.1002/wcm.83)

Second-generation error concealment for video transportover error-prone channels

Trista Pei-chun Chen and Tsuhan Chen*,†

Electrical and Computer EngineeringCarnegie Mellon UniversityPittsburghPAU.S.A.

Summary

Video transport over error-prone channels may resultin loss or erroneous decoding of the video data.Error concealment is an effective mechanism toreconstruct the video data. In this paper, we reviewerror-concealment methods and introduce a newframework, which we refer to as second-generationerror concealment. All the error-concealmentmethods reconstruct the lost video data by makinguse of certain a priori knowledge about the videocontent. First-generation error concealment buildssuch a priori in a heuristic manner. The proposedsecond-generation error concealment builds the apriori by modeling the statistics of the videocontent explicitly, typically in the region of interest(ROI). Context-based models are trained with thecorrectly received video data and then used toreplenish the lost video data. Trained models capturethe statistics of the video content and thusreconstruct the lost video data better thanreconstruction by heuristics. A new dynamic model‘updating principal components’ (UPC) is proposedas a model for second-generation error concealment.UPC can be applied to pixel values to conceal lossof pixel data. In addition, UPC can be applied tomotion vectors, which results in ‘updatingeigenflows’ (U-Eigenflow), to conceal loss ofmotion vectors. With UPC applied to both pixelvalues and motion vectors, hybrid temporal/spatialerror concealment can be achieved. The proposedsecond-generation error-concealment methodprovides superior performances to first-generationerror-concealment methods. Copyright 2002 JohnWiley & Sons, Ltd.

ŁCorrespondence to: Tsuhan Chen, Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213,U.S.A.†E-mail: [email protected]/grant sponsor: Industrial Technology Research Institute

Copyright 2002 John Wiley & Sons, Ltd.

608 TRISTA PEI-CHUN CHEN AND TSUHAN CHEN

KEY WORDSsecond-generation error concealmentfirst-generation error concealmentupdating principal components (UPC)updating eigenflows (U-Eigenflow)Projection onto Convex Sets (POCS)error concealmenterror resilient video codingwireless multimedia

1. Introduction

When transmitting video data over error-prone chan-nels, the video data may suffer from losses orerrors. Error concealment is an effective way torecover the lost information at the decoder. Comparedto other error-control mechanisms such as forwarderror correction (FEC) [1] and automatic retransmis-sion request (ARQ) [2], error concealment has theadvantages of not consuming extra bandwidth asFEC and not introducing retransmission delay asARQ. On the other hand, error concealment canbe used to supplement FEC and ARQ when bothFEC and ARQ fail to overcome the transmissionerrors [3].

Error concealment is performed after error detec-tion. That is, error concealment needs to be pre-ceded with some error-detection mechanism to knowwhere the errors in the decoded video are located.For example, error detection provides informationas which part of the received video bitstream iscorrupted. Various methods, such as checking thevideo bitstream syntax, monitoring the packet num-bers of the received video data, and so on, canbe applied [4,5]. In this paper, we assume that theerrors are located and that such information is avail-able to us. We focus on the reconstruction for thelost video.

In general, spatial, spectral, or temporal redundan-cies of the received video data are utilized to performerror concealment [6]. Hybrid or dynamic switchingof spatial/temporal error-concealment methods is alsopossible [7–9]. In this paper, we will review theseerror-concealment methods.

All error-concealment methods reconstruct the lostvideo content by making use of some a prioriknowledge about the video content. Most existing

error-concealment methods, which we refer to as first-generation error concealment, build such a priori ina heuristic manner by assuming smoothness or con-tinuity of the pixel values, and so on. The proposedsecond-generation error concealment methods traincontext-based models as the a priori. Methods of sucha framework have advantages over first-generationerror concealment, as the context-based model is cre-ated specifically for the video content and hence cancapture the statistical variations of the content moreeffectively.

It is important for a second-generation error-con-cealment approach to choose a model that can repre-sent the video content effectively. Principal compo-nent analysis (PCA) has long been used to modelvisual content of images. The most well-knownexample is using eigenfaces to represent humanfaces [10]. In this paper, we introduce a new dynamicmodel ‘updating principal components’ (UPC) [11]for second-generation error concealment. UPC is verysuitable for error-concealment applications in that itupdates with nonstationary video data. UPC can beapplied to pixel values in regions of interest (ROI)in video frames. In addition, UPC can be appliedto motion vectors (MVs), which results in ‘updatingeigenflows’ (U-Eigenflow ). With both UPC for pixelvalues and UPC for MVs, hybrid temporal/spatialerror concealment can be achieved.

This paper is organized as follows. In Section2, we review first-generation error concealment byproviding a survey of conventional error-concealmentmethods. We introduce the new framework of second-generation error concealment in Section 3. A detaileddescription of using UPC for error concealment isprovided in Section 4. Both UPC for pixel valuesand for MVs will be discussed. We conclude inSection 5.

Copyright 2002 John Wiley & Sons, Ltd. Wirel. Commun. Mob. Comput. 2002; 2:607–624

SECOND-GENERATION ERROR CONCEALMENT 609

2. First-generation Error Concealment

Error concealment relies on some a priori to recon-struct the lost video content. First-generation error-concealment methods build the a priori for recon-structing the lost video content in a heuristic manner.A simple example is to assume the pixel valuesto be smooth across the boundary of the lost andretained regions. Methods of this framework assumesmoothness or continuity of the video data in differentdomains such as spatial, spectral, temporal, or sometransforms of these domains. To recover lost data withthe smoothness assumption, interpolation or optimiza-tion based on certain objective functions are oftenused. Since first-generation error-concealment meth-ods perform error concealment with such heuristicknowledge, we also call them heuristic-based errorconcealment.

According to the domains in which smoothnessassumptions are applied, first-generation error-concealment methods fall into two categories:spatial/spectral and temporal, as follows.

2.1. Spatial/spectral Error Concealment

Spatial error concealment assumes that images aresmooth in nature. Lost image content can be recon-structed by interpolation of the neighboring pix-els. Work by Wang et al. [12] and Hemami andMeng [13] are earlier examples of using spatial inter-polation to accomplish the task of error concealment.However, spatial interpolation approaches often suf-fer from blurring in the edges of the image. Sev-eral approaches have been proposed to solve thisproblem. Suh and Ho [14] proposed to find edgesfirst and interpolate along the edge directions. Zhuet al. [15] proposed to use a second-order derivative-based method to reduce the blur across the edgewhile enforcing the smoothness along the edge. Zengand Liu [16] proposed to perform directional inter-polation based on the neighbor’s geometric structure.Robie and Mersereau [17] proposed to use the Houghtransform to determine the best orientation for eitherdirectional filtering or interpolation. Interpolation canbe applied not only to the spatial domain but alsoto the spectral domain such as the discrete cosinetransform (DCT) domain, as proposed by Chunget al. [18]. Some other methods are based on pro-jection onto convex sets (POCS), which iterativelyuses the smoothness assumption and pixel value orDCT coefficient range information for error conceal-ment [19,20].

An extension to the assumption that natural imagesare smooth and the values are continuous spatially orspectrally is to assume that images can be modeledby Markov random fields (MRF) [21]. MRF-basederror-concealment methods were first proposed bySalama et al. [22–24]. Later Shirani et al. [25] pro-posed to adaptively adjust the MRF model parameterswithout increasing the model order and showed thatthe adaptive MRF outperformed conventional MRFmethods. Multiscale MRF (MMRF) by Zhang andMa [26] is another extension of MRF. MMRF mod-els image blocks instead of image pixels. Work byZhang et al. [27] models the DCT coefficients as afirst-order Markov process and uses Laplacian dis-tribution to model the density function of the DCTcoefficients.

2.2. Temporal Error Concealment

Temporal error-concealment methods use the tempo-ral neighbor, that is, the previous frame or the nextframe, to conceal the loss of the current frame. Tem-poral error-concealment methods assume the videocontent to be smooth or continuous in time. A basicapproach is to replace the lost block of the cur-rent frame with the content of the previous frameat the same block location. An advanced approachis to replace the lost block with the content of theprevious frame at the motion-compensated location.This advanced temporal error-concealment schemeneeds motion vector information to find the cor-responding block location in the previous frame.However, in the process of transmission, MVs canbe lost as well. Without MVs, temporal error con-cealment with motion compensation cannot providesatisfactory reconstruction results. Therefore, tech-niques to estimate the lost MVs are explored. Bound-ary matching algorithm (BMA) proposed by Lamet al. [28] is a popular method to estimate lost MVs.Extensions to BMA can be found in References [29to 32]. Decoder motion vector estimation (DMVE)proposed by Zhang et al. [33,34] treats the loss ofMVs as a motion estimation problem, in the decoderinstead of in the encoder. Motion field interpola-tion (MFI) and its extensions proposed by Al-Muallaet al. [35,36] estimate the MVs from neighbors witha single or multiple reference frames. Furthermore,Lee et al. [37] extended translational block motion toaffine transform for motion-compensated error con-cealment.



3. Second-generation Error Concealment

Second-generation error concealment goes beyondheuristics. A more sophisticated way to obtain thea priori for error concealment is through training.Second-generation error concealment builds the apriori by training a context-based model for an objector ROI and uses this trained model to recover the lostdata. With object-based video coding standards suchas MPEG-4 [38], the video bitstream already containsobject, or ROI, information, which makes second-generation error concealment possible. In cases inwhich the ROI information is not available in thevideo bitstream, object trackers [39] can be used toextract the ROI information. Figure 1 shows threevideo frames with face regions tracked and specifiedas the ROI. Since the context-based model is createdspecifically for the object, it captures the statisticalvariations of the object effectively and yields goodconcealment results. Since second-generation errorconcealment methods train and apply context-basedmodels for error concealment, we also call themmodel-based error concealment.

As mentioned, PCA has been widely used to modelimage statistics. Face images, the most commonexamples of ROI, especially in video telephony orvideoconferencing, can be modeled well with PCA.Figure 2 shows an example of using PCA with amean and two eigenvectors to represent face images.The mean captures the average face appearance andthe eigenvectors characterize variations such as poseor expression variations.

With PCA as the model to describe the objectstatistics, we can train the PCA model, that is, themean and the eigenvectors, with pixel values in theROI from correctly received frames. Then, we projectany corrupted ROI to the PCA model to recover thelost data in the corrupted ROI. Using face images

(a) (b) (c)

Fig. 2. PCA for face images: (a) mean; (b) firsteigenvector; and (c) second eigenvector.

1

2 3

Project

Reconstruct

Fig. 3. Error concealment with PCA.

as an example of ROI, we illustrate such an error-concealment scheme in Figure 3. The PCA modelshown in ©2 is trained in advance. The corruptedROI shown in ©1 is projected to the PCA model toget the recovered ROI as shown in ©3 .

POCS can be used in the second-generation error-concealment scheme based on PCA modeling of theROI. Error concealment based on POCS formulateseach constraint about the unknowns as a convexset. The optimal solution is obtained by iterativelyprojecting the previous solution onto each convexset. The projections refer to (i) projecting the datawith some losses to the PCA model that is built onerror-free data, and (ii) replacing the projection resultwith the correctly received data in the correspondingregion. Illustration of POCS-based error concealment

InterviewAkiyo

(a) (b) (c)

Fig. 1. Frames from sequences (a) ‘Akiyo’ and (b), (c) ‘Interview’, with objects/ROI specified.



Remain correctlydecoded data

Reconstruct

Project

2 3

1

Fig. 4. POCS-based error concealment with PCA.

with PCA is shown in Figure 4. The PCA modelshown in ©2 is pretrained with correctly receivedROI. Any corrupted ROI in ©1 is projected to thePCA model to obtain the recovered ROI shown in©3 . After the reconstructed ROI in ©3 is obtained,the region in ©3 in which the data was correctlyreceived is replaced with the correctly received data.This result is projected again to the PCA model andso on until the reconstruction result is satisfactory.

In addition to PCA that can represent the ROIstatistics of the training data, ‘mixture of principalcomponents’ (MPC) [40] can model the multimodalcharacteristic of the data. ‘Updating mixture of prin-cipal components’ (UMPC) [11] can further adapt themodel itself with the nonstationary characteristics ofthe ROI content. Focusing on the dynamic adaptingnature of the UMPC model, we will elaborate on asimplified case of the UMPC model with a single-mixture component, which we refer to as ‘updatingprincipal components’ (UPC). We will illustrate in thenext session about how to use UPC as the model forsecond-generation error concealment to recover thelost video data.

In addition to PCA, MRF model can be used formodel-based error concealment. Shirani et al. [41]proposed to use an appropriate form of the MRF tomodel the shape information of a MPEG-4 video.The MRF parameters are obtained from the edgedirections of the neighbors. A maximum a posteriori(MAP) estimation gives the most likely reconstruc-tion result using such an MRF model. Furthermore,model-based error concealment can use models thatwere originally proposed for model-based video cod-ing. These include 3-D model-based approaches inwhich a 3-D model of the object appearance is builtbefore coding, and 2-D model-based approaches thatuse deformable segmentation of the image and theaffine motion model. A good overview of model-based video coding can be obtained by Aizawa andHuang [42] and Pearson [43].

Temporal model-based error concealment is alsopossible. Models can be built for MVs. The recon-structed MVs are then used for motion compensation.For example, eigenflows proposed in Reference [44]can be used to model MVs and reconstruct anylost MV. We will detail in the next session how toupdate eigenflows, which we refer to as ‘updatingeigenflows’ (U-Eigenflow), for temporal error con-cealment.

4. Error Concealment with UpdatingPrincipal Components (UPC)

It is important for a second-generation error-conceal-ment method to choose a model that can represent thevideo content effectively. We propose to use UPC asthe model for error concealment because UPC adaptsitself to the nonstationary video data. Section 4.1 willdescribe the UPC model.

Video sequences can be either Intra or Inter codedwith coding standards such as MPEG-4 [38] andH.263 [45]. The coded bitstream consists of headerinformation, MVs, DCT coefficients, and so on.Therefore, loss of video data could be loss of anyof the above information or a combination of them.In this paper, we perform error concealment for theloss of MVs and/or DCT coefficients.

In the Intra coding scenario, DCT coefficientscould be lost in the transmission process. We can useUPC to reconstruct the pixel values in the ROI of acorrupted video frame. That is, spatial error conceal-ment with UPC for pixel values in the ROI is per-formed (UPC for ROI). We will describe more aboutUPC for ROI for Intra coded videos in Section 4.2.

In the Inter coding scenario, both MVs and DCTcoefficients could be lost in the transmission process.We can use UPC to reconstruct MVs of a frame. Wecall UPC applied to MVs, U-Eigenflow. The recon-structed MVs are then used for motion compensation.That is, U-Eigenflow for MVs is performed for a tem-poral error-concealment scheme to recover the lostMVs. This temporal error-concealment scheme can beused together with spatial error concealment to forma hybrid temporal/spatial error-concealment scheme.After motion compensation, UPC is further applied topixel values in the ROI if the DCT coefficients insidethis ROI are lost. The hybrid temporal/spatial error-concealment scheme with U-Eigenflow for MVs andUPC for ROI for Inter coded videos will be detailedin Section 4.3.



4.1. Updating Principal Components (UPC)

Given a set of data, we try to model the data withminimum representation error. The data given can benonstationary, that is, the stochastic properties of thedata are time-varying as shown in Figure 5(a). Forexample, at time instant n, the data are distributedas shown by Figure 5(a). At time instant n0, the dataare distributed as shown by Figure 5(b). We see thatthe mean of the data is shifting and that the mostrepresentative axes of the data are also rotating.

At any time instant, we attempt to represent thedata as a weighted sum of the mean and the prin-cipal axes. As time proceeds, the model changes itsmean and principal axes as shown in Figure 6, fromFigure 6(a) and (b), so that it always models thecurrent data effectively. To accomplish this, the repre-sentation/reconstruction error of the model evaluatedat time instant n should have less contribution fromthe data that are further away in time from the currenttime instant n.

The optimization objective function at time instantn, which tries to minimize the sum of weighted

(a) (b)

*

*

*

**

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

*

***

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

*

***

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

Fig. 5. Nonstationary data at (a) time n and (b) time n0.

u1(n)

m(n)

m(n)

u1(n ')

(a) (b)

*

*

*

**

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

*

*

*

**

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

*

*

*

**

*

*

*

*

*

*

*

*

**

*

***

*

**

*

*

**

*

*

***

***

***

***

**

*

**

* **

**

*

** * *

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

**

***

*

*

***

**

*

**

** * *

***

*

*

**

* *

***

** *

* * *

*

**

* ** **

**

** **

*

**

*

Fig. 6. UPC for nonstationary data at (a) time n and (b)time n0.

reconstruction errors of all data, can be written as

minm�n�,U�n�

1∑iD0

˛i

ð

∥∥∥∥∥∥∥∥∥∥∥∥∥xn�i �

m�n� CP∑

kD1[�xn�i �m�n��Tu�n�

k

]u�n�

k︸︷︷︸x̂n�i

∥∥∥∥∥∥∥∥∥∥∥∥∥

2

�1�

The notations are organized as follows:

n : Current time indexD : Dimension of the data vectorP : Number of eigenvectorsxn�i : Data vector at time n� i, where i represents

how far away the data are from the currenttime instant

m�n� : Mean at time nu�n�

k : kth eigenvector at time nU�n� : Matrix with P columns of u�n�

k , k D 1 ¾ Px̂n�i : Reconstruction of xn�i

˛ : Decay factor, 0 < ˛ < 1

The reconstruction errors contributed by previousdata are weighted by powers of the decay factor ˛.The powers are determined by how far away thissample of data is from the current time instant. Atany time instant n, we try to reestimate or update theparameter (mean or eigenvector) given the parameterestimated at the previous time instant n� 1 andthe new data xn, by minimizing Equation (1). Thesolution of mean m�n� that minimizes Equation (1) attime n is

m�n� D ˛m�n�1� C �1� ˛� xn �2�

We can see that m�n� is obtained from the previousestimated m�n�1� and the current input xn. The decayfactor ˛ tells how fast the new estimation m�n� adaptsto the new data xn. The smaller the decay factor,the faster the estimated m�n� adapts to the new data.Similarly, the covariance matrix C�n� that minimizesEquation (1) at time n is

C�n� D ˛C�n�1� C �1� ˛�[�xn �m�n��xn �m�n��T]�3�

Again, C�n� is obtained by the previous estimatedC�n�1� and the current input xn. The decay factor ˛controls how fast the eigenvectors adapt to the newdata xn. Interested readers can read the appendix in



Section A1 for ‘updating mixture of principal com-ponents’ (UMPC), a more general case of UPC withthe number of mixture components greater than one.

4.2. Spatial Error Concealment with UPC

4.2.1. UPC for ROI

As mentioned in Section 3, if ROI information isavailable, second-generation error concealment canbe applied to pixel values in the ROI with the trainedmodel. In this section, we consider spatial error con-cealment for Intra coded videos using UPC for ROI.

When the video decoder receives a video framewith an error-free ROI, it can use the data in the ROIto update the existing UPC model with the processesdescribed in Section 4.1. In this paper, all availableerror-free ROI are used to update the UPC model.Less frequent update to reduce the computationalcomplexity is possible at the expense of less adaptiv-ity. In the experiment, the time consumed on an IntelPentium III 650 PC to update the UPC model, withsix eigenvectors, is about 100 ms per ROI. Practicalsystem design can consider updating the UPC modelwith the incoming error-free ROI when the error torepresent this ROI with the current UPC model islarger than a threshold.

When the video decoder receives a frame of videowith corrupted macroblocks (MBs) in the ROI, it usesUPC to reconstruct the pixel values in this corruptedROI. We adopt the POCS- based error-concealmentscheme as illustrated in Figure 4. Iterations of projec-tions and replacements are repeated until the result issatisfactory. As to reconstructing the corrupted ROIwith the UPC model based on POCS, the time con-sumed on an Intel Pentium III 650 PC is almostnegligible with 5 ms per ROI.

Error concealment for Intra coded videos is sum-marized in Table I.

4.2.2. Experiment

Two test video sequences ‘Akiyo’ and ‘Interview’ areused. Both video sequences are in quarter common

Good Bad

1−p 1−qp

q

Fig. 7. Two-state Markov chain for error simulation.

Time 20 Time 22 Time 60

Mean

Firsteigenvector

Secondeigenvector

Thirdeigenvector

Fourtheigenvector

Fiftheigenvector

Sixtheigenvector

Fig. 8. Updated means and eigenvectors at time instants20, 22, and 60.

intermediate format (QCIF). The video codec usedin this paper is H.263 [45]. One sample frame of‘Akiyo’ with ROI specified is shown in Figure 1(a),and two frames of ‘Interview’ with ROI specifiedare shown in Figure 1(b) and (c). Note that ‘Inter-view’ consists of two different objects of character at

Table I. Error concealment for Intra coded videos: spatial error concealment with UPC for ROI.

Data lost Error concealment: training and reconstruction

Training for pixel values Reconstruction

DCT coefficients only Train UPC for ROI with ROI pixelvalues from frames with correctlyreceived DCT coefficients

Apply UPC to corrupted ROI pixelvalues if some DCT coefficientsin this ROI are lost



(a) (b) (c)

Fig. 9. Sample reconstructed frames of Intra coded ‘Akiyo’ with (a) no concealment; (b) concealment with spatialinterpolation; or (c) concealment with UPC for ROI.

different time instances. We use a two-state Markovchain [46] to simulate the bursty error to corruptthe DCT coefficients in MBs as shown in Figure 7.‘Good’ and ‘Bad’ correspond to error-free and erro-neous states, respectively. The overall error rate εis related to the transition probabilities p and q byε D p/�pC q�. We use ε D 0.1 and p D 0.01 in theexperiment. The UPC model used is with six eigen-vectors, P D 6. In the error-concealment stage whenerroneous MBs are received, five iterations of POCSare performed.

Figure 8 shows the means and eigenvectors ofUPC at three different time instants 20, 22, and60 for the sequence ‘Interview’. Notice that thereis a character change at time instant 21. The firstcharacter is in video frames from time 1 to 20 andthe second character is in video frames from time 21to 80. We can see that UPC describes more about thefirst character at time 20, as opposed to describingmore about the second character at time 60. TheUPC model shows a transition at time instant 22as expected.

Figure 9 shows sample reconstructed frames ofIntra coded ‘Akiyo’ with no concealment, spatialinterpolation based on POCS, or second-generationconcealment with UPC for ROI based on POCS.Spatial interpolation method interpolates the spatialneighbors of the lost MBs to recover the pixel val-ues. The white-bounded boxes in Figure 9 are to showthe ROI regions. All evaluations in peak signal-to-noise ratio (PSNR) are calculated inside the ROIregion. Figure 10 shows the frame-by-frame PSNRof the three methods. Figure 11 shows sample recon-structed frames of Intra coded ‘Interview’ with thesame three methods. Figure 12 shows the frame-by-frame PSNR of the three methods. The over-all PSNR comparisons are shown in Figure 13 forboth sequences.

10

15

20

25

30

35

Frame number

PS

NR

(dB

)

Akiyo

0 60 120 180 240 300

None Interpolation UPC

Fig. 10. Frame-by-frame PSNR of Intra coded ‘Akiyo’with no concealment (none), concealment with spatial

interpolation, or concealment with UPC for ROI.

We can observe that the ones without error conceal-ment, Figures 9(a) and 11(a), have bad visual quality.The ones with spatial interpolation, Figures 9(b) and11(b), provide some replenishment while missing thedetailed texture information in the lost regions of thereconstructed ROI. Figures 9(c) and 11(c), which useUPC for ROI, provide the best reconstruction resultsamong the three methods.

4.3. Hybrid Temporal/spatial Error Concealmentwith UPC

4.3.1. U-Eigenflow for MVs and UPC for ROI

In the Inter coded video bitstream, MVs as well asDCT coefficients can be lost. We apply U-Eigenflowfor MVs to reconstruct the lost MVs for tempo-ral error concealment. In addition, we further applyUPC for ROI for ROI that contain corrupted DCTcoefficients. Therefore, we propose a new hybridtemporal/spatial error-concealment scheme with U-Eigenflow for MVs and UPC for ROI for Intercoded videos.



(a) (b) (c)

Fig. 11. Sample reconstructed frames of Intra coded ‘Interview’ with (a) no concealment; (b) concealment with spatialinterpolation; or (c) concealment with UPC for ROI.

PS

NR

(dB

)

10

15

20

25

30

35

Frame number0 10 20 30 40 50 60 70 80

Interview

None Interpolation UPC

Fig. 12. Frame-by-frame PSNR of Intra coded ‘Interview’with no concealment (none), concealment with spatial

interpolation, or concealment with UPC for ROI.

When the video decoder receives an Intra codedvideo frame with error-free ROI, it uses the data inthe ROI to update the existing UPC model. It is calledUPC for ROI . When the video decoder receives anInter coded video frame with all the MVs correctinside, it uses the MVs to update the existing U-Eigenflow. It is called U-Eigenflow for MVs .

When the video decoder receives a video framewith lost MVs, it uses U-Eigenflow to reconstruct thelost MVs. Motion compensation is followed using

the reconstructed MVs. If no MV is lost, motioncompensation uses the correctly received MVs. Ifthere are lost DCT coefficients inside the ROI, UPCis further applied to the corrupted ROI. Both U-Eigenflow for MVs and UPC for ROI constitute thehybrid temporal/spatial error concealment for Intercoded videos.

Error concealment for Inter coded videos is sum-marized in Table II.

4.3.2. Experiment

The same two test video sequences ‘Akiyo’ and‘Interview’ are used as in the Intra coded case. Weuse the same two-state Markov chain to simulatethe bursty error to corrupt the DCT coefficients inMBs. The parameters are ε D 0.1 and p D 0.01. Asto simulate the bursty error to corrupt the MVs, weuse ε D 0.05 and p D 0.005 assuming that MVs areusually better protected than DCT coefficients.

U-Eigenflow for MVs and UPC for ROI both usesix eigenvectors, P D 6. Figure 14 shows the meansand the eigenvectors of U-Eigenflow at three differenttime instants 20, 22, and 60 for the sequence ‘Inter-view’. Notice again that there is a character change

Akiyo

31.09

PS

NR

(dB

)

Interview

15.75

29.73 30.22

PS

NR

(dB

)

(b)(a)10

15

20

25

30

35

10

15

20

25

30

35

NoneInterpolationUPC

NoneInterpolationUPC18.44

28.81

Fig. 13. Overall PSNR of Intra coded (a) ‘Akiyo’ and (b) ‘Interview’ with no concealment (none), concealment with spatialinterpolation, or concealment with UPC for ROI.



Table II. Error concealment for Intercoded videos: hybrid temporal/spatial error concealment with U-Eigenflow for MVs/UPC for ROI.

Data lost Error concealment: training and reconstruction

Training Reconstruction

MVs in addition to DCT coefficients 1. Training for MVs—Train U-Eigenflowfor MVs with correctly received MVs

1. Retain the lost MVs by U-Eigenflow ifsome MVs in a frame are lost

2. Training for pixel values—Train UPCfor ROI with ROI pixel values fromIntra frames with correctly receivedDCT coefficients

2. Perform motion compensation for thisframe with the retained MVs from Step1, or correctly received MVs

3. Apply UPC to ROI pixel values in themotion-compensated frame if someDCT coefficients in this ROI are lost

Time 20 Time 22 Time 60

Mean

Firsteigenvector

Secondeigenvector

Thirdeigenvector

Fourtheigenvector

Fiftheigenvector

Sixtheigenvector

Fig. 14. Updated mean and eigenvectors of U-Eigenflow at time instants 20, 22, and 60.



Interview

0.104

0.137

0.073

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

MA

D Spatial-nbrTemporal-nbrU-Eigenflow

(b)

Akiyo

0.0082 0.0096 0.0081

MA

D Spatial-nbrTemporal-nbrU-Eigenflow

(a)0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Fig. 15. MAD of reconstructed MVs of Inter coded (a) ‘Akiyo’ and (b) ‘Interview’ with spatial-nbr, temporal-nbr, orU-Eigenflow for MVs.

at time instant 21. We can see that U-Eigenflowdescribes more about the first character at time 20, asopposed to describing more about the second char-acter at time 60. The first character moves his headand body a lot, while the second character movesmainly with his head. The U-Eigenflow model showsa transition at time instant 22.

The two MV reconstruction methods for compari-son are ‘spatial-nbr’ and ‘temporal-nbr’. ‘Spatial-nbr’reconstructs the lost MVs for the center MB by tak-ing the median of the eight MVs of the neighboringMBs. ‘Temporal-nbr’ copies the MV of the same cen-ter MB from the previous frame. After MVs havebeen reconstructed using these three methods, motion

Fig. 16. Sample reconstructed frame of Inter coded‘Akiyo’ with U-Eigenflow for MVs and UPC for ROI.

compensation is performed. Finally, UPC for ROI isapplied to ROI pixel values if DCT coefficients in theROI are lost.

Figure 15(a) and (b) show the mean absolutedifference (MAD) of the MVs reconstructed bythese three methods for ‘Akiyo’ and ‘Interview’,respectively. ‘Akiyo’ has little motion throughout thesequence and all three methods can perform equallywell. Figure 15(b) shows that U-Eigenflow for MVsperforms the best among the three in ‘Interview’, inwhich MVs are larger.

Let us look at some sample MVs and samplereconstructed frames with the hybrid temporal/spatialerror concealment. As mentioned in the last para-graph, ‘Akiyo’ has little motion and all three meth-ods are equally good. Figure 16 shows the samplereconstructed frame of Inter coded ‘Akiyo’ with U-Eigenflow for MVs and UPC for ROI. Let us lookat the results of ‘Interview’. Figure 17 shows that‘temporal-nbr’ and U-Eigenflow for MVs reconstructthe MVs better than ‘spatial-nbr’. Figure 18 reflectsthe performance differences of these three meth-ods visually. Figure 19 shows that ‘spatial-nbr’ andU-Eigenflow for MVs reconstruct the MVs betterthan ‘temporal-nbr’. Figure 20 reflects the perfor-mance differences of these three methods visually.We can see that the face in Figure 20(b) has off-set MBs.

(a) (b) (c) (d) (e)

Fig. 17. Sample MVs at time 19 of Inter coded ‘Interview’: (a) regions indicating corrupted MVs (black blocks); (b) realMVs; (c) MVs reconstructed by spatial-nbr; (d) MVs reconstructed by temporal-nbr; and (e) MVs reconstructed by

U-Eigenflow for MVs.



(a) (b) (c)

Fig. 18. Sample reconstructed frame at time 19 of Inter coded ‘Interview’ with MVs reconstructed by (a) spatial-nbr; (b)temporal-nbr; and (c) U-Eigenflow for MVs, followed by UPC for ROI for all three MV reconstruction methods.

(a) (b) (c) (d) (e)

Fig. 19. Sample MVs at time 35 of Inter coded ‘Interview’: (a) regions indicating corrupted MVs (black blocks); (b) realMVs; (c) MVs reconstructed by spatial-nbr; (d) MVs reconstructed by temporal-nbr; and (e) MVs reconstructed by

U-Eigenflow for MVs.

(a) (b) (c)

Fig. 20. Sample reconstructed frame at time 35 of Inter coded ‘Interview’ with MVs reconstructed by (a) spatial-nbr; (b)temporal-nbr; and (c) U-Eigenflow for MVs, followed by UPC for ROI for all three MV reconstruction methods.

Akiyo

20

23

26

29

32

35

0 40 80 120 160 200 240 280

Frame number

PS

NR

(dB

)

Spatial-nbr Temporal-nbr U-Eigenflow

Fig. 21. Frame-by-frame PSNR of Inter coded ‘Akiyo’with MVs reconstructed by spatial-nbr, temporal-nbr, orU-Eigenflow for MVs, followed by UPC for ROI for all

three MV reconstruction methods.

Interview

15

20

25

30

35

0 10 20 30 40 50 60

Frame number

PS

NR

(d

B)

Spat ia l -nbr Temporal-nbr U-Eigenf low

Fig. 22. Frame-by-frame PSNR of Inter coded ‘Interview’with MVs reconstructed by spatial-nbr, temporal-nbr, orU-Eigenflow for MVs, followed by UPC for ROI for all

three MV reconstruction methods.



Akiyo

28.50 28.45 28.49

27.00

27.30

27.60

27.90

28.20

28.50

28.80

PS

NR

(dB

)

Spatial-nbrTemporal-nbrU-Eigenflow

Interview

23.70

22.97

23.83

22.20

22.60

23.00

23.40

23.80

24.20

PS

NR

(dB

)

Spatial-nbrTemporal-nbrU-Eigenflow

(a) (b)

Fig. 23. Overall PSNR of Inter coded (a) ‘Akiyo’ and (b) ‘Interview’ with MVs reconstructed by spatial-nbr, temporal-nbr,or U-Eigenflow for MVs, followed by UPC for ROI for all three MV reconstruction methods.

Figures 21 and 22 show frame-by-frame PSNRperformance of these three methods. The overallPSNR comparisons are shown in Figure 23 forboth sequences. The hybrid error concealment withU-Eigenflow for MVs and UPC for ROI providespromising results.

5. Conclusion

In this paper, we proposed a new second-generation error-concealment framework. Second-generation error-concealment methods train andreconstruct the lost video content by context-basedmodels and thus provide better error-concealmentresults than heuristic-based error-concealment meth-ods. ‘Updating principal components’ (UPC) wasproposed to construct such models. UPC can beapplied to reconstruct the lost motion vectors (MVs)as well as to replenish the corrupted pixel val-ues in the region of interest (ROI). The pro-posed hybrid temporal/spatial error concealment withU-Eigenflow for MVs and UPC for ROI providessuperior performance to conventional first-generationerror-concealment methods.

References

1. Wicker S. Error Control Systems for Digital Communicationand Storage. Prentice Hall: New York, 1995.

2. Lin S, Costello DJ, Miller MJ. Automatic repeat requesterror control schemes. IEEE Communication Magazine 1984;22(12): 5–17.

3. Bystrom M, Parthasarathy V, Modestino JW. Hybrid errorconcealment schemes for broadcast video transmission overATM networks. IEEE Transactions on CSVT 1999; 9(6):868–881.

4. Hemami SS. Robust image transmission using resynchronizingvariable-length codes and error concealment. IEEE Journal onSelected Areas in Communications 2000; 18(6): 927–939.

5. Aign S. Error concealment improvements for MPEG-2 usingenhanced error detection and early re-synchronization. ICASSP1997 1997; 4: 2625–2628.

6. Wang Y, Zhu Q-F. Error control and concealment for videocommunication: a review. Proceedings of the IEEE 1998;86(5): 974–997.

7. Valente S, Dufour C, Groliere F, Snook D. An efficient errorconcealment implantation for MPEG-4 video streams. IEEETransactions on Consumer Electronics 2001; 47(3): 568–578.

8. Delicado F, Cuenca P, Garrido A, Orozco-Barbosa L, Qui-les F. On the capabilities of error concealment in MPEG-2communications over wireless ATM. ICME 2000 2000; 3:1443–1446.

9. Cei S, Cosman P. Comparison of error concealment strategiesfor MPEG video. WCNC 1999 1999; 1: 329–333.

10. Turk M, Pentland A. Eigenfaces for recognition. Journal ofCognitive Neuroscience 1991; 3(1): 71–86.

11. Chen TP-C, Chen T. Updating mixture of principal compo-nents for error concealment. In ICIP 2002, Rochester NY; toappear.

12. Wang Y, Zhu Q, Shaw L. Coding and cell-loss recovery inDCT based packet video. IEEE Transactions on CSVT 1993;3(3): 248–258.

13. Hemami S, Meng T. Transform coded image reconstructionexploiting interblock correlation. IEEE Transactions on ImageProcessing 1995; 4(7): 1023–1027.

14. Suh J-W, Ho YS. Error concealment based on directionalinterpolation. IEEE Transactions on Consumer Electronics1997; 43(3): 295–302.

15. Zhu W, Wang Y, Zhu Q-F. Second-order derivative-basedsmoothness measure for error concealment. IEEE Transactionson CSVT 1998; 8(6): 713–718.

16. Zeng W, Liu B. Geometric structured based error concealmentwith novel applications in block-based low-bit-rate coding.IEEE Transactions on CSVT 1999; 9(4): 648–665.

17. Robie DL, Mersereau RM. The use of Hough transformsin spatial error concealment. ICASSP 2000 2000; 4:2131–2134.

18. Chung YJ, Kim J, Kuo CC-J. Real-time streaming video withadaptive bandwidth control and DCT-based error concealment.IEEE Transactions on CS II: Analog and Digital SignalProcessing 1999; 46(7): 951–956.

19. Sub H, Kwok W. Concealment of damaged block transformcoded images using projections onto convex sets. IEEETransactions on Image Processing 1995; 4: 470–477.

20. Yu G-S, Liu MM-K, Marcellin MW. POCS-based errorconcealment for packet video using multiframe overlapinformation. IEEE Transactions on CSVT 1998; 8(4):422–434.

21. Geman S, Geman D. Stochastic relaxation, Gibbs distributionand the Bayesian restoration of images. IEEE Transactions onPAMI 1984; PAMI-6: 721–741.



22. Salama P, Shroff N, Delp EJ. A Bayesian approach to errorconcealment in encoded video streams. ICIP 1996 1996; 1:49–52.

23. Salama P, Shroff N, Delp EJ. A fast suboptimal approach toerror concealment in encoded video streams. ICIP 1997 1997;2: 101–104.

24. Salama P, Shroff N, Delp EJ. Error concealment in MPEGvideo streams over ATM networks. IEEE Journal on SelectedAres in Communications 2000; 18(6): 1129–1144.

25. Shirani S, Kossentini F, Ward R. An adaptive Markovrandom field based on error concealment method for videocommunication in an error prone environment. ICASSP 19991999; 6: 3117–3120.

26. Zhang Y, Ma K-K. Error concealment for image transmissionby multiscale Markov random field modeling. ICME 20002000; 1: 513–516.

27. Zhang R, Regunathan SL, Rose K. Optimal estimation forerror concealment in scalable video coding. 2000 AsilomarConference on Signals, Systems and Computers 2000; 2:1374–1378.

28. Lam WM, Reibman A, Liu B. Recovery of lost or erroneouslyreceived motion vectors. ICASSP 1993 1993; 5: 417–420.

29. Park CS, Ye J, Lee SU. Lost motion vector recoveryalgorithm. ISCAS 1994 1994; 3: 229–232.

30. Feng J, Lo KT, Mehrpour H, Karbowiak AE. Cell lossconcealment method for MPEG video in ATM networks.GLOBECOM 1995 1995; 3: 1925–1929.

31. Feng J, Lo KT, Mehrpour H. Error concealment for MPEGvideo transmissions. IEEE Transactions on ConsumerElectronics 1997; 43(2): 183–187.

32. Suh J-W, Ho YS. Motion vector recovery for errorconcealment based on distortion modeling. 2001 AsilomarConference on Signals, Systems and Computers 2001; 1:190–194.

33. Zhang J, Arnold JF, Frater MR, Pickering MR. Video errorconcealment using decoder motion vector estimation.TENCON 1997 1997; 2: 777–780.

34. Zhang J, Arnold JF, Frater MR. A cell-loss concealmenttechnique for MPEG-2 coded video. IEEE Transactions onCSVT 2000; 10(4): 659–665.

35. Al-Mualla ME, Canagarajah N, Bull DR. Error concealmentusing motion field interpolation. ICIP 1998 1998; 3:512–516.

36. Al-Mualla ME, Canagarajah N, Bull DR. Multiple-referencetemporal error concealment. ISCAS 2001 2001; 5: 149–152.

37. Lee S-H, Choi D-H, Hwang C-S. Error concealment usingaffine transform for H.263 coded video transmissions.Electronics Letters 2001; 37(4): 218–220.

38. Motion Pictures Experts Group. Overview of the MPEG-4Standard. ISO/IEC JTC1/SC29/WG11 N2459, 1998.

39. Huang J, Chen T. Tracking of multiple faces for human-computer interfaces and virtual environments. In ICME 2000,New York, 2000.

40. Turaga DS, Chen T. Model-based error concealment forwireless video. IEEE Transactions on CSVT 2002; toappear.

41. Shirani S, Erol B, Kossentini F. Error concealment for MPEG-4 video communication in an error prone environment. ICASSP2000 2000; 4: 2107–2110.

42. Aizawa K, Huang TS. Model-based image coding: Advancedvideo coding techniques for very low bit-rate applications.Proceedings of the IEEE 1995; 83: 259–271.

43. Pearson DE. Developments in model-based video coding.Proceedings of the IEEE 1995; 83: 892–906.

44. Liu X, Chen T, Vijaya Kumar BVK. Face authenticationfor multiple subjects using eigenflow. Pattern Recognition.(Special Issue on Biometrics) 2001; in press.

45. ITU-T Recommendation H. 263, 27 January, 1998.

46. Yajnik M, Moon S, Kurose J, Towsley D. Measurement andmodeling of the temporal dependence in packet loss. InINFOCOM 1999, March 1999; pp. 345–352.

A1. Appendix

We can extend ‘updating principal components’(UPC) to more than one mixture components bymodifying the objective function in Equation (1) as

minm�n�

j,U�n�

j,wi

8i,j

( 11∑

iD0

˛i

) 1∑iD0

˛i

∥∥∥∥∥xn�i �M∑

jD1

wn�i,j

ð[

m�n�j C

P∑kD1

[�xn�i �m�n�j �Tu�n�

jk ]u�n�jk︸︷︷︸

x̂n�i,j

]∥∥∥∥∥2

�A1�

The notations are organized as follows:

n : Current time indexD : Dimension of the data vectorM : Number of mixture componentsP : Number of eigenvectors in each mixture

componentxi : Data vector at time im�n�

j : Mean of the jth mixture component esti-mated at time n

u�n�jk : kth eigenvector of the jth mixture compo-

nent estimated at time nU�n�

j : Matrix with P columns of u�n�jk , k D 1 ¾ P

x̂ij : Reconstruction of xi with mixture compo-nent j

X̂i : Matrix with M columns of x̂ij, j D 1 ¾ Mwij : Weight of x̂ij to reconstruct xi

wi : Vector with M entries of wij

˛ : Decay factor, 0 < ˛ < 1q, r : Index for the mixture component

The mean m�n�q of mixture component q at time n is

m�n�q D

1� w2

nq1∑

iD0

˛iw2n�i,q

m�n�1�

q

C

wnq1∑

iD0

˛iw2n�i,q

xn �

M∑jD1,j 6Dq

wnjx̂nj

�A2�



The covariance matrix C�n�r of mixture component r

at time n is

C�n�r D ˛C�n�1�

r C �1� ˛�

Ð�xn � wnrm�n�

r ��M∑

jD1,j 6Dr

wnjx̂nj

Ð [wnr�xn �m�n�r �]T C [wnr�xn �m�n�

r �]

Ð�xn � wnrm�n�

r ��M∑

jD1,j 6Dr

wnjx̂nj

T

� w2nr�xn �m�n�

r ��xn �m�n�r �T

�A3�

Finally, the solution for weights is

[wn�i

�

]D

[2X̂T

n�iX̂n�i 11T 0

]�1 [2X̂T

n�ixn�i

1

]�A4�

where 1 D [1 Ð Ð Ð 1]T is an Mð 1 vector. The deriva-tions of Equations (A2) to (A4) are described in thefollowing sections.

A1.1. Solution for Updating Mixture of PrincipalComponents

A1.1.1. Solution for the means

The optimization criterion in Equation (A1) can berewritten as

minm�n�

q

11∑

iD0

˛i

1∑iD0

˛ijjxn�i � X̂n�iwn�ijj2 �A5�

where X̂n�i is defined as

x̂n�i,j D U�n�j U�n�T

j �xn�i �m�n�j �Cm�n�

j

D

ID � U�n�

j U�n�T

j︸︷︷︸Aj

m�n�

j C U�n�j U�n�T

j︸︷︷︸Bj

xn�i

X̂n�i DM∑

jD1

x̂n�i,jeTj , where

ej D

0...1...0

� jth position

Note:

Since U�n�j U�n�T

j D IM

) ATj Aj D Aj D AT

j &ATj Bj D 0 �A6�

We can see that Bj is the projection matrix thatprojects data onto a subspace spanned by the eigen-vectors U�n�

j . Aj is the projection matrix that projectsdata onto a subspace that is orthogonal to the sub-space spanned by the eigenvectors U�n�

j . ReplacingX̂n�i in Equation (A5) with Aj and Bj in Equation(A6), we get

minm�n�

q

11∑

iD0

˛i

1∑

iD0

˛i

∥∥∥∥∥∥xn�i � M∑

jD1

�Ajm�n�j C Bjxn�i�eT

j

wn�i

∥∥∥∥∥∥2

�A7�Note that wT

n�iej D eTj wn�i D wn�i,j. Let us expand

Equation (A7) and drop the terms that are indepen-dent of m�n�

j to get

minm�n�

q

11∑

iD0

˛i

1∑

iD0

˛i

ð

�2M∑

jD1

wn�i,jm�n�T

j ATj xn�i

CM∑

jD1

M∑kD1

wn�i,jwn�i,km�n�T

j ATj Akmk

CM∑

jD1

M∑kD1

wn�i,jwn�i,km�n�T

j ATj Bkxn�i

CM∑

jD1

M∑kD1

wn�i,jwn�i,kxTn�iB

Tj Akm�n�

k

�A8�



To find the m�n�q that minimizes the optimization cri-

terion, we take the derivatives of Equation (A8) withrespect to m�n�

q and set the result to zero.

1∑iD0

˛i

�wn�i,qATq xn�i

CM∑

jD1,j 6Dq

wn�i,qwn�i,j

Ð ATq Ajm�n�

j C w2n�i,qAT

q Aqm�n�q

CM∑

kD1

wn�i,qwn�i,kATq Bkxn�i

D 0

�A9�

We simplify Equation (A9) and obtain the solutionas follows:

m�n�q D

11∑

iD0

˛iw2n�i,q

1∑

iD0

˛iwn�i,q

Ðxn�i �

M∑jD1,j 6Dq

wn�i,jx̂n�i,j

�A10�

Equation (A13) can also be written in a recur-sive form:

m�n�q D

1� w2

nq1∑

iD0

˛iw2n�i,q

m�n�1�

q

C

wnq1∑

iD0

˛iw2n�i,q

xn �

M∑jD1,j 6Dq

wnjx̂nj

�A11�

A1.1.2. Solution for the eigenvectors

The optimization criterion in Equation (A1) can berewritten as

minu�n�

rs

11∑

iD0

˛i

1∑iD0

˛i

∥∥∥∥∥∥xn�i �M∑

jD1

wn�i,j

Ð[

m�n�j C

P∑kD1


jk ]u�n�jk

]∥∥∥∥∥∥∥∥∥∥

2

�A12�

Let us expand Equation (A12) and drop the terms thatare independent of u�n�

jk to get

minu�n�

rs

11∑

iD0

˛i

1∑iD0

˛i

Ð

�2xTn�i

M∑jD1

wn�i,j

ÐM∑

kD1

�xn�i �m�n�j �Tu�n�

jk u�n�jk

C 2M∑

jD1

M∑aD1

wn�i,jwn�i,am�n�T

j

ÐP∑

bD1

�xn�i �m�n�a �Tu�n�

ab u�n�ab

CM∑

jD1

M∑aD1

wn�i,jwn�i,a

ÐP∑

kD1

P∑bD1


jk ]

Ð [�xn�i �m�n�a �Tu�n�

ab ]�u�n�T

jk u�n�ab �

�A13�

Now let us simply Equation (A13) with the knowl-edge that eigenvectors of the same mixture compo-nent are orthogonal to each other.

minu�n�

rs

11∑

iD0

˛i

1∑iD0

˛i



Ð

�2M∑

jD1

wn�i,j

M∑kD1


jk ]

Ð �xTn�iu

�n�jk �C 2

M∑jD1

M∑aD1

wn�i,jwn�i,a

ÐP∑

bD1

[�xn�i �m�n�a �Tu�n�

ab ]�m�n�T

j u�n�ab �

CM∑

jD1

M∑aD1,a 6Dj

wn�i,jwn�i,a

ÐP∑

kD1

P∑bD1


jk ]

Ð [�xn�i �m�n�a �Tu�n�

ab ]�u�n�T

jk u�n�ab �

CM∑

jD1

w2n�i,j

P∑kD1


jk ]2

�A14�

With the additional constraint that eigenvectorsneed to be normal,

u�n�T

jk u�n�jk D 1 �A15�

we can apply Lagrange optimization algorithm to findout the eigenvectors u�n�

rs that minimizes the optimiza-tion criterion. Let us abbreviate the terms inside Equa-tion (A14) as �Ł�, the Lagrangian function is therefore,

�Ł�C ��u�n�T

jk u�n�jk � 1� �A16�

Taking the derivatives of the Lagrangian functionwith respect to u�n�

rs , we get,

11∑

iD0

˛i

1∑iD0

˛i

Ð

�2wn�i,r[�xn�i �m�n�r �xT

n�i C xn�i�xn�i

�m�n�r �T]u�n�

rs C 2M∑

jD1

wn�i,jwn�i,r[�xn�i

�m�n�r �m�n�T

j Cm�n�j �xn�i �m�n�

r �T]u�n�rs

C2M∑

jD1,j 6Dr

wn�i,jwn�i,r

P∑kD1


jk ]

Ð[�xn�i �m�n�r �u�n�T

jk C u�n�jk �xn�i �m�n�

r �T]u�n�rs

C2w2n�i,r�xn�i �m�n�

r ��xn�i �m�n�r �Tu�n�

rs

D �2�u�n�rs �A17�

We can see that u�n�rs is the eigenvector of the follow-

ing matrix C�n�r :

C�n�r D

11∑

iD0

˛i

1∑iD0

˛iwn�i,r

Ð

[�xn�i �m�n�r �xT

n�i C xn�i�xn�i �m�n�r �T]

�M∑

jD1

wn�i,j[�xn�i �m�n�r �m�n�T

j

Cm�n�j �xn�i �m�n�

r �T]�M∑

jD1,j 6Dr

wn�i,j

ÐP∑

kD1


jk ]

Ð [�xn�i �m�n�r �u�n�T

jk C u�n�jk �xn�i �m�n�

r �T]

� wn�i,r�xn�i �m�n�r ��xn�i �m�n�

r �T

�A18�

The first P eigenvectors of C�n�r are the solution for

u�n�rs , s D 1 ¾ P. Equation (A18) can also be written

in a recursive form:


r C �1� ˛�wnr

Ð

[�xn �m�n�r �xT

n C xn�xn �m�n�r �T]

�M∑

jD1

wnj[�xn �m�n�r �m�n�T

j

Cm�n�j �xn �m�n�

r �T]�M∑

jD1,j 6Dr

Ð wnj

P∑kD1

[�xn �m�n�j �Tu�n�

jk ]

Ð [�xn �m�n�r �u�n�T

jk C u�n�jk �xn �m�n�

r �T]

� wnr�xn �m�n�r ��xn �m�n�

r �T

�A19�

We can rearrange Equation (A19) and get


r C �1� ˛�



Ð

wnr�xn �m�n�

r �CM∑

jD1,j 6Dr

wnj�xn � x̂nj�

Ð [wnr�xn �m�n�r �]T C [wnr�xn �m�n�

r �]

Ðwnr�xn �m�n�

r �CM∑

jD1,j 6Dr

Ð wnj�xn � x̂nj�

T

� w2nr�xn �m�n�

r �

Ð �xn �m�n�r �T

�A20�

A1.1.3. Solution for the weights

The weights are solved individually for each of thevectors xn�i. We may drop the summation over allvectors. The optimization criterion in Equation (A1)can be rewritten as

minwn�i

∥∥∥∥∥∥xn�i �M∑

jD1

wn�i,jx̂n�i,j

∥∥∥∥∥∥2

D minwn�i

jjxn�i � X̂n�iwn�ijj2 �A21�

There is also the constraint that the weights wn�i

for xn�i should be summed up to one. Again,using the Lagrange optimization algorithm, we obtainLagrangian function,

�xn�i � X̂n�iwn�i�T�xn�i � X̂n�iwn�i�

C ��wTn�i1� 1� �A22�

where 1 D [1 Ð Ð Ð 1]T is an Mð 1 vector. Taking thederivatives of Equation (A22) with respect to wn�i

and � and setting the result to zero, we get,[2X̂T

n�iX̂n�i 11T 0

] [wn�i

�

]D

[2X̂T

n�ixn�i

1

]�A23�

The solution for weights is therefore[wn�i

�

]D

[2X̂T

n�iX̂n�i 11T 0

]�1 [2X̂T

n�ixn�i

1

]�A24�

Authors’ Biographies

Trista Pei-chun Chen received the B.S. degree andthe M.S. degree from National Tsing Hua University,Hsinchu, Taiwan, in 1997 and 1999, respectively. SinceAugust 1999, she has been working towards her Ph.D.degree in Electrical and Computer Engineering at CarnegieMellon University, Pittsburgh, Pennsylvania. From July1998 to June 1999, she was a software engineer developingfingerprint identification algorithms at Startek EngineeringIncorporated, Hsinchu, Taiwan. During the summer of2000, she was with HP Cambridge Research Laboratory,Cambridge, Massachusetts, conducting research in imageretrieval for massive databases. During the summer of 2001,she was with Pittsburgh Sony Design Center, Pittsburgh,Pennsylvania, designing circuits for Video Watermarking(VWM). Her research interests are in the areas ofnetworked video, watermark/data hiding, image processing,and biometric signal processing. She is a student memberof the IEEE.

Tsuhan Chen has been with the Department of Electricaland Computer Engineering, Carnegie Mellon University,Pittsburgh, Pennsylvania, since October 1997, wherehe is now a Professor. He directs the AdvancedMultimedia Processing Laboratory. His research interestsinclude multimedia signal processing and communication,audio-visual interaction, biometrics, processing of 2D/3Dgraphics, bioinformatics, and building collaborative virtualenvironments. From August 1993 to October 1997,he worked in the Visual Communications ResearchDepartment, AT&T Bell Laboratories, Holmdel, NewJersey, and later at AT&T Labs-Research, Red Bank,New Jersey.

Tsuhan helped create the Technical Committee onMultimedia Signal Processing, as the founding chair, andthe Multimedia Signal Processing Workshop, both in theIEEE Signal Processing Society. He has recently beenappointed as the Editor-in-Chief for IEEE Transactionson Multimedia for 2002–2004. He has co-edited abook titled Advances in Multimedia: Systems, Standards,and Networks.

Tsuhan received the B.S. degree in electrical engineeringfrom the National Taiwan University in 1987, and theM.S. and Ph.D. degrees in electrical engineering from theCalifornia Institute of Technology, Pasadena, California,in 1990 and 1993, respectively. He is a recipient of theNational Science Foundation CAREER Award.


Date post:	22-Sep-2020
Category:	Documents
Upload:	others
View:	6 times
Download:	0 times

Second-generation error concealment for video transport...

Documents