REDUCING BLURRING-EFFECT IN HIGH RESOLUTION MOSAIC GENERATION

RamazanSavas¸ AygunandAidongZhang

Departmentof ComputerScienceandEngineeringStateUniversityof New York at Buffalo

Buffalo,NY 14260-2000�aygun,azhang � @cse.buffalo.edu

ABSTRACT

The mosaicgenerationmethodsbenefitfrom recentglobalmotionestimation(GME) methods,which yield almostac-curateestimationof motionparameters.However, thegen-eratedmosaicsareusuallymoreblurredthanoriginalframesdueto imagewarpingstageanderrorsin motionestimation.ThetransformedcoordinatesresultingfromGME aregener-ally realnumberswhereasimagesaresampledinto integervalues. Although GME methodsgeneratepropermotionparameters,a slight error in motion estimationmay prop-agateto subsequentmosaicgenerationsteps. In this pa-per, we proposea methodto generateclearermosaicsfromvideo.Thetemporalintegrationof imagesis performedus-ing the histemporal filter basedon the histogramof valueswithin an interval. The initial framein the videosequenceis registeredat a higherresolutionto generatehigh resolu-tion mosaic. Insteadof warpingof eachframe,the framesarewarpedinto the mosaicat intervals. This reducestheblurring in themosaic.

1. INTRODUCTION

Mosaicgenerationhasbeenstudiedfor bothcontent-basedretrieval andvideo compression.MPEG-4[1] enablesde-codingandencodingof layeredspritesfor the objectsandthebackground.Thedifferenttypesof mosaicsandmosaicgenerationmethodsarecoveredin [2, 3]. The initial stageof amosaicgenerationis GlobalMotion Estimation(GME).Themotionis usuallymodeledusingperspective,affine,translation-zoom-rotation,or translationalmotion models.Most of the GME techniquesconcentrateon the accuracyof motionparametersof thechosenmotionmodels[4, 5, 6].Thesemethodsusually includean initial estimationof thesubsetof the motion parametersand thenadjustingof themotionparametersusingahierarchicalpyramidof low-passfilteredimages.

This researchis supportedby NSFgrantIIS-9733730andNSFgrantIIS-9905603.

Differentrepresentationsof mosaicslikestatic,dynamicandsynopsismosaichave beeninvestigatedin [2]. Directmethodsare applied to align imagesand to generatethemosaic. A spritecreationmethodbasedon connectedop-eratorsis presentedin [7]. A detailedwork onestimationofmotion parametersandgenerationof spriteshasbeenpre-sentedin [5]. A highresolutionmosaicis generatedby slid-ing themosaicandwarpingthenext frameinto themosaic[6]. Sincewarpingoccursfor every frame, the generatedmosaiccanstill beblurred.Moreover, temporalintegrationmethodsareusedaccordingto the type of the mosaicthatwill be generated.The temporalintegrationmethodsalsocausesblurring in themosaic.

In this paper, we proposea methodfor generatinghighresolutionmosaicfrom video. The framesare integratedusingthehistemporal filter. Thehistemporal filter is a gen-eralizedfilter andkeepsthe histogramof valuesthat mapto a specificinterval. The initial mosaicis maintainedat ahigherresolutionto reducethe blurring dueto real-valuedtransformedcoordinates.The framesare warpedinto themosaicat intervals. Sincewarpingof framesis performedusingbilinearinterpolation,a low-passeffect is introduced.Therefore,ignoringunnecessaryframesyieldsclearmosaicgeneration.After developingour method,experimentsareconductedon standardMPEGtestsequences.

This paperis organizedas follows. The motion esti-mation is explainedin Section2. High resolutionmosaicgenerationandhistemporal filter arepresentedin Section3.Theexperimentsandresultsarereportedin Section4. Thelastsectionconcludesour paper.

2. MOTION ESTIMATION

Themosaicshouldincludeeverysectionthatisvisiblethrough-out thevideosequence.If thereis no a priori motioninfor-mationfor avideosequence,themotionhasto beestimatedbetweeneachsequentialframe.

Therearedifferenttypesof motionmodelsthatareusedin GME dependingon thecameraoperationsandthestruc-ture of the scene. In this paper, we detectcameramotion

thatis parameterizedby perspectivemotionmodel:�������� ����� ������������������ � ������� � ��������� �! ����"#��������$ ���� � ������� � ������� (1)

where %'& , % � , %)( , %)* , %,+ , %)- , %). , and %)/ aremotionparam-etersand 0 �1��#2 ����43 is thetransformedcoordinatefor 0 � � 2 � � 3 .This modelturnsinto affine motion when 05%'. �7682 %)/ �6,3 , translation-zoom-rotationmotionwhen 05% + �:9 % * 2 % - �% ( 2 % . �;6<2 % / �;6'3 , and translationalmotion when0=% ( �?>@2 % - �:>,2 % + �A6<2 % - �A6<2 % . �B682 % / �A6,3 .

Theerrorbetweentwo framescanbedeclaredasC �EDBFHG (� (2)

whereG � �BI'� 0 �1��#2 ����53@9JI 0 � � 2 � � 3 , I 0 � � 2#� � 3 is theintensityat0 � � 2#� � 3 in thepreviousframe,and I,� 0 �1��#2 ����43 is theintensityat the transformedcoordinatein thecurrentframe. Error Cis computedfor pixelsoverlappingin two frames.

Theiterativedescentmethodsarelikely to betrappedinlocal minima. Our mosaicgenerationmethodskipssomeof theframesto reduceblurring in themosaic.Themotionestimationis performedbetweeneachsequentialframeandalsobetweenframesatspecificintervals.Motion estimationbetweenfartherframesis moreproneto errorsdue to theinitial estimationand possiblelarge displacement.Accu-mulatedmotionparametersor relative motionwith respectto the initial framein the interval is a goodapproximationof themotionparameters.In matrix form, affinemotiones-timationcanbewrittenas

K � ����ML � K %)(N%'*% + % - LMO K � �PLRQ K %'&% � LSO (3)

More generally, thiscanbewrittenasT � �AUET QWV (4)

where U containsthe motion parametersfor the first ma-trix and V containsthetranslationalparameters.Therelativemotionis computedasT � � �AU � 0 UET QWV 3 QXV � �YU � UET Q 0 U � V�QXV � 3 (5)

whereT)� � is thevectorfor thenew transformedcoordinates;U:� and V � hold thecurrentmotionparameters;and U and Vhold themotionparametersup to thecurrentframe.

To increasetherobustnessof motionestimation,weuseM-estimators[4, 5] andtheerroris expressedas:FZ [ 0 G � 3 (6)

where[ 0 G � 3\�]G (� in the original formulation. Sincethis

function givesmoreweight to large errors,it is biasedby

local motion(which areoutliersfor globalmotion). To de-creasetheeffect of outliers,thetruncatedquadraticmotionis used: [ 0 G � 3^�E_ G (� 2 if ` G � `)a V682 if ` G � `)b V (7)

where V is a thresholdselectedaccordingto the histogramof theerrors.

3. MOSAIC GENERATION

3.1. Histemporal Filter

The linear temporalfilters like averaging,recursive filterslike Kalmanfilter [8], andorder-statisticfilters like medianfilters have beenusedfor noisereductionor removal in im-agesequences.Temporalaveragingyields a blurred mo-saic,if thevideoincludesmoving objectsor motioncannotbe estimatedaccurately. Median filters requireenormousstorageto detectthetemporalmedianfilter. Moreover, tem-poral medianmay yield erroneousresults,if the expectedmediancantakeseveralvalues.For example,thefrequencyof pixel values100 and101 is 20 and22 for a pixel coor-dinatein themosaic,respectively. Althoughthis differencemayresultfrom theilluminancechangein theenvironment,they are treatedas different. If the frequency of anotherpixel valueis 25, this valuewill be chosenby mistake. Infact,averaging(of 100and101)would yield a betterresult.Histemporal filter is a temporal filter that is basedon thehistogramof intensityvalueswithin a specificinterval.

The ced V G�fgT %)h determinesthe precisionof temporalin-tegrationin mosaicgeneration.For a 8-bit per pixel gray-scaleimage,all the pixels lay in i 682�j@k,kgl . Therewill bem (�-#.�onqp=rts�u ��v#w slots in the histogram. If ced V GxfgT %)h is j,kzy , his-temporalfilter becomestemporalaveraging.If c4d V G�fgT %'h is1, thetemporalinterval becomestemporalmedianfilter.

Two datastructuresareusedto obtain the histemporalfilter: frequencyarrayandaverage array. Frequencyarraykeepsthe frequency of eachinterval of the histogram.Astheframesareprocessed,thefrequency of aninterval is in-creasedfor eachpixel valuebelongingto theinterval. Aver-agearraymaintainstheaverageof thevaluesasnew valuesareprocessedfor eachslot. Histemporalfilter returnstheaveragevalueof the interval having the highestfrequency.Figure1 (a) shows a histogramwhere ced V GxfgT %)h is 16. Theinterval i { >@2#|,yql hasthehighestfrequency. Figure1 (b) dis-playsthe frequenciesof thevaluesthat lay in this interval.During histemporalfilter computation,theaverageof thesevaluesis computedasthey arrive.

3.2. High Resolution Mosaic Generation

Motion parametersthat are obtainedfrom Equation1 areusuallyrealnumbersandyield real-valuedtransformedco-ordinates.Theoriginal imagesaresampledinto integerdo-

Figure1: Histemporalfilter.

main. The ordinary techniquescreatea mosaichaving aresolutionof the initial frame in the sequence.The pixellocationsin the mosaicmay not correspondto the integer-valuedpixel locationsin the new frame. Approacheslikebilinearinterpolationareusedto estimatethepixel valueatthelocation.Bilinear interpolationtakestheweightedaver-ageof theclosestpixelsandblurstheimage.

A high resolutionvideo mosaickingapproachis pro-posedin [9]. A high resolutionmosaicis generatedwherea mosaicalsocontainshalf-peldata.Whena new frameisprocessed,a shift (diagonal,vertical, or horizontal)on themosaicis assumed,and the frame is warpedinto the cor-respondingareain the mosaic. This usuallypreservestheoriginal sharpnessof the image. However, this approachdoesnot considerthe precisionof the transformedcoordi-natesandwarpingstill occursata low resolutionbecauseofshifting. In ourcase,warpingoccursathighresolution(Fig-ure2). Every pixel in thewarpingregion is updatedduringwarping.

Figure2: High resolutionmosaic.

Themotionparametersarealsoaffectedby themovingobjectsandapertureproblem. This causessomedeviationfrom the original valuesof motion parameters.Whenthemotion estimationis performedfrom frame to frame, theerror accumulatesandpropagatesto the later motion esti-mationandwarping. In addition,warpingat every framealsointroducesblurring. Thus,insteadof warpingat every

frame, the framesarewarpedinto the mosaicat intervals.But themotionestimationhasto beperformedfor eachse-quential frame. The previous frame is mappedfrom themosaicto avoid error accumulation.Threethresholdsareused: maximumaccumulateddisplacement0=}\%)~ 3 , maxi-mumscalefactor 0=}��g� 3 andmaximuminterval length 0�}\ceh 3 .The motion betweenconsecutive framesare accumulateduntil thedisplacementis lessthan }�%)~ andscale(zooming)factoris lessthan }��g� . Otherwise,themotionbetweenthefirst frameandthe last frame in the interval may increasesignificantly, andmotionestimationmethodsmayyield lessaccurateparameters.If thereis no significantmotionin thesequence,the relative motion is computedfor at most }\cehframes.This upper-boundis neededto remove the objectsfrom thebackgroundmosaic.Theframeis alsowarpedintothemosaicwhenthedirectionof cameramotionchanges.

4. EXPERIMENTS

In our experiments,theresolutionof themosaicis twice asthe initial frameof the sequence,thusresultingin half-pelaccuracy. The }�%)~ and }�cth arebothselectedas10. Figure3 shows anordinaryblurredmosaicgeneratedfrom ’coast-guard’MPEGtestsequence.If themotioncanbemodeledusingtranslationalmodel,theimagescanbewarpedaccord-ing to the precisionof transformedcoordinates. MPEG-4 test sequence’coastguard’can be modeledusing trans-lational model. Figure 4 shows the high resolutionback-groundmosaicgeneratedafter 300 frames. No segmen-tation mask is usedin the generation. The water textureis smoothedbecauseof temporaltexture,andhasbeenre-movedfrom themosaic.Theright sideof thefigureincludespartsthatarenot filled by frames.Thereforethe right sidelooks darker. Theselocationsarefilled with bilinear inter-polation. Thesmoothedregionsin theordinarymosaicareclearlyvisible in thehigh resolutionmosaic.

Figure3: OrdinaryMosaic.

Thereis no standardizedperformancetestsfor generat-ing mosaics.Themostcommonmethodis averagingPSNRvaluesfor a video. Although PSNRis a good indicationof similarity betweenimages,averageof PSNRvaluesisnot alwaysa goodmeasurefor video. Figure5 shows themosaicgeneratedfor ’foreman’ MPEGtestsequencefromframes195 to 240. The correspondingPSNRvaluesforframesthataregeneratedfrom high resolutionmosaicandordinarymosaicaregivenin Figure6. We have usedaffinemotionmodelfor ’foreman’ sequence.

Figure4: High resolutionmosaicfrom coastguard.

Figure5: Mosaicgeneratedfrom ’foreman’.

195 200 205 210 215 220 225 230 235 24023.5

24

24.5

25

25.5

26

26.5

27

27.5

28PSNR for foreman

PS

NR

in d

b

frame numbers

High resolution affineOrdinary affine

Figure6: PSNRfor foremansequence.

5. CONCLUSION

In thispaper, wepresentedamethodfor highresolutionmo-saicgenerationfrom video.Motion estimationis performedbetweeneachconsecutive frame not to miss visible areasin the sequencefor mosaicgeneration.Theblurring in themosaicgenerationis reducedby warpingat intervalsandatahigherresolution.Althoughhigh resolutionmosaicwarp-ing increaseselapsedtime, this is compensatedby warpingat intervals.Thenumberof framesthatareskippedcanfur-therbedecreased,but this is left asa furtherresearch.

6. REFERENCES

[1] T. Sikora, “The mpeg-4 video standardverificationmodel,” IEEE Trans.Circuits Syst.VideoTechnology,vol. 7, pp.19–31,February1997.

[2] M. Irani and P. Anandan, “Video indexing basedonmosaicrepresentations,” in Proceedingsof IEEE, May1998,pp.905–921.

[3] R. Szeliski and H-Y. Shum, “Creating full viewpanoramicimagemosaicsandenvironmentmaps,” inComputerGraphicsProceedings,Annual ConferenceSeries, 1997,pp.251–258.

[4] F. Dufaux andJ. Konrad, “Efficient, robust, and fastglobal motion estimationfor video coding,” IEEETransactionson Image Processing, vol. 9, no. 3, pp.497–501,March2000.

[5] T. SikoraA. SmolicandJ.-R.Ohm, “Long-termglobalmotionestimationandits applicationfor spritecoding,contentdescriptionandsegmentation,” IEEE Transac-tionsonCircuitsandSystemsfor VideoTechnology, vol.9, no.8, pp.1227–1242,December1999.

[6] A. Smolic andJ.-R.Ohm, “Robust global motion es-timation usinga simplified m-estimatorapproach,” inProc.ICIP2000,IEEEInternationalConferenceonIm-ageProcessing, September2000.

[7] P. Salembier, O. Pujol, andL.Garrido, “Connectedop-eratorsfor spritecreationandlayeredrepresentationofimagesequences,” in IV EuropeanSignalProcessingConference, September1998,pp.2105–2108.

[8] O. Munkelt C. Ridder and H. Kirchner, “Adaptivebackgroundestimationand foreground detectionus-ing kalman-filtering,” in Proceedingsof InterenationalConferenceon RecentAdvancesin Mechatronics, June1995,pp.193–199.

[9] A. SmolicandT. Wiegand,“High-resolutionvideomo-saicing,” in Proc. ICIP2001,IEEE InternationalCon-ferenceon ImageProcessing, October2001.