REDUCING BLURRING-EFFECT IN HIGH RESOLUTION MOSAIC GENERATION Ramazan Savas ¸ Ayg¨ un and Aidong Zhang Department of Computer Science and Engineering State University of New York at Buffalo Buffalo, NY 14260-2000 aygun,azhang @cse.buffalo.edu ABSTRACT The mosaic generation methods benefit from recent global motion estimation (GME) methods, which yield almost ac- curate estimation of motion parameters. However, the gen- erated mosaics are usually more blurred than original frames due to image warping stage and errors in motion estimation. The transformed coordinates resulting from GME are gener- ally real numbers whereas images are sampled into integer values. Although GME methods generate proper motion parameters, a slight error in motion estimation may prop- agate to subsequent mosaic generation steps. In this pa- per, we propose a method to generate clearer mosaics from video. The temporal integration of images is performed us- ing the histemporal filter based on the histogram of values within an interval. The initial frame in the video sequence is registered at a higher resolution to generate high resolu- tion mosaic. Instead of warping of each frame, the frames are warped into the mosaic at intervals. This reduces the blurring in the mosaic. 1. INTRODUCTION Mosaic generation has been studied for both content-based retrieval and video compression. MPEG-4 [1] enables de- coding and encoding of layered sprites for the objects and the background. The different types of mosaics and mosaic generation methods are covered in [2, 3]. The initial stage of a mosaic generation is Global Motion Estimation (GME). The motion is usually modeled using perspective, affine, translation-zoom-rotation, or translational motion models. Most of the GME techniques concentrate on the accuracy of motion parameters of the chosen motion models [4, 5, 6]. These methods usually include an initial estimation of the subset of the motion parameters and then adjusting of the motion parameters using a hierarchical pyramid of low-pass filtered images. This research is supported by NSF grant IIS-9733730 and NSF grant IIS-9905603. Different representations of mosaics like static, dynamic and synopsis mosaic have been investigated in [2]. Direct methods are applied to align images and to generate the mosaic. A sprite creation method based on connected op- erators is presented in [7]. A detailed work on estimation of motion parameters and generation of sprites has been pre- sented in [5]. A high resolution mosaic is generated by slid- ing the mosaic and warping the next frame into the mosaic [6]. Since warping occurs for every frame, the generated mosaic can still be blurred. Moreover, temporal integration methods are used according to the type of the mosaic that will be generated. The temporal integration methods also causes blurring in the mosaic. In this paper, we propose a method for generating high resolution mosaic from video. The frames are integrated using the histemporal filter. The histemporal filter is a gen- eralized filter and keeps the histogram of values that map to a specific interval. The initial mosaic is maintained at a higher resolution to reduce the blurring due to real-valued transformed coordinates. The frames are warped into the mosaic at intervals. Since warping of frames is performed using bilinear interpolation, a low-pass effect is introduced. Therefore, ignoring unnecessary frames yields clear mosaic generation. After developing our method, experiments are conducted on standard MPEG test sequences. This paper is organized as follows. The motion esti- mation is explained in Section 2. High resolution mosaic generation and histemporal filter are presented in Section 3. The experiments and results are reported in Section 4. The last section concludes our paper. 2. MOTION ESTIMATION The mosaic should include every section that is visible through- out the video sequence. If there is no a priori motion infor- mation for a video sequence, the motion has to be estimated between each sequential frame. There are different types of motion models that are used in GME depending on the camera operations and the struc- ture of the scene. In this paper, we detect camera motion
Transcript
REDUCING BLURRING-EFFECT IN HIGH RESOLUTION MOSAIC GENERATION
RamazanSavas¸ AygunandAidongZhang
Departmentof ComputerScienceandEngineeringStateUniversityof New York at Buffalo
The mosaicgenerationmethodsbenefitfrom recentglobalmotionestimation(GME) methods,which yield almostac-curateestimationof motionparameters.However, thegen-eratedmosaicsareusuallymoreblurredthanoriginalframesdueto imagewarpingstageanderrorsin motionestimation.ThetransformedcoordinatesresultingfromGME aregener-ally realnumberswhereasimagesaresampledinto integervalues. Although GME methodsgeneratepropermotionparameters,a slight error in motion estimationmay prop-agateto subsequentmosaicgenerationsteps. In this pa-per, we proposea methodto generateclearermosaicsfromvideo.Thetemporalintegrationof imagesis performedus-ing the histemporal filter basedon the histogramof valueswithin an interval. The initial framein the videosequenceis registeredat a higherresolutionto generatehigh resolu-tion mosaic. Insteadof warpingof eachframe,the framesarewarpedinto the mosaicat intervals. This reducestheblurring in themosaic.
1. INTRODUCTION
Mosaicgenerationhasbeenstudiedfor bothcontent-basedretrieval andvideo compression.MPEG-4[1] enablesde-codingandencodingof layeredspritesfor the objectsandthebackground.Thedifferenttypesof mosaicsandmosaicgenerationmethodsarecoveredin [2, 3]. The initial stageof amosaicgenerationis GlobalMotion Estimation(GME).Themotionis usuallymodeledusingperspective,affine,translation-zoom-rotation,or translationalmotion models.Most of the GME techniquesconcentrateon the accuracyof motionparametersof thechosenmotionmodels[4, 5, 6].Thesemethodsusually includean initial estimationof thesubsetof the motion parametersand thenadjustingof themotionparametersusingahierarchicalpyramidof low-passfilteredimages.
This researchis supportedby NSFgrantIIS-9733730andNSFgrantIIS-9905603.
Differentrepresentationsof mosaicslikestatic,dynamicandsynopsismosaichave beeninvestigatedin [2]. Directmethodsare applied to align imagesand to generatethemosaic. A spritecreationmethodbasedon connectedop-eratorsis presentedin [7]. A detailedwork onestimationofmotion parametersandgenerationof spriteshasbeenpre-sentedin [5]. A highresolutionmosaicis generatedby slid-ing themosaicandwarpingthenext frameinto themosaic[6]. Sincewarpingoccursfor every frame, the generatedmosaiccanstill beblurred.Moreover, temporalintegrationmethodsareusedaccordingto the type of the mosaicthatwill be generated.The temporalintegrationmethodsalsocausesblurring in themosaic.
In this paper, we proposea methodfor generatinghighresolutionmosaicfrom video. The framesare integratedusingthehistemporal filter. Thehistemporal filter is a gen-eralizedfilter andkeepsthe histogramof valuesthat mapto a specificinterval. The initial mosaicis maintainedat ahigherresolutionto reducethe blurring dueto real-valuedtransformedcoordinates.The framesare warpedinto themosaicat intervals. Sincewarpingof framesis performedusingbilinearinterpolation,a low-passeffect is introduced.Therefore,ignoringunnecessaryframesyieldsclearmosaicgeneration.After developingour method,experimentsareconductedon standardMPEGtestsequences.
This paperis organizedas follows. The motion esti-mation is explainedin Section2. High resolutionmosaicgenerationandhistemporal filter arepresentedin Section3.Theexperimentsandresultsarereportedin Section4. Thelastsectionconcludesour paper.
2. MOTION ESTIMATION
Themosaicshouldincludeeverysectionthatisvisiblethrough-out thevideosequence.If thereis no a priori motioninfor-mationfor avideosequence,themotionhasto beestimatedbetweeneachsequentialframe.
Therearedifferenttypesof motionmodelsthatareusedin GME dependingon thecameraoperationsandthestruc-ture of the scene. In this paper, we detectcameramotion
whereG � �BI'� 0 �1��#2 ����53@9JI 0 � � 2 � � 3 , I 0 � � 2#� � 3 is theintensityat0 � � 2#� � 3 in thepreviousframe,and I,� 0 �1��#2 ����43 is theintensityat the transformedcoordinatein thecurrentframe. Error Cis computedfor pixelsoverlappingin two frames.
Theiterativedescentmethodsarelikely to betrappedinlocal minima. Our mosaicgenerationmethodskipssomeof theframesto reduceblurring in themosaic.Themotionestimationis performedbetweeneachsequentialframeandalsobetweenframesatspecificintervals.Motion estimationbetweenfartherframesis moreproneto errorsdue to theinitial estimationand possiblelarge displacement.Accu-mulatedmotionparametersor relative motionwith respectto the initial framein the interval is a goodapproximationof themotionparameters.In matrix form, affinemotiones-timationcanbewrittenas
K � ����ML � K %)(N%'*% + % - LMO K � �PLRQ K %'&% � LSO (3)
More generally, thiscanbewrittenasT � �AUET QWV (4)
where U containsthe motion parametersfor the first ma-trix and V containsthetranslationalparameters.Therelativemotionis computedasT � � �AU � 0 UET QWV 3 QXV � �YU � UET Q 0 U � V�QXV � 3 (5)
whereT)� � is thevectorfor thenew transformedcoordinates;U:� and V � hold thecurrentmotionparameters;and U and Vhold themotionparametersup to thecurrentframe.
To increasetherobustnessof motionestimation,weuseM-estimators[4, 5] andtheerroris expressedas:FZ [ 0 G � 3 (6)
where[ 0 G � 3\�]G (� in the original formulation. Sincethis
function givesmoreweight to large errors,it is biasedby
local motion(which areoutliersfor globalmotion). To de-creasetheeffect of outliers,thetruncatedquadraticmotionis used: [ 0 G � 3^�E_ G (� 2 if ` G � `)a V682 if ` G � `)b V (7)
where V is a thresholdselectedaccordingto the histogramof theerrors.
3. MOSAIC GENERATION
3.1. Histemporal Filter
The linear temporalfilters like averaging,recursive filterslike Kalmanfilter [8], andorder-statisticfilters like medianfilters have beenusedfor noisereductionor removal in im-agesequences.Temporalaveragingyields a blurred mo-saic,if thevideoincludesmoving objectsor motioncannotbe estimatedaccurately. Median filters requireenormousstorageto detectthetemporalmedianfilter. Moreover, tem-poral medianmay yield erroneousresults,if the expectedmediancantakeseveralvalues.For example,thefrequencyof pixel values100 and101 is 20 and22 for a pixel coor-dinatein themosaic,respectively. Althoughthis differencemayresultfrom theilluminancechangein theenvironment,they are treatedas different. If the frequency of anotherpixel valueis 25, this valuewill be chosenby mistake. Infact,averaging(of 100and101)would yield a betterresult.Histemporal filter is a temporal filter that is basedon thehistogramof intensityvalueswithin a specificinterval.
The ced V G�fgT %)h determinesthe precisionof temporalin-tegrationin mosaicgeneration.For a 8-bit per pixel gray-scaleimage,all the pixels lay in i 682�j@k,kgl . Therewill bem (�-#.�onqp=rts�u ��v#w slots in the histogram. If ced V GxfgT %)h is j,kzy , his-temporalfilter becomestemporalaveraging.If c4d V G�fgT %'h is1, thetemporalinterval becomestemporalmedianfilter.
Two datastructuresareusedto obtain the histemporalfilter: frequencyarrayandaverage array. Frequencyarraykeepsthe frequency of eachinterval of the histogram.Astheframesareprocessed,thefrequency of aninterval is in-creasedfor eachpixel valuebelongingto theinterval. Aver-agearraymaintainstheaverageof thevaluesasnew valuesareprocessedfor eachslot. Histemporalfilter returnstheaveragevalueof the interval having the highestfrequency.Figure1 (a) shows a histogramwhere ced V GxfgT %)h is 16. Theinterval i { >@2#|,yql hasthehighestfrequency. Figure1 (b) dis-playsthe frequenciesof thevaluesthat lay in this interval.During histemporalfilter computation,theaverageof thesevaluesis computedasthey arrive.
3.2. High Resolution Mosaic Generation
Motion parametersthat are obtainedfrom Equation1 areusuallyrealnumbersandyield real-valuedtransformedco-ordinates.Theoriginal imagesaresampledinto integerdo-
Figure1: Histemporalfilter.
main. The ordinary techniquescreatea mosaichaving aresolutionof the initial frame in the sequence.The pixellocationsin the mosaicmay not correspondto the integer-valuedpixel locationsin the new frame. Approacheslikebilinearinterpolationareusedto estimatethepixel valueatthelocation.Bilinear interpolationtakestheweightedaver-ageof theclosestpixelsandblurstheimage.
A high resolutionvideo mosaickingapproachis pro-posedin [9]. A high resolutionmosaicis generatedwherea mosaicalsocontainshalf-peldata.Whena new frameisprocessed,a shift (diagonal,vertical, or horizontal)on themosaicis assumed,and the frame is warpedinto the cor-respondingareain the mosaic. This usuallypreservestheoriginal sharpnessof the image. However, this approachdoesnot considerthe precisionof the transformedcoordi-natesandwarpingstill occursata low resolutionbecauseofshifting. In ourcase,warpingoccursathighresolution(Fig-ure2). Every pixel in thewarpingregion is updatedduringwarping.
Figure2: High resolutionmosaic.
Themotionparametersarealsoaffectedby themovingobjectsandapertureproblem. This causessomedeviationfrom the original valuesof motion parameters.Whenthemotion estimationis performedfrom frame to frame, theerror accumulatesandpropagatesto the later motion esti-mationandwarping. In addition,warpingat every framealsointroducesblurring. Thus,insteadof warpingat every
frame, the framesarewarpedinto the mosaicat intervals.But themotionestimationhasto beperformedfor eachse-quential frame. The previous frame is mappedfrom themosaicto avoid error accumulation.Threethresholdsareused: maximumaccumulateddisplacement0=}\%)~ 3 , maxi-mumscalefactor 0=}��g� 3 andmaximuminterval length 0�}\ceh 3 .The motion betweenconsecutive framesare accumulateduntil thedisplacementis lessthan }�%)~ andscale(zooming)factoris lessthan }��g� . Otherwise,themotionbetweenthefirst frameandthe last frame in the interval may increasesignificantly, andmotionestimationmethodsmayyield lessaccurateparameters.If thereis no significantmotionin thesequence,the relative motion is computedfor at most }\cehframes.This upper-boundis neededto remove the objectsfrom thebackgroundmosaic.Theframeis alsowarpedintothemosaicwhenthedirectionof cameramotionchanges.
4. EXPERIMENTS
In our experiments,theresolutionof themosaicis twice asthe initial frameof the sequence,thusresultingin half-pelaccuracy. The }�%)~ and }�cth arebothselectedas10. Figure3 shows anordinaryblurredmosaicgeneratedfrom ’coast-guard’MPEGtestsequence.If themotioncanbemodeledusingtranslationalmodel,theimagescanbewarpedaccord-ing to the precisionof transformedcoordinates. MPEG-4 test sequence’coastguard’can be modeledusing trans-lational model. Figure 4 shows the high resolutionback-groundmosaicgeneratedafter 300 frames. No segmen-tation mask is usedin the generation. The water textureis smoothedbecauseof temporaltexture,andhasbeenre-movedfrom themosaic.Theright sideof thefigureincludespartsthatarenot filled by frames.Thereforethe right sidelooks darker. Theselocationsarefilled with bilinear inter-polation. Thesmoothedregionsin theordinarymosaicareclearlyvisible in thehigh resolutionmosaic.
Figure3: OrdinaryMosaic.
Thereis no standardizedperformancetestsfor generat-ing mosaics.Themostcommonmethodis averagingPSNRvaluesfor a video. Although PSNRis a good indicationof similarity betweenimages,averageof PSNRvaluesisnot alwaysa goodmeasurefor video. Figure5 shows themosaicgeneratedfor ’foreman’ MPEGtestsequencefromframes195 to 240. The correspondingPSNRvaluesforframesthataregeneratedfrom high resolutionmosaicandordinarymosaicaregivenin Figure6. We have usedaffinemotionmodelfor ’foreman’ sequence.
Figure4: High resolutionmosaicfrom coastguard.
Figure5: Mosaicgeneratedfrom ’foreman’.
195 200 205 210 215 220 225 230 235 24023.5
24
24.5
25
25.5
26
26.5
27
27.5
28PSNR for foreman
PS
NR
in d
b
frame numbers
High resolution affineOrdinary affine
Figure6: PSNRfor foremansequence.
5. CONCLUSION
In thispaper, wepresentedamethodfor highresolutionmo-saicgenerationfrom video.Motion estimationis performedbetweeneachconsecutive frame not to miss visible areasin the sequencefor mosaicgeneration.Theblurring in themosaicgenerationis reducedby warpingat intervalsandatahigherresolution.Althoughhigh resolutionmosaicwarp-ing increaseselapsedtime, this is compensatedby warpingat intervals.Thenumberof framesthatareskippedcanfur-therbedecreased,but this is left asa furtherresearch.
6. REFERENCES
[1] T. Sikora, “The mpeg-4 video standardverificationmodel,” IEEE Trans.Circuits Syst.VideoTechnology,vol. 7, pp.19–31,February1997.
[2] M. Irani and P. Anandan, “Video indexing basedonmosaicrepresentations,” in Proceedingsof IEEE, May1998,pp.905–921.
[3] R. Szeliski and H-Y. Shum, “Creating full viewpanoramicimagemosaicsandenvironmentmaps,” inComputerGraphicsProceedings,Annual ConferenceSeries, 1997,pp.251–258.
[4] F. Dufaux andJ. Konrad, “Efficient, robust, and fastglobal motion estimationfor video coding,” IEEETransactionson Image Processing, vol. 9, no. 3, pp.497–501,March2000.
[6] A. Smolic andJ.-R.Ohm, “Robust global motion es-timation usinga simplified m-estimatorapproach,” inProc.ICIP2000,IEEEInternationalConferenceonIm-ageProcessing, September2000.
[7] P. Salembier, O. Pujol, andL.Garrido, “Connectedop-eratorsfor spritecreationandlayeredrepresentationofimagesequences,” in IV EuropeanSignalProcessingConference, September1998,pp.2105–2108.
[8] O. Munkelt C. Ridder and H. Kirchner, “Adaptivebackgroundestimationand foreground detectionus-ing kalman-filtering,” in Proceedingsof InterenationalConferenceon RecentAdvancesin Mechatronics, June1995,pp.193–199.
[9] A. SmolicandT. Wiegand,“High-resolutionvideomo-saicing,” in Proc. ICIP2001,IEEE InternationalCon-ferenceon ImageProcessing, October2001.
