ADAPTIVE, OPTIMAL-RECOVERY IMAGE INTERPOLATION D. Darian Muresan and Thomas W. Parks School of Electrical and Computer Engineering Cornell University darian, [email protected]ABSTRACT We consider the problem of image interpolation using adaptive optimal recovery. We adaptively estimate the local quadratic signal class of our image pixels. We then use op- timal recovery to estimate the missing local samples based on this quadratic signal class. This approach tends preserve edges, interpolating along edges and not across them. 1. INTRODUCTION Image interpolation is becoming an increasingly important topic in digital image processing, especially as consumer digital photography is becoming ever more popular. From enlarging consumer images to creating large artistic prints, interpolation is at the heart of it all. It has been known for some time that classical interpolation techniques such as lin- ear and bi-cubic interpolation are not good performers since these methods tend to blur and smooth edges. Wavelets have been successfully used in interpolation [1, 4, 6]. These methods assume the image has been passed through a low pass filter before decimation and then try to estimate the missing details, or wavelet coefficients from the low resolution scaling coefficients. One drawback to these approaches is that they assume the knowledge of the low pass filter. Directional interpolation algorithms try to first detect edges and then interpolate along edges, avoiding interpola- tion across edges [5]. In this class, there are algorithms that do not require the explicit detection of edges. Rather, the edge information is built into the algorithm itself. For ex- ample, [3] uses directional derivatives to generate weights used in estimating the missing pixels from the neighboring pixels. In [2], the local covariance matrix is used for esti- mating the missing pixels. This interpolation tends to adjust to an arbitrarily oriented edge. In this paper we present a new directional interpolation technique based on optimal recovery. The results of our in- terpolation approach can be thought of as an extension to [2]. In regions of high frequency our approach provides This work was supported by NSF MIP9705349, TI and Kodak PSfrag replacements Fig. 1. Geometric Diagram of Ellipsoid Class slightly better results than [2] and in some cases outper- forms [9]. 2. OPTIMAL-RECOVERY In this section we briefly review the theory of optimal re- covery as applied to the interpolation problem [8]. We then apply this theory to develop a new adaptive approach to im- age interpolation. The interpolation problem may be viewed as a problem of estimating missing samples of an image. This latter problem can be examined using the theory of op- timal recovery. The theory of optimal recovery provides a broader setting, which illuminates the process of interpola- tion, by providing error bounds and allowing calculation of worst-case images which achieve these bounds. Locally, at location (Fig. 2), we model the image as belonging to a certain ellipsoidal signal class (1) where is derived from the local image pixels as shown in section 3. Vector is any subset of the image containing the missing pixel . Vector is chosen such that any linear functionals ( ) of are assumed known. If we note the actual values of the functionals by we have . In this paper we assume that the functionals are based on derivatives and/or actual pixel values of the decimated image. The known functionals , in the local image, determine a hyper-plane (Fig. 1). The intersection of the hyper-plane and ellipsoid is a hyper-circle in . The intersection depends upon the known
We considerthe problemof imageinterpolationusingadaptiveoptimalrecovery. We adaptively estimatethelocalquadraticsignalclassof our imagepixels. We thenuseop-timal recovery to estimatethemissinglocal samplesbasedon this quadraticsignalclass.Thisapproachtendspreserveedges,interpolatingalongedgesandnot acrossthem.
1. INTRODUCTION
Imageinterpolationis becomingan increasinglyimportanttopic in digital imageprocessing,especiallyas consumerdigital photographyis becomingever morepopular. Fromenlarging consumerimagesto creatinglargeartisticprints,interpolationis at theheartof it all. It hasbeenknown forsometimethatclassicalinterpolationtechniquessuchaslin-earandbi-cubicinterpolationarenotgoodperformerssincethesemethodstendto blur andsmoothedges.
Waveletshave beensuccessfullyusedin interpolation[1, 4, 6]. Thesemethodsassumetheimagehasbeenpassedthrougha low passfilter beforedecimationandthentry toestimatethemissingdetails,or waveletcoefficientsfrom thelow resolutionscalingcoefficients. Onedrawbackto theseapproachesis that they assumethe knowledgeof the lowpassfilter.
Directional interpolationalgorithmstry to first detectedgesandtheninterpolatealongedges,avoiding interpola-tion acrossedges[5]. In this class,therearealgorithmsthatdo not requirethe explicit detectionof edges.Rather, theedgeinformationis built into the algorithmitself. For ex-ample,[3] usesdirectionalderivativesto generateweightsusedin estimatingthemissingpixels from theneighboringpixels. In [2], the local covariancematrix is usedfor esti-matingthemissingpixels.This interpolationtendsto adjustto anarbitrarily orientededge.
In this paperwe presenta new directionalinterpolationtechniquebasedon optimalrecovery. Theresultsof our in-terpolationapproachcanbe thoughtof asan extensionto[2]. In regions of high frequency our approachprovides
Thiswork wassupportedby NSFMIP9705349,TI andKodak
PSfragreplacements�
Fig. 1. GeometricDiagramof Ellipsoid Class
slightly better resultsthan [2] and in somecasesoutper-forms[9].
2. OPTIMAL-RECOVERY
In this sectionwe briefly review the theoryof optimal re-coveryasappliedto theinterpolationproblem[8]. We thenapplythis theoryto developanew adaptiveapproachto im-ageinterpolation.Theinterpolationproblemmaybeviewedas a problemof estimatingmissingsamplesof an image.This latterproblemcanbeexaminedusingthetheoryof op-timal recovery. The theoryof optimal recovery providesabroadersetting,which illuminatestheprocessof interpola-tion, by providing errorboundsandallowing calculationofworst-caseimageswhich achievethesebounds.
Locally, at location � (Fig. 2), we modelthe imageasbelongingto acertainellipsoidalsignalclass����������� ��������������� (1)
where � is derivedfrom thelocal imagepixelsasshown insection3. Vector � is any subsetof theimagecontainingthemissingpixel � . Vector � is chosensuchthatany � linearfunctionals( ��� �"!#�%$&��'('�')� � ) of � areassumedknown. Ifwe notethe actualvaluesof the functionalsby *+� we have� � , ��-.�/* � . In this paperwe assumethat the functionalsare basedon derivativesand/oractualpixel valuesof thedecimatedimage. The known functionals � � , in the localimage,determineahyper-plane0 (Fig. 1).
The intersectionof the hyper-planeand ellipsoid is ahyper-circlein 0 . Theintersectiondependsupontheknown
For a linear mapping ? , the imageof 1 2 under ? is therangeof valuesthat ?�* can take. The optimal recoveryproblemis to selectthevaluein 0 which is a bestapproxi-mationoverall ?�* in ?@1 2 . We wantto minimizeA �CB�D+E2GFH=I2 JLK�5MN� JTheChebyshev centerachievesthisminimization.TheCheby-shev centerhasbeenshown to betheminimum O -normsig-nal on the hyper-planedeterminedby the known samples.Thesolutionto thisproblemis well-known: see[8, 7].
If the collection of known functionalsis * � , the mini-mum norm signalis PQ . Signal PQ is the uniquesignalin 0with theproperty, :RPQ : = � SUTRVW&XZY 2�[]\I^ X :)��: = (3)
Our estimatessignal PQ mustsatisfy � � , PQ -_�`* � andwe areestimating� , PQ -a�b* . As shown in [8] thereexist vectorsc � ced ��'('�'(� c�f suchthat � , KQ -8� , c �&PQ - = and��� , KQ -g� , c � ��PQ - = (4)
We now dealwith theproblemof determiningO adap-tively from the imagedata. To make this explanationassimpleandasstraightforwardaspossible,we demonstrateourmethodwith asimpletoy example.
3. ADAPTIVE, OPTIMAL-RECOVERYINTERPOLATION
Ouradaptively determinedquadraticsignalclass,or O , willbeameasureof how well thelocaldatamatchesthealreadyknown functionals � � . We want to find an adaptive signalclass� of theform:���o�G�� ��� � �p�q�n��� (6)
To bestunderstandthis process,let’s look at Fig. (2). In
a r1 a r
2 s b 1
a r4 a r
3 t b 4
d u
1 d u
2 c v 1
d u
4 w d
u3 t c v 4
w
y x 1 y x 2 s y x 3
ty x 4 y x y x 6
yy x 7 z y x 8
{ y x 9 |
b 2 s
b 3 t
c v 2
c v 3 t
Fig. 2. Interpolatepixel � . The only known pixelsarethegraypixels.
this small example,the problemis thatof estimatingpixel� . Our first stepis to choosea signal � that containsthemissingpixel � . For reasonsthatwill beclearin a moment,let �@�~} � d �&�4�&�3�k�3���4���3���>�4���4� d �&�4�&�4�&�<� �Next, we assumethat thereexists weight � d �('('�')�"��� suchthat locally, eachpixel can be estimatedby the weightedsumof thefour closestdiagonalpixels.With: K� : � ��BpS�T� : � : �ourmeasureof how well thedatamatchesthenearbypointsis
K� �������������
� d� �� �����<�����...
�(�����������
M������������
� d �k���k���������� d �����k�� d �<� �<� ���� d � d � d � d� � � � � � � �� � � � � � � d� � � � � d � �...
A �B�D+E 2;Fk=I2 JLK�ªMa� Jis equivalentto finding � whichminimizes� � �p� giventheknown functionals� � .
Assuming� is full rank,matrix Q hasfour zeroeigen-valueswith therestof thembeingall one.Thenull spaceofQ is spannedby thecolumnvectorsof � . At first glance,it
seemsthatthesolutionto this problemmight beany vectorin thenull space,sincethatwill give zeroerror. Thathow-ever, is nottruesincethesolutionmustalsosatisfythegivenfunctionals.Unlessthereareonly four known functionals,oursolutionwill not bein thenull spaceof Q .
Known functionalscanbepixel valuesof thedecimatedimage,derivative assumptionsor any otherlinear function-als of the high resolutionimage. In our toy example,thelinear functionalsarethegivenpixels �k�&�«���>� � d � ��� andas-sumptionsaboutthederivatives.In particular, welook atthederivativesin thedirections�>�¬M ��� and �k�¬M� d . We chosethe directionwith the smallestchangeandassumethat thederivativesof theunknown pixelsin thatdirectionareequalwith thederivativesof theknown pixels in thesamedirec-tion. For example,if ����M���� hasthesmallestdifference,thederivativebasedfunctionalswouldbe�&��MN�p�9�<��M ��� and �&�¬MN�p���>�_M�>�Whenthe known functionalsareonly the decimatedpixelvalues,this methodsimplifies to the methodpresentedin[2].
Theformulationof ourproblemandtheadaptive Q ma-trix is alsoquiteusefulwhenweassumethattheimagewentthroughalow passfilter, beforedecimation.In thiscase,ourassumptionis that thepixel valuesof Fig. 2 aresamplesofthefilteredimage.If we let ® bea filtering matrix andweassumethattheimagebeforefiltering is ¯����®7¯ (9)
then Q of (8) becomes�°�±® � Oa® . Thenew Q will stillhave four zeroeigenvalues,but the othereigenvalueswillno longerbeone.
1. Interpolatethe missing pixels with the methodde-scribedabove.
2. Using the interpolatedpixels, return to equation(2)andaddthecalculatedpixelsat thehigherresolutionasextra functionals.
We haven’t provedconvergencehere,but from our experi-mentalresults,repeatingthis processthreetimesseemstobeenough.
4. RESULTS
In obtainingour resultswe first startedwith a high resolu-tion image.We thenfilteredthehigherresolutionimagebya low passfilter (Daubechies1), to simulatecameralenses,and decimatedby two. We then reconstructedthe imageusingdifferentinterpolationapproaches.
For our resultssectionwe have comparedtheadaptive,optimal recovery imageinterpolationalgorithmagainstthealgorithm presentedin [2], againstbi-cubic interpolationandagainsta commerciallyavailablealgorithm[9]. Whencomparedagainst[2] the algorithmoutperformedslightly,especiallyaroundsharpand/orthinedges.Thealgorithmal-waysoutperformedbi-cubicinterpolation.Whencomparedagainst[9] therewereplaceswherethe adaptive, optimal-recoveryinterpolationoutperformed[9], but therewerealsoplaceswhereit under-performed.Somesampleimagesareincludedat theendof this section,but thereaderis encour-agedto view TIF imagesatww.ee.cornell.edu/² splab.
Finally, we would like to thankXin Li for providing uswith his interpolationalgorithm[2].
5. REFERENCES
[1] D. DarianMuresanandThomasW. Parks,“Predic-tion of ImageDetail,” IEEE ICIP, Vancouver, Sept.2000.
[2] Xin Li and Michael T. Orchard, “New Edge Di-rectedInterpolation,” IEEE International Conferenceon Image Processing, Vancouver, Sept.2000.
[4] W. Knox Carey, Daniel B. Chuangand Sheila S.Hemami, “Regularity-PreservingImage Interpola-tion,” IEEE Transactions on Image Processing, Vol.8, No. 9 (Sept.1999),1293-1297.
[5] J. Allebach and P. W. Wong “Edge-directedInter-polation,” IEEE Proceedings of ICIP, 1996,pp.707-710.
[7] C.A. Michelli andT. J.Rivlin, “ A Survey of OptimalRecovery,” in Optimal Estimation in ApproximationTheory, C. A. Michelli and T. J. Rivlin, Eds.NewYork: Plenum1976,pp.1-54.
[8] M. GolombandH. F. Weinberger, “Optimal approx-imationandError Bounds,” On Numerical Approxi-mation, R. E. Langered.,TheUniversityof Wiscon-sinPress,Maddison,pp.117-190,1959.
