Feature Detection and Matching in Images with Radial Distortion
Miguel Lourenco, Joao P. Barreto and Abed MaltiInstitute of Systems and Robotics,
Faculty of Science and Technology,University of Coimbra.3030 Coimbra, Portugal
{miguel,jpbar,amalti}@isr.uc.pt
Abstract— Image keypoints are broadly used in roboticsfor different purposes, ranging from recognition to 3D re-construction, passing by SLAM and visual servoing. Robustkeypoint matching across different views is problematic becauseof the relative motion between camera and scene that causessignificant changes in feature appearance. The problem canbe partially overcome by using state-of-the-art methods forkeypoint detection and matching, that are resilient to commonaffine transformations such as changes in scale and rotation.Unfortunately, these approaches are not invariant to the radialdistortion present in images acquired by cameras with widefield-of-view. This article proposes modifications to the ScaleInvariant Feature Transform (SIFT), that improve the repeata-bility of detection and effectiveness of matching in the presenceof distortion, while preserving the characteristics of invarianceto scale and rotation. These modifications require an approx-imate modeling of the image distortion, and consist in usingadaptative gaussian filtering for detection and implicit gradientcorrection for description. Extensive experiments, with bothsynthetic and real images, show that our method outperformsexplicit distortion correction using image rectification.
I. INTRODUCTION
The Scale-Invariant Feature Transform (SIFT) [1] enableskeypoint detection and description in conventional perspec-tive images, providing invariance to common image trans-formation such as scale, rotation, illumination, and minimalviewpoint changes [2]. In the past SIFT has been successfullyapplied in robotics for performing different tasks such asvisual servoing and SLAM [3], [4]. In addition, roboticsystems can benefit from the usage of wide field-of-viewimages. Panoramic cameras enable a more thorough visualcoverage of the environments, and are highly advantageousin egomotion estimation by avoiding ambiguities betweentranslation and rotation whenever the translation directionlies outside the field of view [5], [6]. However, the projectionin cameras with wide angle lens presents strong radialdistortion caused by the bending of the light rays whencrossing the optics. The distortion increases as we go fara way from the center, and it is typically described by non-linear terms that are function of the image radius. Since theoriginal SIFT algorithm was not designed to handle this typeof image deformation, keypoint detection and matching inwide-angle imagery can be highly problematic [7].
The authors acknowledge the Portuguese Science Foundation, that gen-erously funded this work through grant PTDC/EEA-ACR/68887/2006
The SIFT algorithm performs keypoint detection in ascale-space representation of the image [8], [9] applyingan approximating of Laplacian-of-Gaussian (LoG) by theDifference-of-Gaussian (DoG). The detection is carried in theDoG pyramid by looking for extrema simultaneously in scaleand space, with the extrema being illustrative of the corre-lation between the characteristic length of the signal featureand the standard deviation of the filter !. After the detectionof the keypoints, the processing is carried at the level of thegaussian pyramid where the extrema occurred, and a mainorientation, based on the spatial gradients, is assigned to eachkeypoint. The final descriptor is computed using a patch of16! 16, after rotation according to the previously assignedorientation, providing invariance to image rotation.
Radial distortion (RD) is a non-linear geometric deforma-tion that moves the pixel position along the radial directionand towards the center of distortion. In broad terms, thecompression induced by the RD diminishes the characteristiclength of the signal features and, as a consequence, thecorresponding extrema tend to occur at lower levels ofscale than they would occur in the absence of distortion.In addition, the image gradients are also affected by thepixel shifting induced by RD. The SIFT descriptor, despite ofbeing robust to small changes in the gradient contributions,suffers a considerable deterioration for significant amountsof distortion, which has a negative impact in the recognitionperformance.
Despite of the fact that the SIFT algorithm is not in-variant to RD, it has been applied in the past to imageswith significant distortion. While ones ignore the perniciouseffects of RD and directly apply the original SIFT algorithmover distorted images [10], others perform a preliminarycorrection of distortion through image rectification and thenapply SIFT [11]. This last approach has two major draw-backs: (i) the rectification is computationally expensive,specially when dealing with large sized images; (ii) theimage re-sampling requires interpolation that, dependingon the choice of reconstruction filter, can adulterate thespectrum of the image signal and affect the response of theDoG [12]. Recently, Hansen et al. [13] proposed an approachto extend SIFT for wide angle images. The method assumesthat camera calibration is known and they suggest to back-project the image onto an unitary sphere and build a scale-space representation that is the solution of the diffusion
2010 IEEE International Conference on Robotics and AutomationAnchorage Convention DistrictMay 3-8, 2010, Anchorage, Alaska, USA
equation over the sphere. Such representation minors theproblems inherent to planar perspective projection, enablingRD invariance and extra invariance to rotation. However, theapproach requires perfect camera calibration and tends to behighly complex and computationally expensive.
In contrast with [13], we propose a set of well engineeredmodifications to the original SIFT algorithm to achieve RDinvariance. Every processing step is carried directly in thedistorted image plane and the additional computational costis marginal. The gaussian pyramid is obtained by convolutionwith a gaussian filter, whose shape is modified in terms ofthe image radius. The objective is to take into account thedistortion effect, such that the final DoG representation isequivalent to the one that would be obtained by filteringin the absence of distortion and subsequently applying theRD. In a similar manner, the SIFT descriptors are computeddirectly over the distorted image after correcting the imagegradients using the derivative chain rule. Comparative studiesshow that the modified SIFT algorithm outperforms theapproach of correcting the distortion through image recti-fication in terms of detection repeatability, precision-recallof matching, and computational efficiency, preserving scaleand rotation invariance.
The structure is as follows: Section II briefly reviews theSIFT algorithm and the division model [14] that is assumedfor describing the image distortion. Section III studies theeffect of the radial distortion in keypoint detection, andderives the gaussian adaptative filtering for overcoming theproblems caused by image deformation. Section IV evaluatesthe impact of the distortion in the keypoint description, andproposes implicit gradient correction to account for the RDeffect. Finally, section V conducts tests using real distortedimages taken from different viewpoints.
Notation: Convolution kernels are represented by symbolsin sans serif font, e.g. G, and image signals are denotedby symbols in typewriter font , e.g. I. Vectors and vectorfunctions are typically represented by bold symbols, andscalars are indicated by plain letters, e.g x = (x, y)T andf(x) = (fx(x), fy(x))T. We will also often use RD to referto radial distortion.
II. THEORETICAL BACKGROUND
A. Scale Invariant Features Transform
Lowe adopts a strategy that approximates the Laplacian-of-Gaussian (LoG), used for the scale-space representation[8], [9], by the DoG operator [1]. Let I(x, y) be an imagesignal and G!(x, y) a 2D gaussian function with standarddeviation !. The blurred version of I(x, y) is obtained by itsconvolution with the gaussian
L!(x, y) = I(x, y) " G!(x, y) (1)
and the DoG pyramid is computed as the difference of con-secutive filtered images with the standard deviation differingby a constant multiplicative factor:
DoG(x, y, kn+1!) = Lkn+1!(x, y)# Lkn!(x, y) (2)
In the pyramid of DoG images each pixel is comparedwith its neighborhood pixels in order to find local extrema inscale and space. These extrema are subsequently filtered andrefined to obtain the detected keypoints. After the detectionof the keypoint, the next steps concern the computation ofthe final descriptor using the image gradients of a local patcharound the point. In order to achieve scale invariance, allthe computations are performed at the scale of selection ofthe keypoint in the gaussian pyramid. The method starts byfinding the dominant orientation of the local gradients, anduses it for rotating the image patch towards a normalizedposition in order to achieve invariance to rotation transfor-mations. For the main orientation assignment, an histogramfor 36 bins (10 degrees per bin). Each sample is weightedby a gaussian of 1.5! to give less emphasis to contributionsfar from the keypoint. The normalizing rotation is performedand the final SIFT descriptor is computed from a patch of16!16 pixels divided into subregions of 4!4 pixels, eachone providing 8 main orientations [1].
B. The Division Model for Radial DistortionThe effect of lens distortion in image acquisition can be
often described using the first order division model [14]. Letx = (x, y) be a point in the distorted image I, and x = (u, v)the corresponding point in the undistorted image I. Theorigin of coordinate system is assumed to be coincident withthe distortion center, which is approximated by the imagecenter [15]. The amount of distortion is quantified by aparameter " (typically " < 0), and undistorted image pointsx are mapped into distorted points x by function f :
x = f(x) =!
fx(x)fy(x)
"=
#
$2u
1+$
1!4"(u2+v2)2v
1+$
1!4"(u2+v2)
%
& , (3)
The distorted image can be rectified using the inverse ofdistortion function :
'x = f!1(x) =!
f!1u (x)
f!1v (x)
"=
(x
1+"(x2+y2)y
1+"(x2+y2)
)(4)
The function f is radially symmetric around the image center,and its action can be understood as a shift of image pointstowards the center along the radial direction. The relationshipbetween undistorted and distorted radius is given by :
r =r
1 + "r2(5)
Radial distortion causes a space compression of the imageinformation, which substantially changes the signal spectrumand introduces new high frequency components. To providethe notion of how much the image is compressed, wewill often express the amount of distortion through thenormalized decrease in the maximum image radius:
%distortion =rM # rM
rM" 100 (6)
with rM and rM denoting respectively the maximum valuesfor the undistorted and distorted image radius. Through thiswork we will always assume that image distortion followsthe division model.
1029
Fig. 1. Some of the images used for the synthetic experiments and itscorrespondent distorted views. The data set comprises a broad variety ofscenes and visual contexts.
III. SIFT DETECTION UNDER RADIAL DISTORTION
If we apply SIFT directly over a distorted image, thecorresponding multi-scale representation is different from theone that would be obtained from the equivalent perspectiveimage in the absence of RD. The distortion compressesthe intensity spectrum of the image, introducing new highfrequency components. This leads to the detection of someunstable points, which would not be detected in the undis-torted image, as well as the non-detection of others.
A. Evaluation using Images with Artificially added RD
To study SIFT detection under RD, we used a set ofimages from the internet, and we artificially injected radialdistortion (Fig. 1). We decide to perform such syntheticexperiment in order to control the amount of distortion, toknow the positions where keypoint detection should occur(ground truth), and because it would not be practicallyfeasible to acquire multiple images with different distortionsfrom the same viewpoint. Let’s consider an image of thedata set and one of its distorted versions, and assume S0
and S as being the set of keypoints detected in the originaland distorted images, respectively. The elements of S caneither be points already detected in the original image, or newkeypoints that appear due to the high frequency componentsintroduced by radial distortion. Henceforth, we will denotethe former by Sd and the latter by Snew such that:
S = Sd % Snew (7)
Sd = S0 & S (8)
The set Sd contains keypoints in the distorted image detectedat a correct spatial location. However, the correct assignmentof scale is fundamental for achieving reliable matchingacross different views. Therefore set Sd is split in twosubsets: Sc containing the points detected at correct scaleand location, and Sws being the set of points close in spacebut not in scale (detections at wrong scale).
Sc = S0 & (S # (Snew % Sws)) (9)
From the subset introduced, the repeatability in keypointdetection is evaluated using the following metric:
%Repeatability =#Sc
#S0! 100 (10)
with # denoting the number of keypoints in each set. Theoccurrence of new spurious detections due to radial distortionis quantified as follows:
%New detections =#Snew
#S" 100 (11)
And finally the detection at wrong scale is characterizedby the percentage of points detected at incorrect scale withrespect to the points detected at a correct image location [1]:
%Keypoints at wrong scale =#Sws
#Sd" 100 (12)
B. How does RD affect Keypoint Detection?
The compressing effect induced by radial distortion isresponsible for several problems during keypoint detection.Since the level ! of the DoG pyramid at which detectionoccurs reflects the characteristic length of a certain feature inthe image, the compressive effect of RD pushes the extremadetection towards lower values of scales. Since SIFT startsfiltering at !0 = 1.6, some keypoints will no longer bepicked as an extrema because the value of their scale will nolonger be considered in SIFT band-pass filtering. In additionthere will be keypoints detected at different scales and, thehigh frequency components introduced by RD, can even leadto new detections. Fig. 2 shows experimental evidence ofthe degradation of SIFT detection in images with increasingRD. The observed behavior is in accordance with the statedtheoretical interpretation: (i) the loss of repeatability ismore pronounced at lower levels of the DoG pyramid, anddetections at wrong scales arise at coarser levels of scale,which reflects the fact that RD makes the keypoints smaller;(ii) the compression induced by RD in the image spectrumcreates new unstable keypoints that were not detected in theoriginal image.
C. Adaptative gaussian filtering
We introduce a new approach for image adaptative blur-ring that accounts for the RD effect. The objective is togenerate a scale-space representation equivalent to the onethat would be obtained by filtering the image in the absenceof distortion, followed by applying the distortion over all thelevels of the DoG pyramid. Remark that this is different fromthe DoG obtained by simply convolving the distorted imagewith the standard isotropic gaussian kernel, in the sense thatin this case the action of the distortion is before (and notafter) the gaussian filtering. To achieve such goal we willperform the distortion correction in an implicit manner, byadapting the convolution kernel that is used directly over thedistorted image.
Let G! be a bi-dimensional gaussian function with stan-dard deviation !, I the undistorted image, and I the distortedimage. The value of the blurred undistorted image L! at pixel(s, t) is given by
L!(s, t) =+"*
u=!"
+"*
v=!"I(u, v) G!(s# u, t# v) (13)
1030
0 10 20 30 40 50 60 7010
20
30
40
50
60
70
80
% of distortion
% R
epe
ata
bili
ty
(a) Repeatability of the detection
0 10 20 30 40 50 60 70
5
10
15
20
25
30
35
40
45
% of distortion
% K
eyp
oin
t a
t w
rong
sca
le
(b) Detections at wrong scale
0 10 20 30 40 50 60 7024
26
28
30
32
34
36
38
40
42
% of distortion
% N
ew
de
tectio
ns
(c) New detections
Fig. 2. Experimental evaluation of the SIFT detector under RD images. As it can be seen the SIFT detector is severely affected by RD, being therepeatability clearly affected even for lower levels of distortion.
This is the convolution that SIFT performs for the case of theimage being rectified for correcting the distortion. However,and since we want to work directly the with distorted imageI, the undistorted image I can be replaced by its distortedcounterpart, taking into account the inverse of the mappingfunction f() (4). Considering that
I(u, v)) = I(f!1u (x, y), f!1
v (x, y)), (14)
and changing the variables (u, v) by (x, y) in (13), it arises:
L!(s, t) =
1!"!*
x=! 1!"!
1!"!*
y=! 1!"!
I(x, y) G!(s# f!1u (x, y),
t# f!1v (x, y))
(15)Since L! is the distorted version of the smoothed image
L! , we can repeat the reasoning and change the undistortedcoordinates (s, t) by their distorted counterparts (h, k). Itfollows that
L!(h, k) =
1!"!X
x=! 1!"!
1!"!X
y=! 1!"!
I(x, y)G!(f!1u (h, k)" f!1
u (x, y),
f!1v (h, k)" f!1
v (x, y)),(16)
which after some algebraic manipulations leads to
L!(h, k) =
1!"!X
x=! 1!"!
1!"!X
y=! 1!"!
I(x, y)G!
“ h" x + !r2(h"2 " x)1 + !r2(1 + "2 + !r2"2)
,
k " y + !r2(k"2 " y)1 + !r2(1 + "2 + !r2"2)
”.
(17)with +
r =$
h2 + k2
# =,
x2+y2
h2+k2
(18)
Remark that now the smoothing kernel depends on (x, y) and(h, k) and (17) is no longer a straightforward convolution.However, if the pixel coordinates (h, k) is very close to thecenter, then "r2 ' 0 and the expression becomes a standardconvolution. This makes sense because the distortion in thecentral region is negligible and there is no need for thefilter to make any compensation. On the other hand, if thepixel (h, k) is far from the center, then the filtering kernel
only takes significant values for (x, y) close to the location(h, k) (the center of convolution), for which the ratio # isapproximately unitary (# ' 1). In this particular case (17)can be simplified to
L!(h, k) #
1!"!X
x=! 1!"!
1!"!X
y=! 1!"!
I(x, y)G!
“ 11 + !r2
(h" x),
11 + !r2
(k " y)”
(19)The result above is an approximation of (17), and henceforth
we will call it the simplified adaptative filter. While inthe original SIFT detection the image is blurred using astandard isotropic gaussian kernel with standard deviation !,in our case the standard deviation of the filter decreases asa function of the images radius (1+ "r2)!. The convolutionkernel follows the deformation caused by RD, and empha-sizes the contribution of pixels increasingly closer to theconvolution point while the filter moves far from the centerof distortion. The blurring using a standard gaussian filteruses the same kernel mask over the entire image. Moreoverthe computational efficiency of the convolution can be largelyimproved by taking advantage of the decoupling properties inX and Y of the gaussian [9]. Unfortunately, the dependenceof the adaptative filter with respect to the radius requiresusing different kernels for different concentric image circlelocations. However, while the accurate adaptative filtering ofequation (17) cannot be decoupled, the convolution with itssimplified version in (19) can be separately done in X and Ydimensions, adding a minimal computational overhead whencompared to a spatial invariant gaussian filter.
D. Detection Results
In terms of detection evaluation, the repeatability of key-point detection is unarguably the most important property ofa reliable detector [16]. Figure 3 compares the repeatabilityof detection at the correct location and scale by runningdifferent approaches over the synthetically distorted imagery.The properties of the derived adaptative filters allow toovercome the main limitations of SIFT under RD (Fig. 3).For the initial octaves of the scale pyramid, the adaptativegaussian filters allow to detect points that original SIFT does
1031
0 10 20 30 40 50 60 7010
20
30
40
50
60
70
80
90
100%
of R
epeata
bili
ty
% of distortion
SIFT in RD imagesSIFT in rectified imagesAccurate adaptative filterSimplified adaptative filter
(a) Repeatability of the detection
0 10 20 30 40 50 60 700
5
10
15
20
25
30
35
40
45
% N
ew
dete
ctions
% of distortion
SIFT in RD imagesSIFT in rectified imagesAccurate adaptative filterSimplified adaptative filter
(b) New detections
Fig. 3. We compare the proposed adaptative filters against the originalSIFT algorithm ran in both distorted and rectified images (Fig.1). Therepeatability of detection for different amounts of distortion is shown in(a). The adaptative filtering provides the highest repeatability rates foramounts of RD up to ! 35%. The performance of accurate and simplifiedadaptation is very similar, and henceforth we will consider the latter becauseof computational efficiency reasons. The graphic (b) concerns the percentageof new spurious detection showing that the improved repeatability of theadaptative filtering is not achieved at the expenses of an increase in detectionspecificity.
not consider anymore. They also allow to model the structuresize at higher levels of the pyramid in order to avoid detec-tions at wrong scale (Fig. 2(b)). It is somewhat surprising thefact that adaptive filtering outperforms image rectificationfor medium-small amounts of RD. The image re-samplingfor distortion compensation implicitly requires reconstructingthe discrete signal. Depending on the type of low-band passfiltering (in our case we use first order interpolation), thereconstruction can either remove high frequency componentsand/or introduce new spurious frequencies [12]. Thus, therectification causes changes in the image spectrum that haveconsequences in terms of the detection repeatability. Theskeptical reader can easily verify this by performing a linearimage rescaling (expansion to avoid aliasing effects) andcompare the SIFT detections. Contrary to the expected, notevery keypoint in the original images is detected in the scaledversion.
For high amounts of RD (above 35%) the image rectifica-tion outperforms the adaptive filtering. When the compres-sive effect of RD is too high there are image structures thatvanish and become impossible to detect without performingsome kind of image reconstruction. This partially explainsthis experimental observation.
IV. MATCHING IN RADIAL DISTORTED SPACE
By applying a certain amounts of distortion to an im-age, the pixels are shifted towards the center along theradial direction. This will deform the image gradients andconsequently corrupt the SIFT descriptors (see Fig. 4 (c)).However, if we consider that the distortion can be reversedusing the inverse mapping of equation (14), we can computethe distorted image gradients and correct them by applyinga chain-rule derivation. The correction of image gradientsusing the chain rule can be carried only on the neighborhoodof detected keypoints at the scale of selection in the distortedscale-space, which avoids a significant computational over-head.
Applying the chain rule derivation on (14), we obtain#
$#I#u
#I#v
%
& =
#
-$
#I#x
#fx
#u + #I#y
#fy
#u
#I#x
#fx
#v + #I#y
#fy
#v
%
.& = J
#
$#I#x
#I#y
%
& (20)
J denotes the jacobian of function f , which can be expressedin terms of point coordinates in the distorted image
J =1 + "r2
1# "r2
!1# "(r2 # 8x2) 8"xy
8"xy 1# "(r2 # 8y2)
"(21)
with r denoting the radius of the distorted pixel (x, y).Thus, instead of correcting the distortion in the entire
image using rectification, we propose to apply the gradientcompensation around the keypoints detected in the distortedimage. This provides gains in computational efficiency andavoids interpolation artifacts that can change the local ap-pearance of the features.
A. Evaluating Matching Performance
In order to evaluate the effectiveness of image gradientcorrection, we will match features extracted in the original(undistorted) image of the data set shown in Fig. 1, withfeatures detected in the corresponding artificially distortedimages. The gradient correction is compared against match-ing results obtained by applying standard SIFT and byapplying SIFT over corrected images after interpolation.The performance evaluation is described using Recall vs 1-Precision curves [2].
We can observe that, even for low amounts of distortion,the SIFT descriptor starts to be affected by radial distortion(Fig. 4(c)). It is easy to understand that when the image iscompressed the local patch around each keypoint receivescontributions that do not occur for the original undistortedimage. As mentioned early, the SIFT descriptor is preparedto deal with small shifts inside each subregion histogram.However, for high amounts of distortion this effect becomestoo noticeable and, as a consequence, the descriptor drifts inthe feature space precluding a successful match.
Another relevant constraint for the SIFT descriptor usagein RD images is that the gaussian weighting, used to givemore emphasis to contributions close to the keypoint, startsto loose its effectiveness. As we increase the distortion,some pixels, that were initially far from the keypoint, areshifted inside its neighborhood and actively contribute forthe descriptor building. We reduce this pernicious effect byconsidered a weighting gaussian function with standard devi-ation (1+"r2)!, instead of a standard deviation of ! as usedin the original SIFT approach. This allows to have similarcontributions in the distorted and in the original undistortedimage, and then improve the descriptor resilience.
From experimental evidence, it is clear that for low levelsof distortion the method of implicit gradient correctionoutperforms the classic approaches, Fig. 4. It is also observedthat the image rectification is the most valid approach forhigher levels of distortion. Nevertheless, our method alwaysprovides better matching results when comparing with the
1032
0 0.1 0.2 0.3 0.4 0.5 0.6 0.70.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
1!precision
Re
call
5% of distortion
15% of distortion
25% of distortion
35% of distortion
45% of distortion
(a) Implicit gradient correction
0 0.1 0.2 0.3 0.4 0.5 0.60.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
1!precision
Re
call
5% of distortion
15% of distortion
25% of distortion
35% of distortion
45% of distortion
(b) Image rectification
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
1!precision
Re
call
5% of distortion
15% of distortion
25% of distortion
35% of distortion
45% of distortion
(c) SIFT in radial distorted image
516 542 569 598 628 659 692 727 763 8011
2
3
4
5
6
7
8
9
10
Image side (square image)
Co
mp
uta
tion
al t
ime
(se
con
ds)
SIFT in distorted images
SIFT in rectified images
Our Method
Fig. 4. The curves (a)-(c) show the recall against 1-precision for increasing amounts of radial distortion. The recall indicates the percentage of correctmatches obtained (true positives) over the entire set of possible correct matches (Sc subset). The 1-precision is a measure of specificity corresponding tothe percentage of false positives over the total number of matches obtained. We can observe that the rectification from distortion allows high percentagesof successful matches for all levels of distortion. However, until ! 25% of RD the implicit gradient correction outperforms the rectification, being themost suitable approach for moderate levels of distortion. In (d) can be seen the comparison of computational time varying image size at constant distortionof 25%. Our method (simplified adaptative filter with implicit gradient correction) adds minimal computational complexity to the original method whencompared with the explicit distortion correction.
(a) Playmobil data set (! 15% of distortion)
(b) Smiles data set (! 30% of distortion)
Fig. 5. The Playmobil data set was acquired with a lens of ! 15% ofdistortion and the Smiles data set with one of! 30%. Both data sets englobea set of images taken form different viewpoint angles of a planar surface.As it can be seen the image appearance considerably changes due to thedistortion effect allied to the viewpoint in which is taken.
use of SIFT directly in distorted images. The implicit gradi-ent correction technique allows to minimize the effect of thepixels shifting for moderate amounts of radial distortion.
V. VALIDATION WITH REAL IMAGES
The tests performed so far with synthetic imagery providereliable ground truth and enable to test the uniquely RDinvariance. However, we aim to match images with differentacquisition conditions, like scale, rotation and viewpointchanges. In this section we will carry on tests with realimages with radial distortion undergoing significant view-point changes. This enables to evaluate the resilience toRD and also the invariance to rotation and scale that theoriginal SIFT provides. The data set is composed by a set ofimages of a textured planar surface. This means that everytwo images are related by an homography that enables togenerate the ground truth between images. In order to do this,a estimation of the distortion parameter [17] is performed.Then, the homography between the different image viewswas generated by hand using 10 correspondences. Then,this homography is used to select hundreds of automaticallydetected and matched keypoints between the two views anda new estimation based on these points is performed. We
PlayMobil 2 PlayMobil 3 PlayMobil 40
10
20
30
40
50
60
70
80
90
100
% R
epeata
bili
ty
Simplified Adaptative Filter
Accurate Adaptative Filter
SIFT in rectified images
SIFT in RD images
(a) Detection repeatability
0 0.1 0.2 0.3 0.4 0.5 0.6 0.70
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1!precision
Re
ca
ll
SIFT in RD images
SIFT in rectified images
Implicit Gradient Correction
(b) Curves of precision-recall
Fig. 6. Evaluation of the playmobil data set. In (a) is compared therepeatability of the detection for the 4 methods in evaluation in Fig. 3.The images undergo significant scale and viewpoint changes (Fig. 5), whilethe RD invariance is preserved. In (b) it can be seen that the implicitgradient correction overcomes the main limitations of the SIFT descriptor formoderate amounts of distortion. Since the images suffer scale and rotationchanges, we can conclude that the invariances of the original SIFT descriptorare preserved.
considered two data sets one with 15% of distortion and theother with RD of ' 30% (Fig. 5).
The playmobil data set is composed by set of images withmoderate distortion, undergoing scale, rotation and viewpointchanges. We observe in Fig.6(a) that the proposed filtersallow an improvement in detection under RD, outperformingthe explicit distortion correction. We also confirmed that,for moderate amount of distortion, the implicit gradientcorrection performs better than the two classic approaches(Fig. 6(b)).
The smiles data set presents a set of images undergoingconsiderable viewpoint changes, with the estimated valueof distortion being ' 30%. In terms of detection (Fig.7(a)), our method is the one with higher score of successfuldetections. The derived filters preserve the scale invariantin feature detection, as can be observed for the real datasets repeatability. As proved under simulation, the implicitgradient correction starts to be affected by high values ofdistortion in the same manner as the original SIFT descriptorcomputed over RD images. In here, the rectification providesbetter performance than our method (Fig. 7(b)). However,since the recall measure depends on the Sc set, we can
Fig. 8. Matches between smile2 and smile3. Our method provides better matching results in the image periphery, where the RD makes the others methodsfail. To obtain the matches we use the ambiguity distance [13] and the threshold of 0.8 proposed by Lowe [1] to compute the descriptors distance. Theoutliers were discarded recurring to the homography between the two views.
Smile 2 Smile 30
10
20
30
40
50
60
70
80
90
% R
epeata
bili
ty
Simplified Adaptative Filter
Accurate Adaptative Filter
SIFT in rectified images
SIFT in RD images
(a) Detection repeatability
0.1 0.2 0.3 0.4 0.5 0.6 0.70
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1!precision
Recall
SIFT in RD images
SIFT in rectified images
Implicit Gradient Correction
(b) Smile
Fig. 7. Evaluation of the smiles data set. In (a) is compared the repeatabilityof the detection for the 4 method in evaluation in Fig. (3). We can observethat, although the repeatability diminishes as we augment the viewpoint,our method is the more resilient to distortion, showing the highest ratesof repeatability. The final matching performance of our method is slightlypoorer than image rectification in terms of precision-recall. However, theadaptive filtering still provides in absolute terms the highest number ofcorrect matches (see Fig. 8)
argue that if a method for image descriptor presents lowerperformance in terms of recall but if the integrated detectoris really efficient, the algorithm can provide better retrievalperformance. Our method is advantageous when the imagesare acquired with lens that induce radial distortion sincethey allow an improvement of keypoints tracking acrossdifferent views of the same scene, Fig. 8. The methods hereinproposed allies the invariance of the original SIFT to a moreresilient detection and description under radial distortion.From the experimental results (simulation and real cases),we can conclude that our method is a suitable approach foruse in cameras where the lens induce radial distortion.
VI. CONCLUSIONS
In this paper we presented modifications to the broadlyused SIFT algorithm that enhance it with invariance toimage radial distortion. Extensive experiments prove thatour method outperforms explicit image rectification for con-siderable amounts of distortion, preserving all the originalinvariance of SIFT with respect to scale and rotation. Theproposed modifications add a minimal computational over-head to the original method, being potentially applicable toseveral robotic tasks. As future work we aim to improve theresilience of the descriptor built using the derivative chainrule. The proposed detection using adaptative filtering is
extremely effective under considerable amounts of distortion,however increasing the distinctiveness of the descriptor is apriority in order to improve global performance.
REFERENCES
[1] D. G. Lowe, “Distinctive image features from scale-invariant key-points,” Int. J. Comput. Vision, 2004.
[2] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas,F. Schaffalitzky, T. Kadir, and L. V. Gool, “A comparison of affineregion detectors,” Int. J. Comput. Vision, vol. 65, no. 1/2, 2005.
[3] S. Se, D. Lowe, and J. Little, “Mobile robot localization and mappingwith uncertainty using scale-invariant visual landmarks,” Int. J. ofRobotics Research, vol. 21, 2002.
[4] T. Nierobisch, J. Krettek, U. Khan, and F. Hoffmann, “Optimal largeview visual servoing with sets of sift features,” in IEEE Int. Conf. onRobotics and Automation, 2007.
[5] P. Baker, C. Fermuller, Y. Aloimonos, and R. Pless, “A spherical eyefrom multiple cameras (makes better models of the world,” in IEEEInt. Conf. on Comput. Vision and Pattern Recognition, 2001.
[6] J. Gluckman and S. Nayar, “Egomotion and omnidirectional cameras,”in IEEE Int. Conf. on Comput. Vision, 1998.
[7] M. Lourenco, “Techniques for keypoint detection and matchingin endoscopic images,” Coimbra, July 2009. [Online]. Available:http://sites.google.com/site/miguelrlourenco/research-interest
[8] J. L. Crowley and A. C. Parker, “A representation for shape basedon peaks and ridges in the difference of low-pass transform,” IEEETrans. Pattern Anal. Mach. Intell., 1984.
[9] T. Lindeberg, “Feature detection with automatic scale selection,” Int.J. Comput. Vision, vol. 30, no. 2, 1998.
[10] D. Burschka, M. Li, R. H. Taylor, and G. D. Hager, “Scale-invariantregistration of monocular endoscopic images to ct-scans for sinussurgery,” in MICCAI (2), 2004.
[11] R. Castle, D. Gawley, G. Klein, and D. Murray, “Towards simultaneousrecognition, localization and mapping for hand-held and wearablecameras,” in IEEE Int. Conf. on Robotics and Automation, April 2007.
[12] L. Velho, A. Frery, and J. Gomes, Image Processing for ComputerGraphics and Vision. Springer London, 2008.
[13] P. Hansen, P. Corke, W. Boles, and K. Daniilidis, “Scale-invariantfeatures on the sphere,” in Int. Conf. on Comput. Vision, Oct. 2007.
[14] A. Fitzgibbon, “Simultaneous linear estimation of multiple viewgeometry and lens distortion,” in Int. Conf. on Comput. Vision andPattern Recognition, 2001.
[15] R. Willson and S. Shaffer, “What is the center of the image,” Int. Conf.on Comput. Vision and Pattern Recognition, 1993.
[16] T. Tuytelaars and K. Mikolajczyk, “Local invariant feature detectors:a survey,” Found. Trends. Comput. Graph. Vis., vol. 3, no. 3, 2008.
[17] J. P. Barreto and H. Araujo, “Geometric properties of central cata-dioptric line images and their application in calibration,” IEEE Trans.Pattern Anal. Mach. Intell., 2005.
