The impact of radial distortion on the self-calibration of rotating cameras Ben Tordoff and David W Murray Department of Engineering Science, University of Oxford Parks Road, Oxford OX1 3PJ, UK Abstract Recent methods of automatically calibrating the intrinsic parameters of cameras under- going pure rotation are based on the infinite homography constraint, and have been found to be sensitive to radial distortion in the imagery. This paper develops a straightforward argument based on geometrical optics to show that increasing pin-cushion radial distortion will produce a gently worsening under-estimate of the lens’ focal length, whereas increas- ing barrel radial distortion will produce a more sharply increasing over-estimate followed by failure of the calibration. A second geometrical argument uses the approximation of a barrel-distorted image to a spherical projection to estimate the degree of distortion at which breakdown is likely to occur. The predictions are verified experimentally using data from real scenes with varying degrees of distortion and noise added. The paper also considers four methods of correcting the radial distortion within self- calibration. The first method pre-calibrates the distortion as a function of focal length, but the remainder assume no such prior knowledge. Although these prior-less methods are suc- cessful to an extent, everyday scenes are unlikely to provide image feature data of sufficient density and quality to make them fully viable. Key words: Camera calibration, radial distortion, zoom lenses, intrinsic parameters. 1 Introduction A growing number of applications in machine vision depend on the visual geome- try of imagery captured under changing intrinsic parameters, particularly changing zoom. Examples include surveillance using pan-tilt heads, inserting graphical arti- facts into TV and movie sequences, and manipulating imagery captured by domes- tic hand-held camcorders. Preprint submitted to Elsevier Science 14 May 2004
Transcript
The impact of radial distortion on the self-calibrationof rotating cameras
Ben Tordoff and David W Murray
Department of Engineering Science, University of OxfordParks Road, Oxford OX1 3PJ, UK
Abstract
Recent methods of automatically calibrating the intrinsic parameters of cameras under-going pure rotation are based on the infinite homography constraint, and have been foundto be sensitive to radial distortion in the imagery. This paper develops a straightforwardargument based on geometrical optics to show that increasing pin-cushion radial distortionwill produce a gently worsening under-estimate of the lens’ focal length, whereas increas-ing barrel radial distortion will produce a more sharply increasing over-estimate followedby failure of the calibration. A second geometrical argument uses the approximation of abarrel-distorted image to a spherical projection to estimate the degree of distortion at whichbreakdown is likely to occur. The predictions are verified experimentally using data fromreal scenes with varying degrees of distortion and noise added.
The paper also considers four methods of correcting the radial distortion within self-calibration. The first method pre-calibrates the distortion as a function of focal length, butthe remainder assume no such prior knowledge. Although these prior-less methods are suc-cessful to an extent, everyday scenes are unlikely to provide image feature data of sufficientdensity and quality to make them fully viable.
Key words: Camera calibration, radial distortion, zoom lenses, intrinsic parameters.
1 Introduction
A growing number of applications in machine vision depend on the visual geome-try of imagery captured under changing intrinsic parameters, particularly changingzoom. Examples include surveillance using pan-tilt heads, inserting graphical arti-facts into TV and movie sequences, and manipulating imagery captured by domes-tic hand-held camcorders.
Preprint submitted to Elsevier Science 14 May 2004
Studies of the optical parameterisation of zoom lenses indicate that for most, onceset at a particular zoom value and after correction for radial distortion, the usualperspective projection model is applicable [1]. Calibration of the camera/lens sys-tem can therefore be carried out quasi-statically using one or other of the varietyof methods based on known scene geometry (e.g. [2,3]). Look-up tables for fo-cal length, principal point, distortion, and so on can be constructed, all indexedby the mechanical setting of the lens. This approach is accurate but laborious, andlikely to be applicable only to the most expensive commercial zoom lenses that feedback positional information and exhibit little mechanical hysteresis. (Stepper motordrives finesse the problem of feedback, but zoom lenses move so slowly that espe-cial care must be taken not to make a new demand before the current step sequenceis completed.)
The impracticality of calibrating in this way has spurred efforts to devise methodsof auto- or self-calibrating camera/lens combinations. Maybank and Faugeras [4]and Pollefeys et al. [5] considered the self-calibration problem for fixed and vary-ing intrinsic parameters respectively when the camera undergoes general motionrelative to the scene. Our interest here though is in cameras which rotate but whichdo not translate, and general motion methods do not perform gracefully in spe-cial motion cases. Hartley considered the self-calibration of a rotating camera withfixed intrinsics [6], and de Agapito, Hayman and Reid [7,8] and Seo and Hong [9]devised methods for rotating cameras with varying intrinsics. These latter methodsare based on the observation that under pure rotation the projections of the viewedstatic structure in successive frames are related by planar (
�����) homographies,
even when the camera’s intrinsic parameters change.
While much is known about the ways in which the optical properties of lenses af-fect static calibration, rather little is known of how these same properties affectself-calibration. Of particular concern here is the effect of radial distortion on theself-calibration of rotating cameras, and the paper offers two straightforward phys-ical insights which allow, first, the prediction that self-calibration algorithms forrotating cameras will be tolerant of pin-cushion distortion but highly sensitive tobarrel-distortion, and, secondly, the estimation of the level of barrel distortion atwhich a self-calibration algorithm will break down.
The paper goes on to explore four methods of recovering and correcting for theradial distortion during self-calibration. The first method requires that the camera-lens combination be available for pre-calibration of distortion as a function of focallength, and a method of optimization by scaling of the focal length is employed tocorrect the self-calibration. The remaining three methods assume no prior calibra-tion is available. The first of these allows for independent distortions in all imagesand attempts, unsuccessfully, to solve the complete problem by bundle adjustment.The second assumes that the distortion is modelled by a simple function over a fewadjacent frames, and the third assumes a simple relationship between focal lengthand distortion. Although the last two methods are successful to an extent, they are
2
both sensitive to noise.
2 Self-calibration of rotating cameras
As noted in the introduction, methods of self-calibrating rotating cameras withchanging intrinsics involve the infinite homography constraint. Here the methodof de Agapito, Hayman and Reid [7,8] is used, and a brief summary of it follows.
Let the Euclidean scene structure be � , and its projection in image � be ����� ��� .If the camera/lens combination undergoes pure rotation about its optic centre then� �������������� ��� , where ��� is the matrix of camera intrinsic parameters (focal length � ,aspect ratio � , principal point �! #"%$'&("*) , and skew + ) in frame �
� ��,-----.� +/ 0"1 �2�3&("141 5
68777779 � $and ��� is a rotation matrix between some fiducial frame, frame 1 say, and frame � .Thus corresponding image points in images � and : are related by an homography;=< � � < ;=< �>�?�#@� < � < �BABC� �BABC� ���EDSince ��� < F� < �GABC� is a rotation matrix it has the property that ��� < H�0A�I� < , and so� < � I< ; � < � �J� I� ; I� < , a statement of the infinite homography constraint [10,6,11].The constraint may also be expressed using the image of the absolute conicK < L�M� < � I< )*ABCN ; A�I< � K � ; ABC< � ; A�I< C K C ; ABC< C (1)
where at the last step the fiducial frame is accessed by multiplication through thechain of homographies,
;O< C ;=<8P < ABC ;=< ABC P < A�Q DRDRD ; Q�C .In [7], de Agapito et al. proposed a non-linear method to recover the intrinsic pa-rameters, but later they and Hartley [12] devised a linear solution which is the oneused here. The key observation is that in any frame : , the six functions of the in-ternal parameters contained in the symmetric matrix K < are linearly related to thesame functions in the fiducial frame, provided the skew + is assumed zero. FromEq. (1),
K < ,-----.S < C S < Q S <8TS < Q S <8U S <8VS <WT S <WV S <8X
68777779 5� Q,-----.
5 1 Y 0"1 5[Z � Q Y &=" Z � QY G" Y &=" Z � Q � Q]\ Q" \ & Q" Z � Q68777779 < D (2)
3
But skew is zero in all images : giving the constraint^ :`_ S < Q a� ; A�I< C K C ; ABC< C ) C!Q 1 D (3)
Each image : then contributes a row to the design matrix for the functions of theinternal parameters in the reference frame
,-----....
......
......
...b < C b < Q b <WT b <8U b <WV b <WX......
......
......
68777779,--------.S C�CS C!Q...S C X
68777777779 ��cDThe
b’s here are functions of the measured elements of the
;’s and derived from
Eq. (3). Although sufficient to solve for the calibration, de Agapito et al. found thatstability is much improved by adding in knowledge of the principal point, and thisis done in the experiments reported here. Then there are three rows per frame inthe design matrix, and the least-squares solution is found by eigen-decomposition.With K C found, the linear relationships are used in reverse to find K < , and the in-trinsic parameters for frame : are then easily recovered from Eq. (2).
2.1 Experimental Example
Fig. 1(b) shows a plot of the pan-tilt and zoom settings made during a deliberatelylarge rotation and in/out zoom motion of an EIA (Electronique Informatique Appli-cations) high precision computer-controlled zoom lens and Sony XC-75CE CCDcamera mounted on the pan-tilt head shown in Fig. 1(a). The rotating camera wasviewing a static scene. Corner features were detected and matched during simul-taneous robust computation of the inter-image homographies over 50 consecutiveframes. Fig. 1(c) compares the self-calibration of �(d and �[e ( ��]�fd ) obtained with-out compensation for radial distortion with a conventional static calibration usinga 3D grid but with compensation for radial distortion. The g 5f1
% underestimatefrom self-calibration was consistent and repeatable for the EIA lens.
The following sections show that the substantial under-estimate is due to radial dis-tortion with a positive distortion coefficient — pin-cushion distortion. This is at firstsurprising, because neglecting pin-cushion distortion in a conventional static cali-bration using known scene points results in a small over-estimate of focal length.
4
(a)
−5
0
5
−5
0
5
1500
2000
2500
3000
Tilt (degrees)Pan (degrees)
Foca
l Len
gth
(pix
els)
1000 2000 3000 1000
2000
3000
4000
Foca
l Len
gth
Fy (p
ixel
s)
Motor Demand
1000 2000 3000 1000
2000
3000
4000
Foca
l Len
gth
Fx (p
ixel
s)
Calibrated data True data
(b) (c)
Fig. 1. (a) An EIA Servolens/Sony XC-75CE lens/camera combination undergoing rota-tion by a pan-tilt head during self-calibration. (b) The pan, tilt and zoom settings during therotation. (c) The crosses show the recovered focal length without compensating for radialdistortion, compared with those obtained from a conventional calibration after compensat-ing for radial distortion.
3 Distortion and the calibration of rotating cameras
A non-ideal lens or group of lenses does not form a focussed image on the im-age plane, but rather on some more complex surface close-by. High quality zoomlens systems can use some 15 lenses to flatten the focus surface, removing variousaberrations and distortions. For a typical lens the most prominent of these weredescribed by von Seidel in the 1850’s: spherical aberration, coma, astigmatism,curvature of field, and radial distortion. Radial distortion (Fig. 2) arises from theinevitable misplacements of the stop irises used to improve the focussing proper-ties of zoom lenses [13].
5
(a)
(b)
(c)
Fig. 2. Placing an iris (or stop) in the system reduces the spread of rays and improves thedepth of field. Unless the chief ray is chosen (a) improved focus is at the expense of creatingpin-cushion (b) or barrel distortion (c).
3.1 Describing radial distortion
Von Seidel expressed the displacement of an ideal image point as a Taylor series inits radial distance, h%i , from the centre of distortion
hRjk�h%i�� 5 \ml C h Qi \ml Q h Ui \ml T h Xi \ DRDRD')where l � are the distortion parameters and hfj the resulting distorted position [13].For ideal spherically convergent rays there is an analytical relationship between thedistortion parameters l C , l Q etc, but for complex lens systems this is rarely the case,and it is usual to treat the distortion terms as independent (see [14], chapter 7).
For the lenses used in Section 5, the distortion is relatively small and the first dis-tortion term l C dominates. It will be shown that estimating even a single distortionparameter during self-calibration is difficult, and so discussion is restricted to afirst order model. Here we adopt that due to Harris [15], with approximations forthe forward distortion and backward correction of
6
hRjnoh%i ,. 5p 5qYsr l C h Qi69 [forward] (4)
h%i]ohRj ,. 5p 5 \ r l C h Qj69 [backward].
Harris’ approximation has the advantage of equal complexity in distortion and cor-rection whilst producing topologically consistent images, which is not the casewhen the series is just truncated. For the low levels of distortion found in prac-tice the Harris model is also accurate over a greater range than truncation and isequivalent, again to first order, to expressions used by other researchers who haveinvestigated radial distortion, eg. Tsai [2], Li and Lavest [16] and Weng et al [17].Below l C will be referred to as just l .
3.2 Distortion and rotating cameras
As illustrated in Fig. 3(a), an essential requirement for the self-calibration algorithmdescribed in Section 2 to succeed is not merely that all rays should pass through therotation centre, but that the 3D lines joining sets of corresponding points should allmeet at the rotation centre. Only then is the infinite homography constraint valid.
This is no longer the case when the lens exhibits radial distortion. Fig. 3(b) showsan example for pin-cushion distortion. Actual image points lie further away fromthe principal point than their ideal counterparts, and most lines joining pairs of cor-responding points meet between the rotation centre and the image planes, as thesymmetric pair in the figure suggests. In contrast, Fig. 3(c) shows that with mod-erate barrel distortion the meeting points are pushed beyond the rotation centre. Asbarrel distortion further increases these match lines become parallel and eventuallydiverge, pushing the intersection point to the other side of the image planes.
Different pairs of rays will of course have different meeting points, as shown inFig. 3(e,f). An impression of their distribution can be obtained by considering themeeting of symmetrically disposed rays for some fixed rotation angle
rOtbetween
the images � and : . Consider two ideal rays at uwv to the line of symmetry. Assumingthe focal length � is fixed between the views, the one at \ v strikes images � and :at displacements x �n Y �zy*{O|?� t}Y vn) x < \ �~y'{O|� t \ vn)from the principal points of � and : . Applying distortion l givesx��
� x � Z p 5qY�r l x Q�
x��< x < Z p 5qY�r l x Q< D7
Image
Rotation andoptic centre
Image
Scene
xui j
X
Principalpoint
ujx
i
i
z
φθ
xdi
xdj
X
ζ
xdi
X
djx
(a) ���3� (b) ���o� (c) �����
−600 −400 −200 0 200 400 600−2000
−1000
0
1000
2000
3000
4000
X (pixels)
Z (p
ixel
s)
−600 −400 −200 0 200 400 600−2000
−1000
0
1000
2000
3000
4000
X (pixels)
Z (p
ixel
s)
−600 −400 −200 0 200 400 600−2000
−1000
0
1000
2000
3000
4000
X (pixels)
Z (p
ixel
s)
(d) ���3� (e) ����� (f) �����Fig. 3. Top: The effect of radial distortion on match-lines for (a) no distortion, (b)pin-cushion distortion, and (c) barrelling distortion. Bottom: By examining a large num-ber of match-lines for realistic focal-lengths and image rotations it is evident that (d) forzero distortion all the lines pass through the rotation centre, but that (e,f) with distortion nomatch-lines pass through the rotation centre, instead forming an envelope around it.
The point of intersection of the rays is at � along the line of symmetry, where
�G��v$ t )NL���z����� t}Y x �� ���J| t ) \ ���}���J| t \ x �� �%��� t )� x �� Yx �< )��%�O� t \ r �}���>| t �
x �� \
x �< )��W�J| t D (5)
Fig. 4 shows the dependence of � on v witht
fixed at1 D 5 rad. It is evident that:
(i) Pin-Cushion distortion results in line intersections which are almost exclusivelycontained between the rotation centre and the image planes, and furthermore themajority lie consistently close to the true rotation centre. (The exceptions are raysclose to the intersection point of the two images � and : — i.e., those with v�� 1
.)Pin-cushion distortion will cause the self-calibration algorithm to under-estimatefocal length, but the calibration should always succeed.
(ii) Barrel distortion results in line intersections which are spread across a widerange. The most common value is a substantial over-estimate of the focal length,but the large range of values must result in a large variance in, or even completefailure of, the self-calibration.
8
−300 −200 −100 0 100 200 300−2000
−1000
0
1000
2000
3000
4000
5000
6000
Image plane
True camera centre
Image point position (pixels)
Inte
rsec
tion
dept
h O
z (pix
els) κ = +2×10−8
−300 −200 −100 0 100 200 300−2000
−1000
0
1000
2000
3000
4000
5000
6000
Image plane
True camera centre
Image point position (pixels)
Inte
rsec
tion
dept
h O
z (pix
els) κ = −2×10−8
(a) (b)
Fig. 4. Graphs showing the dependence of the position along the optic axis of the inter-section of two symmetric rays as a function of increasing angular spread between the rays,plotted for (a) pin-cushion distortion, and (b) barrelling distortion.
−0.5 0 0.5 1x 10
−7
0
500
1000
1500
2000
2500
3000
3500
4000
True focal length
κ (pixels−2)
Foca
l Len
gth
(pix
els)
Fig. 5. The focal length recovered by the self-calibration algorithm operating on radiallydistorted image features. The different characteristics of the calibration for negative andpositive distortion are as predicted.
3.3 Experimental results for self-calibration in the presence of distortion
The predictions of the geometrical study are readily confirmed by applying theself-calibration algorithm itself to features derived from an image sequence andthen distorted over a range of l values from negative to positive. Fig. 5 shows thefocal lengths recovered from trials at each of 20 different values of distortion. Ineach trial 30 images were generated from a camera undergoing rotation. The focallength was held constant at 2000 pixels throughout. The recovered focal lengthsand standard deviations are derived from 100 trials at each distortion setting.
As expected, pin-cushion distortion causes under-estimation of the focal lengthwhich increases gently with increasing distortion. Because of the clustering of in-
9
Scene point
Planar
Barrel
Spherical
Pin−cushion
Optic centre
Optic axisφ
Fig. 6. Radial distortion can be modelled as a projection onto a curved image.
tersection points discussed above, the standard deviation is small. In contrast, in-creasing barrel distortion causes a rapid increase in both the mean estimate of land its standard deviation. For this focal length, the calibration becomes unreliableat l � Yz���35f1 A�� , and breaks down catastrophically at around l � Y~����5f1 A�� .4 Predicting calibration failure
Although the first geometrical argument suggests that self-calibration using the in-finite homography constraint will break down under increased barrel distortion, asecond argument allows prediction of the value of l at which breakdown is likelyto occur.
Consider the scene points projected onto the four surfaces shown in Fig. 6. Theideal image is that projected onto the planar surface. If the lens introduces pin-cushion distortion, the resulting image is as if first projected onto a surface curv-ing upwards, then warped flat so that image distances equal those measured in thecurved surface. The barrel-distorted image surface curves in the opposite direction.For normal ranges of focal length, image size and distortion, these projection sur-faces are well approximated by paraboloids.
Now contrast projection onto a planar surface with that onto a spherical imagesurface, and consider the motion of points. In 2-D, denoting the distances along theplanar and spherical projection surfaces as +�� and +f� , respectively,
In the spherical case there is complete ambiguity between focal length and rotation,
10
and focal length changes can only be determined up to scale. The term � �v has nodependence on the position in the image, and on its own is of little use in determin-ing � unless the angular velocity is known. The term +(� �� Z � does depend on pointposition and so more than one point allows calculation of �� Z � , the relative changein focal length. In the planar case there is also the extra term + Q� �v Z � which, togetherwith the � �v term, allows recovery of both � and �v .
Only the middle term in the planar optic-flow equation allows absolute calibrationof the focal length. In the spherical case, calculation of the absolute focal length �is impossible, but changes in focal length +(� �� Z � will be recovered correctly. Thisimplies that the focal length curves produced by self-calibrating will have the rightoverall shape, but will have arbitrary scaling.
Both pin-cushion and barrel distortion bend the image, but barrel distortion bendsit towards the degenerate case of a spherical projection where recovery of absolutefocal length is impossible. For some value of distortion l#¢ , the image plane closelymatches the spherical projection, so that the distorted planar distance closely ap-proximates the spherical distance, +=� H�Gy*{O| ABC ��+'� Z ��) . That is breakdown shouldoccur when +'�p 5£Y¤r lB¢ + Q� �¥�Gy*{(| ABC ��+'� Z �n)¡DRe-arranging and taking first order approximations on both sides gives5 \¦lB¢ +RQ� � y*{(| ABC ��+'� Z ��)+'� Z � � 5£Y 5� ��+'� Z �n)8Qand thus the breakdown value is l0¢ � Y 5� � Q DFor a focal length of 2000 pixels, as used for Fig. 5, this predicts ln¢ Yz� D ���5f1 A�� . As this level of distortion is reached we should expect self-calibration to failcompletely, and this is indeed evident in the experimental results in Fig. 5.
5 Estimating radial distortion during self-calibration
Because radial distortion is a non-linear process described in the ideal image plane,it is strictly not possible to correct it from measurements in digitised imagery with-out first knowing elements of the intrinsic calibration. In practice, however, therequirements are far less onerous, for three reasons. First, the perfection of imagingsemiconductor fabrication and electronic timing gives rise to negligible skew. Sec-ondly, the distortion centre does not necessarily coincide with the principal point
5.1 Method 1: Recovering l using a known distortion model
Lenses with fixed (but unknown) focal length will have fixed distortion, and in hiswork on self-calibrating rotating cameras with fixed intrinsics Hartley was able toadd the extra parameter l to the optimization [18]. Sawhney and Kumar [19] andStein [20] have used similar processes.
However, with the greater degrees of freedom introduced by varying intrinsics ourfirst approach here is to model the relationship between distortion and focal length.To develop it, a number of zoom cameras were calibrated using Tsai’s method [2].The observed variations of l with focal length are shown in Fig. 7(a) for three zoomlenses: (i) a bottom-end motorized surveillance lens (TECSEC) and (ii) a top-endcomputer controlled lens (EIA) both attached to a Sony XC-75CE PAL camera;and (iii) and a high-quality Sony VL500 Firewire camera with integral zoom lens.Error bars are omitted for clarity, but typical repeatability over ten trials at each often focal lengths was u ° ��5R1 A�� pixels A�Q . Also shown is the curve at the thresholdfor algorithm breakdown, lG¢ Y�5[Z(� � Q .Wiley and Wong [21] suggested that such curves are well represented by joiningtwo quadratic splines, but Collins and Tsin [22] proposed a simpler l l " Z � Qmodel so that at fixed distortion, l " , increasing the focal length causes the imageonly to scale up and not to further distort (see Eq. 4). However this model is itselfbased on an idealized model, not on lens mechanisms, and we find that the data areonly well-fitted by allowing offsets on both the distortion and focal length, so that
l ²± Y ³��´ \ ��) Q D (7)
The resulting fitted curves are shown in Fig. 7(b).
Using this function, the method proceeds by achieving consistency between a set ofpostulated focal lengths �Gµ>¶J·¹¸»º� and the corresponding set �#¼J½Mº ¾� returned by the self-
12
1000 1500 2000 2500 3000 3500 4000−3
−2
−1
0
1
2
3 x 10−7
EIA Servolens
TECSEC Servolens
Sony VL500
Focal Length (pixels)
κ (p
ixel
s−2)
1000 1500 2000 2500 3000 3500 4000−3
−2
−1
0
1
2
3 x 10−7
Focal Length (pixels)
κ (p
ixel
s−2)
(a) (b)
Fig. 7. (a) Measurements of the distortion parameter against focal length for three lenses.The typical repeatability over ten trials at each of ten focal lengths was ¿ÁÀ2Â~Ã*� A�� pixels A�Q .(b) Best-fit inverse-square curves for three lenses mentioned in the text. In both subfigures,the curve � ¢ �ÅÄ£ÃÇÆRÈfÉ Q is shown dashed.
calibration after correcting the images �} 5 $RDRD%D%$*Ê using l C ��� µ>¶J·¹¸»º� ) . To keep theoptimization manageable, the empirical and theoretical evidence of Figs. 1 and 5is exploited — that under substantial pin-cushion distortion any uncorrected self-calibration will underestimate all the focal lengths by approximately the same fac-tor.
The algorithm is as follows:
(1) Distort all features positively by significant amount and self-calibrate togive a set of focal lengths � ¼J½Mº ¾� . Positive distortion ensures that the result-ing calibration is stable, but the resulting focal lengths are likely to beunder-estimates.
(2) Choose a scale Ë¦Ì 5to test. Scale up the focal lengths recovered in step
(1), so that �0µJ¶>·¹¸»º� �Ë¡�G¼J½Mº ¾� .(3) Look up the distortion parameters l �����0µ>¶J·¹¸»º� ) , and correct the original image
points.(4) Re-calibrate from these new points to update ��¼>½Mº ¾� .(5) Determine the cost Í¥�ΪÏ�sÐ �G¼>½Mº ¾� Y �0µ>¶J·¹¸»º� Ñ Q .(6) Repeat from step (2) for a new scale, chosen to move towards a minimum
in Í .
Fig. 8 shows two sets of cost curves that are descended using this technique forthe measured optical characteristics of (a) the Sony VL500 and (b) the EIA Serv-olens/Sony XC-75 combination. The sequences used contained 30 images duringwhich the camera was rotated and zoomed. Between 50 and 100 features weretracked between consecutive images, and Gaussian white noise added with stan-dard deviation varied from 0 to 1.5 pixels. In the figure, the scale axis is normalized
to an average “true” value, and in Fig. 8(a) the minima do indeed occur aroundË Z Ë µ>¶JÕ'½ g 5. In subfigure (b) the results exhibit two minima, symmetrically posi-
tioned either side of unity. This tends to occur when a lens exhibits both positiveand negative distortion within a sequence, when the use of a single “true” scale isan obvious over-simplification (Fig. 5).
5.2 Methods 2-4: Recovering l without a known distortion model
When the distortion curve cannot be obtained in advance, the distortion must be es-timated alongside the calibration. This may occur because the camera is either notavailable at all or is inconveniently located for traditional calibration. Pre-recordedvideo, films and remote cameras (such as surveillance cameras) are likely to fallinto this category. Three possibilities have been considered, increasing in special-ization from having no prior constraints on the distortion to having the specificmodel constraints just introduced.
5.2.1 Method 2: no constraints
In this approach no constraints are placed on the distortion, with the value of lin each of the Ê frames being treated as an independent parameter in the self-calibration. Bundle-adjustment was used to minimize errors in the image [23]. Ex-perimentation with this method by the present authors and by others in our labo-ratory [8] showed that typical cost surfaces had many local minima, and the cali-bration failed unless all the parameters were initialised unreasonably close to theirtrue values.
The third method determines the distortion in Ö consecutive frames of a run of Ê ,by simultaneously computing the pairwise inter-image homographies and minimiz-ing the discrepancies between modelled and observed image positions. Method 2indicated that it is impractical to recover Ö independent distortions, so instead itis assumed that, because the focal length is a smoothly varying function of framenumber, so too is the distortion.
Fig. 9(a) shows the results of using cubics to model the distortion in overlappingsets of Ö× 5f1
consecutive frames. The camera was rotated, while the focal length(and hence distortion) of the EIA servolens was increased then decreased back to
15
its original value. The fitting was performed using Chebyshev polynomials, allow-ing some overlap between consecutive sets, and shows good agreement with theveridical distortion data obtained by a static calibration. With the distortion found,the image data can be corrected, and the self-calibration method used to find an un-biased estimate of the focal length and camera motion (Fig. 9(b,c)). The resultingl vs. � values can be used to build a model of the sort described in the previoussection, as shown in Fig. 9(d).
Although these results appear promising, further experimentation showed that suc-cess depended heavily both on obtaining matches around the periphery of the imageand on the noise in the point positions being unreasonably low relative to the sizeof the distortion. These requirements appear too demanding for general use.
5.2.3 Method 4: Global parametrization
The final method of recovering the distortion during self-calibration parametrizesl as a function of focal length using the model of Eq. (7) and performs an iterativeminimization for the parameters and calibration.
The first stage follows that of the known model case, and all image features aredistorted by a positive factor to ensure stable self-calibration. It is then assumed thatthe focal length �[�Ø" so recovered for every frame � are correct up to a single changeof scale, so that the distortions are determined by three parameters Ù=±�"f$ ³ � " $'´ � "%Ú ,
l �Ø"Á²±�" Y ³ "�!´Ç" \ Ën"*�[�Ø"Ç) Q @± " Y ³ � "��´ � " \ �f�Ø"Ç) Q D (8)
These parameters are determined by minimizing the radial component of the trans-fer error under the inter-image homographies. The homographies are initialized attheir pre-correction values, and limits can be placed on the other parameters as itis known that ³ ��Û 1
and ´ ��Ü 1. Once the homographies are refined, the self-
calibration can be re-run, yielding new focal lengths �(� C , and the process iterated.
Fig. (10) shows the resulting estimated curve obtained for the Sony VL500, andit is found in reasonable agreement with the curve obtained earlier using staticcalibration.
However the success of even this more global approach to modelling is difficultto assure. Fig. 11 shows two example cross sections of the three-dimensional pa-rameter space obtained from synthetic data when there was zero noise added tofeature positions used in matching and when Gaussian noise with a standard de-viation of 0.2 pixel was added. Even this small amount of noise washes out thesharp minimum into a mostly flat surface with only the mildest depression at theveridical value. As with the patch fitting above, this technique only works well withextremely accurate feature localisation or large distortion.
16
1400 1600 1800 2000 2200 2400−3
−2
−1
0x 10
−7
Focal Length (pixels)
Kap
pa (p
ixel
s−2)
Estimated curveTrue curve
Fig. 10. The variation of distortion with focal length recovered during self-calibration byMethod 4 (Eq. 8), compared with that from a static calibration. The static calibration valuesare Ý � �a��Þ¹�f�[ßfà , á � �âÄ¡��Þäã�à , and the minimum for the 0.2 pixel noise case is found atÝ � �3��Þ¹�f�[ã%À , á � �ÅÄ¡��Þäãfß%À .
00.005
0.010.015
0.02
−0.25−0.2
−0.15
−15
−10
−5
bc
resi
dual
s (lo
g)
00.005
0.010.015
0.02
−0.25−0.2
−0.15
−6
−5.5
−5
−4.5
−4
bc
resi
dual
s (lo
g)
(a) (b)
Fig. 11. Error surfaces over two of the three distortion curve parameters for the SonyVL500, with (a) zero noise and (b) 0.2 pixels noise.
6 Conclusions
In this paper it has been shown that the small and varying amounts of radial distor-tion present in imagery obtained using zoom lenses cause serious problems for self-calibration algorithms based on rotation and the infinite homography constraint.This constraint does not strictly require all rays to pass through the rotation cen-tre, but rather that all the lines joining corresponding image points intersect at thecentre of rotation. Arguments based on geometric optics showed that with pin cush-ion distortion the intersection points tend to cluster some way between the rotationcentre and the image plane, so that self-calibration should be stable but produce agradually worsening under-estimate of focal length. With barrel distortion however,the meeting points do not cluster, and under increasing distortion move rapidly be-
17
yond the rotation centre, through u infinity and back behind the image plane, sug-gesting that self-calibration should at first produce an overestimate then collapsecompletely. A second argument was used to predict the value of barrel distortionat which self-calibration algorithms based on rotation should break down. Botheffects were observed as predicted.
The paper suggested four methods for calculating the distortion during self-calibration,one involving a pre-computed model of distortion against focal length, and three forthe case where no such model is known beforehand. In the latter case it was shownthat some sort of constraint between the distortion in neighbouring frames is essen-tial, but a fully satisfactory method proved elusive. Most success was obtained withthe last method which recovered a global three-parameter model of the sort used inthe known-model case.
Acknowledgements
This work was supported by Grants GR/L58668 and GR/N03266 from the UK’sEngineering and Physical Science Research Council. The authors are grateful toTorfi Thorhallsson for valuable discussions.
References
[1] R. Willson, Modelling and calibration of automated zoom lenses, Ph.D. thesis,Carnegie Mellon University (1994).
[2] R. Tsai, A versatile camera calibration technique for high-accuracy 3d machine visionmetrology using off-the-shelf tv cameras and lenses, IEEE Journal of Robotics andAutomation 3 (4) (1987) 323–344.
[3] O. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint, MITPress, Cambridge, Massachusetts, 1993.
[4] S. Maybank, O. Faugeras, A theory of self-calibration of a moving camera, Int. Journalof Computer Vision 8 (2) (1992) 123–151.
[5] M. Pollefeys, R. Koch, L. VanGool, Self-calibration and metric reconstruction inspite of varying and unknown internal camera parameters, in: Proc 6th Int Conf onComputer Vision, Bombay, 1998, pp. 90–95.
[6] R. Hartley, Self-calibration from multiple views with a rotating camera, in: Proc 3rdEuropean Conf on Computer Vision, Stockholm, 1994, pp. A:471–478.
[7] L. de Agapito, E. Hayman, I. Reid, Self-calibration of a rotating camera with varyingintrinsic parameters, in: Proc 9th British Machine Vision Conf, Southampton, 1998,pp. 105–114.
18
[8] L. de Agapito, E. Hayman, I. Reid, Self-Calibration of Rotating and ZoomingCameras, Int. Journal of Computer Vision 45 (2).
[9] Y. Seo, K. Hong, Auto-calibration of a rotating and zooming camera, in: Proc 5thIAPR Workshop on Machine Vision Applications, 1998.
[10] Q.-T. Luong, T. Vieville, Canonic representations for the geometries of multipleprojective views, in: Proc 3rd European Conf on Computer Vision, Stockholm, 1994,pp. 589–599.
[11] Q.-T. Luong, T. Vieville, Canonical representations for the geometries of multipleprojective views, Computer Vision and Image Understanding 64 (2) (1996) 193–229.
[12] L. de Agapito, R. Hartley, E. Hayman, Linear self-calibration of a rotating andzooming camera, in: Proc of the IEEE Conf on Computer Vision and PatternRecognition, 1999, pp. 15–21.
[13] M. Born, E. Wolf, Principles of Optics, 7th edition, Cambridge University Press, 1999.
[14] M. Freeman, Optics, 10th edition, Butterworths, 1990.
[15] C. Harris, Camera calibration, Tech. rep., Roke Manor Research, Siemens, UK (1992).
[16] M. Li, J.-M. Lavest, Some aspects of zoom-lens camera calibration, Tech. Rep.CVAP172, Computational Vision and Active Perception Laboratory, KTH, Stockholm,Sweden (February 1995).
[17] J. Weng, P. Cohen, M. Herniou, Camera calibration with distortion models andaccuracy evaluation, IEEE Transactions on Pattern Analysis and Machine Intelligence14 (10) (1992) 965–980.
[18] R. Hartley, Self calibration of stationary cameras, Int. Journal of Computer Vision22 (1) (1997) 5–23.
[19] H. Sawhney, R. Kumar, True multi-image alignment and its application to mosaicingand lens distortion correction, in: Proc of the IEEE Conf on Computer Vision andPattern Recognition, 1997, pp. 450–456.
[20] G. Stein, Lens distortion calibration using point correspondences, in: Proc of the IEEEConf on Computer Vision and Pattern Recognition, 1997, pp. 602–608.
[21] A. G. Wiley, K. W. Wong, Geometric calibration of zoom lenses for computer visionmetrology, Photogrammetric Engineering and Remote Sensing 61 (1) (1995) 69–74.
[22] R. Collins, Y. Tsin, Calibration of an outdoor active camera system, in: Proc of theIEEE Conf on Computer Vision and Pattern Recognition, 1999, pp. 528–534.
[23] W. Triggs, P. McLauchlan, R. Hartley, A. Fitzgibbon, Bundle adjustment: A modernsynthesis, in: W. Triggs, A. Zisserman, R. Szeliski (Eds.), Vision algorithms: theoryand practice, LNCS, Springer Verlag, 2000.
![Radial Distortion Triangulation...radial lens distortion, and many different camera geometry solvers based on this model were proposed recently [7, 15, 6, 21, 20, 24, 27]. In the division](https://static.documents.pub/doc/80x56/613a87990051793c8c011815/radial-distortion-triangulation-radial-lens-distortion-and-many-different-camera.jpg)
