Visual servoing of mobile robots using non-central catadioptriccamerasHadi Aliakbarpour a,b,∗, Omar Tahri a, Helder Araujo a
a Institute of Systems and Robotics, University of Coimbra, Portugalb Computer Science Department, University of Missouri–Columbia, USA
h i g h l i g h t s
• An image-based visual servoing method for mobile robots is proposed.• Using the radial model, new visual features with decoupling properties are derived.• Efficient IBVS approach based on desired value of the interaction matrix is proposed.• Real experiments using a real robot have been carried out (different scenarios).
This paper presents novel contributions on image-based control of a mobile robot using a general cata-dioptric camera model. A catadioptric camera is usually made up by a combination of a conventionalcamera and a curved mirror resulting in an omnidirectional sensor capable of providing 360° panoramicviews of a scene. Modeling such cameras has been the subject of significant research interest in the com-puter vision community leading to a deeper understanding of the image properties and also to differentmodels for different types of configurations. Visual servoing applications using catadioptric cameras haveessentially been using central cameras and the corresponding unified projection model. So far only in afew cases more general models have been used. In this paper we address the problem of visual servo-ing using the so-called radial model. The radial model can be applied to many camera configurations andin particular to non-central catadioptric systems with mirrors that are symmetric around an axis coin-ciding with the optical axis. In this case, we show that the radial model can be used with a non-centralcatadioptric camera to allow effective image-based visual servoing (IBVS) of a mobile robot. Using thismodel, which is valid for a large set of catadioptric cameras (central or non-central), new visual featuresare proposed to control the degrees of freedom of amobile robot moving on a plane. In addition to severalsimulation results, a set of experiments was carried out on Robot Operating System (ROS)-based platformwhich validates the applicability, effectiveness and robustness of the proposed method for image-basedcontrol of a non-holonomic robot.
Vision-based servoing approaches are versatile and effectivemethods to control robot motion by using camera observations. Inpractice, when a conventional camera is used there is no guaran-tee that the features remain in the camera’s field of view (FOV).
∗ Corresponding author at: Institute of Systems and Robotics, University ofCoimbra, Portugal. Tel.: +351 914244403.
In order to overcome the problem of keeping the features inthe camera’s FOV, several methods have been developed namely:based on path planning [1], zoom adjustment [2], and switch-ing control [3]. Simpler and differing approaches use omnidirec-tional vision sensors to increase the FOV usingmirrors.Wide-anglecameras include catadioptric systems that combine mirrors andconventional cameras to create omnidirectional images providing360° panoramic views of a scene, or dioptric fish-eye lenses [4,5].Lately they have been the subject of an increasing interest fromrobotics researchers [6–10].
Having a single viewpoint in omnidirectional imaging systemsis very practical [4,11]. Such systems have a single center of
2 H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) –
projection, in such way that every image point measures the ir-radiance of the light passing through the same viewpoint. Onecan model a central imaging system as two consecutive projec-tions: spherical and perspective. Geyer and Daniilidis in [12] de-rived the geometric model of these systems, and called it theunified model. In this model perspective projection corresponds toa special configuration. This formulation has been used by manyresearch works in the area of visual servoing. Tahri et al. in [9] pro-posed an image-based visual servoing (IBVS)method to control thetranslational degrees of freedom (DOFs) which is invariant to rota-tional motion. In [7] an IBVS is proposed. This method is based onthe auto-epipolar condition, which occurs when the current anddesired catadioptric views undergo a pure translation. Themethodhas been applied to control a holonomic mobile robot. Adaptationof the classical image-based visual servoing to a generalized imag-ing model was proposed in [13], by modeling the cameras as setsof 3D viewing rays. In [6], the projection of 3-D straight lines inthe image plane on a central catadioptric system is used to con-trol a 6DOFs holonomic robot and a non-holonomic mobile robot.Although the existing methods are effective for single viewpointcatadioptric systems, in practice just a few realistic configurationslead to a single-viewpoint catadioptric system, asmentioned in [4].
The problem ofmodeling the general case of a non-central cata-dioptric camera is a hard problem and still has only been tackledpartially in computer vision. For this reason, iterative approachesare usually applied by some researchers to determine the reflectionpoint on themirror. Recently, a forward projectionmodel has beenproposed for the case of non-central catadioptric cameras consist-ing on a perspective camera and a rotationally symmetric conic re-flector [14]. In the latter work, the optical path from a given 3Dpoint to the given viewpoint is obtained by solving a 6th degreepolynomial equation for general conic mirrors. For a spherical mir-ror, the forward projection equation reduces to a 4th degree poly-nomial, resulting in a closed form solution. In [15], an analyticalforward projection equation for the projection of a 3D point re-flected by a quadric mirror into the image plane of a perspectivecamera, with no restrictions on the camera placement is derived.They show that the equation is a 8th degree polynomial in a sin-gle unknown. In the absence of an analytical and simple forwardmodel, the determination of some elements like the interactionmatrix required for image-based servoing becomes difficult.
Becerra et al. in [16] proposed a method for image-based con-trol of a mobile robot, employing only the angular (bearing) infor-mation provided by an omnidirectional camera, where a 1D radialcameramodelwas used. In theirwork, the 1D trifocal tensormodelwas computed using the angles of visual features as a tool for thecontrol. In [17] the authors proposed a method to control a mo-bile robot by exploiting panoramic vision. In this work, the corre-sponding features between omnidirectional images are employedto define a control strategy which moves a ground robot along asequence of desired positions. It is assumed that the robot tracksvisual features in panoramic views of the environment that it ac-quires as it moves.
For scene reconstruction or control purposes, a completeknowledge of the projectionmodel is not always required. In [18], atechnique to linearly estimate the radial distortion of a wide-anglelens given three views of a real-world plane has been proposedusing the radial projectionmodel. Based on [18], linearmethods forthe estimation of multi-view geometry of 1D radial cameras havebeen studied in [10] and [19].
In this paper, it will be shown that the simple radial projectionmodel can be sufficient for mobile robot control using a largefamily of catadioptric cameras. More precisely, the contributionsof this paper are:• An image-based visual servoingmethod formobile robotsmov-
ing on a plane, valid for a large set of catadioptric cameras (in-cluding radially symmetric non-central cameras) is proposed.
• Using the radial model, new visual features with decouplingproperties are derived.
• An efficient image-based visual servoing approach based on thedesired value of the interaction matrix is proposed. Unlike theexisting visual servoing methods, we incorporate the depth ofthe features (scene points) in the control task.
• The feasibility and effectiveness of the proposed method havebeen demonstrated with real experiments using a real robot.The rest of this paper is organized as follows:
• The proposed radial camera model and its usage for visualservoing are introduced in Section 2.
• The issues related to the selection of adequate visual featuresfrom images are discussed in Section 3. A control lawwhichusesthe proposed features is introduced in the same section.
• In Section 4 we present our experimental setup and achievedresults using ROS platform and real robot.
• Conclusions and future works are presented and discussed inSection 5.
1.1. Notation and symbols
Throughout this article we use the following notations:• Scalars are typeset in regular lower-case.• Vectors are denoted by lower-case boldface.• Matrices are typeset in capital boldface.• Variables with the * (star) as exponent denote they are com-
puted using the information corresponding to the robot’s goalpositions.
2. Radial camera model for visual servoing
2.1. Radial camera model
A catadioptric systemmade up by the combination of a conven-tional pinhole camera and a rotationally symmetric mirror, shownin Fig. 1, is considered. The camera is positioned in such a way asto have its optical axis aligned with the mirror axis. Using the ra-dial projection model, 3D point p = (X, Y , Z) is reflected first ona point on the mirror pr = [Xr Yr , Zr ] before being projected ontothe image plane asxm (expressed in metric homogeneous coordi-nates):
xm = (xm, ym, 1) =pr
Zr. (1)
Pointxm is projected into the camera’s image plane atxd = (xd,yd, 1), expressed in pixels and can be obtained fromxm using:xd = Kxm (2)where K is the matrix of the camera intrinsic parameters, fx and fybeing the focal lengths, µx and µy are the principle point coordi-nates (and zero skew) as follows:
K =
fx 0 µx0 fy µy0 0 1
. (3)
From the laws of reflection, we have: (a) vector n, the center ofthe projection c, the 3D pointsp andpr belonging to the sameplaneπ as shown in Fig. 1(b) the angle between the incident ray and n isequal to the angle between the reflected ray andn. In [19] and [10],the intersection of the planes π defined by the image points frommultiple views has been used to recover linearly the structure ofthe scene.
The mirror is rotationally symmetric, and therefore the opticalaxis also belongs to π. Further, for symmetry reasons, the centerof distortion (in our case the center of the image) (crad) and theprincipal point coincide. In this paper, the normalized coordinatesof xm are used as follows:
xn =xm
∥xm∥(4)
H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) – 3
so that they belong to the unit circle. Later, xn will be used in thederivation of the new features and image servoing algorithm.
Note that the computation of xn from the image points ex-pressed in pixel only requires the knowledge of the principal pointcoordinates (which coincides with the distortion center) and theratio of the focal length parameters. One can prove it as follows:from (4):
xn =xm
x2m + y2m
yn =ym
x2m + y2m.
(5)
On the other hand, using (2), xm and ym can be expressed as follows:xm =
xd − µx
fx
ym =yd − µy
fy.
(6)
In (5) xm and xy can be substituted by their expressions from (6)yielding:
xn =xd − µx
(xd − µx)2 + ρ2f (yd − µy)2
yn =yd − µy
ρ−2f (xd − µx)2 + (yd − µy)2
(7)
where
ρf =fxfy
. (8)
Eq. (7) implies that further than the coordinates of the distor-tion’s center, only the ratio between the two focal lengths needs tobe known in order to obtain xn. Note that the center of the image(center of distortion) can be approximated by estimating the cen-ter of the mirror border (assumed to be a circle or an ellipse) [20].
Letxu = (xu, yu, 1) =pZ be the point coordinates (homoge-
neous) inmetric units of the projection of p using pinholemodel asshown in Fig. 1. Let xu = [xu, yu] be the non-homogeneous coordi-nates corresponding toxu. Since the center of the pin-hole cameraand p belong to plane π, the point xu also belongs to the intersec-tion of this plane with the image plane. Therefore, crad, xu and xmbelong to the same line. We have then xn =
xu∥xu∥
, which leads to:
xn =
xnyn
=
1√X2 + Y 2
XY
. (9)
Note that xn is not defined only if the 3D point belongs to the cam-era optical axis (which does not happen in the case of the catadiop-tric camera). In the following xn will be used to define new visualfeatures to control themotion of amobile robotmoving on a plane.
2.2. Visual servoing
In visual servoing the time variation s of the visual features s canbe linearly expressed with respect to the relative camera–objectkinematics screw:s = Lsτ, (10)where Ls is the interaction matrix related to s. Usually, the controlscheme is designed to reach an exponential decoupled decrease ofdifferences on the visual features to their goal value s∗. If we con-sider an eye-in-hand system observing a static object, the corre-sponding control law is:
τc = −λLs+(s − s∗), (11)
where Ls is a model or an approximation of Ls, Ls+ the pseudo-inverse of Ls, λ a positive gain tuning the time to convergence,
Fig. 1. Axial catadioptric system.
and τc = (υc, ωc) the camera velocity sent to the low-level robotcontroller. In practice the matrix Ls could be chosen as the cur-rent value of the interaction matrix Ls. This choice (except in thecase of a singularity) ensures an exponential decrease of the erroron features in image. Unfortunately, further to the problem of lo-cal minima and singularities, computing the exact current value ofthe interaction matrix requires the knowledge of the depth infor-mation. Determining this information can be time consuming, butalso subject to instabilities. In order to avoid these problems, usingthe depth information of the goal values in Ls can be an alternative(i.e Ls(Z∗)). Unfortunately, in that case, the exponential decrease ofthe feature errors in the image will no longer be ensured. Further-more, if the value of Ls changes, its pseudo-inverse should also becomputed at each iteration, which also can be time consuming ifthe size of the features vector increases significantly.
In order to avoid computing the depth information and invert-ing Ls at each time that the velocities have to be computed, astraightforward choice is to use the constant matrix Ls∗ computedfor the goal pose. Using Ls∗ also permits to avoid the problem ofthe singularities (except if the desired position corresponds to asingular value of Ls∗ ). Despite the mentioned advantages above, inpractice using Ls∗ ensures a local and limited domain of conver-gence around the desired position as compared to the case whereLs is used. Furthermore, the behavior of the feature errors in the im-age as well as in 3D space is neither always predictable nor alwayssatisfactory. Combining Ls∗ and Ls in a single control law has beenstudied in [21] and [22] to improve the stability and 3D behavior.Unfortunately, once again, and as far as Ls is involved in the controllaw, the depth information has to be determined and Ls to be in-verted. Actually, the limited domain of convergence and the unpre-dictable behavior obtained using Ls∗ results, in large part, from theproblem of the tensor frame change. Indeed, Ls∗ expresses the vari-ations of features as a function of the camera velocities expressedin the goal frame. Therefore, if the current and the goal frames havedifferent orientations, the tensor change of frame has to be takeninto account since the velocities are to be applied in the currentcamera frame. This problem has been highlighted in [23] for in-stance. More precisely, instead of using the average Ls =
Ls∗+Ls2 , as
proposed in [21], [23] proposed to use Ls =Ls+Ls∗ T−1
2 after inte-grating the spatial motion transform T. In this paper we only usethe goal value of the interaction matrix in the control law. Moreprecisely, the velocity computed using Ls = Ls∗ in the control law(11) has to be multiplied by a spatial transformation T. A method
4 H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) –
to effectively approximate the tensor change of frame in the caseof amobile robot (to avoid reconstructing depth data and invertingLs at each iteration of the control loop) will be described next.
3. Visual features selection and control law
In the next paragraph, new visual features are proposed andtheir corresponding interactionmatrices are derived. A control lawusing the goal values of the interaction matrices is also proposedand derived.
3.1. Visual features suitable to control camera translational velocities
In this paragraph, we introduce new visual features which aresuitable for the control of the translation component of the cameramovement. To do so, we use the inner product between the coor-dinates of two points xn i and xnj in the image:
cij = x⊤
n i xnj. (12)
Note that at most Np(Np − 1)/2 of cij can be defined, Np being thenumber of the feature points. By taking the derivative of (12), weobtain:
cij = x⊤
nj xni + x⊤
ni xnj. (13)
The interactionmatrix corresponding to xn can be obtained by tak-ing the derivative of (9):
Lxn =Lxnυ Lxnω
(14)
with:
Lxnυ =
−(1 − x2n)
dxnynd
0
xnynd
−(1 − y2n)
d0
(15)
and
Lxnω =
−xnynzn −(1 − x2n)zn yn
(1 − y2n)zn xnynzn −xn
(16)
where d =√X2 + Y 2 and zn = Z/d. By combining (14) and (13),
the interaction matrix Lcij = [Lcijυ Lcijω] corresponding to cij can bethen obtained by:
Lcijυ =
−1dj
+cijdi
x⊤
n i +
−1di
+cijdj
x⊤
n j 0
(17)
and
Lcijω =ynjzij + ynizji −xnjzij − xnizji 0
(18)
where zij = zni − cijznj and zji = znj − cijzni. From (17) and (18),it can be seen that cij is invariant to the motion around the opticalaxis (which corresponds to the normal of the plane of motion). Weassume that the camera is mounted on the mobile robot so thatthe translational motion takes place on the plane defined by thevectors x and y of the camera frame. Therefore, only the first twoentries of the matrix Lcijυ are useful for the control of the transla-tional motion with respect to the x-axis and the y-axis. In the nextparagraph, we explain how to select an adequate feature to controlthe remaining DOF, namely the rotation around the optical axis.
3.2. Visual features suitable to control camera rotation
In this paragraph, we introduce visual features which are suit-able for the control of the rotational component of robot motion.
We consider α as a natural feature in the image that can be ob-tained from points xn to control the rotation of the robot on aplane as:
α = atan2(yn, xn). (19)
The time variation of α can then be obtained by:
α =xn yn − yn xn
x2n + y2n= xn yn − yn xn. (20)
By combining (20) and (14), the interaction matrix correspondingto α can be obtained by:
Lα =
ynd
−xnd
0 xn zn yn zn −1. (21)
From (21), we can notice the direct link betweenα and the rotationaround the z-axis. For the sake of robustness, all projected pointsxn have to be used. A simple way to do it is by stacking all the an-gles αi in a feature vector. A better choice can be combining all thepoints in a single and unique feature to control the rotation aroundthe z-axis. A straightforward and simple way to use all the rotationangles could be using their average αa =
1N
Ni=1 atan2(yni, xni).
Such feature is directly related to ωz . However, the arithmeticaverage of rotations does not correspond to the real average of ro-tations, especially when the difference between the rotations con-sidered is large. For instance, for a rotation angle close to π , anddue to the effect of noise or due to translational motion, the com-puted rotation angles can have opposite signs. Therefore, the ro-tation angle corresponding to their arithmetic mean would have avalue close to 0 instead of π or −π generating some discontinu-ities in the estimation of αa. In this paper, we propose to define arotation angle αm for a virtual point whose coordinates are com-puted as a linear combination of the projections of points on thecircle. Let p1 be the point defined by:
p1 =
Ni=1
aixni. (22)
From p1, we define a new point v1 belonging to the unit circle by:
v1 =p1
∥p1∥. (23)
By taking the derivative of (23), the interactionmatrix correspond-ing to v1 can be obtained by:
Lv1 =I2 − v1 v⊤
1
∥p1∥
Ni=1
aiLxni . (24)
Let αm be the angle defined by:
αm = atan2(v1y, v1x). (25)
By taking the derivative of (25), it can be obtained:
αm = v1xv1y − v1yv1x. (26)
By combining (26) with (24), Lαmωz = −1 is obtained. As a resultone can conclude that αm varies linearly with respect to the veloc-ity ωz .
The parameters ai have to be determined before defining pointv1 on the unit circle. More precisely, we have to define a virtualpoint p∗
1 and next represent it as a linear combination of the de-sired projected points on the circle x∗
ni. For the sake of simplicity,
p∗
1 is chosen to be unitary (∥p∗
1∥ = 1 then v∗
1 =p∗1
∥p∗1∥
= p∗
1). Letp∗
2 be also a unit vector perpendicular to p∗
1 . As a result p∗
1 and p∗
2form a direct orthogonal frame basis V∗
= [p∗
1; p∗
2]. It is possibleto represent any given frame basis V∗ as a linear combination ofthe coordinates of a set of points. For instance, V∗ could be set as
H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) – 5
Fig. 2. (a) Egocentric polar coordinate system with respect to the observer. (b) Camera frame position.
the desired frame of the camera. In any given frame basis V∗, eachprojected point onto the circle can be expressed as:
x∗
ni = b1iv∗
n1 + b2iv∗
n2. (27)
Let B be the 2 × N matrix that defines the coordinates of all theprojected points on the new frame basis. We have:
X∗
nt = V∗ B (28)
where X∗nt = [x∗
n1 x∗
n2 . . . x∗
nN ], and B = V∗⊤X∗nt . From (28), V∗ can
be represented as a linear combination of X∗nt by:
V∗= X∗
nt B+ (29)
B+ is a N × 2 matrix corresponding to the pseudo-inverse of B.Therefore, the ai can be chosen as the first columns of B+.
3.3. Control law
Let sc be the feature vector obtained by stacking the features cijand s∗c their goal values. Let Lsc be the interaction matrix obtainedby stacking the two first entries vx and vy of the interaction ma-trix corresponding to each feature cij. Only the two first entries aretaken into account because we are only concerned with a planarmotion and cij is invariant to the rotation around z-axis. Let us con-sider that the goal is to move the desired camera position towardsthe initial one. Therefore, the velocities that have to be applied tothe goal position of the camera using its corresponding interactionmatrix are obtained from:
υ∗
xυ∗
y
= −λL+
sc∗(s∗
c − sc)
ω∗
z = λ(α∗
m − αm) − Lαmυx∗ υ∗
x − Lαmυy∗ υ∗
y
(30)
where Lαmυx∗ and Lαmυy∗ represent the variation ofαm with respectto the velocities υx and υy respectively (Notice that υx, υy and ωzare the only non-zero components of the camera motion.). Let usconsider the three frames shown in Fig. 2(b). Let Fc and Fc∗ repre-sent respectively the current and the goal camera frames and Fcian intermediate frame that has the same position of the center asFc∗ but the orientation of Fc . As it can be seen from Fig. 2(b), thetranslational velocity to be applied to the frame Fc to move it to-wards its desired position is equal to the negative of the velocitiesthat move Fci towards Fc . Therefore, to control the translationalmotion of the current camera position, it is more adequate to usethe interaction matrix corresponding to sc computed for the posi-tion corresponding to Fci:
υxυy
= −λL+
sci(sc − sci). (31)
In the case of the projection onto the sphere, it was shown in [9]that two interaction matrices L2in and L1in related to an invariant tothe 3D rotation in and computed respectively for two camera poses
1 and 2 separated by a rotational motion are related by equation:
L2in = L1in1R2 (32)
where 1R2 is the rotationmatrix. Similarly, it can be shown for fea-ture sci that if only a rotation is considered betweenFci andFc∗, Lscican be obtained from Lsc∗ by:
Lsci = Lsc∗c∗Ri (33)
where c∗Ri is the 2-dimensional rotation matrix corresponding tothe rotation angle γ between Fc∗ and Fci. We continue to provethat cij will be invariant to any rotation around z axis. Based on(12), cij is a function of xn i and xnj:
cij = g(xn i, xnj) = x⊤
n i xnj. (34)
Now we apply an arbitrary rotation R to the inputs of thefunction g:
Eq. (35) shows that g(R xn i,R xnj) is equal to g(xn i, xnj) and as a re-sult the feature cij is invariant to any rotation around z axis. Since cijis invariant to a z-axis rotation one obtains sci = sc∗. By combiningthis result and (33) in (31), we obtain:
υxυy
= −
iRc∗λL+
sc∗(sc − sc∗). (36)
By combining (30) and (36), we finally obtain:υxυy
= −
iRc∗
υx∗υy∗
. (37)
On the other hand, since the z-axis has the same orientation in thecurrent and the goal camera poses, we choose ωz = −ωz∗. In thenext section, we explain how to effectively approximate iRc∗.
4. Experiments
In this section we first present a set of simulations and after-wards some results obtained in a real scenario will be presented.For both cases, a differential drive mobile robot is considered asnon-holonomic vehicle. The uniformcoordinate systemused as therobot model is shown in Fig. 2(a). The control law proposed in [24]is used to transform the camera velocities into linear and steer-ing velocities to be applied to themobile robot. More precisely, thesteering velocity is defined by:
ω =υl
r
k2(δ − arctan(−k1 θ)) +
1 +
k11 + (k1θ)2
sin(δ)
(38)
where k1 and k2 are two positive constants, υl is the linear velocity,r is the distance between the current position of the robot and thegoal position, θ is the orientation of the goal T with respect to theline of sight defined between the current and the desired position
6 H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) –
a b
c d
Fig. 3. Trajectory performed by the robot using the real 3D parameters and the 3D parameters estimated from an IBVS in the case of four different rotation angles: (a) 0°,(b) 90°, (c) 180°, (d) 270°.
of the robot (T ); δ is the orientation of the vehicle heading withrespect to the line of sight. To apply the control law (38), it is nec-essary to represent the parameters υl, r , θ and δ as a function of thecartesian camera velocities obtained by IBVS. First, the linear veloc-ity can be defined as υl =
υ2x + υ2
y . The linear velocity becomesnull when the translational motion is null (because of the invari-ance of the feature sc). The angle δ can also be estimated as thedirection of the velocity to be applied to the current camera posefrom δ = atan2(υy, υx) (since the camera is rigidly attached to therobot). The distance from the initial to the desired camera pose can
be approximated by r =
√υ2x +υ2
yλ
after removing the timeunit. Thisis equivalent to setting υl
r = λ in (38). Finally, angle θ can be de-fined as the rotation angle between the initial and the goal camerapose. More precisely, we choose θ =
ωzλ
− δ. Note also that θ asdefined in Fig. 2(a) is equal to γ as defined in Fig. 2(b). The rotationmatrix iR∗ is also estimated using γ =
ωzλ
as a rotation angle.
4.1. Simulation results
In the first set of simulation results, we compare the applicationof the IBVS described in this paper with the application of the ex-act 3D parameters θ , r and δ in the control law (38).More precisely,four examples of robot parking are considered: all cases start fromthe same initial pose and have to converge towards four differentdesired poses obtained after shifting the initial pose by the transla-tional motion defined by [4 4] meters but with different orienta-tions corresponding respectively to the angles 0, π
2 , π and 3π2 . The
images of the following set of 8 points defined in the 3D environ-ment were used to compute the velocities for IBVS:
quired to compute the interaction matrix, are assumed to beknown (they could, for example, be estimated using the multiple-view scene reconstruction proposed in [15]). The simulations havebeen performed using the ISFfMR Integrated Simulation Frame-work for Mobile Robots [25]. The constants k1 = 4 and k2 = 12were used in the control law (38) to control the robot using IBVSand the real 3Dparameters. Fig. 3 shows the trajectories performedby the robot using IBVS and using the real 3D data. From this fig-ure, it can be seen that the trajectories are similar and that IBVShas a performance similar (and as satisfactory) as using the 3Dreal data. Video 1 (attachment to this paper1) shows the behaviorof the robot along the performed trajectories. The video confirmsthe similarities between the two trajectories and the good conver-gence towards the desired robot pose. Only the velocities of therobot along these trajectories differ: the convergence using IBVS isslightly slower than using the 3D real data for this experiment. Thisis due to the fact that the amplitude of the translation estimatedusing IBVS is smaller than the real one.
In the second set of simulations, a wrong scale for the positionof the points is used (r∗ = 1.3 r∗ is used as depth instead of the realvalues). The same cases of robot parking considered in the first setof simulations are considered here. Fig. 4 compares the trajecto-ries performed by the robot using the real value of ϵr∗ and r∗ tocompute the desired value of the interaction matrices. From theseplots, it can be seen that the errors on the scene scale have neg-ligible influence on the trajectories performed, which means thatthe curvature defined by the ratio between the steering velocityand the linear velocity ω
vlis not very sensitive to the scene scale.
Videos2 2 and 3 compare the behavior of the robot motions along
H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) – 7
a b
c d
Fig. 4. Trajectory performed by the robot using the real 3D parameters and the 3D parameters estimated from an IBVS in the case of four different rotation angles: (a) 0°,(b) 90°, (c) 180°, (d) 270°.
Fig. 5. Images of the robot and of images acquired by the robot on its position. (a) and (c): the robot in the goal position and the corresponding image acquired by the robot,respectively. (b) and (d) the robot on an arbitrary initial position and its corresponding image, respectively.
the performed trajectories and the point motions in the image us-ing the correct and an erroneous scene scale. The two videos showthat the robot converges in all cases, but with different velocities:the convergence using r∗ = 1.3 r∗ to compute the interaction
matrices is faster than when using the real depth r∗. This is dueto the fact that the scene scale used is bigger than the real one,and therefore the amplitude of the translational motions to be per-formed is amplified.
8 H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) –
Fig. 6. Diagram of errors in terms of points and features during the convergence: Fig. (a) shows the reduction of errors between the nine goal image points and theircorrespondences in the current image. Using these nine image points, 36 features were defined. The errors between the goal features and current features are plotted in (b).
Fig. 7. Plots for the velocities during the iterations: (a) depicts the camera linear velocity. (b) shows the linear and angular components of the velocity of the robot duringthe convergence.
Fig. 8. Error in the camera’s angle corresponding to αm in (25).
4.2. Results using real scenario
Here we present some experiments performed in real scenarioby a real robot, using the open-source meta-operating system,ROS [26]. A Pioneer 3-DX [27] has been used as differential drivenon-holonomic vehicle. Two different catadioptric systems wereused in our servoing experiments, both made up of a hyperbolicalmirror (50 mm diameter) and a perspective camera (6 mm focallength). The specifications of the hyperbolic mirrors are as follows:
Equation Parameters
Mirror A X2
a2−
Y2
b2= −1 a = 20.8485, b = 26.8578
Mirror B X2
a2−
Y2
b2= −1 a = 23.6634, b = 27.1301
In the first experiment we use mirror A and later it will be sub-stituted bymirror B,while preserving the same configuration. Oncethe robot is placed on the goal position (Fig. 5(a)) in the scene, animage is grabbed as the goal image (Fig. 5(c)). As seen in the figures,some black circles (here nine circles) were fixed on the walls and
used as visual beacons. The centers of these beacons are automat-ically extracted (using openCV library) and used as our goal pointsfor the goal image. The proposed method requires the knowledgeof the depths of all the points in the scene, observed from thegoal position, in order to compute the interaction matrices. There-fore in our experiment, the distance (depth) between each of thefixed beacons to the robot is measured manually, while the robotis parked in this position. One can conclude that a coarse measure-ment is enough and there is no need to have precise depth, as dis-cussed in the simulation part.
As already mentioned, one of the advantages of the methodproposed in this paper is that the camera does not need to be fullycalibrated. In our experiments, ρf has been given the value 1 andthe center of the image/distortion was computed as the center ofthemirror border image. It has as coordinatesµx = 643,µy = 535(automatically obtained through applying the Hough transform forcircle detection defining the mirror border in the image).
The robot is placed on an arbitrary position, the initial position(Fig. 5(b)). Then the robot moves, so that it can reach the goal posi-tion with the goal orientation. After the extraction of features fromboth goal and initial images, the proposed control law is appliedto the robot. In these experiments, k1 = 1, k2 = 1 are used asconstant parameters in the control law (38). As expected the robotperforms a smooth trajectory, converges and stops nearly on thegoal position. Fig. 6(a) depicts the errors (distances in pixels) be-tween the goal image points and their corresponding points in thecurrent as a function of servoing iterations. As one can see, the con-vergence is quite smooth. There is a negligible spacial error (lessthan 1 cm) between the parked and the goal position (the behaviorof the robot is shown in video 3). The same behavior is shown inFig. 6(b) where the errors between the features (c∗
ij − cij) are plot-ted. Note that all 36 available cij (from9 tracked features points) areused in this experiment. The velocities of the camera and robot areshown in Fig. 7(a) and (b), respectively. Fig. 8 presents the errors inthe angle αm values during servoing.
H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) – 9
Fig. 9. Images acquired by the robot on its goal position using two different mirrors. Left: mirror A, right mirror B.
For the sake of evaluating the robustness of the proposed ap-proach in case of using different mirrors, a new experiment hasbeen carried out: mirror A (the mirror used so far) has been re-placed by mirror B. Fig. 9-left and -right respectively show the im-ages taken bymirrors A andB, once the robot is placed on the desireposition. In this position, an image is taken while mirror B ismounted. Features are extracted from the newly taken image andkept as the desired features. Then mirror A has been placed backon the robot. Afterwards, the previously presented visual servoingexperiment has been repeated. As one can see from video3 4, thesystem’s behavior is robust despite of using a goal image taken byother camera.
5. Conclusion
In this paper, we have used the radial camera model to proposea novel IBVS. New features were extracted based on this modeland their corresponding interaction matrices were derived. Themethod does not require a fully calibrated camera. The only cali-bration parameters that are required are the ratio of the two focallengths (ρf =
fxfy) and the coordinates of the principal point (µx
and µy). In general, both these parameters can be estimated auto-matically by using the image of themirror border (circle or ellipse).Furthermore, only the goal value of the interaction matrix is usedto compute the velocities, which allows avoiding the estimation ofthe depths of the points, as well as the inversion of the interactionmatrix, during servoing.
As a result of using a simple radial model, the proposed IBVSmethod can be applied for a large class of catadioptric cameras,both central and non-central. The proposed method has beenimplemented using a ROS-based robotic platform. The results ob-tained show the validity and effectiveness of the proposed ap-proach. Our future work includes extending the method for a6 DOFs robot and the use of global visual features as well asmultiple-view geometry in the control law.
Acknowledgments
This paper is an extension of the work originally presented inthe IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS’12) [28]. Hadi AliAkbarpour, Omar Tahri and HelderAraujo are with Institute for Systems and Robotics, Polo II 3030-290 Coimbra, Portugal. Hadi AliAkbarpour is alsowithUniversity of
3 https://sites.google.com/site/otahri/.
Missouri–Columbia, USA. The authors would like to thank the sup-port of project Morfeu–PTDC/EEA-CRO/108348/2008 funded bythe Portuguese Science Foundation (FCT) and H. Araujo would liketo thank the support of project FCT/PTDC/EIA-EIA/122454/2010,funded also by the Portuguese Science Foundation (FCT) by meansof national funds (PIDDAC) and co-funded by the European Fundfor Regional Development (FEDER) through COMPETE OperationalProgramme Competitive Factors (POFC).
References
[1] Y. Mezouar, F. Chaumette, Path planning for robust image-based control, IEEETrans. Robot. Autom. 18 (4) (2002) 534–549.
[2] S. Benhimane, E. Malis, Vision-based control with respect to planar andnon-planar objects using a zooming camera, in: The 11th InternationalConference on Advanced Robotics Coimbra, Portugal, Coimbra, Portugal, 2003,pp. 863–866.
[3] G. Chesi, K. Hashimoto, D. Prattichizzo, A. Vicino, Keeping features in the fieldof view in eye-in-hand visual servoing: a switching approach, IEEE Trans.Robot. 20 (5) (2004) 908–914.
[4] S. Baker, S. Nayar, A theory of catadioptric image formation, Int. J. Comput. Vis.35 (2) (1999) 175–196.
[5] J. Courbon, Y. Mezouar, L. Eck, M.Martinet, A generic fisheye cameramodel forrobotic applications, in: IROS, 2007, pp. 1683–1688.
[6] H. Hadj-Abdelkader, Y. Mezouar, P. Martinet, F. Chaumette, Catadioptric visualservoing from 3d straight lines, IEEE Trans. Robot. 24 (3) (2008) 652–665.
[7] G.L. Mariottini, D. Prattichizzo, Image-based visual servoing with centralcatadioptric camera, Int. J. Robot. Res. 27 (2008) 41–57.
[8] P. Corke, D. Strelow, S. Singh, Omnidirectional visual odometry for a planetaryrover, in: In IEEE/RSJ International Conference on Intelligent Robots andSystems, vol. 4, Sendai, Japan, 2004, pp. 4007–4012.
[9] O. Tahri, Y. Mezouar, F. Chaumette, P. Corke, Decoupled image-based visualservoing for cameras obeying the unified projection model, IEEE Trans. Robot.26 (4) (2010) 684–697.
[10] C. Sagues, A. Murillo, J. Guerrero, T. Goedeme, T. Tuytelaars, L. Van Gool,Localization with omnidirectional images using the radial trifocal tensor, in:IEEE Int. Conf. on Robotics and Automation, Orlando, FL, 2006, pp. 551–556.
[11] T. Svoboda, T. Pajdla, Epipolar geometry for central catadioptric cameras, Int.J. Comput. Vis. 49 (1) (2002) 23–37.
[12] C. Geyer, K. Daniilidis, Mirrors in motion: epipolar geometry and motionestimation, Int. J. Comput. Vis. 45 (3) (2003) 766–773.
[13] A. Comport, R. Mahony, F. Spindler, A visual servoing model for generalisedcameras: case study of non-overlapping cameras, in: 2011 IEEE InternationalConference on Robotics and Automation (ICRA), 2011, pp. 5683–5688.http://dx.doi.org/10.1109/ICRA.2011.5979678.
[14] A. Agrawal, Y. Taguchi, S. Ramalingam, Analytical forward projection for axialnon-central dioptric and catadioptric cameras, in: K. Daniilidis, P. Maragos,N. Paragios (Eds.), ECCV 2010, in: Lecture Notes in Computer Science,vol. 6313/2010, 2010, pp. 129–143.
[15] A. Agrawal, Y. Taguchi, S. Ramalingam, Beyond alhazen’s problem: analyticalprojection model for non-central catadioptric cameras with quadric mirrors,in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011,pp. 2993–3000.
[16] H. M. Becerra, G. López-Nicolás, C. Sagüés, Omnidirectional visual control ofmobile robots based on the 1d trifocal tensor, Robot. Auton. Syst. 58 (6) (2010)796–808. http://dx.doi.org/10.1016/j.robot.2010.02.011.
[17] A. Argyros, K. Bekris, S. Orphanoudakis, L. Kavraki, Robot homing by exploitingpanoramic vision, Auton. Robots 19 (1) (2005) 7–25. http://dx.doi.org/10.1007/s10514-005-0603-7.
10 H. Aliakbarpour et al. / Robotics and Autonomous Systems ( ) –
[18] S. Thirthala, M. Pollefeys, The radial trifocal tensor: a tool for calibrating theradial distortion of wide-angle cameras, in: IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005,pp. 321–328.
[19] S. Thirthala, M. Pollefeys, Multi-view geometry of 1d radial cameras and itsapplication to omnidirectional camera calibration, in: Tenth IEEE InternationalConference on Computer Vision, vol. 2, Beijing, China, 2005, pp. 1539–1546.
[20] C. Mei, P. Rives, Single view point omnidirectional camera calibrationfrom planar grids, in: IEEE Int. Conf. on Robotics and Automation, 2007,pp. 3945–3950. http://dx.doi.org/10.1109/ROBOT.2007.364084.
[21] E. Malis, Improving vision-based control using efficient second-order mini-mization techniques, in: IEEE Int. Conf. on Robotics and Automation, vol. 2,New Orleans, Louisiana, 2004, pp. 1843–1848.
[22] M. Marey, F. Chaumette, Analysis of classical and new visual servoing controllaws, in: IEEE Int. Conf. on Robotics and Automation, ICRA’08, Pasadena,California, 2008, pp. 3244–3249.
[23] O. Tahri, Y. Mezouar, On visual servoing based on efficient second orderminimization, Robot. Auton. Syst. 58 (5) (2010) 712–719.
[24] J.J. Park, B. Kuipers, A smooth control law for graceful motion of differentialwheeled mobile robots in 2d environment, in: 2011 IEEE InternationalConference on Robotics and Automation (ICRA), Shanghai, China, 2011,pp. 4896–4901.
[25] J.C.G. Cadavid, Isffmr integrated simulation framework for mobile robots,https://sites.google.com/site/isffmr/.
[26] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger,R. Wheeler, A.Y. Ng, Ros: an open-source robot operating system, in: Open-Source Software workshop of the International Conference on Robotics andAutomation (ICRA), 2009.
[27] S. Zaman, W. Slany, G. Steinbauer, Ros-based mapping, localization and auto-matic navigation using pioneer 3-dx robot and their relevant issues, in: SaudiInternational Electronics, Communications and Photonics Conference, IEEE,Riad, Saudi-Arabia, 2011, pp. 1–5.
[28] O. Tahri, O. Araujo, Non-central catadioptric cameras visual servoing for mo-bile robots using a radial camera model, in: IEEE/RSJ International Conferenceon Intelligent Robots and Systems, IROS 2012, 2012, pp. 1683–1688.
Hadi Aliakbarpour is currently a post-doctorate fellowin the Computer Science Department at the University ofMissouri. He received his PhD in Automation and Roboticsfrom University of Coimbra (Department of Electricaland Computer Engineering), Portugal. He has been a re-searcher at the Institute of Systems and Robotic (ISR), dur-ing September 2007 till March 2013. His research interestsinclude Computer Vision, Wide Area Motion Imagery(WAMI), Robotics and Computational Human BehaviourAnalysis.
Omar Tahri was born in Fez, Morocco, in 1976. He gothis Masters in photonics, images and system control fromthe Louis Pasteur University, Strasbourg, France, in 2000and received his Ph.D degree in computer science from theUniversity of Rennes, France, in March 2004. Since 2008,he is working as a researcher with the computer visionlab (http://labvis.isr.uc.pt/) of the institute for Systemsand Robotics, Coimbra, Portugal (http://www.isr.uc.pt/).His current research interests include robotics, computervision, vision-based control, and mobile robots
Helder Araujo is a Professor in the Department of Electri-cal and Computer Engineering of the University of Coim-bra, Portugal and also an Invited Professor at Blaise PascalUniversity, Clermont-Ferrand, France. He is a researcher atthe Institute for Systems and Robotics—Coimbra. His re-search interests include computer and robot vision, robotnavigation and localization, and sensormodeling for robotnavigation and localization.
