Automated Extrinsic Laser and Camera Inter-Calibration UsingTriangular Targets
Stefano Debattisti, Luca Mazzei, Matteo Panciroli∗VisLab – Dipartimento di Ingegneria dell’Informazione
Universita degli Studi di Parma, ITALYhttp://www.vislab.it
{deba, mazzei, panci∗}@vislab.it
Abstract— This paper presents a method for solving theextrinsic calibration between camera and multi-layer laserscanner for outdoor multi-sensorized vehicles. The proposedmethod is designed for intelligent vehicles within the au-tonomous navigation task where usually distances betweensensor and targets become relevant for safety reasons, thereforehigh accuracy across different measures must be kept. Thecalibration procedure takes advantage of triangular shapes stillpresent in scenarios, it recovers three virtual points as targetpose in the laser and camera reference frames and then computeextrinsic information of each camera sensor with respect toa laser scanner by minimizing a geometric distance in theimage space. To test algorithm correctness, and accuracy aset of simulations are used reporting absolute error results andsolution convergence, then tests on robustness and reliability(i.e., outliers management) are based on a wide set of datasetsacquired by VIAC prototypes.
Index Terms— Calibration, Inter Calibration, Multi-LayerLIDAR, Camera, Extrinsic Calibration, Triangular calibrationtargets
I. INTRODUCTION
In the mobile robotics field and particularly in the in-telligent vehicle context, the autonomous navigation task isachieved by heterogeneous suit of perception sensors in orderto acquire more rich information content than using onlyactive or passive sensors. For their complementary purposesLIDARs and camera are widely used together on suchvehicles, Zehang et al. in [1] reviews the problem of on-roadvehicles detection using optical sensors and details pros andcons by introducing active sensor in object perception. In thelast year several advanced driver assistance systems coupledcamera and laser devices, such as vehicle detection [2] whereLIDAR provide good range information to define obstacleswhile it is hard to recognize vehicles and cameras thatallows better recognition but does not provide good rangeinformation.
Since laser and camera are complementary sources, inorder to fuse their information the extrinsic calibration ofthese heterogeneous sensors is required. This paper presentan extrinsic calibration algorithm between slave and mastersensors which are a vision system and a multi-layer LIDARrespectively. By detecting a planar triangular board besidethe road, and in front of the sensors, the proposed methodsolves the Location Determination Problem (LDP) in the
∗ Corresponding author.
laser frame by a 3D reconstruction of the triangular targets,than minimizing its projection in the image space. Two mainsteps compose such algorithm: a target detection in eachsensor frame and triangle vertex estimation, and computeextrinsic in a optimization phase that minimizing a geometricdistance. A preliminary comparison between different extrin-sic estimation techniques based on geometric and algebraicminimization functions (i.e., SVD rigid body method orIterative Closest Point) lead us to chose for a non-linearoptimization algorithm based on geometrical constraints inthe image space.
II. PREVIOUS WORKS
The extrinsic camera calibration problem can be solvedusing laser range finders by making correspondences be-tween features seen by laser and the same features seenby the camera. However, produce correspondences betweenthese heterogeneous sensors is complex and arise severalproblems. First of all, the collisions between laser beamsand objects are not visible, since standard automotive camerasystems do not operate in the same frequency spectrum ofthe LRF. Moreover, camera and laser have different errormodels, since cameras are affected by perspective projectiondistortion and laser range measurements provide constantprecision over distance. To deal with these known problemsthere are different methods present in literature that estimate3D features such as points that lying on depth edges than thecamera pose is measured by minimizing a geometric, ratherthen an algebraic distance either projecting LRG featuresin the image plane or projecting camera features in the 3Dspace.
The procedure in [3] describe a extrinsic calibration al-gorithm by placing a planar chessboard at different posi-tions and orientations in front of the sensors, the proposedmethod solves the problem based on 3D reconstruction of thechessboard and geometric constraints between views fromthe stereo vision system and the LIDAR. The three principlesteps of the approach are: 3D corner points triangulation, 3Dplane least-squares estimation, solving extrinsic parametersby applying a non-linear optimization algorithm based onthe geometric constraints. To evaluate the performance ofthe algorithm, experiments based on computer simulationand real data are performed. The proposed approach is also
compared with a popular calibration method to show itsadvantages.
In 2007 Li [4] propose an extrinsic calibration methodbased on line features detection and a Gauss Newton opti-mization with a geometrical distance function. The extrinsicparameters are obtained by minimizing the distance formthe calculated projection of the intersection point to theprojected edges of the checkerboard in the image. Thisprocedure takes advantages by getting easily laser pointusing well known target object, moreover it performs betterresults than previous procedure (i.e.:Zhang-Pless [5], andWasielewski-Strauss [6]). Despite this contribution fit forindoor mobile robot and calibration procedure is achievedby short range distances, it employ a camera with 1024 ×768 pixel resolution and a laser scanner SICK LMS221-30206 that provides 100 FoV and measurement up to 80m with accuracy ±50mm, which are usable also for outdoorvehicles.
Another important contribution is provided by [7] in 2011based on the same ideas of Wasielewski [6] and Li [4],in which extrinsic calibration parameters are estimated byminimizing the distance between corresponding features pro-jected on the 2D image plane. The features are lines asmentioned in [4] although, calibration target is a particulartwo plane panel arranged in a v-shape. Then weightedmeasure of distances coupled with a penalizing functionare used in order to exclude outliers and estimates extrinsiccamera parameters. This procedure extends previous worksand improves performances by remove bias at depth edgesand removing outliers.
A similar platform to ours is considered in [8] 2008. Usingmulti-layer LIDAR and mono camera system, the proposedmethod employ a circular ring as calibration target and solveextrinsic and intrinsic camera parameters. Further resultson error propagation and confidence intervals are useful todefine the scope of this procedure in the automotive field.
III. PROBLEM STATEMENT
Perception system employed on vehicle is composed by afour layer laser scanner and a vision system both placed infront of the vehicle facing the road (i.e. four layer LRF anda stereo system in figure 5 detailed in V). The calibrationtarget designed for sensor registration consists of a planarisosceles triangular shape (of size wt = 0.6m, ht = 1m)of a specific white bright color to easier detect the shapein image. Hence, a Target Reference Frame (TRF), CameraReference Frame (CRF) and Laser Reference Frame (LRF)are defined, notice that a permutation have to be consideredto convert from sensor frame to image frame as shown infigure 1.
Basic hypothesis is that triangular calibration target willbe fully observable by a laser scanner and a camera, sinceits geometry is known triangle vertexes can be computedand used as virtual constraints. Partial observed object leadto errors in optimization phase. The transformation of targetvertex points from LRF to CRF is represented by a rigidtransformation composed by a 3 × 3 rotation matrix R and
TargetReference Frame
ht
wt
Ot
Oc
YcZc
Xc
Ol
YlZl
Xl
Xt
Yt
Zt
Camera
LRF
[ cRt | cTt ]
[ lRt | lTt ]
Fig. 1. Reference frames. CRF and LRF are placed in front of the vehiclefacing the same region of interest and TRF is placed in the middle bottomof the calibration target. Target must be seen completely and simultaneouslyby laser and camera.
a translation vector t
Pc =c Rl · Pl +c tl (1)
Given two set of N 3D points Xi and Yi in the LRFframe and camera frame respectively, where correspondencesbetween points are known. The problem statement consist tosolve the rigid pose transformation [RΘ|t] of the cameraframe with respect to the LRF frame in order to minimizethe overlapping error e between X and Y ,
e = mint,Θ
N∑i
‖X − s(RΘY + t)‖2 (2)
where t = (tx, ty, tz)> is translation vector and Θ =
(roll, pitch, yaw)> is a rotation vector.
IV. EXTRINSIC CALIBRATION
The proposed method is based on three main steps that areexplained in the next three sections: acquiring synchronizeddata from laser and camera (i.e., with the aid of an hardwaretrigger), then select regions of interest and estimate positionof calibration targets with respect to cameras and laserframes, in which targets pose is computed in 2D and 3Drespectively. Finally, an optimization procedure based ongeometric minimization involves image and laser correspon-dences simultaneously by fitting virtual points.
A. Target Detection in laser frame
In this section detail on the range images acquired by laserscanner is given. In this studies approaches are based on fourlayer LIDAR SICK or similarly IBEO Lux that have narrowelevation field of views and high scanning resolution.
The first step is to filter out far points (up to 10m ), since itis not possible to detect related target objects in image withhigh accuracy. Then, a clusterization and recognition phasegroups raw laser points by an euclidean distance policy and
697
Algorithm 1 Extrinsic camera registrationRequire: 4-plane LIDAR.Require: Single or stereo camera.Ensure: Pose of camera frame w.r.t. laser frame
1: Acquiring simultaneously camera and laser frames2: Detect targets in laser frame and generate 3D target pose3: Detect targets in camera frame and generate 2D target
pose4: Estimate rigid transformation [R|t] of cameras frame
w.r.t. laser frame
all only opportune planar thin object will be kept. Next, givena set of N potential targets Tj with j = . . . N a procedureto estimate pose of each Tj is computed. RANSAC fittingis used to generate target plane π represent by a centroid Cand a normal vector n model.
π : C + nx = 0 (3)
All points Xji owned by Tj are projected on π, getting a
new set of points Xπi and a plane reference frame [Rπ|tπ]
apply to Tj , where Rπ = [xπ, xπ × n, n] and tπ =¯Xji . The
virtual point Ot (shown in figure 1 ) in Tj will be computedon π considering only two edge point sets S1 and S2 closerto target segments AC and AB respectively. The procedureto obtain Ot is composed of two steps: first, an intersectionpoint A = (Ax, Ay)> is obtained by intersection of twopolar lines r1 and r2
r1 : x cos(θ1) + y sin(θ1) = ρ1
r1 : x cos(θ2) + y sin(θ2) = ρ2
θ2 = θ1 + α
(4)
solving a constrained non-linear optimization problem de-fined as loss function e in equation 7 with a constrainedangle α,
e = LSS1(r1) + LSS2
(r2) (5)
solution of equation 6 is guaranteed by constrain α.(cos(θ1) sin(θ1)
cos(θ2) sin(θ1)
)(Ax
Ay
)=
(ρ1
ρ2
)(6)
Finally point Ot is obtained by shifting of h from A towardBC then through the point cloud Xj
i . Resulting a targetreference [RT |tT ] frame as follow
RT = [n, v, v × n, ]t = Ot
(7)
B. Target Detection in Camera Frames
Aim of this procedure is to recover target pose in imageframe with less assumptions. Two procedures have beendescribed to recover targets position in camera referenceframe based on monocular view and stereo images.
(a)
(b)
Fig. 2. Target detection with laser data. Plane projection and temporaryreference frame (a). On 2D plane triangle fitting and final result in (d).
1) Mono images: In this section the targets identificationprocedure based on singular camera is detailed. To find atriangular target in the image is necessary to use a imageprocessing algorithm. There are not assumption on the targetorientation and position and moreover it is not possible tocompute distances from uncalibrated sensor. Than first stepof the automatic calibration procedure is to recognize alltriangular shape in the scene with no constraints on the poseof such target.
The overall schema of the algorithm applied to findtriangular targets on a 2D image I is showed in algorithm 2.
Firstly a preprocessing stage involves sequentially colorfiltering, edges extraction, and an Hough transformationthat are applied to I generating image E and accumulatorspace H(ρ, θ) to be able to detect lines on I (as shownin Preprocessing function of algorithm 2). At the endof preprocessing stage parametric equations L of linearcomponents are computed from local maxima extraction on
698
H and applied on I by relation 8.
x cos(θ) + y sin(θ) = ρ (8)
Second algorithm stage is concerning match between linesL obtained by the previous stage and edges in E. Hence,a set of lines Li are obtained as intersection between Eand L where white pixel amount over each line is higherwith respect to a specific threshold set with respect tothe target dimensions in pixels. This stage is detailed onthe ExtractLines function of the algorithm 2. Accord-ing to the line extraction algorithm presented in [9], theSearchTargets procedure is able to extract N segmentsSi for each line L, and compose it in a large ones witha double thresholded algorithm. Finally, a set of line Si,each with a unique segment, is processed by a targets finderstep. This step tests each possible triangle composed as linetriplet t = {Si, Sj , Sk} ∀i, j, k with i 6= j 6= k by matchingcouple of segment vertex, whether all vertex couples differeach other by less than a Manhattan threshold an intersectionpoint of the lines will be computed. Only when three validintersection points are obtained, a triangular shape is detectedby algorithm.
Algorithm 2 2D triangular shape detection on mono imagesInput: 2D image I = {xi,j , i = 1 . . . w, j = 1 . . . h}.Output: Set T of n vector Ti with i = 1 . . . n, where Ti =
[x1, x2, x3]> ∈ I .1: procedure SEARCH TARGETS(I ,E,H)2: [E,H]← Preprocessing(I)3: [L]← ExtractLinesfrom(H)4: Segments S ← L ∩ E5: for all tuple t = {Si, Sj , Sk}wherei 6= j do6: if Are closer segments ? then7: T ← t8: end if9: end for
10: end procedure
2) Stereo images: In this procedure the relative posebetween left and right camera are supposed to be known inorder to assume that distances in the sensor frame are correctand only the absolute pose need to be recorded. This proce-dure is quite similar to the laser procedure due to the samerange information, however a different error distribution ispresent and must be consider in model extraction.
C. Pose Estimation
Point x in LRF is represented on image plane with pixelp, as follow
p = F · x
F is composed by a rigid transformation
x′ = T · x
with T = [RΘ|t] shown in equation 1 and a perspectiveprojection K.
(a) (b)
(c)
Fig. 3. Image processing phases.
p = K · x′
where K is a pinhole model defined as follow
K =
ku s u0
0 kv v0
0 0 1
To be able to compute the extrinsic parameters in T
intrinsic parameters pixel focal length ku, kv , principal point(u0, v0) and a skew s must be known. Extrinsic parame-ters registration problem can be formulated as a non-linearproblem of finding the optimal solution of rotation vector Θand translation vector t that minimize the euclidean distancebetween image points.
minΘ,t‖pi − Fxi‖2 (9)
between image point pi and image projection of laserpoints xi, where index i represent the i-th constraint pair< p,x > that match a triangle vertex in CRF and LRFrespectively. The non-linear optimization is solved by usingthe Levenberg-Marquardt algorithm.
V. EXPERIMENTAL RESULTS
Experiments on synthetic and real data have been con-ducted to evaluate algorithms behaviors.
and a rotation vector Θ = (0.0, 0.0, 0.0)>. Simulated cam-era had a resolution of 1280×960 pixels, 7.50 mm of focallength, 25 Hz frame rate, with a 1/2” (6.4×4.8 mm) CCDsensor, and without distortion. Simulated laser scanner had85 degrees of azimuth FOV with 340 beams, and 3.2 degreesof elevation FOV with 4 beams. Laser provided data atfrequency of 25 Hz. Concerning system resolution, angularazimuth cell size was 0.125 degree, and angular elevationcell size was 0.8 degrees, with a 0.04 m range cell size.
Tables I and II refers to an average value of extrinsicparameters that is evaluated with minimum number of poses.
With 2 targets per frame as shown in figure 4.(b), positionerror (figure 4.(d)) decreases up to 20 cm when more 5 posesare used. In this case orientation error in figure 4.(f) decreasein a range of ±1 degree for each angles. In the other hand,test in figure 4.(a) considers one target per frame and provesthat error on yaw angle does not converge, affecting otherparameters convergence. This behavior is shown in 4.(c)(e).
B. Experiments on real data
Experiments on real data were made with an intelligentvehicle (figure 5.(a)) equipped by VisLab during VIACexpedition. Vehicle was equipped with a autonomous drivingstate-of-art suite of sensors as detailed in [10]. Concerningcomputer vision navigation systems, perception tasks weremade by a 3 synchronized cameras panoramic vision system,a front and rear stereo cameras and three different mounted
(a) Simulation with a target. (b) Simulation with 2 targets.
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Number of poses
Positio
n E
rror
[m]
(c) Position errors with 1 target.
2 4 6 8 10 12 14 16 18 20 220
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Number of poses
Positio
n E
rror
[m]
(d) Position errors with 2 targets.
0 5 10 15 20 25 300
0.5
1
1.5
2
2.5
Number of poses
Orienta
tion E
rror
[degre
e]
(e) Orientation errors with 1 target.
2 4 6 8 10 12 14 16 18 20 220
0.5
1
1.5
2
2.5
Number of poses
Orienta
tion E
rror
[degre
e]
(f) Orientation errors with 2 targets.
Fig. 4. Errors on position and orientation on simulated data. In theorientation error graphs dash line representing yaw is in blue, roll in red,and pitch in green.
laser scanners suites to perform 360 degrees environmentalperception.
Two stereo rigs, shown in figure 5.(b), was used forexperimentation, a large baseline couple of PointGrey Fireflycameras, and a PointGrey Bumblebee X3 for the smallestone. The large stereo rig had a 0.80 m baseline, each camerashad 752×480 pixels resolution. Point Grey Bumblebee X3had a 24 cm baseline, a resolution of 1280×960 pixels, anda 3.8 mm focal length.
Laser scanner used for experiments was a 4-layers SickLD-MRS-400001. Laser scanner had 3 different angularresolution 0.125◦/ 0.25◦/ 0.5◦ with scanning frequency of12.5 Hz / 25 Hz / 50 Hz respectively. Distance resolutionwas 40 mm, and the statistical error σ1 was 100 mm with arange up to 50 m at 10% reflectivity.
Real test was conducted with 2 calibration targets perframe in a fixed position 5 m far from vehicle, detailed infigure 5.(c). Results on 20 frames is shown in Table III, thathighlights imprecision concerning target pose estimation inLRF due to high level of noise. This problem can be solvedacquiring frame with different targets pose.
This article facing the multi-sensorized intelligent vehiclesextrinsic calibration problem. Cameras and laser scannersextrinsic inter-calibration is performed by the aim of trian-gular shape targets. Proposed method might be used bothfor monocular and stereo vision systems, where intrinsicparameters are considered as known. Furthermore, inter-calibration can be coupled with stereo auto-calibration to beable to recover correct distance measure.
The proposed method reports good results in simulated en-vironments, however on real scenarios it is strongly affectedby laser scanner statistical error. Therefore a multi posecalibration is preferred to come through laser uncertaintylimitation as discussed in experimental results section.
Further works are focused on using this procedure in urbanenvironment by enforcing the shape detection phase, andusing such method with known objects beside the road, evenwith road signs whenever a laser scanner can percepts it.
[1] Z. Sun, G. Bebis, and R. Miller, “On-road vehicle detection: a review,”Pattern Analysis and Machine Intelligence, IEEE Transactions on,vol. 28, no. 5, pp. 694 –711, may 2006.
[2] L. Huang and M. Barth, “Tightly-coupled lidar and computer visionintegration for vehicle detection,” in Intelligent Vehicles Symposium,2009 IEEE, june 2009, pp. 604 –609.
[3] Y. Li, Y. Ruichek, and C. Cappelle, “3d triangulation based extrinsiccalibration between a stereo vision system and a lidar,” in IntelligentTransportation Systems (ITSC), 2011 14th International IEEE Confer-ence on, oct 2011, pp. 797 –802.
[4] G. Li, Y. Liu, L. Dong, X. Cai, and D. Zhou, “An algorithm forextrinsic parameters calibration of a camera and a laser range finderusing line features,” in Intelligent Robots and Systems, 2007. IROS2007. IEEE/RSJ International Conference on, 29 2007-nov. 2 2007,pp. 3854 –3859.
[5] Q. Zhang and R. Pless, “Extrinsic calibration of a camera and laserrange finder (improves camera calibration),” in Intelligent Robots andSystems, 2004. (IROS 2004). Proceedings. 2004 IEEE/RSJ Interna-tional Conference on, sept.-2 oct. 2004, pp. 2301 – 2306 vol.3.
[6] S. Wasielewski and O. Strauss, “Calibration of a multi-sensor systemlaser rangefinder/camera,” in Intelligent Vehicles ’95 Symposium.,Proceedings of the, sep 1995, pp. 472 –477.
(a)
(b)
(c)
Fig. 5. VIAC prototypes used for test on real data and results. (a) Vehiclefrontal perception system. (b) Detail on two stereo camera systems withboth sort and wide baselines. (c) Data obtained by the perception systemafter sensor registration.
[7] K. Kwak, D. Huber, H. Badino, and T. Kanade, “Extrinsic calibrationof a single line scanning lidar and a camera,” in Intelligent Robots andSystems (IROS), 2011 IEEE/RSJ International Conference on, sept.2011, pp. 3283 –3289.
[8] S. Rodriguez F, V. Fremont, and P. Bonnifait, “Extrinsic calibrationbetween a multi-layer lidar and a camera,” in Multisensor Fusion andIntegration for Intelligent Systems, 2008. MFI 2008. IEEE Interna-tional Conference on, aug 2008, pp. 214 –219.
[9] R. Grompone von Gioi, J. Jakubowicz, J.-M. Morel, and G. Randall,“On straight line segment detection,” Journal of Mathematical Imagingand Vision, vol. 32, pp. 313–347, 2008.
[10] L. Mazzei, P. Medici, and M. Panciroli, “A lasers and camerascalibration procedure for viac multi-sensorized vehicles,” in IntelligentVehicles Symposium (IV), 2012 IEEE, june 2012, pp. 548 –553.
![Calibration of mirror position and extrinsic parameters in ... · 8 1.1. Previous work on catadioptric calibration 9 Centralcatadioptricsystems[1]allowforasingle-viewpointprojectionmodel,](https://static.documents.pub/doc/80x56/5f89e03fa053072c1633220c/calibration-of-mirror-position-and-extrinsic-parameters-in-8-11-previous-work.jpg)
