1
Camera Calibration 1
2008-1
Multi View Geometry (Spring '08)
Prof. Kyoung Mu LeeSoEECS, Seoul National University
Computation of the Camera Matrix P
Camera Calibration 2
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Computation of the Camera Matrix - DLT
• Problem:Given point correspondences , how to find a camera matrix P, such that ?
• DLT:
ii xX ↔ii PXx =
i
ii
i
ii
i
i
i
i
i
i vyuxZYX
vu
ωωω
==
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡, ,
1
P
0PPP
X0XXX0
0PPP
X0XXX0
0PPP
0XXX0X
XX0
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡
−−
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡
−−
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−
3
2
1
3
2
1
3
2
1
Tii
TTi
Tii
Ti
T
Tii
TTii
Tii
Tii
T
TTii
Tii
Tii
TTii
Tii
Tii
T
xy
uv
uvu
v
ωω
ωω
0pA =i
0PXx =× ii
2
Camera Calibration 3
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Computation of the Camera Matrix - DLT
• Using n correspondences, we have
Minimal solutionqP has 11 dof, 2 independent eq./pointsqThus, 6 correspondences are needed
Over-determined solutionqMore than 6 correspondencesqThis can be solved by minimizing the algebraic error (DLT)
with constraintsTppp ),(ˆ where,1ˆor 1 33,3231
33 === ppp
0pA =×122n
Apmin3p̂
=P
Camera Calibration 4
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Degenerate Configurations
• More complicate situations than 2D case
When the camera and points are on a twisted cubic
When points lie on plane or single line passing through the projection center
3
Camera Calibration 5
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
DLT- Data Normalization
• Normalization of 3D data
Simple as before in 2D
Anistropic scaling
32
2D 3D
Camera Calibration 6
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
DLT - Lines
• Extend DLT to Line correspondencesAssume an imaged line l and two points X0 and X1 on the 3D lineSince the plane back-projected from l is PTl, the two points must satisfy
Two linear equations on the entries of P for a line
Thus for n correspondences, we have
0pA =×122n
l
X0
X1PTl
C
why?
4
Camera Calibration 7
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Geometric error
• Assume Xi are known
• The geometric error in the image:
• MLE of P:
points estimated:ˆpoints measured:
ii
i
PXxx=
Camera Calibration 8
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Gold Standard Algorithm
ObjectiveGiven n≥6 3D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelihood Estimation of P
Algorithm(i) Linear solution:
(a) Normalization: (b) DLT:
(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:
(iii) Denormalization:
ii UXX =~
ii Txx =~
UPTP ~1-=
( )2~~,~∑i
iid XPx
5
Camera Calibration 9
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Calibration Example
• Use Canny edge detector• Detect straight lines from the edgels• Find corners by intersecting the lines • Subpixel accuracy < 1/10 • # of constraints > 5 x # of unknowns (11) => more than
28 points are required
Camera Calibration 10
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Errors in the world points
• Errors in the world
• Errors in the image and in the world
ii XPx)
=
6
Camera Calibration 11
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Geometric Interpretation of Algebraic Error
• Normalized DLT minimizes
• Note invariance to 2D and 3D similarity transform given proper normalization
( )2)ˆ,(ˆ∑i
iiid xxω
( ) iTiii yx PX=1,ˆ,ˆω̂ );depth(ˆˆ 3 PXp±=iω
)ˆ,(~)ˆ,(ˆ iiiii fdd XXxxω
then1ˆ if therefore, 3 =p
Tiiii
Tiiii
yx
ZYX
)1,,(
)1,,,(
ω=
=
x
X
Camera Calibration 12
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Estimation of an Affine Camera Matrix
• Affine camera matrix: • For
• By stacking n correspondences, and using pseudo inverse
• Algebraic error: (=geometric error, in this case )
)1,0,0,0( ,),,( 3321 == TTA PPPPPT
iiiiT
iiii yxZYX )1,,( and )1,,,( ω== xX
ii
i
iTi
T
TTi
i
iTT
i
Ti
T
yx
xy
bpAPP
X00X
0PP
0XX0
=⇒
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛⎥⎦
⎤⎢⎣
⎡⇒
=⎟⎟⎠
⎞⎜⎜⎝
⎛−
+⎟⎟⎠
⎞⎜⎜⎝
⎛⎥⎦
⎤⎢⎣
⎡ −
8
2
1
2
1
bApbpA +=⇒= 8888
( ) ( ) ( )∑∑ =−+−=−i
iii
iT
iiT
i dyx222212 ˆ, xxXPXPApb
7
Camera Calibration 13
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Gold Standard Algorithm - Affine Camera
ObjectiveGiven n≥4 3D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelihood Estimation of P(remember P3T=(0,0,0,1))
Algorithm(i) Normalization:(ii) For each correspondence
(iii) solution is and
(iv) Denormalization:
bpA =88bAp += 88
ii UXX =~
ii Txx =~
UPTP ~1-=
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛⎥⎦
⎤⎢⎣
⎡
i
iTi
T
TTi
yx~~
~~
~~
2
1
PP
X00X
)1,0,0,0(~3 =TP
Camera Calibration 14
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Decompose P into K, R, and t
• We note that M=KR is the product of an upper triangular and rotation matrix
Factor M into KR using the QR matrix decomposition. This determines K and R.Then
• Note that this produces a matrix with nonzero skew s
with , and θ is the angle between the image axes
Tppp ),,( 3424141−= Kt
θtan=s
8
Camera Calibration 15
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Restricted Camera Estimation
• Camera estimation with some known parameter information:
The skew is zero s = 0The pixels are square The principal point (x0, y0) is knownThe internal parameter matrix K is known
• Minimizing geometric error:impose constraint through parameterizationSuppose , then2D Image error case: LM minimization 3D and 2D error case: LM minimization
yx αα =
nf 29 RR: →nnf 593 RR: →+
yxs αα == ,0
Camera Calibration 16
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Restricted Camera Estimation
• Minimizing algebraic error:Assume map from param q → , i.e. p=g(q)Minimize
• Reduced measurement matrixReplace with such that
By using SVD
By using QR factorization
Thus,
found be toparameters theis where,)( qqAAp g=
12 x 2nA 12 x 12Â
AAAAPAAPAPAPpAAp ˆˆˆˆˆ TTTTTT =⇒=⇒=
AADV(VDUDV(VDUAA
DVAUDVAˆˆ))())(
then,ˆ and Let TTTTT
TT
===
==
AAAQQA(AA
AQAˆˆ)ˆ)(ˆ
then,ˆLet TTTT ==
=
129 RR:)(ˆ: →⇒→ fgf qAq
]~|[ CRRKP −=
9
Camera Calibration 17
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Restricted Camera Estimation
• InitializationFind initial P using DLTDecompose P and clamp fixed parameters to their desired values e.g. s=0, αx= αy (hard constraint).Set other variables to their values obtained by the new decompositionIt can sometimes cause big jump in error
• Alternative initializationUse general DLTImpose soft constraints
gradually increase weights
Camera Calibration 18
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Exterior Orientation
• If the camera is calibrated (internal parameters are known), then the position and orientation of the camera should be determined
→ Pose estimation• 6 DOF ⇒ 3 points minimal (4 solutions in general)
10
Camera Calibration 19
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Experimental Evaluation
Camera Calibration 20
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Covariance Estimation
• ML residual error
• Example:
37.0 ,365.0 ,197 === σε resn
σεσε
↔−=
res
res nd2/1)2/1(
11
Camera Calibration 21
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Covariance for Estimated Camera
• Compute Jacobian at ML solution, then
variance per parameter can be found on diagonal
• Confidence ellipsoid for the camera center
( )+−= JΣJΣ xP 1T
2χ(chi-square distribution=distribution of sum of squares)
cumulative-1
)(12 α−= nFk
Camera Calibration 22
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Covariance for Estimated Camera
Camera center covariance ellipsoids
12
Camera Calibration 23
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Radial Distortion
center opticalcenter distortion ),( ≈cc yx
),( yx )ˆ,ˆ( yxr),( cc yx
long focal lengthshort focal length
Camera Calibration 24
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Radial Distortion
• Correction of distortion
• Distortion function: Taylor series
• Distortion parameters can be determined by some constraints or cost function:
Straight line should be straightDeviation from the linear mapping
222 )()( cc yyxxr −+−=
L++++= 332
211)( rrrrL κκκsufficient is 1)( 2rrL κ+=
13
Camera Calibration 25
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Radial Distortion
Camera Calibration 26
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Radial Distortion
/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False
/Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice