1
Geometric Models & Camera Calibration
Reading: Chap. 2 & 3
2CS4243: Geometric Models
Introduction
We have seen that a camera captures both geometric and photometric information of a 3D scene.
Geometric: shape, e.g. lines, angles, curvesPhotometric: color, intensity
What is the geometric and photometric relationship between a 3D scene and its 2D image?We will understand these in terms of models.
3CS4243: Geometric Models
Models
Models are approximations of reality.“Reality” is often too complex, or computationally intractable, to handle.Examples:
Newton’s Laws of Motion vs. Einstein’s Theory of RelativityLight: waves or particles?
No model is perfect.Need to understand its strengths & limitations.
4CS4243: Geometric Models
Camera Projection Models
3D scenes project to 2D imagesMost common model: pinhole camera model
From this we may derive several types of projections:
orthographicweak-perspective para-perspectiveperspective
least accurate
most accurate
5CS4243: Geometric Models
Pinhole Camera Model
real image plane virtual image plane
f f
YZ
optical centre
object
image is inverted image is not inverted
6CS4243: Geometric Models
Real vs. Virtual Image PlaneThe image is physically formed on the real image plane (retina).The image is vertically and laterally inverted. We can imagine a virtual image plane at a distance of fin front of the camera optical center, where f is the focal length.The image on this virtual image plane is not inverted.
This is actually more convenient.Henceforth, when we say “image plane” we will mean the virtual image plane.
7CS4243: Geometric Models
Perspective Projection
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
1ZYX
⎥⎦
⎤⎢⎣
⎡''
yx
f
u
8CS4243: Geometric Models
Perspective ProjectionThe imaging process is a many-to-one mapping
all points on the 3D ray map to a single image point.
Therefore, depth information is lost
From similar triangles, can write down the perspective equation:
,','ZY
fy
ZX
fx
==
9CS4243: Geometric Models
Perspective Projection
MPu
ZYX
ff
wvu
yx
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇔⎥
⎦
⎤⎢⎣
⎡
or1
000
1000000
''
This assumes coordinate axes is at the pinhole.
10CS4243: Geometric Models
t
X
Y
Z
fi
fj
fk
World coordinatesystem
Camera coordinatesystem
3D Scene point P
t : World origin w.r.t. cameracoordinate axes
Rotation R
P : 3D position of scene pointw.r.t world coordinate axes
R : Rotation matrix to align world coordinate axes to camera axes
11CS4243: Geometric Models
Perspective Projection
[ ]
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
×=
×=
100sin0cot-
matrixparameterintrinsic33:
matrixprojection43:
0
0
vu
KtRK
MMPu
θβ
θαα
α, β : scaling in image u, v axes, respectivelyΘ : skew angle, that is, angle between u, v axesu0, v0 : origin offsetNote: α=kf, β=lf where k, l are the magnification factors
12CS4243: Geometric Models
Intrinsic vs. Extrinsic parametersIntrinsic: internal camera parameters.
6 of them (5 if you don’t care about focal length)Focal length, horiz. & vert. magnification, horiz. & vert. offset, skew angle
Extrinsic: external parameters6 of them3 rotation angles, 3 translation param
Imposing assumptions will reduce # params.Estimating params is called camera calibration.
13CS4243: Geometric Models
Representing Rotations
Euler angles
pitch: rotation about x axis :
yaw: rotation about y axis:
roll: rotation about z axis:X
Y
Z
ωφ
κ
14CS4243: Geometric Models
Rotation Matrix
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−++−
=
=
)cos()cos()sin()cos()sin()sin()cos()cos()sin()sin()cos()cos()sin()sin()sin()cos()sin()sin()sin()cos()sin()cos()cos()sin()sin()sin()cos(cos)cos(
ωφωφφωκωφκωκωφκφκωκωφκωκωφκφκ
xyz RRRR
Two properties of rotation matrix:
R is orthogonal: RTR = Idet(R) = 1
15CS4243: Geometric Models
Orthographic Projection
Projection rays are parallel
Camera centre
opticalaxis
Image plane
Parallel lines
16CS4243: Geometric Models
Orthographic Projection Equations
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
10
10100
000010001
2
1
y
x
T
T
T
T
tt
rr
tRM
TT rr 21 ,
yx tt ,: first two rows of R
: first two elements of t
This projection has 5 degrees of freedom.
17CS4243: Geometric Models
Image plane Parallel lines
Weak-perspective Projection
18CS4243: Geometric Models
Weak-perspective Projection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
101000000
2
1
y
x
T
T
T
tt
rr
M βα
This projection has 7 degrees of freedom.
19CS4243: Geometric Models
Parallel linesImage plane
Para-perspective Projection
20CS4243: Geometric Models
Para-perspective Projection
Camera matrix given in textbook Table 2.1, page 36.Note: Written a bit differently.This has 9 d.o.f.
21CS4243: Geometric Models
Real cameras
Real cameras use lensesLens distortion causes distortion in image
Lines may project into curvesSee Chap 1.2, 3.3 for details
Change of focal length (zooming) scales the image
not true if assuming orthographic projection)
Color distortions too
22CS4243: Geometric Models
Color Aberration
Bad White Balance
23CS4243: Geometric Models
Color Aberration
Purple Fringing
24CS4243: Geometric Models
Color Aberration
Vignetting
25CS4243: Geometric Models
Geometric Camera Calibration
26CS4243: Geometric Models
Recap
A general projective matrix:
This has 11 d.o.f. , i.e. 11 parameters5 intrinsic, 6 extrinsic
How to estimate all of these?Geometric camera calibration
[ ]PtRKuMPu
==
27CS4243: Geometric Models
Key Idea
Capture image of a known 3D object (calibration object).Establish correspondences between 3D points and their 2D image projections.Estimate MEstimate K, R, and t from MEstimation can be done using linear or non-linear methods.
We will study linear methods first.
28CS4243: Geometric Models
Setting things up
Calibration object:3D object, or2D planar object captured at different locations
29CS4243: Geometric Models
Setting it up
Suppose we have N point correspondences:P1, P2, … PN are 3D scene points u1, u2, … uN are corresponding image points
Let mT1, mT
2, mT3 be the 3 rows of M
Let (ui, vi) be the (non-homogeneous) coords of the i th image point.Let Pi be the homogeneous coords of i th scene point.
30CS4243: Geometric Models
Math
For the i th point
Each point gives 2 equations.
Using all N points gives 2N equations:
0)(
0)(
32
31
=−
=−
iT
i
iT
i
Pmvm
Pmum
31CS4243: Geometric Models
Math
00
0
00
3
2
1111
111
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
mmm
PvPu
PP
PvPPuP
TNN
TNN
TN
T
T
TN
TTT
TTT
A m = 0
A : 2N x 12 matrix, rank is 11m : 12 x 1 column vectorNote that m has only 11 d.o.f.
32CS4243: Geometric Models
Math
Solution?m is in the Nullspace of A !
Use SVD: A = U ∑ VT
m = last column of V m is only up to an unknown scaleIn this case, SVD solution makes || m || = 1
33CS4243: Geometric Models
Estimating K, R, t
Now that we have M (3x4 matrix)
c is the 3D coords of camera center wrt World axesLet ĉ be homogeneous coords of cIt can be shown that M ĉ = 0
So ĉ is in the Nullspace of MAnd t is computed from –Rc, once R is known.
[ ] [ ]Rct
pBtRKM−=
==
34CS4243: Geometric Models
Estimating K, R, t
B, the left 3x3 submatrix of M is KRPerform “RQ” decomposition on B.The “Q” is the required rotation matrix RAnd the upper triangular “R” is our K matrix !
Impose the condition that diagonals of K are positive
We are done!
35CS4243: Geometric Models
RQ factorization
Any n x n matrix A can be factored into A = RQWhere both R, Q are n x nR is upper triangular, Q is orthogonal
Not the same as QR factorizationTrick: post multiply A by Givens rotation matrices: Qx ,Qy, Qz
θθ sin,coswhere10000
,0
0100
,00
001
==
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−=
sc
cssc
Qcs
scQ
csscQ zyx
36CS4243: Geometric Models
RQ factorization
c, s chosen to make a particular entry of A zeroFor example, to make a21 = 0, we solve
Choose Qx ,Qy, Qz such that
222
221
21222
221
22
2221
,
0
aaas
aaac
saca
+=
+
−=⇒
=+
RQQQRQA
RQQAQTx
Ty
Tz
zyx
==⇒
=
ngularupper tria
37CS4243: Geometric Models
Degeneracy
Caution: The 3D scene points should not all lie on a plane.
Otherwise no solution
In practice, choose N >= 6 points in “general position”
Note: the above assumes no lens distortion.See Chap 3.3 for how to deal with lens distortion.
38CS4243: Geometric Models
Summary
Presented pinhole camera modelFrom this we get hierarchy of projection typesPerspective, Para-perspective, Weak-perspective, Orthographic
Showed how to calibrate camera to estimate intrinsic + extrinsic parameters.