Theory of Bouguet’s MatLab Camera CalibrationToolbox

Yuji Oyamada

1HVRL, University

2Chair for Computer Aided Medical Procedure (CAMP)Technische Universitat Munchen

June 26, 2012

Introduction Functions Theory

MatLab Camera Calibration Toolbox

• Calibration using a planar calibration object.

• Points correspondence requires manual click on object’scorners.

• Core calibration part is fully automatic.

• C implementation is included in OpenCV.

• Web

Introduction Functions Theory

Functions

• Extract grid corners: Points correspondence

• Calibration: Core calibration

• Show Extrinsic: Visualization of estimated extrinsicparameters

• Reproject on images: Visualization of reprojection error

• Analyse error:

• Recomp. corners: Recompute the corners using estimatedparameters

Introduction Functions Theory

Extract grid cornersInput:

• n sq {x,y}: Number of square along {x,y} axis

• d{X,Y}: Size of square in m unit.

• I: Input image

Output:

• {x i}, i = 1, . . . ,N: Set of detected 2D points.

• {X i}, i = 1, . . . ,N: Set of known 3D points.

where x i and X i have Mi elements as

x i =

[x i(1) · · · x i(j) · · · x i(Mi )y i(1) · · · y i(j) · · · y i(Mi )

]and

X i =

X i(1) · · · X i(j) · · · X i(Mi )Y i(1) · · · Y i(j) · · · Y i(Mi )Z i(1) · · · Z i(j) · · · Z i(Mi )

Introduction Functions Theory

Calibration

Input:

• {x i}: Set of detected 2D points.

• {X i}: Set of known 3D points.

Output:

• Intrinsic parameters

• Extrinsic parameters

Method:

1. Initialize intrinsic and extrinsic params.

2. Non-linear optimization to refine all parameters.

Introduction Functions Theory

Camera parameters

Intrinsic parameters:

• fc ∈ R2: Camera focal length

• cc ∈ R2: Principal point coordinates

• alpha c ∈ R1: Skew coefficient

• kc ∈ R5: Distortion coefficients

• KK ∈ R3×3: The camera matrix (containing fc, cc, alpha c)

Extrinsic parameters for i-th image:

• omc i ∈ R3: Rotation angles

• Tc i ∈ R3: Translation vectors

• Rc i ∈ R3×3: Rotation matrix computed asRc i = rodrigues(omc i)

Introduction Functions Theory

Initialization 1/2: Intrinsic parameters

Input

• {x i}: Set of detected 2D points.

• {X i}: Set of known 3D points.

Output

• Initialized intrinsic parameters

• Initialized extrinsic parameters

Introduction Functions Theory

Initialization 1/2: Intrinsic parameters

1. Roughly compute intrinsic params.

2. Refine intrinsic params.

3. Compute extrinsic params.

Step1:

• fc : Computed based on all the vanishing points.

• cc : Center of image.

• alpha c = 0.

• kc = 0.

Introduction Functions Theory

Initialization 1/2: Intrinsic parametersStep2: Given N sets of corresponding points {xk ,Xk},

1. Compute Homography {Hk}.2. Build linear equations Vb = 0 and solve the equations Detail .3. Compute A from estimated b.

V ≡

v>1,12

(v1,11 − v1,22)>

· · ·v>k,12

(vk,11 − vk,22)>

· · ·v>N,12

(vN,11 − vN,22)>

vk,ij = [hk,i1hk,j1, hk,i1hk,j2 + hk,i2hk,j1, hk,i2hk,j2,

hk,i3hk,j1 + hk,i1hk,j3, hk,i3hk,j2 + hk,i2hk,j3, hk,i3hk,j3]

Introduction Functions Theory

Initialization 2/2: Extrinsic parameters

Step3: Compute each extrinsic parameters from estimated A andpre-computed Hk as

rk,1 = λkA−1hk,1

rk,2 = λkA−1hk,2

rk,3 = rk,1 × rk,2

tk = λkA−1hk,3

where

λk =1

||A−1hk,1||=

1

||A−1hk,2||

Introduction Functions Theory

Non-linear optimization

Solves the following equations:

p = argminp‖x− f (p)‖2

2

where p = {fc, cc , alpha c , kc , {omc i}, {Tc i}} and the functionf projects the known 3D points X onto 2D image plane with theargument p.Sparse Levenberg-Marquardt algorithm iteratively updates p withan initial estimate p0 as

while(∥∥x− f (pi−1)

∥∥2

2> ε)

pi = pi−1 + ∆pi ;

i = i + 1;

Introduction Functions Theory

Image projectionGiven

• xp = (xp, yp, 1)>: Pixel coordinate of a point on the imageplane.

• xd = (xd , yd , 1)>: Normalized coordinate distorted by lens.

• xn = (xn, yn, 1)>: Normalized pinhole image projection.

• k ∈ R5: Lens distortion parameters.

• KK ∈ R3×3: Camera calibration matrix.

where

KK =

fc(1) alpha c(1) cc(1)0 fc(2) cc(2)0 0 1

xp = KKxd = KK(cdistxn + delta x)

Introduction Functions Theory

Lens distortion

Given

• xn = x: Normalized pinhole image projection.

• k ∈ R5: Lens distortion parameters.

Compute

• xd = xd: Normalized coordinate distorted by lens.

• cdist: Radial distortion

• delta x: Tangential distortion

xd = cdist ∗ x + delta x

=

[cdist ∗ x + delta x(1)cdist ∗ y + delta x(2)

]

Introduction Functions Theory

Radial distortion

xd1 =

[x ∗ cdisty ∗ cdist

]cdist = 1 + k(1) ∗ r2 + k(2) ∗ r4 + k(5) ∗ r6

where

r2 = x2i + y2

i ,

r4 = r22,

r6 = r23

Introduction Functions Theory

Tangential distortion

delta x =

[k(3) ∗ a1 + k(4) ∗ a2k(3) ∗ a3 + k(4) ∗ a1

]where

a1 = 2xiyi

a2 = r2 + 2x2i

a3 = r2 + 2y2i

Introduction Functions Theory

Closed form solution for initialization

s

uv1

= A[r1 r2 r3 t

] XY01

= A[r1 r2 t

] XY1

= H

XY1

Since r1 and r2 are orthonormal, we have following two constraints:

h>1 A−>A−1h2 = 0

h>1 A−>A−1h1 = h>2 A

−>A−1h2,

where A−>A−1 describes the image of the absolute conic.

Introduction Functions Theory

Closed form solution for initialization

Let

B = A−>A−1 ≡

B11 B21 B31

B12 B22 B32

B13 B23 B33

=

1α2 − γ

α2βv0γ−u0βα2β

− γα2β

γ2

α2β2 + 1β2 −γ(v0γ−u0β)

α2β2 − v0β2

v0γ−u0βα2β

−γ(v0γ−u0β)α2β2 − v0

β2(v0γ−u0β)2

α2β2 +v2

0β2 + 1

Note that B is symmetric, defined by a 6D vector

b ≡[B11,B12,B22,B13,B23,B3

]>

Introduction Functions Theory

Closed form solution for initialization

Let the i-th column vector of H be hi =[hi1, hi2, hi3

]>.

Then, we have following linear equation

h>i Bhj = v>ij b

where

vij = [hi1hj1, hi1hj2 + hi2hj1, hi2hj2

hi3hj1 + hi1hj3, hi3hj2 + hi2hj3, hi3hj3]

Introduction Functions Theory

Closed form solution for initialization

The above two constraints can be rewritten as two homogeneousequation w.r.t. unknown b as[

v>12

(v11 − v22)>

]b = 0

Given n images, we have Vb = 0, where V ∈ R2n×6.

Introduction Functions Theory

Least Squares

• Let f (·) a function projecting 3D points X onto 2D points xwith projection parameters p as bmx = f (p) = pX.

• Our task is to find an optimal p such that minimizes‖x− pX‖2

2.

In the case of non-linear function f (·), we iteratively update theestimate p with an initial estimate p0 as

while(‖x− pX‖22 > ε)

pi = pi−1 + ∆pi ;

i = i + 1;

Introduction Functions Theory

Least Squares: Newton’s method

Given a previous estimate pi , estimate an update ∆i s.t.

∆i = argmin∆‖f (pi + ∆)− X‖2

2

= argmin∆‖f (pi ) + J∆− X‖2

2

= argmin∆‖εi + J∆‖2

2

where J = ∂f∂p .

Then, solve the normal equations

J>J∆ = −J>εi

Introduction Functions Theory

Levenberg-Marquardt (LM) iteration

The normal equation is replaced by the augmented normalequations as

J>J∆ = −J>εi→ (J>J + λI)∆ = −J>εi

where I denotes the identity matrix.An initial value of λ is 10−3 times the average of the diagonalelements of N = J>J.

Introduction Functions Theory

Sparse LM

• LM is suitable for minimization w.r.t. a small number ofparameters.

• The central step of LM, solving the normal equations,• has complexity N3 in the number of parameters and• is repeated many times.

• The normal equation matrix has a certain sparse blockstructure.

Introduction Functions Theory

Sparse LM

• Let p ∈ RM be the parameter vector that is able to bepartitioned into parameter vectors as p = (a>,b>)>.

• Given a measurement vector x ∈ RN

• Let∑

x be the covariance matrix for the measurement vector.

• A general function f : RM → RN takes p to the estimatedmeasurement vector x = f (p).

• ε denotes the difference x− x between the measured and theestimated vectors.

Introduction Functions Theory

Sparse LM

The set of equations Jδ = ε solved as the central step in the LMhas the form

Jδ = [A|B]

(δaδb

)= ε.

Then, the normal equations J>∑∑∑−1

x Jδ = J>∑∑∑−1

x ε to be solvedat each step of LM are of the form[

A>∑∑∑−1

x A A>∑∑∑−1

x B

B>∑∑∑−1

x A B>∑∑∑−1

x B

](δaδb

)=

(A>∑∑∑−1

x ε

B>∑∑∑−1

x ε

)

Introduction Functions Theory

Sparse LM

Let

• U = A>∑∑∑−1

x A

• W = A>∑∑∑−1

x B

• V = B>∑∑∑−1

x B

and ·∗ denotes augmented matrix by λ.The normal equations are rewritten as[

U∗ WW> V∗

](δaδb

)=

(εAεB

)→[U∗ −WV∗−1W> 0

W> V∗

](δaδb

)=

(εA −WV∗−1εB

εB

)This results in the elimination of the top right hand block.

Introduction Functions Theory

Sparse LM

The top half of this set of equations is

(U∗ −WV∗−1W>)δa = εA −WV∗−1εB

Subsequently, the value of δa may be found by back-substitution,giving

V∗δb = εB −W>δa

Introduction Functions Theory

Sparse LM

p = (a>,b>)>, where a = (fc>, cc>, alpha c>, kc>)> andb = ({omc i> Tc i>})>The Jacobian matrix is

J =∂x

∂p=

[∂x

∂a,∂x

∂b

]= [A,B]

where

∂x

∂a=

[∂x

∂fc,∂x

∂cc,

∂x

∂alpha c,∂x

∂kc

]∂x

∂b=

[∂x

∂omc 1,

∂x

∂Tc 1, · · · , ∂x

∂omc i

∂x

∂Tc i· · · , ∂x

∂omc N

∂x

∂Tc N

]

Introduction Functions Theory

Sparse LM

The normal equation is rewritten as([A>

B>

] [A B

]+ λI

)∆p = −

[A>

B>

]εx

→

N∑i=1

A>i Ai

N∑i=1

A>i Bi

N∑i=1

B>i Ai

N∑i=1

B>i Bi

+ λI

∆p = −[A>εxB>εx

]

Introduction Functions Theory

Sparse LM

J>J =

N∑i=1

A>i Ai A>1 B1 · · · A>i Bi · · · A>NBN

B>1 A1 B>1 B1...

. . .

B>i Ai B>i Bi...

. . .

B>NAN B>NBN

Introduction Functions Theory

Sparse LM

J>εx =

N∑i=1

A>i εx

B>1 εx...

B>i εx...

B>Nεx

Introduction Functions Theory

Sparse LM

When each image has different number of corresponding points(Mi 6= Mj , if i 6= j), each Ai and Bi have different size as

Introduction Functions Theory

Sparse LM

However, the difference does not matter because

A>A ∈ Rdint×dint

A>B ∈ Rdint×dex

B>A ∈ Rdex×dint

B>B ∈ Rdex×dex

where dint denotes dimension of intrinsic params and dex denotesdimension of extrinsic params.