Camera calibration techniquewprowadzenie teoretyczne

Krzysztof WegnerChair of Multimedia Telecommunications and Microelectronics

Poznań University of Technology, Poland

1

Goal of the calibration Knowledge about

Intrinsic camera parameters Focal length Optical center

Extrinsic camera parameters -Position of the camera in 3D world

Orientation of the camera Translation

2

Goal of the calibration Knowledge about

Intrinsic camera parameters Focal length Optical center

Extrinsic camera parameters -Position of the camera in 3D world

Orientation of the camera Translation Common word coordinate system

3

Camera parameters Intrinsic camera parameters

Focal length Optical center

Extrinsic camera parameters -Position of the camera in 3D world

Orientation of the camera Translation

4

𝑨=[ 𝑓 𝑢 𝛾 𝑜𝑢

0 𝑓 𝑣 𝑜𝑣

0 0 1 ]

𝑹=[𝒓𝟏 𝒓𝟐 𝒓𝟑 ]𝑻=[𝑡𝑥𝑡𝑦𝑡 𝑧 ]

Camera model Projection of a 3D point Onto a point at image plane

s is a scale – distance to the point

5

𝑠 ∙[𝑢𝑣1 ]=𝐴 ∙ [𝑅 −𝑅 ∙𝑇 ] ∙ [𝑋𝑌𝑍1 ]

Zhang’s Algorithm Allows estimation of

intrinsic parameters - matrix extrinsic parameters – rotation matrix and

translation vector Planar template

6Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝑠 ∙[𝑢𝑣1 ]=𝐴 ∙ [𝑟 1 𝑟2 𝑟 3~𝑡 ] ∙ [ 𝑋𝑌𝑍1 ]

Zhang’s algorithm Planar template

7Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Zhang’s algorithm Planar template

8Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Estimating Homography H We know position of the pattern’s feature points From registrated image we know

So

9Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Estimating Homography H We know position of the pattern’s feature points From registrated image we know

10Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Estimating Homography H

Let’s assign

We have 2 equations and 9 variables so we need at least 5 points to solve uniquely for

11Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Estimating Homography H Defined up to a scale factor We know position of the pattern’s feature points From registrated image we know

Multiplication of both side by don’t change known

So we don’t know whether we obtain or So we don’t know scale of the scene

12Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝜆 ∙𝑠 ∙ [𝑢𝑣1 ]=𝝀 ∙𝑯 ∙[ 𝑋𝑌1 ] 𝜆 ∙𝑠 ∙[𝑢𝑣1 ]=~𝑯 ∙[ 𝑋𝑌1 ]

Homography H Defined up to a scale factor We know position of the pattern’s feature points From registrated image we know

13Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

Homography H Defined up to a scale factor We know position of the pattern’s feature points From registrated image we know

Multiplication of both side by don’t change known

14Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝜆 ∙𝑠 ∙ [𝑢𝑣1 ]=𝝀 ∙𝑯 ∙[ 𝑋𝑌1 ]𝜆 ∙𝑠 ∙ [𝑢𝑣1 ]=~𝑯 ∙[ 𝑋𝑌1 ]

Homography H Defined up to a scale factor

Multiplication of both side by don’t change known

So we don’t know whether we obtain or So we don’t know Z scale of the scene

15Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝜆 ∙𝑠 ∙ [𝑢𝑣1 ]=𝝀 ∙𝑯 ∙[ 𝑋𝑌1 ]𝜆 ∙𝑠 ∙ [𝑢𝑣1 ]=~𝑯 ∙[ 𝑋𝑌1 ]

Constraints on the intrinsic Vectors , are orthonormal so taking dot product

gives

and length of , should be the same

For

we have

and

16Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒓𝟏𝑻 ∙𝒓𝟐=𝟎

‖𝒓 𝟏‖𝟐=‖𝒓𝟐‖

𝟐⟹𝒓𝟏𝑻 ∙𝒓 𝟏=𝒓𝟐

𝑻 ∙𝒓𝟐

Constraints on the intrinsic Puting all together

17Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒓𝟏𝑻 ∙𝒓𝟐=𝟎

𝒓𝟏𝑻 ∙𝒓𝟏=𝒓 𝟐

𝑻 ∙𝒓𝟐

(𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟏 )𝑻 ∙𝜆−1 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙𝜆−1∙ 𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎

(𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟏 )𝑻 ∙ 𝜆−1 ∙ 𝑨−𝟏 ∙𝒉𝟏=(𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐 )𝑻 ∙𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟐

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝜆−1 ∙𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐

𝑻 ∙𝑨−𝑻 ∙𝜆− 1 ∙𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐

𝑻 ∙ 𝑨−𝑻 ∙𝑨−𝟏 ∙𝒉𝟐

Constraints on the intrinsic Puting all together

18Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒓𝟏𝑻 ∙𝒓𝟐=𝟎

𝒓𝟏𝑻 ∙𝒓𝟏=𝒓 𝟐

𝑻 ∙𝒓𝟐

(𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟏 )𝑻 ∙𝜆−1 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙𝜆−1∙ 𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎

(𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟏 )𝑻 ∙ 𝜆−1 ∙ 𝑨−𝟏 ∙𝒉𝟏=(𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐 )𝑻 ∙𝜆−1 ∙𝑨−𝟏 ∙𝒉𝟐

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝜆−1 ∙𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐

𝑻 ∙𝑨−𝑻 ∙𝜆− 1 ∙𝜆− 1 ∙ 𝑨−𝟏 ∙𝒉𝟐

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐

𝑻 ∙ 𝑨−𝑻 ∙𝑨−𝟏 ∙𝒉𝟐

Closed form solution Try to solve

Intrinsic matrix

Let’s assign

19Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝑨=[ 𝑓 𝑢 𝛾 𝑜𝑢

0 𝑓 𝑣 𝑜𝑣

0 0 1 ]𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐𝑻 ∙ 𝑨−𝑻 ∙𝑨−𝟏 ∙𝒉𝟐

𝑩=𝑨−𝑻 ∙ 𝑨−𝟏

𝑩=[1𝑓 𝑢2

−𝛾𝑓 𝑢2 𝑓 𝑣

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝛾𝑓 𝑢2 𝑓 𝑣

1𝑓 𝑣

2+𝛾2

𝑓 𝑢2 𝑓 𝑣

2

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢)𝑓 𝑢2 𝑓 𝑣

2

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )𝑓 𝑢2 𝑓 𝑣

2 1+𝑜𝑣

2

𝑓 𝑣2 +

(𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )2

𝑓 𝑢2 𝑓 𝑣

2]=[𝐵11 𝐵12 𝐵13

𝐵12 𝐵22 𝐵23

𝐵13 𝐵23 𝐵33]

Closed form solution Let’s assign

is symetrical Let’s assign

20Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝑩=𝑨−𝑻 ∙ 𝑨−𝟏

𝑩=[1𝑓 𝑢2

−𝛾𝑓 𝑢2 𝑓 𝑣

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝛾𝑓 𝑢2 𝑓 𝑣

1𝑓 𝑣

2+𝛾2

𝑓 𝑢2 𝑓 𝑣

2

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢)𝑓 𝑢2 𝑓 𝑣

2

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )𝑓 𝑢2 𝑓 𝑣

2 1+𝑜𝑣

2

𝑓 𝑣2 +

(𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )2

𝑓 𝑢2 𝑓 𝑣

2]=[𝐵11 𝐵12 𝐵13

𝐵12 𝐵22 𝐵23

𝐵13 𝐵23 𝐵33]

𝒃=[𝐵11 𝐵12 𝐵13 𝐵22 𝐵23 𝐵33 ]

Closed form solution We try to solve

Let’s see pattern in the equations

21Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟐=𝟎

𝒉𝟏𝑻 ∙𝑨−𝑻 ∙ 𝑨−𝟏 ∙𝒉𝟏=𝒉𝟐

𝑻 ∙ 𝑨−𝑻 ∙𝑨−𝟏 ∙𝒉𝟐

𝑩=𝑨−𝑻 ∙ 𝑨−𝟏

𝒉𝟏𝑻 ∙𝑩 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑩 ∙𝒉𝟏=𝒉𝟐𝑻 ∙𝑩 ∙𝒉𝟐

𝒉𝒊𝑻 ∙𝑩 ∙𝒉 𝒋=𝒗𝒊𝒋

𝑻 ∙𝒃=

¿𝐵11 ∙ h𝑖1 ∙ h 𝑗 1+𝐵12 ∙ h𝑖1 ∙ h 𝑗2+𝐵13 ∙ h𝑖1 ∙ h 𝑗3+¿

¿ 𝒗𝒊𝒋𝑻 ∙𝒃

+ ++

𝒗 𝒊𝒋𝑻=[h𝑖 1 ∙ h 𝑗 1 h𝑖1 ∙ h 𝑗 2+h𝑖2 ∙ h 𝑗1 h𝑖1 ∙h 𝑗3+h𝑖3 ∙ h 𝑗 1 h𝑖2 ∙ h 𝑗 2 h𝑖 2 ∙ h 𝑗 3+h𝑖 3 ∙ h 𝑗2 h𝑖3 ∙ h 𝑗3 ]

Closed form solution We try to solve

so

22Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒉𝟏𝑻 ∙𝑩 ∙𝒉𝟐=𝟎𝒉𝟏

𝑻 ∙𝑩 ∙𝒉𝟏−𝒉𝟐𝑻 ∙𝑩 ∙𝒉𝟐=𝟎

𝒉𝒊𝑻 ∙𝑩 ∙𝒉 𝒋=𝒗𝒊𝒋

𝑻 ∙𝒃

𝒗𝟏𝟐𝑻 ∙𝒃=𝟎𝒗𝟏𝟏𝑻 ∙𝒃−𝒗𝟐𝟐

𝑻 ∙𝒃=𝟎𝒗𝟏𝟐𝑻 ∙𝒃=𝟎

(𝒗𝟏𝟏𝑻 −𝒗𝟐𝟐

𝑻 ) ∙𝒃=𝟎

[ 𝒗𝟏𝟐𝑻

(𝒗𝟏𝟏−𝒗𝟐𝟐 )𝑻 ] ∙𝒃=𝟎

Solving for b

Two equation are defined but have 6 unknowns. So at least 3 images are required to uniquly solve for Because all images are captured with the same

camera we can stack equation for together

23Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

[ 𝒗𝟏𝟐𝑻

(𝒗𝟏𝟏−𝒗𝟐𝟐 )𝑻 ] ∙𝒃=𝟎

[ 𝒗𝟏𝟐′𝑻

(𝒗 ′𝟏𝟏−𝒗 ′𝟐𝟐)𝑻 ] ∙𝒃=𝟎

[ 𝒗 ′ ′𝟏𝟐𝑻

(𝒗 ′ ′𝟏𝟏−𝒗 ′ ′𝟐𝟐 )𝑻 ]∙𝒃=𝟎

[ 𝒗𝟏𝟐𝑻

(𝒗𝟏𝟏−𝒗𝟐𝟐 )𝑻 ] ∙𝒃=𝟎

[𝒗𝟏𝟐𝑻

(𝒗𝟏𝟏−𝒗𝟐𝟐 )𝑻

𝒗𝟏𝟐′𝑻

(𝒗 ′𝟏𝟏−𝒗 ′𝟐𝟐 )𝑻

𝒗 ′ ′𝟏𝟐𝑻

(𝒗 ′ ′𝟏𝟏−𝒗 ′ ′𝟐𝟐)𝑻] ∙𝒃=𝟎𝑽=[

𝒗𝟏𝟐𝑻

(𝒗𝟏𝟏−𝒗𝟐𝟐 )𝑻

𝒗𝟏𝟐′𝑻

(𝒗 ′𝟏𝟏−𝒗 ′𝟐𝟐)𝑻

𝒗 ′ ′𝟏𝟐𝑻

(𝒗 ′ ′𝟏𝟏−𝒗 ′ ′𝟐𝟐 )𝑻]

Solving for b

There is trivia solution

But we look for non trivial solution so Such solution is given by eigenvector of

asociated with the smallest eigenvalue (right singulart vector of

24Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝑽 ∙𝒃=𝟎

𝑽 𝑻 ∙𝑽

Retriving intrinsic parameters Once we have We can calculate intrinsic parameters from

25Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝑩=[1𝑓 𝑢2

−𝛾𝑓 𝑢2 𝑓 𝑣

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝛾𝑓 𝑢2 𝑓 𝑣

1𝑓 𝑣

2+𝛾2

𝑓 𝑢2 𝑓 𝑣

2

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢)𝑓 𝑢2 𝑓 𝑣

2

𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢

𝑓 𝑢2 𝑓 𝑣

−𝑜𝑣❑

𝑓 𝑣2 −

𝛾 (𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )𝑓 𝑢2 𝑓 𝑣

2 1+𝑜𝑣

2

𝑓 𝑣2 +

(𝑜𝑣𝛾− 𝑓 𝑣𝑜𝑢 )2

𝑓 𝑢2 𝑓 𝑣

2]=[𝐵11 𝐵12 𝐵13

𝐵12 𝐵22 𝐵23

𝐵13 𝐵23 𝐵33]

𝒃=[𝐵11 𝐵12 𝐵13 𝐵22 𝐵23 𝐵33 ]

𝑜𝑣=𝐵12 ∙𝐵13−𝐵11 ∙𝐵23

𝐵11 ∙𝐵22−𝐵122𝑓 𝑢=√ 𝜆

𝐵11

𝜆=𝐵33−𝐵132 +𝑜𝑣 (𝐵12 ∙𝐵13−𝐵11 ∙𝐵23 )

𝐵11

𝑓 𝑣=√ 𝜆 ∙𝐵11

𝐵11 ∙𝐵22−𝐵122

𝛾=−𝐵12 ∙ 𝑓 𝑢2 ∙ 𝑓 𝑣

𝜆𝑜𝑢=

𝛾 ∙𝑜𝑣

𝑓 𝑣−𝐵13

𝑓 𝑢2

𝜆

Retriving position We know constraints

To complete rotation matrix we calculater third column

Finally we ortonormalize rotation matrix

26Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330–1334, 2000

𝒓𝟑=𝒓𝟏×𝒓 𝟐

‖𝒓 𝟏‖=‖𝒓𝟐‖=‖𝒓𝟑‖=𝟏

Summary Camera parameters requires at least 5

point pattern Intrinsic camera parameters estimation

requires at least 3 images at different orientation

All parameters are defined up to a unknown scale

M37232, October 2015, Geneve 27