Single-view geometry

Odilon Redon, Cyclops, 1914

Our goal: Recovery of 3D structure• Recovery of structure from one image is

inherently ambiguous

X?X?

X?X?

x

Our goal: Recovery of 3D structure• Recovery of structure from one image is

inherently ambiguous

Our goal: Recovery of 3D structure• Recovery of structure from one image is

inherently ambiguous

Ames Room

http://en.wikipedia.org/wiki/Ames_room

Our goal: Recovery of 3D structure• We will need multi-view geometry

Recall: Pinhole camera model

• Principal axis: line from the camera center• Principal axis: line from the camera center perpendicular to the image plane

• Normalized (camera) coordinate system: camera center is at the origin and the principal axis is the z-axis

Recall: Pinhole camera model

)/,/(),,( ZYfZXfZYX a ),(),,( ff

⎟⎞

⎜⎛⎤⎡

⎟⎞

⎜⎛⎟

⎞⎜⎛

0X

fXfX

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

0100

ZY

ff

ZYfXf

ZY

a PXx =⎟⎟⎠

⎜⎜⎝⎥⎦⎢⎣

⎟⎠

⎜⎝⎟⎟

⎠⎜⎜⎝ 1

011

Z

Principal point

py

Principal point (p) point here principal a is intersects the

px

• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)

• Normalized coordinate system: origin is at the principal pointy g p p p• Image coordinate system: origin is in the corner• How to go from normalized coordinate system to image

coordinate system?

Principal point offset

)(principal point: ),( yx pppy

)/,/(),,( pZYfpZXfZYX ++a

px

)/,/(),,( yx pZYfpZXfZYX ++a

⎟⎞

⎜⎛⎤⎡⎞⎛ +⎟

⎞⎜⎛

0X

pfpZXfX

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛++

⎟⎟⎟⎟⎟

⎜⎜⎜⎜⎜

0100

ZY

pfpf

ZpZYfpZXf

ZY

y

x

y

x

a

⎟⎟⎠

⎜⎜⎝⎥⎦⎢⎣

⎟⎠

⎜⎝⎟⎟

⎠⎜⎜⎝ 1

011

Z

Principal point offset

)(principal point: ),( yx pp

⎟⎟⎞

⎜⎜⎛

⎥⎤

⎢⎡⎥⎤

⎢⎡

⎟⎞

⎜⎛ + 01

YX

pfZpXf xx

⎟⎟⎟⎟

⎠⎜⎜⎜⎜

⎝⎥⎥⎥

⎦⎢⎢⎢

⎣⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎟⎟⎟

⎠⎜⎜⎜

⎝

+

10101

1ZY

pfZ

ZpYf yy

⎠⎝ 1

⎥⎥⎤

⎢⎢⎡

=x

pfpf

K calibration matrix [ ]0|IKP =⎥⎥⎦⎢

⎢⎣ 1

ypfK calibration matrix [ ]0|IKP

Pixel coordinates

11Pixel size:

yx mm11

×

mx pixels per meter in horizontal direction, m pixels per meter in vertical direction

⎥⎤

⎢⎡

⎥⎤

⎢⎡⎥⎤

⎢⎡ xxxx pfm βα

my pixels per meter in vertical direction

⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎥⎥⎥

⎦⎢⎢⎢

⎣⎥⎥⎥

⎦⎢⎢⎢

⎣

=111

yyyy pfmK βα

⎦⎣⎦⎣⎦⎣pixels/m m pixels

Camera rotation and translation

• In general, the camera di t f illcoordinate frame will

be related to the world coordinate frame by a yrotation and a translation

( )C~-X~RX~ cam = ( )cam

coords. of point in camera frame

coords. of camera center in world frame

d f i tcoords. of a pointin world frame (nonhomogeneous)

Camera rotation and translation

In non-homogeneouscoordinates:

( )C~-X~RX~ cam =

coordinates:

~~~ ⎤⎡⎞⎛⎤⎡X

10CRR

1X

10CRRXcam ⎥

⎦

⎤⎢⎣

⎡ −=⎟⎟

⎠

⎞⎜⎜⎝

⎛⎥⎦

⎤⎢⎣

⎡ −=

[ ] [ ]XC~R|RKX0|IKx −== [ ],t|RKP = C~Rt −=[ ] [ ]XCR|RKX0|IKx cam [ ],t|RKP CRt

Note: C is the null space of the camera projection matrix (PC=0)

Camera parameters• Intrinsic parameters

• Principal point coordinates• Focal length ⎥

⎤⎢⎡

⎥⎤

⎢⎡⎥⎤

⎢⎡ xxxx pfm βα

• Focal length• Pixel magnification factors• Skew (non-rectangular pixels)

⎥⎥⎥

⎦⎢⎢⎢

⎣

=⎥⎥⎥

⎦⎢⎢⎢

⎣⎥⎥⎥

⎦⎢⎢⎢

⎣

=111

yyyy pfmK βα

• Radial distortion

Camera parameters• Intrinsic parameters

• Principal point coordinates• Focal length• Focal length• Pixel magnification factors• Skew (non-rectangular pixels)• Radial distortion

• Extrinsic parameters• Rotation and translation relative to world coordinate system• Rotation and translation relative to world coordinate system

Camera calibration

[ ]XRK[ ]XtRKx =

⎥⎤

⎢⎡

⎥⎤

⎢⎡

⎥⎤

⎢⎡ ****

YX

xλ

⎥⎥⎥⎥

⎢⎢⎢⎢

⎥⎥⎥

⎢⎢⎢=

⎥⎥⎥

⎢⎢⎢ ****

ZY

yλ

⎥⎦

⎢⎣

⎥⎦⎢⎣⎥⎦⎢⎣ 1****λ

Source: D. Hoiem

Camera calibration• Given n points with known 3D coordinates Xi

and known image projections xi, estimate the tcamera parameters

XXi

xi

?P

Camera calibration: Linear method

ii PXx =λ 0PXx =× ii0XP

XP

2

1

=⎥⎥⎥⎤

⎢⎢⎢⎡

×⎥⎥⎤

⎢⎢⎡

iT

iT

i

i

yx

ii iiXP1 3⎥⎦⎢⎣⎥

⎥⎦⎢

⎢⎣ i

T

PXX0 ⎞⎛⎤⎡ TT

0PP

X0XXX0

2

1

=⎟⎟⎟⎞

⎜⎜⎜⎛

⎥⎥⎥⎤

⎢⎢⎢⎡

−−

Tii

Ti

Tii

Ti

xy

P0XX 3⎟⎠

⎜⎝⎥⎦⎢⎣−

Tii

Tii xy

Two linearly independent equations

Camera calibration: Linear method

PX0XXX0

1

111

⎟⎞

⎜⎛⎥⎥⎤

⎢⎢⎡

−−

TTT

TTT

xy

0pA =0PPP

XX0

X0X

2

1111

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥⎥

⎢⎢⎢⎢

TTT

xLLL

PX0XXX0 3

⎟⎠

⎜⎝⎥⎥⎦⎢

⎢⎣ −

−Tnn

TTn

Tnn

Tn

T

xy

• P has 11 degrees of freedom (12 parameters, but scale is arbitrary)

• One 2D/3D correspondence gives us two linearly independent equationsH l t• Homogeneous least squares

• 6 correspondences needed for a minimal solution

Camera calibration: Linear method

PX0XXX0

1

111

⎟⎞

⎜⎛⎥⎥⎤

⎢⎢⎡

−−

TTT

TTT

xy

0Ap =0PPP

XX0

X0X

2

1111

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥⎥

⎢⎢⎢⎢

TTT

xLLL

PX0XXX0 3

⎟⎠

⎜⎝⎥⎥⎦⎢

⎢⎣ −

−Tnn

TTn

Tnn

Tn

T

xy

• Note: for coplanar points that satisfy ΠTX=0,we will get degenerate solutions (Π 0 0)we will get degenerate solutions (Π,0,0), (0,Π,0), or (0,0,Π)

Camera calibration: Linear method• Advantages: easy to formulate and solve• Disadvantages

• Doesn’t directly tell you camera parameters• Doesn’t model radial distortion• Can’t impose constraints, such as known focal length andCan t impose constraints, such as known focal length and

orthogonality

• Non linear methods are preferred• Non-linear methods are preferred• Define error as difference between projected points and

measured points• Minimize error using Newton’s method or other non-linear

optimization

Source: D. Hoiem

Multi-view geometry problems• Structure: Given projections of the same 3D point in two

or more images, compute the 3D coordinates of that point

?

Camera 3R3,t3 Slide credit:

Noah Snavely

Camera 1Camera 2R1,t1 R2,t2

Multi-view geometry problems• Stereo correspondence: Given a point in one of the

images, where could its corresponding points be in the other images?other images?

Camera 3R3,t3

Camera 1Camera 2R1,t1 R2,t2 Slide credit:

Noah Snavely

Multi-view geometry problems• Motion: Given a set of corresponding points in two or

more images, compute the camera parameters

Camera 1Camera 2 Camera 3

R1,t1 R2,t2 R3,t3? ? ? Slide credit:

Noah Snavely

Triangulation• Given projections of a 3D point in two or more

images (with known camera matrices), find th di t f th i tthe coordinates of the point

X?

x x2x12

O1 O2

Triangulation• We want to intersect the two visual rays

corresponding to x1 and x2, but because of i d i l th d ’t tnoise and numerical errors, they don’t meet

exactlyR1R2

X?2

x x2x12

O1 O2

Triangulation: Geometric approach• Find shortest segment connecting the two

viewing rays and let X be the midpoint of that tsegment

X

x x2x12

O1 O2

Triangulation: Linear approach

XPXPx 111 =

λλ

0XP0XPx 11 =×

0XP][x0XP][x 11 =×

XPx 222 =λ 0XPx 22 =× 0XP][x 22 =×

0 ⎤⎡⎤⎡ − baa

Cross product as matrix multiplication:

baba ][0

00

×=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−=× y

x

xz

yz

bbb

aaaa

0 ⎥⎦⎢⎣⎥⎦⎢⎣− zxy baa

Triangulation: Linear approach

XPXPx 111 =

λλ

0XP0XPx 11 =×

0XP][x0XP][x 11 =×

XPx 222 =λ 0XPx 22 =× 0XP][x 22 =×

Two independent equations each in terms of three unknown entries of X

Triangulation: Nonlinear approachFind X that minimizes

)()( 22 XPxdXPxd + ),(),( 2211 XPxdXPxd +

X?

x x2

x’1x1

2x’2

O1 O2