Structure from motion

• Multi-view geometry• Affine structure from motion• Projective structure from motion

Planches :– http://www.di.ens.fr/~ponce/geomvis/lect4.ppt – http://www.di.ens.fr/~ponce/geomvis/lect4.pdf

Epipolar Constraint

• Potential matches for p have to lie on the corresponding epipolar line l’.

• Potential matches for p’ have to lie on the corresponding epipolar line l.

Epipolar Constraint: Calibrated Case

Essential Matrix(Longuet-Higgins, 1981)

Properties of the Essential Matrix

• E p’ is the epipolar line associated with p’.

• E p is the epipolar line associated with p.

• E e’=0 and E e=0.

• E is singular.

• E has two equal non-zero singular values (Huang and Faugeras, 1989).

T

T

Epipolar Constraint: Small MotionsTo First-Order:

Pure translation:Focus of Expansion

Epipolar Constraint: Uncalibrated Case

Fundamental Matrix(Faugeras and Luong, 1992)

Properties of the Fundamental Matrix

• F p’ is the epipolar line associated with p’.

• F p is the epipolar line associated with p.

• F e’=0 and F e=0.

• F is singular.

T

T

The Eight-Point Algorithm (Longuet-Higgins, 1981)

|F | =1.

Minimize:

under the constraint2

Non-Linear Least-Squares Approach (Luong et al., 1993)

Minimize

with respect to the coefficients of F , using an appropriate rank-2 parameterization.

The Normalized Eight-Point Algorithm (Hartley, 1995)

• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’.

• Use the eight-point algorithm to compute F from thepoints q and q’ .

• Enforce the rank-2 constraint.

• Output T F T’.T

i i i i

i i

Data courtesy of R. Mohr and B. Boufama.

With

out n

orm

aliz

atio

nW

ith n

orm

aliz

atio

nMean errors:10.0pixel9.1pixel

Mean errors:1.0pixel0.9pixel

Trinocular Epipolar Constraints

These constraints are not independent!

Trinocular Epipolar Constraints: Transfer

Given p and p , p can be computed

as the solution of linear equations.

1 2 3

Trifocal Constraints

Trifocal Constraints

All 3x3 minorsmust be zero!

Calibrated Case

Trifocal Tensor

Trifocal ConstraintsUncalibrated Case

Trifocal Tensor

Trifocal Constraints: 3 Points

Pick any two lines l and l through p and p .Do it again.

2 3 2 3T( p , p , p )=01 2 3

Properties of the Trifocal Tensor

Estimating the Trifocal Tensor

• Ignore the non-linear constraints and use linear least-squaresa posteriori.

• Impose the constraints a posteriori.

• For any matching epipolar lines, l G l = 0.

• The matrices G are singular.

• They satisfy 8 independent constraints in theuncalibrated case (Faugeras and Mourrain, 1995).

2 1 3T i

1i

For any matching epipolar lines, l G l = 0. 2 1 3T i

The backprojections of the two lines do not define a line!

Multiple Views (Faugeras and Mourrain, 1995)

Two Views

Epipolar Constraint

Three Views

Trifocal Constraint

Four Views

Quadrifocal Constraint(Triggs, 1995)

Geometrically, the four rays must intersect in P..

Quadrifocal Tensorand Lines

Scale-Restraint Condition from Photogrammetry

The Euclidean (perspective) Structure-from-Motion Problem

Given m calibrated perspective images of n fixed points Pj we can write

Problem: estimate the m 3x4 matrices Mi = [Ri ti] and

the n positions Pj from the mn correspondences pij .

2mn equations in 11m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

The Euclidean Ambiguity of Euclidean SFM

If Ri, ti, and Pj are solutions,

So are Ri’, ti’, and Pj’, where

In fact, the absolute scale cannot be recovered since:

When the intrinsic and extrinsic parameters are known

Euclidean ambiguity up to a similarity transformation.

The Affine Structure-from-Motion Problem

Given m images of n fixed points P we can write

Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .

ij ij

2mn equations in 8m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

The Affine Ambiguity of Affine SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and

Q is an affinetransformation.

When the intrinsic and extrinsic parameters are unknown

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

Affine Epipolar Geometry

The Affine Fundamental Matrix

where

With

out n

orm

aliz

atio

nW

ith n

orm

aliz

atio

nMean errors:10.0pixel9.1pixel

Mean errors:1.0pixel0.9pixel

Perspective case..

Mean errors: 3.24 and 3.15pixel (without normalization160.92 and 158.54pixel).

Affine case..

An Affine Trick..

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

An Affine Trick.. Algebraic Scene Reconstruction Method

Affine reconstruction. Mean relative error: 3.2%

The Affine Structure of Affine Images

Suppose we observe a static scene with m fixed cameras..

The set m-tuples of allimage points in a sceneis a 3D affine space!

When do m+1 points define a p-dimensional subspace Y of ann-dimensional affine space X equipped with some coordinateframe basis?

Writing that all minors of size (p+2)x(p+2) of D are equal tozero gives the equations of Y.

Rank ( D ) = p+1, where

has rank 4!

From Affine to Vectorial Structure

Idea: pick one of the points (or their center of mass)as the origin.

Singular Value Decomposition

Singular Value Decomposition square roots of

Singular Value Decomposition

Singular Value Decomposition

What if we could factorize D? (Tomasi and Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

Affine reconstruction. Mean relative error: 2.8%

Back to perspective:Euclidean motion from E (Longuet-Higgins, 1981)

• Given F computed from n > 7 point correspondences, and its SVD F= UWVT, compute E=U diag(1,1,0) VT.

• There are two solutions t’ = u3 and t’’ = -t’ to ETt=0.

• Define R’ = UWVT and R” = UWTVT where

(It is easy to check R’ and R” are rotations.)

• Then [tx’]R’ = -E and [tx’]R” = E. Similar reasoning for t”.

• Four solutions. Only two of them place the reconstructedpoints in front of the cameras.

100001010

W

Euclidean reconstruction. Mean relative error: 3.1%

A different view of the fundamental matrix

• Projective ambiguity ! M’Q=[Id 0] MQ=[A b].

• Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1)T.

• This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b.

• Or: z (b x p) = z’ (b x Ap’).

• Finally: pTFp’ = 0 with F = [bx] A.

Projective motion from the fundamental matrix

• Given F computed from n > 7 point correspondences, compute b as the solution of FTb=0 with |b|2=1.

• Note that: [ax]2 = aaT - |a|2Id for any a.

• Thus, if A0 = - [bx] F,

[bx] A0 = - [bx]2 F = - bbTF + |b|2 F = F.

• The general solution is M = [A b] with

A = A0 + ( b | b | b).

Two-view projective reconstruction. Mean relative error: 3.0%

Bundle adjustment

Use nonlinear least-squares to minimize:

Bundle adjustment. Mean relative error: 0.2%

Projective SFM from multiple images

z11p11 … z1np1n

… … …zm1pm1 … zmnpmn

M1

…Mm

P_1 … P_n= , D = MP

• If the zij’s are known, can be done via SVD. In principlethe zij’s can be found pairwise from F (Triggs 96).

• Alternative, eliminate zij from the minimization of E=|D-MP|2

• This reduces the problem to the minimization ofE = ij |pij x MiPj|2

under the constraints |Mi|2=|Pj|2=1 with |pij|2=1.

• Bilinear problem.

Bilinear projective reconstruction. Mean relative error: 0.2%

From uncalibrated to calibrated cameras

Weak-perspective camera:

Calibrated camera:

Problem: what is Q ?

Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

What is some parameters are known?

Weak-perspective camera:

Zero skew:

Problem: what is Q ?

0

Self calibration!

П1

Chasles’ absolute conic: x12+x2

2+x32 = 0, x4 = 0.

Kruppa (1913); Maybank & Faugeras (1992)

Triggs (1997);Pollefeys et al. (1998,2002)

, u0, v0

The absolute quadric u0 = v0 = 0The absolute quadratic complex 2 = 2, = 0

u0

v0

kl

f

x’ ≈ P ( H H-1 ) xH = [ X y ]

Relation between K, , and *