Computing FClass 8

Read notes Section 4.2

C1

C2

l2

l1

e1

e20m m 1

T2 F

Fundamental matrix (3x3 rank 2

matrix)

1. Computable from corresponding points

2. Simplifies matching3. Allows to detect wrong

matches4. Related to calibration

Underlying structure in set of matches for rigid scenes

l2

C1m1

L1

m2

L2

M

C2

m1

m2

C1

C2

l2

l1

e1

e2

m1

L1

m2

L2

M

l2lT1

Epipolar geometry

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

Other entities besides points?

Lines give no constraint for two view geometry(but will for three and more views)

Curves and surfaces yield some constraints related to tangency

(e.g. Sinha et al. CVPR’04)

The projective reconstruction theorem

If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent

allows reconstruction from pair of uncalibrated images!

Today’s menu

Computation of F• Linear (8-point)• Minimal (7-point)• Robust (RANSAC)• Non-linear refinement (MLE, …)

• Practical approach

Epipolar geometry: basic equation

0Fxx'T

separate known from unknown

0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx

0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx

(data) (unknowns)(linear)

0Af

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

the singularity constraint

0Fe'T 0Fe 0detF 2Frank

T333

T222

T111

T

3

2

1

VσUVσUVσUVσ

σσ

UF

SVD from linearly computed F matrix (rank 3)

T222

T111

T2

1

VσUVσUV0

σσ

UF'

FF'-FminCompute closest rank-2 approximation

the minimum case – 7 point correspondences

0f1''''''

1''''''

777777777777

111111111111

yxyyyxyxyxxx

yxyyyxyxyxxx

T9x9717x7 V0,0,σ,...,σdiagUA

9x298 0]VA[V T8

T ] 000000010[Ve.g.V

1...70,)xλFF(x 21T iii

one parameter family of solutions

but F1+F2 not automatically rank 2

F1 F2

F

3

F7pts

0λλλ)λFFdet( 012

23

321 aaaa

(obtain 1 or 3 solutions)

(cubic equation)

0)λIFFdet(Fdet)λFFdet( 1-12221

the minimum case – impose rank 2

Compute possible as eigenvalues of (only real solutions are potential solutions)

1-12 FF

B.detAdetABdet

0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100

!Orders of magnitude differencebetween column of data matrix least-squares yields poor results

the NOT normalized 8-point algorithm

Transform image to ~[-1,1]x[-1,1]

(0,0)

(700,500)

(700,0)

(0,500)

(1,-1)

(0,0)

(1,1)(-1,1)

(-1,-1)

1

1500

2

10700

2

normalized least squares yields good results (Hartley, PAMI´97)

the normalized 8-point algorithm

Robust estimation

• What if set of matches contains gross outliers?(to keep things simple let’s consider line fitting first)

RANSACObjective

Robust fit of model to data set S which contains outliers

Algorithm

(i) Randomly select a sample of s data points from S and instantiate the model from this subset.

(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.

(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate

(iv) If the size of Si is less than T, select a new subset and repeat the above.

(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

Distance threshold

Choose t so probability for inlier is α (e.g. 0.95)

• Often empirically• Zero-mean Gaussian noise σ then

follows distribution with m=codimension of model

2d2

m(dimension+codimension=dimension space)

Codimension

Model t 2

1 line,F 3.84σ2

2 H,P 5.99σ2

3 T 7.81σ2

How many samples?Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99

sepN 11log/1log

peNs 111

proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 117

7Note: Assumes that inliers allow to identify other inliers

Acceptable consensus set?

• Typically, terminate when inlier ratio reaches expected ratio of inliers

neT 1

Adaptively determining the number of samples

e is often unknown a priori, so pick worst case, i.e. 0, and adapt if more inliers are found, e.g. 80% would yield e=0.2

• N=∞, sample_count =0• While N >sample_count repeat

• Choose a sample and count the number of inliers• Set e=1-(number of inliers)/(total number of

points)• Recompute N from e• Increment the sample_count by 1

• Terminate

sepN 11log/1log

Other robust algorithms

• RANSAC maximizes number of inliers

• LMedS minimizes median error

• Not recommended: case deletion, iterative least-squares, etc.

100%

50%

residual(pixels)

inlier percentile

1.25

Non-linear refinment

algebraic minimization

possible to iteratively minimize algebraic distance subject to det F=0 (see H&Z if interested)

Geometric distance

Gold standard

Symmetric epipolar distance

Gold standard

Maximum Likelihood Estimation

i

iiii dd 22 'x̂,x'x̂,x

(= least-squares for Gaussian noise)

0x̂F'x̂ subject to T

iXt],|[MP'0],|[IP Parameterize:

Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi

iiii XP'x̂,PXx̂

Minimize cost using Levenberg-Marquardt(preferably sparse LM, e.g. see H&Z)

(overparametrized)

Gold standard

Alternative, minimal parametrization (with a=1)

(note (x,y,1) and (x‘,y‘,1) are epipoles)

problems: • a=0 pick largest of a,b,c,d to fix to 1

• epipole at infinity

pick largest of x,y,w and of x’,y’,w’

4x3x3=36 parametrizations!

reparametrize at every iteration, to be sure

Symmetric epipolar error

2

221

2

2T2

1T

T

FxFx

1

Fx'Fx'

1Fxx'

i

iiii dFd2T2 x'F,xx,x'

Some experiments:

Some experiments:

Some experiments:

Some experiments:

i

iiii dFd2T2 x'F,xx,x'

(for all points!)

Residual error:

Recommendations:

1. Do not use unnormalized algorithms

2. Quick and easy to implement: 8-point normalized

3. Better: enforce rank-2 constraint during minimization

4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

Special case:

Enforce constraints for optimal results:Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof)

Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do

Step 3.1 select minimal sample (i.e. 7 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90%

80%

70% 60%

50%

#samples

5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

Automatic computation of F

Abort verification early

Assume percentage of inliers p, what is the probability P to pick n or more wrong samples out of m?stop if P<0.05 (sample most probably contains outlier) (P=cum. binomial distr. funct. (m-n,m,p ))

To avoid problems this requires to also verify at random!(but we already have a random sampler anyway)

(inspired from Chum and Matas BMVC2002)

O O O O O I O O I O O O OI I I O I I I I O O I O I I I I I I O I I I I I I O I O O I I I I…

O O O O O O O

O O O O I O O O O

restrict search range to neighborhood of epipolar line (e.g. 1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches

• Degenerate cases• Planar scene• Pure rotation

• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)

• Solution 1: Model selection (Torr et al., ICCV´98, Kanatani, Akaike)

• Compare H and F according to expected residual error (compensate for model complexity)

• Solution 2: RANSAC• Compare H and F according to inlier count

(see next slide)

Degenerate cases:

RANSAC for quasi-degenerate cases

• Full model (8pts, 1D solution)

• Planar model (6pts, 3D solution)

Accept if large number of remaining inliers

• Plane+parallax model (plane+2pts)

Sample for out of plane points among outliers

closest rank-6 of Anx9 for all plane inliers

(accept inliers to solution F)

(accept inliers to solution F1,F2&F3)

• Absence of sufficient features (no texture)• Repeated structure ambiguity

(Schaffalitzky and Zisserman, BMVC‘98)

• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition

More problems:

RANSAC for ambiguous matching

• Include multiple candidate matches in set of potential matches

• Select according to matching probability (~ matching score)

• Helps for repeated structures or scenes with similar features as it avoids an early commitment

(Tordoff and Murray ECCV02)

geometric relations between two views is fully

described by recovered 3x3 matrix F

two-view geometry

Next class: rectification and stereo

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)