Date post: | 22-Dec-2015 |
Category: |
Documents |
View: | 224 times |
Download: | 2 times |
3D photography course schedule(tentative)
Lecture Exercise
Sept 26 Introduction -
Oct. 3 Geometry & Camera model Camera calibration
Oct. 10 Single View Metrology Measuring in images
Oct. 17 Feature Tracking/matching(Friedrich Fraundorfer)
Correspondence computation
Oct. 24 Epipolar Geometry F-matrix computation
Oct. 31 Shape-from-Silhouettes(Li Guan)
Visual-hull computation
Nov. 7 Stereo matching Project proposals
Nov. 14 Structured light and active range sensing
Papers
Nov. 21 Structure from motion Papers
Nov. 28 Multi-view geometry and self-calibration
Papers
Dec. 5 Shape-from-X Papers
Dec. 12 3D modeling and registration Papers
Dec. 19 Appearance modeling and image-based rendering
Final project presentations
• Brightness constancy assumption
Optical flow
(small motion)
• 1D example
possibility for iterative refinement
• Brightness constancy assumption
Optical flow
(small motion)
• 2D example
(2 unknowns)
(1 constraint)?
isophote I(t)=Iisophote I(t+1)=I
the “aperture” problem
Optical flow• How to deal with aperture problem?
Assume neighbors have same displacement
(3 constraints if color gradients are different)
Revisiting the small motion assumption
• Is this motion small enough?• Probably not—it’s much larger than one pixel
(2nd order terms dominate)• How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
image It-1 image I
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
slides fromBradsky and Thrun
image Iimage J
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
slides fromBradsky and Thrun
Feature tracking
• Identify features and track them over video• Small difference between frames• potential large difference overall
• Standard approach: KLT (Kanade-Lukas-Tomasi)
Good features to track
• Use same window in feature selection as for tracking itself
• Compute motion assuming it is small
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
differentiate:
Example
Simple displacement is sufficient between consecutive frames, but not to compare to reference template
Good features to keep tracking
Perform affine alignment between first and last frameStop tracking features with too large errors
(i) Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the
corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?
Three questions:
Two-view geometry
The epipolar geometry
epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)
The fundamental matrix F
algebraic representation of epipolar geometry
l'x
we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F
The fundamental matrix F
geometric derivation
xHx' π
x'e'l' FxxHe' π
mapping from 2-D to 1-D family (rank 2)
The fundamental matrix F
algebraic derivation
λCxPλX IPP
PP'e'F
xPP'CP'l'
(note: doesn’t work for C=C’ F=0)
xP
λX
The fundamental matrix F
correspondence condition
0Fxx'T
The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images 0l'x'T
The fundamental matrix F
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)
(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x
e’TF=0, similarly Fe=0(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(v) F is a correlation, projective mapping from a point x to
a line l’=Fx (not a proper correlation, i.e. not invertible)
Fundamental matrix for pure translation
PP'e'F
0]|K[IP t]|K[IP'
0KP
-1
00
0e'F
xy
xz
yz
eeeeee
General motion
Pure translation
for pure translation F only has 2 degrees of freedom
The fundamental matrix F
relation to homographies
lHl' -T
π FHe'
π
valid for all plane homographies
eHe'π
The fundamental matrix F
relation to homographies
FxlxH'xππ
requires
πl
πx
x x
Fe'H e.g. 0e'e'T 0e'lT
π
Projective transformation and invariance
-1-T FHH'F̂ x'H''x̂ Hx,x̂
Derivation based purely on projective concepts
X̂P̂XHPHPXx -1
F invariant to transformations of projective 3-space
X̂'P̂XHHP'XP'x' -1
FP'P,
P'P,F
unique
not unique
canonical form
m]|[MP'0]|[IP
MmF
PP'e'F
Projective ambiguity of cameras given Fprevious slide: at least projective ambiguitythis slide: not more!
Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’
~ ~
~ ~
]a~|A~
['P~
0]|[IP~
a]|[AP' 0]|[IP
A
~a~AaF
T1 avAA~
kaa~ kandlemma:
kaa~Fa~0AaaaF2rank
TavA-A~
k0A-A~
kaA~
a~Aa
kkIkT1
1
v0H
'P~
]a|av-A[
v0a]|[AHP'
T1
T1
1
kk
kkIk
(22-15=7, ok)
The projective reconstruction theorem
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
allows reconstruction from pair of uncalibrated images!
C1
C2
l2
l1
e1
e20m m 1
T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong
matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Epipolar geometry
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Other entities besides points?
Lines give no constraint for two view geometry(but will for three and more views)
Curves and surfaces yield some constraints related to tangency
(e.g. Sinha et al. CVPR’04)
Computation of F
• Linear (8-point)• Minimal (7-point)• Robust (RANSAC)• Non-linear refinement (MLE, …)
• Practical approach
Epipolar geometry: basic equation
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx
(data) (unknowns)(linear)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differencebetween column of data matrix least-squares yields poor results
the NOT normalized 8-point algorithm
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
normalized least squares yields good results (Hartley, PAMI´97)
the normalized 8-point algorithm
the singularity constraint
0Fe'T 0Fe 0detF 2Frank
T333
T222
T111
T
3
2
1
VσUVσUVσUVσ
σσ
UF
SVD from linearly computed F matrix (rank 3)
T222
T111
T2
1
VσUVσUV0
σσ
UF'
FF'-FminCompute closest rank-2 approximation
the minimum case – 7 point correspondences
0f1''''''
1''''''
777777777777
111111111111
yxyyyxyxyxxx
yxyyyxyxyxxx
T9x9717x7 V0,0,σ,...,σdiagUA
9x298 0]VA[V T8
T ] 000000010[Ve.g.V
1...70,)xλFF(x 21T iii
one parameter family of solutions
but F1+F2 not automatically rank 2
F1 F2
F
3
F7pts
0λλλ)λFFdet( 012
23
321 aaaa
(obtain 1 or 3 solutions)
(cubic equation)
0)λIFFdet(Fdet)λFFdet( 1-12221
the minimum case – impose rank 2
Compute possible as eigenvalues of (only real solutions are potential solutions)
1-12 FF
B.detAdetABdet
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
Automatic computation of F
RANSAC
restrict search range to neighborhood of epipolar line (e.g. 1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
• (Mostly) planar scene (see next slide)• Absence of sufficient features (no texture)• Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
Issues:
Computing F for quasi-planar scenes QDEGSAC
17% success for RANSAC
100% for QDEGSAC #i
nlie
rsdata rank
337 matches on plane, 11 off plane
%inclusion of out-of-plane inliers