Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 243 times |
Download: | 2 times |
ComputerVision
Multiple View Geometry& Stereo
Marc PollefeysCOMP 256
ComputerVision
2
C1
C2
l2
l1
e1
e20p p 1
T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong
matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1p1
L1
p2
L2
M
C2
p1
p2
C1
C2
l2
l1
e1
e2
p1
L1
p2
L2
M
l2lT1
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Last class: epipolar geometry
ComputerVision
3
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
ComputerVision
4
Multiple Views (Faugeras and Mourrain, 1995)
ComputerVision
5
Two Views
Epipolar Constraint
ComputerVision
6
Three Views
Trifocal Constraint
ComputerVision
7
Four Views
Quadrifocal Constraint(Triggs, 1995)
ComputerVision
8
Geometrically, the four rays must intersect in P..
ComputerVision
9
Quadrifocal Tensorand Lines
ComputerVision
10
Quadrifocal tensor
• determinant is multilinear
thus linear in coefficients of lines !
• There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor
wi
wi
yi
yi
xi
xii
Ti MlMlMl Ml
Twi
yi
xi lll ][
s
r
q
p
pqrs
MMMM
Q
4
3
2
1
det
ComputerVision
11
from perspective to omnidirectional cameras
perspective camera(2 constraints / feature)
radial camera (uncalibrated)(1 constraints / feature)
3 constraints allow to reconstruct 3D point
more constraints also tell something about cameras
multilinear constraints known as epipolar, trifocal and quadrifocal constraints
(0,0)
l=(y,-x)
(x,y)
ComputerVision
12
Radial quadrifocal tensor
• Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
• Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
(2x2x2 tensor)
Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation• Radial trifocal tensor Tijk from 7 points in 3 views
• Reconstruct 2D panorama and use it for calibration
(x,y)
ComputerVision
13
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radius
angle
• Models fish-eye lenses, cata-dioptric systems, etc.
ComputerVision
14
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radiusangle
90o
• Models fish-eye lenses, cata-dioptric systems, etc.
ComputerVision
15
STEREOPSIS
Reading: Chapter 11.
• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras
ComputerVision
16
An Application: Mobile Robot Navigation
The Stanford Cart,H. Moravec, 1979.
The INRIA Mobile Robot, 1990.
Courtesy O. Faugeras and H. Moravec.
ComputerVision
17
Reconstruction / Triangulation
ComputerVision
18
(Binocular) Fusion
ComputerVision
19
Reconstruction
• Linear Method: find P such that
• Non-Linear Method: find Q minimizing
ComputerVision
20
Rectification
All epipolar lines are parallel in the rectified image plane.
ComputerVision
21
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
He001
ComputerVision
23
ComputerVision
24
ComputerVision
25
Polar re-parameterization around epipoles
Requires only (oriented) epipolar geometry
Preserve length of epipolar linesChoose so that no pixels are
compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
ComputerVision
26
polar rectification: example
ComputerVision
27
polar rectification: example
ComputerVision
28
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
ComputerVision
29
Example: Béguinage of Leuven
ComputerVision
30
Reconstruction from Rectified Images
Disparity: d=u’-u. Depth: z = -B/d.
ComputerVision
31
Stereopsis
Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.
ComputerVision
32
Human Stereopsis: Reconstruction
Disparity: d = r-l = D-F.
d=0
d<0
In 3D, the horopter.
ComputerVision
33
Human Stereopsis: experimental horopter…
ComputerVision
34
Iso-disparity curves: planar retinas
X0X1
C1
X∞
C2
Xi Xj
ii
i
i
1
1:
1
01
1
01
11
:
11
10
11
0
the retina act as if it were flat!
ComputerVision
35
What if F is not known?
Human Stereopsis: Reconstruction
Helmoltz (1909):
• There is evidence showing the vergence anglescannot be measured precisely.
• Humans get fooled by bas-relief sculptures.
• There is an analytical explanation for this.
• Relative depth can be judged accurately.
ComputerVision
36
Human Stereopsis: Binocular Fusion
How are the correspondences established?
Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one??• There is anecdotal evidence for the latter (camouflage).
• Random dot stereograms provide an objective answerBP!
ComputerVision
37
A Cooperative Model (Marr and Poggio, 1976)
Excitory connections: continuity
Inhibitory connections: uniqueness
Iterate: C = C - w C + C .e i 0
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
ComputerVision
38
Correlation Methods (1970--)
Normalized Correlation: minimize instead.
Slide the window along the epipolar line until w.w’ is maximized.
2Minimize |w-w’|.
ComputerVision
39
Correlation Methods: Foreshortening Problems
Solution: add a second pass using disparity estimates to warpthe correlation windows, e.g. Devernay and Faugeras (1994).
Reprinted from “Computing Differential Properties of 3D Shapes from Stereopsis without 3D Models,” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994). 1994 IEEE.
ComputerVision
40
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
• Edges are found by repeatedly smoothing the image and detectingthe zero crossings of the second derivative (Laplacian).• Matches at coarse scales are used to offset the search for matchesat fine scales (equivalent to eye movements).
ComputerVision
41
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
One of the twoinput images
Image Laplacian
Zeros of the Laplacian
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
ComputerVision
42
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
ComputerVision
43
The Ordering Constraint
But it is not always the case..
In general the pointsare in the same orderon both epipolar lines.
ComputerVision
44
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
ComputerVision
45
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.
ComputerVision
46
Three Views
The third eye can be used for verification..
ComputerVision
47
More Views (Okutami and Kanade, 1993)
Pick a reference image, and slide the correspondingwindow along the corresponding epipolar lines of allother images, using inverse depth relative to the firstimage as the search parameter.
Use the sum of correlation scores to rank matches.
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
ComputerVision
48
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
ComputerVision
49
Hierarchical stereo matchingD
ow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
ComputerVision
50
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
ComputerVision
51
Example: reconstruct image from neighboring images
ComputerVision
52
I1 I2 I10
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
ComputerVision
53
Multi-view depth fusion
• Compute depth for every pixel of reference image– Triangulation– Use multiple views– Up- and down sequence– Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
ComputerVision
54
Real-time stereo on graphics hardware
• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate
results over support region• Trade-off between small and large support window
Yang and Pollefeys CVPR03
140M disparity hypothesis/sec on Radeon 9700pro140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hze.g. 512x512x20disparities at 30Hz
Shape of a kernel for summing up 6 levels
ComputerVision
55
Sample Re-Projections
near far
ComputerVision
56
(1x1)
(1x1+2x2)
(1x1+2x2 +4x4+8x8)
(1x1+2x2 +4x4+8x8 +16x16)
Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass
video
ComputerVision
57
Cool ideas
• Space-time stereo (varying illumination, not shape)
ComputerVision
58
More on stereo …
The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/
Recommended reading
D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).
ComputerVision
59
Next class:Optical Flow: where do pixels move to?