Post on 21-Dec-2015
transcript
Assignment 3
• Collect potential matches from all algorithms for all pairs• Matlab ASCII format, exchange data
• Implement RANSAC that uses combined match dataset
• Compute consistent set of matches and epipolar geometry• Report thresholds used, match sets used, number of
consistent matches obtained, epipolar geometry, show matches and epipolar geometry (plot some epipolar lines).
Due next Tuesday, Nov. 2
naming convention: firstname_ij.dat
chris_56.dat
[F,inliers]=FRANSAC([chris_56; brian_56; …])
http://www.unc.edu/courses/2004fall/comp/290/089/assignment3/
Papers
• Each should present a paper during 20-25 minutes followed by discussion. Partially outside of class schedule to make up for missed classes.(When?)
• List of proposed papers will come on-line by Thursday, feel free to propose your own (suggestion: something related to your project).
• Make choice by Thursday, assignments will be made in class.
• Everybody should have read papers that are being discussed.
PapersChris
Nathan
Brian
Li
Chad
Seon Joo
Jason
Sudipta
Sriram
Christine
http://www.unc.edu/courses/2004fall/comp/290b/089/papers/
3D photography course schedule
Introduction
Aug 24, 26 (no course) (no course)
Aug.31,Sep.2
(no course) (no course)
Sep. 7, 9 (no course) (no course)
Sep. 14, 16 Projective Geometry Camera Model and Calibration
(assignment 1)
Feb. 21, 23 Camera Calib. and SVM Feature matching(assignment 2)
Feb. 28, 30 Feature tracking Epipolar geometry(assignment 3)
Oct. 5, 7 Computing F Triangulation and MVG
Oct. 12, 14 (university day) (fall break)
Oct. 19, 21 Stereo Active ranging
Oct. 26, 28 Structure from motion SfM and Self-calibration
Nov. 2, 4 Shape-from-silhouettes Space carving
Nov. 9, 11 3D modeling Appearance Modeling Nov.12 papers(2-3pm SN115)
Nov. 16, 18 (VMV’04) (VMV’04)
Nov. 23, 25 papers & discussion (Thanksgiving)
Nov.30,Dec.2
papers & discussion papers and discussion Dec.3 papers(2-3pm SN115)
Dec. 7? Project presentations
Ideas for a project?
Chris Wide-area display reconstruction
Nathan ?
Brian ?
Li Visual-hulls with occlusions
Chad Laser scanner for 3D environments
Seon Joo Collaborative 3D tracking
Jason SfM for long sequences
SudiptaCombining exact silhouettes and photoconsistency
Sriram Panoramic cameras self-calibration
Christine desktop lamp scanner
Dealing with dominant planar scenes
• USaM fails when common features are all in a plane
• Solution: part 1 Model selection to detect problem
(Pollefeys et al., ECCV‘02)
Dealing with dominant planar scenes
• USaM fails when common features are all in a plane• Solution: part 2 Delay ambiguous computations
until after self-calibration(couple self-calibration over all 3D
parts)
(Pollefeys et al., ECCV‘02)
Non-sequential image collections
4.8im/pt64 images
3792
po
ints
Problem:Features are lost and reinitialized as new features
Solution:Match with other close views
For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure
Relating to more views
Problem: find close views in projective frame
For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmFor all close views k
Compute two view geometry k/i and matchesInfer new 2D-3D matches and add to list
Refine pose using all 2D-3D matchesRefine existing structureInitialize new structure
Determining close views
• If viewpoints are close then most image changes can be modelled through a planar homography
• Qualitative distance measure is obtained by looking at the residual error on the best possible planar homography
Distance = m´,mmedian min HD
9.8im/pt
4.8im/pt
64 images
64 images
3792
po
ints
2170
po
ints
Non-sequential image collections (2)
Hierarchical structure and motion recovery
• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction
FT
H
PM
Stitching 3-view reconstructions
Different possibilities1. Align (P2,P3) with (P’1,P’2) -1
23-1
12H
HP',PHP',Pminarg AA dd
2. Align X,X’ (and C,C’) j
jjAd HX',XminargH
3. Minimize reproj. error
jjj
jjj
d
d
x',HXP'
x,X'PHminarg 1-
H
4. MLE (merge) j
jjd x,PXminargXP,
Refining structure and motion
• Minimize reprojection error
• Maximum Likelyhood Estimation (if error zero-mean
Gaussian noise)• Huge problem but can be solved
efficiently (Bundle adjustment)
m
k
n
iikD
ik 1 1
2
kiM̂,P̂
M̂P̂,mmin
Sparse bundle adjustment
U1
U2
U3
WT
W
V
P1 P2 P3 M
Non-linear min. requires to solve Jacobian of has sparse block
structure
J JJN T
12xm 3xn(in general
much larger)
im.pts. view 1
m
k
n
iikD
1 1
2
ki M̂P̂,m
Needed for non-linear minimization
0
T-1T JJJ e
Sparse bundle adjustment• Eliminate dependence of
camera/motion parameters on structure parametersNote in general 3n >> 11m
WT V
U-WV-1WT
NI0WVI 1
11xm 3xn
Allows much more efficient computations
e.g. 100 views,10000 points,
solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms
Motivation
• Avoid explicit calibration procedure• Complex procedure• Need for calibration object • Need to maintain calibration
Motivation
• Allow flexible acquisition• No prior calibration necessary• Possibility to vary intrinsics• Use archive footage
Projective ambiguity
Reconstruction from uncalibrated images
projective ambiguity on reconstruction
´M´M))((Mm 1 PTPTP
Stratification of geometry
15 DOF 12 DOFplane at infinity
parallelism
More general
More structure
Projective Affine Metric
7 DOFabsolute conicangles, rel.dist.
Constraints ?
Scene constraints• Parallellism, vanishing points, horizon, ...• Distances, positions, angles, ...Unknown scene no constraints
Camera extrinsics constraints–Pose, orientation, ...
Unknown camera motion no constraints Camera intrinsics constraints
–Focal length, principal point, aspect ratio & skew
Perspective camera model too general some constraints
Euclidean projection matrix
tRRKP TT
1yy
xx
uf
usf
K
Factorization of Euclidean projection matrix
Intrinsics:
Extrinsics: t,R
Note: every projection matrix can be factorized,
but only meaningful for euclidean projection matrices
(camera geometry)
(camera motion)
Constraints on intrinsic parameters
Constant e.g. fixed camera:
Knowne.g. rectangular pixels:
square pixels: principal point known:
21 KK
0s
1yy
xx
uf
usf
K
0, sff yx
2,
2,
hwuu yx
Self-calibration
Upgrade from projective structure to metric structure using constraints on intrinsic camera parameters• Constant intrinsics
• Some known intrinsics, others varying
• Constraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)
(Faugeras et al. ECCV´92, Hartley´93,
Triggs´97, Pollefeys et al. PAMI´99, ...)
(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)
(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)
A counting argument
• To go from projective (15DOF) to metric (7DOF) at least 8 constraints are needed
• Minimal sequence length should satisfy
• Independent of algorithm• Assumes general motion (i.e. not critical)
8#1# fixedmknownm
The Dual Absolute Quadric
00
0I*T
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. 8 dof2. plane at infinity π∞ is the nullvector of Ω∞
3. Angles:
2*
21*
1
2*
1
ππππ
ππcos
TT
T
Absolute Dual Quadric and Self-calibration
Eliminate extrinsics from equation
Equivalent to projection of Dual Abs.Quadric
))(Ω)((Ω *1* TTTTT PTTTPTPPKK
Dual Abs.Quadric also exists in projective world
T´Ω´´ * PP Transforming world so thatreduces ambiguity to similarity
** ΩΩ´
*
*
projection
constraints
Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics
Tii
Tiii Ωω KKPP
Absolute Dual Quadric and Self-calibration
Projection equation:Projection equation:
Translate constraints on K through projection equation to constraints on *
Constraints on *
1
ω 22
222
*
yx
yyyyxy
xyxyxx
cc
ccfccsf
cccsfcsf
Zero skew quadratic m
Principal point linear 2m
Zero skew (& p.p.)
linear m
Fixed aspect ratio (& p.p.& Skew)
quadratic m-1
Known aspect ratio (& p.p.& Skew)
linear m
Focal length (& p.p. & Skew)
linear m
*23
*13
*33
*12 ωωωω
0ωω *23
*13
0ω*12
*11
*22
*22
*11 ω'ωω'ω
*22
*11 ωω
*11
*33 ωω
condition constraint type #constraints
Linear algorithm
Assume everything known, except focal length
0Ω
0Ω
0Ω
0ΩΩ
23T
13T
12T
22T
11T
PP
PP
PP
PPPP
(Pollefeys et al.,ICCV´98/IJCV´99)
TPP *2
2
*
100
0ˆ0
00ˆ
ω
f
f
Yields 4 constraint per imageNote that rank-3 constraint is not enforced
Linear algorithm revisited
0Ω
0Ω
0Ω
0ΩΩ
23T
13T
12T
22T
11T
PP
PP
PP
PPPP
100
0ˆ0
00ˆ2
2
f
fTKK
9
1
9
1
)3log()1log()ˆlog( f)1.1log()1log()log( ˆ
ˆ
y
x
f
f1.00xc1.00yc
0s
0ΩΩ
0ΩΩ
33T
22T
33T
11T
PPPP
PPPP
(Pollefeys et al., ECCV‘02)
1.0
11.0
101.0
12.0
1
assumptions
Weighted linear equations
Projective to metric
Compute T from
using eigenvalue decomposition of and then obtain metric
reconstruction as
00
0
~ withΩ
~or Ω
~ **T
T-1-T IITITTTI
M and TPT-1
Ω*
Alternatives: (Dual) image of absolute conic
• Equivalent to Absolute Dual Quadric
• Practical when H can be computed first• Pure rotation (Hartley’94, Agapito et al.’98,’99)
• Vanishing points, pure translations, modulus constraint, …
T** ωω HH ea)( HH
TPP ** Ωω
1ω 22
22
*
yx
yyyyx
xyxxx
ccccfcc
ccccf
22222222
22
22
220
01
ω
yxxyyxyxxy
yxx
xyy
yx cfcfffcfcf
cff
cff
ff
Note that in the absence of skew the IAC can be more practical than the DIAC!
Kruppa equations
Limit equations to epipolar geometryOnly 2 independent equations per pairBut independent of plane at infinity
T*TT*T* ωe'ωe'e'ωe' FFHH
Refinement
• Metric bundle adjustment
Enforce constraints or priors on intrinsics during minimization(this is „self-calibration“ for photogrammetrist)
Critical motion sequences
• Self-calibration depends on camera motion
• Motion sequence is not always general enough
• Critical Motion Sequences have more than one potential absolute conic satisfying all constraints
• Possible to derive classification of CMS
(Sturm, CVPR´97, Kahl, ICCV´99, Pollefeys,PhD´99)
Critical motion sequences:constant intrinsic parameters
Most important cases for constant intrinsics
Critical motion type
ambiguity
pure translation affine transformation (5DOF)pure rotation arbitrary position for (3DOF)orbital motion proj.distortion along rot. axis
(2DOF)planar motion scaling axis plane (1DOF)
Note relation between critical motion sequences and restricted motion algorithms
Critical motion sequences:varying focal length
Most important cases for varying focal length (other parameters known)Critical motion type
ambiguity
pure rotation arbitrary position for (3DOF)forward motion proj.distortion along opt. axis
(2DOF)translation and rot. about opt. axis
scaling optical axis (1DOF)
hyperbolic and/or elliptic motion
one extra solution
Critical motion sequences:algorithm dependent
Additional critical motion sequences can exist for some specific algorithms• when not all constraints are enforced
(e.g. not imposing rank 3 constraint)• Kruppa equations/linear algorithm: fixating
a pointSome spheres also project to circles located in the image and hence satisfy all the linear/kruppa self-calibration constraints
Non-ambiguous new views for CMS
• restrict motion of virtual camera to CMS• use (wrong) computed camera parameters
(Pollefeys,ICCV´01)