ECCV 2010 TUTORIAL
NONRIGID STRUCTURE FROM MOTION
YASER SHEIKHThe Robotics InstituteCarnegie Mellon UniversityPittsburgh, USAhttp://cs.cmu.edu/~yaser
SOHAIB KHANComputer Vision LabLUMS School of Science & Engineering Lahore, PAKISTANhttp://web.lums.edu.pk/~sohaib
http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE
x
z
3D Structure That Deforms Over Time
4D DYNAMIC STRUCTURE
IMAGE MOTION
OBJECT MOTION CAMERA MOTION
IMAGE MOTION
OBJECT MOTION CAMERA MOTIONAND
RIGID STRUCTURE FROM MOTION
NONRIGID STRUCTURE FROM MOTION
NONRIGID STRUCTURE FROM MOTION
NONRIGID STRUCTURE FROM MOTION
NONRIGID STRUCTURE FROM MOTION
ONLY ONE VIEW PER 3D CONFIGURATON: ILL-POSED PROBLEM EQUIVALENT TO FINDING 3D FROM SINGLE IMAGE
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
ASSUMPTIONS� Orthographic Camera� At least 3 images� Rigid Scene� Camera Motion� Corresponding points available
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
NOTATION� P 3D points seen in F frames
� Xj = [Xj, Yj, Zj] is jth 3D point 1��j ��P
� xij = [xij, yij] is the projection of Xj in ith frame 1��i ��F
� Pi is the camera projection matrix if the ith frame 1��i ��F
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
orthographicprojection
matrix2D image
point
3D scenepoint
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation
2 rows of a 3D rotation
matrix
imageoffset
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation
2 rows of a 3D rotation
matrix
imageoffset
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
PROJECTION OF PP 3D POINTS IN ith IMAGE
PROJECTION OF P 3D POINTS IN F IMAGES
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
PROJECTION OF PP 3D POINTS IN F IMAGES
W R
S
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
Image Observations Matrix, W
×
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
Image Observations Matrix, W
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
HOW TO SOLVE FOR Q� Observation: The correct Q will result in an R whose rows
are pair-wise orthonormal
� The ith image results in the following 3 constraints on Q
� Total 3F constraints on 6 terms of QQT
� Can be solved linearly for G = QQT for F ����
ORTHONORMALITYCONSTRAINTS
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE NONRIGID STRUCTURE
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE NONRIGID STRUCTURE
NONRIGID STRUCTURE FROM MOTIONComparison with Rigid Structure from Motion
RIGID SFM NONRIGID SFM
W R
S
W RS
S(1)
S(2)
S(F)
S(3)...
R1R2
R3� � �
RF
NONRIGID STRUCTURE FROM MOTION
RIGID SFM� Inputs:
100 pts x 40 sec x 30 fps x 2 (x,y)= 240,000 observations
� Unknowns:100 points x 3 (X,Y,Z) = 300 unknowns
NONRIGID SFM
Explosion of Unknowns
Example: Given a 40 second video with 100 tracked points
� Inputs:100 pts x 40 sec x 30 fps x 2= 240,000 observations
� Unknowns:100 points x 40 sec x 30 fps x 3 = 360,000 unknowns
NONRIGID STRUCTURE FROM MOTIONExplosion of Unknowns
IN GENERAL, NRSFM HAS MORE UNKNOWNS THAN CONSTRAINTS
ILL-POSED PROBLEM: Additional assumptions are necessary to constrain the solution.
HOWEVER…Motion is not random: 3D points are often highly correlated in space and timePoints move because an actuator exerts force on them
Hence their acceleration is limited by the actuating forceTherefore, shape does not deform arbitrarily over time
4D STRUCTURE OFTEN LIES IN A LOW DIMENSIONAL SUBSPACE
NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches
Shape Basis3D points at each time instant lie in a low dimensional subspace
Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace
EXAMPLES OF APPLICATIONSMatch Moving in Movies
Akhter et al. NIPS 2008
EXAMPLES OF APPLICATIONSMotion-Capture
Input Video Two views of the reconstruction
Akhter et al. NIPS 2008
EXAMPLES OF APPLICATIONSMotion-Capture Cleanup
ReconstructionUnlabeled DataVideoInput Output
Disney Research, Pittsburgh
EXAMPLES OF APPLICATIONSTracking in 2D and 3D
Credit: Iain Matthews
EXAMPLES OF APPLICATIONSAnimation
Jain et al. SCA 2010
EXAMPLES OF APPLICATIONSBrowsing Image Collections
Credit: Hyun Soo Park
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
time
spac
e
DYNAMIC STRUCTURE
S3F×P =
⎡⎢⎢⎢⎣
X11 X12 · · · X1PX21 X22 · · · X2P...
......
XF1 XF2 · · · XFP
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
DYNAMIC STRUCTUREUNDER ORTHOGRAPHIC PROJECTION
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦=
W = RX
Saturday, September 4, 2010
LINEAR SHAPE MODEL[T. Cootes et al. 91, Bregler et al. 97]
w1 × + w2 × + … + wK × = α1 α2 αk
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦ =
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
LINEAR SHAPE MODEL
5Saturday, September 4, 2010
LINEAR SHAPE MODEL RECONSTRUCTION
5 Basis 15 Basis 25 Basis
6Saturday, September 4, 2010
LINEAR SHAPE MODEL UNDER ORTHOGRAPHIC PROJECTION
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦=
=
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
2F x P 2F x 3F (6F) 3F x P
2F x 3F (6F) 3F x 3k 3k x P
Saturday, September 4, 2010
KNOWNS VS UNKNOWNS
2F × P6F + (3F × k) + (k × P )
2F × P ≥ 6F + (3F × k) + (k × P )
KNOWNS:UNKNOWNS:
Saturday, September 4, 2010
LINEAR SHAPE MODEL
w1 × + w2 × + … + wK × = α1 α2 αk
Y = R(
K∑i=1
ωibi) +T
RIGID COMPONENT
NONRIGID COMPONENT
IDEA: RIGID COMPONENT GETS FOLDED INTO PROJECTION
Saturday, September 4, 2010
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
BREGLER et al. 2000Nested SVD
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦=
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkRF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦=
2F x 3k 3k x P
Saturday, September 4, 2010
BREGLER et al. 2000Outer SVD
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦=
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkRF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
2F x 3k 3k x P
SVD
W H B=
W = UDVT
W = (UD12 )(D
12VT )
W = ĤB̂
Saturday, September 4, 2010
BREGLER et al. 2000Inner SVD
SVD
W = ĤB̂⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H
[ω11r
11 ω11r
21 ω11r
31 · · · ω1kr11 ω1kr21 ω1kr31
ω11r41 ω11r
51 ω11r
61 · · · ω1kr41 ω1kr51 ω1kr61
]
⎡⎢⎢⎢⎣
ω11r11 ω11r
21 ω11r
31 ω11r
41 ω11r
51 ω11r
61
ω12r11 ω12r
21 ω12r
31 ω12r
41 ω12r
51 ω12r
61
......
ω1kr11 ω1kr
21 ω1kr
31 ω1kr
41 ω1kr
51 ω1kr
61
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11ω12...
ω1k
⎤⎥⎥⎥⎦[r11 r
21 r
31 r
41 r
51 r
61
]
rank 1
=
=h1
h′1 =
h′1 = udvT = ω̂r̂
=
METRIC RECTIFICATION USING ORTHONORMALITY CONSTRAINTS
Saturday, September 4, 2010
BREGLER et al. 2000OVERVIEW
• OUTER SVD: PERFORM SVD ON W TO GET ESTIMATES OF:• H: CAMERA PROJECTIONS AND COEFFICIENTS
• INNER SVD: PERFORM SVD ON H TO GET ESTIMATES OF:• OMEGA: COEFFICIENTS• R: CAMERA PROJECTIONS
• METRIC RECTIFY USING ORTHONORMALITY CONSTRAINTS• B: THE SHAPE BASIS
Saturday, September 4, 2010
RESULTS
Saturday, September 4, 2010
BREGLER et al. 2000IN PERSPECTIVE
• SEMINAL WORK: SHOWED THAT FACTORIZATION METHODS CAN BE APPLIED TO NONRIGID OBJECTS
• CASCADING ERROR: ANY OUTER SVD ESTIMATION ERROR CASCADES INTO INNER SVD ESTIMATION
• AMBIGUITY ERROR: ESTIMATION OF METRIC RECTIFICATION• NUMBER OF BASIS: LARGE NUMBER OF BASIS REQUIRED• MISSING DATA: NEEDS COMPLETE W MATRIX
Saturday, September 4, 2010
METRIC RECTIFICATIONAMBIGUITY
W = ĤB̂
W = ĤGG−1B̂
H = ĤG B = G−1B̂
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H ==
⎡⎣ Ĥ
⎤⎦⎡⎣ | | |g1 g2 . . . gk
| | |
⎤⎦
G3k X 3k
Saturday, September 4, 2010
METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H ==
⎡⎣ Ĥ
⎤⎦⎡⎣ | | |g1 g2 . . . gk
| | |
⎤⎦
⎡⎣ Ĥ
⎤⎦⎡⎣ |gk
|
⎤⎦ =
⎡⎢⎢⎢⎣
ω1kR1ω2kR2
...ωFkRF
⎤⎥⎥⎥⎦
ωikRi = Ĥ2i−1:2igk
RiRTi = I
ORTHONORMALITY CONSTRAINT
H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI
Saturday, September 4, 2010
METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT
RiRTi = I
ORTHONORMALITY CONSTRAINT
H2igkgTk Ĥ2i−1 = 0
H2igkgTk Ĥ2i = ω
2ikH2i−1gkg
Tk Ĥ2i−1 = ω
2ik
H2i−1gkgTk Ĥ2i−1 = H2igkgTk Ĥ2i
H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI =
[ω2ik 00 ω2ik
]
Saturday, September 4, 2010
CHALLENGE?AMBIGUITY
Saturday, September 4, 2010
OPTIMIZATION
Saturday, September 4, 2010
CHALLENGEMISSING DATA
• A.M. Buchanan and A.W. Fitzgibbon, “Damped Newton Algorithms for Matrix Factorization with Missing Data,” IEEE International Conference on Computer Vision and Pattern Recognition, 2005.
• L. Torresani, A. Hertzmann, and Christoph Bregler, “Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors,” Transactions on Pattern Analysis and Machine Intelligence, 2008.
• SPANISH FOLKS
• CVPR 2010 BEST PAPER
• BRANCH AND BOUND
Saturday, September 4, 2010
CHALLENGESOVERVIEW
• MISSING DATA• BEST K• TRILINEAR OPTIMIZATION
Saturday, September 4, 2010
LINEAR SHAPE MODEL PERSPECTIVE PROJECTION
Saturday, September 4, 2010
LINEAR SHAPE MODEL MAXIMUM LIKELIHOOD SOLUTION
Saturday, September 4, 2010
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches
Shape Basis3D points at each time instant lie in a low dimensional subspace
Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace
DYNAMIC STRUCTURE
Shape
Traj
ecto
ry
DYNAMIC STRUCTURE
Shape
Shape Representation
LINEAR SHAPE MODEL
DYNAMIC STRUCTURE
Traj
ecto
ry
Trajectory Representation
LINEAR TRAJECTORY MODEL
X-co
ordi
nate
of
traj
ecto
ry o
f han
d
…+ + +a1 a2 ak=
time
DYNAMIC STRUCTURE
Traj
ecto
ry
Trajectory Representation
LINEAR TRAJECTORY MODEL
X-co
ordi
nate
of
traj
ecto
ry o
f han
d
time
kth trajectory basis vector
Trajectory of jth point (X-component only)
Trajectory Coefficient Contribution of kth basis in the trajectory of jth point
TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE
X-co
ordi
nate
of
traj
ecto
ry
…+ + +a1 a2 ak=
time
TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE X-component of trajectory
of jth point as linear combination of K basis trajectories
X-component of trajectory of all point as linear combination of K basis trajectories
X-component of trajectory of all points
X, Y and Z-components of trajectory of all points
TRAJECTORY REPRESENTATION
SStructure
£Basis
ACoefficents
of Dynamic Structure
TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection
W RS
R1R2
R3� � �
RF
TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection
W R
R1R2
R3� � �
RF£
A
Structure S, in trajectory subspace represented by K trajectory basis
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
W £A= R
W B= R
SHAPE FACTORIZATION
TRAJECTORY FACTORIZATION
Weights
Traj basis
Shape basis
Weights
DUALITYWeights and Bases
Shape weights are trajectory basis and trajectory weights are shape basis
����
�
�
����
�
�
�
FPFPFPFFF
PPP
PPP
PF
ZYXZYX
ZYXZYXZYXZYX
�
�
�
�
111
222212121
111111111
*3S
� rank of columns = rank of rows� Shape model and trajectory model has equal compaction
power
������PPP ZYX� 11YY1XX ��
��������
PPP ZYX� 22YY2�
�FPFPFP� �FPFPFP ZYFX F�����
�
YX Z
� ZYX 1111YY11XXZYX 2121YY21
� FFF 111 FFF ZYFX F 111
DUALITYWeights and Bases
����
�
�
����
�
�
�
FPFPFPFFF
PPP
PPP
PF
ZYXZYX
ZYXZYXZYXZYX
�
�
�
�
111
222212121
111111111
*3S
PROOF OF DUALITYWeights and Bases
Consider rearranged structure matrix SS*
PROOF OF DUALITYWeights and Bases
Consider rearranged structure matrix SS*
where
1st shape basis
PROOF OF DUALITYWeights and Bases
To link shape to jjth trajectory, we select the coefficients related to jth point
PROOF OF DUALITYWeights and Bases
Can be rewritten as Compare to Trajectory Representation
ILLUSTRATION OF DUALITYSVD Shape and Trajectory Basis for Mocap Structure
Shape Coefficients ��������������� Trajectory Coefficients ������������
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
1. Deformation constrained by physical actuation2. Trajectories vary smoothly and not randomly3. Can be compactly represented by predefined basis
e.g. Discrete Cosine Transform
Object Independent Basision
DCT BASIS
PREDEFINING TRAJECTORY BASIS� We showed that PCA approaches DCT (Discrete
Cosine Transform) on CMU’s body MOCAP database.
67
21st – 26thbasis
1st – 6thbasis
41st – 46thbasis
PCA DCT
COMPACTNESS OF DCT BASIS
68
DCT RECONSTRUCTION
35 Basis 50 Basis 65 Basis
69
SA \� AS �ˆ
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
�
WA
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
�
1. By SVD, compute2. Correct solution differs by a linear transform
3. Solving for QQ ?
Solution
FINDING QThe correct Q will yield the correct form of �
We can just estimate first 3 columns of Q instead of estimating full Q
If Q|||is known:� Compute R� Compute �� Compute A
FINDING QIIIThe correct Q will yield the correct form of �
Orthonormality Constraints
Each image yields 3 constraints because � is known
F images yield 3F constraintsAt least 3K images needed to constrain the solution
RESULTS
Traj Basis
Xiao0
2
4
6
8
10
12
QUANTITATIVE RESULTSStructure Error Rotation Error
Traj BasisTorresani
Xiao0
0.2
0.4
0.6
0.8
1
1.2
1.4
Traj Basis
Torresani
Xiao
We use synthetic and Motion captured data for quantitative experiments
MOTION CAPTURE DATASETS
Input Video Two views of the reconstruction
DANCE DATASET75 points, 264 frames, K=5
Torresani et al. 2005 Xiao et al. 2004
MOTION CAPTURE DATASETS
Input Data Two views of the reconstruction
STRETCH DATASET41 points, 370 frames, K=12
MOTION CAPTURE DATASETS
Input Data Two views of the reconstruction
PICKUP DATASET41 points, 357 frames, K=12
Two views of the reconstruction
CUBES SEQUENCES14 points, 200 frames, K=2
RESULTS ON REAL VIDEOS
Two views of the reconstruction
MATRIX SEQUENCE30 points, 93 frames, K=3
RESULTS ON REAL VIDEOS
Two views of the reconstruction
PIE DATASET68 points, 240 frames, K=2
RESULTS ON REAL VIDEOS
Two views of the reconstruction
DINOSAUR SEQUENCE49 points, 231 frames, K=12
RESULTS ON REAL VIDEOS
AS CAMERA MOTION INCREASESAS OBJECT MOTION DECREASES
RECONSTRUCTION STABILITY INCREASES
SHAPE MODEL VS. TRAJECTORY MODELShape Trajectory
Model Can be learnt Hard to specialize
Specificity Object dependent Generalize
Ordering of frames Irrelevant Exploited
Ordering of points Exploited Irrelevant
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand Limitations
Shape Estimation
Tuesday, September 7, 2010
3D TRAJECTORY ESTIMATIONECCV 2010
Tuesday, September 7, 2010
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
Tuesday, September 7, 2010
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
]
Tuesday, September 7, 2010
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
Tuesday, September 7, 2010
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
QiXi = −qi
Tuesday, September 7, 2010
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
QiXi = −qi2x3 3x1 2x1
Tuesday, September 7, 2010
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
Tuesday, September 7, 2010
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
Tuesday, September 7, 2010
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
IDEA: ESTIMATE CAMERA FROM RIGID PART
Tuesday, September 7, 2010
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
RECONSTRUCTION EVENTS
7Trajectory Reconstruction
Tuesday, September 7, 2010
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
Tuesday, September 7, 2010
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
Tuesday, September 7, 2010
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
2Fx3F 3Fx1 2Fx1
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
2Fx3F 3Fx1 2Fx1
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
QX = q
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
Tuesday, September 7, 2010
QX = q
=
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
Tuesday, September 7, 2010
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦k×1
= Θβ
Trajectory Reconstruction
35 Basis 50 Basis 65 Basis
9Tuesday, September 7, 2010
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦k×1
= Θβ
Trajectory Reconstruction
35 Basis 50 Basis 65 Basis
9Tuesday, September 7, 2010
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×3k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦3k×1
= Θβ
X = Θβ
=
Tuesday, September 7, 2010
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
Tuesday, September 7, 2010
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
Tuesday, September 7, 2010
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3
Multiple ViewReconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
Tuesday, September 7, 2010
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3
Multiple ViewReconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
ΘLinear
Transform
R2
R2
R2
R3k
Trajectory Reconstruction
Tuesday, September 7, 2010
LINEAR SOLUTION
=
QX = q
2Fx3F3Fx1 2Fx1
QΘβ = q
=
2Fx13Fx3k 3kx12Fx3F
Tuesday, September 7, 2010
LINEAR SOLUTION
=
QX = q
2Fx3F3Fx1 2Fx1
βkβ2β1 …+ + +=
X = Θβ
=
3Fx3k3Fx1 3kx1
QΘβ = q
=
2Fx13Fx3k 3kx12Fx3F
Tuesday, September 7, 2010
ALGORITHM
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS
Tuesday, September 7, 2010
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS
• BUNDLE ADJUSTMENT
Tuesday, September 7, 2010
MISSING DATA
14
QΘβ = q
=2Fx1
3Fx3k
3kx12Fx3F
Tuesday, September 7, 2010
MISSING DATA
14
QΘβ = q
=2Fx1
3Fx3k
3kx12Fx3F
Tuesday, September 7, 2010
PARK ET AL., ECCV 20103D Reconstruction of a Moving Point from a Series of 2D Projections
Tuesday, September 7, 2010
16Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand Limitations
Shape Estimation
Tuesday, September 7, 2010
AMBIGUITY
20Tuesday, September 7, 2010
AMBIGUITY
20
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1
Tuesday, September 7, 2010
AMBIGUITY
20
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) =
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
THEOREM 2: limη→inf
β = β̂
±1
Tuesday, September 7, 2010
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1
X = aC+ b
C
CORRELATED X and C
Tuesday, September 7, 2010
HYPERPLANE OF SOLUTIONS
Single View Reconstruction
Trajectory Reconstruction
X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C
C
X X
C
X̂ X̂
SINGLE VIEW 3D RECONSTRUCTION
MULTIPLE VIEWDYNAMIC 3D
RECONSTRUCTION
Tuesday, September 7, 2010
HYPERPLANE OF SOLUTIONS
Single View Reconstruction
Trajectory Reconstruction
X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C
C
X X
C
X̂ X̂
SINGLE VIEW 3D RECONSTRUCTION
MULTIPLE VIEWDYNAMIC 3D
RECONSTRUCTION
Tuesday, September 7, 2010
QΘβ = qR3F
GEOMETRY OF C AND X
Trajectory Reconstruction
Tuesday, September 7, 2010
QΘβ = q
C
R3F
GEOMETRY OF C AND X
Trajectory Reconstruction
C
Tuesday, September 7, 2010
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
Θ
Θ⊥
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
Θ
Θ⊥
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
X̂
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
X̂
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
e
X̂
Tuesday, September 7, 2010
X = aC
23
X
C
X̂
Θ
Θ⊥
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
X = aC
23
X
C
X̂
Θ
Θ⊥
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
Θ
Θ⊥
X = aC
23
X
C
X̂
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
Θ
Θ⊥
X = aC
23
X
C
X̂
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
= ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
η ∝ = 1‖Θ⊥β⊥X‖HOW WELL THE BASIS DESCRIBES X= ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂C
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂Θ⊥β⊥CC
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂
Θ⊥β⊥X
Θ⊥β⊥CC
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
X̂ = AX+ (1−A)C
X
C
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂Θ⊥β⊥C
Θ⊥β⊥X
Tuesday, September 7, 2010
WHAT DOES THIS TELL US?
• DE-CORRELATE CAMERA AND OBJECT MOTION
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
Tuesday, September 7, 2010
PARK ET AL. 2010IN PERSPECTIVE
• PROBLEMS SOLVED:• PERSPECTIVE RECONSTRUCTION• HANDLES MISSING DATA• LINEAR SOLVE (FAST, GLOBAL OPTIMUM)
• OPEN PROBLEMS:• HANDLING SMOOTHLY MOVING CAMERAS• AUTOMATIC COMPUTATION OF K• EXPLOITING DEPENDENCIES BETWEEN TRAJECTORIES• “PHYSICS-AWARE” ESTIMATION
Tuesday, September 7, 2010
http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html [email protected]
37
COLLABORATORS
Ijaz AkhterHyun Soo Park
Eakta JainMoshe Mahler
Takaaki ShiratoriIain MatthewsJessica HodginsTakeo Kanade
Tuesday, September 7, 2010