+ All Categories
Home > Documents > ECCV 2010 TUTORIAL NONRIGID STRUCTURE FROM OTIONyaser/TutorialSlides-Integrated-small.pdf ·...

ECCV 2010 TUTORIAL NONRIGID STRUCTURE FROM OTIONyaser/TutorialSlides-Integrated-small.pdf ·...

Date post: 22-Oct-2020
Category:
Upload: others
View: 3 times
Download: 0 times
Share this document with a friend
190
ECCV 2010 TUTORIAL NONRIGID STRUCTURE FROM MOTION YASER SHEIKH The Robotics Institute Carnegie Mellon University Pittsburgh, USA http://cs.cmu.edu/~yaser SOHAIB KHAN Computer Vision Lab LUMS School of Science & Engineering Lahore, PAKISTAN http://web.lums.edu.pk/~sohaib http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html
Transcript
  • ECCV 2010 TUTORIAL

    NONRIGID STRUCTURE FROM MOTION

    YASER SHEIKHThe Robotics InstituteCarnegie Mellon UniversityPittsburgh, USAhttp://cs.cmu.edu/~yaser

    SOHAIB KHANComputer Vision LabLUMS School of Science & Engineering Lahore, PAKISTANhttp://web.lums.edu.pk/~sohaib

    http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE

    x

    z

    3D Structure That Deforms Over Time

    4D DYNAMIC STRUCTURE

  • IMAGE MOTION

    OBJECT MOTION CAMERA MOTION

  • IMAGE MOTION

    OBJECT MOTION CAMERA MOTIONAND

  • RIGID STRUCTURE FROM MOTION

  • NONRIGID STRUCTURE FROM MOTION

  • NONRIGID STRUCTURE FROM MOTION

  • NONRIGID STRUCTURE FROM MOTION

  • NONRIGID STRUCTURE FROM MOTION

    ONLY ONE VIEW PER 3D CONFIGURATON: ILL-POSED PROBLEM EQUIVALENT TO FINDING 3D FROM SINGLE IMAGE

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    ASSUMPTIONS� Orthographic Camera� At least 3 images� Rigid Scene� Camera Motion� Corresponding points available

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    NOTATION� P 3D points seen in F frames

    � Xj = [Xj, Yj, Zj] is jth 3D point 1��j ��P

    � xij = [xij, yij] is the projection of Xj in ith frame 1��i ��F

    � Pi is the camera projection matrix if the ith frame 1��i ��F

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    orthographicprojection

    matrix2D image

    point

    3D scenepoint

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation

    2 rows of a 3D rotation

    matrix

    imageoffset

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation

    2 rows of a 3D rotation

    matrix

    imageoffset

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    PROJECTION OF PP 3D POINTS IN ith IMAGE

    PROJECTION OF P 3D POINTS IN F IMAGES

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    PROJECTION OF PP 3D POINTS IN F IMAGES

    W R

    S

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    Image Observations Matrix, W

    ×

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    Image Observations Matrix, W

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

    HOW TO SOLVE FOR Q� Observation: The correct Q will result in an R whose rows

    are pair-wise orthonormal

    � The ith image results in the following 3 constraints on Q

    � Total 3F constraints on 6 terms of QQT

    � Can be solved linearly for G = QQT for F ����

    ORTHONORMALITYCONSTRAINTS

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

  • FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992

  • NONRIGID STRUCTURE3D Structure That Deforms Over Time

    RIGID STRUCTURE

  • NONRIGID STRUCTURE3D Structure That Deforms Over Time

    RIGID STRUCTURE NONRIGID STRUCTURE

  • NONRIGID STRUCTURE3D Structure That Deforms Over Time

    RIGID STRUCTURE NONRIGID STRUCTURE

  • NONRIGID STRUCTURE FROM MOTIONComparison with Rigid Structure from Motion

    RIGID SFM NONRIGID SFM

    W R

    S

    W RS

    S(1)

    S(2)

    S(F)

    S(3)...

    R1R2

    R3� � �

    RF

  • NONRIGID STRUCTURE FROM MOTION

    RIGID SFM� Inputs:

    100 pts x 40 sec x 30 fps x 2 (x,y)= 240,000 observations

    � Unknowns:100 points x 3 (X,Y,Z) = 300 unknowns

    NONRIGID SFM

    Explosion of Unknowns

    Example: Given a 40 second video with 100 tracked points

    � Inputs:100 pts x 40 sec x 30 fps x 2= 240,000 observations

    � Unknowns:100 points x 40 sec x 30 fps x 3 = 360,000 unknowns

  • NONRIGID STRUCTURE FROM MOTIONExplosion of Unknowns

    IN GENERAL, NRSFM HAS MORE UNKNOWNS THAN CONSTRAINTS

    ILL-POSED PROBLEM: Additional assumptions are necessary to constrain the solution.

    HOWEVER…Motion is not random: 3D points are often highly correlated in space and timePoints move because an actuator exerts force on them

    Hence their acceleration is limited by the actuating forceTherefore, shape does not deform arbitrarily over time

    4D STRUCTURE OFTEN LIES IN A LOW DIMENSIONAL SUBSPACE

  • NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches

    Shape Basis3D points at each time instant lie in a low dimensional subspace

    Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace

  • EXAMPLES OF APPLICATIONSMatch Moving in Movies

    Akhter et al. NIPS 2008

  • EXAMPLES OF APPLICATIONSMotion-Capture

    Input Video Two views of the reconstruction

    Akhter et al. NIPS 2008

  • EXAMPLES OF APPLICATIONSMotion-Capture Cleanup

    ReconstructionUnlabeled DataVideoInput Output

    Disney Research, Pittsburgh

  • EXAMPLES OF APPLICATIONSTracking in 2D and 3D

    Credit: Iain Matthews

  • EXAMPLES OF APPLICATIONSAnimation

    Jain et al. SCA 2010

  • EXAMPLES OF APPLICATIONSBrowsing Image Collections

    Credit: Hyun Soo Park

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • time

    spac

    e

    DYNAMIC STRUCTURE

    S3F×P =

    ⎡⎢⎢⎢⎣

    X11 X12 · · · X1PX21 X22 · · · X2P...

    ......

    XF1 XF2 · · · XFP

    ⎤⎥⎥⎥⎦

    Saturday, September 4, 2010

  • DYNAMIC STRUCTUREUNDER ORTHOGRAPHIC PROJECTION

    ⎡⎢⎢⎢⎣

    x11 · · · x1Px21 x2P...

    ...xF1 · · · xFP

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    X11 · · · X1PX21 X2P...

    ...XF1 · · · XFP

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦=

    W = RX

    Saturday, September 4, 2010

  • LINEAR SHAPE MODEL[T. Cootes et al. 91, Bregler et al. 97]

    w1 × + w2 × + … + wK × = α1 α2 αk

    ⎡⎢⎢⎢⎣

    X11 · · · X1PX21 X2P...

    ...XF1 · · · XFP

    ⎤⎥⎥⎥⎦

    Saturday, September 4, 2010

  • ⎡⎢⎢⎢⎣

    X11 · · · X1PX21 X2P...

    ...XF1 · · · XFP

    ⎤⎥⎥⎥⎦ =

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    LINEAR SHAPE MODEL

    5Saturday, September 4, 2010

  • LINEAR SHAPE MODEL RECONSTRUCTION

    5 Basis 15 Basis 25 Basis

    6Saturday, September 4, 2010

  • LINEAR SHAPE MODEL UNDER ORTHOGRAPHIC PROJECTION

    ⎡⎢⎢⎢⎣

    x11 · · · x1Px21 x2P...

    ...xF1 · · · xFP

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    X11 · · · X1PX21 X2P...

    ...XF1 · · · XFP

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦=

    =

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    2F x P 2F x 3F (6F) 3F x P

    2F x 3F (6F) 3F x 3k 3k x P

    Saturday, September 4, 2010

  • KNOWNS VS UNKNOWNS

    2F × P6F + (3F × k) + (k × P )

    2F × P ≥ 6F + (3F × k) + (k × P )

    KNOWNS:UNKNOWNS:

    Saturday, September 4, 2010

  • LINEAR SHAPE MODEL

    w1 × + w2 × + … + wK × = α1 α2 αk

    Y = R(

    K∑i=1

    ωibi) +T

    RIGID COMPONENT

    NONRIGID COMPONENT

    IDEA: RIGID COMPONENT GETS FOLDED INTO PROJECTION

    Saturday, September 4, 2010

  • CHALLENGETRILINEAR ESTIMATION

    W = RΩB

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    Saturday, September 4, 2010

  • BREGLER et al. 2000Nested SVD

    ⎡⎢⎢⎢⎣

    x11 · · · x1Px21 x2P...

    ...xF1 · · · xFP

    ⎤⎥⎥⎥⎦=

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11R1 · · · ω1kR1ω21R2 ω2kR2

    ......

    ωF1RF · · · ωFkRF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦=

    2F x 3k 3k x P

    Saturday, September 4, 2010

  • BREGLER et al. 2000Outer SVD

    ⎡⎢⎢⎢⎣

    x11 · · · x1Px21 x2P...

    ...xF1 · · · xFP

    ⎤⎥⎥⎥⎦=

    ⎡⎢⎢⎢⎣

    ω11R1 · · · ω1kR1ω21R2 ω2kR2

    ......

    ωF1RF · · · ωFkRF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    2F x 3k 3k x P

    SVD

    W H B=

    W = UDVT

    W = (UD12 )(D

    12VT )

    W = ĤB̂

    Saturday, September 4, 2010

  • BREGLER et al. 2000Inner SVD

    SVD

    W = ĤB̂⎡⎢⎢⎢⎣

    ω11R1 · · · ω1kR1ω21R2 ω2kR2

    ......

    ωF1RF · · · ωFkR1

    ⎤⎥⎥⎥⎦H

    [ω11r

    11 ω11r

    21 ω11r

    31 · · · ω1kr11 ω1kr21 ω1kr31

    ω11r41 ω11r

    51 ω11r

    61 · · · ω1kr41 ω1kr51 ω1kr61

    ]

    ⎡⎢⎢⎢⎣

    ω11r11 ω11r

    21 ω11r

    31 ω11r

    41 ω11r

    51 ω11r

    61

    ω12r11 ω12r

    21 ω12r

    31 ω12r

    41 ω12r

    51 ω12r

    61

    ......

    ω1kr11 ω1kr

    21 ω1kr

    31 ω1kr

    41 ω1kr

    51 ω1kr

    61

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11ω12...

    ω1k

    ⎤⎥⎥⎥⎦[r11 r

    21 r

    31 r

    41 r

    51 r

    61

    ]

    rank 1

    =

    =h1

    h′1 =

    h′1 = udvT = ω̂r̂

    =

    METRIC RECTIFICATION USING ORTHONORMALITY CONSTRAINTS

    Saturday, September 4, 2010

  • BREGLER et al. 2000OVERVIEW

    • OUTER SVD: PERFORM SVD ON W TO GET ESTIMATES OF:• H: CAMERA PROJECTIONS AND COEFFICIENTS

    • INNER SVD: PERFORM SVD ON H TO GET ESTIMATES OF:• OMEGA: COEFFICIENTS• R: CAMERA PROJECTIONS

    • METRIC RECTIFY USING ORTHONORMALITY CONSTRAINTS• B: THE SHAPE BASIS

    Saturday, September 4, 2010

  • RESULTS

    Saturday, September 4, 2010

  • BREGLER et al. 2000IN PERSPECTIVE

    • SEMINAL WORK: SHOWED THAT FACTORIZATION METHODS CAN BE APPLIED TO NONRIGID OBJECTS

    • CASCADING ERROR: ANY OUTER SVD ESTIMATION ERROR CASCADES INTO INNER SVD ESTIMATION

    • AMBIGUITY ERROR: ESTIMATION OF METRIC RECTIFICATION• NUMBER OF BASIS: LARGE NUMBER OF BASIS REQUIRED• MISSING DATA: NEEDS COMPLETE W MATRIX

    Saturday, September 4, 2010

  • METRIC RECTIFICATIONAMBIGUITY

    W = ĤB̂

    W = ĤGG−1B̂

    H = ĤG B = G−1B̂

    ⎡⎢⎢⎢⎣

    ω11R1 · · · ω1kR1ω21R2 ω2kR2

    ......

    ωF1RF · · · ωFkR1

    ⎤⎥⎥⎥⎦H ==

    ⎡⎣ Ĥ

    ⎤⎦⎡⎣ | | |g1 g2 . . . gk

    | | |

    ⎤⎦

    G3k X 3k

    Saturday, September 4, 2010

  • METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT

    ⎡⎢⎢⎢⎣

    ω11R1 · · · ω1kR1ω21R2 ω2kR2

    ......

    ωF1RF · · · ωFkR1

    ⎤⎥⎥⎥⎦H ==

    ⎡⎣ Ĥ

    ⎤⎦⎡⎣ | | |g1 g2 . . . gk

    | | |

    ⎤⎦

    ⎡⎣ Ĥ

    ⎤⎦⎡⎣ |gk

    |

    ⎤⎦ =

    ⎡⎢⎢⎢⎣

    ω1kR1ω2kR2

    ...ωFkRF

    ⎤⎥⎥⎥⎦

    ωikRi = Ĥ2i−1:2igk

    RiRTi = I

    ORTHONORMALITY CONSTRAINT

    H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI

    Saturday, September 4, 2010

  • METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT

    RiRTi = I

    ORTHONORMALITY CONSTRAINT

    H2igkgTk Ĥ2i−1 = 0

    H2igkgTk Ĥ2i = ω

    2ikH2i−1gkg

    Tk Ĥ2i−1 = ω

    2ik

    H2i−1gkgTk Ĥ2i−1 = H2igkgTk Ĥ2i

    H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI =

    [ω2ik 00 ω2ik

    ]

    Saturday, September 4, 2010

  • CHALLENGE?AMBIGUITY

    Saturday, September 4, 2010

  • OPTIMIZATION

    Saturday, September 4, 2010

  • CHALLENGEMISSING DATA

    • A.M. Buchanan and A.W. Fitzgibbon, “Damped Newton Algorithms for Matrix Factorization with Missing Data,” IEEE International Conference on Computer Vision and Pattern Recognition, 2005.

    • L. Torresani, A. Hertzmann, and Christoph Bregler, “Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors,” Transactions on Pattern Analysis and Machine Intelligence, 2008.

    • SPANISH FOLKS

    • CVPR 2010 BEST PAPER

    • BRANCH AND BOUND

    Saturday, September 4, 2010

  • CHALLENGESOVERVIEW

    • MISSING DATA• BEST K• TRILINEAR OPTIMIZATION

    Saturday, September 4, 2010

  • LINEAR SHAPE MODEL PERSPECTIVE PROJECTION

    Saturday, September 4, 2010

  • LINEAR SHAPE MODEL MAXIMUM LIKELIHOOD SOLUTION

    Saturday, September 4, 2010

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches

    Shape Basis3D points at each time instant lie in a low dimensional subspace

    Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace

  • DYNAMIC STRUCTURE

    Shape

    Traj

    ecto

    ry

  • DYNAMIC STRUCTURE

    Shape

    Shape Representation

    LINEAR SHAPE MODEL

  • DYNAMIC STRUCTURE

    Traj

    ecto

    ry

    Trajectory Representation

    LINEAR TRAJECTORY MODEL

    X-co

    ordi

    nate

    of

    traj

    ecto

    ry o

    f han

    d

    …+ + +a1 a2 ak=

    time

  • DYNAMIC STRUCTURE

    Traj

    ecto

    ry

    Trajectory Representation

    LINEAR TRAJECTORY MODEL

    X-co

    ordi

    nate

    of

    traj

    ecto

    ry o

    f han

    d

    time

    kth trajectory basis vector

    Trajectory of jth point (X-component only)

    Trajectory Coefficient Contribution of kth basis in the trajectory of jth point

  • TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE

    X-co

    ordi

    nate

    of

    traj

    ecto

    ry

    …+ + +a1 a2 ak=

    time

  • TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE X-component of trajectory

    of jth point as linear combination of K basis trajectories

    X-component of trajectory of all point as linear combination of K basis trajectories

  • X-component of trajectory of all points

    X, Y and Z-components of trajectory of all points

  • TRAJECTORY REPRESENTATION

    SStructure

    £Basis

    ACoefficents

    of Dynamic Structure

  • TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection

    W RS

    R1R2

    R3� � �

    RF

  • TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection

    W R

    R1R2

    R3� � �

    RF£

    A

    Structure S, in trajectory subspace represented by K trajectory basis

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • W £A= R

    W B= R

    SHAPE FACTORIZATION

    TRAJECTORY FACTORIZATION

    Weights

    Traj basis

    Shape basis

    Weights

    DUALITYWeights and Bases

    Shape weights are trajectory basis and trajectory weights are shape basis

  • ����

    ����

    FPFPFPFFF

    PPP

    PPP

    PF

    ZYXZYX

    ZYXZYXZYXZYX

    111

    222212121

    111111111

    *3S

    � rank of columns = rank of rows� Shape model and trajectory model has equal compaction

    power

    ������PPP ZYX� 11YY1XX ��

    ��������

    PPP ZYX� 22YY2�

    �FPFPFP� �FPFPFP ZYFX F�����

    YX Z

    � ZYX 1111YY11XXZYX 2121YY21

    � FFF 111 FFF ZYFX F 111

    DUALITYWeights and Bases

  • ����

    ����

    FPFPFPFFF

    PPP

    PPP

    PF

    ZYXZYX

    ZYXZYXZYXZYX

    111

    222212121

    111111111

    *3S

    PROOF OF DUALITYWeights and Bases

    Consider rearranged structure matrix SS*

  • PROOF OF DUALITYWeights and Bases

    Consider rearranged structure matrix SS*

    where

    1st shape basis

  • PROOF OF DUALITYWeights and Bases

    To link shape to jjth trajectory, we select the coefficients related to jth point

  • PROOF OF DUALITYWeights and Bases

    Can be rewritten as Compare to Trajectory Representation

  • ILLUSTRATION OF DUALITYSVD Shape and Trajectory Basis for Mocap Structure

    Shape Coefficients ��������������� Trajectory Coefficients ������������

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • W R£

    A

    ESTIMATING STRUCTURE VIA TRAJECTORY MODEL

  • W R£

    A

    ESTIMATING STRUCTURE VIA TRAJECTORY MODEL

    1. Deformation constrained by physical actuation2. Trajectories vary smoothly and not randomly3. Can be compactly represented by predefined basis

    e.g. Discrete Cosine Transform

    Object Independent Basision

  • DCT BASIS

  • PREDEFINING TRAJECTORY BASIS� We showed that PCA approaches DCT (Discrete

    Cosine Transform) on CMU’s body MOCAP database.

    67

    21st – 26thbasis

    1st – 6thbasis

    41st – 46thbasis

    PCA DCT

  • COMPACTNESS OF DCT BASIS

    68

  • DCT RECONSTRUCTION

    35 Basis 50 Basis 65 Basis

    69

    SA \� AS �ˆ

  • W R£

    A

    ESTIMATING STRUCTURE VIA TRAJECTORY MODEL

  • WA

    ESTIMATING STRUCTURE VIA TRAJECTORY MODEL

    1. By SVD, compute2. Correct solution differs by a linear transform

    3. Solving for QQ ?

    Solution

  • FINDING QThe correct Q will yield the correct form of �

    We can just estimate first 3 columns of Q instead of estimating full Q

    If Q|||is known:� Compute R� Compute �� Compute A

  • FINDING QIIIThe correct Q will yield the correct form of �

    Orthonormality Constraints

    Each image yields 3 constraints because � is known

    F images yield 3F constraintsAt least 3K images needed to constrain the solution

  • RESULTS

  • Traj Basis

    Xiao0

    2

    4

    6

    8

    10

    12

    QUANTITATIVE RESULTSStructure Error Rotation Error

    Traj BasisTorresani

    Xiao0

    0.2

    0.4

    0.6

    0.8

    1

    1.2

    1.4

    Traj Basis

    Torresani

    Xiao

    We use synthetic and Motion captured data for quantitative experiments

  • MOTION CAPTURE DATASETS

    Input Video Two views of the reconstruction

    DANCE DATASET75 points, 264 frames, K=5

  • Torresani et al. 2005 Xiao et al. 2004

  • MOTION CAPTURE DATASETS

    Input Data Two views of the reconstruction

    STRETCH DATASET41 points, 370 frames, K=12

  • MOTION CAPTURE DATASETS

    Input Data Two views of the reconstruction

    PICKUP DATASET41 points, 357 frames, K=12

  • Two views of the reconstruction

    CUBES SEQUENCES14 points, 200 frames, K=2

    RESULTS ON REAL VIDEOS

  • Two views of the reconstruction

    MATRIX SEQUENCE30 points, 93 frames, K=3

    RESULTS ON REAL VIDEOS

  • Two views of the reconstruction

    PIE DATASET68 points, 240 frames, K=2

    RESULTS ON REAL VIDEOS

  • Two views of the reconstruction

    DINOSAUR SEQUENCE49 points, 231 frames, K=12

    RESULTS ON REAL VIDEOS

  • AS CAMERA MOTION INCREASESAS OBJECT MOTION DECREASES

    RECONSTRUCTION STABILITY INCREASES

  • SHAPE MODEL VS. TRAJECTORY MODELShape Trajectory

    Model Can be learnt Hard to specialize

    Specificity Object dependent Generalize

    Ordering of frames Irrelevant Exploited

    Ordering of points Exploited Irrelevant

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality

    Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand limitations

    Shape Estimation

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand Limitations

    Shape Estimation

    Tuesday, September 7, 2010

  • 3D TRAJECTORY ESTIMATIONECCV 2010

    Tuesday, September 7, 2010

  • CHALLENGETRILINEAR ESTIMATION

    W = RΩB

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    Tuesday, September 7, 2010

  • xi ∈ R2

    Xi ∈ R3

    SINGLE VIEW RECONSTRUCTION

    4

    Single View Reconstruction

    Tuesday, September 7, 2010

  • xi ∈ R2

    Xi ∈ R3

    SINGLE VIEW RECONSTRUCTION

    4

    Single View Reconstruction

    si

    [xi1

    ]= Pi

    [Xi1

    ]

    Tuesday, September 7, 2010

  • xi ∈ R2

    Xi ∈ R3

    SINGLE VIEW RECONSTRUCTION

    4

    Single View Reconstruction

    si

    [xi1

    ]= Pi

    [Xi1

    ][

    xi1

    ]×Pi

    [Xi1

    ]= 0

    Tuesday, September 7, 2010

  • xi ∈ R2

    Xi ∈ R3

    SINGLE VIEW RECONSTRUCTION

    4

    Single View Reconstruction

    si

    [xi1

    ]= Pi

    [Xi1

    ][

    xi1

    ]×Pi

    [Xi1

    ]= 0

    QiXi = −qi

    Tuesday, September 7, 2010

  • xi ∈ R2

    Xi ∈ R3

    SINGLE VIEW RECONSTRUCTION

    4

    Single View Reconstruction

    si

    [xi1

    ]= Pi

    [Xi1

    ][

    xi1

    ]×Pi

    [Xi1

    ]= 0

    QiXi = −qi2x3 3x1 2x1

    Tuesday, September 7, 2010

  • ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    STRUCTURE FROM MOTION

    5

    R2

    R2

    R2

    R3

    Multiple ViewReconstruction

    Tuesday, September 7, 2010

  • ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    STRUCTURE FROM MOTION

    5

    R2

    R2

    R2

    R3

    Multiple ViewReconstruction

    Tuesday, September 7, 2010

  • ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    STRUCTURE FROM MOTION

    5

    R2

    R2

    R2

    R3

    Multiple ViewReconstruction

    IDEA: ESTIMATE CAMERA FROM RIGID PART

    Tuesday, September 7, 2010

  • CHALLENGETRILINEAR ESTIMATION

    W = RΩB

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    Tuesday, September 7, 2010

  • CHALLENGETRILINEAR ESTIMATION

    W = RΩB

    ⎡⎢⎢⎢⎣

    R1R2

    . . .

    RF

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    ω11 · · · ω1kω21 ω2k...

    ...ωF1 · · · ωFk

    ⎤⎥⎥⎥⎦

    ⎡⎢⎢⎢⎣

    −b1−−b2−

    ...−bk−

    ⎤⎥⎥⎥⎦

    Tuesday, September 7, 2010

  • RECONSTRUCTION EVENTS

    7Trajectory Reconstruction

    Tuesday, September 7, 2010

  • RECONSTRUCTION EVENTS

    7

    R2

    R3

    R2

    R3

    R2

    R3

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • si

    [xi1

    ]= Pi

    [Xi1

    ]

    RECONSTRUCTION EVENTS

    7

    R2

    R3

    R2

    R3

    R2

    R3

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • si

    [xi1

    ]= Pi

    [Xi1

    ]

    RECONSTRUCTION EVENTS

    7

    R2

    R3

    R2

    R3

    R2

    R3

    Trajectory Reconstruction

    ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    si

    [xi1

    ]= Pi

    [X1

    ]

    2Fx3 3x1 2Fx1

    Structure from Motion

    Tuesday, September 7, 2010

  • si

    [xi1

    ]= Pi

    [Xi1

    ]

    RECONSTRUCTION EVENTS

    7

    R2

    R3

    R2

    R3

    R2

    R3

    Trajectory Reconstruction

    ⎡⎢⎣

    Q1. . .

    QF

    ⎤⎥⎦⎡⎢⎣

    X1...

    XF

    ⎤⎥⎦ =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    2Fx3F 3Fx1 2Fx1

    ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    si

    [xi1

    ]= Pi

    [X1

    ]

    2Fx3 3x1 2Fx1

    Structure from Motion

    Tuesday, September 7, 2010

  • si

    [xi1

    ]= Pi

    [Xi1

    ]

    RECONSTRUCTION EVENTS

    7

    R2

    R3

    R2

    R3

    R2

    R3

    Trajectory Reconstruction

    ⎡⎢⎣

    Q1. . .

    QF

    ⎤⎥⎦⎡⎢⎣

    X1...

    XF

    ⎤⎥⎦ =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    2Fx3F 3Fx1 2Fx1

    ⎡⎢⎣

    Q1...

    QF

    ⎤⎥⎦X =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    si

    [xi1

    ]= Pi

    [X1

    ]

    2Fx3 3x1 2Fx1

    Structure from Motion

    Tuesday, September 7, 2010

  • QX = q

    ⎡⎢⎣

    Q1. . .

    QF

    ⎤⎥⎦⎡⎢⎣

    X1...

    XF

    ⎤⎥⎦ =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    Tuesday, September 7, 2010

  • QX = q

    =

    ⎡⎢⎣

    Q1. . .

    QF

    ⎤⎥⎦⎡⎢⎣

    X1...

    XF

    ⎤⎥⎦ =

    ⎡⎢⎣

    q1...qF

    ⎤⎥⎦

    Tuesday, September 7, 2010

  • ⎡⎢⎢⎢⎣

    X11X12...

    X1F

    ⎤⎥⎥⎥⎦3F×1

    =

    ⎡⎣ | | |θ1 θ2 · · · θk

    | | |

    ⎤⎦3F×k

    ⎡⎢⎢⎢⎣

    β1β2...βk

    ⎤⎥⎥⎥⎦k×1

    = Θβ

    Trajectory Reconstruction

    35 Basis 50 Basis 65 Basis

    9Tuesday, September 7, 2010

  • ⎡⎢⎢⎢⎣

    X11X12...

    X1F

    ⎤⎥⎥⎥⎦3F×1

    =

    ⎡⎣ | | |θ1 θ2 · · · θk

    | | |

    ⎤⎦3F×k

    ⎡⎢⎢⎢⎣

    β1β2...βk

    ⎤⎥⎥⎥⎦k×1

    = Θβ

    Trajectory Reconstruction

    35 Basis 50 Basis 65 Basis

    9Tuesday, September 7, 2010

  • ⎡⎢⎢⎢⎣

    X11X12...

    X1F

    ⎤⎥⎥⎥⎦3F×1

    =

    ⎡⎣ | | |θ1 θ2 · · · θk

    | | |

    ⎤⎦3F×3k

    ⎡⎢⎢⎢⎣

    β1β2...βk

    ⎤⎥⎥⎥⎦3k×1

    = Θβ

    X = Θβ

    =

    Tuesday, September 7, 2010

  • Trajectory Reconstruction

    11

    R2

    R3

    Single View Reconstruction

    Tuesday, September 7, 2010

  • Trajectory Reconstruction

    11

    R2

    R3

    Single View Reconstruction

    R2

    R2

    R2

    R3R

    3 R3

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • Trajectory Reconstruction

    11

    R2

    R3

    Single View Reconstruction

    R2

    R2

    R2

    R3

    Multiple ViewReconstruction

    R2

    R2

    R2

    R3R

    3 R3

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • Trajectory Reconstruction

    11

    R2

    R3

    Single View Reconstruction

    R2

    R2

    R2

    R3

    Multiple ViewReconstruction

    R2

    R2

    R2

    R3R

    3 R3

    Trajectory Reconstruction

    ΘLinear

    Transform

    R2

    R2

    R2

    R3k

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • LINEAR SOLUTION

    =

    QX = q

    2Fx3F3Fx1 2Fx1

    QΘβ = q

    =

    2Fx13Fx3k 3kx12Fx3F

    Tuesday, September 7, 2010

  • LINEAR SOLUTION

    =

    QX = q

    2Fx3F3Fx1 2Fx1

    βkβ2β1 …+ + +=

    X = Θβ

    =

    3Fx3k3Fx1 3kx1

    QΘβ = q

    =

    2Fx13Fx3k 3kx12Fx3F

    Tuesday, September 7, 2010

  • ALGORITHM

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT

    CORRESPONDENCES:

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT

    CORRESPONDENCES:

    • CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT

    CORRESPONDENCES:

    • CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS

    Tuesday, September 7, 2010

  • ALGORITHM

    • GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT

    CORRESPONDENCES:

    • CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS

    • BUNDLE ADJUSTMENT

    Tuesday, September 7, 2010

  • MISSING DATA

    14

    QΘβ = q

    =2Fx1

    3Fx3k

    3kx12Fx3F

    Tuesday, September 7, 2010

  • MISSING DATA

    14

    QΘβ = q

    =2Fx1

    3Fx3k

    3kx12Fx3F

    Tuesday, September 7, 2010

  • PARK ET AL., ECCV 20103D Reconstruction of a Moving Point from a Series of 2D Projections

    Tuesday, September 7, 2010

  • 16Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • NONRIGID STRUCTURE FROM MOTION

    Introduction to Nonrigid SFM

    Shape Representation

    Ambiguity of Orthogonality Constraints

    Trajectory Representation

    Shape-Trajectory Duality

    TrajectoryEstimation

    Tutorial Outline

    Reconstructibilityand Limitations

    Shape Estimation

    Tuesday, September 7, 2010

  • AMBIGUITY

    20Tuesday, September 7, 2010

  • AMBIGUITY

    20

    THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1

    Tuesday, September 7, 2010

  • AMBIGUITY

    20

    THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) =

    η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖

    THEOREM 2: limη→inf

    β = β̂

    ±1

    Tuesday, September 7, 2010

  • THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1

    X = aC+ b

    C

    CORRELATED X and C

    Tuesday, September 7, 2010

  • HYPERPLANE OF SOLUTIONS

    Single View Reconstruction

    Trajectory Reconstruction

    X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C

    C

    X X

    C

    X̂ X̂

    SINGLE VIEW 3D RECONSTRUCTION

    MULTIPLE VIEWDYNAMIC 3D

    RECONSTRUCTION

    Tuesday, September 7, 2010

  • HYPERPLANE OF SOLUTIONS

    Single View Reconstruction

    Trajectory Reconstruction

    X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C

    C

    X X

    C

    X̂ X̂

    SINGLE VIEW 3D RECONSTRUCTION

    MULTIPLE VIEWDYNAMIC 3D

    RECONSTRUCTION

    Tuesday, September 7, 2010

  • QΘβ = qR3F

    GEOMETRY OF C AND X

    Trajectory Reconstruction

    Tuesday, September 7, 2010

  • QΘβ = q

    C

    R3F

    GEOMETRY OF C AND X

    Trajectory Reconstruction

    C

    Tuesday, September 7, 2010

  • QΘβ = q

    X

    C

    R3F

    GEOMETRY OF C AND X

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    QΘβ = q

    X

    C

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    QΘβ = q

    X

    C

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    QΘβ = q X̂ = AX+ (1−A)C

    X

    C

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    QΘβ = q X̂ = AX+ (1−A)C

    X

    C

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    QΘβ = q X̂ = AX+ (1−A)C

    X

    C

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    Trajectory Reconstruction

    X

    C

    e

    Tuesday, September 7, 2010

  • X = aC

    23

    X

    C

    Θ

    Θ⊥

    R3F

    THEOREM 1: CORRELATED C AND X

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • X = aC

    23

    X

    C

    Θ

    Θ⊥

    R3F

    THEOREM 1: CORRELATED C AND X

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    X = aC

    23

    X

    C

    R3F

    THEOREM 1: CORRELATED C AND X

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • Θ

    Θ⊥

    X = aC

    23

    X

    C

    R3F

    THEOREM 1: CORRELATED C AND X

    Trajectory Reconstruction

    X

    C

    Tuesday, September 7, 2010

  • η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖

    26

    THEOREM 2: limη→inf

    β = β̂

    RECONSTRUCTIBILITY

    Tuesday, September 7, 2010

  • η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖

    26

    THEOREM 2: limη→inf

    β = β̂

    = ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C

    RECONSTRUCTIBILITY

    Tuesday, September 7, 2010

  • η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖

    26

    THEOREM 2: limη→inf

    β = β̂

    η ∝ = 1‖Θ⊥β⊥X‖HOW WELL THE BASIS DESCRIBES X= ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C

    RECONSTRUCTIBILITY

    Tuesday, September 7, 2010

  • X

    Θ

    Θ⊥

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    e

    Trajectory Reconstruction

    X

    C

    X̂C

    X̂ = AX+ (1−A)C

    Tuesday, September 7, 2010

  • X

    Θ

    Θ⊥

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    e

    Trajectory Reconstruction

    X

    C

    X̂Θ⊥β⊥CC

    X̂ = AX+ (1−A)C

    Tuesday, September 7, 2010

  • X

    Θ

    Θ⊥

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    e

    Trajectory Reconstruction

    X

    C

    Θ⊥β⊥X

    Θ⊥β⊥CC

    X̂ = AX+ (1−A)C

    Tuesday, September 7, 2010

  • X̂ = AX+ (1−A)C

    X

    C

    Θ

    Θ⊥

    R3F

    GEOMETRY OF C AND X

    βkβ2β1 …+ + +=

    e

    Trajectory Reconstruction

    X

    C

    X̂Θ⊥β⊥C

    Θ⊥β⊥X

    Tuesday, September 7, 2010

  • WHAT DOES THIS TELL US?

    • DE-CORRELATE CAMERA AND OBJECT MOTION

    Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • Tuesday, September 7, 2010

  • PARK ET AL. 2010IN PERSPECTIVE

    • PROBLEMS SOLVED:• PERSPECTIVE RECONSTRUCTION• HANDLES MISSING DATA• LINEAR SOLVE (FAST, GLOBAL OPTIMUM)

    • OPEN PROBLEMS:• HANDLING SMOOTHLY MOVING CAMERAS• AUTOMATIC COMPUTATION OF K• EXPLOITING DEPENDENCIES BETWEEN TRAJECTORIES• “PHYSICS-AWARE” ESTIMATION

    Tuesday, September 7, 2010

  • http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html [email protected]

    [email protected]

    37

    COLLABORATORS

    Ijaz AkhterHyun Soo Park

    Eakta JainMoshe Mahler

    Takaaki ShiratoriIain MatthewsJessica HodginsTakeo Kanade

    Tuesday, September 7, 2010


Recommended