Space Time Tracking Space Time Tracking ECCV 2002ECCV 2002

Lorenzo TorresaniLorenzo Torresani

Christoph BreglerChristoph Bregler

OutlineOutline

ProblemProblemBackgroundBackgroundStructure from MotionStructure from MotionMatrix DecompositionMatrix DecompositionNon-Rigid Motion EstimationNon-Rigid Motion EstimationNon-Rigid Shapes EstimationNon-Rigid Shapes EstimationResultsResults

ProblemProblem

““To track feature points on non-rigid objects To track feature points on non-rigid objects without using any prior model”without using any prior model”

Rank of a MatrixRank of a Matrix

N

M

Rank (A) = Number of linearly independent vectors in

N

M

Rank (B) = Number of linearly independent vectors in

For M x N the Rank of A ≤ min (M,N)

Rank of a Matrix Rank of a Matrix cont’dcont’d

Rank C = ?

Columns of C are Linear combination of Columns of A

There are only N independent vectors in A Rank C = N

SVDSVD

SVD for a matrix A writes A as a product of three matrices:

• U • D• V

• Every m x n matrix has a singular value decomposition

Am x n

Um x n

Dn x n

VT

n x n

U,V have orthogonalcolumns

Frame 1 Frame 2 ………… Frame F

Tomasi Kanade Structure from MotionTomasi Kanade Structure from Motion

Given N 2D trajectories taken over F images, recover 3D Given N 2D trajectories taken over F images, recover 3D structure and motion (Camera pose)structure and motion (Camera pose)

• Assumption:• 3D Object is rigid• Orthographic Projection

• Tracks can be computed using any standard tracker (KLT etc)

Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d

Assume a set of P 3D points on a rigid object (structure)

S = [P1, P2 …….. PP ]

Orthographic Projection

where (u,v) are image coordinates and M is orthographic projection where (u,v) are image coordinates and M is orthographic projection matrixmatrix

Subtract mean of all u’s and v’s to center the world coordinate frame at the Subtract mean of all u’s and v’s to center the world coordinate frame at the center of the object.center of the object. This will get rid of T in the above equationThis will get rid of T in the above equation

2D coordinates of N points over F images can be 2D coordinates of N points over F images can be written in one matrixwritten in one matrix

W is called the measurement/tracking matrixW is called the measurement/tracking matrix

Rank 3

Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d

From W to R and SFrom W to R and S

Force the rank of W to be 3

SVD

StepsSteps

Matrix Decomposition of W matrix for non-Matrix Decomposition of W matrix for non-rigid objectsrigid objects

Estimate Motion Matrix using reliable set Estimate Motion Matrix using reliable set of pointsof points

Estimate shape basis (S) for all other Estimate shape basis (S) for all other feature points (unreliable)feature points (unreliable)

For Non Rigid ConstraintFor Non Rigid Constraint

3D Non Rigid Shape Model 3D Non Rigid Shape Model

Linear Combination of K Basis ShapesLinear Combination of K Basis Shapes Each basis shape is SEach basis shape is Sii 3 x P3 x P matrix describing P points matrix describing P points

S1 S2………………

SK

= l1S1 l2S2+ + … + lKSKS

Courtesy Christopher Bregler

Matrix DecompositionMatrix Decomposition Project P points of shape SProject P points of shape S Scaled Orthographic ProjectionScaled Orthographic Projection

Move world coordinate to object centeroid (This will get Move world coordinate to object centeroid (This will get rid of T)rid of T)

W Q M

-

Tracking MatrixTracking Matrix

Complete 2D Tracks or Flow

Rank of W 3K

2F x 3K 3K x P

In Tomasi Kanade it was 3

Non Rigid Motion EstimationNon Rigid Motion Estimation

Since Since W W isis rank-deficient, rank-deficient, Q Q can be estimated w/o can be estimated w/o the full availability of the full availability of WW

r <= 3Kr <= 3K point tracks will span the space of the point tracks will span the space of the trajectories of all the points (as rank of W is r)trajectories of all the points (as rank of W is r)

? = ?

W Q’ M’

known reliable tracks

Courtesy Christopher Bregler

r = 9

Trajectory ConstraintTrajectory Constraint

Q’

3D positions of point i for the K modes of deformation

t=2

t=1

t=F

. . . .= .

...frames

miwi : full trajectory

• Generate m trajectories (hypothesis) using Factored Sampling• Evaluate w by computing sum of square difference around point i.

Courtesy Christopher Bregler

• Each column mi of unreliable M is computed as expected value of posterior.

ResultsResults