U. of Milano, 8.7.04 P^k to P^2 1
Multi-linear Systems for 3D-from-2D Interpretation
Lecture 4
Dynamic Tensors
Amnon Shashua
Hebrew University of JerusalemIsrael
P k → P2
U. of Milano, 8.7.04 P^k to P^2 2
Material We Will Cover Today
• Two Applications for
• The general problem of reconstruction from “line of sights”
P4 → P2
• An Application for P6 → P2
• The geometry of P4 → P2
• Indexing into “action” (an example).
Wolf, Shashua ICCV01, IJCV’02 (Honorable Mention, Marr Prize)Levin,Wolf,Shashua CVPR’01
Wolf,Shashua ICCV’01: (translating planes)Wolf,Shashua CVPR’01: (segmentation tensors)
not discussed here
U. of Milano, 8.7.04 P^k to P^2 3
Dynamic configurations:
• Constant Velocity• Points in 3D• Straight-line Motion
(of points)• General Camera Motion• Trajectories Span
a 3D, 2D or 1D space
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 4
Dynamic configurations:
As Seen in the Image Space
• Recover Camera motionfrom image observations
• How many points/views?
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 5
Dynamic configurations:(multi-body)
• 2 Rigid Configurations• Bodies moving in relative
translation to each other
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 6
Dynamic configurations:(multi-body)
As Seen in the Image Space
• Recover Camera motionfrom image observations
• How many points/views?• How many “segmented”
points are needed?
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 7
A General Scheme for ConstructingMulti-view Tensors of Dynamic Scenes
23 PP →
22 PP → projections of (static) planar configuration
projections of (static) 3D configuration
2PPk →6,5,4,3=k
projections of dynamic and multi-body scenes
• We describe the multi-view constraints from these projection matrices.
• We show how to extract the structure and motion for each application.
U. of Milano, 8.7.04 P^k to P^2 8
When Camera Trajectory is known
U. of Milano, 8.7.04 P^k to P^2 9
Triangulation
U. of Milano, 8.7.04 P^k to P^2 10
Triangulation?
U. of Milano, 8.7.04 P^k to P^2 11
When Camera Trajectory is known
U. of Milano, 8.7.04 P^k to P^2 12
How Many Views for a Unique Line?
4 views - finite number of solutions (nonlinear)5 views - unique solution
U. of Milano, 8.7.04 P^k to P^2 13
NotationsL
ip
il
iM
U. of Milano, 8.7.04 P^k to P^2 14
Plucker Coordinates
( )11111 ZYXP =( )12222 ZYXP =
( )342423141312
21
LLLLLL
PPL
=∧=
222
111
1
1
ZYX
ZYX
For example:
=2
112
1
1det
X
XL
Given two 3D points
The 3D line is:
U. of Milano, 8.7.04 P^k to P^2 15
Linear Solution
0=lpT
( ) ( )21 MPMPl ×=
LMl ~= 433
2
1
×
=m
m
m
M
63
~
21
13
32
×
∧∧∧
=mm
mm
mm
M
( ) 0~ =LMpT
U. of Milano, 8.7.04 P^k to P^2 16
Dynamic
• Reconstruction using 5 views with Known Camera Motion.
• Planar scene, 3 views, unknown camera motion.
Generalization to 3D-from-2D becomes unwieldy:3x3x3x3x3x3 tensor with 729 observations per view!
23 PP →
Wolf, Shashua ICCV’01
What do we have so far:
Additional Constraints are Required
ii jVP + Position of the feature point at frame j
U. of Milano, 8.7.04 P^k to P^2 17
General Idea
Describe the dynamic 23 PP → applications
as a static 2PPk → problem
• 26 PP → Multi-point, 3D, constant-velocity
• 25 PP → Multi-point, 3D, coplanar constant-velocity
• 24 PP → Multi-point, 3D, collinear constant-velocity
Multi-point, 2D, constant-velocity
Two-body, 3D, Segmentation (pure trans)
• 23 PP → Two-body, 2D, Segmentation (pure trans)
U. of Milano, 8.7.04 P^k to P^2 18
3D segmentation tensor
ii PMp 43×≅
ii PMp 43ˆˆ ×≅
if point iP belongs to object 1
if point iP belongs to object 2
Because the motion between the two objects is pure translation:
][ vAM =
]ˆ[ˆ vAM =
24 PP →
U. of Milano, 8.7.04 P^k to P^2 19
3D segmentation tensor
≅ ×
0
1
]ˆ[ 53 i
i
i
i Z
Y
X
vvAp
≅ ×
1
0
]ˆ[ˆ 53 i
i
i
i Z
Y
X
vvAp
U. of Milano, 8.7.04 P^k to P^2 20
3D segmentation tensor
jM~
Let be the j’th projection matrix from 24 PP →
=1
ji
ji
ji y
x
p be the projection of 4~PPi ∈ in the j’th view
ijj
i PMp~~=
− jix
0
1
− jiy
1
0
0
1
0
1
=
−
ji
ji
T
ji
y
x
x
0
1
1
0
=
−
ji
ji
T
ji
y
x
y
Note:
U. of Milano, 8.7.04 P^k to P^2 21
1 0 −xi
1( ) ˜ M 1
0 1 −yi
1( ) ˜ M 1
1 0 −xi
2( ) ˜ M 2
0 1 −yi
2( ) ˜ M 2
l3T ˜ M 3
˜ P i = 0
3x3x3 tensor (27 coefs)
Each triplet of matchingPoints p,p’ ,p’’Provides 2 constraints
We need 13 points (from both objects!)
Once the tensor is recovered, 4 segmented pointsare sufficient to segment the entire scene
3D segmentation tensor
U. of Milano, 8.7.04 P^k to P^2 22
24 PP →Another Application of : 2D Dynamic Motion under const. velocity
Points moving under const. velocity
Recall from Homography Tensors lecture: 26 triplets are required for general motionalong straight line paths !
Let:
three homography matrices
projections of
Note: because of the const. Velocity constraint, the reconstruction could be up to 3D affineSo we are allowed to set the 3rd coordinate to be 1.
13 matching triplets would be sufficient !
U. of Milano, 8.7.04 P^k to P^2 23
)1,,,( iiii ZYXP =
)0,,,( iiii dZdYdXV =
ii jVP +
Let ni ,...,1=
mj ,...,0=
be a 3D point configurationwhere each point moving independently along a direction
At time the position of each point isLet jM 43×be the projection matrix
)( iijj
i jVPMp +≅
=
≅
i
i
i
i
i
i
j
i
i
i
i
i
i
jjj
i
j
i
dZ
dY
dX
Z
Y
X
M
dZ
dY
dX
Z
Y
X
jMMy
x
1~
1][
1
*
26 PP →
Multi-point, 3D, Constant Velocity
U. of Milano, 8.7.04 P^k to P^2 24
1 0 −xi
1( )˜ M 1
0 1 −yi
1( )˜ M 1
1 0 −xi
2( )˜ M 2
0 1 −yi
2( )˜ M 2
1 0 −xi
3( )˜ M 3
0 1 −yi
3( )˜ M 3
l4T ˜ M 4
X i
Yi
Zi
1
dX i
dYi
dZi
= 0
3x3x3x3 tensor (81 coefs)
Each quadruple of matchingPoints p,p’ ,p’’ ,p’’’Provides 2 constraints
We need 40 points
Multi-point, 3D, constant velocity
U. of Milano, 8.7.04 P^k to P^2 25
List of tensors
U. of Milano, 8.7.04 P^k to P^2 26
The term extensor (cf. Barnabei,Brini,Rota 1985) is used to describe thelinear space spanned by a collection of points.
Extensor of step 1 is a pointExtensor of step 2 is a lineExtensor of step 3 is a planeExtensor of step k in is a hyperplane
P k
In , the union (“join”) of extensors of step and whereis an extensor of step
k1P n k2 k1 + k2 ≤ n +1k1 + k2
The intersection (“meet”) of extensors of step and is an extensorof step
k1 k2k1 + k2 − (n +1)
Mathematical Background
U. of Milano, 8.7.04 P^k to P^2 27
In , the union (“join”) of extensors of step and whereis an extensor of step
k1P n k2 k1 + k2 ≤ n +1k1 + k2
The intersection (“meet”) of extensors of step and is an extensorof step
k1 k2k1 + k2 − (n +1)
Mathematical Background
The following statements follow immediately:
1. The center of projection of a projection is an extensor of stepP k → P2 k − 2
Recall, the center of projection is the null space of the projectionmatrix.
3× (k +1)
2. The line of sight (image ray) joins the COP and a point (on the image plane):
P3 → P2
P4 → P2
image ray is a line, extensor of step 1+1=2
image ray is a plane, extensor of step 2+1=3
P6 → P2 image ray is an extensor of step 4+1=5
U. of Milano, 8.7.04 P^k to P^2 28
In , the union (“join”) of extensors of step and whereis an extensor of step
k1P n k2 k1 + k2 ≤ n +1k1 + k2
The intersection (“meet”) of extensors of step and is an extensorof step
k1 k2k1 + k2 − (n +1)
Mathematical Background
The following statements follow immediately:
3. The intersection of two lines of sight (triangulation) is the meet of the twoextensors.
P3 → P2
P4 → P2
the intersection is generally empty 2+2-4=0
intersection is a point 3+3-5=1 (no constraint from two views!)
P6 → P2 intersection is a plane 5+5-7=3
4 views are required for a constraint.(intersection of 3 rays: 3+5-7=1)
U. of Milano, 8.7.04 P^k to P^2 29
In , the union (“join”) of extensors of step and whereis an extensor of step
k1P n k2 k1 + k2 ≤ n +1k1 + k2
The intersection (“meet”) of extensors of step and is an extensorof step
k1 k2k1 + k2 − (n +1)
Mathematical Background
The following statements follow immediately:
P3 → P2
P4 → P2
the epipole is a point (1+1) +3 - 4 =1
the epipole is a line (2+2) + 3 -5=2
P6 → P2 the epipole is not defined because 4+4 > 7 (the join of the twoCOPs is larger than the space dimension.
4. The epipole is the intersection between the join of the two COPs and an image plane.
U. of Milano, 8.7.04 P^k to P^2 30
The Geometry of 24 PP → Projection Matrices
23 PP →
p
Center of projection
)( 43×Mnull
Line of sight
•
24 PP →
Center of Projection is a Line
• Two planes intersect (always) at a point
Three views are necessary for a multilinear constraint
• Line of sight is a plane
)~
( 53×Mnull
1533 =−+
• A plane is the intersection of two hyperplanes
4 + 4 − 5 = 3
Cπ =π1
π 2
2×5
Cπ P =0
0
∀P ∈ π
U. of Milano, 8.7.04 P^k to P^2 31
The Geometry of 26 PP → Projection Matrices
23 PP →
p
Center of projection
)( 43×Mnull
•
26 PP →
Center of Projection is a 4D-space
• Two line-of-sights intersect in a plane
Four views are necessary for a multilinear constraint
• Line of sight is a 5D-space
)~
( 73×Mnull
Line of sight
3755 =−+A line-of-sight and a plane intersect at a point
1735 =−+•
U. of Milano, 8.7.04 P^k to P^2 32
1 0 −xi
1( ) ˜ M 1
0 1 −yi
1( ) ˜ M 1
1 0 −xi
2( ) ˜ M 2
0 1 −yi
2( ) ˜ M 2
l3T ˜ M 3
5×5
˜ P i = 0
The Geometry of P4 → P2
Aijk
The coefficients of the determinant expansionLive in a mixed tensor of the type
The constraint: pi p' j l' 'k Aijk = 0
A triplet of matching points provides 2 constraints
kkij
ji pApp ''' ≅
plane plane intersection of two planes: 3+3-5=1 is a point (projected onto )''p
Goal: given (recovered from 13 matching triplets) we wish to find the three3x5 projection matrices, and then to recover the 3D-2D underlying matrices.
Aijk
U. of Milano, 8.7.04 P^k to P^2 33
δ j Aijk ≅ Hi
kδ∨
→
2 plane by the induced
31matrix homography a
C
2C
kj
kij
i EA ≅δ δ∨→
1 plane by the induced
32matrix homography a
C
δ
1C
p
3C
pHδ
kij
j ApH δδ =
The Geometry of P4 → P2
Homography slices:
U. of Milano, 8.7.04 P^k to P^2 34
23 PP → 24 PP →
e31e13
C3
C33Ψ
1Ψ
Π
),( ofvector -eigen
dgenerelize a is
0)(
21
21
13
13
3113
ππ
ππ
π
λ
HH
e
eHH
eeH
⇒
≅−⇒
≅
e31 e13
3Ψ1Ψ
Π
) :form theof slices(
),( ofvector -eigen
dgenerelize a is
21
13
3113
kij
j
TT
T
A
HH
e
eeH
δππ
π
−−
−
⇒
≅TH −
π
The Geometry of P4 → P2
Recovering Epipoles from Homography slices: eij = ˜ M i(null( ˜ M j ))
U. of Milano, 8.7.04 P^k to P^2 35
Claim: There exist a canonical projective frame such that:
p ≅ I3x3 03 03[ ]p
ηµ
p'= Hπ e1 e2[ ]
p
ηµ
TT neneH 2211 ++πλ
The Geometry of P4 → P2
The canonical form of projection matrices:
Where is a homography matrix from views 1 to 2, andare two points on the epipole (a line) on view 2.
The family of homography matrices between views 1 to 2 has the generalForm with 7 degrees of freedom:
U. of Milano, 8.7.04 P^k to P^2 36
The Geometry of P4 → P2
The canonical form of projection matrices:
Proof:
Let be the two 3x5 projections matrices, a point P in space and matchingpoints p,p’ satisfying:
Let C,C’ be two points spanning the center-of-projection of the first camera:
Let a 5x5 projective change of basis W be defined: where U is some5x3 matrix chosen such that
Our goal will be to create the 5x3 matrix U such that is a homography matrix.
U. of Milano, 8.7.04 P^k to P^2 37
The Geometry of P4 → P2
The canonical form of projection matrices:
Proof:
Consider the following construction of U: (cols 1-3 from the 5x5 matrix)
Note:
We have:
∀P ∈ π
We have thus shown that for all matching points arising from the plane
U. of Milano, 8.7.04 P^k to P^2 38
Comparison With
[ ][ ]eHM
IM
PP
x
=
=
→
2
3331
23
~0
~:
1. adds :,between Scales
point. 1 determines
points. 3 determines
:points 5by determined is basisA
1
eH
C
H
[ ][ ]21122
33331
24
~00
~:
eeHM
IM
PP
x
=
=
→
1. adds :,,between Scales
line). (apoint 2 determines
points. 3 determines
:points 6by determined is basisA
2112
1
12
eeH
C
H
23 PP →
Note: the third camera matrix can be recovered from and the tensor
U. of Milano, 8.7.04 P^k to P^2 39
Given a new matching triplet reconstruct and if then the point belongs to object 1.
Performing the Segmentation• Points from each object are located on a different hyper-plane.
• Points on the first objectare on a hyper-plane:
• Four points are sufficient to recover this hyper-plane.
0
1
i
i
i
Z
Y
X
Take 4 matching points which are known to come fromone of the objects and reconstruct the points
Solve for the hyperplane
U. of Milano, 8.7.04 P^k to P^2 40
Recovery of structure
• Done in two stages:– Reconstruction of 3x(k+1) camera matrices per
view in Pk
– Reconstruction of the constituent 3x4 camera matrices in 3D.
• We get Affine 3D structure
U. of Milano, 8.7.04 P^k to P^2 41
Experiments
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 42
Traditional solution
• Pick 7 points from each object.
• For each object compute the trifocal tensor separately.
• Does not require relative translation.
U. of Milano, 8.7.04 P^k to P^2 43
Computing the tensor
• We use all points to compute tensor.
U. of Milano, 8.7.04 P^k to P^2 44
segmenting
• Only 4 known points from oneobject are needed.(instead of 7)
U. of Milano, 8.7.04 P^k to P^2 45
Results
U. of Milano, 8.7.04 P^k to P^2 46
The Geometry of 26 PP → Projection Matrices
23 PP →
p
Center of projection
)( 43×Mnull
•
26 PP →
Center of Projection is a 4D-space
• Two line-of-sights intersect in a plane
Four views are necessary for a multilinear constraint
• Line of sight is a 5D-space
)~
( 73×Mnull
Line of sight
3755 =−+A line-of-sight and a plane intersect at a point
1735 =−+•
U. of Milano, 8.7.04 P^k to P^2 47
• There is no such thing as epipole: the join of the two camera centers is larger then the whole space.
• We use joint epipole: the projection of the intersection of two camera centers.
• :the projection of the intersection of the first two camera centers onto the third view.
Joint Epipole
312C
U. of Milano, 8.7.04 P^k to P^2 48
Reconstruction of projection matrices from the tensor
3. in view C,C of epipolejoint theis
Cin contained plane a of 32 homography a is
Cin contained plane a of 31 homography a is
:
21321
123
213
c
H
H
where
→→
M4 can be determined linearly from the tensor and M1, M2, M3
U. of Milano, 8.7.04 P^k to P^2 49
Constant Velocity Experiments
The scene is 3D, The velocities are coplanar:A situations for a tensorP5 → P2
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 50
Computing the tensor
• We use all tracked points across three images to compute the tensor.
U. of Milano, 8.7.04 P^k to P^2 51
Results: 3-D velocity projected to first view
� Arrow’s base – point in first view
� Arrow’s tip – 3-D point in time = 2 projected to the first view
U. of Milano, 8.7.04 P^k to P^2 52
Indexing into “Action”
Training Sequence Single View: is this a view of a person sitting?
QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.
U. of Milano, 8.7.04 P^k to P^2 53
0),,...,( 1 =jppf k
Algebraic Polynomials for Indexing into Action
If the collection of image points at time j are a projection of the dynamic configuration.
Example: Affine Camera Model
+
=
×1
42
ii
j
j
ij
ij jVP
b
a
y
x
U. of Milano, 8.7.04 P^k to P^2 54
PiT + jVi
T 1 x ij( ) a j
−1
= 0
PiT + jVi
T 1 y ij( ) b j
−1
= 0
+
=
×1
42
ii
j
j
ij
ij jVP
b
a
y
x
0
1
1
01
11
01
555
444
33
22
11
=
+++++
jTT
jTT
TT
TT
TT
xjVP
xjVP
jVP
jVP
jVP
0
1
1
11
01
01
555
444
33
22
11
=
+++++
jTT
jTT
TT
TT
TT
yjVP
yjVP
jVP
jVP
jVP
U. of Milano, 8.7.04 P^k to P^2 55
0),,( 541 =jppf 0),,( 542 =jppf
The coefficients of these polynomials depend on shape P1,…,P5 and action V1,…,V5
These coefficients can be recovered using 8 views.
U. of Milano, 8.7.04 P^k to P^2 56
0 5 10 15 20 25 30 35 400
5
10
15
20
25
30
35
40
45
50
“time” increases monotonically for motions of the same “class”
Model Sequence
U. of Milano, 8.7.04 P^k to P^2 57
0 5 10 15 20 25 30 35 400
5
10
15
20
25
30
35
40
45
50
0 10 20 30 40 50 60−500
−400
−300
−200
−100
0
100
200
300
400
0 5 10 15 20 25 305
10
15
20
25
30
35
40
change of j for a “sitting” motion change of j for a “standing-up” motion
change of j for a “walking” motion using “sitting” po lynomials
U. of Milano, 8.7.04 P^k to P^2 58
1 2 3 4 5
Residuals of 5 tensors on images of Action 1
iiii VjUjOP )sin()cos( λλ ++=
U. of Milano, 8.7.04 P^k to P^2 59
The End