1

Trajectory estimation problems in vision

–temporal tracking• deterministic: Kalman Filter• stochastic: Particle Filter

–curve tracing• deterministic: LiveWire (Mortensen&Barrett, 95)• stochastic: JetStream (Perez et al. 01)

–stereo vision• deterministic: Dynamic Programming• stochastic: ??

Inferring the cyclopean image A forward-backward algorithm for stereo matching

• Stereo vision– seeing in depth

– cyclopean vision (cf. double vision)

• Applications to – teleconferencing

– virtual reality

collaborators: P. Torr, I.Cox, A. Criminisi

2

Virtual reality

(Anandan, Criminisi, Kang, Szeliski, Uyttendale, Microsoft Research)

Teleconferencing: failure of eye contact

screencamera

camera

3

Problem: estimate cyclopean intensity

Estimate parallax:

Cyclopean image:

L(x+d) = R(x)Parallax:

?? Occlusion?? Prior on d i) Epipolarity ii) Ordering

Epipolargeometry

(tutorial by Andrea Fusiello )

4

Stereo disparity on epipolar lines

Epipolar matchingas optimal path finding

m

n

1 2 3 4

1

2

3

4zero parallax

Mapping:

where

minimisingFind (Dijkstra)

(Ohta & Kanade, 1985;Cox, Hingorani & Rao, 1996)

5

Ordering constraint

m

n

Nail illusion

Gaze correction: virtual cyclopean camera

Dijkstra solution (10 fps)

6

Robust virtual cyclopean camera

• Family of paths

• Dijkstra framework won’ t do it

• Viterbi framework could work.

JetStream: particle filter(Perez et al.)

7

Stereo disparity in cyclopean coordinates

Cyclopean Coordinates

m

nk

d

1 2 3 4

1

2

3

4

12

3

-1-2

-3

2

4

6

8

Warp:

-- even or odd

8

Posterior distribution for cyclopean image

*

Obtain marginals for d--- check decomposition of p(z|I,d) and P(d)

*

-- robust estimate

Observation modelObservation density:

Generative form(matched)

Marginalise (step 1)

indept. Gaussians

(improper Gaussian)

uniform

Check decomposition of p(z|I,d) (for step 3)

9

Disparity prior

Decomposition (for step 2)

For example:

eveneven

... Disparity prior

m

n k

1

1-2q

q

q

eveneven

possibly occluded

10

Conventional forward algorithm?Time-stamped observations

Forward probabilities

Forward algorithm

But observations are not time-stamped – two streams:

observation history?

1 2 3 4

1

2

3

4

History for stereo observations

‘Past’ depends not only on time k but also on value dk

m

nk

d

11

Forward and backward probabilities

Define past observations:

and future observations:

(even hk)

Forward probabilities:

Backward probabilities:

(odd hk)

Forward step

Recall observation density

Forward step

for1 otherwise

12

...Forward step

m

n k

f(..)

1-2q

q

q

... Posterior distribution for cyclopean image

� Assume pointwise loss, so require only

for each

13

... Posterior distribution for cyclopean image

so

Posterior expectations of cyclopean intensity

m

n k

d

14

DijkstraFB

DijkstraFB

15

16

Dijkstra: c=0.005 FB: c=0.005, q=0.1 FB: c=0.005, q=0.02 FB: c=0.005, q=0.3

FB: disparity distributions

row 87row 90

disparity

17

FB vs. Dijkstracf. ground truth

Haloing around occlusions

18

Substituting background into occlusions

Cyclopean Smoothing --- open questions

• Non-cyclopean virtual views

• Regression to fronto-parallel

• Explicit occlusion labels/process

• Intra-scan-line constraints

• Robust intensity estimators

19

Right viewLeft view

(data: Scharstein & Szeliski 2002)

Labelling of occlusion

Occluded regions Disparity

End