Binocular Stereo #1

Topics

1. Principle2. binocular stereo basic equation3. epipolar line4. features and strategies for matching

Binocular stereo

single image is ambiguous

A

another image taken from a different direction gives the unique 3D point

a’a”

Epipolar line

Epipolar plane

Epipolar line constraints

Corresponding points lie on the Epipolar lines

Epipolar line constratints

Base line

One image pointPossible line of sight

Epipolar geometry (multiple points)

C1

C2

e1e2

Epipoles:• intersections of baseline with image planes• projection of the optical center in another image• the vanishing points of camera motion direction

Examples of epipolar geometry

Examples of epipolar geometry

Examples of epipolar geometry

Characteristics of epipolar line

•rectification

Basic binocular stereo equation

A physical point

focal length

right image point

z

left image point

base line length

right image planeleft image plane

World coordinate systemleft image centerright image center

Camera Model

Pinhole camera

Camera Model

geometry

(X, Y, Z)

Image plane

X

Y

-Z

xy

(x, y)

f : focal length

Z

Yf

Z

Xfyx ,),(

Perspective projection

View point

(Optical center) (sX, sY, sZ)

Basic binocular stereo equation

z=-2df/(x”-x’)x”-x’: disparity2d : base line length

x” x’

-z

fd d

z

d + x

)("

"

dxz

fx

f

x

z

xd

d - x

)('

'

dxz

fx

f

x

z

xd

dz

fdxdx

z

fxx 2)('"

Classic algorithms for binocular Stereo

Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio

Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade

BakerHannahMoravec

Barnard-Thompson

MIT group

CMU group

Stanford group

Features for matching

a. brightness

b. edges

c. edge intervals

d. interest points

10 11 1210 11 12

10 11 1210 11 1211 15 16

a. relaxation

b. coarse to fine

c. dynamic programming

local optimam local optimam

Strategies for matching

global optimam

),(),()(),,( 32321211321 xxfxxfxfxxxf

10 10 1010 5 1010 10 10

10 10 1010 5 1010 10 10

10 10 1010 10 1010 10 10

Main purpose of development

simulate human stereosimulate human stereo

map makingmap makingmap makingmap making

map makingnavigationnavigation

navigation

Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio

Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade

BakerHannahMoravec

Barnard-Thompson

Features for matching

points(random dots)edgesintervals

brightness(gradient)intervalsbrightnessbrightness

edgesinterest pointsinterest points

interest points

Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio

Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade

BakerHannahMoravec

Barnard-Thompson

Strategies for matching

relaxationcoarse to finecoarse to fine

relaxationdynamic programmingRelaxation 　 (Kalman filter)relaxation

dynamic programmingcoarse to finecoarse to fine

relaxation

Marr-PoggioMarr-Poggio-GrimsonNishihara-Poggio

Lucas-KanadeOhta-KanadeMatthie-KanadeOkutomi-Kanade

BakerHannahMoravec

Barnard-Thompson

Summary

1.binocular stereo takes two images of 3D point from two different positions and determines its 3D coordinate system.2. Epipolar line

2D matching ↓1D matching

3. Features for matching---brightness,edges,edge interval,interest point

4. Strategies for matching---relaxation,coarse to fine,dynamic programming

5. ReadB&B pp.88-93Horn pp.299-303

Binocular Stereo #2

Topics

case studyarea-based stereoMarr-poggio stereosimulate human visual systemOhta-Kanade stereoaerial image analysisMoravec stereonavigation

Classification of stereo method

1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniqueness

Area-based stereo

1. method

b c

b

c

2. problema. trade-off of window size and resolutionb. dull peak

b c

1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniqueness

Area-based stereo

Marr-Poggio Stereo(`76)

Simulating human visual system(random dot stereo gram)

Marr,Poggio “Coopertive computation of stereo disparity” Science 194,283-287

Input : random dot stereo

left image

random dot

shift the catch pat

right image

we can see the height different between the central and peripheral area

Constraints– Epipolar line constraint

– Uniqueness constraint» each point in a image has only one depth value

O.K. No.

– Continuity constraint» each point is almost sure to have a depth value near the values o

f neighbors

O.K. No.

Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines

continuity constraint attracts more matching on a diagonal line

ABC

D E F

D E F

A

B

C

A

B

C

(E-A)

(E-B)

(E-C)

prohibit

attract

attract

(D-A)

(E-B)

(F-C)Same depth

n n+1

relaxation

10 10 1010 5 1010 10 10

10 10 1010 5 1010 10 10

10 10 1010 10 1010 10 10

),( jicn

)1,( jicn

),(1 jicn

)1,( jicn

),1( jicn

),1( jicn

Pr

''''1

''''

),(),(),(ji

nExji

nn jicjicjic

),(0 jic ),(1 jic ),(1 jicn

1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenesssimulate the human visual system (MIT)

Marr-Poggio Stereo(`76)

Ohta-Kanade Stereo(`85)

Map making

Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming” ,IEEE Trans.,Vol. PAMI-7,No.2,pp.139-14

now matching become 1D to 1D

yet, N line * ML * MR (512 * 100 * 100 * 10 m sec = 15 hours)

L1L2L3L4L5L6

R1R2R3R4R5R6

L

R

disparity

Path Search

Matching problem can be considered as a path search problem

define a cost at each candidate of path segment based some ad-hoc function

10 100 100

Dynamic programming

We can formalize the path finding problem as the following iterative formula

optimum cost to K

cost between M and K

)();(min)(}{

kDkMdMDk

)1()1;0(),2()2;0(),3()3;0(min)0( DdDdDdD

3 0

2 1

Optimum costs are known

stereo pair

edges

path disparity

depth

stereo pair

edges

depth

1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenessaerial image analysis (CMU)

Ohta-Kanade Stereo(`85)

Brightness of interval

Moravec Stereo(`79)

navigation

Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp.598-600 (1979)

Moravec’s cart

Slide stereo

Motion stereo

Slider stereo (9 eyes stereo)

9C2 = 36 stereo pairs!!! each stereo has an uncertainty measure uncertainty = 1 / base-line

each stereo has a confidence measure

22

2

ba

ab

long base line

large uncertainty

Coarse to fine

expand

expand

matching

matching

matching

σ

estimated distance

σ:uncertainty measure

area:confidence measure

9C2 = 36 curves

Interest point

1. Features for matchinga. brightness valueb. pointc. edged. region2. Strategies for matchinga. brute-force (not a strategy ???)b. coarse-to-finec. relaxationd. dynamic programming3. Constraints for matchinga. epipolar linesb. disparity limitc. continuityd. uniquenessnavigation (Stanford)

Moravec Stereo(`81)

interest point

Summary

1. Two images from two different positions give depth information

2. Epipolar line and plane

3. Basic equationZ=-2df/(x”-x’)x”-x’: disparity 2d : base line length

4. case studyarea-based stereoMarr-poggio stereo simulate human visual systemOhta-Kanade stereo aerial image analysisMoravec stereo navigation

5. Read Horn pp.299-303

F matrix

Camera Model

Pinhole camera

Camera Model

geometry

(X, Y, Z)

Image plane

X

Y

-Z

xy

(x, y)

f : focal length

Z

Yf

Z

Xfyx ,),(

Perspective projection

View point

(Optical center) (sX, sY, sZ)

Camera Model

Z

Yf

Z

Xfyx ,),( Perspective projection

RsZ

Y

X

f

f

y

x

s

10100

000

000

1

formularization

Perspective projection

(Non-linear)

Affine projection

(Linear)

Projection matrix

Affine Camera Models

General formularization

11000

0010

0001

1Z

Y

X

y

x

s•Orthographic

10100

000

000

1Z

Y

X

f

f

y

x

s•Perspective

•Affine camera

10001 34

24232221

14131211

Z

Y

X

a

aaaa

aaaa

y

x

s

Affine Cameras

perspective orthographic

Focal length

Distance from camera

Intrinsic parameters

Image plane : an ideal image

CCD : an actual picture

Not equal !

CCD elements

Intrinsic parameters

yAn ideal image on the Image plane

x

u

v

θ An actual picture

u0

v0

(x, y)

(u, v)

1100

sin0

cot

10

0

y

x

vk

ukk

v

u

v

uu

Intrinsic parameters

1100

sin0

cot

10

0

y

x

vk

ukk

v

u

s v

uu

e.g. perspective projection

10100

000

000

100

sin0

cot

0

0

Z

Y

X

f

f

vk

ukk

v

uu

XPAZ

Y

X

vfk

ufkfk

v

uu

10100

0010

0001

100

sin0

cot

0

0

Intrinsic matrix

Projection matrix (normalized)

T

Extrinsic parameters

Y

X

Z

P

),,( zyxp

i

j

k

Extrinsic parameters

T

Y

X

Z

P

),,( zyxp

i

j

k

TkzjyixP

TixPi tt

ti

Extrinsic parameters

TkzjyixP

TkzPk

TjyPj

TixPi

tt

tt

tt

T

k

j

i

P

k

j

i

z

y

x

t

t

t

t

t

t

Extrinsic parameters

T

k

j

i

P

k

j

i

z

y

x

t

t

t

t

t

t

tPR

TRPRp

R : rotation matrix t : translation vector

Summary (intrinsic & extrinsic parameters)

Y

X

Z (X,Y,Z)

World coordinate

R, t

(u, v)

picture

)(

1

tPRApAv

u

s

Camera coordinate

World coordinate

Summary (intrinsic & extrinsic parameters)

Y

X

Z (X,Y,Z)

World coordinate

R, t

(u, v)

picture

11

Z

Y

X

tRAt

Z

Y

X

RAv

u

s

3 × 4 matrix MtRAms~~

Epipolar geometry

C1

C2

t

xx

xRt

R

p

tt

tt

tt

ptpt

0

0

0

12

13

23 xRt

0 xRtx t

Essential matrix : E

Essential & Fundamental matrix

x x

0 xEx t Image planes (ideal)

Pictures (actual)

m

m

xAm

0)(

)()(1

21

1

111

mAEAm

mAEmA

xEx

tt

t

t

Fundamental matrix : F

Image 1 Image 2

F matrix

m

m

(u, v, 1) (u’, v’, 1)

0 mFmt

F & (u, v) known

0 mFmt 0 cvbua

Calculate the epipolar line

picture 1 picture 2

0..0,for

eFeimFmm t

2 picture in the epipole theis e

1 picture in the epipole theis ,0 satisfies, similaly,

eFe t

Computing F matrix (Linear solution)

0 mFmt

0

1

1

333231

232221

131211

v

u

fff

fff

fff

vu

01

33

32

31

23

22

21

13

12

11

f

f

f

f

f

f

f

f

f

vuvvvvuuuvuu

819)(33

matrix F of freedom) of (degree D.O.F

scaleambiguity

Corner detector

Extract interest points in each images

x

y

2

22

2

2

2

y

I

yx

Iyx

I

x

I

C

04.0

)(det 2

k

traceCkCR

Harris corner detector

Matching

),( jiI ),( jiI

)()(

),(

II

IICovCorr

or

2),(),( jiIjiId

Computing F matrix (Linear solution)

0

0

0

0

0

0

0

0

1

1

33

32

31

23

22

21

13

12

11

888888888888

111111111111

f

f

f

f

f

f

f

f

f

vuvvvvuuuvuu

vuvvvvuuuvuu

Suppose we found 8 pairs of corresponding points ·····

12

33

2

12

2

11 fff

Computing F matrix (Singularity constraint)

Epipolar pencil by linear solution (due to noise and error)

Computing F matrix (Singularity constraint)

Singular value decomposition (SVD)

321

3

2

1

00

00

00

VUF

2rank F Without noise, σ3 must be 0

modification

VUF

000

00

00

2

1

Computing F matrix (Singularity constraint)

VUF

3

2

1

00

00

00

VUF

000

00

00

2

1

Summary Pinhole camera and Affine camera

Intrinsic and extrinsic camera parameter

Epipolar geometry

Fundamental matrix