The Correspondence Problemand “Interest Point” Detection

Václav HlaváčCenter for Machine Perception

Czech Technical University Praguehlavac@fel.cvut.cz

Courtesy Martin Urban (the talk is based on his presentation) and Jana Kostkova, Jan Kybic, Jiri Matas, Radim Sara

2

1. The Correspondence Problem (CP) lies at the core of a number of computer vision problems:

• tracking (correspondence in consecutive frames• narrow-baseline stereo• wide-baseline stereo• egomotion (camera motion) estimation • motion segmentation in sequences with a moving camera• recognition, categorisation, …

2. Demo of typical applications3. The CP is commonly solved by robust matching of

“Interest Points”4. “Interest Points” are regions with distinguishing

property, allowing there detection in a viewpoint and illumination invariant manner

5. Harris “Corner” (HC) Detection Algorithm• HC are rotation and translation invariant interest points• HC are interest points most commonly used in tracking and stereo

Lecture Overview

3

“Ideal” solution: a pixel to pixel mapping from A to B (B ⋃ NULL)

What it is “The correspondence problem”?

A B

Ex. result: x-shift

Applications: 3D Reconstruction

3D Reconstruction

Camera motion tracking ⇒ image stabilization

original stabilized

original stabilized

Camera motion tracking ⇒ 3D animation

Motion keying / segmentation

input sequence

Background sequence Foreground sequence

Motion Detection

Input:

Output:

Medical imaging – image registration

from the atlas

test slice

deform. field

before registration

after

11

“Ideal” solution: a pixel to pixel mapping from A to B

The correspondence problem

A B

How to do it?

12

The correspondence problem

A BIntuitive approach:

For ∀ pxl ∈ A, find a pxl ∈ B with the most similar neighbourhood.

Problems: - How to measure similarity of image patches? - undistinguishable regions (e.g. texture-less) - not surjective map (onto) due to occlusions - not bijective map (one to one) due to scale changes

- huge data ⇒ very hard / impossible to recover the mapping by direct

minimisation

13

The correspondence problem

Conclusion:It is very hard / impossible to recover dense pxl 2 pxl mapping between two images.

Solution: 1. Recover the correspondence relation just between several well distinguished image features (interest points / corners / Harris points). 2. Estimate multi-view transformation (e.g. epipolar geometry, camera motion).

3. Having epipolar geometry, try to find dense correspondences.

14

Corner Detection: Introduction

Corner detector detects points with distinguished neighbourhood(*) well suited for matching verification.

undistinguished patches:

distinguished patches:

15

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

16

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

17

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

18

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

19

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

20

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

21

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

22

-

Demo of a point + with well distinguished neighbourhood.

+

> 0

Corner Detection: Introduction

23

Example of detected points

Corner Detection: Introduction

24

Corner Detection: Basic principle

undistinguished patches: distinguished patches:

Image gradients ∇I(x,y) of undist. patches are (0,0) or have only one principle component.

Image gradients ∇I(x,y) of dist. patches have two principle components.

⇒ rank ( ∑ ∇I(x,y)* ∇I(x,y) ⊤ ) = 2

25

Algorithm (C. Harris, 1988)

1. filter the image by gaussian (2x 1D convolution), sigma_d

2. compute the intensity gradients ∇I(x,y), (2x 1D conv.)

3. for each pixel and given neighbourhood, sigma_i:

- compute auto-correlation matrix

A = ∑ ∇I(x,y)* ∇I(x,y) ⊤

- and evaluate the response function R(A):

R(A) >> 0 for rank(A)=2, R(A) → 0 for rank(A)<2

4. choose the best candidates (non-max suppression and thresholding)

26

Corner Detection: Algorithm (R. Harris, 1988)

Harris response function R(A):

R(A) = det (A) – k*trace 2(A) ,

[lamda1,lambda2] = eig(A)

27

Corner Detection: Algorithm (R. Harris, 1988)

Algorithm properties:

+ “invariant” to 2D image shift and rotation

+ invariant to shift in illumination

+ “invariant” to small view point changes

+ low numerical complexity

- not invariant to larger scale changes

- not invariant to high contrast changes

- not invariant to bigger view point changes

28

Corner Detection: Algorithm (C. Harris, 1988)

Exp.: Harris points and view point change

29

Corner Detection: Harris points versus sigma_d and sigma_i

Sigma_I →

↑Sigma_d

30

Corner Detection: Application

Algorithm:

1. Corner detection

2. Tentative correspondences- by comparing similarity of the corner neighb. in the searching window (e.g. cross-correlation)

3. Camera motion geometry estimation (e.g. by RANSAC)- finds the motion geometry and consistent correspondences

4. 3D reconstruction- triangulation, bundle adjustment

3D camera motion tracking / 3D reconstruction

31

Corner Detection: Camera Tracking Application - Boujou

Input sequence

32

Corner Detection: Camera Tracking Application - Boujou

Harris points

33

Corner Detection: Camera Tracking Application

points consistent with 3D camera motion

34

Corner Detection: Camera Tracking Application

3D points and 3D camera motion

35

Corner Detection: Camera Tracking Application

3D animation

36

Corner Detection: Camera Tracking Application - Boujou

Input sequence

37

Corner Detection: Camera Tracking Application - Boujou

Harris points

38

Corner Detection: Camera Tracking Application - Boujou

points consistent with 3D camera motion

39

for the presentation I copied almost done presntation• Martin Urban

for used demo images and software:• Jana Kostková, CMP - (slide 3: images, disparity map)• Radim Šára, CMP - (slide 5: images, 3D face reconstruction

demo)• Jan Kybic, CMP - (slide 10: medical image registration)

• 2d3 – (slide 31-38: Boujou Demo, slide 6: img. sequences) • ImagineerSystems – (slide 8: MoKey Demo, slide 8: img.

sequences)• The Pixel Farm – (slide 31-33: img. sequence)

Acknowledgements