Feature points extraction

Many slides are courtesy of Darya Frolova, Denis Simakov

A low level building block in many applications: Structure from motion

Object identification:

Video Google

Objects recognition.

A motivating application Building a panorama

• We need to match/align/register images

Building a panorama

1) Detect feature points in both images

Building a panorama1. Detect feature points in both images

2. Find corresponding pairs

Building a panorama1. Detect feature points in both images

2. Find corresponding pairs

3. Find a parametric transformation (e.g. homography)

4. Warp (right image to left image)

11

pair matching

333231

232221

131211

rightleft

yx

hhh

hhh

hhhyx

Matching with Features

•Detect feature points in both images

•Find corresponding pairs

•Find a parametric transformation 2 (n) views geometry

Today's

talk

Criteria for good Features

Repeatable detector

Distinctive descriptor

Accurate 2D position

• Property 1:– Detect the same point independently in both

images

no chance to match!

Repeatable detector

Distinctive descriptor

• Property 2:– Reliable matching of a corresponding point

?

Accurate 2D position

• Property 3: Localization – Where exactly is the point

?

Sub-pixel accurate 2D position

11

333231

232221

131211

rightleft

yx

hhh

hhh

hhhyx

Examples of commonly used features

1. Harris, Corner Detector (1988)2. KLT Kanade-Lucas-Tomasi (80’s 90’s)

3. Lowe, SIFT (Scale Invariant Features Transform)

4. Mikolajczyk &Schmid, “Harris Laplacian” (2000)

5. Tuytelaars &V.Gool. Affinely Invariant Regions6. Matas et.al. “Distinguished Regions”7. Bay et.al. “SURF” (Speeded Up Robust Features) (2006)

Corner detectors

Harris & KLT

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

Descriptor: a small window around it (i.e., matching by SSD, SAD)

Detection: points with high “Cornerness” (next slide)

Localization: peak of a fitted parabola that approximates the “cornerness” surface

Lucas Kanade. An Iterative Image Registration Technique 1981.

Tomasi Kanade. Detection and Tracking of Point Features. 1991.

Shi Tomasi. Good Features to Track 1994.

Cornerness (formally)

20 0( , ) det traceR x y M k M

0 0

2

0 0 2, neighborhood( , )

( , ) ( , ) x x y

x y x y x y y

I I IM x y w x y

I I I

where M is a 22 “structure matrix”

computed from image derivatives:

“Cornerness” R (x0, y0) of a point is defined as:

And k – is a scale constant, and w(x,y) is a weight function

Descriptors & Matching- Descriptors ROI around the point (rectangle /

Gaussian ) typical sizes 8X8 up to 16X16.

- Matching: (representative options)- Sum Absolute Difference- Sum Square Difference- Correlation (Normalized Correlation)

Localization

• Fit a surface / parabola P(x,y)

(using 3x3 R values)

• Compute its maxima

Yields a non integer position.

0,0 dy

dP

dx

dP

Harris corner detector is motivated by accurate localization

Find points such that:

small shift high intensity change

Hidden assumption:Good localization in one image good localization in another image

Harris Detector Cont.

2,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Change of intensity for the shift [u,v]:

IntensityShifted intensity

Window function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

Cornerness ≈High change of intensity for every shift

E

u v

Harris Detector: Basic Idea

“flat” region: “edge”: “corner”:

2),(),(),( yxIvyuxIvuE

yx

yx yxIvIuIyxI,

2),(),(

yx

yx vIuI,

2

yx

yxyyxx vIuvIIvuIIuI,

2222

yx yxy

yxx

v

u

III

IIIvu

,2

2

Measuring the “properties” of E()

yx yxy

yxx

III

IIIM

,2

2

M depends on image properties

Harris Detector Cont.

( , ) ,u

E u v u v Mv

For small shifts [u,v] we have a bilinear approximation:

2

2,

( , ) x x y

x y x y y

I I IM w x y

I I I

where M is a 22 matrix computed from image derivatives:

“properties” of E() ↔ “properties” of M

( , ) ,u

E u v u v Mv

1, 2 – eigenvalues of M

direction of the fastest change

direction of the slowest change

(min)-1/2

(max)-1/2

Ellipse E(u,v) = const

Bilinear form and its eigenvalue

UUM T

2

1

0

0

KLT

“Cornerness” of a point R(x0, y0) is defined as:

And k – is a scale constant, and w(x,y) is a weight function

|)||,(| 21 MinR

Classification of image points using eigenvalues of M:

1

2

“Corner”1 and 2 are large,

1 ~ 2;

E increases in all directions

1 and 2 are small;

E is almost constant in all directions “Edge”

1 >> 2

“Edge” 2 >> 1

“Flat” region

Harris corner detector

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

20 0( , ) det traceR x y M k M

“Cornerness” of a point R(x0, y0) > threshold >0:

And 0<k<0.25 (~0.05) is a scale constant,

2212100 )(),( kyxR

Computed using 2 tricks:

Harris Detector

1

2 “Corner”

“Edge”

“Edge”

“Flat”

• R depends only on eigenvalues of M

• R is large for a corner

• R is negative with large magnitude for an edge

• |R| is small for a flat region

R > 0

R < 0

R < 0|R| small

20 0( , ) det traceR x y M k M

Harris Detector (summary)

• The Algorithm:– Detection: Find points with large corner

response function R (R > threshold)– Localization:

Approximate (parabola) local maxima of R- Descriptors ROI around (rectangle) the point.

Matching : SSD, SAD, NC.

Harris Detector: Workflow

Harris Detector: WorkflowCompute corner response R

Harris Detector: WorkflowFind points with large corner response: R>threshold

Harris Detector: WorkflowTake only the points of local maxima of R

Harris Detector: Workflow

Detector Properties

Properties to be “Invariant” to

2D rotations

Illumination Scale Surface orientationViewpoint (base line between 2 cameras)

If I detected this point Will I detect this point If I detected this point Will I detect this point If I detected this point Will I detect this point

Harris Detector: Properties

• Rotation invariance

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response R is invariant to image rotation

Harris Detector: Properties

• Partial invariance to intensity change Only derivatives are used to build M => invariance to intensity shift I I + b

R

x (image coordinate)

threshold

R

x (image coordinate)

Harris Detector: Properties

• Non-invariant to image scale!

points “classified” as edges Corner !

Harris Detector: Properties

• Non-invariant for scale changes

Repeatability rate is:# correspondences

# possible correspondences

C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

“Correspondences” in controlled setting (i.e., take an image and scale it) is trivial

Rotation Invariant Detection

• Harris Corner Detector

C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Examples of commonly used features

1. Harris, Corner Detector (1988)2. KTL Kanade-Lucas-Tomasi

3. Lowe, SIFT (Scale Invariant Features Transform)

4. Mikolajczyk &Schmid, “Harris Laplacian” (2000)

5. Tuytelaars &V.Gool. Affinely Invariant Regions6. Matas et.al. “Distinguished Regions”7. Bay et.al. “SURF” (Speeded Up Robust Features) (2006)

Scale Invariant problem illustration

• Consider regions (e.g. circles) of different sizes around a point

Scale invariance approach• Find a “native” scale.

The same native scale should redetected

(at images of different scale).

Scale Invariant Detectors

• Harris-LaplacianFind local maximum of: Harris corner detector for set of Laplacian images

1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

scale

x

y

Harris

L

apla

cian

SIFT (Lowe)

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

Find local maximum of Difference of Gaussians

scale

x

y

DoG

D

oG

Difference of Gaussians images

• Functions for determining scale

2 2

21 22

( , , )x y

G x y e

( , , ) ( , , )DoG G x y k G x y

DOG Imagef Kernels:

where Gaussian

Note: both kernels are invariant to scale and rotation

(Difference of Gaussians)

SIFT Localization

• Fit a 3D quadric D(x,y,s)

(using 3x3X3 DoG values)

• Compute its maxima

Yields a non integer position (in x,y) .

Brown and Lowe, 2002

D(x,y,s) is also used for pruning non-stable maxima

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

Scale Invariant Detectors

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

• Experimental evaluation of detectors w.r.t. scale change

Repeatability rate:

# correspondences# possible correspondences

SIFT Descriptors

SIFT – Descriptor • A vector of 128 values each between [0 -1]

We also computed

location

scale

“native” orientation

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

“native” orientation• Peaks in a gradient orientation histogram

Gradient is computed at the selected scale36 bins (resolution of 10 degrees). Many times (15%) more than 1 peak !?

The “weak chain” in SIFT descriptor.

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

Computing a SIFT descriptor

– Determine scale (by maximizing DoG in scale and in space),

– Determine local orientation (direction dominant gradient). define a native coordinate system.

– Compute gradient orientation histograms (of a 16x16 window)

– 16 windows 128 values for each point/ (4x4 histograms of 8 bins)

– Normalize the descriptor to make it invariant to intensity change

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

Matching SIFT Descriptors • vectors of 128 values

Using L2 norm.

A search for NN (or KNN) cannot be commuted trivially , and is implemented using a KD-tree

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

• SIFT Empirically found2 to show good performance,

2 K.Mikolajczyk, C.Schmid. “A Performance Evaluation of Local Descriptors”. CVPR 2003

Scale = 2.5Rotation = 450

SIFT - empirically found2 to show good performance,

Descriptors Invariant to Scale/Orientation

• Use the scale/orientation to determined by detector to in a normalized frame.

• compute a descriptor in this frame.

Scale example:• moments integrated over an adapted window• derivatives adapted to scale: sIx

Scale & orientation example:Resample all points/regions to 11X11 pixels

• PCA coefficients •Principle components of all points.

Other invariant features (not part of this class)

• K.Mikolajczyk, C.Schmid. “Harris Laplacian”

• T.Tuytelaars, L.V.Gool. Affinely Invariant Regions

• J.Matas et.al. “Distinguished Regions”

• Kadir & Brady Max entropy

• Bay et.al. “SURF” (Speeded Up Robust

Features)

Thank You

Note, home assignment 2.

Affine Invariant Detection (Tuytelaars)

• Take a local intensity extremum as initial point

• Go along every ray starting from this point and stop when extremum of function f is reached

T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

0

10

( )( )

( )t

o

t

I t If t

I t I dt

f

points along the ray

• We will obtain approximately corresponding regions

Remark: we search for scale in every direction

Affine Invariant Detection

• The regions found may not exactly correspond, so we approximate them with ellipses

• Geometric Moments:

2

( , )p qpqm x y f x y dxdy

Fact: moments mpq uniquely

determine the function f

Taking f to be the characteristic function of a region (1 inside, 0 outside), moments of orders up to 2 allow to approximate the region by an ellipse

This ellipse will have the same moments of orders up to 2 as the original region

Affine Invariant Detection

• Algorithm summary (detection of affine invariant region):– Start from a local intensity extremum point

– Go in every direction until the point of extremum of some function f

– Curve connecting the points is the region boundary

– Compute geometric moments of orders up to 2 for this region

– Replace the region with ellipse

T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

Affine Invariant Detection (Matas)

• Maximally Stable Extremal Regions– Threshold image intensities: I > I0

– Extract connected components(“Extremal Regions”)

– Find a threshold when an extremalregion is “Maximally Stable”,i.e. local minimum of the relativegrowth of its square

– Approximate a region with an ellipse

J.Matas et.al. “Distinguished Regions for Wide-baseline Stereo”. Research Report of CMP, 2001.

Affine Invariant DetectionKadir & Brady

• Algorithm summary

• Entropy(x,r) = entropy of circle of radiuses r around x• Look for local maxima (in r). scale• At each peak find change in pdf function of scale (r).• Replace the circle with ellipse ( a greedy search)

Kadir et. Al, Affine invariant salient region detector ECCV 04