Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer...

Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2

(2004), pp. 91-110

Presented by: Shalomi EldarBased (in part) on slides by Ofir Pele, Kirill Dyagilev and Ayelet

Dominitz

SIFTSIFTScale Invariant Feature Transform

Vision Topics Seminar 2009

[ ]c n[ ]c n[ ]c n

2

Image Matching

Introduction

Detection Description

Applications

Motion Tracking

Fundamental aspect of many problems:

Object Recognition

3D structuresStereo Correspondence

3

Features Detection

Introduction


Applications

Give comprehensive description of

image.

Images from: M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the ) the International Conference on Computer Vision (ICCV2003

Enables matching!

What are the desired features’ features?

4

Features’ Features

Introduction


Applications

Robustness => Invariance to changes in

illumination, scale, rotation, affine, perspective.

Locality => robustness to occlusion and clutter.

Distinctiveness => easy to match to a large

database of objects.

Quantity => many features can be generated for

even small objects.

Efficiency => computationally “cheap”, real-time

performance.

5

SIFT Algorithm

Introduction


Applications

SIFT - Scale Invariant Feature Transform

Input: Image nxm.

Output: Set of descriptors of image’s

features.

6

SIFT Algorithm

2. Keypoint localization.

1. Scale-space extrema detection.

3. Orientation assignment.

4. Keypoint descriptor.

Introduction


Applications

Typical image of size 500x500

pixels produces about 2000 stable

keypoints Near Real-time performance:

Cascade approach – keep heavy operations only to keypoints that “survive”.

7

SIFT Algorithm

Introduction


Applications





=> Application

- matching.

We need only 3 keypoints matches for reliable

identification!

8

Today’s Talk Orientation

Extracting Keypoints

Distinctive Description Matching Keypoints

Extrema detectionCorrect localizationChoosing robust keypoints only

Local invariant orientationBuilding keypoint descriptor

Nearest-neighbor algorithmFinding 3 matchesLeast-square affine approximation

Detection Description Application

1st part of the talk 2nd part of the talk Last part of the talk

Z

Z

Z Z

Z

Introduction


Applications

9

SIFT Algorithm





Introduction


Applications

Collecting keypoint candidates…

10

Introduction


Applications

Why Extrema?

We want to find points that give us information about the objects in the image.

We will represent the image in a way that gives us these edges as this representations extrema points.

The information about the objects is in the object’s edges.

11

Scale-space Representation

Introduction


Applications

, , , , ,L x y G x y I x y

2 2 22

2

1, ,

2

x yG x y e

12

Low computation time - Only subtraction of smoothed

images!

Introduction


Applications

Scale-space Representation

, , , , , , ,

, , , ,

D x y G x y k G x y I x y

L x y k L x y

Difference-of-Gaussian (DoG):

13

Here’s what we get:

Introduction


Applications

DoG Pyramid

Different frequencies features

Scale invariance

14

X is selected if it is larger or smaller than all 26 neighbors.

Introduction


Applications


Low cost - only several usually checked.

15

Extrema detection product:

Introduction


Applications


Not all of them are good…

233x189 image => 832 DoG Keypoints.Each Keypoint is represented as . , ,x y

16

Problematic Keypoints

Introduction


Applications

Inaccurate localization (due to scaling and sampling).

Low contrast - sensitive to noise.Strong edge responses.

17

SIFT Algorithm




4. Keypoint descriptor.Filtering Keypoints…

Introduction


Applications

18

Inaccurate Keypoint Localization

Introduction


Applications

xSampling

Detected Extrema

True Extrema

The Problem:

19

Inaccurate Keypoint Localization

Introduction


Applications

The Solution:

xx

Dxx

x

DDxD

TT

T

2

2

2

1

Taylor expansion:

Minimize to find accurate extrema:

x

D

x

Dx

1

2

2

ˆ

Brown & Lowe 2002

If offset from sampling point is larger than 0.5 - Keypoint should be in a different sampling point.

20

Low Contrast Keypoints

Introduction


Applications

Function value at the extremun - xD ˆIf (pixel values in range [0,1]) - keypoint is discarded.

03.0ˆ xD

=> down to 729 Keypoints after min. contrast threshold.

21

Eliminating Edge Responses

Introduction


Applications

The Problem:“Edge“ keypoints are poorly

determined.

Point can move along edgePoint detection Point detection

=> unstable.

22

Eliminating Edge Responses

Introduction


Applications

The Solution:Check Keypoints “cornerness”.

Point constrained

High “cornerness” No dominant principal curvature component.

23

Finding “Cornerness”

Introduction


Applications

xx xy

xy yy

D DH

D D

Principal curvature are proportional to eigenvalues of Hessian matrix:max min,

2 2max

min

) ) ) 1)

) )

Tr H rr

Det H r

Harris (1988) showed:

Threshold: if r < 10 - ratio is too great, keypoint discarded.

24

Stable Keypoints

Introduction


Applications

=> down to 536 Keypoints after “cornerness” threshold.

25









Introduction


Applications

DEMO

26

SIFT Algorithm




4. Keypoint descriptor.Representation invariant to Rotation.

Introduction


Applications

27

Gradients

Introduction


Applications

yxLyxL

yxLyxLyx

yxLyxLyxLyxLyxm

,1,1

)1,)1,tan,

)1,)1,,1,1,

1

22

For each sample point we compute gradient’s magnitude and orientation:

28

Keypoints Orientation

Introduction


Applications

Create gradient histogram (36 bins) weighted by magnitude and Gaussian window ( is 1.5 times that of the scale of a keypoint)

Any histogram peak within 80% of highest peak is assigned to keypoint (multiple assignments possible).

29

SIFT Algorithm





Distinctive (yet invariant) Keypoint Representation.

Introduction


Applications

30

What do we have (and what don’t)

For each Keypoint we have assigned location, scale and orientation.Provides invariance to these parameters.

Introduction


Applications

Do:

Sufficient distinctiveness.

Invariance to other parameters, such as 3D viewpoint and change of illumination.

Don’t:

31

Keypoint Descriptor

Introduction


Applications

Create 16 gradient histograms (8 bins) weighted by magnitude and Gaussian window ( is 0.5 times of the window)

Keypoint Descriptor -

128 (4x4x8) element vector

32

Change of Illumination

Introduction


Applications

Change of brightness => doesn’t effect gradients (difference of pixels value).

Change of contrast => doesn’t effect gradients (up to normalization).

Saturation (non-linear change of illumination) => affects magnitudes much more than orientation.=> Threshold gradient magnitudes to 0.2

and renormalize.

33









Introduction


Applications

34

Object RecognitionFor training images:

Extracting keypoints by SIFT.Creating descriptors database.

Performing detailed geometric fit check for each cluster.

Introduction


Applications

For query images:Extracting keypoints by SIFT.For each descriptor - finding nearest neighbor in DB. Finding cluster of at-least 3 keypoints.

35

Keypoint MatchingBest candidate mach:

Introduction


Applications

Problem: There are keypoints that do not have correct matches

(background clutter/were not detected in training images)

Nearest neighbor.

36

Keypoints MatchingSolution:

Introduction


Applications

nearest neighborsecond nearest neighbor

closest neighbor

found

closest neighbor

from different

object

Threshold:

< 0.8

Eliminates 90% of false matches while discarding less than 5% of correct matches.

37

No good algorithm for high dimensional spaces.

Finding Nearest Neighbor

Introduction


Applications

=> Use Best-Bin-First (BBF) algorithm (Beis and Lowe, 1997).

=> Returns closest neighbor with high probability.

Low cost - cutting off search after checking specific number of

candidates. Good enough - we only need to show 0.8 ratio between first and

second neighbor.

38

Clustering

We cluster 3 keypoints using Hough Transform.

In order to identify an object with high probability - we need more than one match:

Introduction


Applications

39

Geometric VerificationHough Transform found clusters of keypoints.

Introduction


Applications

We would like to verify that these points do match geometrically to trained image.

In order to do that we use least-squares approach to find affine transformation from training image to query image.

Now we can be pretty sure we got the right object!

40

Examples

Introduction


Applications

Training images: Query image: Recognition (clutter, occlusion, illumination, etc.):

41

Examples

Introduction


Applications

Training images: Query image: Recognition (different viewpoint, non-distinctive):

Total time to recognize all object in both examples is less

than 0.3 sec. (on 2GHz Pentium 4 processor)

42

Image Registration

Introduction


Applications

[Brown & Lowe 2003]

43

Examples

Introduction


Applications

Real time Object-Recognition:

Nose DEMO

Motion Tracking:

Recognition DEMO

44



Summary

Introduction


Applications






45

Thank you

Date post:	21-Dec-2015
Category:	Documents
View:	233 times
Download:	1 times

Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer...

Documents