1/30/2012

1

EE 6882 Visual Search Engine

Prof. Shih‐Fu  Chang, Jan. 30, 2012Lecture #2

Visual Features: Global features and matching

Evaluation metrics

(Many slides from A. Efors, W. Freeman, C. Kambhamettu, L. Xie, and likely others)(Slides preparation assisted by Rong‐Rong Ji)

2

Course Format

Lectures + two hands‐on homeworks (due 2/13, 2/27)

Mid‐term project Review and implement topics of interest, 2 students each team

Proposal due 3/5, narrated slides due 3/26

Selected projects presented and discussed in class (3/26‐4/9)

Final project  Extension of mid‐term projects encouraged, 2 students each team

Proposal due 4/2, narrated slides due 4/30

Selected projects presented and discussed in class (4/30‐5/7)

Grading: Class participation (20%), homework (20%), mid‐term (20%), final (40%) 

Late policy: a total “budget” of 4 days for late submissions.  No other delays accepted.

1/30/2012

2

Image Features

Why features are needed? Finding similar images in database

Classifying images to categories

Tracking objects in video

Creating panorama

Stereo matching ‐> 3D

Desired properties Compact (~100 – 1000 dimensions) 

Easy to compute (30 fps for video)

Robust (invariant to photometric, geometric, content variations)

3

photoguides.net

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

merl.com

Desired Properties of Visual Features Invariance:

Rotation, scaling, cropping, shift, etc. illumination, pose, clutter, occlusion,

viewpoint

1/30/2012

3

Invariant Local Features

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

Features Descriptors(Slide of A. Efros)

6

(review) Imaging Formation

R G R G R

G B G B G

R G R G R

G B G B G

R G R G R

Lens CCD Sensor

Demosaicking

Filter

Camera Response Function

Additive Noise

DSP(White

Balance, Contrast

Enhancement… etc)

irradiance Image intensity

1/30/2012

4

Color Spaces and Color Order Systems Color Spaces

RGB – cube in Euclidean space

Standard representation used in color displays Drawbacks

RGB basis not related to human color judgments Intensity should be one of the dimensions of color Important perceptual components of color are

hue, saturation, and brightness

Perceptual color spaces: HIS, HSV

R G Br g b

R G B R G B R G B

8

Understanding HSI from RGB Turn the RGB cube so that Black‐

White axis is vertical

Each plane containing the B‐W axis and a color point contains all the colors of the same hue 

Hue represented as angle between the plane and a reference plane (e.g. Red)

Saturation: distance to axis, less saturated by mixing more grey colors

Intensity measured by intersection with the B‐W axis.

Cross section shape: triangle – hexagon ‐ triangle

Images from Gonzalez and Woods

1/30/2012

5

9

Colors on the HSI color cone Cross section approximated by triangle or 

circle

HSI values computed by various geometrical models, e.g., 

More suitable for measuring perceptual distance

Can be quantized unevenly, e.g.,Columbia VisualSEEK System: 16M colors (in RGB) quantized to 166 HSV colors (18 Hue, 3 Sat, 3 Val, 4 Gray)

B

G

R

V

V

I

06/16/1

6/26/16/1

3/13/13/1

2

1

)(tan1

21

V

VH

2/122

21 )( VVChroma

10

Manipulations in the HSI space

HSI values of primary/secondary colors

HSI allows independent manipulations of colors

Hue of Green & Blue set to 0. Saturation of Cyan reduced by

half. Intensity of White reduced by

half.

1/30/2012

6

Color Histogram Feature extraction from color images

Choose GOOD color space Quantize color space to reduce number of colors

Invariance? Scale, shift, rotation, crop, view angle, illumination, clutter,

occlusion Advantages

Easy to compute and compare Cons

Lack spatial information, dimension may be high

1 [ , ] , [ , ] , [ , ][ , , ]

0R G B

RGBm n

if I m n r I m n g I m n bh r g b

otherwise

Color Moments Is there a more compact representation than

color histogram? Compute moment statistics in each color channel.

?

1/30/2012

7

Localizing

http://www.ai.mit.edu/courses/6.801/Fall2002/

Color Layout Search

Query

results

Columbia VisualSEEk (Smith & Chang, ’96) IBM QBIC  (Flickner et al ’95)

1/30/2012

8

Color correlogram

http://www.ai.mit.edu/courses/6.801/Fall2002/

http://www.ai.mit.edu/courses/6.801/Fall2002/

1/30/2012

9

http://www.ai.mit.edu/courses/6.801/Fall2002/

EEAABB

EEAABB

EAABBB

AABDCB

AABCBB

AABBCB

Color Coherence Vector (CCV) (Pass et al, 1997)

1 1 1 1

1 1

, ,..., , , ,..., ,I n n I n n

n n

G i i i i H i i i ii i

G H

G G

by triangular inequality

Not just count of colors, also check adjacency

331122

331122

311222

112312

112122

112212

5131512

33121

size

color

EDCBAregions

103

51512

321

CCV

colorCoherent!Size threshold: 3

Region segmentation

= =

1/30/2012

10

Lp distance

Quadratic distance

Histogram Intersection

Mohalanobis distance

Distance Metrics between Feature Vectors

where Cx is the covariance matrixNormalize distance in the major/minor axes

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

p

i

pp ixixD /1

21 ))()((

)()(

))()(),()()((

2121

2121

xxCxx

jxjxjiCixixD

T

j iq

C(i,j): color distance

Mohalanobis Metric 2 1

1 2 1 2

1

(1,1) (1, 2) ... (1, )

... ... ... ...

( ,1) ( , 2) ... ( , )

( , ) ( ) ( ) ( ) ( ) / 1, :

Tmah x

x

N

k kk

D x x C x x

c c c d

covariance matrix C

c d c d c d d

c i j x i m i x j m j N N number of samples

oo o

oxi

xj

oo o

oxi

xj

ooo

oo

xi

xj

o oo o

o

xi

xj

oo o

oo

xi

xj

oo

i jc s s 1

2 i jc s s 0c 1

2 i jc s s i jc s ssi, sj: std. deviation

d: dimension of features

1/30/2012

11

Mohalanobis Metric

1 2 1 2 1 2

1 11 2 1 2 1 2

| ... | ( , ,..., ) | ... |

| ... | ( ( , , ..., )) | ... |

Tx d d d

Tx d d d

C e e e diag e e e

C e e e diag e e e

e1e2

Project data to the eigen vectors, divide with the sd of each eigen dimension, and compute Euclidian distance

where Cx is the covariance matrix

Normalize distance in the eigen vector axes

Mohalanobis Metric (cont.) Advantages of Mahalanobis metric

Account for scaling of coordinate axes Invariant under linear transformation

Correct for correlation Produce curved as well as linear decision boundaries

Potential issue Need enough training data to estimate Cov. Matrix

2 2,Ty x y xIf y Ax C AC A D D

.

km

cm ........

.

. ........

..... . ............

.

Maha. Dist.

Maha. Dist.c1

m1

cc

mc

xiMinimumSelector

Selected class

1/30/2012

12

Earth Mover’s Distance (EMD) Rubner, Tomasi, Guibas ’98 Mallow’s distance in statistics in 1950’s

Transportation Problem [Dantzig’51]

I Jcij

I: set of suppliersJ: set of consumerscij : cost of shipping a unit of supply from i to j

Problem: find the optimal flows fij

0, ,

,

,

i j iji I i I

ij

ij ji I

ij ij J

j ij J

minimize c f s.t.

f i I j J (No reverse shipping)

f y j J (satisfy each consumer need /cacacity)

f x i I (bounded by each supplier's limit)

y x (

i I

feasibility)

EMD of Color Histogram

1 1

1 1

, ,..., , , ,..., , ( ) ( )

,

j i

M N

ij iji j

M N

iji j

h h 1 h 2 h M g= g 1 g 2 h N assume g j h i

C f

EMD h g

f

Earth Hole

1 1 1

/M N N

ij ij ji j j

ij

ij ij

= C f g Fill up each hole

C : distance between color i in color space h and color j in color space g

f : move f units of mass from color i in h to color j in g

Normalization by the denominator term Avoid bias toward low mass distributions (i.e., small images) what’s the difference if both h and g are normalized first?

1/30/2012

13

Evaluation

Detection

False Alarms

Misses

Correct Dismissals

2/)(

)/(

)/(

)/(

1 RP

RPF

DBBF

BAAP

CAAR

1-N0 "Irrelevant" 0

Relevant"" 1

n

Vn

BVD

AVC

VB

VA

N

n n

N

n n

K

n n

K

n n

))1((

)(

)1(

1

0

1

0

1

0

1

0

N Images K Returned Results

Recall

Precision

Fallout

Combined

D B CA

“Returned”“Relevant Results”

Ground truth search

DB

Evaluation MeasuresPrecision at depth K

Precision Recall Curve

Receiver Operating Characteristic (ROC Curve)

) vs( RPP

R

KVP kn nk /)( 1

0

BA vs

A(hit)

B (false)

1/30/2012

14

Evaluation Metric: Average Precision

Ranked list of data in response to a query

3/73/63/53/42/31/21/1Precision

0001101truth Ground

DDDDD s......2163815

functionindicatorItotalR

DIPRK

AP j

K

jj

: data,relevant of#:

)]correct is ([ ),min(

1

1

AP approximates areas under PR curve

0 1 2 3 4 5 6 7

Precision

j

3 iP

AP1.0

Example:

Evaluation Metric: Average Precision Observations (AP)

AP depends on the rankings of relevant data and the size of the relevant data set. E.g., R=10

Case I: + + + + + + + + + - - - - --+Pre: 1 1 1 1 1 1 1 1 1 0 0 0 0 001 AP=1

Case II: - +Pre: 1/2 AP=1/2

- + - + - + - + - + - + - + - + - +1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2

Case II: Pre:

- - - --- - - -- + + + + + + + + +1/11 2/12 10/20… … AP~0.3

1/30/2012

15

Homework #1 Given a small image database and a few queries

Implement codes to extract color histogram

Implement codes to measure L2 image similarity

Use image object labels to measure precision/recall curves

Bonus:

Add new color features or similarity metrics to improve performance

Design GUI for result browsing

Reading List

• Rui, Y., T.S. Huang, and S.‐F. Chang, Image retrieval: current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, 1999. 10(4): p. 39‐62.

• Smith, J.R. and S.‐F. Chang. VisualSEEk: a Fully Automated Content‐Based Image Query System. in ACM International Conference on Multimedia. 1996. Boston, MA.

• David G. Lowe, Distinctive Image Features from Scale‐Invariant Keypoints, International Journal of Computer Vision, 60(2), 2004, pp91‐110.

• Randen, T. and J. Husoy, Filtering for texture classification: A comparative study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2002. 21(4): p. 291‐310.

• Mikolajczyk, K. and C. Schmid, A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005: p. 1615‐1630.

• Brown, M., R. Szeliski, and S. Winder. Multi‐image matching using multi‐scale oriented patches. in IEEE CVPR. 2005.