Computer Vision

Spring 2012 15-385,-685

Instructor: S. Narasimhan

WH 5409

T-R 10:30am – 11:50am

Lecture #23

Classification and SVM

Credits: Guru Krishnan and Shree Nayar

Eric P. Xing

Classification (Supervised Learning)

X = {X1, X2, … , Xn}Data:

Y = {y1, y2, … , yn}, yi {-1,1} Label:

X =

Y = 1 1 1 -1 -1 -1

Xt -1 or 1?Test:

XtFace or not?

Classifier: f: X Y

Classification and Computer Vision

Face

Object recognitionScene classification

Handwritten

characters

Classification and Computer Vision

Pedestrian detection

Yes / No

Procedure of Classification

(1) Gather positive / negative training data

(2) Feature Extraction

X =

Y = 1 1 1 -1 -1 -1

⋮ ⋮ ⋮ ⋮ ⋮ ⋮

Rn

Very challenging in reality…

Feature Extraction

⋮

Rn

PixelColor

histogram

Texture

filters

SIFT

Binary codes are available !

Haar

filter

Feature Space

f2

fN

f1

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

(3) Learn classifier!

Let the n-dimensional feature vector F be a point an

n-D space Rn(Feature space)

f2

fN

f1

Nearest Neighbor Classifier

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

Nearest samples decide the result of the classifier.

Test Image

f2

fN

f1

Nearest Neighbor Classifier

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

Nearest samples decide the result of the classifier.

Face

f2

fN

f1

Nearest Neighbor Classifier

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

Nearest samples decide the result of the classifier.

Not Face

f2

fN

f1

Nearest Neighbor Classifier

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

False Positive

Larger the training set, more robust the NN classifier

f2

fN

f1

Nearest Neighbor Classifier

°°

°

° °

Training Data

of Face

Training Data

of Non-Face

°

Larger the training set, slower the NN classifier

f2

fN

f1

Decision Boundary

°

Training Data

of Face

Training Data

of Non-Face

A simple decision boundary separating the face and

non-face classes will suffice.

°°

°

° °

Decision Boundary

Find Decision Boundary in feature space

Decision Boundary

°°°°

°°°°

°°°

Faces

Non-Faces

WTF+b=0

WTF+b>0

WTF+b<0

Decision Boundary

How to find the optimal decision boundary?

°°°°

°°°°

°°°

Face Class

Non-Face Class

Evaluating a Decision Boundary

Margin or Safe Zone: The width that the boundary

could be increased by before hitting a data point.

Margin

°°°°

°°°

°°

°°

Evaluating a Decision Boundary

Choose Decision Boundary with the Maximum Margin!

Margin I

° °°

°

°° °°

Margin II

° °°

°

°° °°

++

Decision I: Face Decision II: Non-Face

Support Vector Machine (SVM)

Margin

°°°°

°°°

°°

°°

Classifier optimized to maximize Margin

Support Vectors: Closest data samples to the boundary.

Decision Boundary and the Margin depend only on the

Support Vectors.

Finding Decision Boundary

°°

°°

°°

Distance between a point x to a plan

WTF+b=0

wTx + b

||w||d= (Hessian normal form)

WTF+b=0

WTF+b=c

-ρ

WTF+b=-c

ρ

wTxs + b

||w||ρ= where wTxs + b=c

c

||w||ρ=

max 2c

||w||w

s.twTxi + b

||w||≥

c

||w||for all Xi with yi=1

wTxi + b

||w||≤

-c

||w||for all Xi with yi=-1

Founding Decision Boundary

max 2c

||w||w

s.twTxi + b

||w||≥

c

||w||for all Xi with yi=1

wTxi + b

||w||≤

-c

||w||for all Xi with yi=-1

max 2c

||w||w

s.twTxi + b

||w||≥

c

||w||for all Xi

yi

min ||w||w

s.t (wTxi + b) ≥ 1 for all Xiyi

Learning classifier Compute w by solving the QP

Kernel Trick

°°

°°

°°

Linearly separable

How about these points in 1-D?

Think about a quadratic mapping Φ(x)=x2.

° °°0

° °°

0

No!

Yes!

Kernel Trick

°°

°°

°°

°°Mathematically hard to describe!

Φ(x)

°°

°°° °°

°