Support Vector Machine basierte Klassifikation in der ... · Geographisches Institut der...

Abteilung Geomatik

Geographisches Institut der Humboldt-Universität zu Berlin

Support Vector Machine

basierte Klassifikation

in der Geofernerkundung

*[email protected]

Andreas Rabe*

Sebastian van der Linden

Patrick Hostert

26. Oktober 2011


2

Bad news:

• wrong usage leads to overfitting or underfitting

• mostly used as a black box (complex mathematics)

• nearly never used with one- or two-dimensional date

Support Vector Machines (SVMs)

What would be of interest for the audience?

Good news:

• a SVM is a state-of-the-art classifier (fits arbitrary class boundaries)

• is widely used inside remote sensing applications

• works well in high-dimensional feature spaces (hyperspectral data)

Take-home-message:

• you can always avoid overfitting or underfitting when using SVM

• you can use SVM as a black box, ...

• ... but you could gain a deeper understanding by looking at simple

one- or two-dimensional examples


3

Support Vector Machines (SVMs)

What would be of interest for the audience?

This talk...

• is not about the mathematics and theory behind SVMs.

• is not about specific remote sensing applications → colored maps are not helpful!

• is about understanding the concepts behind SVM and the influence of parameters.

• is about learning from simple one- or two-dimensional examples, to be able to

generalize to high-dimensional, real world problems.


4

linear / separable

linear /non-separable

non-linear / separable

non-linear /non-separable

simple

kernel function

regularization

regularizationand

kernel function

Different settings for binary classification in 2D

positive class negative class

To train a SVM we need to set appropriate parameter values for the kernel function

(e.g.RBF kernel with parameter g) and for the regularization (parameter C).


5

SVM overview

A Support vector machine (SVM) ...

... is a universal learning machine for

- pattern recognition (classification),

- regression estimation and

- distribution estimation.

... can be seen as an implementation of Vapnik's Structural Risk Minimisation

principle inside the context of Statistical Learning Theory (Vapnik1998).


6

The optimal separating hyperplane.

Suppose the training set:

(x1,y1),..., (xl,yl), x Rn, y {+1,−1},

can be separated by a hyperplane

(w∙x)-b = 0.

The optimal separating hyperplane

separates the vectors without error and

maximizes the margin between the

closest vectors to the hyperplane.

(Burges1998)

SVM classification overview

H1: (w∙x)-b = -1

H2: (w∙x)-b = +1


7

To construct the optimal separating hyperplane one has to

solve a quadratic optimization problem:

Formulated as lagrange functional:

Maximize the functional

under the constraints:

1

2L w w w

1 if 1

1 if 1

i i

i i

b , y

b , y

w x

w x

1 1

1

2

l l

i i j i j i ji i , j

W α α α y yα x x

1

0 and 0.l

i i ii

α y α

Minimize the functional





8

Let α0 = (α10,..., αl

0) be a solution to this quadratic optimization problem.

The optimal hyperplane w0 is a linear combination of the vectors of the training set.

0

0 0 01

l

i i ii

f b α y bx w x x x

y sign fx x

0

01

l

i i ii

α yw x

The decision rule y(x) is based on the sign of the decision function f(x):




9

1d

i iK ,x x x x

Kernel Function

When looking at the lagrange functional:

1 1

1

2

l l


W α α α y yα x x

it can be observed, that only dot products between vectors

in the input space are calculated.

The idea is to replace the dot product in the input space by the dot product in a

higher dimensional feature space, defined by a kernel function K(x,xi).

Polynomial kernel:

Gaussian RBF kernel: 2

expi i

K , gx x x x


0

01

l

i i ii

f α y K , bx x x

This leads to a non-linear decision function:


10

The concept of maximizing the margin between classes must

be modified, to be able to handle non-separable classes.

We introduce so-called slack variables = ( 1,..., l), one for

each vector in the training set.

1

1

2

l

ii

L Cw w w

1 if 1

1 if 1

0

i i i

i i i

i

b , y

b , y

i

w x

w x

Minimize the functional


(Burges1998)

Regularization



11

1 1

1

2

l l


W α α α y y K ,α x x

1

0 and 0l

i i ii

α y α C

Formulated as lagrange functional:

Maximize the functional


Regularization



12

simple separable example

2D example - separable, linear

(Burges1998)

1D example - separable, non-linear

2

1

expl

i i ii

f α y g bx x xclass sign fx x

2D example - separable, non-linear

(www.mblondel.org)


13

simple non-separable example

1D example - non-separable, non-linear

2D example - non-separable, linear

(Burges1998)

2

1

expl

i i ii

f α y g bx x xclass sign fx x

2D example - non-separable, linear

(www.mblondel.org)


14

influence of parameters

good fit

numerical

problems

overfitting

underfitting

cg

kernel parameter g and penalty parameter c


15

influence of parameters

kernel parameter g and penalty parameter c

good fit

numerical

problems

overfitting

underfitting

cg


16

imageSVM inside EnMAP-Box software (remote sensing software)

A SVM implementation for classification and regression imageSVM is freely

available inside EnMAP-Box software (contact [email protected]).

Suitable parameters are estimated via grid search and cross-validation.


17

underfitting

outlook - SVM regression

SVM regression - kernel parameter g and penalty parameter c

cg

good fit

overfitting


18

SVM regression - epsilon-loss function

outlook - SVM regression


19

Thank you very much for your attention.

Any questions?

References

Burges, C. J. C. (1998). "A Tutorial on Support Vector Machines for Pattern Recognition." Data Mining and Knowledge Discovery 2(2): 121-167.

Chang, C.-C. and C.-J. Lin (2001). LIBSVM: a Library for Support Vector Machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

Vapnik, V. (1999). The Nature of Statistical Learning Theory, Springer-Verlag.

Date post:	05-Jul-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Support Vector Machine basierte Klassifikation in der ... · Geographisches Institut der...

Documents