SUPPORT VECTOR MACHINE IN MATLAB
Weicheng Guo
PROBLEM DEFINITION
2
We classify red points (in a circle with radius of 1) as 1 and green points (in a ring with radius from 1 to 2) as -1 in this figure. We want to find the hyperplane separating these two classes. Our decision boundary will have equation:
𝑤𝑤𝑇𝑇x+b=0
FINDING THE DECISION BOUNDARY
3
Let {x1,…, xn} be the training data set and let yi∈{1,-1} be the class label of xi
The decision boundary should classify all points correctly with
The decision boundary can be found by solving the following constrained optimization problem
𝑦𝑦𝑖𝑖 𝑤𝑤𝑇𝑇𝑥𝑥𝑖𝑖 + 𝑏𝑏 ≥ 1 − 𝜉𝜉𝑖𝑖
min 12‖𝑤𝑤‖2 + 𝐶𝐶�𝜉𝜉𝑖𝑖
𝑛𝑛
𝑖𝑖=1𝑠𝑠𝑠𝑠𝑏𝑏𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑡𝑡 𝑦𝑦𝑖𝑖 𝑤𝑤𝑇𝑇𝑥𝑥𝑖𝑖 + 𝑏𝑏 ≥ 1 − 𝜉𝜉𝑖𝑖
REFORMULATING AS A LAGRANGIAN
4
We can introduce Lagrange multipliers to represent the condition and thus have the following formulation:
j1 1 1
1
1max x x2
subject to 0 , 0
n n nT
i i j i j ii i j
n
i i ii
y y
C y
α αα
α α
= = =
=
−
≤ ≤ =
∑ ∑∑
∑
KERNEL TRICK
5
Since it is a nonlinear classification in this case, we can use kernel functions that operates on the lower dimension vectors xi and xj to produce a value equivalent to the dot product of the higher-dimensional vectors. Therefore, the data can be linearly separated in that space and the formulation is transformed to:
1 1 1
1
1max ( , )2
subject to 0 , 0
n n n
i i j i j i ji i j
n
i i ii
y y K x x
C y
α αα
α α
= = =
=
−
≤ ≤ =
∑ ∑∑
∑
KERNEL FUNCTIONS
6
Linear kernel
Polynomial kernel with γ, d and r
Radial basis function kernel with γ
Sigmoid kernel with γ and r
𝐾𝐾(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗) = 𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗
𝐾𝐾(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗) = 𝛾𝛾𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗 + r 𝑑𝑑
𝐾𝐾 𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗 = exp(−𝛾𝛾||𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑗𝑗||2)
𝐾𝐾 𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑗𝑗 = tanh(𝛾𝛾𝑥𝑥𝑖𝑖𝑇𝑇𝑥𝑥𝑗𝑗 + r)
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
7
LIBSVM[1] is an open source machine learning library developed at the National Taiwan University and written in C++ though with a C API. LIBSVM implements the SMO algorithm for kernelized support vector machines (SVMs), supporting classification and regression. [1] https://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html
Differences between LIBSVM and SVM function in Matlab
LIBSVM SVM function in Matlab
Classification or regression Both Classification
Classification type Two-class and multi-class Two-class
SVM formulations C-SVC, nu-SVC, one-class SVM, epsilon-SVR, nu-SVR C-SVC
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
8
PROCEDURE 1. Transform data to the format of an SVM package
2. Conduct simple scaling on the data
3. Consider the RBF kernel K(𝑥𝑥𝑖𝑖,𝑥𝑥𝑗𝑗) = exp(−γ| 𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑗𝑗 |2)
4. Find the best parameter C and γ
5. Use the best parameter C and γ to train the whole training data
6. Test
TRAINING RESULT
9
GAMMA=0.1
TRAINING RESULT
10
GAMMA=3
TRAINING RESULT
11
GAMMA=50
TESTING RESULT
12
GAMMA=3
TESTING RESULT
13
GAMMA=3
CONCLUSION
14
SVM is a new method of machine learning based on statistics theory. In contrast to ‘black box’ learning approaches (artificial neural network), SVM is supported by certain mathematical models.
The training of SVM is relatively easy. Unlike in neural network, SVM can get global optimum and the training time does not depend on dimensionality of feature space any more by using the kernel trick.
The key to training SVM is to select a kernel function and its parameters, which cannot be conducted by a principled manner.