Classification
Summer School
“Achievements and Applications of Contemporary Informatics,
Mathematics and Physics” (AACIMP 2011)
August 8-20, 2011, Kiev, Ukraine
Erik Kropat
University of the Bundeswehr Munich Institute for Theoretical Computer Science,
Mathematics and Operations Research
Neubiberg, Germany
Examples
Clinical trials
In a clinical trial 20 laboratory values of 10.000 patients are collected together with the diagnosis ( ill / not ill ).
We measure the values of a new patient.
Is he / she ill or not?
Credit ratings
An online shop collects data from its customers together with some information about the credit rating ( good customer / bad customer ).
We get the data of a new customer.
Is he / she a good customer or not?
Machine-Learning / Classification
Labeled training
examples
Machine learning
algorithm
Classification rule
New Example
Predicted classification
k Nearest Neighbor Classification
kNN
k Nearest Neighbor Classification
Compare the new object with the k “nearest” objects (“nearest neighbors”)
Idea: Classify a new object with regard to a set of training examples.
+
+ +
+
+ + +
+ + +
+
+ Objects in class 1 Objects in class 2
New object
4-nearest neighbor
k Nearest Neighbor Classification
• Required
− Training set, i.e. objects and their class labels
− Distance measure
− The number k of nearest neighbors
+ +
+ +
+ + +
+ + +
+
5-nearest neighbor
• Classification of a new object
− Calculate the distance between the objects of the training set.
− Identify the k nearest neighbors.
− Use the class label of the k nearest neighbors to determine the class of the new object (e.g. by majority vote).
New object
k Nearest Neighbor Classification
+
+
+ +
1-nearest neighbor
+
+
+ +
+
+
+ +
2-nearest neighbor 3-nearest neighbor
Classification ? + Class label:
Decision by distance
1-nearest neighbor ⇒ Voronoi diagram
kNN k Nearest Neighbor Classification
• The distance between the new object and the objects in the set of training samples is usually measured by the Euclidean metric or the squared Euclidean metric.
• In text mining the Hamming-distance is often used.
Distance
kNN k Nearest Neighbor Classification
• The class label of the new object is determined by the list of the k nearest neighbors. This could be achieved by
− Majority vote with regard to the class labels of the k nearest neighbors.
− Distance of the k nearest neighbors.
Class label of the new object
kNN k Nearest Neighbor Classification • The value of k has a strong influence on the classification result.
− k too small: Noise can have a strong influence.
− k too large: Neighborhood can contain objects from different classes (ambiguity / false classification)
+
+ +
+ + + +
+ +
+
Support Vector Machines
Support Vector Machines
A set of training samples with objects in Rn is divided in two categories:
positive objects and negative objects
Goal: “Learn” a decision rule from the training samples.
Assign a new example into the “positive” or the “negative” category. Idea: Determine a separating hyperplane.
New objects are classified as
positive, if they are in the half space of positive examples
negative, if they are in the half space of negative examples.
Support Vector Machines
INPUT: Sample of training data T = { (x1, y1),...,(xk, yk) | xi ∈ Rn , yi ∈ { -1, +1 } }, with xi ∈ Rn data
and yi ∈ {-1, +1} class label
Data from patients with confirmed diagnosis
Laboratory values
Disease: Yes / No
Decision rule: f : Rn → {-1, +1}
INPUT: Laboratory values of a new patient Decision: Disease: Yes / No
Support Vector Machines
Separating Hyperplane
A separating hyperplane is determined by
− a normal vector w and
− a parameter b
Idea: Choose w and b, such that the hyperplane separates the set of training samples in an optimal way.
H = { x ∈ Rn | ⟨ w, x ⟩ b = 0 }
H
w scalar product
‖ ‖ w b ____
Offset of the hyperplane from the origin along w:
What is a good separating hyperplane?
There exist many separating hyperplanes
Will this new object be in the “red” class?
Question: What is the best separating hyperplane?
Answer: Choose the separating hyperplane so that the distance from it
to the nearest data point on each side is maximized.
support vector
support vector
H
margin
maximum-margin hyperplane
Scaling of Hyperplanes
• A hyperplane can be defined in many ways:
For c ≠ 0: { x ∈ Rn | ⟨ w, x ⟩ + b = 0 } = { x ∈ Rn | ⟨ cw, x ⟩ + cb = 0 } • Use trainings samples to choose (w, b), such that Min | ⟨ w, xi ⟩ + b | = 1
xi canonical hyperplane
Definition
A training sample T = {(x1, y1),...,(xk, yk) | xi ∈ Rn , yi ∈ {-1, +1} } is separable by the hyperplane H = { x ∈ Rn | ⟨ w, x ⟩ + b = 0 }, if there exists a vector w ∈ Rn and a parameter b ∈ R, such that ⟨ w, xi ⟩ + b ≥ +1 , falls yi = +1
⟨ w, xi ⟩ + b ≤ 1 , falls yi = 1 for all i ∈ {1,...,k}.
H
w
⟨ w, x ⟩ + b = -1
⟨ w, x ⟩ + b = 1
Maximal Margin
• The above conditions can be rewritten:
yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 for all i ∈ {1,...,k}
• Distance between the two margin hyperplanes:
⇒ In order to maximize the margin we must minimize
‖ ‖ w 2 ____
‖ ‖ w
H
w
⟨ w, x ⟩ + b = -1
⟨ w, x ⟩ + b = 1
s.t. yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 for all i ∈ {1,...,k}
Optimization problem
Find a normal vector w and a parameter b, such that the distance between
the training samples and the hyperplane defined by w and b is maximized.
Minimize 2 ‖ ‖ w 1 2 __
⇒ quadratic programming problem
H
w
Dual Form
Find parameters α1,...,αk, such that
with αi ≥ 0 for all i = 1,...,k
i = 1 Σ k
αi 1 2 i, j = 1
Σ k
αi αj yi yj ⟨ xi, xj ⟩ Max
αi yi = 0 i = 1 Σ k
The maximal margin hyperplane (= the classification problem)
is only a function of the support vectors. ⇒
Kernel function
k ( xi, xj ) := ⟨ xi, xj ⟩
Dual Form
• When the optimal Parameters α1,...,αk are known, the normal vector w*
of the separating hyperplane is given by
* *
αi yi * xi w* = • The parameter b* is given by
b* = 1 2 _ _ max { ⟨ w*, xi ⟩ | yi = 1 } min { ⟨ w*, xi ⟩ | yi = +1 } +
i = 1 Σ k
training data
Classifier
• A decision function f maps a new object x ∈ Rn to a category f(x) ∈ {-1, +1} :
, if ⟨ w*, x ⟩ + b* ≥ +1
, if ⟨ w*, x ⟩ + b* ≤ 1 f (x) =
+1
1
H
w
-1
+1
Support Vector Machines
Soft Margins
Soft Margin Support Vector Machines
• Until now: Hard margin SVMs
The set of training samples can be separated by a hyperplane. • Problem: Some elements of the trainings samples can have a false label
The set of training samples can not be separated by a hyperplane
and SVM is not applicable.
• Idea: Soft margin SVMs
Modified maximum margin method for mislabeled examples.
• Choose a hyperplane that splits the training set as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples.
• Introduce slack variables ξ1,…, ξ n which measure the degree of misclassification.
Soft Margin Support Vector Machines
• Interpretation
The slack variables measure the degree of misclassification of the training examples with regard to a given hyperplane H.
H
ξ i
ξ j
Soft Margin Support Vector Machines
• Replace the constraints
by
yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 for all i ∈ {1,...,n}
yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 ξ i for all i ∈ {1,...,n}
Soft Margin Support Vector Machines
H
ξ i
• Idea
If the slack variables ξ i are small, then:
ξ i = 0 ⇔ xi is correctly classified
0 < ξ i < 1 ⇔ xi is between the margins.
ξ i ≥ 1 ⇔ xi is misclassified [ yi · ( ⟨ w, xi ⟩ + b ) < 0 ]
Constraint: yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 ξ i for all i ∈ {1,...,n}
H
ξ i
Soft Margin Support Vector Machines
• The sum of all slack variables is an upper bound for the total training error:
H
ξ i
ξ j
i = 1 Σ n
ξ i
Soft Margin Support Vector Machines
Find a hyperplane with maximal margin and minimal training error.
2
regularisation
Soft Margin Support Vector Machines
Minimize 2 ‖ ‖ w 1 2 __
i = 1 Σ n
ξ i
+
C
*
s.t. yi · ( ⟨ w, xi ⟩ + b ) ≥ 1 ξ i for all i ∈ {1,...,n }
* *
≥ 0 for all i ∈ {1,...,kn ξ i
Support Vector Machines
Nonlinear Classifiers
Support Vector Machines Nonlinear Separation
Question: Is it possible to create nonlinear classifiers?
Idea: Map data points into a higher dimensional feature space where a linear separation is possible.
Ф
Rn Rm
Support Vector Machines Nonlinear Separation
Nonlinear Transformation
Ф
Rn Rm original feature space high dimensional feature space
Kernel Functions
Assume: For a given set X of training examples we know a function Ф,
such that a linear separation in the high-dimensional space is possible.
Decision: When we have solved the corresponding optimization problem,
we only need to evaluate a scalar product
to decide about the class label of a new data object.
i = 1 Σ n
αi yi * f(xnew) ⟨ Ф (xi), Ф(xneu) ⟩ + b* ) ( = sign ∈ {-1, +1}
Kernel functions
Introduce a kernel function The kernel function defines a similarity measure between the objects xi and xj. It is not necessary to know the function Ф or the dimension of H !!!
K(xi, xj) = ⟨ Ф (xi), Ф(xj) ⟩
Kernel Trick
Example: Transformation into a higher dimensional feature space
Ф (x1,x2) = ( x1 , 2 x1 x2, x2 ) 2 ___
2 Ф : R → R , 2 3
Input: An element of the training sample x,
a new object x
⟨ Ф ( x ), Ф( x ) ⟩ ^
^
The scalar product in the higher dimensional space (here: R )
can be evaluated in the low dimensional original space (here: R ).
3
2
= x1 x1 + 2 x1 x1 x2 x2 + x2 x2 ^ ^ 2 2 ^ 2 2 ^
= ⟨ ( x1 , 2 x1 x2, x2 ), ( x1 , 2 x1 x2, x2 ) ⟩ ^ ^ ^ ^ 2 2 2 2
= ( x1 x1 + x2 x2) ^ ^ 2
= ⟨ x , x ⟩ ^ = K ( x , x ) ^ 2
√
___
√ ___
√
It is not necessary to apply the nonlinear function Ф to transform
the set of training examples into a higher dimensional feature space. Use a kernel function instead of the scalar product in the original optimization problem and the decision problem.
K(xi, xj) = ⟨ Ф (xi), Ф(xj) ⟩
Kernel Trick
Kernel Functions
Linear kernel K(xi, xj) = ⟨ xi, xj ⟩ Radial basis function kernel K(xi, xj) = exp
Polynomial kernel K(xi, xj) = (s ⟨ xi, xj ⟩ + c) d
Sigmoid kernel K(xi, xj) = tanh (s ⟨ xi, xj ⟩ + c) Convex combinations of kernels K(xi, xj) = c1K1(xi, xj) + c2K2(xi, xj) Normalization kernel K(xi, xj) =
xi xj ‖ ‖ 2
___________ 2 σ 2
0
___________________ K (xi, xj) ,
√ K (xi, xi) K (xj, xj)
σ 2 0 = ‖ ‖ xi xj 2 mean ;
, ,
Summary
• Support vector machines can be used for binary classification.
• We can handle misclassified data if we introduce slack variables.
• If the sets to discriminate are not linearly separable we can use kernel functions.
• Applications → binary decisions − Spam filter (spam / no spam)
− Face recognition ( access / no access)
− Credit rating ( good customer / bad costumer)
Literature
• N. Christianini, J.Shawe-Taylor
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods.
Cambridge University Press, Cambridge, 2004. • T. Hastie, R. Tibshirani, J. Friedman
The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2011.
Thank you very much!
