1

Today’s Topics

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Support Vector Machines (SVMs)

Three Key Ideas

– Max Margins

– Allowing Misclassified Training Examples

– Kernels (for non-linear models; in next lecture)

Three Key SVM Concepts

• Maximize the MarginDon’t choose just any separating plane

• Penalize Misclassified ExamplesUse soft constraints and ‘slack’ variables

• Use the ‘Kernel Trick’ to get Non-LinearityRoughly like ‘hardwiring’ the input HU

portion of ANNs (so only need a perceptron)

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 2

Support Vector MachinesMaximizing the Margin between Bounding Planes

Support Vectors

?

Margin

2||w||2

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 3

SVMs define some inequalities we want satisfied. We then useadvanced optimization methods (eg, linear programming) to find the satisfying solutions, but in cs540 we’ll do a simpler approx

Margins and Learning Theory

Theorems exist that connect learning (‘PAC’) theory to the size of the margin

– Basically the larger the margin, the better the expected future accuracy

– See, for example, Chapter 4 of Support Vector Machines by N. Christianini & J. Shawe-Taylor, Cambridge Press, 2000(not an assigned reading)

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 4

y Support Vectors

‘Slack’ VariablesDealing with Data that is not Linearly Separable

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 5

For each wrong example, we pay a penalty, which is the distance we’d have to move it to get on the right side of the decision boundary (ie, the separating plane)

If we deleted any/all of the non support vectors we’d get the same answer!

SVMs and Non-Linear Separating Surfaces

f1

f2 +

+

_

_

h(f1, f2)

g(f1, f2) +

+

_

_

Non-linearly map to new space

Linearly separate in new spaceResult is a non-linear

separator in original space

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 6

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 7

Math Review: Dot Products

11/10/15

X Y X1 Y1 + X2 Y2 + … + Xn Yn

So if X = [4, 5, -3, 7] and Y = [9, 0, -8, 2]

Then X Y = 49 + 50 + (-3)(-8) + 72 = 74

(weighted sums in ANNs are dot products)

Some Equations

1 posxW

xW

+

++

++

-

-

--

--

-Separating Plane

For all positive examples

1 negxW

For all negative examples

weights input features threshold

These 1’s result from dividingthrough by a constant for convenience (it is the distance from the dashed lines to the green line)

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 8

Idea #1: The Margin(derivation not on final)

)cos(|| || || || )( BABA xxWxxW

xi 2)( BA xxW

1 AxW

1 BxW

Subtracting (ii) from (i) gives

(i)

(ii)

(iii)

WxA

xB

mar

gin

= 1 sinceparallel lines(iv)

The green line is the set of all pts that satisfy this equation (ditto for red line)

Combining (iii) and (iv) we get||W ||

2 || || BA xx

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 9

xj

Our Initial ‘Mathematical Program’

min ||w|| (this is the ‘1-norm’ length of the weight vector, which is the sum of the absolute values of the weights;

some SVMs use quadratic programs, but 1-norms have some preferred properties)

such that

w · xpos ≥ + 1 // for ‘+’ ex’s

w · xneg ≤ – 1 // for ‘–’ ex’s

w,

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 10

1

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

The ‘p’ Norm – Generalization of the Familiar Euclidean Distance (p=2)

11/10/15 11

Our Mathematical Program (cont.)

Note: w and are our adjustable parameters (we could, of course, use the ANN ‘trick’ and move to the left side of our inequalities and treat as another weight)

We can now use existing math programming optimization s/w to find a sol’n to our current program (covered in cs525)

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 12

Idea #2: Dealing with Non-Separable Data

• We can add what is called a ‘slack’ variable to each example

• This variable can be viewed as = 0 if example correctly separatedelse = ‘distance’ we need to move ex to get it correct (ie, distance from decision boundary)

• Note: we are NOT counting #misclassified would be nice to do so, but that becomes [mixed] integer programming, which is much harder

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 13

min ||w||1 + μ ||S||1

such that

w · xposi + Si ≥ + 1

w · xnegj – Sj ≤ – 1

Sk ≥ 0

The Math Program with Slack Vars (this is the linear-programming version; there also is a quadratic-prog version - in cs540 we won’t worry about the difference)

w, s,

Dim = # of input features

Dim = # of training

examples

scalar

Scaling constant (use tuning set to

select value)

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

The S’s are how far we would need to move an example in order for it to be on the proper side of the decision surface

14

Notice we are solving the perceptron task with acomplexity penalty (sum of wgts) – Hinton’s wgt decay!

Slack’s and Separability

• If training data is separable, will all Si = 0 ?

• Not necessarily!

– Might get a larger margin by misclassifying a few examples (just like in d-tree pruning)

– This can also happen when using gradient-descent to minimize an ANN’s cost function

11/10/15 CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10 15

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Brief Intro to Linear Programs (LP’s) - not on final

• We need to convert our task into

A z ≥ bwhich is the basic form of an LP (A is a constant matrix, b is a constant vector, z is a vector of variables)

• Note Can convert inequalities containing ≤ into ones using ≥ by multiplying both sides by -1 eg, 5x ≤ 15 same as -5x ≥ -15

• Can also handle = (ie, equalities) could use ≥ and ≤ to get =, but more efficient methods exist

11/10/15 16

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Brief Intro to Linear Programs (cont.) - not on final

In addition, we want to

min c z under the linear Az ≥ b constraints

Vector c says how to penalize settings for variables in vector zHighly optimized s/w for solving LP exists(eg, CPLEX, COINS [free])

11/10/15

Yellow region are those points that satisfy the constraints; dotted lines are iso-cost lines

Lecture #21, Slide 17

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Review: Matrix Multiplication

11/10/15 18

From (code also there): http://www.cedricnugteren.nl/tutorial.php?page=2

A B = C

Matrix A is K by M

Matrix B is N by K

Matrix C is M by N

f

e/2

e/2

1

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Aside: Our SVM as an LP (not on

final)

Apos 1 0 -1 0

-Aneg 0 1 1 0

0 1 0 0 0

0 0 1 0 0

-1 0 0 0 1

1 0 0 0 1The 1’s are identity matrices (often written as I)

| f | e/2 | e/2 | 1 | f |

e/2

e/2

e/2

e/2

f

f

≥

e/2

e/2

e

f

f

W

Spos

Sneg

Z

1

1

0

0

0

Let Apos = our positive training examples

Aneg = our negative training examples (assume 50% pos and 50% neg for notational simplicity)

#features

#exa

mpl

es

11/10/15 19

f

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Our C Vector (determines the cost we’re minimizing, also not on final)

min [ 0 μ 0 1 ] W

S

Z

= min μ ● S + 1 ● Z

Aside: could also penalize (but would need to add more

variables since can be negative)

CNote we min Z’s not

W’s since only Z’s ≥ 0

= min μ ||S||1 + ||W||1since all S are non-negative and

the Z’s ‘squeeze’ the W’s

11/10/1520

Note here: S = Spos concatenated with Sneg

CS 540 - Fall 2015 (Shavlik©), Lecture 22, Week 10

Where We are so Far

• We have an ‘objective’ function that we can optimize by Linear Programming– min ||w||1 + μ ||S||1 subject to some constraints

– Free LP solvers exist– CS 525 teaches Linear Programming

• We could also use gradient descent– Perceptron learning with ‘weight decay’

quite similar, though uses SQUARED wgts and SQUARED error (the S is this error)

11/10/15 21