Fisher’s linear disc.

Fisher’s Linear DiscriminantIntuition

Write this in terms of w

We are looking for a good projection

So,

Next,

Define the scatter matrix:

Then,

Thus the denominator can be written:

Rayleigh QuotientMaximizing equivalent tosolving:

With solution:

Margins in data space

b

Larger margins promote uniqueness forunderconstrained problems

– In conclusion, a linear discriminant function dividesthe feature space by a hyperplane decision surface

– The orientation of the surface is determined by thenormal vector w and the location of the surface isdetermined by the bias

†

x = xp +rww

since g(xp ) = 0 and wtw = w 2

g(x) = wtx + w0 fi wt xp +rww

Ê

Ë Á

ˆ

¯ ˜ + w0

= g(xp ) + wtw rw

fi r =g(x)

w

in particular d([0,0],H) =w 0

w

H

w

x

xt w

r

xp

Nomenclature• Given x1, x2,…, xn sample points, with true category labels:

y1, y2,…,yn

• Decision are made according to:

• Now these decisions are wrong when wtxi is negative andbelongs to class w1.Let zi = ai xi Then zi >0 when correctly labelled,negative otherwise.†

if wt x i' = w t xi + b > 0 class w1 is chosen

if wt x i' = w t xi + b < 0 class w2 is chosen†

yi =1yi = -1

¸ ˝ ˛

if point xi is from class w1

if point xi is from class w2

Constrained Optimization ProblemsMinimize enforcing Equality ConstraintsFind: such that

Lagrange Multiplier

†

r x * =r x min

†

h( r x *)=0

),(min 21 xxf

†

s.t. h(x1 , x2) = 0

),(),(),,( 212121 xxhxxfxxL uu += ˜̃¯

ˆÁÁË

Ê

multiplierLagrange

funcLagrangeL

:

:

u

0),(),(),(

0),(),(),(

2

*2

*1

2

*2

*1

2

*2

*1

1

*2

*1

1

*2

*1

1

*2

*1

=∂

∂+

∂

∂=

∂

∂

=∂

∂+

∂

∂=

∂

∂

x

xxf

x

xxf

x

xxL

x

xxf

x

xxf

x

xxL

u

u

At the candidate minimum point, gradients of the cost andconstraint func are along the same line.

(In other words, is a linear combination of

Therefore constrained optimization is converted tounconstrained optimization.

†

—L(x*) = —f (x*) + n—h(x*) = 0—f (x*) = -n —h(x*) geometrical meaning

†

—f

†

—L(x*,n *) = 0

†

L(x,n ) = f (x) +n T h(x)

†

—h

• The "Milkmaid problem"• It's milking time at the farm, and the

milkmaid has been sent to the field to getthe day's milk. She is in quite a hurry,because she has a date, so she wants tofinish her job as quickly as possible.However, before she gathers the milk,she has to rinse out her bucket in thenearby river.

• Just when she reaches point M, ourheroine spots the cow, at point C. She isin a hurry, so she wants to take theshortest possible path from where she isto the river and then to the cow. If thenear bank of the river is a curvesatisfying the function g(x,y) = 0, what isthe shortest path for the milkmaid totake? (Assume that the field is flat anduniform and that all points on the riverbank are equally good.)

• Problem:• Minimize f(P) = d(M,P) + d(P,C),

– such that g(P) = 0.

F(P,a) = f(P) - a g(P).

†

—F = 0∂f∂P

+ a∂g∂P

= 0∂F∂a

= 0 Æ g(P) = 0