Machine Learning Algorithms (IFT6266 A07)Prof. Douglas Eck, Université de Montréal

These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning

by Christopher Bishop

1

A note: Methods

• We (perhaps unwisely) skipped Bishop 2.1--2.4 until just before our graphical models lectures

• Give it a look. . . .

2

2

Nonparametric Methods

• Parametric methods: fit a small number of parameters to a data set.

• Must make assumptions about the underlying distribution of data.

• If those assumptions are wrong, model is flawed. Eg. trying to fit Gaussian model to multimodal data.

• Nonparametric methods: fit a large number of parameters to a data set.

• Number of parameters scales with number of data points.

• In general one parameter per data point

• Main advantage is that we need make only very weak assumptions about the underlying distribution of data.

3

3

Histograms

4

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

pi =ni

N!i!p(x) dx = 1

4

Histograms

• Discontinuities at bin boundaries

• Curse of dimensionality: if we divide each variable in a D-dimensional space into M bins, we get MD bins

• Locality is important (density defined via evaluation of points in a local neighborhood)

• Smoothing achieved by bin count; we want neither “too much” nor “too little” smoothing

• Relates to model complexity and regularization in parametric modeling

5

5

Kernel density estimators• Assume Euclidean distance, observe from unknown density p(x)

• Consider small region R containing x. Mass for region is:

• Distribution of points is binomial:

• From chp 2.1 on binary variables:

6

P =!

Rp(x) dx

Bin(K|N,P ) =N !

K!(N !K)!PK(1! P )N!K

p(x = 1|µ) = µ

Bin(m|N,µ) =!

N

m

"µm(1! µ)N!m

!N

m

"! N !

(N "m)!m!

[m] !N!

m=0

mBin(m|N,µ) = Nµ

var[m] !N!

m=0

(m" [m])2Bin(m|N,µ) = Nµ(1" µ)

6

Kernel density estimators

• Thus we see that mean fraction of points falling into region is E[K/N]=P

• Variance is var[K/N] = P(1-P) / N

• For large N, distribution will be sharply peaked around the mean so K ≅ NP

• If region R is sufficiently smally that the density p(x) is constant then we have P ≅ p(x)V where V is the volume of R

• Combining we get p(x) = K / NV

• Depends on contradictory assumptions (that R is small enough to have constant density over region yet sufficiently large such that the number K of points falling into the region generates a sharply peaked distribution

• If we fix K and determine V we get K-nearest-neighbors

• If we fix V and determine K we get kernal density estimator

7

7

Kernel density estimators

• Wish to determine density around x in region R centered on x.

• We will count points:k(u) = 1, |ui|≤ .5 i = 1,..., D 0, otherwise

• k(u) is a kernel function, here called a Parzen window

• Total number of data points inside of cube of side h defined by k(u) :

• Substitution into p(x) = K / NV (where volume of hypercube of side h in D dimeinsions is V = 1/hd)

8

K =N!

n=1

k

"x! xn

h

#

P (X) =1N

N!

n=1

1hD

k

"x! xn

h

#

8

Parzen estimator

• Model suffers from discontinuities at hypercube boundaries

• Substitute Gaussian (where h represents standard deviation of Gaussian components):

• As expected, h acts to smoothTradeoff between noise sensitivityand oversmoothing

• Any kernel can be used provided:k(u) ≥ 0 m, integral k(u)du = 1

• No computation for training phase

• But must store entire data set to evaluate

9

P (X) =1N

N!

n=1

1(2!h2)1/2

exp"! ||x! xn||2

2h2

#

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

9

Nearest neighbor methods• One weakness of kernel density estimation is that kernel width h is fixed

for all lerneks.

• Fix K and determine V

• Consider small sphere centered at x, allow sphere to grow until it contains precisely K points. Estimate of density p(x) given by same formula: P(X) = K / NV

• K governs degree of smoothing. Compare Parzens (left) to KNN (right)

10

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

0 0.5 10

5

10

Classification using KNN

• Can do classification by applying KNN to each class and applying Bayes:

• To classify new point x, draw a sphere containning precisely K points.

• Suppose sphere contains Kk points from class Ck. Then model p(x) = K/NV provides a density estimate:

• We also obtain unconditional density:

• With class priors:

• Applying Bayes’ theorem yields:

11

p(x|Ck) =Kk

NkV

p(x) =K

NV

p(Ck) =Nk

N

p(Ck|x) =p(x|Ck)p(Ck)

p(x)=

Kk

K

11

Classification using KNN

• To minimize misclassification rate, always choose class with largest Kk/K. For K=1, yields decision boundary composed of hyperplanes that form perpendicular bisectors of pairs from different classes

12

x1

x2

(a)x1

x2

(b)

Left: K=3, Right: K=1

12

KNN Example

• K acts as regularizer.

• Tree-base search can be used to find approximate near neighbors.

13

Oil dataset. Left K=1, Middle K=3, Right K=31

0 1 20

1

2

0 1 20

1

2

0 1 20

1

2

13