Kernel Methods Arie Nakhmani. Outline Kernel Smoothers Kernel Density Estimators Kernel Density...

transcript

Kernel Methods

Arie Nakhmani

Outline Kernel Smoothers Kernel Density Estimators Kernel Density Classifiers

Kernel Smoothers – The Goal Estimating a function by

using noisy observations, when the parametric model for this function is unknown

The resulting function should be smooth

The level of “smoothness” should be set by a single parameter

( ) : pf X

Example

0 1 2 3 4 5 6-2

N=100 sample points

What is it: “smooth enough” ?

0 1 2 3 4 5 6-2

Example

Y in X , X ~ U 0,2 , ~ (0,1/ 4)s N

N=100 sample points

Exponential Smoother

0 1 2 3 4 5 6-2

ˆ ˆ( ) (1 ) ( 1) ( )ˆ(1) (1) 0 1

sorted

Y i Y i Y i

Smaller smoother line, but more delayed

Exponential Smoother

Simple Sequential Single parameter Single value memory Too rough Delayed

ˆ ˆ( ) (1 ) ( 1) ( )sortedY i Y i Y i

Moving Average Smoother

Y(1) = Y (1)

Y(2) = (Y (1) + Y (2) + Y (3))/3

Y(3) = (Y (1) + Y (2) + Y (3) + Y (4) + Y (5))/5

Y(4) = (Y (2) + Y (3) + Y (4) + Y

sorted

sorted sorted sorted

sorted sorted sorted sorted sorted

sorted sorted sorted sort

(5) + Y (6))/5...

ed sorted

Moving Average Smoother

0 1 2 3 4 5 6-2

Larger m smoother, but straightened line

Moving Average Smoother Sequential Single parameter: the window size

m Memory for m values Irregularly smooth What if we have p-dimensional

problem with p>1 ???

0 1 2 3 4 5 6-2

YNearest Neighbors Smoother

0 0ˆ( ) | ( )i i mY x Average y x Neighborhood x

ˆ( )Y x

Larger m smoother, but biased line

Nearest Neighbors Smoother Not sequential Single parameter: the number of

neighbors m Trivially extended to any number of

dimensions Memory for m values Depends on metrics definition Not smooth enough Biased end-points

Low Pass Filter

0 1 2 3 4 5 6-2

2nd order Butterworth: 2

22 1( ) 0.0078

3 0.77z zH z

Why do we need kernel smoothers ???

Low Pass Filter

0 1 2 3 4 5 6-2

The same filter…for log function

Low Pass Filter Smooth Simply extended to any number of

dimensions Effectively, 3 parameters: type,

order, and bandwidth Biased end-points Inappropriate for some functions

(depends on bandwidth)

Kernel Average Smoother

0 1 2 3 4 5 6-2

0ˆ( )Y x

0 0ˆ( ) |i i iY x Average w y x x

Kernel Average Smoother Nadaraya-Watson kernel-weighted average:

with the kernel:

for Nearest Neighbor Smoother for Locally Weighted Average

( , )ˆ( )

K x x yY x

( , )( )

x xK x x D

0( )h x 0 0 [ ]( )m mh x x x

Popular Kernels Epanechnikov kernel:

Tri-cube kernel:

Gaussian Kernel:

23(1 ) / 4 1( )0,

t if tD totherwise

3 3(1 ) 1( )0,

t if tD totherwise

21( ) ( ) exp22tD t t

-3 -2 -1 0 1 2 3

EpanechnikovTri-cubeGaussian

Non-Symmetric Kernel Kernel example:

Which kernel is that ???

(1 ) , 0, 0 1( )0,

t tD totherwise

2 11 2 1

ˆ(1 )

ˆ (1 ) (1 ) ... (1 )

ii i i i

Y Y Y Y Y

0(1 ) 1i

-3 -2 -1 0 1 2 30

Kernel Average Smoother Single parameter: window width Smooth Trivially extended to any number

of dimensions Memory-based method – little or

no training is required Depends on metrics definition Biased end-points

Local Linear Regression Kernel-weighted average

minimizes:

Local linear regression minimizes:

20 0 0 0

ˆmin ( , ) ( ) ( ) ( )N

i ix i

K x x y x Y x x

20 0 0

( ), ( ) 1

0 0 0 0

min ( , ) ( ) ( )

ˆ( ) ( ) ( )

i i ix x i

K x x y x x x

Y x x x x

Local Linear Regression Solution:

where:

Other representation:

10 0 0 0

ˆ( ) 1, ( ) ( )T TY x x x x

B W B B W y

1 1 ... 1...

Nx x x

0 0( ) ( , )i N Nx diag K x x W

ˆ( ) ( )N

Y x l x y

equivalent kernel01

( ) 1N

Local Linear Regression

0 1 2 3 4 5 6-2

0ˆ( )Y x

Equivalent Kernels

Local Polynomial Regression Why stop at local linear fits? Let’s minimize:

0 0 0( ), ( ), 1,..., 1 1

0 0 0 01

min ( , ) ( ) ( )

ˆ( ) ( ) ( )

i i j ix x j d i j

K x x y x x x

Y x x x x

Local Polynomial Regression

Variance Compromise 2 2 2

0 0ˆ( ) ( ) ( ) ; , 0i iVar Y x l x for y f x Var E

0.2 tri-cube kernel

Conclusions Local linear fits can help bias dramatically at

the boundaries at a modest cost in variance. Local linear fits more reliable for extrapolation.

Local quadratic fits do little at the boundaries for bias, but increase the variance a lot.

Local quadratic fits tend to be most helpful in reducing bias due to curvature in the interior of the domain.

λ controls the tradeoff between bias and variance. Larger λ makes lower variance but higher bias

Local Regression in Radial kernel:

0( , )

( )x x

K x x Dh x

0 0 0( ) 1

2 21 2 1 2 1 2

ˆ( ) arg min ( , ) ( ) ( )

( ) 1, , , , , ...

ˆˆ( ) ( ) ( )

i i ix i

x K x x y b x x

b X X X X X X X

Y x b x x

Popular Kernels Epanechnikov kernel

Tri-cube kernel

Gaussian kernel

23(1 ) / 4, 1( )0,

t if tD t

otherwise

331 , 1( )0,

t if tD totherwise

21( ) ( ) exp22tD t t

Example

0 1 2 3 4 5 6 70

sin( )~ [0,2 ]

~ (0,0.2)

Z XX U

Higher Dimensions The boundary estimation is problematic

Many sample points are needed to reduce the bias

Local regression is less useful for p>3 It’s impossible to maintain localness

(low bias) and sizeable samples (low variance) at the same time

Structured Kernels Non-radial kernel:

Coordinates or directions can be downgraded or omitted by imposing restrictions on A.

Covariance can be used to adapt a metric A. (related to Mahalanobis distance)

Projection-pursuit model

0 0, 0

( ) ( )( , ) ; 0

Tx x x xK x x D

Structured Regression Divide into a set (X1,X2,…,Xq) with

q<p and the remainder of the variables collect in vector Z.

Conditionally linear model:

For given Z fit a model by locally weighted least squares:

1 1( ) ( ) ( ) ... ( )q qf X Z Z X Z X

20 0 1 1 0 0

( ), ( ) 1min ( , ) ( ) ( ) ... ( )

i i i qi qz z i

K z z y z x z x z

Density Estimation

-10 -5 0 5 10 15 200

original distribution

constant window estimation

sample set0

0# Neighbor(x )ˆ ( ) i

Mixture of two normal distributions

6000.3

Kernel Density EstimationSmooth Parzen estimate: 0 0

1ˆ ( ) ( , )N

f x K x xN

Comparison

-10 -5 0 5 10 15 200

Data SamplesNearest NeighborsEpanechnikovGaussian

Mixture of two normal distributions

Usually Bandwidth selection is more important than kernel function selection

Kernel Density Estimation Gaussian kernel density estimation:

where denote the Gaussian density with mean zero and standard deviation .

Generalization to :

1ˆ ˆ( ) ( ) * ( )N

f x x x F xN

20 02 /2

1ˆ ( ) exp 0.5 /(2 )

f x x xN

Kernel Density Classification

ˆˆ ( )Pr | ; 1,...,

ˆˆ ( )

f xG j X x j J

For a J class problem:

Pr( | )jf x

X x G j

Radial Basis Functions Function f(x) is represented as expansion in

basis functions:

Radial basis functions expansion (RBF):

where the sum-of-squares is minimized with respect to all the parameters (for Gaussian kernel):

1( ) ( )Mj jjf x h x

( ) ,j

M M jj j

xf x K x D

0 1 2{ , , } 1

( ) ( )min exp

Mj j j

TN M i j i ji jj

Radial Basis Functions When assuming constant j= : the problem of

“holes”

The solution - Renormalized RBF: 1

D xh x

Additional Applications Local likelihood Mixture models for density

estimation and classification Mean-shift

Conclusions Memory-based methods: the model is the

entire training data set Infeasible for many real-time applications Provides good smoothing result for

arbitrary sampled function Appropriate for interpolation and

extrapolation When the model is known, better use

another fitting methods

Kernel Methods Arie Nakhmani. Outline Kernel Smoothers Kernel Density Estimators Kernel Density...

Documents