Kernel MethodsDept. Computer Science & Engineering, Shanghai Jiao Tong University

Kernel Methods

*Kernel Methods*OutlineOne-Dimensional Kernel SmoothersLocal RegressionLocal LikelihoodKernel Density estimationNaive BayesRadial Basis FunctionsMixture Models and EM

Kernel Methods

*Kernel Methods*One-Dimensional Kernel Smoothersk-NN:

30-NN curve is bumpy, since is discontinuous in x.The average changes in a discrete way, leading to a discontinuous .

Kernel Methods

*Kernel Methods*Nadaraya-Watson Kernel weighted average:

Epanechnikov quadratic kernel:One-Dimensional Kernel Smoothers

Kernel Methods

*Kernel Methods*One-Dimensional Kernel SmoothersMore general kernel:

: width function that determines the width of the neighborhood at x0.For quadratic kernelFor k-NN kernel Variance constantThe Epanechnikov kernel has compact support

Kernel Methods

*Kernel Methods*Three popular kernel for local smoothing:

Epanechnikov kerneland tri-cube kernel are compact but tri-cube has two continuous derivativesGaussian kernel is infinite supportOne-Dimensional Kernel Smoothers

Kernel Methods

*Kernel Methods*Boundary issueBadly biased on the boundaries because of the asymmetry of the kernel in the region.Linear fitting remove the bias to first orderLocal Linear Regression

Kernel Methods

*Kernel Methods*Local Linear RegressionLocally weighted linear regression make a first-order correctionSeparate weighted least squares at each target point x0:

The estimate:b(x)T=(1,x); B: Nx2 regression matrix with i-th row b(x)T;

Kernel Methods

*Kernel Methods*Local Linear RegressionThe weights combine the weighting kernel and the least squares operationsEquivalent Kernel

Kernel Methods

*Kernel Methods*The expansion for , using the linearity of local regression and a series expansion of the true function f around x0

For local regression The bias depends only on quadratic and higher-order terms in the expansion of .Local Linear Regression

Kernel Methods

*Kernel Methods*Local Polynomial RegressionFit local polynomial fits of any degree d

Kernel Methods

*Kernel Methods*Local Polynomial RegressionBias only have components of degree d+1 and higher.The reduction for bias costs the increased variance.

Kernel Methods

*Kernel Methods* k- k/N-

Kernel Methods

*Kernel Methods*Structured Local RegressionStructured kernels

Introduce structure by imposing appropriate restrictions on AStructured regression function

Introduce structure by eliminating some of the higher-order terms

Kernel Methods

*Kernel Methods*Any parametric model can be made local:Parameter associated with :Log-likelihood:Model likelihood local to :

A varying coefficient model Local Likelihood & Other Models

Kernel Methods

*Kernel Methods*Logistic Regression

Local log-likelihood for the J class model

Center the local regressions at Local Likelihood & Other Models

Kernel Methods

*Kernel Methods*A natural local estimate

The smooth Parzen estimate

For Gaussian kernel The estimate becomeKernel Density Estimation

Kernel Methods

*Kernel Methods*Kernel Density EstimationA kernel density estimate for systolic blood pressure. The density estimate at each point is the average contribution from each of the kernels at that point.

Kernel Methods

*Kernel Methods*Bayes theorem:

The estimate for CHD uses the tri-cube kernel with k-NN bandwidth.

Kernel Density Classification

Kernel Methods

*Kernel Methods*Kernel Density ClassificationThe population class densities and the posterior probabilities

Kernel Methods

*Kernel Methods*Nave BayesNave Bayes model assume that given a class G=j, the features Xk are independent:

is kernel density estimate, or Gaussian, for coordinate Xk in class j.If Xk is categorical, use Histogram.

Kernel Methods

*Kernel Methods*Radial Basis Function & KernelRadial basis function combine the local and flexibility of kernel methods.

Each basis element is indexed by a location or prototype parameter and a scale parameter , a pop choice is the standard Gaussian density function.

Kernel Methods

*Kernel Methods*Radial Basis Function & KernelFor simplicity, focus on least squares methods for regression, and use the Gaussian kernel.RBF network model:

Estimate the separately from the .A undesirable side effect of creating holesregions of IRp where none of the kernels has appreciable support.

Kernel Methods

*Kernel Methods*Renormalized radial basis functions.

The expansion in renormalized RBFRadial Basis Function & Kernel

Kernel Methods

*Kernel Methods*Mixture Models & EMGaussian Mixture Model

are mixture proportions, EM algorithm for mixturesGiven log-likelihood:

Suppose we observe Latent Binary

BadGood

Kernel Methods

*Kernel Methods*Mixture Models & EMGiven ,compute

In Example

Kernel Methods

*Kernel Methods*Mixture Models & EMApplication of mixtures to the heart disease risk factor study.

Kernel Methods

*Kernel Methods*Mixture Models & EMMixture model used for classification of the simulated data

Kernel Methods