Home >Documents >Kernel Methods Dept. Computer Science & Engineering, Shanghai Jiao Tong University

Kernel Methods Dept. Computer Science & Engineering, Shanghai Jiao Tong University

Date post:19-Jan-2016
Category:
View:225 times
Download:0 times
Share this document with a friend
Transcript:
  • Kernel MethodsDept. Computer Science & Engineering, Shanghai Jiao Tong University

    Kernel Methods

  • *Kernel Methods*OutlineOne-Dimensional Kernel SmoothersLocal RegressionLocal LikelihoodKernel Density estimationNaive BayesRadial Basis FunctionsMixture Models and EM

    Kernel Methods

  • *Kernel Methods*One-Dimensional Kernel Smoothersk-NN:

    30-NN curve is bumpy, since is discontinuous in x.The average changes in a discrete way, leading to a discontinuous .

    Kernel Methods

  • *Kernel Methods*Nadaraya-Watson Kernel weighted average:

    Epanechnikov quadratic kernel:One-Dimensional Kernel Smoothers

    Kernel Methods

  • *Kernel Methods*One-Dimensional Kernel SmoothersMore general kernel:

    : width function that determines the width of the neighborhood at x0.For quadratic kernelFor k-NN kernel Variance constantThe Epanechnikov kernel has compact support

    Kernel Methods

  • *Kernel Methods*Three popular kernel for local smoothing:

    Epanechnikov kerneland tri-cube kernel are compact but tri-cube has two continuous derivativesGaussian kernel is infinite supportOne-Dimensional Kernel Smoothers

    Kernel Methods

  • *Kernel Methods*Boundary issueBadly biased on the boundaries because of the asymmetry of the kernel in the region.Linear fitting remove the bias to first orderLocal Linear Regression

    Kernel Methods

  • *Kernel Methods*Local Linear RegressionLocally weighted linear regression make a first-order correctionSeparate weighted least squares at each target point x0:

    The estimate:b(x)T=(1,x); B: Nx2 regression matrix with i-th row b(x)T;

    Kernel Methods

  • *Kernel Methods*Local Linear RegressionThe weights combine the weighting kernel and the least squares operationsEquivalent Kernel

    Kernel Methods

  • *Kernel Methods*The expansion for , using the linearity of local regression and a series expansion of the true function f around x0

    For local regression The bias depends only on quadratic and higher-order terms in the expansion of .Local Linear Regression

    Kernel Methods

  • *Kernel Methods*Local Polynomial RegressionFit local polynomial fits of any degree d

    Kernel Methods

  • *Kernel Methods*Local Polynomial RegressionBias only have components of degree d+1 and higher.The reduction for bias costs the increased variance.

    Kernel Methods

  • *Kernel Methods* k- k/N-

    Kernel Methods

  • *Kernel Methods*Structured Local RegressionStructured kernels

    Introduce structure by imposing appropriate restrictions on AStructured regression function

    Introduce structure by eliminating some of the higher-order terms

    Kernel Methods

  • *Kernel Methods*Any parametric model can be made local:Parameter associated with :Log-likelihood:Model likelihood local to :

    A varying coefficient model Local Likelihood & Other Models

    Kernel Methods

  • *Kernel Methods*Logistic Regression

    Local log-likelihood for the J class model

    Center the local regressions at Local Likelihood & Other Models

    Kernel Methods

  • *Kernel Methods*A natural local estimate

    The smooth Parzen estimate

    For Gaussian kernel The estimate becomeKernel Density Estimation

    Kernel Methods

  • *Kernel Methods*Kernel Density EstimationA kernel density estimate for systolic blood pressure. The density estimate at each point is the average contribution from each of the kernels at that point.

    Kernel Methods

  • *Kernel Methods*Bayes theorem:

    The estimate for CHD uses the tri-cube kernel with k-NN bandwidth.

    Kernel Density Classification

    Kernel Methods

  • *Kernel Methods*Kernel Density ClassificationThe population class densities and the posterior probabilities

    Kernel Methods

  • *Kernel Methods*Nave BayesNave Bayes model assume that given a class G=j, the features Xk are independent:

    is kernel density estimate, or Gaussian, for coordinate Xk in class j.If Xk is categorical, use Histogram.

    Kernel Methods

  • *Kernel Methods*Radial Basis Function & KernelRadial basis function combine the local and flexibility of kernel methods.

    Each basis element is indexed by a location or prototype parameter and a scale parameter , a pop choice is the standard Gaussian density function.

    Kernel Methods

  • *Kernel Methods*Radial Basis Function & KernelFor simplicity, focus on least squares methods for regression, and use the Gaussian kernel.RBF network model:

    Estimate the separately from the .A undesirable side effect of creating holesregions of IRp where none of the kernels has appreciable support.

    Kernel Methods

  • *Kernel Methods*Renormalized radial basis functions.

    The expansion in renormalized RBFRadial Basis Function & Kernel

    Kernel Methods

  • *Kernel Methods*Mixture Models & EMGaussian Mixture Model

    are mixture proportions, EM algorithm for mixturesGiven log-likelihood:

    Suppose we observe Latent Binary

    BadGood

    Kernel Methods

  • *Kernel Methods*Mixture Models & EMGiven ,compute

    In Example

    Kernel Methods

  • *Kernel Methods*Mixture Models & EMApplication of mixtures to the heart disease risk factor study.

    Kernel Methods

  • *Kernel Methods*Mixture Models & EMMixture model used for classification of the simulated data

    Kernel Methods

Popular Tags:

Click here to load reader

Embed Size (px)
Recommended