NIPS 2000 Workshop on Kernel methods Frame, Reproducing Kernel and Learning Alain Rakotomamonjy...

transcript

NIPS 2000 Workshop on Kernel methods

Frame, Reproducing Kernel and Learning

Alain RakotomamonjyStéphane Canu

http://asi.insa-rouen.fr/~arakotomAlain.Rakoto,Stephane.Canu@insa-rouen.fr

Perception, Systèmes et InformationInsa de Rouen,76801 St Etienne du RouvrayFrance

Motivations

Wavelet-based approximation (wavelet or ridgelet networks) are regularization networks?

Construction of multiresolution scheme of approximation

kernel adapted to the structures of function to be learned

Motivations Ctd.

Frame based framework for learning

Approximating highly oscillating structure Without losing

regularity in smooth region

Road Map

Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives

Frame : A definition

H : Hilbert Space dot product nn A sequence of elements of H

nn is a frame of H if there exists A,B > O s.t

HfB,ffAHf

A,B are the frame bounds

Frame : definition Ctd.

Frame intepretation

Frame allows stable representationas for all f in H

n*n,ff

Frame = "Basis" + linear dependency + redundancy

n*n being a dual frame of n in H

Particular cases of Frame

Tight FrameFrame with bounds s.t A=B

Orthonormal BasisA=B=1

Riesz BasisFrame elements are linearly independent

Hn f,f,Hf

]np[, n*p

Examples of Frame

Tight Frame of IR2

Frame of L2(IR)

23,f,IRf

2Zn,jk,j )t(

jk,j aanut

1)t( is an admissible wavelet

Road Map

Frameable RKHS

Condition for having a RKHSSuppose H is a Hilbert space of function IRIR n

nnand a frame of H

n*n )t()(,t

The Reproducing Kernel is

n*n )t()s()t,s(K

H is a RKHS if Htt fM)t(ft.s0M,t,Hf

On a frameable Hilbert Space, this is equivalent to

Construction of Frameable RKHS

A Practical way to build a RKHSF is a Hilbert Space of function IRIR:f n

N..1nn A finite set of F elements such that

Fn,N...1n

M)t(t,N...1n,IRM n

Fn ,,span is a RKHS with {n} as frame elements

Example of Frameable RKHS

frameable RKHS included in L2(IR)i : L2 function (e.g i is a wavelet) span {i}i=1…N is a RKHS

span a RKHS with kernel

*i )t()s()t,s(K

3 wavelets at same scale jExample

Road Map

Semiparametric EstimationContext Learning from training set (xi,yi)i=1..N

One looks for the minimizer of the risk functional 2

1iii f)x(f,yC

in a space H + span{i}i=1…m H being a RKHS

1iiHijj

* )x,x(Kb)x(a)x(f

Under general conditions,

Semiparametric framework

span{i}i=1…m : parametric hypothesis space

Semiparametric EstimationParametric hyp. space is a frameable

RKHSP is a frameable RKHS spanned by {n}, with P H, H RHKS

PPH PPKKK

1iii f)x(f,yC

One looks for the minimizer in H of

1iiPijj

* )x,x(Kb)x(a)x(f

As spaces are orthogonal, backfitting is sufficient for estimating f*

Semiparametric estimation on H with P as a parametric hyp. space

Semiparametric Estimation

H= P + N P : Frameable RKHS, N : Frameable RKHS

N: "unknown component" to be regularized

P : "known component" not to regularized

Frame view point H frameable

H defined by kernel K

NPH KN=KH-KP

P N : due to linear dependency of frame

P : Frameable RKHS

Multiscale approximation

H a frameable RKHS

1m...1iFHH 1i1ii

m1m10 HHHH And any space Hi or Fi is a RKHS

Hi : Trend Spaces Fi : Details Spaces

H is splitted in different spaces {Fi}i=1…m-1 and H0

Multiscale Approximation Ctd.

At each step j, trend obtained at step j-1 is decomposed in trend and details

ii y,x

k,2k,2d k

ik,2k,2 )x,x(Kc Resid.

k,1k,1d k

k,0k,0d k

Multiscale Approximation Ctd.

ValidityAt each step, representer Theorem

Hypothesis must be verified

Solution

Details

1iijj,i

)x(d)x,x(Kc)x(f

Illustration on toy problem

)2t(5))2t(5sin(

)5t())5t(sin()xsin()x(s

Function to be learnedData xi : N points from the random sampling of [0, 10]

),0(N)x(sy ii

Algorithm - SVM Regression- Multiscale Regularisation networks on Frameable RKHS

Sin/Sinc based kernelWavelet based kernel

Results

SVM Wavelet Kernel Sinc Kernel

L2 error

1 ± 0.096

1 ± 0.028

0.9297 ± 0.312

0.8280± 0.025

0.5115 ± 0.098

0.7252± 8.022

N=902 Results are averagerad over 300 experiments and

normalized with regards to SVM performance

Plots of typical results

Road Map

Summary new design of kernel based on frame

elements algorithm for multiscale learning

But no explicit definition of kernel

Time-consuming

Future work

Multidimensional extensionTight Frame of multidimensional wavelet

Using a priori knowledge on the learning problem

How to choose the frame elements?Theoretical justification and analysis

of multiscale approximation

NIPS 2000 Workshop on Kernel methods Frame, Reproducing Kernel and Learning Alain Rakotomamonjy...

Documents