Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 11
Support Vector Regression
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 22
SVR
Drawings and illustrations from Bernhard Schölkopf, and Alex Smola: Learning with Kernels (MIT Press, Cambridge, MA, 2002)
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 33
SVR - HistoryBased on Learning Theory,
consisting of few axioms on learning errors
Started in 1960’s, still actively developed
SVRs recently outperformed NNs in recognition tests on US Postal Service’s standard set of handwritten characters
libSVM by Chih-Chung Chang and Chih-Jen Lin provides fast and simple to use implementation, extended as requests (e.g. from HEP) come in
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 44
Training sample X, observed results YGoal: f with y=f(x)
Simplicity: • Linear case,•
Formulation of Problem
miyxf
bxwxf
ii ,,1
,
1 ,1y
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 55
Optimal confidence = maximal margin
Minimize quadratic problem
with Quadratic problem: Unique solution!
Optimizing the Confidence
m
i iiim
i ii
m
ji jijjii
y
xx
1
*
1
*
1,
** ,21
m
i iiim
i ii xyw11
* ;0
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 66
Non-Linearity
bxxkbxwxf
xxkxxxxm
i ii
, ,
,, ,
1
*
:Introduce mapping to higher dimensional space
e.g. Gaussian kernel:
2
2
2exp,
xx
xxk
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 77
Calculation
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 88
L2 b Tagger Parameters
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 99
L2 b Tagger Parameters
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1010
L2 b Tagger Output
SVR NN
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1111
L2 b Tagger Discussion • Complex problem increases number of SVs• Almost non-separable classes still almost non-
separable in high dimensional space• High processing time due to large number of
SVs
NNs show better performance for low-information, low-separability problems
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1212
Higgs Parameters
Higgs SVR analysis by Daniel Whiteson, UC Berkley
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1313
Higgs Parameters
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1414
Higgs Output
Background Signal Background Signal
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1515
Higgs Purity / EfficiencyPurity
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1616
Kernel Width
k
kkji
ji
xx
xx
2
22
2/exp
: widthsionalmultidimen
2/exp
:original
Kernel Width
Inte
grat
ed S
igni
fican
ce
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1717
SummarySVR often superior to NN• Not stuck in local minima: unique solution• Better performance for many problemsImplementation exists, actively supported by the
development community
Further information: www.kernel-machines.org
Time for SVR @ HEP!
Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The
NetherlandsNetherlands
June 24, 2002June 24, 2002ACAT02, MoscowACAT02, Moscow 1818
L2 b Tagger Correlation
b udcsSVR
SVR
NN
NN
