Complex Support Vector Machines For Quaternary Classification

Introduction

Support Vector Machines

The Complex Case

Complex Support Vector Machines For

Quaternary Classification

P. Bouboulis, E. Theodoridou, S. Theodoridis

Department of Informatics and TelecommunicationsUniversity of Athens

Athens, Greece

23-09-2013

P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 1 / 47

Introduction


The Complex Case

Outline

1 Introduction

Reproducing Kernel Hilbert Spaces

Complex RKHS

2 Support Vector Machines

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case


Complex RKHS

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case


Complex RKHS

Reproducing Kernel Hilbert Spaces.

Consider a linear class H of real (complex) valued functions f

defined on a set X (in particular H is a Hilbert space), for which

there exists a function (kernel) κ : X × X → R(C) with the

following two properties:


Introduction


The Complex Case


Complex RKHS






1 For every x ∈ X , κ(x , ·) belongs to H.


Introduction


The Complex Case


Complex RKHS







2 κ has the so called reproducing property, i.e.,

f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)


Introduction


The Complex Case


Complex RKHS







2 κ has the so called reproducing property, i.e.,

f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)

Then H is called a Reproducing Kernel Hilbert Space (RKHS)

associated to the the kernel κ.


Introduction


The Complex Case


Complex RKHS

Kernel Trick

The notion of RKHS is a popular tool for treating non-linear

learning tasks.


Introduction


The Complex Case


Complex RKHS

Kernel Trick


learning tasks.

Usually this is attained by the so called “kernel trick”.


Introduction


The Complex Case


Complex RKHS

Kernel Trick


learning tasks.


If

X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,


Introduction


The Complex Case


Complex RKHS

Kernel Trick


learning tasks.


If

X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,

then the inner product in H is given as a function computed on

X :

κ(x , y) = 〈κ(x , ·), κ(y , ·)〉H kernel trick


Introduction


The Complex Case


Complex RKHS

Developing Learning Algorithms in RKHS

The black box approach.


Introduction


The Complex Case


Complex RKHS



Develop the learning Algorithm in X .


Introduction


The Complex Case


Complex RKHS




Express it, if possible, in inner products.


Introduction


The Complex Case


Complex RKHS





Choose a kernel function κ.


Introduction


The Complex Case


Complex RKHS





Choose a kernel function κ.Replace inner products with kernel evaluations according to

the kernel trick.


Introduction


The Complex Case


Complex RKHS





Choose a kernel function κ.Replace inner products with kernel evaluations according to

the kernel trick.

Work directly in the RKHS, assuming that the data have

been mapped and live in the RKHS H, i.e.,

X ∋ x → Φ(x) := κ(x , ·) ∈ H.


Introduction


The Complex Case


Complex RKHS

Advantages

Advantages of kernel-based learning tasks:


Introduction


The Complex Case


Complex RKHS

Advantages


The original nonlinear task is transformed into a linear one.


Introduction


The Complex Case


Complex RKHS

Advantages


The original nonlinear task is transformed into a linear one.

Different types of nonlinearities can be treated in a unified

way.


Introduction


The Complex Case


Complex RKHS

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case


Complex RKHS

Complex RKHS

Although the theory of RKHS holds for complex spaces

too, most of the kernel-based learning techniques were

designed to process real data only.


Introduction


The Complex Case


Complex RKHS

Complex RKHS




Moreover, in the related literature the complex kernel

functions have been ignored.


Introduction


The Complex Case


Complex RKHS

Complex RKHS






Recently, however, a unified kernel-based framework,

which is able to treat complex data, has been presented.


Introduction


The Complex Case


Complex RKHS

Complex RKHS








This machinery transforms the input data into a complex

RKHS, i.e.,

Φ(z) = κC(·, z).


Introduction


The Complex Case


Complex RKHS

Complex RKHS








This machinery transforms the input data into a complex

RKHS, i.e.,

Φ(z) = κC(·, z).

and employs the Wirtinger’s Calculus to derive the

respective gradients.


Introduction


The Complex Case


Complex RKHS

Complex RKHS

Definitions:

H denotes a complex RKHS.


Introduction


The Complex Case


Complex RKHS

Complex RKHS

Definitions:


H denotes a real RKHS.


Introduction


The Complex Case


Complex RKHS

Complex RKHS

Definitions:



The complex RKHS can be expressed as H = H + iH.


Introduction


The Complex Case


Complex RKHS

Complex RKHS

Definitions:



The complex RKHS can be expressed as H = H + iH.

H is isomorphic to the doubled real space H2.


Introduction


The Complex Case


Complex RKHS

Complex Kernels

The complex Gaussian kernel:

κ(z ,w) = exp

(

−∑d

i=1(zi−w∗i)2

σ2

)

,


Introduction


The Complex Case


Complex RKHS

Complex Kernels


κ(z ,w) = exp

(

−∑d

i=1(zi−w∗i)2

σ2

)

,

The Szego kernel: κ(z ,w) = 11−wHz

,


Introduction


The Complex Case


Complex RKHS

Complex Kernels


κ(z ,w) = exp

(

−∑d

i=1(zi−w∗i)2

σ2

)

,

The Szego kernel: κ(z ,w) = 11−wHz

,

Bergman kernel: κ(z,w) = 1(1−wHz)2 .


Introduction


The Complex Case


Complex RKHS

Wirtinger Calculus

Complex differentiability is a very strict notion.


Introduction


The Complex Case


Complex RKHS

Wirtinger Calculus


In learning tasks that involve complex data, we often

encounter functions (e.g., the cost functions, which are

defined in R) that ARE NOT complex differentiable.


Introduction


The Complex Case


Complex RKHS

Wirtinger Calculus





Example: f (z) = |z|2 = zz∗.


Introduction


The Complex Case


Complex RKHS

Wirtinger Calculus





Example: f (z) = |z|2 = zz∗.

In these cases one has to express the cost function in

terms of its real part fr and its imaginary part fi , and use

real derivation with respect to fr , fi .


Introduction


The Complex Case


Complex RKHS

Wirtinger’s Calculus

This approach leads usually to cumbersome and tedious

calculations.


Introduction


The Complex Case


Complex RKHS



calculations.

Wirtinger’s Calculus provides an alternative equivalent

formulation.


Introduction


The Complex Case


Complex RKHS



calculations.


formulation.

It is based on simple rules and principles.


Introduction


The Complex Case


Complex RKHS



calculations.


formulation.

It is based on simple rules and principles.

These rules bear a great resemblance to the rules of the

standard complex derivative.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

Suppose we are given training data, which belong to two

separate classes C+,C−,i.e.,

{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then

the n-th sample belongs to C+, while if dn = −1, then the n-th

sample belongs to C−.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

Suppose we are given training data, which belong to two

separate classes C+,C−,i.e.,

{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then

the n-th sample belongs to C+, while if dn = −1, then the n-th

sample belongs to C−.

For example:

Figure: Training points belonging to two classes.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

The goal of the SVM task is to estimate the maximum margin

hyperplane (wT x + c = 0), that separates the points of the two

classes as best as possible


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The primal problem




minimizew∈X ,c∈R

12‖w‖2

H+ C

N

N∑

n=1

ξn

subject to

{

dn

(

wT xn + c)

≥ 1 − ξn

ξn ≥ 0

for n = 1, . . . ,N,


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The primal problem




minimizew∈X ,c∈R

12‖w‖2

H+ C

N

N∑

n=1

ξn

subject to

{

dn

(

wT xn + c)

≥ 1 − ξn

ξn ≥ 0

for n = 1, . . . ,N,

Note that C is chosen a priori.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

Physical justification

minimizew∈X,c∈R

12‖w‖2

H+ C

N

N∑

n=1

ξn

subject to

{

dn

(

wT xn + c)

≥ 1 − ξn

ξn ≥ 0for n = 1, . . . , N,


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

Physical justification

minimizew∈X,c∈R

12‖w‖2

H+ C

N

N∑

n=1

ξn

subject to

{

dn

(

wT xn + c)

≥ 1 − ξn

ξn ≥ 0for n = 1, . . . , N,

Figure: Linear SVM


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The dual problem

To solve this task, usually we consider the dual problem derived

by the Lagrangian:

maximizea∈RN

N∑

n=1

an −1

2

N∑

n,m=1

anamdndmxTmxn

subject to

N∑

n=1

andn = 0 and an ∈ [0,C/N].


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick

Choose a positive definite kernel κR.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick


In the dual problem, replace the inner products xTn xm with

the respective kernel evaluations, i.e., κR(xn,xm).


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick




The application of the kernel trick leads to the nonlinear

SVM:


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick




The application of the kernel trick leads to the nonlinear

SVM:

maximizea∈RN

N∑

n=1

an −1

2

N∑

n,m=1

anamdndmκR(xm,xn)

subject to

N∑

n=1

andn = 0 and an ∈ [0,C/N].


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

Mapping to the feature space

The application of the kernel trick to the dual problem is

equivalent to the following procedure:


Introduction


The Complex Case

Linear SVMs

Non-linear SVM




Choose a positive definite kernel κR, that is associated to a

specific RKHS H.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM





specific RKHS H.

Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM





specific RKHS H.

Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.

Solve the linear SVM task on the infinite dimensional

RKHS H, for the training data {(Φ(xn),dn); n = 1, . . . ,N}.


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

−4

−2

0

2

4

6

−1

−1

−1

−1

0

0

00

0

0

0

1

1

1

Figure: Non linear SVM classification, C = 2, gaussian kernel

(σ = 2).


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

−4

−2

0

2

4

6

−1

−1

−1

−1

−1

0

00

0

0 0

0

1

1

1

1


(σ = 2).


Introduction


The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

−4

−2

0

2

4

6

−1 −1

−1

−1

−1

−1

−1

−1

0

0

0

0

0

00

0

1

1

1

1

1


(σ = 2).


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Real Hyperplanes

Recall that in any real Hilbert space H, a hyperplane

consists of all the elements f ∈ H that satisfy

〈f ,w〉H + b = 0, (2)

for some w ∈ H, b ∈ R.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Real Hyperplanes

Recall that in any real Hilbert space H, a hyperplane

consists of all the elements f ∈ H that satisfy

〈f ,w〉H + b = 0, (2)

for some w ∈ H, b ∈ R.

Moreover, any hyperplane of H divides the space into two

parts, H+ = {f ∈ H; 〈f ,w〉H + b > 0} and

H− = {f ∈ H; 〈f ,w〉H + b < 0}.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization


The output variable y is retained to be real.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization



The set of functions considered is in one way or another of

a special structure, so that the inner product is real.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization



The set of functions considered is in one way or another of

a special structure, so that the inner product is real.

The difficulty is that the set of complex numbers is not an

ordered one, and thus one may not assume that a

complex version of the aforementioned relation divides the

space into two parts, as H+ and H− cannot be defined.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

In order to be able to generalize the SVM rationale to

complex spaces, we need first to develop an appropriate

definition for a complex hyperplane.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization




We begin by considering the following two relations,

Re (〈f ,w〉H + c) = 0,

Im (〈f ,w〉H + c) = 0,

for some w ∈ H, c ∈ C, where f ∈ H.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization




We begin by considering the following two relations,

Re (〈f ,w〉H + c) = 0,

Im (〈f ,w〉H + c) = 0,

for some w ∈ H, c ∈ C, where f ∈ H.

It is not difficult to see, that this couple of relations

represent two orthogonal hyperplanes of the doubled real

space, i.e., H2.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

To overcome this constraint and be able to define arbitrarily

placed hyperplanes, we need to employ the so called

widely linear estimation functions, i.e.,

Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

for some w , v ∈ H, c ∈ C, where f ∈ H.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

To overcome this constraint and be able to define arbitrarily

placed hyperplanes, we need to employ the so called

widely linear estimation functions, i.e.,

Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

for some w , v ∈ H, c ∈ C, where f ∈ H.

Depending on the values of w , v , these hyperplanes may

be placed arbitrarily on H2. We define this complex couple

of hyperplanes as the set of all f ∈ H that satisfy either one

of the two relations, for some w , v ∈ H, c ∈ C.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The aforementioned arguments demonstrate the significant

difference between complex linear estimation and widely

linear estimation functions, which has been pointed out by

many other authors, in the context of regression tasks.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization





In the current context of classification, we have just seen

that confining to complex linear modeling is quite

restrictive, as the corresponding couple of complex

hyperplanes are always orthogonal.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization





In the current context of classification, we have just seen

that confining to complex linear modeling is quite

restrictive, as the corresponding couple of complex

hyperplanes are always orthogonal.

On the other hand, the widely linear case is more general

and covers all cases.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The complex couple of hyperplanes divides the space H (or

H2) into four parts, i.e.,

H++ =

{

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0

}

,

H+− =

{

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0

}

,

H−+ =

{

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0

}

,

H−− =

{

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0

}

.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

Figure: A complex couple of hyperplanes separates the space of

complex numbers (i.e., H = C) into four parts.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem

Suppose we are given training data, which belong to four

separate classes C++,C+−,C−+,C−−, i.e.,

{(zn,dn); n = 1, . . . ,N} ⊂ X × {±1 ± i)}. If dn = +1 + i , then

the n-th sample belongs to C++, i.e., zn ∈ C++, if dn = 1 − i ,

then zn ∈ C+−, e.t.c.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem






As zn is complex, we denote by xn its real part and by yn its

imaginary part respectively, i.e.,

zn = xn + iyn, n = 1, . . . ,N.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem






As zn is complex, we denote by xn its real part and by yn its

imaginary part respectively, i.e.,

zn = xn + iyn, n = 1, . . . ,N.

Our objective is to develop an SVM rationale for the complex

training data.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Consider the complex RKHS, H, with respective kernel κC.

Following a similar rationale to the real case, we transform

the input data from X to H, via the feature map ΦC.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation




The goal of the SVM task is to estimate a complex couple

of maximum margin hyperplanes, that separates the points

of the four classes as best as possible. To this end, we

formulate the primal complex SVM as


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation




The goal of the SVM task is to estimate a complex couple

of maximum margin hyperplanes, that separates the points

of the four classes as best as possible. To this end, we

formulate the primal complex SVM as

minw ,v ,c

1

2‖w‖2

H+ 1

2‖v‖2

H+ C

N

N∑

n=1

(ξrn + ξi

n)

s. to

d rn Re (〈ΦC(zn),w〉H + 〈Φ∗

C(zn), v〉H + c) ≥ 1 − ξr

n

d in Im (〈ΦC(zn),w〉H + 〈Φ∗

C(zn),w〉H + c) ≥ 1 − ξi

n

ξrn, ξ

in ≥ 0

for n = 1, . . . ,N.

(3)

for some C > 0.P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 35 / 47

Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Consequently, the Lagrangian function becomes

L(w , v , a, a,b, b) =1

2‖w‖2

H +1

2‖v‖2

H +C

N

N∑

n=1

(ξrn + ξi

n)

−

N∑

n=1

an (drn Re (〈ΦC(zn),w〉H + 〈Φ∗

C(zn), v〉H + c)− 1 + ξrn)

−

N∑

n=1

bn

(

d in Im (〈ΦC(zn),w〉H + 〈Φ∗

C(zn),w〉H + c)− 1 + ξin

)

−

N∑

n=1

ηnξrn −

N∑

n=1

θnξin,

where an,bn, ηn, θn are the positive Lagrange multipliers of the

respective inequalities, for n = 1, . . . ,N.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Employing the notion of Wirtinger’s calculus to derive therespective gradients and exploiting the saddle point conditionsof the Lagrangian function, it turns out that the dual problemcan be split into two separate maximization tasks:

maximizea

N∑

n=1

an −1

2

N∑

n,m=1

anamdrnd

rmκ

rC(zm, zn)

subject to

N∑

n=1

andrn = 0

0 ≤ an ≤ CN

for n = 1, . . . , N

(4a)

and

maximizea

N∑

n=1

bn −1

2

N∑

n,m=1

bnbmdind

imκ

rC(zm, zn)

subject to

N∑

n=1

bndin = 0

0 ≤ bn ≤ CN

for n = 1, . . . , N,

(4b)


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

We observe that these problems are equivalent with two distinct

real SVM (dual) tasks employing the induced real kernel κrC

:

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation



:

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may

split the (output) data to their real and imaginary parts,


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation



:

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may


solve two real SVM tasks employing any one of the

standard algorithms and, finally,


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation



:

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may


solve two real SVM tasks employing any one of the

standard algorithms and, finally,

combine the solutions to take the complex labeling

function:

g(z) = signi

(

N∑

n=1

(and rn + ibnd i

n)κrC(zn, z) + cr + ic i

)

,

where signi

(z) = sign(Re(z)) + i sign(Im(z)).


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Pure Complex SVM

Figure: Pure Complex Support Vector Machines.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification

An alternative path is the so called complexification

procedure.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification


procedure.

We employ a real kernel κR and transform the input data

from X to the complexified space H, i.e.,

x → ΦR(x) + iΦR(x).


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification


procedure.




We can similarly deduce that the dual of the complexified

SVM task is equivalent to two real SVM tasks employing

the kernel 2κR.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification


procedure.




We can similarly deduce that the dual of the complexified

SVM task is equivalent to two real SVM tasks employing

the kernel 2κR.

We conclude that, in both cases, we end up with two real

SVM tasks (although employing different types of kernels).


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Binary Classification

Although both scenarios are developed naturally for

quaternary classification, they can be easily adapted to the

binary case also.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Binary Classification

Although both scenarios are developed naturally for

quaternary classification, they can be easily adapted to the

binary case also.

This can be done by considering that the labels of the data

are real numbers (i.e., dn ∈ R) taking the values ±1. In this

case we solve one problem instead of two.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction


Complex RKHS


Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST

We use the popular MNIST database of handwritten digits.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST


Each digit is encoded as an image file with 28 × 28 pixels.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST



MNIST contains 60000 handwritten digits (from 0 to 9) for

training and 10000 handwritten digits for testing.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST



MNIST contains 60000 handwritten digits (from 0 to 9) for

training and 10000 handwritten digits for testing.

To exploit the structure of complex numbers, we perform a

Fourier transform to each training image and keep only the

100 most significant coefficients.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

We compare a standard one-versus-all SVM scenario that

exploits the original (real) data (images of 28 × 28 = 784

pixels) with


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment



pixels) with

a complex one versus all variant exploiting the

complexified binary SVM, where we use only the 100 most

significant (complex) Fourier coefficients of each picture.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment



pixels) with




In both scenarios we use the first 6000 digits of the MNIST

training set to train the learning machines and test their

performances using the 10000 digits of the testing set.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment



pixels) with







We used the gaussian kernel with t = 1/64 and

t = 1/1402 respectively.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment



pixels) with







We used the gaussian kernel with t = 1/64 and

t = 1/1402 respectively.

The SVM parameter C has been set equal to 100.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

The error rate of the standard real-valued scenario is

3.79%, while the error rate of the complexified

(one-versus-all) SVM is 3.46%.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment




In both learning tasks we used the SMO algorithm to train

the SVM.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment




In both learning tasks we used the SMO algorithm to train

the SVM.

The total amount of time needed to perform the training of

each learning machine is almost the same for both cases

(the complexified task is slightly faster).


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Second Experiment - Quaternary Classification

This is a quaternary classification problem.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments



Using the complex approach, such a problem can be

solved using only 2 distinct SVM tasks, instead of the 4

SVM tasks needed by the standard 1-versus-all or the

1-versus-1 strategies.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments







We compare a complex quaternary SVM task with the

1-versus-all scenario.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments







We compare a complex quaternary SVM task with the

1-versus-all scenario.

To this end we use the first 6000, 0, 1, 2 and 3 digits of the

MNIST training set and compare the performances of the

two algorithms using the respective digits of the MNIST

training set.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments


The error rate of the 1-versus-all SVM was 0.721%,


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments



while the error rate of the complex SVM was 0.866%.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments




In terms of speed the 1-versus-all SVM task required about

double the time for training, compared to the complex SVM.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments






This is expected, as the latter solves half as many distinct

SVM tasks as the first one.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments








In both experiments we used the gaussian kernel with

t = 1/49 and t = 1/1602 respectively.


Introduction


The Complex Case

Complex Hyperplanes

Problem formulation

Experiments








In both experiments we used the gaussian kernel with

t = 1/49 and t = 1/1602 respectively.

The SVM parameter C has been set equal to 100 in this

case also.


Date post:	14-Jun-2015
Category:	Education
Upload:	pantelis-bouboulis
View:	704 times
Download:	4 times

Complex Support Vector Machines For Quaternary Classification

Education