Clustering using Spiking Neural Networks

Biological Neuron:The Elementary Processing Unit of the Brain

Biological Neuron:A Generic Structure

Dendrite SomaSynapse Axon Axon

Terminal

Biological Neuron:Nerve Impulse Transiting

Action Potential(Spike) Postsynaptic

Potential

Membrane Potential

Action Potential(Spike)

Spike-After Potential

Biological Neuron:Soma Firing Behavior

)(tu

)(rest tut

t

t

in1t

in2t

in3t

in4t

outt

a)

b)

c)

Synchrony is the main factor of soma firing

Biological Neuron:Information Coding

Neurons communicate via exact spike timing

Firing rate alone does not carry all the relevant information

Neuroscience Models of Neuron:The Hodgkin-Huxley Model

Alan Lloyd Hodgkin and Andrew Huxley received the Nobel Prize in Physiology and Medicine in 1963

The Hodgkin-Huxley model is too complicated model of neuron to be used in artificial neural networks

Neuroscience Models of Neuron:Leaky Integrate-And-Fire Model

orLeaky Integrate-And-Fire model disregards the refractory capability of neuron

Neuroscience Models of Neuron:Spike-Response Model

Spike-Response model captures the major elements of a biological neuron behavior

Biological Neuron – Computational Intelligence Approach:The First Generation

The first artificial neuron was proposed by W. McCulloch & W. Pitts in 1943

n

iiji

n

iiji

j

xwif

xwify

1

1

,0

,1

Biological Neuron – Computational Intelligence Approach:The Second Generation

Multilayered Perception is a universal approximator

n

iijij xwfy

1

Biological Neuron – Computational Intelligence Approach:Artificial Neurons – Too Artificial?

0

1y

spike occurrence

spike absence

From neurophysiology point of view, y is existence of an output spike

y Number of spikesTime frame

From neurophysiology point of view, y is firing rate

Spike timing is not considered at all!

Biological Neuron – Computational Intelligence Approach:The Third Generation

Spiking neuron model was introduced by J. Hopfield in 1995

Spiking neural networks are - biologically more plausible, - computationally more powerful, - considerably faster than networks of the second generation

Spiking Neural Network:Overall Architecture 1,1RN

1,2RN

1,hRN

2,1RN

2,hRN

2,2RN

nRN ,1

nRN ,2

hnRN

)(1 kx

)(2 kx

)(kxn

s

s

s

1,1,1MS

mhnMS

))((]1[1 kxt

))((]1[2 kxt

))((]1[ kxtm

∫

∫

mSN

∫

2SN

1SN

RN is a receptive neuron

MS is a multiple synapse

SN is a spiking neuron

Spiking neural network is a heterogeneous two-layered feed-forward network with lateral connections in the second hidden layer

Spiking Neural Network:Population Coding

Pool of Receptive Neurons

iliili ckxtkxt ,)(1))(( ]0[]0[max

]0[Input spike:

)(kxi )(sxi

)(sxi

0))((]0[ ,1 kxt i

iRN ,1 iRN ,2 iRN ,3

4))((]0[ ,2 kxt i

1))((]0[ ,1 sxt i

2))((]0[ ,2 sxt i

Spiking Neural Network:Multiple Synapse

))))(((()( ]0[ pli

pjli

pjli dkxttwtu

)(1exp)( tHtt

tpjli

q

p

pjlijli tutu

1

)()(

n

i

h

ljlij tutu

1 1

)()(

Delayed postsynaptic potential:

Spike-response function:

Total postsynaptic potential:

Membrane potential:

1,1RN

hnRN

liRN ∫

s

Σ

qjliw

... ...

1d

2d

1jliw

2jliw

1,1,jMS

jhnMS

jliMS

...

...

...

...jSN

)(1,1, tu j

)( tu jli

))((]0[ kxt nhn

))((]0[ kxt ili

))(( 1]0[

1,1 kxt

)(tu jhn

qd

Spiking Neural Network:Hebbian Learning – WTA and WTM

,~),(

,~),()()()1( w

jjKw

jjtLKKwKw

pjli

pjli

pjlip

jli

))(())(( ]1[]0[ kxtdkxtt jp

ilipjli

0

0

t

)( tL

0

12

)1(2exp1)(

2pjlip

jli

ttL

1ln2

12

Winner-Takes-All:

Winner-Takes-More*:

)()()()()1( ~wpjlijj

pjli

pjli tLtKKwKw

))(())(( ]1[]1[~~ kxtkxtt jjjj

*Proposed for the first time on the 11th International Conference on Science and Technology “System Analysis and Information Technologies” (Kyiv, Ukraine, 2009) by Ye. Bodyanskiy and A. Dolotov

Spiking Neural Network:Image Processing*

Original Image SOM at 50 epoch SNN at 4 epoch

*In Bionics of Intelligence: 2007, 2 (67), pp. 21-26 by Ye. Bodyanskiy and A. Dolotov

Spiking Neuron:The Laplace Transform Basis

Thus, transformation of action potential to postsynaptic potential taken into synapse is nothing other than pulse-position – continuous-time transformation, and soma transformation is just reverse one, continuous-time – pulse-position transformation

From control theory point of view, action potential (spike) is a signal in pulse-position form:

skxtekxttL ))(())((

Spiking Neuron Synapse:A 2nd order critically damped response unit *

*Proposed for the first time on the 6th International Conference “Information Research and Applications” (Varna, Bulgaria, 2009) by Ye. Bodyanskiy, A. Dolotov, and I. Pliss

111

)(21

ss

sG

21

21

1)(~

tt

eet

21 1

21

)(~

t

et

t

)(~)( tet

t

et

t1

)(

21)(

se

sGSynapse

Spiking Neuron:Technically Plausible Description*

Incoming Spike: Time Delay:

Spike-Response Function: Membrane Potential:

Relay: Outgoing Spike:

skxtli

liekxttL ))((]0[ ]0[

))(( sdTimeDelay

p

esG )(

21)(

se

sGSRF

n

i

h

l

q

p

sdkxtpjli

j s

ewsu

pli

1 1 12

))((1

1)(

]0[

2

1))(sign();(

sj

sj

tutu sjj tusLkxttL );())((]1[

*Proposed for the first time on the 6th International Conference “Information Research and Applications” (Varna, Bulgaria, 2009) by Ye. Bodyanskiy, A. Dolotov, and I. Pliss

Spiking Neuron:Analog-Digital Architecture*

1,1RN

1,lRN

1,hRN

iRN ,1

liRN

hiRN

nRN ,1

lnRN

hnRN

sde1 1

jliw

sdpe 2PSP

PSP

1s

e pjliw

sdqe 2PSP

PSP

1s

e qjliw

jliMS

)(1 kx

)(kxi

)(kxn

))((]0[ kxtt ili

1,1,jMS

jhnMS ))((]0[ kxtt nhn

))(( 1]0[1,1 kxtt

s)(suj

)(tuj

t

skxt je ))((]1[

))((]1[ kxtt j

jSN

)(tujli

)(suqjli

)(1 tujli

)(tupjli

)(tuqjli

)(1 sujli

)(supjli )(sujli

t]0[

maxt

skxt ilie ))((]0[

t]0[

maxt

skxte ))(( 1]0[1,1

t]0[

maxt

skxt nhne ))((]0[ )(sujhn

)(1,1, suj

)(tujhn

)(1,1, tuj

s.n. 2PSP

PSP

1s

e

]1[maxt

+

+

+

++

+

sde spike

2SAP 11 sSAPw

-

s.n.

* Proposed for the first time in Image Processing / Ed. Yung-Sheng Chen: In-Teh, Vukovar, Croatia, pp. 357-380 by Ye. Bodyanskiy and A. Dolotov,

Analog-digital spiking neurons corresponds to spike-response model entirely

Fuzzy Receptive Neurons*:

*Proposed for the first time in Information Technologies and Computer Engineering: 2009, 2(15), pp. 51-55 by Ye. Bodyanskiy and A. Dolotov

ix)(kxi

iX ,1 iX ,2 iX ,3 ihX ,1 hiX

r.n.

))((,2 kxii

))((,3 kxii

)( ix ...

Pool of receptive neurons is a linguistic variable, and a receptive neuron within a pool is a linguistic term.

Fuzzy Spiking Neural Network:Fuzzy Probabilistic Clustering*

*Proposed for the first time in Sci. Proc. of Riga Technical University: 2008, 36, P. 27-33 by Ye. Bodyanskiy and A. Dolotov

1,1FRN

1,2FRN

1,hFRN

2,1FRN

2,hFRN

2,2FRN

nFRN ,1

nFRN ,2

hnFRN

)(1 kx

)(2 kx

)(kxn

s

s

s

1,1,1MS

mhnMS

Fuzzy Clustering Layer

))(( kxj

))(( kxm

1SN

2SN

mSN

))((1 kx

mj

j

kxt

kxtkx

1

12

]1[

12

]1[

))((

))(()(

There is no need to calculate cluster centers!

Fuzzy Spiking Neural Network:Fuzzy Possibilistic Clustering*

*Proposed for the first time on the 15th Zittau East-West Fuzzy Colloquium (Zittau, Germany, 2008) by Ye. Bodyanskiy, A. Dolotov, I. Pliss, and Ye. Viktorov

1,1FRN

1,2FRN

1,hFRN

2,1FRN

2,hFRN

2,2FRN

nFRN ,1

nFRN ,2

hnFRN

)(1 kx

)(2 kx

)(kxn

s

s

s

1,1,1MS

mhnMS

Fuzzy Clustering Layer

))(( kxj

))(( kxm

1SN

2SN

mSN

))((1 kx

1

11

2]1[ ))((1)(

j

jj

kxtkx

N

kj

N

kjj

j

kx

kxt

1

1

2]1[

))((

))((

Fuzzy Spiking Neural Network:Image Processing*

*In Proceeding of the 4th International School-Seminar “Theory of Decision Making“ (Uzhhorod, Ukraine, 2008) by Ye. Bodyanskiy, A. Dolotov, and I. Pliss

Original image

Training set

FSNN at 4th epoch

SOM at 40th epoch

Fuzzy Spiking Neural Network:Image Processing*

*In Proceeding of the 11th International Biennial Baltic Electronics Conference "BEC 2008“ (Tallinn/Laulasmaa, Estonia, 2008) by Ye. Bodyanskiy and A. Dolotov

Original image

Training set

FSNN at 3rd epoch

FCM at 29th epoch

Fuzzy Spiking Neural Network:Image Processing*

*In Image Processing / Ed. Yung-Sheng Chen: In-Teh, Vukovar, Croatia, pp. 357-380 by Ye. Bodyanskiy and A. Dolotov

Original image

Training set

FSNN at 1st epoch

FSNN at 3rd epoch

FCM at 3rd epoch

FCM at 30th epoch