NEURAL NETWORK THEORY

NEURAL DYNAMIC1:ACTIVATIONHS AND SIGNALS

MAIN POINTS:

• NEURONS AS FUNCTIONS(神经元函数)• SIGNAL MONOTONICITY（信号单调性）• BIOLOGICAL ACTIVATIONS AND SIGNALS（生物激励与信号）• NEURON FIELDS（神经域）• NEURONAL DYNAMICAL SYSTEMS（神经诊断系统）• COMMON SIGNAL FUNCTION（一般信号方程）• PULSE-CODED SIGNAL FUNCTION（脉冲编码信号方程）

NEURONS AS FUNCTION

Figure 1. Neuron Structure Model

ij

n

jjii xI

1 jIfy

,

Relationship of input-output:

•Common nonlinear transduction description: a sigmoidal or S-shaped curve

Fig.2 s(x) is a bounded monotone-nondecreasing function of x

Signal Function: cxe

xS

1

1)( )0(0)1(' cScSdxdSS

Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

NEURONS AS FUNCTION

SIGNAL MONOTONICITY

0)(2

cexS cx

• In general, signal functions are monotone nondecreasing S’>=0. In practice this means signal functions have an upper bound or saturation value. • An important exception: bell-shaped signal function or Gaussian signal functions

xScxeS cx ',2'2

The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.

SIGNAL MONOTONICITY

])(2

1exp[)( 22

n

j

ijj

ii xxS

2i

:

•Generalized Gaussian signal function define potential or radial basis function:

nn Rxxx ),,( 1 input activation vector:

variance:

mean vector: ),,( 1in

ii

)( ii xS

Because the function depend on all neuronal activations not just the ith activation, we shall consider only scalar-input signal functions:

SIGNAL MONOTONICITY

• A property of signal monotonicity: semi-linearity

• Comparation:

a. Linear signal functions: computation and analysis is comparatively easy; do not suppress

noise.

b. Nonlinear signal functions: Increases a network’s computational richness and facilitates noise suppression; risks computational and analytical intractability;

SIGNAL MONOTONICITY

xSdtdx

dxdSS '

•Signal and activation velocities

the signal velocity:

=dS/dtS

Signal velocities depend explicitly on action velocities. This dependence will increase the number of unsupervised learning laws.

BIOLOGICAL ACTIVATIONS AND SIGNALS

Fig3. Key functional units of a biological neuron

•Introduction to units ：Dendrite: input

Axon: output

Synapse: transduce signal

Membrane: potential difference between inside and outside of neuron

BIOLOGICAL ACTIVATIONS AND SIGNALS

•Competitive Neuronal Signal

Signal values are usually binary and bipolar.

Bipolar signal functions :

Binary signal functions :

TxTx

xs01

TxTx

xs1

1

NEURON FIELDS In general, neural networks contain many fields of neurons. Neurons within a field are topological.

Denotation:ZYX FFF

XF : input field zF : output field

xs mm yxyxyx ,......,,, 2211

Neural system samples the function m times to generate the associated

•Classification: Zeroth-order topological (simplest)

Three-dimensional and volume topological (complex)

pairs

NEURONAL DYNAMICAL SYSTEMS •Description:

A system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials

Activation differential equations:

.....,,......,

.....,,......,

21

21

yyyFFhyxxxFFgx

YX

YX

ii yx , denote the activation time functions of the ith neuron in XF

and jth neuron in YF

•Classification: Automomous systems: activations are independent of t

Nonautonomous systems: depend on t

NEURONAL DYNAMICAL SYSTEMS

pnYX RRFF ,

ZYX FFF ]|[

p

n

nx

RtytytYRtxtxtX

,...,...,

1

1

•Neuronal State spaces

So the state space of the entire neuronal dynamical system is:

Augmentation : pnz RF

Concatenate fields have different computational, metrical or other properties

NEURONAL DYNAMICAL SYSTEMS

Signal state spaces as hypercubes

)))((,)),((())(( 11 txStxStXS nXn

X

Fig.4 Neural and fuzzy computations conincide.

NEURONAL DYNAMICAL SYSTEMS

•Neuronal activations as short-term memory

Short-term memory(STM) : activation

Long-term memory(LTM) : synapse

COMMON SIGNAL FUNCTION 1 、 Liner Function

S(x) = cx + k , c>0

x

S

o

k

COMMON SIGNAL FUNCTION

2. Ramp Function r if x≥θS(x)= cx if |x|<θ

-r if x≤-θr>0, r is a constant.

r

-r

θ -θ x

S

COMMON SIGNAL FUNCTION

elsecxifcxif

cxxS 0

101

)(

)),max(,min()( cxxS 01

3 、 threshold linear signal function: a special Ramp Function

Another form:

0cS '

COMMON SIGNAL FUNCTION

xcxc

xc

cx

ee

ee

xS22

2

11)(

4 、 logistic signal function:

Where c>0.01 )(' ScSS

So the logistic signal function is monotone increasing.

COMMON SIGNAL FUNCTION

5 、 threshold signal function:

TxifTxifTxif

xSxSk

k

k

kk

1

1

1

1

0)(

1)(

Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

COMMON SIGNAL FUNCTION

)tanh()( cxxS

01 2 )(' ScS

6 、 hyperbolic-tangent signal function:

Another form:

cxcx

cxcx

eeeecx

)tanh(

COMMON SIGNAL FUNCTION

),min()( cxexS 1

1cxe

0 cxceS '

7 、 threshold exponential signal function:

When

02 cxecS ''

0 cxnn ecS )(

COMMON SIGNAL FUNCTION

),max()( cxexS 10

0x

0 cxceS '

8 、 exponential-distribution signal function:

When

0'' 2 cxecS

COMMON SIGNAL FUNCTION

),max()( n

n

xcxxS

0

1n

9 、 the family of ratio-polynomial signal function:

An example

For

02

1

)(' n

n

xccnxS

PULSE-CODED SIGNAL FUNCTION •Description: Pulse trains arriving in a sampling interval seems to be the bearer

of neuronal signal information.

t

tsii dsesxtS )()(

t

tsjj dsesytS )()(

Pulse-coded formulation:

tatpulsenoiftatoccurspulseaif

txi

01

)(where

ii yx , denote binary pulse functions that summarize the excitation of

membrane potential.

PULSE-CODED SIGNAL FUNCTION

)()()( tStxtS iii

)()()( tStytS jjj

tatpulsenoiftatoccurspulseaif

txi

01

)(

•Velocity-difference property of pulse-coded signals

Current pulse and current signal or expected pulse frequency are available quantities.

Another computational advantage:

If

arrivedpulsenotsarrivedpulsesustainedats

001

A simple form for the signal velocity: