Tutorial : Echo State Networks

Dan Popovici

University of Montreal (UdeM)

MITACS 2005

Overview

1. Recurrent neural networks: a 1-minute primer

2. Echo state networks

3. Examples, examples, examples

4. Open Issues

1 Recurrent neural networks

Feedforward- vs. recurrent NN

......

...... ...

...

Input InputOutput Output

• connections only "from left to right", no connection cycle

• activation is fed forward from input to output through "hidden layers"

• no memory

• at least one connection cycle

• activation can "reverberate", persist even with no input

• system with memory

recurrent NNs, main properties

• input time series output time series

• can approximate any dynamical system (universal approximation property)

• mathematical analysis difficult

• learning algorithms computationally expensive and difficult to master

• few application-oriented publications, little research

......

Supervised training of RNNs

A. Training

Teacher:

Model:

B. Exploitation

Input: Correct (unknown) output:

Model:

in

out

in

out

Backpropagation through time (BPTT)

• Most widely used general-purpose supervised training algorithm

• Idea: 1. stack network copies, 2. interpret as feedforward network, 3. use backprop algorithm.

. . .

original RNN

stack of copies

What are ESNs?• training method for

recurrent neural networks

• black-box modelling of nonlinear dynamical systems

• supervised training, offline and online

• exploits linear methods for nonlinear modeling

... +

+

Previously

ESN training

Introductory example: a tone generator

Goal: train a network to work as a tuneable tone generator

input: frequency

setting

output: sines of desired frequency

20 40 60 80 1000.25

0.35

0.4

0.45

0.5

20 40 60 80 100

0.1

0.2

0.3

0.4

Tone generator, sampling• For sampling period, drive fixed "reservoir" network with teacher input

and output.

20 40 60 80 1000.25

0.35

0.4

0.45

0.5

20 40 60 80 100

0.1

0.2

0.3

0.4

• Observation: internal states of dynamical reservoir reflect both input and output teacher signals

20 40 60 80 100

-0.5-0.4-0.3-0.2-0.1

20 40 60 80 100

-0.8

-0.6

-0.4

-0.2

20 40 60 80 100

-0.75-0.5

-0.25

0.250.5

0.75

20 40 60 80 100

0.40.50.60.70.80.9

Tone generator: compute weights

• Determine reservoir-to-output weights such that training output is optimally reconstituted from internal "echo" signals.

20 40 60 80 1000.25

0.35

0.4

0.45

0.5

20 40 60 80 100

0.1

0.2

0.3

0.4

20 40 60 80 100

-0.5-0.4-0.3-0.2-0.1

20 40 60 80 100

-0.8

-0.6

-0.4

-0.2

20 40 60 80 100

-0.75-0.5

-0.25

0.250.5

0.75

20 40 60 80 100

0.40.50.60.70.80.9

Tone generator: exploitation

• With new output weights in place, drive trained network with input.

• Observation: network continues to function as in training.

– internal states reflect input and output

– output is reconstituted from internal states

• internal states and output create each other

20 40 60 80 1000.25

0.35

0.4

0.45

0.520 40 60 80 100

0.1

0.2

0.3

0.4

20 40 60 80 100

-0.5-0.4-0.3-0.2-0.1

20 40 60 80 100

-0.8

-0.6

-0.4

-0.2

20 40 60 80 100

-0.75-0.5

-0.25

0.250.5

0.75

20 40 60 80 100

0.40.50.60.70.80.9

echo

reconstitute

Tone generator: generalization

The trained generator network also works with input different from training input

50 100 150 200

0.1

0.2

0.3

0.4

50 100 150 2000.25

0.35

0.4

0.45

0.5

200

0.6

0.7

0.80.9

200

- 0.6- 0.5- 0.4- 0.3

200

- 0.7- 0.6- 0.5- 0.4

200

- 0.75- 0.5

- 0.25

0.250.50.75

A. step input B. teacher and learned output

C. some internal states

Dynamical reservoir

• large recurrent network (100 - units)

• works as "dynamical reservoir", "echo chamber"

• units in DR respond differently to excitation

• output units combine different internal dynamics into desired dynamics

......

input units output un its

recurrent "dynam ical reservoir"

Rich excited dynamics

...

10 20 30 40 50

-0.8-0.6-0.4-0.2

0.20.4

10 20 30 40 50-0.2

0.2

0.4

0.6

10 20 30 40 50-0.2

0.2

0.4

10 20 30 40 50

-0.4-0.2

0.20.40.6

10 20 30 40 50

0.2

0.40.6

0.81

excitation

responses

Unit impulse responses should vary greatly.

Achieve this by, e.g.,

• inhomogeneous connectivity

• random weights

• different time constants

• ...

Notation and Update Rules

))(),1(),1((()1(

))()()1(()1(

)(),(

),(),(

))'(),...,(()(

))'(),...,(()(

))'(),...,(()(

1

1

1

nynxnuWfny

nyWnWxnuWfnx

wWwW

wWwW

nynyny

nxnxnx

nununu

outout

backin

backij

backoutij

out

ijin

ijin

L

N

K

Learning: basic idea

Every stationary deterministic dynamical system can be defined by an equation like

),2(),1(,),1(),()( tdtdtutuhtd

where the system function h might be a monster.

Combine h from the I/O echo functions by selecting suitable DR-to-output weights :

iii

iii

tytuhw

txwtytd

),...)1(),...,((

)()()(

iw

)(txi iw)(ty)(tu

Offline training: task definition

i

ii txwty )()(

Let be the teacher output. . )(td

Compute weights such that mean square error

])()([])()([ 22 txwtdEtytdE ii

is minimized.

Recall

Offline training: how it works

1. Let network run with training signal teacher-forced.

2. During this run, collect network states , in matrix M

3. Compute weights , such that

)(td

)(txi

iw ])()([ 2 txwtdE ii

is minimized

MSE minimizing weight computation (step 3) is a standard operation.

Many efficient implementations available, offline/constructive and online/adaptive.

TMw 1

Practical Considerations

backin WWW ,, Chosen randomly

• Spectral radius of W < 1

• W should be sparse

• Input and feedback weights have to be scaled “appropriately”

• Adding noise in the update rule can increase generalization performance

Echo state network training, summary

• use large recurrent network as "excitable dynamical reservoir (DR)"

• DR is not modified through learning• adapt only DR output weights• thereby combine desired system function from I/O

history echo functions• use any offline or online linear regression algorithm

to minimize error

]))()([( 2tytdE

3 Examples, examples, examples

Short-term memories

Delay line: scheme

...

t

in p u t: ( )s t

o u tp u ts : ( ) ,..., ( )s t-d 1 s t-d n

s t( )s t-d( 1 )s t-d n( ) ...

Delay line: example

• Network size 400• Delays: 1, 30, 60, 70, 80, 90, 100, 103, 106, 120 steps• Training sequence length N = 2000

10 20 30 40 50

-0.4

-0.2

0.2

0.4

training signal: random walk with resting states

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

10 20 30 40 50-0.4-0.2

0.20.4

results

correct delayed signals ( ) and network outputs ( )

-1 -30 -60 -90

-100 -103 -106 -120

traces of some DR internal units

1020304050-0.001-0.00050.00050.0010.0015

1020304050-0.002-0.0015-0.001-0.00050.00050.0010.0015

1020304050-0.0004-0.00020.00020.0004

1020304050-0.0015-0.001-0.00050.00050.001

10 20 30 40 50-0.2-0.10.10.20.30.4

10 20 30 40 50-0.2-0.1

0.10.2

10 20 30 40 50

-0.3-0.2-0.1

0.10.2

10 20 30 40 50-0.2-0.1

0.10.2

10 20 30 40 50-0.3-0.2-0.10.10.20.30.4

10 20 30 40 50-0.3-0.2-0.10.10.20.30.4

10 20 30 40 50

0.10.20.30.4

10 20 30 40 50-0.2-0.10.10.20.30.4

Delay line: test with different input

correct delayed signals ( ) and network outputs ( )

-1 -30 -60 -90

-100 -103 -106 -120

traces of some DR internal units

1020304050-0.001

-0.0005

0.00050.001

1020304050-0.0015-0.001-0.00050.00050.0010.0015

1020304050-0.0004-0.00020.00020.00040.0006

1020304050-0.0005-0.000250.000250.00050.000750.001

3.2 Indentification of nonlinear systems

Identifying higher-order nonlinear systems

9,...0

1.0)9()(5.1)()(05.0)(3.0)1(k

nunuknynynyny

A tenth-order system

...

20 40 60 80 100

0.1

0.2

0.3

0.4

20 40 60 80 100

0.3

0.4

0.5

0.6

Training setup

)(ny)(nu

Results: offline learning

augmented ESN (800 Parameters) :

NMSEtest = 0.006

previous published state of the art1):

NMSEtrain = 0.24

D. Prokhorov, pers. communication2):

NMSEtest = 0.004

50 100 150 200

0.3

0.4

0.5

0.6

0.7

1) Atiya & Parlos (2000), IEEE Trans. Neural Networks 11(3), 697-708

2) EKF-RNN, 30 units, 1000 Parameters.

The Mackey-Glass equation

• delay differential equation

• delay > 16.8: chaotic

• benchmark for time series prediction

)(1.0))(1(/)(2.0)( 10 txtxtxtx

50 100 150 200 250 300

0.4

0.6

0.8

1.2

50 100 150 200 250 300

0.6

0.8

1.2

0.4 0.6 0.8 1.2

0.4

0.6

0.8

1.2

0.4 0.6 0.8 1.2

0.4

0.6

0.8

1.2

= 17

= 30

Learning setup

• network size 1000• training sequence N = 3000• sampling rate 1

50 100 150 200 250 300

0.6

0.8

1.2

Results for = 17

Error for 84-step prediction:

NRMSE = 1E-4.2

(averaged over 100 training runs on independently created data)

With refined training method:

NRMSE = 1E-5.1

previous best:

NRMSE = 1E-1.7

original

0.4 0.6 0.8 1.2

0.4

0.6

0.8

1.2

0.4 0.6 0.8 1.2

0.4

0.6

0.8

1.2 learnt model

Prediction with model

visible discrepancy after about 1500 steps

. . .

. . .

Comparison: NRMSE for 84-step prediction

-1.2

-1.2

-1.3

-1.3

-1.7

-1.7

-4.2

-5.1

-1.7

ESN (refined) -5.1

ESN (1+2 K) -4.2

PCR Local Model (McNames 99, 2 K) -1.7

SOM (Vesanto 97, 3K) -1.7

DCS-LLM (Chudy & Farkas 98, 3K) -1.7

AMB (Bersini et al 98, ? K) * -1.3

Neural Gaz (Martinez et al 93, ~4K) -1.3

EPNet (Yao & Liu 97, 0.5 K) -1.2

BPNN (Lapedes & Farber 87, ? K) * -1.2

*) data from survey in Gers / Eck /Schmidhuber 2000

log10(NRMSE)

3.3 Dynamic pattern recognition

Dynamic pattern detection1)

Training signal:

output jumps to 1 after occurence of pattern instance in input

)(ny)(nu

1) see GMD Report Nr 152 for detailed coverage

Single-instance patterns, training setup

1. A single-instance, 10-step pattern is randomly fixed

4 6 8 10

-0.4

-0.2

0.2

0.4

2. It is inserted into 500-step random signal at positions 200 (for training)350, 400, 450, 500 (for testing)

3. 100-unit ESN trained on first 300 steps (single positive instance! "single shot learning), tested on remaining 200 steps

50 100 150 200

-0.4

-0.2

0.2

0.4

test data: 200 steps with 4 occurances of pattern on random background, desired output: red impulses

the pattern

50 100 150 200

-0.75-0.5

-0.25

0.250.5

0.751

50 100 150 200-0.02

0.020.040.060.080.1

50 100 150 200-0.05

0.050.1

0.150.2

0.250.3

Single-instance patterns, results

1. trained network response on test data

50 100 150 200-0.1

0.1

0.2

0.3

2. network response after training 800 more pattern-free steps ("negative examples")

3. like 2., but 5 positive examples in training data

DR: 12.4DR: 12.1DR: 6.4

4. comparison: optimal linear filter

DR: 3.5

discrimination ratio DR:

)]([/)]([ 22 ndEndE

Event detection for robots(joint work with J.Hertzberg & F. Schönherr)

Robot runs through office environment, experiences data streams (27 channels) like...

50 100 150 200

0.20.40.60.8

1

50 100 150 200-0.5-0.250.250.5

0.75

50 100 150 200

0.20.40.60.8

1

50 100 150 200

0.20.40.60.8

1

10 sec

infrared distance sensor

left motor speed

activation of "goThruDoor"

external teacher signal, marking event category

Learning setup

......

50 100 150 200

0.20.40.60.8

1

50 100 150 200

0.20.40.60.8

1

50 100 150 200

0.20.40.60.8

1

50 100 150 200

0.20.40.60.8

1

27 (raw) data channels unlimited number of event detector channels100 unit RNN

• simulated robot (rich simulation)

• training run spans 15 simulated minutes

• event categories like

• pass through door

• pass by 90° corner

• pass by smooth corner

Resultscorners w � o corridor end

200 400 600 800 1000-0.2

0.20.40.60.8

corner + corridor end

200 400 600 800 1000-0.2

0.20.40.60.8

corridor end

200 400 600 800 1000

0.20.40.60.8

enter Room from corridor

200 400 600 800 1000

0.20.40.60.8

go through any door

200 400 600 800 1000-0.2

0.20.40.60.8

• easy to train event hypothesis signals

• "boolean" categories possible

• single-shot learning possible

Network setup in training

......

_

a

z

29 input channelscode symbols

. . .

29 output channels for next symbol hypotheses

400 units

Trained network in "text" generation

......

decision mechanism, e.g. winner-take-all

!!

winning symbol is next input

Results

Selection by random draw according to output

yth_upsghteyshhfakeofw_io,l_yodoinglle_d_upeiuttytyr_hsymua_doey_sammusos_trll,t.krpuflvek_hwiblhooslolyoe,_wtheble_ft_a_gimllveteud_ ...

Winner-take-all selection

sdear_oh,_grandmamma,_who_will_go_and_the_wolf_said_the_wolf_said_the_wolf_said_the_wolf_said_the_wolf_said_the_wolf_said_the_wolf ...

4 Open Issues

4.2 Multiple timescales

4.3 Additive mixtures of dynamics

4.4 "Switching" memory

4.5 High-dimensional dynamics

Multiple time scales

This is hard to learn (Laser benchmark time series):

100 200 300 400 500

-1

-0.8

-0.6

-0.4

-0.2

0.2

0.4

Reason: 2 widely separated time scales

Approach for future research: ESNs with different time constants in their units

Additive dynamics

This proved impossible to learn:

Reason: requires 2 independent oscillators; but in ESN all dynamics are mutually coupled.

Approach for future research: modular ESNs and unsupervised multiple expert learning

50 100 150 200 250 300

0.1

0.2

0.3

0.4

0.5

0.6

0.7

)311.0Sin()2.0Sin()( nnny

"Switching" memory

This FSA has long memory "switches":

Generating such sequences not possible with monotonic, area-bounded forgetting curves!

a a

b

c

baaa....aaacaaa...aaabaaa...aaacaaa...aaa...

boundedarea

unbounded width

An ESN simply is not a model for long-term memory!

High-dimensional dynamics

High-dimensional dynamics would require very large ESN. Example: 6-DOF nonstationary time series one-step prediction

200-unit ESN: RMS = 0.2; 400-unit network: RMS = 0.1; best other training technique1): RMS = 0.02

Approach for future research: task-specific optimization of ESN

100 200 300 400 500 600

0.2

0.4

0.6

0.8

feudu

ucubuauny

21

2122

21)(

1)Prokhorov et al, extended Kalman filtering BPPT. Network size 40, 1400 trained links, training time 3 weeks

Spreading trouble...

• Signals xi(n) of reservoir can be interpreted as vectors in

(infinite-dimensional) signal space • Correlation E[xy] yields inner product < x, y > on this space

• Output signal y(n) is linear combination of these xi(n)

• The more orthogonal the xi(n), the smaller the output weights:

y

yx1x2

x2

x1y = 30 x1 28 x2 y = 0.5 x1 0.7 x2

• Eigenvectors vk of correlation matrix R = (E[xi x j] ) are orthogonal

signals

• Eigenvalues k indicate what "mass" of reservoir signals xi (all

together) is aligned with vk

• Eigenvalue spread max/ min indicates overall "non-orthogonality"

of reservoir signals

vmaxx1

x2

x2

x1

vmin vmax

vmin max/ min 20 max/ min 1

Large eigenvalue spread large output weights ...

• harmful for generalization, because slight changes in reservoir signals will induce large changes in output

• harmful for model accuracy, because estimation error contained in reservoir signals is magnified (applies not to deterministic systems)

• renders LMS online adaptive learning useless vmaxx1

x2 vmin

max/ min 20

Summary

• Basic idea: dynamical reservoir of echo states + supervised teaching of output connections.

• Seemed difficult: in nonlinear coupled systems, every variable interacts with every other. BUT seen the other way round, every variable rules and echoes every other. Exploit this for local learning and local system analysis.

• Echo states shape the tool for the solution from the task.

Thank you.

References

• H. Jaeger (2002): Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the "echo state network" approach. GMD Report 159, German National Research Center for Information Technology, 2002

• Slides used by Herbert Jaeger at IK2002