8/3/2019 NeuralNetworks-2

1/18

G51IAIIntroduction to AI

Andrew Parkes

Neural Networks 2

8/3/2019 NeuralNetworks-2

2/18

Recap

Neuron Weighted sum of inputs

Activation function

Single-Layer Perceptron linearly separable function only

cannot do XOR Multiple-Layer Networks

can represent XOR

8/3/2019 NeuralNetworks-2

3/18

Using Neural Networks

Handwriting recognition (simplified example)

NN is to recognise handwritten Y vs. N

Training set: input : pixels for the hand-writing output : 1 if picture is a Y, 0 if it is an N

use lot of such instances and train the weights so that theoutputs are correct

Usage: give the NN a new picture use the output to predict whether it is a Y or an N

8/3/2019 NeuralNetworks-2

4/18

Instant vs. Time-Stepped Instant nets

effects propagate immediately to the output

computes a (complicated) function of theinputs

Time-Stepped nets

regard neurons as one time-step to compute

state of net does not depend on current inputonly, but also previous inputs

8/3/2019 NeuralNetworks-2

5/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

It takes one time step for a signal to pass

over one connection.

-1

2

2X1

X2

X3

Y

8/3/2019 NeuralNetworks-2

6/18

The First Neural Networks If we touch something cold we perceive

heat

If we keep touching something cold wewill perceive cold

If we touch something hot we will

perceive heat

8/3/2019 NeuralNetworks-2

7/18

The First Neural Networks To model this we will assume that time is discrete

If cold is applied for one time step then heat will beperceived

If a cold stimulus is applied for two time steps thencold will be perceived

If heat is applied then we should perceive heat

8/3/2019 NeuralNetworks-2

8/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

X1

X2

Z1

Z2

Y1

Y2

Heat

Cold

2

2

2

12

-1

1

Hot

Cold

NOT

FEED-FORWARD

8/3/2019 NeuralNetworks-2

9/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

X1

X2

Z1

Z2

Y1

Y2

Heat

Cold

2

2

2

12

-1

1

Hot

Cold

It takes time for the

stimulus (applied at

X1 and X2) to make

its way to Y1 and Y2

where we perceiveeither heat or cold

A

t t(0), we apply a stimulus to X1 and X2 At t(1) we can update Z1, Z2 and Y1

At t(2) we can perceive a stimulus at Y2

At t(2+n) the network is fully functional

8/3/2019 NeuralNetworks-2

10/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

We want the system to perceive cold if a coldstimulus is applied for two time steps

Y2(t) = X2(t 2) AND X2(t 1)

X2(t 2) X2( t 1) Y2(t)

1 1 1

1 0 00 1 0

0 0 0

8/3/2019 NeuralNetworks-2

11/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural NetworksWe want the system to perceive heat if either a hot stimulus is

applied or a cold stimulus is applied (for one time step) and

then removed

Y1(t) = [X1(t 1) ] OR [X2(t 3)AND NOTX2(t 2) ]

X2(t 3) X2(t 2) AND NOT X1(t 1) OR

1 1 0 1 1

1 0 1 1 1

0 1 0 1 1

0 0 0 1 1

1 1 0 0 0

1 0 1 0 1

0 1 0 0 0

0 0 0 0 0

8/3/2019 NeuralNetworks-2

12/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

The network shows

Y1(t) =X1(t 1) ORZ1(t 1)

Z1(t 1) =Z2( t 2) AND NOTX2(t 2)

Z2(t 2) =X2(t 3)

Substituting, we get

Y1(t) = [X1(t 1) ] OR [X2(t 3) AND NOTX2(t 2) ]

which is the same as our original requirements

8/3/2019 NeuralNetworks-2

13/18

G5G51IAI1IAI Neural NetworksNeural Networks

The First Neural Neural Networks

You can confirm that Y2 works correctly

You can also check it works on the

spreadsheet

Threshold

2

Tim e Hea t (X 1) C ol d (X 2) Z1 Z2 Hot (Y1) Col d (Y2)

0 0 1

1 0 0 0 1

2 0 0 1 0 0 0

3 0 0 1 0

Time Heat (X1) Cold (X2) Z1 Z2 Hot (Y1) Cold (Y2) X1 X2 Z1 Z2

0 0 1 Z1 -1 2

1 0 1 0 1 Z2 2

2 0 0 0 1 0 1 Y1 2 2

Y2 1 1

Tim e Hea t (X 1) C ol d (X 2) Z1 Z2 Hot (Y1) Col d (Y2)

0 1 0

1 1 0 0 0

2 0 0 0 0 1 0

Read across to see the inputs to

each neuron

Apply cold for one time step

and we perceive heat

Apply cold for two time ste ps

and we perceive cold

See Fausett, 1994, pp 31 - 35

Apply heat and we perceive

heat

8/3/2019 NeuralNetworks-2

14/18

Recurrent Networks

In the previous slides the sidewaysedge from Z2 to Z1 allowed use of

inputs from different time-steps

A recurrent neural network is a

neural network where the connectionsbetween the units form a directedcycle. (Wikipedia)

8/3/2019 NeuralNetworks-2

15/18

Recurrent Networks

The network structure has backwards edges suchthat signals can go around a loop

due to the time-stepping this gives a form of memory

recurrent networks can be used to model processes thathave some memory can learn and predict patterns in a time-series data

e.g. stock market

Contrast these with feed-forward networks signals move forward only usually used to compute a function of the current inputs

output value is independent of previous inputs

no memory input is a static quantity (a snap-shot)

8/3/2019 NeuralNetworks-2

16/18

Expectations

Understand utility of time-steppedcomputation

keep some state, memory, betweeninputs

Recurrent networks

can model memory be able to contrast with feed-forward

networks

8/3/2019 NeuralNetworks-2

17/18

G5G51IAI1IAI Neural NetworksNeural Networks

And Finally.

If the brain were so simple

that we could understand it

then wed be so simple that

we couldnt

LyallWatson

8/3/2019 NeuralNetworks-2

18/18

G51IAI

Introduction to AI

Andrew Parkes

End of Neural Networks