Ming-Feng Yeh 1

CHAPTER 4CHAPTER 4

Perceptron Perceptron Learning Learning

RuleRule

Ming-Feng Yeh 2

ObjectivesObjectives

How do we determine the weight matrix and bias for perceptron networks with many inputs, where it is impossible to visualize the decision boundaries?

The main object is to describe an algorithm for training perceptron networks, so that they can learn to solve classification problems.

Ming-Feng Yeh 3

History : 1943 History : 1943

Warren McCulloch and Walter Pitts introduced one of the first artificial neurons in 1943.The main feature of their neuron model is that a weighted sum of input signals is compared to a threshold to determine the neuron output.They went to show that networks of these neurons could, in principlein principle, compute any arithmetic or logic function.Unlike biological networks, the parameters of their networks had to designed, as no training method was available.

Ming-Feng Yeh 4

History : 1950sHistory : 1950s

Frank Rosenblatt and several other researchers developed a class of neural networks called perceptrons in the late 1950s.Rosenblatt’s key contribution was the introduction of a learning rule for training perceptron networks to solve pattern recognition problems.The perceptron could even learn when initialized with random values for its weights and biases.

Ming-Feng Yeh 5

History : ~1980sHistory : ~1980s

Marvin Minsky and Seymour Papert (1969) demonstrated that the perceptron networks were incapable of implementing certain elementary functions (e.g., XOR gate).It was not until the 1980s that these limitations were overcome with improved (multilayer) perceptron networks and associated learning rules.The perceptron network remains a fast and reliable network for the class of problems that it can solve.

Ming-Feng Yeh 6

Learning RuleLearning Rule

Learning rule: a procedure (training algorithm) for modifying the weights and the biases of a network. The purpose of the learning rule is to train the network to perform some task.Supervised learning, unsupervised learning and reinforcement (graded) learning.

Ming-Feng Yeh 7

Supervised LearningSupervised LearningThe learning rule is provided with a set of examples (the training set) of proper network behavior: {p1,t1}, {p2,t2},…, {pQ,tQ} where pq is an input to the network and tq is the corresponding correct (target) output.As the inputs are applied to the network, the network outputs are compared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network outputs closer to the targets.

Ming-Feng Yeh 8

Supervised LearningSupervised Learning

Ming-Feng Yeh 9

Reinforcement LearningReinforcement Learning

The learning rule is similar to supervised learning, except that, instead of being provided with the correct output for each network input, the algorithm is only given a grade.The grade (score) is a measure of the network performance over some sequence of inputs.It appears to be most suited to control system applications.

Ming-Feng Yeh 10

Unsupervised LearningUnsupervised Learning

The weights and biases are modified in response to network inputs only. There are no target outputs available.

Most of these algorithms perform some kind of clustering operation. They learn to categorize the input patterns into a finite number of classes. This is especially in such applications as vector quantization.

Ming-Feng Yeh 11

Two-Input / Single-Neuron Two-Input / Single-Neuron PerceptronPerceptron

1p

2p

1

1

1

0a

1a

W

n = 0The boundary is always orthogonal to W.

11w

12w

1p

2p n a

1

b 0:

)(hardlim

212111

212111

bpwpwn

nabpwpwbn

boundary Dicision

Wp

011 ,1 ,1

21

1211

ppnbww

11

0

211hardlim

02 T

a

ppoint toward

Ming-Feng Yeh 12

Perceptron Network DesignPerceptron Network Design

W

The input/target pairs for the AND gate are

.1,1

1,0,

0

1,0,

1

0,0,

0

044332211

tttt pppp

AND

Dark circle : 1Light circle : 0

Step 1: Select a decision boundary

Step 2: Choose a weight vector W that is orthogonal to the decision boundary

Step 3: Find the bias b, e.g., picking a point on the decision boundary and satisfying n = Wp + b = 0

22 W

305.1 T bp

Ming-Feng Yeh 13

Test ProblemTest ProblemThe given input/target pairs are

.0,1

0,0,

2

1,1,

2

1332211

ttt ppp

Two-input and one output network without a bias the decision boundary must pass through the origin.

The length of the weight vector does not matter; only its direction is important.

Dark circle : 1Light circle : 0

p1p2

p3

Ming-Feng Yeh 14

W

Constructing Learning RuleConstructing Learning RuleTraining begins by assigning some initial values for

.8.00.1 ,1,2

111

Wp t

the network parameters.

06.0hardlim2

18.00.1hardlim

a

p1p2

p3

The network has not returned the correct value, a = 0 and t1 = 1.

The initial weight vector results in a decision boundary that incorrectly classifies the vector p1.

Ming-Feng Yeh 15

Constructing Learning RuleConstructing Learning Rule

W

p1p2

p3

One approach would be set W equal to p1 such that p1 was classified properly in the future.

Unfortunately, it is easy to construct a problem for which this rule cannot find a solution.

Ming-Feng Yeh 16

W

Constructing Learning RuleConstructing Learning Rule

W

p1p2

p3

Another way would be to add W equal to p1. Adding p1 to W would make W point more in the direction of p1.

. then ,0 and 1 If pWW oldnewat

2.1

0.2

2

1

8.0

0.11pWW oldnew

Ming-Feng Yeh 17

W

Constructing Learning RuleConstructing Learning Rule

The next input vector is p2.

. then ,1 and 0 If pWW oldnewat

14.0hardlim2

12.10.2hardlim

a

A class 0 vector was misclassified as a 1, a = 1 and t2 = 0.

8.0

0.3

2

1

2.1

0.22pWW oldnew

p1p2

p3W

Ming-Feng Yeh 18

W

Constructing Learning RuleConstructing Learning Rule

Present the 3rd vector p3

18.0hardlim1

08.00.3hardlim

ap1

p2

p3

. then ,1 and 0 If pWW oldnewat

A class 0 vector was misclassified as a 1, a = 1 and t2 = 0.

2.0

0.3

1

0

8.0

0.33pWW oldnew

W

Ming-Feng Yeh 19

One iteration

Constructing Learning RuleConstructing Learning Rule If we present any of the input vectors to the neuron,

it will output the correct class for that input vector. The perceptron has finally learned to classify the three vectors properly.

The third and final rule: . then , If oldnewat WW

. then , If. then ,1 and 0 If. then ,0 and 1 If

oldnew

oldnew

oldnew

atatat

WWpWWpWW

Training sequence: p1 p2 p3 p1 p2 p3

Ming-Feng Yeh 20

Unified Learning RuleUnified Learning Rule

Perceptron error: e = t – a

. then ,0 If. then ,1 If. then ,1 If

oldnew

oldnew

oldnew

eee

WWpWWpWW

. then , If. then ,1 and 0 If. then ,0 and 1 If

oldnew

oldnew

oldnew

atatat

WWpWWpWW

pWpWW )( ate oldoldnew

)(1 atbebb oldoldnew

Ming-Feng Yeh 21

Training Multiple-Neuron Training Multiple-Neuron PerceptronPerceptron

TT )( patWepWW oldoldnew

)( atbebb oldoldnew

Learning rate :

10 T epWW oldnew

ebb oldnew

Ming-Feng Yeh 22

Apple/Orange Apple/Orange Recognition ProblemRecognition Problem

1,

1

1

1

,0,

1

1

1

2211 tt pp

50 ,5.015.0 .bW

5.0 ,5.005.011)5.2(hardlim)(hardlim

T11

ebbeateba

oldnewoldnew pWWWp

5.0 ,5.015.010)5.0(hardlim)(hardlim

T22

ebbeateba

oldnewoldnew pWWWp

5.0 ,5.025.011)5.0(hardlim)(hardlim

T11

ebbeateba

oldnewoldnew pWWWp

Ming-Feng Yeh 23

LimitationsLimitations

The perceptron can be used to classify input vectors that can be separated by a linear boundary, like AND gate example.

linearly separable (AND, OR and NOT gates)

Not linearly separable, e.g., XOR gate

Ming-Feng Yeh 24

Solved Problem P4.3Solved Problem P4.3

(1,2)

(1,1)

(2,0)

(2,1)

(1,2)

(2,1)

(1, 1)

(2, 2)

Class 1: t = (0,0)

Class 2: t = (0,1)

Class 4: t = (1,1)

Class 3: t = (1,0)

Design a perceptron network to solve the next problem

A two-neuron perceptron creates two decision boundaries.

Ming-Feng Yeh 25

Solution of P4.3Solution of P4.3

0

0,

2

1,

1

12121 ttpp

Class 1

1

0,

0

2,

1

24343 ttpp

Class 2

0

1,

1

2,

2

16565 ttpp

Class 3

1

1,

2

2,

1

18787 ttpp

Class 4

00

021

2

1

11

013

1

3

T222

T111

pww

pww

b

b

0

1 ,

21

13T

2

T1 bw

wW

0

1 ,

21

13T

2

T1 bw

wW

-3

-1

-2

1

1

0

xin xout

yin yout

Ming-Feng Yeh 26

Solved Problem P4.5Solved Problem P4.5

Train a perceptron network to solve P4.3 problem using the perceptron learning rule.

.1

1)0( ,

10

01)0(

bW

1

1

1

1)0()0(hardlim 11 atebpWa

0

0 ,

1

111 tp

0

0)0()1( ,

01

10)0()1( T

1 ebbepWW

Ming-Feng Yeh 27

Solution of P4.5Solution of P4.5

.0

0)1( ,

01

10)1(

bW

0

0

0

0)1()1(hardlim 22 atebpWa

0

0 ,

2

122 tp

.0

0)1()2(

,01

10)1()2( T

2

ebb

epWW

Ming-Feng Yeh 28

Solution of P4.5Solution of P4.5.

0

0)2( ,

01

10)2(

bW

1

1

0

1)2()2(hardlim 33 atebpWa

1

0 ,

1

233 tp

.1

1)2()3(

,11

02)2()3( T

3

ebb

epWW

)3()4()5()6()7()8()3()4()5()6()7()8(

bbbbbbWWWWWW

Ming-Feng Yeh 29

Solution of P4.5Solution of P4.5

.1

1)8( ,

11

02)8(

bW

0

0 ,

1

111 tp

1

0

1

0)8()8(hardlim 11 atebpWa

.0

1)8()9(

,20

02)8()9( T

1

ebb

epWW -2

0

-2

0

-1

0

xin xout

yin yout