Post on 10-Feb-2017
transcript
1
Using Neural Networks forPattern Classification Problems
Converting an Image• Camera captures
an image
• Image needs to beconverted to a formthat can beprocessed by theNeural Network
2
Converting an Image• Image consists
of pixels• Values can be
assigned tocolor of eachpixel
• A vector canrepresent thepixel values inan image
Converting an Image
• If we let +1represent blackand 0 representwhite
• p = [0 1 0 1 0 10 1 0 0 0 1 0 0 01 0 …..
3
Neural Network PatternClassification Problem
Tank image =[0 1 0 0 1 1 0…. ]
House image= [1 1 0 0 0 10 ….]
Neural NetworkTank orhouse ?
Types of Neural Networks
• Perceptron• Hebbian• Adeline• Multilayer with Backpropagation• Radial Basis Function Network
4
2-Input Single NeuronPerceptron: Architecture
aaa
+n ap1
p2
1
b
w1,1
w1,2
a = hardlims Wp+b( ) = hardlims w1,1
w1,2[ ]
p1
p2
!
" #
$
% & + b
'
( )
*
+ ,
= hardlims w1,1p
1+ w
1,2p
2+ b( ) =
-1, if w1,1p
1+ w
1,2p
2+ b < 0
+1, if w1,1p
1+ w
1,2p
2+ b . 0
/ 0 1
A single neuronperceptron:
Output:
Symmetrical hardlimiter
2-Input Single NeuronPerceptron: Example
a = hardlims w1,1p
1+ w
1,2p
2+ b( )
=!1, w
1,1p
1+ w
1,2p
2+ b < 0
+1, w1,1p
1+ w
1,2p
2+ b " 0
# $ %
Example: w1,1 = -1 w1,2 = 1 b = -1
a =!1, ! p
1+ p
2!1 < 0 or ! p
1+ p
2< 1
+1, ! p1
+ p2!1" 0 or ! p
1+ p
2"1
#
$
%
This separates the inputs p = [p1, p2]T into twocategories separated by the boundary: -p1 + p2 = 1
aaa
+n ap1
p2
1
b
w1,1
w1,2
5
a
p1
p2
-1
1
decision boundary - p1 + p2 = 1
(-2,1)
(2,-1)
2-Input Single NeuronPerceptron: Decision Boundary
a =!1, ! p
1+ p
2!1 < 0 or ! p
1+ p
2< 1
+1, ! p1
+ p2!1" 0 or ! p
1+ p
2"1
#
$
%
Inputs in this region have an outputof +1
Inputs in this regionhave an output of -1
aaa
+n ap1
p2
1
-1
-1
1
aaa
+n ap1
p2
1
-1
-1
1
a
p1
p2
-1
1
decision boundary - p1 + p2 = 1W = [-1, 1]
2-Input Single NeuronPerceptron: Weight Vector
• The weight vector, W, is orthogonal to thedecision boundary
6
2-Input Single NeuronPerceptron: Weight Vector
• W points towards the class with an output of +1
a
p1
p2
-1
1
decision boundary - p1 + p2 = 1
(-2,1)
(2,-1)
W
Simple Perceptron Design
• The design of a simple perceptron isbased upon:– A single neuron divides inputs into two
classifications or categories– The weight vector, W, is orthogonal to the
decision boundary– The weight vector, W, points towards the
classification corresponding to the “1” output
7
Orthogonal Vectors
• For any hyperplane of the form:a1p1 + a2p2 + a3p3 + . . . + anpn = b
the vector c*[ a1, a2, …, an ] is orthogonal tothe hyperplane (where c is a constant).
- p1 + p2 = - 1 * p1 + 1*p2 = 1
W = [ -1 , 1 ]
AND Gate: Description
• A perceptron can be used to implement most logic functions
• Example: Logical AND Truth table:
111001010000
OutputInputs
8
AND Gate: Architecture
“hardlim” is usedhere to provideoutputs of 0 and 1
aaa
+n ap1
p2
1
b
w1,1
w1,2
TwoinputAND
p1 =0
0
!
" #
$
% & , t1 = 0
'
(
)
*
+
,
p2 =0
1
!
" #
$
% & , t2 = 0
'
(
)
*
+
,
p3 =1
0
!
" #
$
% & , t3 = 0
'
(
)
*
+
,
p4 =1
1
!
" #
$
% & , t4 = 1
'
(
)
*
+
,
Input/Target pairs:
AND Gate: GraphicalDescription
• Graphically:
• Where do we place the decision boundary?
a
= zero output= one output
1
1
111001010000
OutputInputs
p1
p2
9
AND Gate: DecisionBoundary
• There are an infinite number of solutions
• What is the corresponding value of W?
One possible decisionboundary
1
1
AND Gate: Weight Vector• W must be orthogonal to the decision boundary• W must point towards the class with an output of 1
• Output:
a
W1.5
1.5
a = hardlim 2 2[ ]p1
p2
!
" #
$
% & + b
' ( )
* + ,
= hardlim 2p1
+ 2p2
+ b{ }
One possiblevalue is [2 2]
1
1
Decision boundary
10
AND Gate: Bias• Decision Boundary:
• At (1.5, 0): 2(1.5) + 2(0) + b = 0 b = -3
2p1
+ 2p2
+ b = 0
a
W1.5
1.5
Passes through (1.5, 0)
1
1
AND Gate: Final Design
• Final Design:
• Test:
a = hardlim 2 2[ ]p1
p2
!
" #
$
% & ' 3
( ) *
+ , -
aaa
+n ap1
p2
1
-32
2
a = hardlim 2 2[ ]0
0
!
" # $
% & ' 3
( ) *
+ , -
= 0
a = hardlim 2 2[ ]0
1
!
" # $
% & ' 3
( ) *
+ , -
= 0
a = hardlim 2 2[ ]1
0
!
" # $
% & ' 3
( ) *
+ , -
= 0
a = hardlim 2 2[ ]1
1
!
" # $
% & ' 3
( ) *
+ , -
=1
11
Perceptron Learning Rule• Most real problems involve input vectors,
p, that have length greater than three• Images are described by vectors with
1000s of elements• Graphical approach is not feasible in
dimensions higher than three• An iterative approach known as the
Perceptron Learning Rule is used
Character RecognitionProblem
• Given: A network has two possible inputs, “x” and “o”.These two characters are described by the 25 pixel (5 x5) patterns shown below.
• Problem: Design a neural network using the perceptronlearning rule to correctly identify these input characters.
x o
12
Character RecognitionProblem: Input Description
• The inputs must be described as columnvectors
• Pixel representation: 0 = white 1 = black
The “x” is represented as: [ 1 0 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 1 0 1 0 0 0 1]T
The “o” is represented as: [ 0 1 1 1 0 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 0 1 1 1 0 ]T
Character RecognitionProblem: Output Description
• The output will indicate that either an “x” or“o” was received
• Let: 0 = “o” received 1 = “x” received
• The inputs are divided into two classesrequiring a single neuron
• Training set:p1 = [ 1 0 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 1 0 1 0 0 0 1]T, t1 = 1
p2 = [ 0 1 1 1 0 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 0 1 1 1 0 ]T , t2 = 0
A hard limiterwill be used
13
Character RecognitionProblem: Network Architecture
aa
+n a
p1p2
1
b
w1,1w1,2
p25
w1,25
“hardlim is used toprovide an outputof “0” or “1”
a = hardlim(Wp+ b)The input,p, has 25components
Perceptron Learning Rule:Summary
• Step 1: Initialize W and b (if non zero) to small randomnumbers.
• Step 2: Apply the first input vector to the network andfind the output, a.
• Step 3: Update W and b based on:
Wnew = Wold + (t-a)pT
bnew = bold + (t-a)
• Repeat steps 2 and 3 for all input vectors repeatedlyuntil the targets are achieved for all inputs
14
Character Recognition Problem:Perceptron Learning Rule
• Step 1: Initialize W and b (if non zero) to small random numbers.
– Assume W = [0 0 . . . 0] (length 25) and b = 0
• Step 2: Apply the first input vector to the network
–
–
• Step 3: Update W and b based on:
Wnew = Wold + (t-a)p1T = Wold + (1-1)p1
T
= [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
bnew = bold + (t-a) = bold + (1-1) = 0
p1 = [ 1 0 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 1 0 1 0 0 0 1]T, t1 = 1
a = hardlim(W(0)p1 + b(0)) = hardlim(0) = 1
Character Recognition Problem:Perceptron Learning Rule
• Step 2 (repeated): Apply the second input vector to the network
–
–
• Step 3 (repeated): Update W and b based on
Wnew = Wold + (t-a)p1T = Wold + (0-1)p2
T
= [0 -1 -1 -1 0 -1 0 0 0 -1 -1 0 0 0 -1 -1 0 0 0 -1 0 -1 -1 -1 0]
bnew = bold + (t-a) = bold + (0-1) = -1
p2 = [0 1 1 1 0 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 0 1 1 1 0 ]T, t2 = 0a = hardlim(W(1)p2 + b(1)) = 1
15
Character Recognition Problem:Perceptron Learning Rule
011p10[1 -1 -1 -1 1 -1 1 0 1 -1 -1 0 1 0 -1 -1 1 0 1 -1 1 -1 -1 -1 1]
000p20[1 -1 -1 -1 1 -1 1 0 1 -1 -1 0 1 0 -1 -1 1 0 1 -1 1 -1 -1 -1 1]
101p1-1[0 -1 -1 -1 0 -1 0 0 0 -1 -1 0 0 0 -1 -1 0 0 0 -1 0 -1 -1 -1 0]
-110p20[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
011p10[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
eatpbW
Character RecognitionProblem: Results
• After three epochs, W and b converge to:– W = [1 -1 -1 -1 1 -1 1 0 1 -1 -1 0 1 0 -1 -1 1 0 1 -1 1 -1 -1 -1 1]
– b = 0
• One possible solution based on the initial conditionselected. Other solutions are obtained when the initialvalues of W and b are changed.
• Check the solution: a = hardlim(W*p + b) both both inputs
16
Character RecognitionProblem: Results
• How does this network perform in the presence of noise?
• For the “x” with noise:
a = hardlim{W*[1 0 0 0 1 0 1 0 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1] + 0} = 1
• For the “o” with noise:
a = hardlim{W*[0 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 0 1 0 1 0 1 1 1 0] + 0} = 0
• The network recognizes both the noisy x and o.
x and o with threepixel errors in each
Character RecognitionProblem: Simulation
• Use MATLAB to perform the following simulation:– Apply noisy inputs to the network with pixel errors ranging from 1
to 25 per character and find the network output
– Each type of error (number of pixels) was repeated 1000 timesfor each character with the incorrect pixels being selected atrandom
– The network output was compared to the target in each case.
– The number of detection errors was tabulated.
17
Character Recognition Problem:Performance Results
111000100016 - 25.891885100015.621616100014.28.9527694813.06.7658759120.400399110.100961000001 - 9
oxoxNo. ofPixel
Errors
Probability of ErrorNo. of Character Errors
An “o” with11 pixelerrors
Perceptrons: Limitations
• Perceptrons only work for inputs thatare linearly separable
a
xx xxo
oo
a
x
xx xooo
Linearly separable Not Linearly separable
18
Other Neural Networks
• How do the other types of neuralnetworks differ from the perceptron?– Topology– Function– Learning Rule
Perceptron Problem: Part 1
• Design a neural network that can identifya tank and a house.– Find W and b by hand as illustrated with the x-
o example.– Use the Neural Network Toolbox to find W
and b
Tank(t = 1)
House(t = 0)
19
Perceptron Problem: Part 2
• Design a neural network that can find atank among houses and trees.– Repeat the previous problem but now with a
tree included.– Both the house and tree have targets of zero.
Tank(t = 1)
House(t = 0)
Tree(t = 0)
Perceptron Problem: Part 3• Design a neural network that can find a
tank among houses, trees and otheritems.– Create other images on the 9 x 9 grid.– Everything other than a tank will have a target
of zero.– How many items can you introduce before
the perceptron learning rule no longerconverges?
Tank(t = 1)
House(t = 0)
Tree(t = 0)
+ ????
20
MATLAB: Neural NetworkToolbox
• >> nntool
MATLAB: Neural NetworksToolbox
• Go to MATLAB Help and review thedocumentation on the Neural NetworksToolbox
• Use the GUI interface (>> nntool) toreproduce the results you obtained for theperceptron (tank vs. house, tree, etc.)
• Data can be imported/exported from theworkspace to the NN Tool.