8/3/2019 L17 My Neural Nw

1/32

241-320 Design Architecture and Engineeringfor Intelligent System

Suntorn Witosurapot

Contact Address:Phone: 074 287369 or

Email: wsuntorn@coe.psu.ac.th

January 2011

8/3/2019 L17 My Neural Nw

2/32

Lecture 17:

Introduction to Neural Networks

8/3/2019 L17 My Neural Nw

3/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 3

What Are Artificial Neural Networks?

An extremely simplified model of the brain

Essentially a function approximator

Transforms inputs into outputs to the best of its ability

8/3/2019 L17 My Neural Nw

4/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 4

What Are Artificial Neural Networks?

Composed of many neurons that co-operate toperform the desired function

8/3/2019 L17 My Neural Nw

5/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 5

What Are They Used For?

Classification

Pattern recognition, feature extraction, imagematching

Noise Reduction

Recognize patterns in the inputs and producenoiseless outputs

Prediction Extrapolation based on historical data

8/3/2019 L17 My Neural Nw

6/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 6

What Are They Used For?(cont.)

Examples: Predicting time-series in financial world () Fraud detection (- )

Object Separation () etc

Very useful in Data Mining

imitate a humans ability to learn from experience

better results are the hope(Adequately designed and trained NN can capture varied patterns)

Drawback models tend to be difficult to understand(Is it necessary to understand?)

8/3/2019 L17 My Neural Nw

7/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 7

What Are They Used For?(cont.)

is not applying some set formula, but balancing herexperience and knowledge of sale prices of similarhousesher knowledge about housing prices is not

static...fine tuning her calculation to fit the latest data

Real EstateAppraiser

8/3/2019 L17 My Neural Nw

8/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 8

Loan Prospector ()

A Neural Network is like a black box that knows how toprocess inputs to create a useful output. The calculation isquite complex and difficult to understand, yet the resultsare often useful

8/3/2019 L17 My Neural Nw

9/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 9

Neural Net Attractions

Ability to learn

NNs figure out how to perform their function on

their own

Determine their function based only upon sampleinputs

Ability to generalize

i.e. produce reasonable outputs for inputs it hasnot been taught how to deal with

8/3/2019 L17 My Neural Nw

10/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 10

Neural Net Limitations

Neural Nets are good for prediction and estimationwhen:

Inputs are well understood

Output is well understood Adequate examples are trained the neural net

Neural Nets are only as good as the training set usedto generate it. The resulting model is static and mustbe updated with more recent examples and retrainingfor it to stay relevant

8/3/2019 L17 My Neural Nw

11/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 11

Feed-Forward Neural Net Examples

One-way flow through the network from inputs to outputs

8/3/2019 L17 My Neural Nw

12/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 12

Feed-Forward Neural Net Examples(cont.)

8/3/2019 L17 My Neural Nw

13/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 13

A Biological neuron

cell andnucleus

Axon(Neurit)

Dendrits

Synapsis

Dendrits: (Input)Getting other activations

Axon: (Output ) forward the activation(from 1mm up to 1m long)

Synapse: transfer of activation:

to other cells, e.g. Dendrits of otherneurons

a cell has about 1.000 to 10.000

connections to other cells Cell Nucleus: (processing)

evaluation of activation

8/3/2019 L17 My Neural Nw

14/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 14

Natural vs. Artificial Neuron

cell andnucleus

Axon(Neurit)

Dendrits

Synapsis

j

jjii ownet

),( iiacti netfact

)(iouti

actfo

w1i w2i wji...

oi

net : input from the network

w : weight of a connectionact : activationfact : activation function :bias/thresholdfout : output function (mostly ID)o : output

8/3/2019 L17 My Neural Nw

15/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 15

Abstraction

Dendrits:weighted (real number) connections

Axon: output: real number

Synapse: ---(identity: output is directly forwarded)

Cell nucleus:unit contains simple functions

input = (many) real numbersprocessing = activation functionoutput = real number (~activation)

8/3/2019 L17 My Neural Nw

16/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 16

Artificial Neuron

8/3/2019 L17 My Neural Nw

17/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 17

Loan Prospector()

8/3/2019 L17 My Neural Nw

18/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 18

NN can have multiple output neurons

8/3/2019 L17 My Neural Nw

19/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 19

How does Neural Net work?

8/3/2019 L17 My Neural Nw

20/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 20

Common transfer functions

8/3/2019 L17 My Neural Nw

21/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 21

Where Do The Weights Come From?

The weights in a neural network are the most importantfactor in determining its function

Training - process of setting the best weights on the

edges connecting all the units in the network

Use the training set to calculate weights such that NNoutput is as close as possible to the desired output foras many of the examples in the training set as possible

8/3/2019 L17 My Neural Nw

22/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 22

Where Do The Weights Come From?(cont.)

Network

changingnetwork parameters

evaluationnetwork errorlearning

examples

8/3/2019 L17 My Neural Nw

23/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 23

Where Do The Weights Come From? (cont.)

There are two main types of training

Supervised Training

Supplies the neural network with inputs and the desired outputs

Response of the network to the inputs is measured

Adjust weights such that differences between desired andactual outputs are minimized

Unsupervised Training

Only supplies inputs

The neural network adjusts its own weights so that similar inputsshould generate the similar outputs

The network identifies the patterns and differences in theinputs without any external assistance

8/3/2019 L17 My Neural Nw

24/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 24

Example: Voice Recognition

Task: Learn to discriminate between twodifferent voices saying Hello

Data Sources

Steve Simpson

David Raubenheimer

Format Frequency distribution (60 bins)

Analogy: cochlea

8/3/2019 L17 My Neural Nw

25/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 25

Network architecture

Feed forward network

60 input (one for each frequency bin) 6 hidden

2 output (0-1 for Steve, 1-0 for David)

8/3/2019 L17 My Neural Nw

26/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 26

Presentingthe data

Steve

David

8/3/2019 L17 My Neural Nw

27/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 27

Presenting the data (untrained network)

Steve

David

0.43

0.26

0.73

0.55

8/3/2019 L17 My Neural Nw

28/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 28

Calculate error

Steve

David

0.430 = 0.43

0.261 = 0.74

0.731 = 0.27

0.550 = 0.55

8/3/2019 L17 My Neural Nw

29/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 29

Backprop error and adjust weights

Steve

David

0.430 = 0.43

0.261 = 0.74

0.731 = 0.27

0.550 = 0.55

1.17

0.82

8/3/2019 L17 My Neural Nw

30/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 30

Repeat process (sweep) for all training pairs Present data

Calculate error Backpropagate error Adjust weights

Repeat process multiple times

8/3/2019 L17 My Neural Nw

31/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks 31

Presenting the data (trained network)

Steve

David

0.01

0.99

0.99

0.01

8/3/2019 L17 My Neural Nw

32/32

241-320 Design Architecture &Engineering for Intelligent System Introduction to Neural Networks

32

Results Voice Recognition

Performance of trained network

Discrimination accuracy between knownHellos 100%

Discrimination accuracy between newHellos

100%

Summary:

Network has learnt to generalise from original data

Networks with different weight settings can havesame functionality

Trained networks concentrate on lower frequencies

Network is robust against non-functioning nodes