Artificial Neural Networks

Prof.Mohy Eldin A. Abo-Elsoud

Course Content I: Introduction

• What are Neural Networks ( NNs) ?• Why study NNs ?• What then is to prevent us from creating a nervous

system in VLSI ?• Robots Learn How to be Human• The Brain• Biological Neuron• Artificial Neuron Model• NN Applications• Where are neural networks being used?

II: Neural Computing

• Definition• A Simple Adaptable Node (The Toy Adaptive Node)• Threshold Logic Unit ( TLU)• McCulloch and Pitts

III: Learning Rules

• Introduction• Hebbien Learning Rule• Perceptron Learning Rule• Delta Learning Rule• Summary of Learning Rules

IV: Classification

• Features and Decision Region• Approximator• Autonomous Land Vehicle in NN

V: Single Layer Perceptron

VI: Multilayer Perceptron

VII: Hopfield Model

What are NNs ?

NNs are information processing systems.

A computer with a special structure : Large number of simple processors(neurons) , Large number of programble connections (synapses) per neuron.

Neural networks process information in a similar way the human brain does.

Why study NNs ?

• NNs are often used for statistical analysis and data modeling, in which their role is perceived as an alternative to standard nonlinear regression or cluster analysis techniques (classification, or forecasting), some examples include :

Image and speech recognition ,handwritten recognition, medical diagnosis, geological survey for oil, and financial market indicator prediction.

Why study NNs ?

◌◌◌Adaptive learning: A ability to learn how to do tasks based on the data given for training .

Self-Organisation: An ANN can create its own organization or representation of the information it receives during learning time.

Real Time Operation: ANN computations may be carried in parallel, and special H/W devices are being designed and manufactured which take advantage of this capability.

Fault Tolerance: Partial demage of a net leads to the corresponding degradation of performance. Howefer,some capabilities may be retained even with major damage.

What then is to prevent us from creating a nervous system in VLSI ?

Neural systems have for greater connectivity than has been possible in standard computer hardware. Many early attempts to create neural systems failed simply because no workable technology existed for realizing systems of the essential complexity.

Sufficient knowledge of the organizing principles involved in neural systems was not available.

The Brain

The Brain

• The Brain as Information Processing System.• According to a simplified account, the human brain consists of

about ten billion neurons -- and a neuron is, on average, connected to several thousand other neurons. By way of these connections, neurons both send and receive varying quantities of energy. One very important feature of neurons is that they don't react immediately to the reception of energy. Instead, they sum their received energies, and they send their own quantities of energy to other neurons only when this sum has reached a certain critical threshold. The brain learns by adjusting the number and strength of these connections. Even though this picture is a simplification of the biological facts, it is sufficiently powerful to serve as a model for the neural net.

Biological Neuron

• The human brain contains about 10 billion nerve cells or neurons. Each neuron is connected to other neurons through about 10 000 synapses.The brain’s network of neurons forms a massively parallel information processing system.

• The biological neuron consists of :

• Each neuron receives inputs from other neurons- Some neurons also connect to receptors- Cortical neurons use spikes to communicate- The timing of spikes is important

• The effect of each input line on the neuron is controlled by a synaptic weight The weights can be positive or negative

• The synaptic weights adapt so that the whole network learns to perform useful computations Recognizing objects, understanding language, making plans,

controlling the body• You have about 1011 neurons each with about 104 weights

A huge number of weights can affect the computation in a very short time. Much better bandwidth than pentium.

How the brain works

11 3

Biological Neuron

Biological Neuron

1- The cell body (soma) is the largest part of neuron which contains the nucleus.

2-Dendrites form a dendritic tree, which is a very fine. These receive information from neurons through axons-long fibres that serveas transmission lines.

3- Axon is a long cylindrical connection that carries impulses from the neuron.

Synapses

Biological Neuron

The properties of nervous systems are :• parallel, distributed information processing • high degree of connectivity among basic units • connections are modifiable based on experience • learning is a constant process, and usually unsupervised • learning is based only on local information • performance degrades gracefully

Artificial Neuron Models

Artificial Neuron Models

Artificial Neuron Models

Linear neurons

• These are simple but computationally limited If we can make them learn we may get insight into

more complicated neurons

Binary threshold neurons• McCulloch-Pitts (1943): influenced Von Neumann!

First compute a weighted sum of the inputs from other neurons Then send out a fixed size spike of activity if the weighted sum

exceeds a threshold. Maybe each spike is like the truth value of a proposition and

each neuron combines truth values to compute the truth value of another proposition!

=y

ii

iwxz ∑=

Linear threshold neurons

=jy

iji

ijj wxbz ∑+=

Sigmoid neurons

• These give a real-valued output that is a smooth and bounded function of their total input. Typically they use the

logistic function They have nice derivatives

which make learning easy (see lecture 3).

• If we treat Y as a probabilityof producing a spike, we get stochastic binary neurons.

Artificial Neuron Models

• A Simple Artificial Neuron• Our basic computational element (model neuron) is often

called a node or unit. It receives input from some other units, or perhaps from an external source. Each input has an associated weight w, which can be modified so as to model synaptic learning. The unit computes some function f of the weighted sum of its inputs:

• Its output, in turn, can serve as input to other units.

Artificial Neuron Models

• The weighted sum is called the net input to unit i, often written neti.

• Note that wij refers to the weight from unit j to unit i (not the other way around).

• The function f is the unit's activation function. In the simplest case, f is the identity function, and the unit's output is just its net input. This is called a linear unit.

Activation Function

• Step Function• The step function is f(x) = 0 if x ≤ 0 and f(x) = 1 if 0< x

. This is also called the hard limiter. Another common variation is for it to take on values -1 and +1 as shown below( binary perceptron).

Activation Function

Activation Function• Logistic Function (Sigmoid)• The sigmoid function is the most common form of

activation function used in the costruction of ANN (continuous perceptron).

• The logistic function has the form • F(x) = 1/ (1+exp(- a.x)

The parameter "a" in the logistic function determines how steep it is.

Activation Function

Activation Function

Activation Function

• Symmetric Sigmoid ( tanh)• The symmetric sigmoid is simply the sigmoid that is

stretched so that the y range is 2 and then shifted down by 1 so that it ranges between -1 and 1. If g(x) is the standard sigmoid then the symmetric sigmoid is f(x) = 2g(x) - 1

Activation Function

Comparison between The Living System and Computer

• Living System Computer

Processing Element

10E+14 Synapses 10E+8 Transistors

Element Size 10E-6 m 10E-6 m

Processing Speed 100 Hz 19E9 Hz

.Energy Use 30 w 30 w

Style of computation

Parallel,distributed Serial,centralized

Power by Metabolic biochemistry Transformer,rectifier

Neural Network Applications

1-Speech processing, in which multilayer nets are used to translate text to speechAnd vice versa.

2-Optimization, in which a Hopfield net is used to solve the optimization problem.

3-Communication, in which a N net is used to solve the communication routing Problem.

4-Medical applications, in which NNs are used to recognize anomalies in ultra-Sound images.

Where are neural networks being used?

• Signal processing: suppress line noise, with adaptive echo canceling, blind source separation

• Control: e.g. backing up a truck: cab position, rear position, and match with the dock get converted to steering instructions. Manufacturing plants for controlling automated machines.

• Siemens successfully uses neural networks for process automation in basic industries, e.g., in rolling mill control more than 100 neural networks do their job, 24 hours a day

• Robotics - navigation, vision recognition • Pattern recognition, i.e. recognizing handwritten characters, e.g. the current version of

Apple's Newton uses a neural net • Medicine, i.e. storing medical records based on case information • Speech production: reading text aloud (NETtalk) • Speech recognition • Vision: face recognition , edge detection, visual search engines • Business,e.g.. rules for mortgage decisions are extracted from past decisions made by

experienced evaluators, resulting in a network that has a high level of agreement with human experts.

• Financial Applications: time series analysis, stock market prediction • Data Compression: speech signal, image, e.g. faces • Game Playing: backgammon, chess, go, ... • Neural nets do not perform miracles. But if used sensibly they can produce some amazing

results.