Outline
• Class organization
• Artificial Neural networks– Introduction– History of Artificial Neural Networks– Applications
• Survey
04/12/23 CAP 5615: Introduction to Neural Networks 2
Programming Approaches
• Programming approaches– Abstract the problem– Design an algorithm to solve the problem
• An algorithm is a precise step-by-step procedure to solve the problem
– Implement the algorithm
• Artificial neural network approaches for solving problems are different than typical programming approaches in computer science
04/12/23 CAP 5615: Introduction to Neural Networks 3
An Example
• No hands across America– Sponsored by Delco Electronics, AssistWare
Technology, and Carnegie Mellon University– Navlab 5 drove from Pittsburgh, PA to San
Diego, CA, using the RALPH computer program. – The trip was 2849 miles of which 2797 miles
were driven automatically with no hands • Which is 98.2%
04/12/23 CAP 5615: Introduction to Neural Networks 4
An Example – cont.
04/12/23 CAP 5615: Introduction to Neural Networks 5
An Example – cont.
04/12/23 CAP 5615: Introduction to Neural Networks 6
An Example – cont.
• RALPH – Rapidly Adapting Lateral Position Handler– Uses video images to determine the location of
the road ahead and the appropriate steering direction to keep the vehicle on the road
– It uses a multi-layer perceptron with backpropagation as the learning algorithm
04/12/23 CAP 5615: Introduction to Neural Networks 7
An Example – cont.
04/12/23 CAP 5615: Introduction to Neural Networks 8
Artificial Neural Network Approaches
• Artificial neural network approaches– Motivated by the biological counterpart
• The brain, which has about 1011 neurons
• All biological neural functions including learning and memory are stored in the neurons and the connections between them
• The brain is also called “biological neural network”
– Try to “learn” from examples
04/12/23 CAP 5615: Introduction to Neural Networks 9
Artificial Neural Networks – cont.
• Inspiration from neuroscience– The brain consists of 1011 highly interconnected
neurons
04/12/23 CAP 5615: Introduction to Neural Networks 10
Artificial Neural Networks – cont.
• Each neuron is a complicated process– The best available mathematical model is called
Hodgkin-Huxley Model
• We are not interested in recreating the neurons but trying to abstract their main functionalities
04/12/23 CAP 5615: Introduction to Neural Networks 11
McCulloch and Pitts Model
• McCulloch and Pitts Model– It is a simple model as a binary threshold unit– The model neuron first computes a weighted sum
of its inputs– It outputs one if the weighted sum is above a
threshold and zero otherwise
otherwise 0
0 if 1(x)
))(()1(
x
tnwtnj
ijiji
04/12/23 CAP 5615: Introduction to Neural Networks 12
McCulloch and Pitts Model – cont.
wi1
win
04/12/23 CAP 5615: Introduction to Neural Networks 13
McCulloch and Pitts Model – cont.
• McCulloch and Pitts networks (consisting of multiple layers of McCulloch and Pitts neurons) are Turing-machine equivalent in turns of computation capability– In other words, a synchronous assembly of
McCulloch-Pitts neurons can perform any computation that an ordinary digital computer can, though they can be slow or cumbersome.
04/12/23 CAP 5615: Introduction to Neural Networks 14
Digital Computers
• All digital computers consist of digital gates to control the dataflow and perform computation– How many different types of logic gates do we
need in order to be able to build any computer?
04/12/23 CAP 5615: Introduction to Neural Networks 15
McCulloch and Pitts Model – cont.
04/12/23 CAP 5615: Introduction to Neural Networks 16
History of Artificial Neural Networks
• General theories of learning– Late 19th and early 20th centuries– Interdisciplinary in physics, psychology, and
neurophysiology– Hermann Helmholtz, Ernst Mack, Ivan Pavlov
04/12/23 CAP 5615: Introduction to Neural Networks 17
History of Artificial Neural Networks – cont.
• McCulloch and Pitts Model– 1940s– Artificial neural networks in principle could
compute any arithmetic and logical functions• Origin of the modern neural networks
– Warren McCulloch and Walter Pitts– McCulloch and Pitts network cannot learn
• A network must be designed for each function
• For computers, we write a program for each function
04/12/23 CAP 5615: Introduction to Neural Networks 18
History of Artificial Neural Networks – cont.
• Hebbian learning rule– 1949– A mechanism for learning in biological neurons
• The first learning algorithm
– Donald Hebb
04/12/23 CAP 5615: Introduction to Neural Networks 19
History of Artificial Neural Networks – cont.
• Perceptron– 1950s– A network can learn to solve pattern recognition
problems– Frank Rosenblatt– Unfortunately, it only solve a limited class of
problems• Linear separability
04/12/23 CAP 5615: Introduction to Neural Networks 20
History of Artificial Neural Networks – cont.
• Widrow-Hoff learning rule– 1960s– Similar to perceptron
• Widely used in adaptive signal processing
• Most long distance phone lines use Widrow-Hoff for echo cancellation
04/12/23 CAP 5615: Introduction to Neural Networks 21
History of Artificial Neural Networks – cont.
• Minsky and Papert’s book– 1969– Pointed out the intrinsic limitations of
Perceptrons and neural networks at that time• The neural network area became quiet for almost
twenty years
04/12/23 CAP 5615: Introduction to Neural Networks 22
History of Artificial Neural Networks – cont.
• New Stage– 1980s and 1990s– New powerful computers– Two new concepts
• Statistical mechanics to analyze neural network behaviors
• Backpropagation algorithm– Overcome the limitation of perceptron
04/12/23 CAP 5615: Introduction to Neural Networks 23
History of Artificial Neural Networks – cont.
• Current stage– A standard tool to solve many practical problems– Theoretical and statistical foundations of neural
networks– New neural network architectures and learning
algorithms• Support vector machines
• Adaboost
04/12/23 CAP 5615: Introduction to Neural Networks 24
Applications
• Aerospace
• Banking
• Defense
• Financial
• Medical
• Securities
• Transportation
• See http://www.calsci.com/
04/12/23 CAP 5615: Introduction to Neural Networks 25
Financial Forecasting
• Use neural networks to predict stock market trends
• Use neural networks to perform credit scoring
• Use neural networks to perform fraud detection
• See http://www.calsci.com/Applications.html#Stock%20Applications
04/12/23 CAP 5615: Introduction to Neural Networks 26
Online Neural Network Resources
• Neural Java– Neural network tutorial with Java
• Neural network resources– http://www.makhfi.com/resources.htm
04/12/23 CAP 5615: Introduction to Neural Networks 27
Topics to Be Covered
• Basic neural network architectures and learning algorithms– Perceptron– Multiple layer perceptron and backpropagation– Associate learning– Competitive networks– Hopfield network– These topics will be on the exams
04/12/23 CAP 5615: Introduction to Neural Networks 28
Topics to Be Covered – cont.
• Advanced topics– Will NOT be on the exams but very important to
have a comprehensive understanding of the area– Radial basis neural networks– Support vector machines– AdaBoost