ANN : An introduction

What is a neural network ? a branch of "Artificial Intelligence". It can be considered as a black box that

is able to predict an output pattern when it recognizes a given input pattern. Once trained, the neural network is able to recognize similarities when presented with a new input pattern, resulting in a predicted output pattern.

Why Neural Network ?

Neural Network is a fascinating technology, 50 years old, but still not fully employed.

And the question is why? Why didn't Neural Network progress as fast as many other technologies?

The concept has been there since early 1950s, but was mostly dormant until the mid 1980s. One of the first NN developed was the perceptron created by a psychologist, Frank Rosenblatt in 1958. The perceptron was a very simple system used to analyze data and visual patterns, which generated a great deal of interest in AI community.

Unfortunately, people started exaggerating the potential of neural networks. Scientists claimed that with enough complexity & speed, the perceptron would be able to solve almost any problem.

In 1969, Marvin Minsky & Seymour Papert of MIT published a book, which showed that the perceptron could never solve a class of problems, and hinted at several other fundamental flaws in the model.

Their analysis combined with unfulfilled, outrageous claims convinced the AI community; and the bodies that fund it; of the fruitlessness of pursuing work with NN, and the majority of researchers turned away from the approach.

Funding was halted & scientists working on NN found it almost impossible to receive funding.

Era of Neural Network crisis

This period of stunted growth lasted through 80s. In 1982 John Hopfield of Caltech presented a paper to the national Academy of Sciences. With clarity & mathematical analysis, he showed how such networks could work and what they could do.

By 1985 the American Institute of Physics began an annual meeting of Neural Networks for Computing. By 1987, the Institute of Electrical and Electronic Engineer's (IEEE) first International Conference on Neural Networks drew more than 1,800 attendees.

Neural Network back in the limelight

Present scenario

Thanks to the availability of cheap microprocessors and recent discoveries about DNA and human brain, artificial intelligence has gone from being a fantasy to becoming a reality. In fact, most AI researchers believe that it's only a matter of 20 to 30 years before machines become at least as intelligent as humans.

Now ... Neural Network is back and this time ... to stay ... Yet, its future, indeed the very key to the whole technology, lies in its commercial use.

In brief Pre-1940: Von Hemholtz, Mach, Pavlov, etc.

– General theories of learning, vision, conditioning– No specific mathematical models of neuron operation

1940s: Hebb, McCulloch and Pitts– Mechanism for learning in biological neurons– Neural-like networks can compute any arithmetic function

1950s: Rosenblatt, Widrow and Hoff– First practical networks and learning rules

1960s: Minsky and Papert– Demonstrated limitations of existing neural networks, some

research suspended 1970s: Amari, Anderson, Fukushima, Grossberg, Kohonen

– Progress continues, although at a slower pace 1980s: Grossberg, Hopfield, Kohonen, Rumelhart, etc.

– Important new developments cause a resurgence in the field

Where is computer industry headed?

In 1982 IBM introduced the first PC with 64 KB Memory & 5 MB hard-drive for about $3000. In 1991 the same money could buy a PC with 16 MB Memory & 1 GB hard-drive. In other words in just 15 years disk storage has increased 200 times. Trend indicates exponential growth in both performance and storage space of computers.

That will obviously create demand for smarter and more flexible software solutions. As example:Businesses are already gathering gigabyte of data daily and that is also rapidly growing. But we are heading towards Terra Bytes of data!

Everybody would agree that a natural direction for computers would be Soft Computing and Artificial Intelligence. A path that will lead us to solutions for our demanding future needs.

Trendy applications such as Data Mining, Business Intelligence and Robotics have recognized that fact and are already utilizing AI in many different ways. From Heuristic Algorithms, Fuzzy Logic, Genetic and Evolutionary Algorithms to Neural Nets.

Neural Networks satisfies many requirements of futuristic applications such as:- Parallel Processing- Fault tolerance- Self-organization- Generalization ability- Continuous adaptivity

Uses A perfect match for Applications such as:

Intelligent Agents, Monitoring and Warning Systems, Advance Decision Support System (ADSS), Process Automation, Intelligent Personal Assistant and Smart devices.

But yet, Neural Nets are not without problems.- The inner workings of neural networks are like "black boxes."

They learn and model based on experience, but they cannot explain and justify their decisions.

Limitations Neural networks might have high accuracy, but not

100%, as we would want. And unfortunately, not all applications can tolerate that level of error.

But they still can be used in conjunction with traditional methods to, for example, cutting down on time in search.

Neural networks require lot of (sample) data for training purpose. It may have been an obstacle few years ago, but as we move forward that isn't a problem. In this era of digital technology, everything is digitally recorded from bank transactions to phone conversations and from medical data to lab results.

Applications

Aerospace : High performance aircraft autopilots, flight path simulations, aircraft control systems, autopilot enhancements, aircraft component simulations, aircraft component fault detectors

Automotive : Automobile automatic guidance systems Banking : Check & other document readers, credit application

evaluators Defense: Weapon steering, target tracking, object discrimination,

facial recognition, new kinds of sensors, sonar, radar and image signal processing including data compression, feature extraction and noise suppression, signal/image identification

Electronics : Code sequence prediction, integrated circuit chip layout, process control, chip failure analysis, machine vision, voice synthesis, nonlinear modeling

Applications (a world survey !) Financial: Real estate appraisal, loan advisor, mortgage screening,

corporate bond rating, credit line use analysis, portfolio trading program, corporate financial analysis, currency price prediction.

Manufacturing: Manufacturing process control, product design and analysis, process and machine diagnosis, real-time particle identification, visual quality inspection systems, beer testing, welding quality analysis, paper quality prediction, computer chip quality analysis, analysis of grinding operations, chemical product design analysis, machine maintenance analysis, project bidding, planning and management, dynamic modeling of chemical process systems.

Medical: Breast cancer cell analysis, EEG and ECG analysis, optimization of transplant times, hospital expense reduction, hospital quality improvement, emergency room test advisement.

Applications Robotics : Trajectory control, forklift robot, manipulator

controllers, vision systems

Speech: Speech recognition, speech compression, vowel classification, text to speech synthesis

Securities: Market analysis, stock trading advisory systems

Telecommunications: Image and data compression, automated information services, real-time, translation of spoken language, customer payment processing systems

Transportation: vehicle scheduling, routing systems

What is an ANN ? It is a system loosely modeled based on the human

brain. The field goes by many names, such as

connectionism, parallel distributed processing, neuro-computing, natural intelligent systems, machine learning algorithms, and artificial neural networks.

It is an inherently multiprocessor-friendly architecture. It has ability to account for any functional dependency. The network discovers (learns, models) the nature of the dependency without needing to be prompted.

a powerful technique to solve many real world problems.

have the ability to learn from experience in order to improve their performance & to adapt themselves to changes in the environment.

In addition to that they are able to deal with incomplete information or noisy data and can be very effective especially in situations where it is not possible to define the rules or steps that lead to the solution of a problem.

They typically consist of many simple processing units, which are wired together in a complex communication network.

There is no central CPU following a logical sequence of rules - indeed there is no set of rules or program. This structure is similar to the physical workings of the brain cells & leads to a new type of computer that is rather good at a range of complex tasks.

A simple neuron model

What it can do? In principle, NNs can compute any

computable function, i.e. they can do everything a normal digital computer can do. Especially anything that can be represented as a mapping between vector spaces can be approximated to arbitrary precision by Neural Networks.

In practice, NNs are especially useful for mapping problems which are tolerant of some errors and have lots of example data available, but to which hard and fast rules can not easily be applied.

Classifications of NN

Neural Network Applications can be grouped into four categories

Function approximation

Clustering

Classification/Pattern recognition

Prediction/Dynamical Systems

Clustering

A clustering algorithm explores the similarity between patterns and places similar patterns in a cluster. Best known applications include

data compression and data mining.

Classification/Pattern recognition

The task of pattern recognition is to assign an input pattern (like handwritten symbol) to one of many classes. This category includes algorithmic implementations such as associative memory.

Function approximation

The tasks of function approximation is to find an estimate of the unknown function f() subject to noise. Various engineering and scientific disciplines require function approximation.

Prediction/Dynamical Systems

The task is to forecast some future values of a time-sequenced data. Prediction has a significant impact on decision support systems. Prediction differs from Function approximation by considering time factor.

Here the system is dynamic and may produce

different results for the same input data based on system state (time).

Neural Network types can be classified based on following attributes

1. Applications* Classification* Clustering* Function approximation* Prediction

Neural Network Types

Connection Type

Static (feed forward)

Dynamic (feedback)

Topology

- Single layer- Multilayer- Recurrent- Self-organized- . . .

Learning Methods

- Supervised

- Unsupervised

Learning can be done in supervised or unsupervised manner.

In supervised training, both the inputs and the outputs are provided.

The network then processes the inputs and compares its resulting outputs against the desired outputs. Errors are then calculated, causing the system to adjust the weights which control the network. This process occurs over and over as the weights are continually tweaked.

In unsupervised training, the network is provided with inputs but not with desired outputs. The system itself must then decide what features it will use to group the input data. This is often referred to as self-organization or adaptation.

Learning Process

One of the most important aspects of Neural Network is the learning process.

Cont. The learning process can be viewed as reshaping

a sheet of metal, which represents the output (range) of the function being mapped. The training set (domain) acts as energy required to bend the sheet of metal such that it passes through predefined points. However, the metal, by its nature, will resist such reshaping. So the network will attempt to find a low energy configuration (i.e. a flat/non-wrinkled shape) that satisfies the constraints (training data).

ANN in the Real World

Neural networks are appearing in ever increasing numbers of real world applications and are making real money.

Indeed, some banks have proven that the failure rate on loans approved by neural networks is lower than those approved by some of their best traditional methods. Also, some credit card companies are using neural networks in their application screening process.

Neural networks (MFNNs and SOFMs) form the core of most commercial data mining packages such as the SAS Enterprise Miner and the IBM Intelligent Miner.

OCR has become one of the biggest commercial applications of NNWs. Caere Inc, a leader in the market generated $55M in 1997 through OmniPage product line.

Biological Neurons Neurons respond slowly The brain uses massively parallel computation 1011 neurons in the brain 104 connections per neuron