Neural Networks:

An Overview

Neural Networks:

An Overview

There are many different kinds of Neural Network they include to name a few:

The Back Propagating NetworksKohonen (Self-Organising) NetworksHopfield Networks

The Boltzmann Machine

This project has used a Kohonen Network.

I shall explain the properties of such a network and why it is best for a small scale project such as this.

All Neural Networks use the concept of Neurons which are gradually trained to give a more appropriate response to input.

The inputs are added together and this result is used to generate an output for the neuron.

The inputs are generally weighted so that some inputs to the neuron may have a greater affect than others.

Kohonen (Self-Organising) Networks

Neurons have inputs and outputs.

This project has used a Kohonen Network.

I shall explain the properties of such a network and why it is best for a small scale project such as this.

w2

w3

w1

∑

A neuron of such as this one is called an Adaptive Linear Element (or Adaline) and may be found own it’s own …or may be found as part of a larger “neural network” where the output of a neuron may be the input for the next “layer”.

U

x3

x2

x1

What is a Kohonen Network?So what is a Neuron?

What is a Kohonen Network?

Kohonen (Self-Organising) Networks

A Kohonen network uses these basic principles however in a Kohonen network there is only one layer of neurons which are arranged in a 2D lattice.

There is an input layer with inputs which feed into every neuron.

x1

x2

What is a Kohonen Network?

Kohonen (Self-Organising) Networks

A Kohonen network uses these basic principles however in a Kohonen network there is only one layer of neurons which are arranged in a 2D lattice.

There is an input layer with inputs which feed into every neuron.

x1

x2

Each neuron evaluates the sum of the weighted inputs and uses this as it’s own output. This is the Euclidian distance of the neuron from the “input pattern”.

Individual neuron output

=∑(xi – wik) 2 (for all i)

Where wik is the weight between input i and neuron k.

The neuron with the smallest Euclidian distance is the one which best represents the input pattern and is known as the winner.

The key to the neural network are the weights as it is they that dictate which neuron will win when the network is presented with a particular input pattern.

How does training work?

How does training work?

Kohonen (Self-Organising) Networks

All the weights in the network are initialised to small random values.An input pattern is presented to the network and the winning neuron is found. The weights to the winning node are updated using the following equation.

wik(t+1) = wik(t) + G(t) (xi(t) –

wik(t))

The updated weight

between input i and

neuron k

The gain term

The original weight between input i and

neuron k

Input i

This is referred to as course mapping and fine mapping.

This promotes large changes away from the initial state of the network but smaller changes later when the network is some way through training.

The gain term is a value between 0 and 1 which starts high and as more patterns are presented to the network is decreased.

The use of a Neighborhood during training.

x1

x2

The use of a Neighborhood during training.

How does training work?

Kohonen (Self-Organising) Networks

Usually the weights of more neurons than the winner are changed.Once the winner has been identified a neighbourhood of a predefined shape and size is drawn around it.

All the neurons in the neighbourhood have their weights updated to better represent the input pattern. This promotes clustering of similar results.

The neighbourhood size starts very large (typically the entire network) and is decreased during training in a similar way to the gain term.

The use of a Neighborhood during training.

How does training work?

Kohonen (Self-Organising) Networks

Usually the weights of more neurons than the winner are changed.Once the winner has been identified a neighbourhood of a predefined shape and size is drawn around it.

A real output lattice will typically have hundreds of neurons.

How does training work?

Kohonen (Self-Organising) Networks

A real output lattice will typically have hundreds of neurons. This self organisation gives the network it’s name and leads to the lattice of nodes often being referred to as a self organising matrix.

Over time the lattice of neurons will organise its self into areas of neurons which respond more highly to certain input patterns.

Then all that is left is to present some input patterns to the network with known real world values (such as PCO2) so that the defined areas of the lattice can be found for the individual problem and to give the areas real world values.

Now when an input pattern is presented to the network the winning neuron should be in or close to a defined area so a value can be assigned.

302.1

295.6

254.3

320.8

The PCO2 problem on a

Kohonen network

The PCO2 problem on a Kohonen network

The inputs

The network has been mainly programmed in Fortran and is made up of many console programs each with a particular function to perform on the network. Where ever possible the programs are arbitrary and so such properties as the number of input in an input pattern can be set for each network. The latter networks have had 5 inputs in each pattern, these are:2 for month

, 1 latitude, 1 longitude

and 1 sea surface temperature.

The aim of this project is to program, implement and test a Kohonen network designed to predict PCO2 levels at the surface of the ocean.

These where used as they where thought to have the greatest affect on the PCO2 concentration. To encourage the network to distinguish between months far apart and close in the calendar and to give a greater emphasis on the month input it was included twice in each pattern. However one was an input based on the Sine of the month and one on the Cosine making the month cyclical.

Lattice size

The PCO2 problem on a Kohonen network

Lattice size

The aim of this project is to program, implement and test a Kohonen network designed to predict PCO2 levels at the surface of the ocean.

Typically a square lattice size of 70x70 neurons was used. Of the 4900 neurons in such a lattice only 18.06% of them will be given exact PCO2 values to report when they are found to be the winner. If an “unclassified” neuron wins then a function must be employed to find the most appropriate PCO2 value.

As the network must find the output of every neuron in the lattice to find the winner the size of the lattice greatly affects the time the network takes to train or run.

Lattice size Classed Apr May Jun Jul Aug Sep Oct Nov Average

175x175 (30625) 6.13% 30.58 4.81 4.75 2.78 2.12 2.73 2.16 2.12 6.5080

70x70 (4900) 18.06% 35.68 4.74 6.93 3.33 2.52 3.13 2.24 2.22 7.6000

28x28 (784) 46.68% 31.82 6.16 6.57 3.75 2.62 4.27 2.36 2.31 7.4833

11x11 (121) 86.78% 11.54 8.86 10.70 4.62 2.91 5.00 4.07 2.82 6.3161

4x4 (16) 100.00% 30.49 13.22 14.19 5.58 11.43 10.98 16.52 3.59 13.2505

Using data as model - 11.54 15.84 15.69 10.57 11.53 9.41 11.77 4.31 11.3331

All data is restricted to the north Atlantic drift area.

This is a table of Χ where

Χ2=Σ(model-data)2 Number of Data points

been tested.

To illustrate the properties of lattice sizes several sizes have

PCO2 data in the North Atlantic drift area from a neural network

The PCO2 problem on a Kohonen networkPCO2 data in the North Atlantic drift area from a neural

networkUsing Levitus data (which gives a uniform grid of latitude and longitude with temperature values for any month) a map of PCO2 values can be constructed.This was done with MatLab.

Each point on the maps has an inside colour set by the PCO2 value given by the network and an outside colour set by the data the network was trained on, this gives a visual grade of how well the network predicts the PCO2 levels.

The network which created these maps had a lattice size of 70x70

Created by a

70x70 lattice

Created by a

175x175 lattice

Created by a

28x28 lattice

Created by a

11x11 lattice

The PCO2 problem on a Kohonen network

Created by a

4x4 lattice

Training method

The training program looks for a file which tells it how to change these properties of the network.

Usually the gain is stepped down while the neighbourhood size gets smaller smoothly.

As mentioned before the gain and the neighbourhood size change during the training process.

The PCO2 problem on a Kohonen networkTraining

method

3000000 65.00 65.00 100.00 30.00 3000000 40.00 40.00 30.00 20.00 6000000 20.00 10.00 20.00 16.00

The number of runs. This is the number of times a random input pattern is chosen and presented to the network.

Gain. This is expressed as a percentage. The gain goes from the first of these to the second over the number of runs. Mostly it is fixed.

Neighbourhood diameter. This is expressed as a percent of the lattice width. The size goes from the first value to the second smoothly over the number of runs.

So two more tests where performed this time changing the training method.

The file used in training the last networks was as follows.

Long Train

3000000 80.00 80.00 100.00 75.00 3000000 65.00 65.00 75.00 60.00 3000000 52.00 52.00 60.00 50.00 3000000 40.00 40.00 50.00 33.00 6000000 30.00 20.00 33.00 20.00 6000000 20.00 10.00 20.00 16.00

These are designed to do the same as original training method but with more or less runs i.e. the patterns are presented to the network more or less times.

The PCO2 problem on a Kohonen networkTraining

methodLong Train

3000000 80.00 80.00 100.00 75.00 3000000 65.00 65.00 75.00 60.00 3000000 52.00 52.00 60.00 50.00 3000000 40.00 40.00 50.00 33.00 6000000 30.00 20.00 33.00 20.00 6000000 20.00 10.00 20.00 16.00

Quick Train 1000000 65.00 65.00 100.00 35.00 500000 40.00 40.00 35.00 25.00 1500000 20.00 20.00 25.00 18.00 The networks both had a 70x70 output lattice so a

comparison with the 70x70 size network trained with the original method is valid.Classed Apr May Jun Jul Aug Sep Oct Nov Average

Long 18.00% 34.83 4.80 6.08 3.45 2.37 3.37 2.07 2.26 7.4038

Original 18.06% 35.68 4.74 6.93 3.33 2.52 3.13 2.24 2.22 7.6000

Quick 19.84% 18.38 4.30 4.99 3.13 2.38 3.24 2.14 2.22 5.0963This is a table of Χ where

Χ2=Σ(model-data)2 Number of Data points

Created by a 70x70 latticeusing the

Long method

Created by a 70x70 latticeusing the

Orig-inal method

Created by a 70x70 latticeusing the

Quick method