Gap filling of eddy fluxes with artificial neural networks

Dario Papale, Antje Moffat, Markus Reichstein

What is an ANN

It is a network of many simple processors ("units"), each possibly having a small amount of local memory.

The units are connected by communication channels ("connections") which usually carry numeric (as opposed to symbolic) data, encoded by any of various means. The units operate only on their local data and on the inputs they receive via the connections.

Input dataset

Output

Input layer

Hidden layer

Output layer= unit= connection

Artificial Neural Networks - ANNs

The unit or node

n1

n2

nn

w1

w2

wn

x1

x2

w n x n

w2 x2

w1 x

1

x n

Σ ga y

1

b

bwxbwx...wxwxwxan

1xiinn332211

agbwxgyn

1xii

βaαy

/ae1

1y

aa

aa

ee

eey

Tase0y

Tase1y

ni = previous layer nodesxi = input to the unitwi = connection weighg = transfer functiony = output from the unit

Transfer functions

Multilayer Feed-forward Back propagation ANN (BPN)

BPN are a class of ANN with these characteristics:

1. They are able to model also complex relationships2. The ANN training is supervised3. The nodes are organized in layers4. The information flow is unidirectional from the input

layer to the output layer5. There are not connections between nodes of the same

layer6. The error in the output estimate is retro-propagated

from the output layer to the input layer

Multilayer Feed-forward Back propagation ANN (BPN)

It has been demonstrated that a BPN with one hidden layer and sigmoid transfer functions is able to approximate any kind of continuous function with any degree of accuracy if there are a sufficient number of hidden nodes in the hidden layer (Bishop 1995, Hagan et al. 1996, Rzempoluck 1998)

Mat_dem

BPN: how it works

The ANN is trained using a dataset of examples with input and correct output.For each example the ANN-output is compared with the correct-output and the error is minimized modifying the connection weighs Once the ANN has been trained, the connections’ weighs are fixed and it is possible to use the Network with only the input data

To avoid overfitting, the dataset used in the ANN training is split in 3 subsets:

- Training set- Test set- Validation set Epochs

Error

p

How to back propagate the error?

There are different training algorithms available. The simplest but also most used is based on the gradient descent technique.

Considering the connection between node k and node j :

wkj

Ekj

a

b

1tkj

kj

kjtkj Δw

w

EΔw

n

1p

2pp yt

21

E

Input dataset

Output

Input layer

Hidden layer

Output layer= node or unit= connection

Input and Output database

Validation set

Training set

Test set

Used to assess the connection weightsUsed to evaluate the errors

Trained ANN

Used to evaluate the trained ANN

Artificial Neural Networks training

There are a lot of tools available, often available for free, simple and funny to use

There are also toolboxes or codes that it is possible to integrate in your software (I’m using the Matlab toolbox)

However, like for all the methods, you can use it or use it correctly (or in a better way…), and this make the difference in the performances…

So let’s look more in depth some characteristics of the ANNs…

Generally, the common questions when you start to use ANNs are:

- Which variables should I use as input?

- How many examples do I need?

- How many of these examples do I have to put in the three sets?

- Which ANN architecture should I use?

ANNs are quite robust in the application if input are wrong or noisy, but they are sensible to wrong data during the training process. The dataset used to train the ANN must be with high quality in both, input and output

ANN training is based on an iterative process and you will not obtain two times the same result from the same network trained with the same data

ANNs are very good in interpolation but bad in extrapolation…

Which variables should I use as input?

Site dependent and different between night and day. Ideally Radiation, Temperature, VPD and Soil Water Content for daytime, no Radiation but Soil temperature or Precipitation in the night.

How many examples do I need?

As much as possible…. But the ANNs are not good in extrapolation so it is important to cover the domain UNIFORMLY.

Year split in 28 sections (13 days)Daytime and Nighttime separately, both split in three sections

84 periods per year

Same number of example from each period in the training set

Extracted randomly

Test set = 50% of the training set, Valid. set the rest of data

Which ANN architecture should I use?

The simplest with the best performances. It is important to train different ANNs with different architectures

4, 5, 6, 7, 8, 4+3 hidden nodes

Best performance based on observed vs modeled regression (Validation set) and on MAE and RMSE

Each network trained 10 times starting from different initial weights

If two ANNs have the same performances (+/- 1%) the simplest is selected

But everything starts with the random extraction of the dataset…

Bootstrapping would be very interesting but too slow… More extractions are better than nothing…

Training, Test and Validation sets are randomly extracted 7 times

For each dataset all the process is repeated

At the and there will be 7 ANNs selected as “best”

The worst is removed and the other 6 applied

The final value used to fill each gap is the mean of the 6 results

Input variables selected

Yearly dataset split in 84 periods

Same number of example extracted randomly from each period

Preprocessing (scale between 0 and 1, chose generalization method between early stopping and regulation)

6 different ANNs are trained

Dataset split in daytime and nightime

Each ANN is trained 10 times

6 best networks applied

If gaps in the input variables -> gapfilled with MDS

If gaps impossible to fill -> MDS then MDV…..

Average of the 6 results for each gap

Qflags added + std of the 6 ANN output

2

7

6

10

2 x 7 x 6 x 10 = 840

Network trained!(for one storage!!)

Next steps to improve gapfilling the method

Method: ANN

Training data from three years: 97, 98 and 99.

Applied with input data from 2001