FUNCTION FUNCTION APPROXIMATIONAPPROXIMATIONSarbjeet SinghNITTTR chandigarh

CONTENTCONTENT

Learning ParadigmsLearning Paradigms

• Training data: A sample from the data source with the correct Training data: A sample from the data source with the correct classification /regression solution already assigned.classification /regression solution already assigned.

• Two Types of Learning-Two Types of Learning-– SUPERVISEDSUPERVISED– UNSUPERVISEDUNSUPERVISED

• Supervised learning Supervised learning = Learning based on training data.= Learning based on training data.• Two steps:Two steps:• 1. Training step: Learn classifier /regressor from training data.1. Training step: Learn classifier /regressor from training data.• 2. Prediction step: Assign class labels/functional values to test data.2. Prediction step: Assign class labels/functional values to test data.• Example:- Perceptron, LDA, SVMs, linear/ridge/kernel ridge regression Example:- Perceptron, LDA, SVMs, linear/ridge/kernel ridge regression

are all supervised methods.are all supervised methods.

Learning Paradigms Contd..Learning Paradigms Contd..

• Unsupervised learningUnsupervised learning: Learning without training data.

• Examples:

• Data clustering. (Some authors do not distinguish between clustering and unsupervised learning.)

• Dimension reduction techniques.• Data clustering: Divide input data into groups of similar points.• → Roughly the unsupervised counterpart to classification.• Note the difference:

• Supervised case: Fit model to each class of training points, then use models to classify test points.

• Clustering: Simultaneous inference of group structure and model.

Learning TasksLearning Tasks

• There are Six learning There are Six learning – Pattern AssociationPattern Association

– Pattern RecognitionPattern Recognition

– Function ApproximationFunction Approximation

– ControllingControlling

– FilteringFiltering

– Beam formingBeam forming

Function ApproximationConsider a non linear input – output mapping

described by the functional relationship

where

Vector x is input.

Vector d is output.

The vector valued function f(.) is assumed to be unknown.

xfd

Function Approximation

To get the knowledge about the function f(.), some set of examples are taken,

A neural network is designed to approximate the unknown function in Euclidean sense over all inputs, given by the equation

Niii dx 1,

xfxF

WhereΕ is a small positive number.Size N of training sample is large enough

and network is equipped with an adequate number of free parameters,

Thus approximation error ε can be reduced.

The approximation problem discussed here would be example of supervised learning.

Function Approximation

UNKNOWUNKNOWN SYSTEMN SYSTEM

ΣΣix

id

iy

ie

Input Input VectorVector

SYSTEM SYSTEM IDENTIFICATIONIDENTIFICATIONBLOCK DIAGRAMBLOCK DIAGRAM

NEURAL NEURAL NETWORK NETWORK

MODELMODEL

System Identification

• Let input-output relation of unknown memoryless MIMO system i.e. time invariant system is

• Set of examples are used to train a neural network as a model of the system.

WhereVector denote the actual output of the

neural network.

xfd

Niii dx 1,

iy

System Identification

denotes the input vector. denotes the desired response. denotes the error signal i.e. the difference between and .

This error is used to adjust the free parameters of the network to minimize the squared difference between the outputsof the unknown system and neural network in a statistical sense and computed over entire training samples.

ix

id

ie

iyid

INVERSE INVERSE MODELINGMODELING

UNKNOUNKNOWN WN

SYSTEMSYSTEMf(.)f(.)

ΣΣInput Input VectorVector

ixixid

iy

ie

SysteSystem m OutputOutput

Model Model OutputOutput

ErrorError

BLOCK DIAGRAMBLOCK DIAGRAM

INVERSINVERSE E

MODELMODEL

Inverse Modeling

• In this we construct an inverse model that produces the vector x in response to the vector d.

• This can be given by the eqution :

Where

denote inverse of .Again with the use of stated examples neural network approximation of is constructed.

dfx 1

1f f

1f

• Here is used as input and as desired response.

• is the error signal between and produced in response to .

• This error is used to adjust the free parameters of the network to minimize the squared difference between the outputs of the unknown system and neural network in a statistical sense and computed over entire training samples.

Inverse Modeling

id ix

ie ix iyid