Chapter 13.Chapter 13.General Least-Squares General Least-Squares

and Nonlinear and Nonlinear RegressionRegression

Gab-Byung ChaeGab-Byung Chae

13.1 Polynomial 13.1 Polynomial RegressionRegression

-> poorly represented by a straight line.

As discussed in Chap. 12, one method to accomplish this objective is to use transformations.

Another alternative is to fit polynomials to the data using polynomial regression.

Least Squares Least Squares RegressionRegression

Minimize some measure of the difference between Minimize some measure of the difference between the approximating function and the given data the approximating function and the given data points.points.

In least squares method, the error is measured as :In least squares method, the error is measured as :

The minimum of The minimum of EE occurs when the partial occurs when the partial derivatives of derivatives of EE with respect to each of the with respect to each of the variables are 0.variables are 0.

2

1

))(( ii

n

i

yxfE

,...0,0

b

E

a

E

Linear Least Squares Linear Least Squares RegressionRegression

f(x) f(x) is in a linear form : is in a linear form : f(x)=ax+bf(x)=ax+b the error :the error :

Is minimized when :Is minimized when :

2

1

)( ii

n

i

ybxaE

01)(2

0)(2

ii

iii

ybaxb

E

xybaxa

E

ii

iiii

ybxa

yxxbxa

1

2

xaybxxn

yxyxna

ii

iiii

,

)( 22

Quadratic Least Squares Quadratic Least Squares ApproximationApproximation

f(x) f(x) is in a quadratic form : is in a quadratic form : f(x)=axf(x)=ax22+bx+c+bx+c the error :the error :

Is minimized when :Is minimized when :

22

1

)( iii

n

i

ycbxxaE

0)(2

0)(2

01)(2

22

2

2

iiii

iiii

iii

xycbxaxa

E

xycbxaxb

E

ycbxaxc

E

iiiii

iiiii

iii

yxxaxbxc

yxxaxbxc

yxaxbnc

2432

32

2

Cubic Least Squares Cubic Least Squares ApproximationApproximation

f(x) f(x) is in a cubic form : is in a cubic form : f(x)=axf(x)=ax33+bx+bx22 +cx+d +cx+d the error :the error :

Is minimized when :Is minimized when :

223

1

)( iiii

n

i

ydcxbxxaE

01)(2

0)(2

0)(2

0)(2

23

23

223

323

iiii

iiiii

iiiii

iiiii

ydcxbxaxd

E

xydcxbxaxc

E

xydcxbxaxb

E

xydcxbxaxa

E

This case can be easily extended to an mth-order polynomial.

Determining the coefficients of an mth-order Determining the coefficients of an mth-order polynomial is equivalent to solving a system polynomial is equivalent to solving a system of m+1 simutaneous linear equations.of m+1 simutaneous linear equations.

The standard error is formulated as The standard error is formulated as

(m+1) data-drived coefficients- a(m+1) data-drived coefficients- a0, 0, aa1, … 1, … aam, m, - - were used to compute Swere used to compute Sr r ..

)1(/

mn

SS r

xy

Example 13.1Example 13.1Fit a second-order polynomial to the data Fit a second-order polynomial to the data

in the first two columns of Table 13.1in the first two columns of Table 13.1

Sol> Sol>

8.2488

6.585

6.152

97922555

2255515

55156

2

1

0

a

a

a

>> N = [6 15 55; 15 55 225; 55 225 979];>> N = [6 15 55; 15 55 225; 55 225 979];

>> r = [152.6 585.6 2488.8]>> r = [152.6 585.6 2488.8]

>> a = N\r>> a = N\r

a = a =

2.47862.4786

2.35932.3593

1.8607 1.8607

28607.13593.24786.2 xxy

99925.0

99851.039.2513

74657.339.2513

1175.1)12(6

74657.3

2

/

r

r

S xy

)( 2 yyS it22

1

)( iii

n

ir ycbxxaS

The standard error

The coefficient of determination

The correlation coefficient

t

rt

S

SSr

2

Sum of the squares of the residuals between the data points(yi) and the mean

Sum of the squares of the residuals between the data points(yi) and regression curve

Fit of a second-order polynomial.Figure 13.2

13.2 Multiple Linear 13.2 Multiple Linear RegressionRegression

An extension of linear regression : y is a An extension of linear regression : y is a linear function of two or more linear function of two or more independent variables.independent variables.

For this two-dimensional case, the For this two-dimensional case, the regression line becomes a plane(Fig. regression line becomes a plane(Fig. 13.3).13.3).

The sum of the squares of the residuals:The sum of the squares of the residuals:

exaxaay 22110

2,22,110

1

)( iii

n

ir xaxaayS

Graphical depiction of multiple linear regression where y is a linear function of x1 and x2.

Figure 13.3

0)(2

0)(2

0)(2

,22,110,22

,22,110,11

,22,1100

iiiir

iiiir

iiir

xaxaayxa

S

xaxaayxa

S

xaxaaya

S

ii

ii

i

iiii

iiii

ii

yx

yx

y

a

a

a

xxxx

xxxx

xxn

,2

,1

2

1

0

2,2,2,1,2

,2,12,1,1

,2,1

Example 13.2 Multiple Example 13.2 Multiple Linear RegressionLinear Regression

Use multiple Use multiple linear regression linear regression to fit this data. to fit this data.

ii

ii

i

iiii

iiii

ii

yx

yx

y

a

a

a

xxxx

xxxx

xxn

,2

,1

2

1

0

2,2,2,1,2

,2,12,1,1

,2,1

3,4,5

100

5.243

54

544814

4825.765.16

145.166

210

2

1

0

aaa

a

a

a

Which gives us

Extension to m Extension to m dimensionsdimensions

exaxaxaay mm 22110

)1(/

mn

SS r

xy

Power equations of the form Power equations of the form

mm

am

aa

xaxaxaay

xxxay m

logloglogloglog 22110

21021

Standard error

13.3 General Linear Least 13.3 General Linear Least SquaresSquares

mm

mm

m

mm

xzxzxzz

xzxzxzz

functionsbasismarezzwhere

ezazazazay

,,,,1

,,,,1

.1,.....,

)7.13(

2210

22110

0

221100

When

We have simple or multiple linear regression.

When

We have polynomial regression.

The functions can be highly nonlinear.The functions can be highly nonlinear. For example:For example:

OrOr

)1(

)sin()cos(

10

210

xaeay

xaxaay

Equation (13.7) can be expressed in matrix Equation (13.7) can be expressed in matrix notation as notation as

where m is the number of variables in the where m is the number of variables in the model and n is the number of data points. model and n is the number of data points.

mnnn

m

m

zzz

zzz

zzz

Z

functionsbasistheof

valuescalculatedtheofmatrixaisZwhere

eaZy

10

21202

11101

Because n>m, mostly Z is not a square Because n>m, mostly Z is not a square matrix.matrix.

n

i

m

jjijir

mT

mT

nT

zayS

residualtheofsquarestheofsumThe

eeee

evectorcolumnThe

yaaa

avectorcolumnThe

yyyy

yvectorcolumnThe

1

2

0

21

21

21

)(

:

:

:

:

By taking its partial derivative with respect By taking its partial derivative with respect to each of the coefficients and setting the to each of the coefficients and setting the resulting equation equal to zero.resulting equation equal to zero.

aZy

asformvectorinwrittenbecanyy

fitsquaresleasttheofpredictiontheywhere

yy

yyr

S

S

S

SSr

yZaZZ

i

i

i

t

r

t

rt

TT

.

)(

)(1

1

2

22

2

Example 13.3 Example 13.3 Polynomial Regression with Polynomial Regression with

MATLABMATLABRepeat example 13.1 Repeat example 13.1 >> x=[0 1 2 3 4 5]';>> x=[0 1 2 3 4 5]';>> y=[2.1 7.7 13.6 27.2 40.9 61.1]';>> y=[2.1 7.7 13.6 27.2 40.9 61.1]';>> Z=[ones(size(x)) x x.^2]>> Z=[ones(size(x)) x x.^2]

Z =Z =

1 0 01 0 0 1 1 11 1 1 1 2 41 2 4 1 3 91 3 9 1 4 161 4 16 1 5 251 5 25

>> Z'*Z>> Z'*Z

ans =ans =

6 15 556 15 55 15 55 22515 55 225 55 225 97955 225 979

>> a=(Z'*Z)\(Z'*y)>> a=(Z'*Z)\(Z'*y)

aa = =

2.47862.4786 2.35932.3593 1.86071.8607

>> Sr = sum((y-Z*a).^2)>> Sr = sum((y-Z*a).^2)

Sr =Sr =

3.74663.7466

>> r2=1-Sr/sum((y-mean(y)).^2)>> r2=1-Sr/sum((y-mean(y)).^2)

r2 =r2 =

0.99850.9985

>> Syx= sqrt(Sr/(length(x)-length(a)))>> Syx= sqrt(Sr/(length(x)-length(a)))

Syx =Syx =

1.11751.1175

13.4 QR factorization and 13.4 QR factorization and the backslash operator the backslash operator

QR factorization and singular value QR factorization and singular value decomposition : beyond the scope of this decomposition : beyond the scope of this book but we can use it in MATLAB which is book but we can use it in MATLAB which is implemented as polyfit and backslashimplemented as polyfit and backslash

{y} = [Z]{a} : general model Eq.(13.8) {y} = [Z]{a} : general model Eq.(13.8) >> x = [ 0 1 2 3 4 5]' ;>> x = [ 0 1 2 3 4 5]' ;>> y=[2.1 7.7 13.6 27.2 40.9 61.1]‘;>> y=[2.1 7.7 13.6 27.2 40.9 61.1]‘;>> z=[ones(size(x)) x x.^2];>> z=[ones(size(x)) x x.^2];>> a=polyfit(x,y,2)>> a=polyfit(x,y,2)>> a =z\y>> a =z\y

13.5 Nonlinear 13.5 Nonlinear regressionregression

Ex : Ex :

The sum of the square The sum of the square

Find aFind a0 0 and aand a1 1 that minimize the function f that minimize the function f Matlab’s fminsearch function can be used for Matlab’s fminsearch function can be used for

this purpose.this purpose.

[x, fval] =fminsearch(fun, x0, options, p1, p2,[x, fval] =fminsearch(fun, x0, options, p1, p2,…)…)

eeay xa )1( 10

20

110 )]1([),( 1 ixa

n

ii eayaaf

Example 13.4 Example 13.4 Nonlinear Regression with Nonlinear Regression with

MATLABMATLAB Recall example 12.4 with the table 12.1: Recall example 12.4 with the table 12.1:

we have we have

This time use nonlinear regression. This time use nonlinear regression. Employ initial guesses of 1 for the Employ initial guesses of 1 for the coefficient.coefficient.

9842.12741.0 vF

SolSol M-file M-file function f = fSSR(a, xm, ym)function f = fSSR(a, xm, ym)yp = a(1)*xm.^a(2);yp = a(1)*xm.^a(2);f =sum((ym-yp).^2);f =sum((ym-yp).^2);

>> x=[10 20 30 40 50 60 70 80 ];>> x=[10 20 30 40 50 60 70 80 ];>> y = [25 70 380 550 610 1220 830 1450];>> y = [25 70 380 550 610 1220 830 1450];>> fminsearch(@fSSR, [1,1],[], x,y)>> fminsearch(@fSSR, [1,1],[], x,y)

ans =ans =

2.53839730236869 1.435853174785852.53839730236869 1.43585317478585 4359.15384.2 vF

Comparison of transformed and untransformed model fits for force versus velocity data from Table 12.1.

Figure 13.4