COM2023 Mathematics Methods for Computing...

COM2023

Mathematics Methods for

Computing II

Lecture 17& 18

Gianne DerksDepartment of Mathematics (36AA04)

http://www.maths.surrey.ac.uk/Modules/COM2023

Autumn 2010

Use channel 04 on your EVS handset

OverviewLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● Regression modelling

◆ Linear regression

● Recap

● Example

● Predictions

● R and linear regression

◆ Quadratic regression

◆ Other non-linear regression

◆ Multiple regression

Linear regression model (§6.1)

Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Linear regression modelling: recapLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Given a data set {(x1, y1), (x2, y2), . . . , (xn, yn)}, which looks linear. Todetermine the linear regression line:

● Determine the means x = 1n

n∑

i=1

xi and y = 1n

n∑

i=1

yi;

● Determine the quantities

Sxx =

n∑

i=1

(xi − x)2 and Sxy =

n∑

i=1

(xi − x)(yi − y);

● The slope is b =Sxy

Sxx

;

● The intercept is a = y − bx;

● The least squares regression line of y on x is y = a+ bx;

● The residuals are εi = yi − (a+ bxi). Check that they are normallydistributed.

Forbes’ data: linear regression plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

22 24 26 28 30

195

200

205

210

atmospheric pressure (inch Hg)

boili

ng p

oint

(F

)

AAAAK

boiling point = 155.3 + 1.9 × atmospheric pressure

Forbes’ data: normal residual checkLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

To check the assumption that the noise in Forbes’ data is normallydistributed, we plot the quantiles of the residuals against the theoreticalquantiles of a normal distribution (QQ plot).

−2 −1 0 1 2

−1.

0−

0.5

0.0

0.5

Theoretical quantiles

For

bes’

dat

a qu

antil

es

Example


Example: Properties of Synthetic RubberLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The following table gives the hardness and the abrasion loss of twelve differentspecimens of synthetic rubber.

Specimen Hardness x Abrasion loss y Product x× y

1 45 372 167402 55 206 113303 61 175 106754 68 113 76845 59 249 146916 74 145 107307 81 55 44558 86 45 44559 83 97 8051

10 71 219 1554911 53 221 1171312 88 64 5632

Properties of Synthetic Rubber: plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

50 60 70 80

5010

015

020

025

030

035

0

Hardness x

Abr

asio

n lo

ss y

Synthetic Rubber: linear regression lineLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The totals of the data are as follows (n=12):

Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get

Sxx = Txx −1nT 2x = and Sxy = Txy −

1nTxTy =

andx = and y =

Thus b =Sxy

Sxx= , a = y − bx = and the line is



Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get

Sxx = Txx −1nT 2x = 2210.667 and Sxy = Txy −

1nTxTy =

andx = and y =

Thus b =Sxy




Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = and y =

Thus b =Sxy




Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = 68.66667 and y =

Thus b =Sxy




Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy




Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = and the line is



Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = 583.845 and the line is



Tx =

n∑

i=1

xi = 824, Ty =

n∑

i=1

yi = 1961 and Txy =

n∑

i=1

xiyi = 121120.

And,

Txx =n∑

i=1

x2i = 58792 and Tyy =n∑

i=1

y2i = 422597.

Using this, we get


1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = 583.845 and the line is

Abrasion loss = −6.123× Hardness+ 583.845.

Synthetic Rubber: linear regression plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

50 60 70 80

5010

015

020

025

030

035

0

Hardness x

Abr

asio

n lo

ss y

��

abrasion loss = 583.845− 6.123 × hardness

Synthetic Rubber: normal residual checkLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5

−40

−20

020

4060

Normal Q−Q Plot

Theoretical Quantiles

Sam

ple

Qua

ntile

s

Predictions (§6.1, page 44-45)


Prediction for regression modellingLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● In order to predict a future y-value, y say, at a given x-value, x∗ say, usethe fitted model and take

y = a+ bx∗.



y = a+ bx∗.

● The fitted model can usually safely be used for interpolation, i.e., x∗ isan x-value inside the minimum and maximum sample x-values.



y = a+ bx∗.


● Be very cautious about using the fitted straight line model forextrapolation. There is no guarantee that the model will be valid or evenpossible for such an x-value outside the sample x-values.



y = a+ bx∗.


● Be very cautious about using the fitted straight line model forextrapolation. There is no guarantee that the model will be valid or evenpossible for such an x-value outside the sample x-values.

● The prediction is likely to be imprecise if

◆ the variation of the residuals is large;

◆ If the number of data points n is small.

Synthetic Rubber example, predictionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Item Hardness x Abrasion loss y

1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64

Linear regression line:

y = 584− 6× x

Predictions:

● What is the expected abrasion lossat hardness 50?



1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64


y = 584− 6× x

Predictions:

● What is the expected abrasion lossat hardness 50?Take x = 50 in regression line,then y = 284, thus the expectedabrasion loss is 284.



1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64


y = 584− 6× x

Predictions:


● What is the expected abrabsionloss at hardness 100?



1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64


y = 584− 6× x

Predictions:


● What is the expected abrabsionloss at hardness 100?We are not sure if the model isvalid for hardness less than 45 orlarger than 88. The answer fromthe regression line is y = 584 −600 = −16, hence not realistic.

Regression and Matlab (§6.1, p.

41-43)


“Women” data set in Matlab


Weight and height of a sample group of women:

58 60 62 64 66 68 70 72

120

130

140

150

160

height

wei

ght

Analysis of data sets in Matlab


● The weight vector is such that weight[i] lbs is the average weight ofwomen who are height[i] inches tall.

55 60 65 70 75100

120

140

160

180W

eigh

t (lb

)

Height (in)




● Check the correlation and rank correlation:

◆ corr(height,weight)=0.9955

◆ corr(height,weight,’Type’,’Spearman’)=1.







● Next we fit a linear model y = a+ bx with x the height, y the weight.

◆ myfit = polyfit(height,weight,1)

◆ This gives myfit = 3.4500 -87.5167,hence the coefficients a and b with a = −87.52 and b = 3.45.

◆ Thus the linear regression line is “weight = 3.45 × height - 87.52”







● Next we fit a linear model y = a+ bx with x the height, y the weight.

◆ myfit = polyfit(height,weight,1)

◆ This gives myfit = 3.4500 -87.5167,hence the coefficients a and b with a = −87.52 and b = 3.45.

◆ Thus the linear regression line is “weight = 3.45 × height - 87.52”

● To get analyse the residuals, use

◆ weight approx = polyval(myfit,height);the model y-values.

◆ residuals = weight-weight approx; gives the residuals.

◆ mean(residuals) gives 2.8422e-14 and var(residuals)=2.1595

Plot for the “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

scatter(height,weight); hold on; plot(height,weight approx)

title(’American women aged 30-39’); xlabel(’Height (in)’);

ylabel(’Weight (lb)’);

58 60 62 64 66 68 70 72

120

130

140

150

160

American women aged 30−39

Height (in)

Wei

ght (

lb)

Plot for the residuals in “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

figure; scatter(height,residuals); hold on;

plot(height,zeros(length(height))); title(’Residual Plot’);

xlabel(’Height (in)’); ylabel(’Residual (lb)’);

58 60 62 64 66 68 70 72

−1

01

23

Residual Plot

Height (in)

Res

idua

l (lb

)

Normal check of residuals “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

figure; qqplot(residuals)

−1 0 1

−1

01

23

normal Q−Q plot for women data


Qua

ntile

s w

omen

dat

a

Quadratic regression (§6.2)


Non-linear regression and women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Looking at the plots obtained from fitting a straight line to the women data,it appears that a non-linear relationship may be more appropriate.

Assume that the appropriate relationship between the variables “weight” and“height” is:

weight = α+ β × height + γ × (height)2 + noise

We will try to fit a quadratic of the form:

weight = a+ b× height + c× (height)2

to the data, where a, b and c have to be found by using the observations.This will be done by using procedures in Matlab.

Using Matlab to fit a quadraticLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Using Matlab, we fit a quadratic of the form


to the women data.




to the women data.

Matlab can fit any order polynomial with polyfit, so now we just use theoption “2” for a second order polynomial:

myfit2 = polyfit(height,weight,2)

This gives c = 0.0831, b = −7.3483 and a = 261.8782.




to the women data.

Matlab can fit any order polynomial with polyfit, so now we just use theoption “2” for a second order polynomial:

myfit2 = polyfit(height,weight,2)

This gives c = 0.0831, b = −7.3483 and a = 261.8782.

Thus the approximate quadratic is

weight = 261.878184− 7.348319× height+ 0.083064× (height)2

Quadratic & linear fit for women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

58 60 62 64 66 68 70 72

120

130

140

150

160

Height (in)

Wei

ght (

lb)

Fitted quadratic modelFitted linear model

Residuals of quadratic fit for women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

58 60 62 64 66 68 70 72

−0.

4−

0.2

0.0

0.2

0.4

0.6

Residual plot

height

resi

dual

s

−1 0 1

−0.

4−

0.2

0.0

0.2

0.4

0.6

Normal Q−Q Plot


quan

tiles

of r

esid

uals

of q

uadr

atic

fit

Another quadratic fitting exampleLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

We “polute” the quadratic y = (x−6)2

4 = 9− 3x+ x2

4 by adding noise and getthe following dataset:

x 1 2 3 4 5 6 7 8 9 10 11 12 13 14

y 7.1 4.2 0.3 3.0 -1.0 -1.5 0.3 -1.0 1.5 1.4 6.3 8.5 13.5 15.4

y 6.3 4.0 2.3 1.0 0.3 0.0 0.3 1.0 2.3 4.0 6.3 9.0 12.3 16.0

Here y is the “poluted” data and y are the “pure” values.

This data is obtained in Matlab by using the following commands:

x = 1:14;

y = 0.25*(x-6).^2 + normrnd(0,1,1,14);

yhat = 0.25*(x-6).^2

Fitting the quadratic dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next we fit a quadratic by using:

myfit = polyfit(x,y,2)

approx y = polyval(myfit,x)

This gives the fitted curve y = 10.3− 3.6x+ 0.29x2 and a set of values y(x),called approx y.

Fitting the quadratic dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next we fit a quadratic by using:

myfit = polyfit(x,y,2)

approx y = polyval(myfit,x)

This gives the fitted curve y = 10.3− 3.6x+ 0.29x2 and a set of values y(x),called approx y.

To plot this, we use

scatter(x,y); hold on; plot(x,approx y);

plot(x,yhat,’--’);

title(’Quadratic regression example’);legend(’Data points’,’Fitted quadratic’,’Original

quadratic’,’Location’, ’West’)

Fitted & original plot of quadraticLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

2 4 6 8 10 12 14

05

1015

Quadratic regression example

x

y

Fitted quadraticOriginal quadratic

Residuals in quadratic regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Finally we analyse the residuals:residuals = y - approx y;

subplot(1,2,1); scatter(x,residuals); plot(x,zeros(14));

title(’Residual plot’);

subplot(1,2,2);

qqplot(residuals)

2 4 6 8 10 12 14

−1

01

2

Residual plot

x

Res

idua

ls

−1 0 1

−1

01

2

Normal Q−Q Plot


Sam

ple

Qua

ntile

s

Regression and Matlab


Given a data set {(x1, y1), (x2, y2), . . . , (xn, yn)}. Using Matlab:

● The coefficients of a polynomial approximation of order n is obtained by

myfit = polyfit(x,y,n)

● The approximate y values are obtained by

approx = polyval(myfit,x)

Multiple regression (§6.2)


Data with three componentsLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The trees data set gives the relation between the height, volume and girth(diameter) of 31 cherry trees.

4.1 4.2 4.3 4.4 4.5 4.62

2.5

3

3.5

4

4.5

5

log(height)

log(

volu

me)

2 2.5 3 3.52

2.5

3

3.5

4

4.5

5

log(girth)

log(

volu

me)

2 2.5 3 3.54

4.2

4.4

4.6

4.8

log(girth)

log(

heig

ht)

corr(Volume’,Height’) = 0.5982

corr(Girth’,Volume’) = 0.9671

corr(Girth’,Height’) = 0.5193

corr(log(Volume)’,log(Height)’)

= 0.6486

corr(log(Girth)’,log(Volume)’)

= 0.9767

corr(log(Girth)’,log(Height)’)

= 0.5302

Linear regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

First linear regression on the data concerning the girth and volume, usinglog(girth) and log(volume), ignoring any information about the height.

2.2 2.4 2.6 2.8 3.0

2.5

3.0

3.5

4.0

log.g

log.

v

Multiple regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next multiple regression, writing log(volume) as a linear combination oflog(girth) and log(height). Shown are the observed volume against the fittedvolume

10 20 30 40 50 60 70

1020

3040

5060

7080

observed volume

fitte

d vo

lum

e

Comparing residuals (1)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

A plot of the residuals of the linear (left) and multiple regression (right)models

2.5 3.0 3.5 4.0

−0.

2−

0.1

0.0

0.1

0.2

Residuals Linear Regression

log(volume)

resi

dual

s

2.5 3.0 3.5 4.0

−0.

15−

0.10

−0.

050.

000.

050.

10

Residuals Multiple Regression

log(volume)

resi

dual

s

The residuals for the linear regression are mainly between −0.2 and +0.2 andthe residuals for the multiple regression are mainly between −0.1 and +0.1.

Comparing residuals (2)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Finally we compare the residuals of the linear regression (left) and multipleregression (right) in normal qqplots:

−2 −1 0 1 2

−0.

2−

0.1

0.0

0.1

0.2

Normal Q−Q Plot


Sam

ple

Qua

ntile

s

−2 −1 0 1 2

−0.

15−

0.10

−0.

050.

000.

050.

10

Normal QQ plot multiple regression


Sam

ple

Qua

ntile

s

Matlab commands used (1)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● Overview plots:scatter(log(Girth),log(Height));

scatter(log(Height),log(Volume));

scatter(log(Girth),log(Volume));

4.1 4.2 4.3 4.4 4.5 4.62

2.5

3

3.5

4

4.5

5

log(height)

log(

volu

me)

2 2.5 3 3.52

2.5

3

3.5

4

4.5

5

log(girth)

log(

volu

me)

2 2.5 3 3.54

4.2

4.4

4.6

4.8

log(girth)

log(

heig

ht)


● Overview plots:scatter(log(Girth),log(Height));

scatter(log(Height),log(Volume));

scatter(log(Girth),log(Volume));

● Linear regression:myfit lin = polyfit(log(Girth),log(Volume),1);

approx lin = polyval(myfit lin,Girth)

scatter(log(Girth),log(Volume)); plot(Girth,approx lin);

2.2 2.4 2.6 2.8 3.0

2.5

3.0

3.5

4.0

log.g

log.

v


● Multiple regression:X = [ones(length(Girth),1),Girth,Height];

mul = regress(Volume,X);

approx mul = mul(1) + mul(2)*Girth + mul(3)*Height;

scatter(volume,exp(approx mul))

plot([Volume(1),Volume(1)],[Volume(end),Volume(end)])

10 20 30 40 50 60 70

1020

3040

5060

7080

observed volume

fitte

d vo

lum

e


● Residuals:res lin = log(Volume)-log(approx lin);

res mul = log(Volume)-log(approx mul);

subplot(1,2,1); plot(log(Volume),res lin);

subplot(1,2,2); plot(log(Volume),res mul);

2.5 3.0 3.5 4.0

−0.

2−

0.1

0.0

0.1

0.2

Residuals Linear Regression

log(volume)

resi

dual

s

2.5 3.0 3.5 4.0

−0.

15−

0.10

−0.

050.

000.

050.

10

Residuals Multiple Regression

log(volume)

resi

dual

s


● Residuals:res lin = log(Volume)-log(approx lin);

res mul = log(Volume)-log(approx mul);

subplot(1,2,1); plot(log(Volume),res lin);

subplot(1,2,2); plot(log(Volume),res mul);

● Quantum-quantum plots:subplot(1,2,1); qqplot(res lin);

subplot(1,2,1); qqplot(res mul);

−2 −1 0 1 2

−0.

2−

0.1

0.0

0.1

0.2

Normal Q−Q Plot


Sam

ple

Qua

ntile

s

−2 −1 0 1 2

−0.

15−

0.10

−0.

050.

000.

050.

10

Normal QQ plot multiple regression


Sam

ple

Qua

ntile

s

Other nonlinear regression


Nonlinear regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Some nonlinear regression can be done by introducing new variables, whichmake the relation linear (and hence we can use the simple lm command in Ras before). For example:

● The exponential regression model: Y = exp(α+ βX).Note that log Y = α+ βX, hence with U = log Y , we have a linearrelation between U and X. This is U = α+ βX;




● The logarithmic regression model: Y = log(α+ βX).Note that expY = α+ βX, hence with V = expY , we have a linearrelation between V and X. This is V = α+ βX;





● The power regression model: Y = αXβ .Note that log Y = logα+ β logX, hence with W = log Y andZ = logX, we have a linear relation between W and Z. This isW = logα+ βZ.





● The power regression model: Y = αXβ .Note that log Y = logα+ β logX, hence with W = log Y andZ = logX, we have a linear relation between W and Z. This isW = logα+ βZ.

However, most nonlinear regression models are more complicated.

Date post:	06-Jun-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

COM2023 Mathematics Methods for Computing...

Documents