COM2023 Mathematics Methods for Computing...

transcript

COM2023

Mathematics Methods for

Computing II

Lecture 17& 18

Gianne DerksDepartment of Mathematics (36AA04)

http://www.maths.surrey.ac.uk/Modules/COM2023

Autumn 2010

Use channel 04 on your EVS handset

OverviewLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● Regression modelling

◆ Linear regression

● Recap

● Example

● Predictions

● R and linear regression

◆ Quadratic regression

◆ Other non-linear regression

◆ Multiple regression

Linear regression model (§6.1)

Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Linear regression modelling: recapLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Given a data set {(x1, y1), (x2, y2), . . . , (xn, yn)}, which looks linear. Todetermine the linear regression line:

● Determine the means x = 1n

xi and y = 1n

● Determine the quantities

(xi − x)2 and Sxy =

(xi − x)(yi − y);

● The slope is b =Sxy

● The intercept is a = y − bx;

● The least squares regression line of y on x is y = a+ bx;

● The residuals are εi = yi − (a+ bxi). Check that they are normallydistributed.

Forbes’ data: linear regression plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

22 24 26 28 30

atmospheric pressure (inch Hg)

boiling point = 155.3 + 1.9 × atmospheric pressure

Forbes’ data: normal residual checkLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

To check the assumption that the noise in Forbes’ data is normallydistributed, we plot the quantiles of the residuals against the theoreticalquantiles of a normal distribution (QQ plot).

−2 −1 0 1 2

Theoretical quantiles

bes’

Example

Example: Properties of Synthetic RubberLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The following table gives the hardness and the abrasion loss of twelve differentspecimens of synthetic rubber.

Specimen Hardness x Abrasion loss y Product x× y

1 45 372 167402 55 206 113303 61 175 106754 68 113 76845 59 249 146916 74 145 107307 81 55 44558 86 45 44559 83 97 8051

10 71 219 1554911 53 221 1171312 88 64 5632

Properties of Synthetic Rubber: plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

50 60 70 80

Hardness x

Synthetic Rubber: linear regression lineLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The totals of the data are as follows (n=12):

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

Sxx = Txx −1nT 2x = and Sxy = Txy −

1nTxTy =

andx = and y =

Thus b =Sxy

Sxx= , a = y − bx = and the line is

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

Sxx = Txx −1nT 2x = 2210.667 and Sxy = Txy −

1nTxTy =

andx = and y =

Thus b =Sxy

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = and y =

Thus b =Sxy

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = 68.66667 and y =

Thus b =Sxy

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = and the line is

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = 583.845 and the line is

xi = 824, Ty =

yi = 1961 and Txy =

xiyi = 121120.

Txx =n∑

x2i = 58792 and Tyy =n∑

y2i = 422597.

Using this, we get

1nTxTy = −13535.33.

andx = 68.66667 and y = 163.4167.

Thus b =Sxy

Sxx= −6.123, a = y − bx = 583.845 and the line is

Abrasion loss = −6.123× Hardness+ 583.845.

Synthetic Rubber: linear regression plotLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

50 60 70 80

Hardness x

��

abrasion loss = 583.845− 6.123 × hardness

Synthetic Rubber: normal residual checkLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5

Normal Q−Q Plot

Theoretical Quantiles

Predictions (§6.1, page 44-45)

Prediction for regression modellingLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● In order to predict a future y-value, y say, at a given x-value, x∗ say, usethe fitted model and take

y = a+ bx∗.

● The fitted model can usually safely be used for interpolation, i.e., x∗ isan x-value inside the minimum and maximum sample x-values.

y = a+ bx∗.

● Be very cautious about using the fitted straight line model forextrapolation. There is no guarantee that the model will be valid or evenpossible for such an x-value outside the sample x-values.

y = a+ bx∗.

● Be very cautious about using the fitted straight line model forextrapolation. There is no guarantee that the model will be valid or evenpossible for such an x-value outside the sample x-values.

● The prediction is likely to be imprecise if

◆ the variation of the residuals is large;

◆ If the number of data points n is small.

Synthetic Rubber example, predictionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Item Hardness x Abrasion loss y

1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64

Linear regression line:

y = 584− 6× x

Predictions:

● What is the expected abrasion lossat hardness 50?

1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64

y = 584− 6× x

Predictions:

● What is the expected abrasion lossat hardness 50?Take x = 50 in regression line,then y = 284, thus the expectedabrasion loss is 284.

1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64

y = 584− 6× x

Predictions:

● What is the expected abrabsionloss at hardness 100?

1 45 3722 55 2063 61 1754 68 1135 59 2496 74 1457 81 558 86 459 83 9710 71 21911 53 22112 88 64

y = 584− 6× x

Predictions:

● What is the expected abrabsionloss at hardness 100?We are not sure if the model isvalid for hardness less than 45 orlarger than 88. The answer fromthe regression line is y = 584 −600 = −16, hence not realistic.

Regression and Matlab (§6.1, p.

41-43)

“Women” data set in Matlab

Weight and height of a sample group of women:

58 60 62 64 66 68 70 72

height

Analysis of data sets in Matlab

● The weight vector is such that weight[i] lbs is the average weight ofwomen who are height[i] inches tall.

55 60 65 70 75100

Height (in)

● Check the correlation and rank correlation:

◆ corr(height,weight)=0.9955

◆ corr(height,weight,’Type’,’Spearman’)=1.

● Next we fit a linear model y = a+ bx with x the height, y the weight.

◆ myfit = polyfit(height,weight,1)

◆ This gives myfit = 3.4500 -87.5167,hence the coefficients a and b with a = −87.52 and b = 3.45.

◆ Thus the linear regression line is “weight = 3.45 × height - 87.52”

● Next we fit a linear model y = a+ bx with x the height, y the weight.

◆ myfit = polyfit(height,weight,1)

◆ This gives myfit = 3.4500 -87.5167,hence the coefficients a and b with a = −87.52 and b = 3.45.

◆ Thus the linear regression line is “weight = 3.45 × height - 87.52”

● To get analyse the residuals, use

◆ weight approx = polyval(myfit,height);the model y-values.

◆ residuals = weight-weight approx; gives the residuals.

◆ mean(residuals) gives 2.8422e-14 and var(residuals)=2.1595

Plot for the “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

scatter(height,weight); hold on; plot(height,weight approx)

title(’American women aged 30-39’); xlabel(’Height (in)’);

ylabel(’Weight (lb)’);

58 60 62 64 66 68 70 72

American women aged 30−39

Height (in)

Plot for the residuals in “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

figure; scatter(height,residuals); hold on;

plot(height,zeros(length(height))); title(’Residual Plot’);

xlabel(’Height (in)’); ylabel(’Residual (lb)’);

58 60 62 64 66 68 70 72

Residual Plot

Height (in)

Normal check of residuals “women” dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

figure; qqplot(residuals)

−1 0 1

normal Q−Q plot for women data

Quadratic regression (§6.2)

Non-linear regression and women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Looking at the plots obtained from fitting a straight line to the women data,it appears that a non-linear relationship may be more appropriate.

Assume that the appropriate relationship between the variables “weight” and“height” is:

weight = α+ β × height + γ × (height)2 + noise

We will try to fit a quadratic of the form:

weight = a+ b× height + c× (height)2

to the data, where a, b and c have to be found by using the observations.This will be done by using procedures in Matlab.

Using Matlab to fit a quadraticLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Using Matlab, we fit a quadratic of the form

to the women data.

Matlab can fit any order polynomial with polyfit, so now we just use theoption “2” for a second order polynomial:

myfit2 = polyfit(height,weight,2)

This gives c = 0.0831, b = −7.3483 and a = 261.8782.

to the women data.

Matlab can fit any order polynomial with polyfit, so now we just use theoption “2” for a second order polynomial:

myfit2 = polyfit(height,weight,2)

This gives c = 0.0831, b = −7.3483 and a = 261.8782.

Thus the approximate quadratic is

weight = 261.878184− 7.348319× height+ 0.083064× (height)2

Quadratic & linear fit for women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

58 60 62 64 66 68 70 72

Height (in)

Fitted quadratic modelFitted linear model

Residuals of quadratic fit for women dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

58 60 62 64 66 68 70 72

Residual plot

height

−1 0 1

Normal Q−Q Plot

Another quadratic fitting exampleLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

We “polute” the quadratic y = (x−6)2

4 = 9− 3x+ x2

4 by adding noise and getthe following dataset:

x 1 2 3 4 5 6 7 8 9 10 11 12 13 14

y 7.1 4.2 0.3 3.0 -1.0 -1.5 0.3 -1.0 1.5 1.4 6.3 8.5 13.5 15.4

y 6.3 4.0 2.3 1.0 0.3 0.0 0.3 1.0 2.3 4.0 6.3 9.0 12.3 16.0

Here y is the “poluted” data and y are the “pure” values.

This data is obtained in Matlab by using the following commands:

x = 1:14;

y = 0.25*(x-6).^2 + normrnd(0,1,1,14);

yhat = 0.25*(x-6).^2

Fitting the quadratic dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next we fit a quadratic by using:

myfit = polyfit(x,y,2)

approx y = polyval(myfit,x)

This gives the fitted curve y = 10.3− 3.6x+ 0.29x2 and a set of values y(x),called approx y.

Fitting the quadratic dataLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next we fit a quadratic by using:

myfit = polyfit(x,y,2)

approx y = polyval(myfit,x)

This gives the fitted curve y = 10.3− 3.6x+ 0.29x2 and a set of values y(x),called approx y.

To plot this, we use

scatter(x,y); hold on; plot(x,approx y);

plot(x,yhat,’--’);

title(’Quadratic regression example’);legend(’Data points’,’Fitted quadratic’,’Original

quadratic’,’Location’, ’West’)

Fitted & original plot of quadraticLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

2 4 6 8 10 12 14

Quadratic regression example

Fitted quadraticOriginal quadratic

Residuals in quadratic regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Finally we analyse the residuals:residuals = y - approx y;

subplot(1,2,1); scatter(x,residuals); plot(x,zeros(14));

title(’Residual plot’);

subplot(1,2,2);

qqplot(residuals)

2 4 6 8 10 12 14

Residual plot

−1 0 1

Normal Q−Q Plot

Regression and Matlab

Given a data set {(x1, y1), (x2, y2), . . . , (xn, yn)}. Using Matlab:

● The coefficients of a polynomial approximation of order n is obtained by

myfit = polyfit(x,y,n)

● The approximate y values are obtained by

approx = polyval(myfit,x)

Multiple regression (§6.2)

Data with three componentsLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

The trees data set gives the relation between the height, volume and girth(diameter) of 31 cherry trees.

4.1 4.2 4.3 4.4 4.5 4.62

log(height)

2 2.5 3 3.52

log(girth)

2 2.5 3 3.54

log(girth)

corr(Volume’,Height’) = 0.5982

corr(Girth’,Volume’) = 0.9671

corr(Girth’,Height’) = 0.5193

corr(log(Volume)’,log(Height)’)

= 0.6486

corr(log(Girth)’,log(Volume)’)

= 0.9767

corr(log(Girth)’,log(Height)’)

= 0.5302

Linear regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

First linear regression on the data concerning the girth and volume, usinglog(girth) and log(volume), ignoring any information about the height.

2.2 2.4 2.6 2.8 3.0

Multiple regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Next multiple regression, writing log(volume) as a linear combination oflog(girth) and log(height). Shown are the observed volume against the fittedvolume

10 20 30 40 50 60 70

observed volume

Comparing residuals (1)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

A plot of the residuals of the linear (left) and multiple regression (right)models

2.5 3.0 3.5 4.0

Residuals Linear Regression

log(volume)

2.5 3.0 3.5 4.0

Residuals Multiple Regression

log(volume)

The residuals for the linear regression are mainly between −0.2 and +0.2 andthe residuals for the multiple regression are mainly between −0.1 and +0.1.

Comparing residuals (2)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Finally we compare the residuals of the linear regression (left) and multipleregression (right) in normal qqplots:

−2 −1 0 1 2

Normal Q−Q Plot

−2 −1 0 1 2

Normal QQ plot multiple regression

Matlab commands used (1)Linear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

● Overview plots:scatter(log(Girth),log(Height));

scatter(log(Height),log(Volume));

scatter(log(Girth),log(Volume));

4.1 4.2 4.3 4.4 4.5 4.62

log(height)

2 2.5 3 3.52

log(girth)

2 2.5 3 3.54

log(girth)

● Overview plots:scatter(log(Girth),log(Height));

scatter(log(Height),log(Volume));

scatter(log(Girth),log(Volume));

● Linear regression:myfit lin = polyfit(log(Girth),log(Volume),1);

approx lin = polyval(myfit lin,Girth)

scatter(log(Girth),log(Volume)); plot(Girth,approx lin);

2.2 2.4 2.6 2.8 3.0

● Multiple regression:X = [ones(length(Girth),1),Girth,Height];

mul = regress(Volume,X);

approx mul = mul(1) + mul(2)*Girth + mul(3)*Height;

scatter(volume,exp(approx mul))

plot([Volume(1),Volume(1)],[Volume(end),Volume(end)])

10 20 30 40 50 60 70

observed volume

● Residuals:res lin = log(Volume)-log(approx lin);

res mul = log(Volume)-log(approx mul);

subplot(1,2,1); plot(log(Volume),res lin);

subplot(1,2,2); plot(log(Volume),res mul);

2.5 3.0 3.5 4.0

Residuals Linear Regression

log(volume)

2.5 3.0 3.5 4.0

Residuals Multiple Regression

log(volume)

● Residuals:res lin = log(Volume)-log(approx lin);

res mul = log(Volume)-log(approx mul);

subplot(1,2,1); plot(log(Volume),res lin);

subplot(1,2,2); plot(log(Volume),res mul);

● Quantum-quantum plots:subplot(1,2,1); qqplot(res lin);

subplot(1,2,1); qqplot(res mul);

−2 −1 0 1 2

Normal Q−Q Plot

−2 −1 0 1 2

Normal QQ plot multiple regression

Other nonlinear regression

Nonlinear regressionLinear regression Example Predictions Matlab Quadratic regression Multiple regression Other nonlinear regression

Some nonlinear regression can be done by introducing new variables, whichmake the relation linear (and hence we can use the simple lm command in Ras before). For example:

● The exponential regression model: Y = exp(α+ βX).Note that log Y = α+ βX, hence with U = log Y , we have a linearrelation between U and X. This is U = α+ βX;

● The logarithmic regression model: Y = log(α+ βX).Note that expY = α+ βX, hence with V = expY , we have a linearrelation between V and X. This is V = α+ βX;

● The power regression model: Y = αXβ .Note that log Y = logα+ β logX, hence with W = log Y andZ = logX, we have a linear relation between W and Z. This isW = logα+ βZ.

However, most nonlinear regression models are more complicated.

COM2023 Mathematics Methods for Computing...

Documents