1

Engineering Systems Analysis

ENGN 2226

ENGINEERING SYSTEMS ANALYSIS

L24: Linear regression

2

Engineering Systems AnalysisLinear Models

• Very many causal relationships can be exactly described by linear models.

• Many causal relationships can be adequately modelled by linear models.

• Linear models are a basic tool that is useful in a variety of situations and are a fundamental tool for engineering systems analysis.

3

Engineering Systems AnalysisIdentifying linear models in data.

Start with a set of data that has several measured variables for each instance of the population.

4

Engineering Systems AnalysisScatter plot

Look for structure in the scatter plot of the two variables.

In this case we are looking for linear structure.

5

Engineering Systems AnalysisLinear predictive model

The model can be used to predict values for values of the response variable that have not been measured.

Question: How does one compute the parameters a and b that define the model.

6

Engineering Systems AnalysisGraphical representation of error

Measurement noise case

Least squares error

Gaussian noise

Noise equally in both variables

Total least squares cost

Gaussian noise

We will deal exclusively with the measurement noise case in this course.

7

Engineering Systems AnalysisMeasurement Noise

8

Engineering Systems AnalysisLinear Model Estimators

• The mean response of the model is a straight line function, the population regression line, of the explanatory variable.

• The measurement yk, yk+1, yk+2 are sampled from a normal distribution with variance 2 around the point a xk+ b .

yk+1

yk+2

yk

k k+1

k+2

9

Engineering Systems AnalysisLinear Model Estimators

• We can see the difference between the actual measurements and the estimated measurements using our linear model. Now we want to try and understand the error between the two caused by our model

yk+1

yk+2

yk

k k+1

k+2

10

Engineering Systems AnalysisLinear Model Estimators

• When we are choosing an estimator we need to determine the objective we want to meet. This will determine the form (and the mathematical formula) of the estimator we need.

• There are lots of possible objectives but the most common is to reduce the sum squared error (SSE) between our linear model and the observations we have.

11

Engineering Systems AnalysisLeast squares error

The error term is the difference between the predicted output and the measured output

Least squares error

12

Engineering Systems AnalysisEffect of minimising LS cost

The LS cost acts to minimise the squared error of the residue equally along the predictive linear model.

13

Engineering Systems AnalysisLeast Squares Estimator

14

Engineering Systems AnalysisProof of LSE

• On the Board

15

Engineering Systems AnalysisLeast Squares Estimator

16

Engineering Systems AnalysisLeast Squares Example (The Data)

• We have a set of data with explanatory and response variables

• How many explanatory and response variable pairs are needed to find an estimate of the linear model?

17

Engineering Systems AnalysisLeast Squares Example (Data Mean)

18

Engineering Systems AnalysisLeast Squares Example (Data Variance)

19

Engineering Systems AnalysisLeast Squares Example (The LSE)

20

Engineering Systems AnalysisLeast Squares Example (The LSE)