Econometrics
1
Lecture 3: Properties of the OLS Estimator For the OLS estimator yXXXb ')'( 1−= define:
• Vector of OLS residuals
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
−
−
=−=
∑
∑
=
=
K
kknkn
K
kkk
bxy
bxy
Xbye
1
111
Econometrics
2
• Vector of OLS fitted or predicted values
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
==
∑
∑
=
=
K
kknk
K
kkk
bx
bx
Xby
1
11
ˆ
Econometrics
3
Residuals and fitted values have properties
a. 0' =eX
b. If the regression has an intercept
bxy '= with
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡=
=
∑
∑
=
=
n
iiK
n
ii
xn
xn
x
1
11
1
)1(1
c. If the regression has an intercept
yy ˆ=
Econometrics
4
d. If the regression has an intercept
∑∑ ∑== =
+−=−n
ii
n
i
n
iii eyyyy
1
22
1 1
2 )ˆ()(
Econometrics
5
Proofs and remarks: Ad a. 0' =eX Proof: 0'')('' =−=−= XbXyXXbyXeX This implies that each column of X is orthogonal to vector e , i.e. for
Kk ,,1…=
∑=
=n
iiik ex
10
If regression has intercept (and hence X has column equal to ι , a vector of 1-s), then
∑=
===n
ii eee
10'ι
Econometrics
6
The fundamental assumption in CLR model 0)|( =XE ε implies that 0)'( =εXE Hence OLS makes sample analog of )'( εXE equal to 0.
Econometrics
7
Note MyyXXXXIXbye =−=−= − )')'(( 1 with ')'( 1 XXXXIM −−= The matrix M has a number of properties 0=MX
MM =' M is symmetric MM =2 M is idempotent
Econometrics
8
Also PyyXXXXXby === − ')'(ˆ 1 with ')'( 1 XXXXP −= Note
PP =' P is symmetric PP =2 P is idempotent Because Xby =ˆ the matrix P projects yon the space spanned by the columns of X . For this reason P is called the least squares projection matrix. Projection matrices are idempotent, because if we project again nothing changes.
Econometrics
9
Because 0' =eX and M is also a projection matrix, e is the projection of y on space orthogonal to the columns of X . Note eyy += ˆ Also 0'''ˆ == eXbey i.e. y and e are orthogonal and y is the sum of two orthogonal vectors.
Econometrics
10
In graph (one observation with 2xxyb = )
y
bxy =ˆ
bxye −=
Econometrics
11
Ad b: bxy '= Proof: If we write [ ]KxxX 1ι= with kx the −k th column of X , then the normal equations can be written as
y
x
xXb
x
x
KK⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
′
′=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
′
′ 22
'' ιι
Econometrics
12
The first equation is yXb '' ιι = or
yn
Xbn
'1'1 ιι =
or ybx ='
Econometrics
13
Ad c: yy ˆ= Proof: Because eyy += ˆ we have
en
yn
yn
'1ˆ'1'1 ιιι +=
or
yy =ˆ In words: sample average of y is predicted exactly.
Econometrics
14
Ad d: ∑∑ ∑== =
+−=−n
ii
n
i
n
iii eyyyy
1
22
1 1
2 )ˆ()(
Proof: Because iii eyyyy +−=− ˆ , we have
∑ ∑∑∑= ===
−++−=−n
i
n
iiii
n
ii
n
ii yyeeyyyy
1 1
2
1
2
1
2 )ˆ(2)ˆ()(
The last term is 0.
Econometrics
15
Terminology
∑=
−n
ii yy
1
2)( Total Sum of Squares (TSS)
∑=
−n
ii yy
1
2)ˆ( Explained Sum of Squares (ESS)
∑=
n
iie
1
2 Residual Sum of Squares (RSS)
Hence RSSESSTSS +=
Econometrics
16
If we divide by TSS
TSSRSS
TSSESS
−= 1
Coefficient of determination
∑
∑
=
=
−
−== n
ii
n
ii
yy
yyR
1
2
1
2
2
)(
)ˆ(
TSSESS
This is a measure of goodness-of-fit.
Econometrics
17
Note 10 2 ≤≤ R For the extreme values
XbynieeR in
ii =⇔==⇔=⇔= ∑
=,,1,001
1
22 …
i.e. a perfect fit.
0,
ˆ0)ˆ(0
21
1
222
====⇔
⇔=⇔=−⇔= ∑=
K
in
ii
bbyb
yyyyR
i.e. varying regressors Kxx ,,2 … do not help to explain y .
Econometrics
18
Because ∑∑==
−−=−n
iii
n
ii yyyyyy
11
2 ))(ˆ()ˆ(
2
1
2
1
2
1
1
2
1
2
2
1
2
1
2
1
2
2
)ˆ()(
))(ˆ(
)ˆ()(
)ˆ(
)(
)ˆ(
∑∑
∑
∑∑
∑
∑
∑
==
=
==
=
=
=
−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
=
=−−
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=−
−=
n
ii
n
ii
n
iii
n
ii
n
ii
n
ii
n
ii
n
ii
yyyy
yyyy
yyyy
yy
yy
yyR
The final expression is the sample correlation of y and y .
Econometrics
19
Note in 2R we use deviations from sample average, because
∑
∑
=
=n
ii
n
ii
y
y
1
2
1
2ˆ
can be made close to 1 by adding a constant to y .
Econometrics
20
Partitioned regression Consider uXy += β Partition X as
[ ]
21
21
KnKnKnXXX
×××=
2
1
2
1
KK
⎥⎦
⎤⎢⎣
⎡=
ββ
β
with KKK =+ 21 . Hence we can write εββ ++= 2211 XXy
Econometrics
21
What is the formula for the OLS estimator of 1β ? Define (compare with M defined previously) 2
12222 )( XXXXIM ′′−= −
2M is symmetric and idempotent ( 2
22 MM = ).
Further define **
222 ybXyyM =−= with yXXXb 2
122
*2 )( ′′= −
In words: *y is vector of OLS residuals of regression of y on 2X .
Econometrics
22
In the same way define 12
*1 XMX =
In words: the −k th column of *
1X is the vectors of OLS residuals of the −k th column of 1X on 2X .
Hence
**
11*
1*11 ')'( yXXXb −=
In words: the OLS estimator in a regression on 1X and 2X , 1b , is the OLS estimator of the OLS residuals of y (in regression on 2X ) on the OLS residuals of 1X (in regression on 2X ).
Econometrics
23
Note *1
*, Xy are purged of the effect of 2X , because OLS residuals are orthogonal to /uncorrelated with 2X . This is how least squares implements the ceteris paribus condition, i.e. 1b is the effect of 1X ‘holding 2X constant’. More like randomized assignment of 1X : *
1X is uncorrelated with the ‘omitted’ 2X , which is relegated to random error if only 1X is included.
Econometrics
24
Special cases
• Columns of 1X and those of 2X are orthogonal/uncorrelated 012 =′ XX , so that 112 XXM = and
yXXXb 1
1111 )( ′′= −
Conclusion: if we omit 2X we get the same estimate.
• ι=2X with ι an −n vector of 1’s. Then
nIIM ')( 1
2ιιιιιι −=′′−= −
yynyyyM ιιι
−=′
−=2
In words: 2M takes y in deviation from its sample average.
Econometrics
25
Conclusion: if we take both y and X in deviation from the sample means we obtain OLS estimators of y on varying regressors.
Econometrics
26
Estimation of 2σ The other parameter in CLR model is the variance of the random error 2σ . Obvious idea: use sample variance of the OLS residuals e . Instead we use (in model with K regressors including constant)
eeKn
eKn
sn
ii ′
−=
−= ∑
=1
22 11
Econometrics
27
The sampling distribution of 2s can be derived from εεβ MXMMye =+== )( so that
εεMKn
s−
=12
It can be shown that 22 )|( σ=XsE so that ( ) 222 )|()( σ== XsEEsE X In words: 2s is an unbiased estimator of 2σ .