OLS Violation of Assumptions - Thus Spake VM...– Wrong inference CDS M Phil Econometrics...

26/10/2009

1

CDS M Phil EconometricsVijayamohan

OLSOLS

Violation ofViolation ofAssumptionsAssumptions

CDS M Phil Econometrics

Vijayamohanan Pillai N

126-Oct-09

CDS M Phil Econometrics Vijayamohan

n2T I)uu(E)u(Var

)u(E)uu(E)uu(E

)uu(E)u(E)uu(E

)uu(E)uu(E)u(E

)uu(E

2n2n1n

n22212

n12121

T

2

2

2

00

00

00

n2I

22ii )u(E)u(Var ni ,...,1

0)u,u(E)u,u(Cov jiji ji

Therefore the requirement for spherical disturbances is

(i)

and

(ii)

homoskedasticity

No autocorrelation

226-Oct-09

Assumption of Spherical DisturbancesAssumption of Spherical Disturbances

26/10/2009

2

CDS M PhilEconometricsVijayamohan

HeteroscedasticityHeteroscedasticity

3

26-Oct-09


426-Oct-09

HeteroskedasticityHeteroskedasticity: Definition: Definition

• Heteroskedasticity is a problem wherethe error terms do not have a constantvariance.

• That is, they may have a largervariance when values of some Xi (orthe Yi’s themselves) are large (orsmall).

22iiuE )(

26/10/2009

3

5

.

xx1 x2

f(y|x)

Example ofExample of HeteroskedasticityHeteroskedasticity

x3

..

E(y|x) = b0 + b1x


26-Oct-09


626-Oct-09


• This often gives the plots of theresiduals by the dependent variableor appropriate independent variablesa characteristic fan or funnel shape.

0

20

40

60

80

100

120

140

160

180

0 50 100 150

Series1

26/10/2009

4


726-Oct-09



Residual Analysis forResidual Analysis forEqual VarianceEqual Variance

Non-constant variance Constant variance

x x

Y

x x

Y

resi

dua

ls

resi

dua

ls

826-Oct-09

26/10/2009

5


9

HeteroskedasticityHeteroskedasticity

26-Oct-09

With nonspherical errors (e.g.heteroskedasticity and/or autocorrelation)

no longer applies.

Iuuu 2 )()( EVar

2)( uuE


1026-Oct-09

)()()(

)()()(

)()()(

)(

221

22212

12121

nnn

n

n

T

uEuuEuuE

uuEuEuuE

uuEuuEuE

E

uu

2

22

21

00

00

00

n

nn

00

00

00

22

11

2

2


26/10/2009

6


11

AutocorrelationAutocorrelation

26-Oct-09

)()()(

)()()(

)()()(

)(

221

22212

12121

nnn

n

n

T

uEuuEuuE

uuEuEuuE

uuEuuEuE

E

uu

1

1

1

21

21

11

2

nn

n

n

2


1226-Oct-09


Given our model, y = X + uwhere X is a non-stochastic matrix with fullcolumn rankE(u) = 0 and 2)uu(E

The OLS estimator of is

uX)XX(ˆ 1

)(E

So OLSE is still unbiased

26/10/2009

7


})ˆ)(ˆ{(E)(Var

12 )XX(s

•The variance matrix is

•Therefore any inference based on

will be incorrect.

Heteroskedasticity

11 X)X(XuuXX)X(E

11 )XX(X)uu(EX)XX( 121 )XX(XX)XX(

112 )XX)(XX()XX(

s2 may be a biased estimator of 21326-Oct-09

HeteroskedasticityHeteroskedasticity: Causes: Causes• It may be caused by:

– Model misspecification - omittedvariable or improper functional form.

– Learning behaviors across time

– Changes in data collection ordefinitions.

– Outliers or breakdown in model.

• Frequently observed in cross sectionaldata sets where demographics areinvolved (population, GNP, etc).

CDS M Phil Econometrics Vijayamohan1426-Oct-09

26/10/2009

8

HeteroskedasticityHeteroskedasticity::ImplicationsImplications

• The regression s are unbiased/consistent.

• But they are no longer the bestestimator.

• They are not BLUE (not minimumvariance - hence not efficient).


HeteroskedasticityHeteroskedasticity::Implications (cont.)Implications (cont.)

• The estimator variances are notasymptotically efficient, and they arebiased.

– So confidence intervals are invalid.

– Wrong inference


26/10/2009

9

HeteroskedasticityHeteroskedasticity::Implications (cont.)Implications (cont.)

• Types of Heteroskedasticity

– There are a number of types ofheteroskedasticity.

• Additive

• Multiplicative

• ARCH (Autoregressive conditionalheteroskedastic) - a time series problem.



1826-Oct-09

Testing forTesting for HeteroskedasticityHeteroskedasticity

A number of formal tests :

•Ramsey RESET test•Park test•Glejser test•Goldfeld-Quandt test•Breusch-Pagan test•White test

26/10/2009

10

19

• Essentially want to test

• H0: Var(u|x1, x2,…, xk) = s2,

equivalent to

• H0: E(u2|x1, x2,…, xk) = E(u2) = s2

• If assume the relationship between u2

and xj is linear, can test H0 as a linearrestriction

• So, for u2 = d0 + d1x1 +…+ dk xk + v

• this means testing

»H0: d1 = d2 = … = dk = 0


Testing for Heteroskedasticity

26-Oct-09

20

TheThe BreuschBreusch--Pagan TestPagan Test• Estimate the residuals from the OLS

regression

• Get that is

• the residuals squared divided by

• Regress zi on all of the xs.

• can have 3 tests:


n/u2i

2

22ii /uz

26-Oct-09

26/10/2009

11

21

TheThe BreuschBreusch--Pagan TestPagan Test

• can have 3 tests:

1. = ½ RSS,

where RSS = regression sum of squares

from regressing zi on all of the xs ;

2(k – 1) df.


26-Oct-09

22


2. The F statistic is just the reported F

statistic for overall significance of the

regression,

• F = [R2/k] / [(1 – R2)/(n – k – 1)],

• which is distributed Fk, n – k – 1


26-Oct-09

26/10/2009

12

23


3. The (Breusch-Pagan-Godfrey) LM

statistic is LM = nR2,

• which is distributed as 2k-1


26-Oct-09


24

TheThe BreuschBreusch--Pagan Test :Pagan Test :An ExampleAn Example

Consumption $Consumption $ Income $Income $5555 80806565 1001007070 85858080 1101107979 120120

8484 1151159898 1301309595 1401409090 1251257575 90907474 105105

110110 160160113113 150150125125 165165108108 145145115115 180180

140140 225225120120 200200145145 240240130130 185185152152 220220144144 210210175175 245245180180 260260135135 190190140140 205205178178 265265

191191 270270137137 230230189189 250250

Statistics:Linear models and related

Regression diagnosticsSpecification tests, etc.

InStata

26-Oct-09

26/10/2009

13


25

22 (1) = 3.84,(1) = 3.84, = 5%= 5%22 (1) = 6.63,(1) = 6.63, = 1%= 1%

F (1, 28) = 4.20,F (1, 28) = 4.20, = 5%= 5%F (1, 28) = 7.56,F (1, 28) = 7.56, = 1%= 1%

26-Oct-09

TheThe BreuschBreusch--Pagan Test :Pagan Test :An ExampleAn Example


26

The White Test:The White Test:White’s GeneralizedWhite’s Generalized HeteroskedasticityHeteroskedasticity

testtest

26-Oct-09

• The Breusch-Pagan test will detect anylinear forms of heteroskedasticity

• The White test allows for nonlinearitiesby using squares and crossproducts ofall the xs

• using an F or LM to test whether all thexj, xj

2, and xjxk are jointly significant

• can get to be unwieldy

26/10/2009

14


2726-Oct-09


testtest•The test proceeds as follows:

•Step 1: Estimate the original equation by least squaresand obtain the residuals

•Step 2: Regress the squared residuals on a constant,all the regressors, the regressors squared and theircross-products (interactions). For example, with twoexplanatory variables

3223

22321 xxxxxxx

•where x1 represents the constant term


2826-Oct-09


testtest

•Step 3: The test statistic is 2)1k(

2 ~nR

•If nR2 > 2

•then we have an issue withheteroskedasticity.

H0: Constant variance

26/10/2009

15

The White Test:The White Test:White’s GeneralizedWhite’s Generalized HeteroskedasticityHeteroskedasticity test:test:

An ExampleAn Example

29CDS M Phil EconometricsVijayamohan

26-Oct-09

The White Test:The White Test:White’s GeneralizedWhite’s Generalized HeteroskedasticityHeteroskedasticity test:test:

An ExampleAn Example


26-Oct-09

Generate variables inGenerate variables in StataStata

26/10/2009

16

NR2 = 15 x 0.3571 = 5.3565.

2 distribution with 5 df = 11.0705, = 5%

Conclusion ?

The White Test:The White Test: An ExampleAn Example


26-Oct-09

• nR2 < 2

•homoskedasticity


3226-Oct-09

AlternativeAlternative form of the White testform of the White test

• Consider that the fitted values from OLS, ŷ,are a function of all the xs

• Thus, ŷ2 will be a function of the squaresand crossproducts and ŷ and ŷ2 can proxyfor all of the xj, xj

2, and xjxk; so

• Regress the residuals squared on ŷ and ŷ2

and use the R2 to form an F or LM statistic

26/10/2009

17

HeteroskedasticityHeteroskedasticity: Tests: Tests(cont.)(cont.)

•• Park testPark test

• An exploratory test, log the residualssquared and regress them on thelogged values of the suspectedindependent variable.

– If the B is significant, thenheteroskedasticity may be a problem.

ln ln ln

ln

u B X v

a B X v

i i i

i i

2 2



•• GlejserGlejser TestTest

– Similar to the park test, except that it uses theabsolute values of the residuals, and a variety oftransformed X’s.

– A significant B2 indicated Heteroskedasticity.

u B B X v

u B B X v

u B BX

v

i i i

i i i

ii

i

1 2

1 2

1 2

1

u B BX

v

u B B X v

u B B X v

i

i

i

i i i

i i i

1 2

1 2

1 22

1


26/10/2009

18


3526-Oct-09


• Goldfeld-Quandt test

– Rank the n cases of the X that you thinkis correlated with ei

2 in descendingorder

– Drop a section of c cases out of themiddle(one-fifth is a reasonable number).


3626-Oct-09



– Run separate regressions on bothupper and lower (equal) samples of1/2(n - m) observations (where n =sample size and m = middleobservations)..

26/10/2009

19


3726-Oct-09

Goldfeld-Quandt test

If the disturbances are homoskedasticthen Var (Ui) should be the same forboth subsamples.

i.e., the ratio of the two residual sums ofsquares should be approximately equalto unity.



3826-Oct-09



• Do F-test for difference in errorvariances

•F has (n - c - 2k)/2 degrees offreedom for each

22

211

22

210

:H

:H

26/10/2009

20


3926-Oct-09

•This depends on the formheteroskedasticity takes.

•Indirect: Re-specify the model;•Use heteroscedastic-consistent SEs

•Direct: GLS (WLS)adjust the variance-covariancematrix

Remedies forRemedies for HeteroskedasticityHeteroskedasticity


40

HeteroskedasticHeteroskedastic ConsistentConsistent SEsSEs

26-Oct-09

OLS estimate: unbiased and consistent. But

112 )XX)(XX()XX()(Var

•This can be re-written as

12i

1 )XX(X)(DiagX)XX()(Var

),...,,(Diag)(Diag 2n

22

21

2i •where

i.e., we need to estimateall the s'2

i

- which is impossible.

2

22

21

2

00

00

00

n

26/10/2009

21


4126-Oct-09

White (1980) argues that all we really need isan estimate of


XX 2

Under very general conditions, it can beshown that

n

1iii

2i

n

1iii

2i

2 xxexxXX

Therefore the adjusted variance is

1n

1iii

2i

1 )XX(xxe)XX()(Var.asym.est


4226-Oct-09


• A consistent estimate of the variance,

• the square root can be used as a standarderror for inference

• Typically known as robust standard errors

• Sometimes the estimated variance iscorrected for degrees of freedom bymultiplying by n/(n – k – 1)

• As n → ∞ it’s all the same, though

26/10/2009

22


4326-Oct-09

HeteroskedasticHeteroskedastic ConsistentConsistent SEsSEs::Robust SEsRobust SEs

Important to remember:

Robust standard errors only haveasymptotic justification –

with small sample sizes t statistics formedwith robust standard errors will not have adistribution close to the t, and inferenceswill not be correct

In Stata, Linear regression:

SE/Robust:

(select robust – default)

26-Oct-09 CDS M Phil EconometricsVijayamohan

44

26/10/2009

23


4526-Oct-09

Generalized Least SquaresGeneralized Least Squares

• It’s always possible to estimate robuststandard errors for OLS estimates,

• But if we know something the specificform of the heteroskedasticity, we canobtain more efficient estimates than OLS

• The basic idea is going to be to transformthe model into one that hashomoskedastic errors –

• called generalized least squares


4626-Oct-09


• Given a positive definite

matrix.

• Any positive definite matrix can be

expressed in the form: PP’, where P is

nonsingular:

• = PP’, so that

• P–1 P–1’ = I and

• P–1P–1’ = –1

2)uu(E

26/10/2009

24


4726-Oct-09


• Now premultiply the model

• y = X + u by P–1 to get

• y* = X* + u*

• Where y* = P–1y ;

• X* = P–1X ;

• u* = P–1u


4826-Oct-09

Given P–1 P–1’ = I and

y* = X* + u*

Where y* = P–1y ; X* = P–1X ; u* = P–1u

Now E(u*u*’) = E(P–1uu’P–1’) = (P–12P–1’) =

(2P–1P–1’) = 2I : Homoscedastic

OLS assumptions satisfied


2)uu(E

26/10/2009

25


4926-Oct-09


• y* = X* + u*

• OLS estimate of is:

• b = (X*’ X*)–1X*’y*

• = (X’–1 X)–1X’ –1 y

• A BLUE of with

• Var(b) = 2 (X*’ X*)–1

• = 2 (X’–1 X)–1

• An unbiased estimate of 2 is:

• Where e = (y – Xb)

b is the GeneralizedLeast Squares(GLS) or Aitkenestimator of

eekn

1ˆ 12


5026-Oct-09


• If u is normally distributed, so is u*

• Thus b is a ML estimator

• So has min var in the class of allunbiased estimators.

26/10/2009

26


5126-Oct-09

Generalized Least Squares:Generalized Least Squares:Weighted Least squaresWeighted Least squares

• GLS is a weighted least squares

(WLS) procedure where each

squared residual is weighted by

the inverse of Var(ui|xi)


52

Weighted Least SquaresWeighted Least Squares

26-Oct-09

• Let heteroskedasticity be modeled as

Var(u|x) = s2h(x),

• where h(x) ≡ hi to be specified.

• Now E(ui/√hi|x) = 0, because

• hi is only a function of x, and

• Var(ui/√hi|x) = s2, because we know

• Var(u|x) = s2hi

• So divide the whole equation by √hi and we

have a model with homoskedastic error

26/10/2009

27


5326-Oct-09

Weighted Least SquaresWeighted Least Squares

For example,

A common specification: var(u) to one of theregressors or its square:

The weighted (transformed) LS regression model:

E(ui/√xki) = 0, and

Var(ui/√xki) = s2 Homoscedastic

2ki

22i x ;x)x(h 2

ki kii xh

kk

22

k

11k

k x

u...

x

x

x

x

x

y

WLS minimizes theweighted sum of

squares(weighted by 1/hi)


5426-Oct-09

Feasible GLSFeasible GLS

• More typical is the case where we don’t

know the form of the heteroskedasticity

• In this case, need to estimate h(xi)

• This is the case of FGLS

26/10/2009

28


5526-Oct-09

Feasible GLSFeasible GLS

• Run the original OLS model,

• save the residuals, û,

• square them

• Regress û2 on all of the independent

variables and

• get the fitted values, ê

• Do WLS using 1/ê as the weight


56

FGLS:FGLS: StataStata

26/10/2009

29


5726-Oct-09



5826-Oct-09


26/10/2009

30


59


Also Download wls0, using


60

-10

0-5

00

50

10

0R

esid

ua

ls

4 5 6 7 8Price (Rs.)

-10

0-5

00

50

10

0R

esid

ua

ls

2.5 3 3.5 4 4.5Advertising (Rs 1000s)

26/10/2009

31


6126-Oct-09


Other weight types abseabse and loge2loge2 andsquared fitted values (xb2xb2).


6226-Oct-09


Other weight types abseabse and loge2loge2 andsquared fitted values (xb2xb2).

26/10/2009

32


6326-Oct-09


Compare with the FGLS done by steps


AutocorrelationAutocorrelation

64

26-Oct-09

26/10/2009

33


• Autocorrelation is correlation of theerrors (residuals) over time

Violates the regression assumption thatresiduals are random and independent

-15

-10

-5

0

5

10

15

0 2 4 6 8Re

sid

ual

s

Time (t)

Time (t) Residual Plot Here, residuals show a

cyclic pattern, notrandom. Cyclicalpatterns are a sign ofpositive autocorrelation

6526-Oct-09

Autocorrelation: DefinitionAutocorrelation: Definition


6626-Oct-09


• The assumption violated is

• Thus the Pearson’s r between theresiduals from OLS and the sameresiduals lagged on period is non-zero.

0)( jiuuE

01 )( ttuuE

26/10/2009

34


6726-Oct-09


• Types of Autocorrelation

– Autoregressive (AR) processes

– Moving Average (MA) processes


6826-Oct-09

• Autoregressive processes AR(p)

– The residuals are related to their precedingvalues.

– This is classic 1st order autocorrelation: AR(1)process

ttt uu 1


26/10/2009

35


6926-Oct-09


• Autoregressive processes (cont.)

– In 2nd order autocorrelation the residuals arerelated to their t-2 values as well – AR(2):

– Larger order processes may occur as well:AR(p)

tttt uuu 2211

tptpttt uuuu ...2211


7026-Oct-09


• Moving Average Processes MA(q)

• The error term is a function of somerandom error and a portion of the previousrandom error.

• MA(1) process

1 tttu

26/10/2009

36


7126-Oct-09


Higher order processes for MA(q) also exist.

The error term is a function of some randomerror and some portions of the previous

random errors.

qtqttttu ...2211


7226-Oct-09


• Mixed processes ARMA(p,q)

• The error term is a complex function ofboth autoregressive {AR(p)} and movingaverage {MA(q)} processes.

qtqtt

tptpttt uuuu

...

...

2211

2211

26/10/2009

37


7326-Oct-09


–AR processes represent shocks tosystems that have long-termmemory.

–MA processes are quick shocks tosystems, but have only shortterm memory.


7426-Oct-09

Autocorrelation: ImplicationsAutocorrelation: Implications

• Coefficient estimates are unbiased,but the estimates are not BLUE

• The variances are often greatlyunderestimated (biased small)

• Hence hypothesis tests areexceptionally suspect.

26/10/2009

38


7526-Oct-09

Autocorrelation: CausesAutocorrelation: Causes

• Specification error

– Omitted variable

• Wrong functional form

• Lagged effects

• Data Transformations

– Interpolation of missing data

– differencing


The possible range is 0 ≤ d ≤ 4

d should be close to 2 if H0 is true

d < 2 positive autocorrelation,d > 2 negative autocorrelation

• The Durbin-Watson statistic is used to test forautocorrelation

H0: residuals are not correlated

H1: positive autocorrelation is present

7626-Oct-09

Autocorrelation: TestsAutocorrelation: Tests

n

tt

n

ttt

u

uu

d

1

2

2

21

ˆ

)ˆˆ(

26/10/2009

39


Calculate the Durbin-Watson test statistic = d

(Using Stata or SPSS)

Decision rule: reject H0 if d < dL

H0: positive autocorrelation does not exist


0 dU

2dL

Reject H0 Do not reject H0

Find the values dL and dU from the D-W table (for samplesize, n and number of independent variables, k)

Inconclusive

7726-Oct-09

Testing for +Testing for +veve AutocorrelationAutocorrelation


Decision rule: reject H0 if d < dL or 4 – dL < d < 4

H0: positive autocorrelation does not exist


0 dU2dL

Do not reject H0

4 – dU

d

44 – dL

7826-Oct-09


26/10/2009

40


Null Hypothesis Decision If

No + autocorrelation Reject 0 < d < dL

No + autocorrelation No Decision dL ≤ d ≤ dU

No - autocorrelation Reject 4 – dL < d < 4

No - autocorrelation No Decision 4 – dU ≤ d ≤ 4 – dL

No +/- autocorrelation Do not reject dU < d < 4 – dL

Durbin-Watson d Test:Decision Rules

7926-Oct-09



• Suppose we have the following timeseries data:

• Is there autocorrelation?

y = 30.65 + 4.7038xR2 = 0.8976

0

20

40

60

80

100

120

140

160

0 5 10 15 20 25 30

Sale

s

Time

(continued)

8026-Oct-09


26/10/2009

41


8126-Oct-09


Example with n = 25:y = 30.65 + 4.7038x

R2 = 0.8976

0

20

40

60

80

100

120

140

160

0 5 10 15 20 25 30

Sa

les

Time

Durbin-Watson Calculations

Sum of SquaredDifference of Residuals 3296.18

Sum of SquaredResiduals 3279.98

Durbin-Watson Statistic 1.00494

1.004943279.98

3296.18

u

)uu(

dT

1t

2

t

T

2t

21tt

ˆ

ˆˆ


• Here, n = 25 and k = 1 : one independent variable

• Using the Durbin-Watson table,

– dL = 1.29 and dU = 1.45

• d = 1.00494 < dL = 1.29,

• Therefore the given linear model is not the appropriatemodel to forecast sales

Decision: reject H0 since

d = 1.00494 < dL

0 dU=1.45 2dL=1.29

Reject H0 Do not reject H0Inconclusive

8226-Oct-09


significant +veautocorrelation exists

26/10/2009

42


83

Autocorrelation: Tests (cont.)Autocorrelation: Tests (cont.)

• Durbin-Watson d (cont.)

– Note that the d is symmetric about 2.0,so that negative autocorrelation will beindicated by a d > 2.0.

– Use the same distances above 2.0 asupper and lower bounds.


8426-Oct-09

Autocorrelation: Tests (cont.)Autocorrelation: Tests (cont.)

• Durbin’s h

– Cannot use DW d if there is a laggedendogenous variable in the model

– Syt-12 is the estimated variance of the Yt-1

term

– h has a standard normal distribution

2

112

1

tyTS

Tdh

26/10/2009

43


8526-Oct-09

Autocorrelation: RemediesAutocorrelation: Remedies

• Generalized Least Squares

• First difference method

–Take 1st differences of your Xsand Y

–Regress Y on X

–Assumes that = +1

• Generalized differences

–Requires that be known.


8626-Oct-09


Cochran-Orcutt method

(1) Estimate model using OLS and obtainthe residuals, ut.

(2) Using the residuals run the followingregression.

t1tt vuˆu

26/10/2009

44


8726-Oct-09


• Cochran-Orcutt method (cont.)

– (3) using the obtained, perform theregression on the generalized differences

– (4) Substitute the values of B1 and B2 intothe original regression to obtain newestimates of the residuals.

– (5) Return to step 2 and repeat – until nolonger changes.

)uu()XX(B)1(B)YY( 1tt1tt211tt

Date post:	09-Nov-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

OLS Violation of Assumptions - Thus Spake VM...– Wrong inference CDS M Phil Econometrics...

Documents