RD-0143 209 ROBUST MODEL SELECTION IN REGRESSION(U) PRINCETON UNIV i/iNJ DEPT OF STHTISTICS E RONCHETTI FEB 84 TR-259-SER-2RRO-i9442.i9-MFI DRRG29-82-K-0iS

UNCLASSIFIED F/G 1211 NL

11 .0 L 1 .51"0 12.2

1.2511.4==E!!. 1I!6

MICROCOPY RESOLUTION TEST CHARTNATIONAL BUREAU OF STANDARDS-1963-A

1,.

SECURITY CLASSIFICATION OF THIS PAGE (When Once Ento.,

REPORT DOCUMENTATION PAGE BFRED COKPLETINORM1. REPORT NUMBER 2.GOVT ACCESSION NO. S. R9CIPIENT'S CATALOG NMBNER

lapa rq~.qv N/A NIA4. TITLEf (mid Subtdte) S. TYPE OF REPORT & PERIOD COVERED

0') Technical Report No. 259 "Robust Model Selection ______________

0in Regression" 6. PERFORMING ORG. REPORT NUMBER

7.(~mxe S. CONTRACT OR GRANT NUMSER(.

(V)Elvezio Ronchetti -DAAG29-82-K-0178

S. PERFORMING ORGANIZATION NAMIE ANID ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK

(Department of Statistics AE OKUI UBR

Princeton UniversityPrinceton, N. J. 08544

II. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

U. S. Army Research Office February 1984Post Office Box 12211 1S. NUMBER OF PAGES

Re~nA ka np Nft, jV f 77 1014. MONITORN AGNbooM IAOESI~If,, m Cmie~ofgng 0111..) It. SECURITY CLASS. (of Chae report)

Unclassified15a. DECLASSIFICATIONIOOWNGRAOING

SCHEDULE

IS. DISTRIBUTION STATEMENT (of thil Repo~t

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of Ike abettedt mitered In Block 1,It Wiferang from Nagpof)

C) NA

LAJ S. SPPLEMENTARY NOTES

__ The view, opinions, and/or findings contained in this report areLL.. those of the author(s) and should not be construed as an official

Department of the Army position, policy, or decision, unless soCMApai 4nnt .A hy nthor dnretimtntatinl.

S19. KCEY WORDS (CmUam. on ,everee side It neceasemy ad Identity by bloek mm bet)

S Akalke Information Criterion; C criterion; M-estimators; Robust tests;Regression models.

M.ATRACr (bdn - reverseob Nf moee mu dIf 5,i b block madhor)

~A robust version of Akaike's model selection procedure for regressionmodels is introduced and its relationship with robust testing proceduresis discussed.

ON 7 0 so.O O 5 IONL UNCLASSIFIED

8 4 0 7 1 2 09 6 SECURITY CLASSIFICATION OF TIS PACE (01"n Data ntw*0E

Robust Model Selection

in Regression

by

A Elvezia Ronchetti

Technical Report No. 259, Series 2Department of Statistics

Princeton UniversityFebruary 1984

Accession For.' TS GRA&Ir:T-- TAB

J, -. if icatio

I; tr i ',ut ion/ -

K1 ,bility Cod esVivail and/or

Dist Special

This work was supported in part by U.S. Army Research Office GrantNumber DAAG29824K-0178.

Robust Model Selection

in Regression

by

Elvezio RonchettiDepartnent of StatisticsPrinceton University

SUMMARY

A robust version of Akaike's model selection procedure for regression models

is introduced and its relationship with robust testing procedures is discussed

4

Some key &i'da6: Akaike Information Criterion; Cp criterion; M-estimators;

Robust tests; Regression models.

.4

r 4 *. ' .~

1. INTRODUCTION

The Akaike Information Criterion is a powerful tool for choosing among

different models that can be used to fit a given data set. If we denote by

Lp the log-likelihood of the model with p parameters, this amounts to choose

the model that minimizes -2Lp +2P . This procedure may be viewed as an

extension of the likelihood principle and is based on a general information

theoretic criterion. In fact 2L P-2P is a suitable estimate of the expected

entropy of the model and by the Akaike Criterion the entropy will be, at

least approximately, maximized; cf. Akalke (1973).

Bhansall and Downham (1977) proposed to generalize the Akaike Criterion

by choosing the model that minimizes for a given fixed a

AIC(p;t) = -2Lp+a'P . (1)

Several proposals have been made for choosing a ; see, for instance,

_- Bhansall and Downham (1977), Atkinson (1980). If we apply (1) to a linear

regression model

Y T XT e + eI 9 l-1,...,n (2)

with n independent identically normally distributed errors with variance G2 ,

AIC(p;G) - K(n,a) + R p/2 + a.p (3)

, ...........

t ',. - ', . ,,=,;.,, ,-. : ., -.-. -. .- -' .- . ". - " . .. .. '-""'"' - "" '" '-'-"----" "----"-"""" ""-"-"" -"" --"" "'"

2

where K(n,A) is a constant depending on the marginal of the xi' C2 is

some estimate of a2 and Rp = T 2 is the residual some of squares

i=11with respect to the least squares estimate 6p . AIC(p;2) is equivalentto Mallows' C statistic; see Mallows (1973).

One of the main goals of robust statistics is to find new statistical

procedures that are not influenced too much by small deviations from the

distributional assumptions of the model. In recent years there has been

a considerable amount of work directed to construct robust estimators and

testing procedures for regression models, but the aspects related to a

robust model choice have been somewhat neglected. Since the AIC statistic

1for regression models is a direct consequence of the normality assumption

on the errors' distribution (see (3)), we cannot use it in this form with

robust estimators and robust tests. The purpose of this note is to intro-

duce a robust selection procedure for regression that, first, allows us to

choose the model which fits the majo4Lty of the data taking into account

that the errors might not be exactly normally distributed, and secondly,

that can be used consistently with new robust estimators and tests.

In Section 2 the new robust procedure is introduced and its relation-

ship to robust testing procedures is discussed. Section 3 presents some

possible choices of the parameter o for the robust selection procedure.

3

I

2. A ROBUST SELECTION PROCEDURE

Let us assume that the errors in (2) follow some distribution with

density g . Then the right hand side of (1) becomes

n

K(n,&) - 2 Z log 9((Yi-XiTn;p)/a) + a'p (4)

where Tn;p denotes the maximum likelihood estimator of e when the errors'

distribution is g . If we replace -log g in (4) by a general function p ,

we obtain the following robust selection procedure. Note that a similar

idea was used by Martin (1980) for autoregressive models.

For a given constant a and a given function p , chooses the model

that minimizes

nAICR(p;ct,p) a 2 E p(r p + p (5)

T

where ri;p (yi-xTT n;p)/a , a is some robust estimate of a and

TB;p is the N-estimator defined as implicit solution of the system of

equations

nZ *(r;p )X - 0, (6)

with *(r) - dp/dr

6:1r' ,r -,. ,, w" ,, , -w. U ",. ",,/. " t. *.*"2. . .;"". P..- . .'.. .. *..". . .,".".. .. .. .".-,-. .-.. .- ...- l . . . " " :% % " " " -" " ? * *"' * .' ' * *"*.' S' b- .

-J -. -77-7. .

* 4 .

The extension of AIC to AICR is the exact counterpart of that of

maximum likelihood estimation to M-estimation; cf. Huber (1981, Section 3.2).

In particular, if we choose p as Huber's function

pc(r) = r2/2 if Irl < o (7)

a cjri - c2/2 otherwise

then Tn;p is Huber's estimator and AICR (p;a,pc) is the generalized

Akalke statistic (1) computed under the least favorable errors' distribution

with density

go(r) = (1-c)(21)-exp(-pc(r)) , (8)

where c is a function of the contamination e ; cf. Huber (1981, Chapter 4).

In this case a robust estimate for a can be obtained using Huber's Proposal 2

(Huber 1981, p. 137) or Hampel's median absolute deviation (Hampel 1974,

p. 388) in the model with all parameters.

Let us now investigate the relationship between AICR and robust testing

procedures. Denote by e( l) the Jth component of the vector 0 and let

HoO() = 0 , j - q+1,...,p

be the null hypothesis in the model (2). Denote by A the likelihood ratio

test statistic and define

5

1. = 2(p-q)"I log A (9)

Then it is easy to see that

tqsp = o - (p-q)I(AIC(p;a) - AIC(q;)) (10)

If we substitute the likelihood ratio test statistic Zq,p by a robust

version, namely

Srob . 2(p-q" (D(R)-D()),*q ,p

nwhere D(F) is the minimum value of 1

p(ri;p) and D(R) is then =

minimum value of Z=I p(r.;p) subject to Ho , the dispersion of the residuals

under the full and reduced models respectively (see Schrader and Hettmansperger,

1980; Ronchetti, 1982), we obtain

pro = (L- (p-q) 1(AICR(p;,p) - AICR(q;,p)) (12)

(12) is the natural counterpart of (10) when using robust estimators and test.

i

%E!9% % % ~ W~ V ****% *~, .. ~ ~ .. ~j

- "* ,, T.:, ., r.-', .,'_ . -,\,- .' \ % ,- , -.;,,-.;-%-w , -W . -.- -. .-.. , .7. _...%-. -.......-. **. ....-

64 ..¢ J.

3. CHOICE OF THE PARAMETER a

In this section we propose a choice for the parameter a in AICR(p;cx,pc)

It is based on the following result due to Stone (1977).

The Akaike statistic AIC(p;2) is asymptotically equivalent to

-2Lp + trace(MIN) , (13)

where -M2 is the (pxp) matrix of the second derivatives (with respect to e )

of the log-likelihood function and M, is the (pxp) matrix of the products

of the first derivatives. Since AICR(p;a,pc) can be viewed as the Akaike

statistic computed under the least favorable errors' distribution g.

(see (8)), we obtain

M -Eqc • ExxT

M2 = E4p • ExxT

where *c(r) - dp/dr = r If Irl C c

= c.sign(r) otherwiseThus, 2 trce MI) - 2(E*c1EvcI)p and we propose to choose a=ac" 2 E*cIEic < 2

Note that a = 2 and AICR(p;a.,p,) = AIC(p;2) which is the classical

Akaike statistic under normality.

IL

I ~~~ ~ ~ ~ ~~. ~* %~ .-.- . - -.- -.V -. % S-... -, ,-. . ".v.-,...-,*.

7

Remark

Hampel obtains another choice for a "by adding the average decreasen

of j p(r1 ) and the average increase of the total mean square error of fit1=1

due to a superfluous parameter under normality" (Hampel, 1983). His choice

for a is

a=+ 2

that differs little from 2 for the usual values of c (e.g. c between 1.3

and 1.6).

ACKNOWLEDGEMENTS

The author is grateful to Prof. F.R. Hampel for stimulating discussions.

Partial support of ARO (Durham) contract #DAAG29-82-K-0178 is also gratefully

acknowledged.

4..

RE FE RENCES

A' ike, H. (1973). Information theory and an extension of the maximumlikelihood principle. Second InteAnationaZ Syfmposi.um on InjowiationTheo'uj. Academial Kiado, Budapest, 267-81.

Atkinson, A.C. (1980). A note on the generalized information criterionfor a choice of a model. Siomet~ika~ 67, 413-8.

Bhansall, R.J. and Dowsham, D.Y. (1977). Some properties of the order C'autoregressive model selected b y a generalization of Akaike's FPE cy erion.8*omet'Lkd 67, 547-51.

Hampel, F.R. (1974). The influence curve and its role in robust estim :,.J. Am. Sttit. Ass&oc. 69, 383-93.

Hampel, F.R. (1983). Some aspects of model choice in robust statistics.Proceedings of the 44th Session of ISI. To appear.

Huber, P.3. (1981). Robus~t SUtatitcs. Wiley. New York.

Mallows, C.L. (1973). Some Commuents on C .*rTecnometmics, 15, 661-75.

Martin, R.D. (1980). Robust estimation of autoregressive models. Dikectionzi Timne Smes'.e. Inst. of Math. Statist., 228-62.

Ronchetti, E. (1982). Robust alternatives to the F-test for the linear model.P'wobabititg and St&ti ta Ynje.'ence. Reidel, Dortrecht, 329-42.

Schrader, R.M. and Hettmansperger, T.P. (1980). Robust analysis of variancebased upon a likelihood ratio criterion. Biometxr.ka. 67, 93-101.

Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. J.R. Statist Soc. B 39, 44-7.

'X0

IF,, 4c :'

' &

14 !.

_K5

- . , N

'.4,:,jot

9 ,.4f