This article was downloaded by: [University of Chicago Library] On: 19 August 2013, At: 12:03 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Technometrics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/utch20 The Analysis of Unreplicated Factorial Experiments Using All Possible Comparisons Arden Miller a a Department of Statistics, University of Auckland, Auckland, New Zealand Published online: 01 Jan 2012. To cite this article: Arden Miller (2005) The Analysis of Unreplicated Factorial Experiments Using All Possible Comparisons, Technometrics, 47:1, 51-63, DOI: 10.1198/004017004000000608 To link to this article: http://dx.doi.org/10.1198/004017004000000608 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions
Transcript
This article was downloaded by: [University of Chicago Library]On: 19 August 2013, At: 12:03Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK
TechnometricsPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/utch20
The Analysis of Unreplicated Factorial ExperimentsUsing All Possible ComparisonsArden Millera
a Department of Statistics, University of Auckland, Auckland, New ZealandPublished online: 01 Jan 2012.
To cite this article: Arden Miller (2005) The Analysis of Unreplicated Factorial Experiments Using All PossibleComparisons, Technometrics, 47:1, 51-63, DOI: 10.1198/004017004000000608
To link to this article: http://dx.doi.org/10.1198/004017004000000608
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose ofthe Content. Any opinions and views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be reliedupon and should be independently verified with primary sources of information. Taylor and Francis shallnot be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and otherliabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
This article proposes a new procedure for analyzing small unreplicated factorial experiments. This pro-cedure is based on using likelihood ratio tests to compare competing models. An easy method of imple-menting the procedure is presented and then demonstrated on a real set of data. Results of a simulationstudy are presented that indicate that the new procedure compares favorably with Lenth’s method. Tablesof constants are supplied that allow the new procedure to be easily applied to the analysis of 8-run, 12-run,and 16-run experiments.
KEY WORDS: Active contrast; Geometric approach; Half-normal plot; Lenth’s method; Likelihoodratio test; Model selection.
1. INTRODUCTION
Small unreplicated experiments, such as two-level fractionalfactorials, are frequently used in a screening capacity near thestart of a systematic investigation of a process. At this stagethe experimenter has identified a set of candidate factors, andthe goal of the experiment is to determine which of these have anonnegligible impact on the response either through a main ef-fect or through an interaction. Such factors are said to be active.Typically, the results of the experiment are used to obtain esti-mates for a set of orthogonal contrasts. For complete factorialdesigns, each contrast represents either a main effect or an inter-action effect, whereas for fractional factorial designs, each con-trast represents a set of effects that are completely confoundedwith each other. In either case, a key step in identifying the ac-tive factors is to first determine which estimates correspondedto active (nonnegligible) contrasts and which estimates merelyreflect experimental variability.
For screening applications, it is often the case that n − 1orthogonal contrasts are estimated, where n is the number ofruns used in the experiment. This is equivalent to fitting a sat-urated linear model, and thus there is no opportunity to ob-tain an independent estimate of experimental variability fromthe data. The experimenter ends up with a set of estimates butno standard errors. Many different approaches to the problemof model selection under these circumstances have been pro-posed in the statistical literature. Typically, these approachesare based on using contrasts that have been standardized sothat the estimates all have a common variance and the assump-tion that at least some of the contrasts are not active. The nor-mal plot and the closely related half-normal plot, introducedby Daniel (1959), remain the most popular methods despitetheir subjective nature. Numerous nonsubjective alternativeshave been proposed by, for example, Birnbaum (1959), Holmsand Berrettoni (1969), Box and Meyer (1986), Voss (1988),Benski (1989), Lenth (1989), Berk and Picard (1991), Le andZamar (1992), Dong (1993), Haaland and O’Connell (1995),Chipman (1996), Venter and Steel (1996, 1998), Langsrud andNæs (1998), Al-Shiha and Yang (1999, 2000), Aboukalam and
Al-Shiha (2001), and Ye, Hamada, and Wu (2001). Hamada andBalakrishnan (1998) compared 24 different procedures via anextensive simulation study. That article and the contributionsmade by the discussants provided an illuminating comparisonof the different approaches to model selection and an insightfuldiscussion of the issues involved in evaluating and comparingprocedures. Despite the considerable attention that this area ofresearch has received, there is still much diversity of opinion.
The present article proposes a novel approach to model selec-tion termed all possible comparisons (APC) because it is basedon using likelihood ratio statistics to consider all possible pair-wise comparisons between the candidate models. Section 2 in-troduces the APC procedure and demonstrates it using real data.Section 3 presents two versions of the APC procedure and pro-vides tables of constants needed to implement these for 8-, 12-,and 16-run experiments. Section 4 compares the performanceof APC to that of Lenth’s method (Lenth 1989). Section 5 con-tains the concluding remarks.
Throughout this article, a geometric interpretation of modelselection is used to clarify the issues involved. This geomet-ric approach is similar to that presented by Saville and Wood(1991, 1996) for standard statistical techniques. A more de-tailed presentation of the geometric interpretation of model se-lection procedures has been given by Miller (2003).
2. MODEL SELECTION USING ALLPOSSIBLE COMPARISONS
This section starts by introducing the geometric interpreta-tion of model selection, which is used to motivate the APCapproach. The APC approach is developed, and an easy-to-implement procedure is presented. Finally, the proposed pro-cedure is demonstrated using real data.
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1DOI 10.1198/004017004000000608
51
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
52 ARDEN MILLER
2.1 The Geometry of Model Selection
To facilitate a geometric interpretation, we present the modelselection problem in vector form. Consider an n-run, unrepli-cated factorial experiment. Let Y represent the response vector,and let X1, . . . ,Xk (where k = n − 1) represent vectors that cor-respond to the contrasts of interest. Assume that the Xi’s arepairwise orthogonal and have been scaled to have equal length.The full (saturated) linear model can be written as
Y ∼ N(µ, σ 2I),(1)
µ = β01 + β1X1 + β2X2 + · · · + βkXk,
where I is the identity matrix and 1 is a vector of 1’s. If the coef-ficients β1, . . . , βk are estimated using least squares, then the es-timates are unbiased, equivariant, independent normal randomvariates. The experimenter must identify which estimates corre-spond to active contrasts, but lacks an estimate of experimentalvariability. To overcome this deficiency, the experimenter mustassume that at least some of β1, . . . , βk are 0. Let m, wherem ≤ k − 1, represent an upper bound placed on the number ofactive contrasts, and define the set of candidate models as beingall models that contain m or fewer contrasts.
Consider the vector Y∗ created by centering the responsevector and scaling to unit length, Y∗ = (Y− Y)/‖Y− Y‖. Notethat Y∗ retains the relevant information for any model selec-tion procedure that is invariant to linear transformations of theresponse, this encompasses all procedures known to us. Thevector Y − Y is an element of the vector space, Vk, spannedby X1, . . . ,Xk, and Y∗ is the projection of Y − Y onto thek-dimensional unit sphere, Sk, in Vk . Thus the domain of Y∗is Sk, and any model selection procedure can be viewed as par-titioning Sk into a set of regions where each region correspondsto the selection of one of the candidate models. We call theseregions selection regions.
The properties of any selection procedure are determined bythe selection regions that it produces. To gain insight into thisconnection and thereby insight into desirable characteristics forthe selection regions, it is instructive to consider the distributionof Y∗ under the various candidate models. For the null model(i.e., µ = β01), Y∗ is uniformly distributed over Sk. Thus theprobability that Y∗ falls in any region defined on Sk is propor-tional to the area of that region. For nonnull models, Y∗ has aprojected normal distribution. This distribution depends on twoparameters that are directly related to the centered mean vectorµ − µ = β1X1 + β2X2 + · · · + βkXk. Let µ∗ be the projectionof µ−µ onto Sk, µ∗ ≡ µ−µ/‖µ−µ‖, and let ν represent thelength of µ − µ measured in number of σ ’s, ν ≡ ‖µ − µ‖/σ .The distribution function of Y∗ at any point y∗ ∈ Sk (adaptedfrom Small 1996, p. 132), is
f (y∗) = (2π)−k/2 exp(−ν2/2)
×∫ ∞
0exp(−r2/2 + rν cosθ)rk−1 dr
where θ represents the angle between y∗ and µ∗. This func-tion can be difficult to work with, but for the purposes of thisarticle it is only necessary to understand its basic properties.For fixed ν > 0, f (y∗) is a decreasing function of θ ; its maxi-mum occurs when y∗ = µ∗ and decreases as the angle between
y∗ and µ∗ increases. As the value of ν increases, f (y∗) becomesmore concentrated near µ∗. Thus µ∗ determines the “expecteddirection” of Y∗, and ν determines the amount of dispersionof Y∗ around µ∗.
Suppose that the true model consists of q active contrasts,and, without loss of generality, assume that the active contrastsare X1, . . . ,Xq when q > 0. The probability of correctly select-ing the true model can be calculated by integrating f (y∗) underthat model over its selection region. Thus, in rough terms, theselection region should contain points for which f (y∗) is largeunder that model. For q = 0, f (y∗) is constant over Sk, and thusall points are equally suited for inclusion in the null model se-lection region. For q > 0, f (y∗) decreases as y∗ moves awayfrom µ∗, and thus the selection region should contain pointsclose to µ∗. However, µ∗ = (β1X1 + · · · + βqXq)/‖µ − µ‖,and thus for a given true model, µ∗ can occur over a range ofpoints on Sk depending on the values of β1, . . . , βq. Note thatfor q = 1, the domain of µ∗ consists of the two points whereX1 intersects Sk; for q = 2, the domain of µ∗ is the circle de-fined by the intersection of the X1, X2-plane and Sk; and ingeneral the domain of µ∗ is the q-dimensional sphere definedby the intersection of the X1, . . . ,Xq-hyperplane and Sk. As aconsequence, the selection region for a given model should con-tain points close to the domain of µ∗ under that model. The fol-lowing section presents a coherent method of dividing Sk intoselection regions based on this premise.
2.2 Likelihood Ratio Compliant Selection Regions
The model selection problem can be considered an exten-sion of hypothesis testing in that a hypothesis test divides thespace of possible outcomes into two regions, whereas a modelselection procedure divides this space into a larger number ofregions. For hypothesis tests, this division is often based onlikelihood ratio statistics. This article proposes using likelihoodratio statistics to create the selection regions for a model selec-tion procedure. The idea is to produce a set of selection regionssuch that for any pair of regions, RA and RB, RA ∪ RB has beenpartitioned in a manner consistent with having used a likelihoodratio statistic to make this division. A set of selection regionsis said to be likelihood ratio compliant if each pair of regionshas this property. The proposed APC procedure generates like-lihood ratio-compliant selection regions by in effect comparingeach pair of candidate models using a likelihood ratio statistic.The tests are devised in a manner that ensures that the results donot contradict each other (details follow). The selection regionfor a particular candidate model consists of all y∗ where thatmodel is ranked above every other model by these tests. We ar-gue that this approach partitions Sk into selection regions on thebasis of how close y∗ ∈ Sk is to the domain of µ∗ under each ofthe nonnull candidate models.
Consider a single comparison between models A and B. Weconsider the likelihood ratio test based on the sampling dis-tribution for Y because it is difficult to work with the sam-pling distribution for Y∗. The relevant likelihood ratio statisticis (RSS[A]/RSS[B])n/2, where RSS[A] and RSS[B] representthe residual sums of squares for the fitted models. Equivalently,the comparison can be based on RSS[A]/RSS[B], which is de-noted by RSS[A/B]. This ratio partitions the space of possible
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 53
outcomes for Y, but because RSS[A/B] is invariant to lineartransformations, it will have the same value for all values of Ythat project onto the same y∗ ∈ Sk. Therefore, RSS[A/B] canalso be used to partition Sk.
The comparison of models A and B involves comparingRSS[A/B] to a suitable constant. For the proposed procedure,every model of a given size is treated in the same manner, andthus this constant will depend only on the numbers of contrastsin the two models. Assume that models A and B contain j and j†
contrasts, and let cjj† be the relevant constant. Further, let A Bindicate that model A is ranked higher than model B and letA ≈ B indicate that the two models are ranked equally. Thenthe proposed comparison can be summarized as (a) A B ifRSS[A/B] < cjj† , (b) B A if RSS[A/B] > cjj† , and (c) A ≈ Bif RSS[A/B] = cjj† . Note that the two models are allowed to beequally ranked. Theoretically, this does not complicate matters,because A ≈ B occurs with probability 0.
This comparison of models A and B has a straightforward in-terpretation in the geometric framework. First, suppose that B isthe null model and that A is a nonnull model. It is well knownthat RSS[B] is equal to the squared length of the centered re-sponse vector, RSS[B] = ‖Y − Y‖2, and that RSS[A] is thesquared perpendicular distance between Y − Y and the vectorspace, VA, spanned by the active contrasts under model A. ThusRSS[A/B] = sin2 φA, where φA is the angle between Y − Y andthe orthogonal projection of Y−Y onto VA. This result can alsobe interpreted in terms of Y∗ and ϒA, where ϒA represents thedomain of µ∗ on Sk under model A. Because Y∗ has the samedirection as Y − Y and ϒA is the intersection of VA with Sk,φA also represents the angle between Y∗ and the nearest pointon ϒA. Thus RSS[A/B] can be considered a measure of the dis-tance between Y∗ and ϒA. Clearly, then, the likelihood ratiotest selects model A over the null model if Y∗ is within a spec-ified distance of ϒA. Next, consider the situation where bothmodels A and B represent nonnull models. Then, adapting thejust-introduced notation in the obvious manner, it is straight-forward to deduce that RSS[A/B] = sin2 φA/ sin2 φB. Thus thelikelihood ratio test chooses between models A and B by com-paring the distance Y∗ is from ϒA to the distance it is from ϒB.
Defining the set of comparisons is equivalent to defining theset of constants {cjj† : j, j† = 0,1, . . . ,m}. The results of theindividual comparisons must be transitive to ensure that theydo not contradict each other. In basic terms, this means that(a) A ≈ B and B ≈ C must imply A ≈ C, (b) A B and B Cmust imply A C, (c) A B and B ≈ C must imply A C, and(d) A ≈ B and B C must imply A C. This transitivity re-quirement can be achieved by placing the following restrictionson the constants:
To demonstrate that (2) ensures requirement (a), consider mod-els A, B, and C with j, j†, and j‡ active contrasts and note that
RSS[A/C] = RSS[A/B] × RSS[B/C]. (3)
For requirement (a), if A ≈ B and B ≈ C, then RSS[A/
B] = cjj† and RSS[B/C] = cj†j‡ . Substitute these into (3) to getRSS[A/C] = cjj† × cj†j‡ . Thus if (2) holds, then A ≈ C. That(2) ensures the other forms of transitivity can be shown in asimilar manner.
From (2), it can readily be shown that cjj = 1 (set j = j† = j‡)and cjj† = 1/cj†j (set j = j‡). Further, it follows directly from (2)that cjj† = cj0/cj†0. Thus the entire set of constants can be gen-erated from c10, . . . , cm0. Therefore, to specify the entire set ofhypothesis tests, it is necessary to specify only the tests thatinvolve the null model.
2.3 The APC Procedure
A simple procedure can be devised that identifies the bestmodel without actually having to perform all pairwise com-parisons between the candidate models. The key is to definea selection score in a manner such that for any pair of models,the preferred model will always have the higher selection score.Thus the best model is simply the one with the highest selectionscore.
Let Mj denote the best model of size j. Note that M0 repre-sents the null model and that for j = 1 to m, Mj is the model thatcorresponds to the j largest estimated effects. Clearly, the bestmodel overall must be one of the Mj’s. For each Mj, define itsselection score as
ss[Mj] = cj0
RSS[Mj/M0] .
Consider comparing Mj and Mj† . Using the relations RSS[Mj/
Mj†] = RSS[Mj/M0]/RSS[Mj†/M0] and cjj† = cj0/cj†0, it is asimple matter to show that selecting the model with the higherselection score is equivalent to comparing RSS[Mj/Mj†] to cjj†
as described previously.Calculating the selection scores can be simplified by us-
ing the fitted coefficients for the full model from (1). Notethat as this model is saturated, Y = Y = β0 + ∑
βjXj, andthe (corrected) total sum of squares is TSS = (
∑k1 β2
i )‖X‖2.It is not difficult to see that RSS[Mj] can be calculated asfollows: Order the β2
i ’s from largest to smallest, label theseas β2
(1), . . . , β2(k), and find RSS[Mj] = (
∑kj+1 β2
(i))‖X‖2. Thus
RSS[Mj/M0] = (∑k
j+1 β2(i))/(
∑k1 β2
i ).Therefore, the following procedure can be adopted:
1. Select a suitable set of constants c10, . . . , cm0. The identi-fication of such constants is considered in Section 3.
2. Order the squared estimated effects from largest to small-est and label these β2
(1), . . . , β2(k). Let Mj represent the
model that corresponds to the j largest squared estimatedeffects (M0 is the null model).
3. For j = 1 to m, calculate
RSS[Mj/M0] =k∑
i=j+1
β2(i)
/ k∑i=1
β2i .
4. Set ss[M0] = 1, and for j = 1 to m, calculate
ss[Mj] = cj0
RSS[Mj/M0] .
5. Select the model corresponding to the largest selectionscore.
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
54 ARDEN MILLER
2.4 Filtration Rate Example
Box, Hunter, and Hunter (1978, pp. 424–427) presentedthe results of an 8-run 27−4 screening experiment used inthe initial stages of an investigation into an unusually slowfiltration rate at a chemical plant. Table 1 contains the datafrom this experiment. The response Y measures filtrationtime in minutes. Each of the contrasts X1, . . . ,X7 representsa set of 16 confounded effects. In each set there is onemain effect and three two-factor interactions. Fitting a lin-ear model of the form given in (1) gives {β1, . . . , β7} ={−5.44,−1.39,−8.29,1.59,−11.41,−1.71, .26}. Note thatthese estimates are half the estimated effects given in the refer-ence. This is due to an alternative definition of effect used andhas no impact on model selection.
Consider using the APC procedure outlined in Section 2.3to identify the active contrasts. The first step is to selectc10, . . . , cm0. Assume that the practitioner has decided to setm = 4 and to use {c10, c20, c30, c40} = {.4068, .1438, .04240,
.01115}. These values were obtained from Table 3 and representwhat we believe is a reasonable choice for this application—the problem of identifying values for the cj0’s is addressed inSection 3. The next step is to square the estimates and or-der these from largest to smallest: β2
(1), . . . , β2(7) = 130.245,
68.683,29.566,2.933,2.520,1.925, .069. Model Mj, for j = 1to m, is identified as the model corresponding to the j largestsquared estimates. The results of the calculations from steps3 and 4 of the procedure are given in Table 2. Because M3 hasthe largest selection score, X5, X3, and X1 are identified asactive contrasts. This agrees with the selection made by Boxet al. (1978, p. 427).
3. SELECTING CONSTANTS FOR THE ALLPOSSIBLE COMPARISONS METHOD
The APC approach developed in the previous section definesa very flexible family of model selection procedures. The prop-erties of a specific procedure are determined by the set of con-stants c10, . . . , cm0. This section considers the selection of theseconstants. First, restrictions on c10, . . . , cm0 are discussed, thentwo strategies for identifying sets of constants are proposed.Both strategies are based on controlling the error rate of theresulting procedure. Tables of constants needed to implementthe APC procedures created using these strategies are given for8-run, 12-run, and 16-run experiments. Finally, some guidancein selecting a specific version of APC from these tables is given.
First, consider the restrictions that must be placed on c10, . . . ,
cm0 to produce a viable procedure. Implicit in setting c10, . . . ,
cm0 is selecting the maximum model size m, that must sat-isfy 1 ≤ m ≤ k − 1. Restrictions on the cj0’s result fromRSS[Mj+1] < RSS[Mj] for j = 0 to m − 1. Trivially, this meansthat we must set cj+1,0 < cj0 because otherwise the larger modelwill always have the higher selection score. This restriction canbe sharpened to cj+1,0 < (k − j − 1)/(k − j) × cj0. To see this,note that RSS[Mj+1/Mj] = ∑k
i=j+2 β2(i)/(β
2( j+1) +
∑ki=j+2 β2
(i)).Because β2
( j+1) must be at least as large as each term in∑ki=j+2 β2
(i), it follows that β2( j+1) ≥ ∑k
i=j+2 β2(i)/(k − j − 1).
Thus RSS[Mj+1/Mj] ≤ (k − j − 1)/(k − j), which leads tothe stated restriction. For most practical APC procedures, thecj0’s will decrease much more quickly than implied by this re-striction. The greater the decrease in the cj0’s, the greater thepenalty being imposed on larger models and thus the greaterthe tendency to select smaller models.
In this article we present two strategies for setting the cj0’s.Both strategies are based on limiting the rate at which inactiveeffects are declared active. The first strategy limits the exper-imentwise error rate (EER), defined as the probability that atleast one inactive contrast is declared active. The second strat-egy limits the individual error rate (IER), defined as the ex-pected proportion of inactive effects that are declared active.For any model selection procedure, the EER and IER dependon the true model. Specifically, they depend on the number ofactive effects t and, if t > 0, on µ∗ and on ν. In practical terms,this means that they depend on the number and sizes of the ac-tive effects and on the error variance. The EER and the IERunder the null model are denoted by EER0 and IER0, and fort �= 0, they are denoted by EERt(µ
∗, ν) and IERt(µ∗, ν).
Consider the EER-control strategy and note that a discussionof the IER-control strategy would be analogous. For a speci-fied m and target experimentwise error rate, EERT , there arean unlimited number of sets of constants {c10, . . . , cm0} thatwould result in EER0 = EERT . Thus the opportunity exists forchoosing among these sets to specify the EER when t �= 0.The constants presented in this article were generated so thatsup[EERt(µ
∗, ν)] ≤ EERT for all t (where the supremum isover µ∗ ∈ ϒt and 0 < ν < ∞), and the equality holds for t = 1to m − 1. This choice should maximize the ability to identifyactive effects while maintaining the target error rate across allpossible true model scenarios. Note that in the statistical litera-ture involving testing multiple hypotheses, this would be called“strong control” of the EER. The constants needed to apply theAPC approach to 8-run, 12-run, and 16-run experiments aregiven in Tables 3–5. The algorithm used to generate the setsof constants is described in the Appendix, along with a justi-fication of why it produces constants that satisfy the foregoingclaims.
To use the APC model-selection procedure, the practitionermust (1) decide whether to control the EER or the IER, (2) se-lect a value of m, and (3) specify the level of EERT or IERT .
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 55
Table 3. APC Constants for the 8-Run Orthogonal Experiments
The choice between controlling the EER or the IER is largelya matter of the user’s preference for a particular application.Philosophically, the difference between the two approaches isin the way that multiple errors are treated. The EER imposes noadditional penalty if more than one error is made, whereas theIER weights the penalty by the number of errors made. Thepractical consequences of this is that an EER-control proce-dure will tend to favor larger models than a roughly compa-rable IER procedure will. For example, compare the constantsin Table 3 for the m = 5 EERT = .35 procedure with those forthe m = 5 IERT = .100 procedure. The value of c10 is slightlyhigher for the IER-control procedure, but the values of c20,c30, c40, and c50 are all higher for the EER-control procedure.Thus if these two procedures are applied to the same dataset,then the EER-control procedure will have a slightly lower se-lection score for M1 but higher selection scores for M2–M5.Next, consider setting the maximum model size m. The idealchoice would have m equal to the number of active effects t,because this would maximize the probability of selecting thecorrect model. But of course, the practitioner does not havethis information. If m is set less than t, then there is a markedloss of power, because at best m of the t active effects willbe correctly identified; however, the procedure often will iden-tify a few of the largest active effects provided that these aresufficiently larger than the remaining active effects. Setting mgreater than t also results in a loss of power, but the impact ismuch smaller. (The next section presents results from a simula-tion study that in part considered this issue). Thus in setting m,it is advisable to err on the high side, and it is recommended thatm be set such that the practitioner is quite confident that m ≥ t.
Finally, consider selecting the error rate. Increasing the value ofEERT or IERT will, besides increasing the error rate, also de-crease the possibility of missing active effects. Thus this choiceshould reflect the practitioner’s judgment as to the relative costof erroneously selecting inactive effects compared with that ofmissing active ones. For screening applications, missing activeeffects is usually considered the more serious error, and thus er-ror rates are often set relatively high; values of EERT ≥ .20 arecommon.
4. EVALUATING THE PROPERTIES OFSELECTION PROCEDURES
This section compares the performances of the APC pro-cedure and Lenth’s method. Lenth’s method was chosen as abenchmark because it appears to be the most popular alterna-tive to normal plots. Note that the results of the simulation studyof Hamada and Balakrishnan (1998) indicated that there werea number of procedures, including Lenth’s method, that per-formed well overall, and that there was little to choose betweenthe members of this group. The section begins by describing aframework for comparing model selection procedures. The geo-metric interpretation of model selection introduced in Section 2is then used to gain insights into how to best implement some ofthe components of this framework. A suitable method of com-parison is identified, and the results of a simulation study thatcompares the APC procedure to Lenth’s method are presented.
4.1 Comparing Model Selection Procedures
Devising a fair method of comparing different model selec-tion procedures is a challenging exercise. (See Hamada andBalakrishnan 1998 for an informative discussion of the issuesinvolved.) A common approach is to select versions of the pro-cedures that have the same error rate under the null model andthen compare the ability of the procedures to correctly identifythe active contrasts over a range of true model scenarios. Thusthree key components must be specified: (a) the version of eachmodel selection procedure, (b) the set of true model scenarios,and (c) the measure of performance used to evaluate the proce-dures for each true model scenario.
In selecting the versions of the procedures, the aim is to maketheir error rates as similar as possible. Because the observederror rate for each procedure will depend on the true model(i.e., on t, µ∗, and ν), in practice it is impossible to get anexact match over all true models. Most procedures have beendesigned to allow the practitioner to specify a nominal valueof either the EER or the IER. Usually, this nominal rate is ex-act under the null model and represents an upper bound undernonnull models. Thus studies typically compare procedures thathave the same nominal value of either the EER or the IER. Fol-lowing this strategy, in this article we compare procedures for12-run experiments that have EER0 = .20. For Lenth’s method,this corresponds to using a critical value of 2.74 (see Wu andHamada 2000, p. 620), which henceforth is referred to as LEN.Four versions of APC are considered. The first two of these,denoted by APC1 and APC2, are taken from Table 4 and cor-respond to m = 6 and m = 7 for EERT = .20. Note that APC1and APC2 have sup[EERt(µ
∗, ν)] = .20 for t = 0, . . . ,m − 1where the supremum is taken over µ∗ ∈ ϒt and ν ∈ (0,∞).
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
56 ARDEN MILLER
Table 4. APC Constants for the 12-Run Orthogonal Experiments
Henceforth this supremum is denoted by EERst . These meth-
ods were deliberately constructed in this manner to make thelikelihood of identifying the active effects as great as possiblewhile maintaining the selected error rate over all possible truemodels. Lenth’s method, on the other hand, does not maintainEERs
t = .20 for t = 0, . . . ,m − 1 but has decreasing values ofEERs
t as t increases. This is a consequence of Lenth’s methodbeing constructed to maintain a constant IER rather than a con-stant EER. Table 6 gives values of EERs
t and IERst for LEN,
APC1, and APC2. These values were estimated via simula-tion using the approach described by Miller (2004). Clearly,LEN has lower IERs
t for all t and lower EERst for all t �= 0.
Thus LEN is at a disadvantage in comparison with APC1 andAPC2. Two other versions of APC that were devised to pro-vide a fairer comparison with LEN are considered as well. Thefirst of these, APC3, was motivated by geometric considera-tions. The constants for APC3 were selected such that the areasof its selection regions would match those of LEN exactly. Re-call that under the null model, Y∗ has a uniform distributionover Sk, and thus the probability of any model being selected
is proportional to the area of its selection region. Thus APC3has exactly the same properties as LEN under the null model;in particular, both EER0 and IER0 are the same for both pro-cedures. Note that if the selection procedures being comparedhave different areas for a particular selection region, then theone with the larger area has an advantage for detecting that truemodel. In this respect, APC3 and LEN are on exactly the samefooting with respect to identifying any true model. However,APC3 does have somewhat higher values of EERs
t for t = 1, 2,and 3 and of IERs
t for t = 1 and 2, as shown in Table 6. ThusLEN might still be considered to be at a disadvantage relativeto APC3. The fourth version of APC was devised to match theEER of LEN as closely as possible. In particular, constants werechosen such that EERs
t would be the same for APC4 as it is forLEN for all t ≤ 7. This was achieved by making an obvious ad-justment to the algorithm described in the Appendix. Table 6indicates that not only does APC4 have the same values of theEERs
t as LEN, but it also has very similar values of IERst .
Next, consider the problem of specifying a set of true modelscenarios. Without loss of generality, the error variance can
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 57
Table 5. APC Constants for the 16-Run Orthogonal Experiments
be set to 1, and this task can be divided into three parts: set-ting the number of active contrasts, setting the relative mag-nitudes of the effects (when there is more than one activecontrast), and setting the overall magnitude of the effects. Interms of the geometric framework, the last two parts corre-spond to specifying µ∗ and ν. For this article, true modelsof size t = 1,2, . . . ,8 were considered. For each t, the truemodel was assumed to be µ = β01 + β1X1 + · · ·+ βtXt , where
β1 > β2 > · · · > βt > 0 and each Xi was assumed to have unitlength. Note that this does not sacrifice generality, because bothselection procedures are unaffected by changes in the signs ofthe effects and to changes in labeling. Using results from Sec-
tion 2.1, µ∗ = (β1X1 + · · · + βtXt)/√
β21 + · · · + β2
t , and thedomain of µ∗ for this model, ϒt , is a t-dimensional unit sphereembedded on Sk. For each t ≥ 2, a set Ut of 20–30 t-tupleswas used as the µ∗’s for the true model. The goal was to se-
Table 6. Supremums of Error Rates for Procedures Used in the Simulation Study
lect this set so that it would be evenly spread over the region ofϒt specified by β1 > β2 > · · · > βt > 0. This was achieved us-ing number-theoretic nets (NT nets) on the t-dimensional unitsphere. Such an NT net is a set of t-tuples spread evenly over thesurface of the unit sphere. The algorithm used to produce theNT net was taken from Fang and Wang (1994, pp. 166–170).Figure 1 illustrates the points in U2 and U3. In Figure 1(a) theunit circle represents ϒ2, and the subregion corresponding toβ1 ≥ β2 ≥ 0 is shaded gray. In Figure 1(b) the unit sphere repre-sents ϒ3, and the subregion corresponding to β1 ≥ β2 ≥ β3 ≥ 0
is shaded gray. A set of values for ν =√
β21 + · · · + β2
t was thenidentified, and simulations were done using each combinationof µ∗ and ν.
The final component of specifying a model selection proce-dure is to identify the measure of performance used to evalu-ate the procedures. In this article we use power (defined as theexpected proportion of active contrasts that are correctly identi-fied) averaged over the elements of Ut. That is, for each setting
of√
β21 + · · · + β2
t , the power was estimated via simulation foreach element of Ut , and the mean of these was recorded. Thesimulation study was devised such that each estimate of averagepower would have a standard error of ≤.0005. For t = 1, 1 mil-lion Monte Carlo simulations of Y were done for each valueof ν. The simulations were divided into 100 groups of 10,000runs each, and the between-group standard error was used tocalculate the standard error for the overall power estimate. Fort ≥ 2, 250,000 simulations of Y were done for each µ∗ ∈ Ut ateach value of ν. Because there were at least 20 elements in eachUt , this means that a total of at least 5 million simulated realiza-tions of Y were used. Standard errors were estimated using the
method described for t = 1. In all cases, all five selection proce-dures were applied to the same set of Y’s, which means that thestandard errors for the differences between estimated means aresmaller than would be calculated if independent estimates wereassumed. For comparisons of any of the APC procedures withLEN, the standard error of the difference is ≤.0004 and is evensmaller for comparisons of pairs of APC procedures. (For com-parisons of APC1 with APC2, it is ≤.0002.)
4.2 Results of the Simulation Study
Figure 2 compares LEN with APC1 and APC2. The plots
present average power versus ν =√
β21 + · · · + β2
t for t =1, . . . ,8. Note that for t = 1, . . . ,5, the lines for APC1 andAPC2 appear superimposed, but in fact APC1 is slightly aboveAPC2 in each case. The plots for each run size reveal thatfor most values of t in both APC1 and APC2 have higherpower than in LEN over a sizable range of ν. This differencein power is often nonnegligible; as a point of reference, fort = 1, the greatest difference in power is approximately .08 be-tween APC1 (or APC2) and LEN. Thus, overall, both APC1and APC2 are more effective at identifying active contrasts thanLEN. At least part of the difference in performance can be at-tributed to APC1 and APC2 having larger error rates than LENfor most values of t > 0, as summarized in Table 6. For a practi-tioner who wants to control the EER, it is an unfortunate prop-erty of LEN that the value of EERs
t decreases as t increases,because this must have a negative impact on power. One of theadvantages of the APC approach is its great flexibility, and thusit was possible to devise versions that maintained EERs
t at themaximum allowable level for t = 1, . . . ,m − 1. From a practi-cal standpoint, it seems sensible to assume that an experimenterwho willing to accept a given error rate under the null modelwould also be willing to accept the same error rate under non-null models. If this is the case, then the results of this simulationstudy indicate APC1 and APC2 are superior to LEN in terms ofthe average power criteria.
Figure 3 compares APC3 and APC4 with LEN. Note thatneither APC3 nor APC4 is being proposed as an alternative toLenth’s method. Rather, they were devised solely to provide
(a) (b)
Figure 1. The Sets of Representative Points for t = 2 and t = 3: (a) U2 and (b) U3.
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 59
Figure 2. Power Curves for LEN (—), APC1 (- - - -), and APC2 (· · · ·).
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
60 ARDEN MILLER
Figure 3. Power Curves for LEN (—), APC3 (- - - -), and APC4 (· · · ·).
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 61
insight into how efficiently the APC approach partitions Sk intoselection regions relative to LEN. Recall that APC3 was de-vised to partition S11 in such a manner that each selection re-gion had exactly the same area as the corresponding region forLEN. The plots in Figure 3 indicate that in almost all situationsAPC3 performs as well as or better than LEN with respect toaverage power. However, this needs to be weighed against thefact that APC3 does have somewhat larger values of EERs
t fort = 1, 2, and 3, as was noted previously. APC4 was devised tohave values of EERs
t that match up as closely as possible withLEN for all t. Figure 3 indicates that APC4 has a clear advan-tage over LEN in terms of average power for t = 1,2,3,4. Fort = 5,6,7,8, LEN has slightly higher power for lower valuesof ν, but APC4 gains clear superiority as ν increases. Overall,it is reasonable to conclude that APC4 compares favorably withLEN.
Finally, note that Figure 2 can be used to demonstrate the ef-fect of using a larger value of m for the APC approach by com-paring APC1 (m = 6) with APC2 (m = 7). For t = 1, . . . ,6, thepower for APC1 is slightly higher than that for APC2. Giventhe scale of the plots in Figure 2, this is difficult to see, butthe raw data indicate a small consistent advantage for APC1 ineach case. The difference is very slight for t = 1 (the biggestestimated difference in power was .0008) but increases as t in-creases (for t = 6, the biggest estimated difference in powerwas .007). For t = 7, APC2 has a clear advantage over APC1,as would be expected because the best APC1 can do in this caseis identify six of the seven active effects. For t = 8, APC2 hasa clear advantage over APC1, but neither performs particularlywell.
5. CONCLUDING REMARKS
The APC approach presented in this article encompasses anextensive range of selection procedures, because any choice ofc10, . . . , cm0 that conforms to the restrictions discussed in Sec-tion 2 will produce a workable procedure. The two versions ofAPC, EER control and IER control (developed in Sec. 3), areproposed as “general purpose” procedures. It is certainly feasi-ble that other versions of APC could be developed to achieveother objectives. Two possibilities would be to control an alter-native definition of error rate at a specified level or to ensurethat a defined measure of “success rate” is achieved for identi-fying active effects that are of at least a certain magnitude (mea-sured in σ ’s). Thus opportunities for further research involvingthe APC procedure exist. However, to implement any new ver-sion of APC, two main obstacles must be overcome: (1) Theobjective must be defined in a way that is both achievable anduniquely specifies an APC procedure, and (2) a method for find-ing c10, . . . , cm0 must be devised. Both of these tasks can bequite challenging. Consider the proposed EER-control versionof APC as an example. The key to implementing this versionwas establishing that it was possible to identify c10, . . . , cm0such that limν→∞ EERt(µ
∗, ν) = EERT for t = 0, . . . ,m − 1and that such a procedure would provide strong control of theEER (see the App.). It was then necessary to conduct a ratherinvolved set of simulations to estimate the constants.
The main motivation for using an APC procedure is that itproduces selection regions that are likelihood ratio compliant.
The underlying idea is simply that the selection region for eachcandidate model should contain points for which f (y∗) is largeunder that model. It seems reasonable to conjecture that a pro-cedure with this characteristic should perform well with respectto performance measures such as power. This intuition is sup-ported by the results of the simulation study presented in Sec-tion 4. A further argument that can be made in favor of APC isthat likelihood ratio–compliant selection regions ensure a cer-tain type of consistency in the model selection process. Con-sider the selection regions for models Mj and Mj† , where j† > j.Because Mj is a submodel of Mj† an F-test can be used to as-sess the evidence in favor of adding the additional variables. Forlikelihood ratio–compliant selection regions, such a test wouldprovide stronger evidence in favor of adding the extra variablesfor any point in the Mj† selection region than it would for anypoint in the Mj selection. For a procedure that is not likelihoodratio compliant, this would not be the case for at least someMj and Mj† .
The APC procedure proposed in this article is not unique inits use of likelihood ratio tests as the basis for a model selectionprocedure for unreplicated factorial experiments. Notably, thechain pooling approach of Holms and Berrettoni (1969) and therelated procedures developed by Venter and Steel (1996, 1998),Langsrud and Næs (1998), and Al-Shiha and Yang (1999, 2000)are based on using sequences of hypothesis tests that are equiv-alent to likelihood ratio tests. What sets the APC approachapart is that it is only method that considers all pairwise com-parisons between candidate models. All of the other methodsare stepwise approaches and as a result do not produce likeli-hood ratio–compliant selection regions. The importance of thiscan be illustrated using the filtration rate data from Section 2.A half-normal plot of the estimated effects, given in Figure 4,suggests that the most plausible models are the null model andthe best three-contrast model. Note that most stepwise pro-cedures would never actually compare these two models. Forexample, a typical stepdown procedure would start with thenull model and test adding contrasts one at a time. The firststep compares the null model to the best one-contrast modeland stops if the null model is preferred. In this case, the bestthree-contrast model is never tested, even though it appearsto be a stronger competitor than the best one-contrast model.A similar problem occurs for a stepup procedure that eliminates
Figure 4. Half-Normal Plot for the Filtration Data.
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
62 ARDEN MILLER
contrasts one at a time. Assume that such a procedure has elim-inated all contrasts except X1, X3, and X5. Then the next stepwould compare the best three-contrast model to the best two-contrast model. If the procedure stops here, then the best three-contrast model is selected without being evaluated against thenull model. The geometric interpretation of this situation is in-teresting. This example represents a situation where, for manyselection procedures, y∗ is close to the boundary between theselection region for the null model and the selection region forthe {X1, X3, X5} model; a small change in y∗ may cause theprocedure to switch from selecting the null model to selectingthe {X1, X3, X5} model or vice versa. The APC procedure en-sures that this boundary has been in effect defined by a likeli-hood ratio test of the null model with the {X1, X3, X5} model.Stepwise procedures that never directly compare the null modelto the {X1, X3, X5} model must establish this boundary usingsome other test.
APPENDIX: ALGORITHM USED TO GENERATEAPC CONSTANTS
The algorithm used to produce the constants in Tables 3–5is based on the premise that identifying constants such thatlimν→∞ EERt(µ
∗, ν) = EERT [or limν→∞ IERt(µ∗, ν)
= IERT ] for t = 0, . . . ,m − 1 will produce a procedure thathas EERt(µ
∗, ν) ≤ EERT [or IERt(µ∗, ν) ≤ IERT ] for all pos-
sible values of t, µ∗, and ν. Here we first give a justification ofthis assertion, and then present the algorithm.
To avoid confusion, keep in mind that k represents the totalnumber of effects, t represents the true number of active ef-fects, and m represents the maximum number of effects thatcan be declared active by the procedure under investigation.Miller (2004) presented results concerning EERt(µ
∗, ν) andIERt(µ
∗, ν) for model selection procedures. The following re-sults are relevant to this discussion:
A1. For any selection procedure and any t, limν→0 EERt(µ∗,
ν) and limν→0 IERt(µ∗, ν) are constant over µ∗ ∈ ϒt .
Further,
limν→0
EERt(µ∗, ν) < EER0 and
limν→0
IERt(µ∗, ν) = IER0
for t = 1, . . . , k−1. Note that for t = k, the definitions ofEER and IER imply that EERk(µ
∗, ν) and IERk(µ∗, ν)
are 0 for all possible µ∗ and ν.A2. For many selection procedures, limν→∞ EERt(µ
∗, ν)
and limν→∞ IERt(µ∗, ν) are constant over µ∗ ∈ ϒt for
all t.A3. For many selection procedures, sup EERt(µ
∗, ν) coin-cides with either
limν→0
EERt(µ∗, ν) or lim
ν→∞ EERt(µ∗, ν),
and the analogous result holds for the IER.
For the sake of brevity, the following discussion focuses onthe EER case; a discussion of the IER case would be verysimilar. The algorithm was created assuming that A2 and A3are valid for the APC procedure. (Justifications of these as-sumptions are given later.) For each target error rate EERT ,
a set of constants was found such that EER0 = EERT andlimν→∞ EERt(µ
∗, ν) = EERT for t = 1, . . . ,m − 1.Given A1–A3, this ensures that sup[EERt(µ
∗, ν)] =limν→∞ EERt(µ
∗, ν) = EERT for t = 0, . . . ,m − 1. For t ≥ m,note that as ν ≡ ‖µ − µ‖/σ , letting ν → ∞ with µ∗ fixedis equivalent to fixing β1, . . . , βk and letting σ → 0. Clearly,βj → βj as σ → 0, and thus for large enough ν, the t largestsquared estimated effects correspond to the active effects. Be-cause t ≥ m, each of the Mj’s (the best model of size j) un-der consideration will contain only active effects, and thusno inactive effects will be declared active. Therefore, fort > m, limν→∞ EERt(µ
∗, ν) = 0 and sup[EERt(µ∗, ν)] =
limν→0 EERt(µ∗, ν) < EERT .
As mentioned earlier, the algorithm depends on A2 and A3being valid for the APC procedure. First consider A2. We ar-gued previously that for any fixed µ∗, if ν is large enough, thenthe t largest squared estimated effects correspond to the activeeffects. As a result, the best model of size j will contain thej largest active effects if j ≤ t and will contain all t active effectsplus j − t inactive effects if j > t. For t ≥ m, it follows that eachof the Mj’s under consideration will contain only active effects,and thus no inactive effects will be declared active. Therefore, itcan be concluded that limν→∞ EERt(µ
∗, ν) = 0 for t ≥ m. Fort < m, it is necessary to consider applying the algorithm in Sec-tion 2.3. From RSS[Mj/M0] = ∑k
i=j+1 β2(i)/
∑ki=1 β2
i (step 3),it can be seen that RSS[Mj/M0] → 0 as σ → 0 (ν → ∞)if j ≥ t but converges to a positive value, say qj, otherwise.Thus ss[Mj] → ∞ for j ≥ t and ss[Mj] → cj0/qj otherwise.It follows that for large enough ν, the selection scores of Mt ,Mt+1, . . . ,Mm will be larger than those of M0, M1, . . . ,Mt−1.Therefore, the selected model must be one of the modelsthat contains all of the active effects. Because the values ofRSS[Mj/M0] for these models depend only on the values of theβj’s for the inactive effects, limν→∞ EERt(µ
∗, ν) will be con-stant across µ∗ ∈ ϒt and limν→∞ EERt(µ
∗, ν) depends only onthe probabilities of selecting models Mt , Mt+1, . . . ,Mm. Nowconsider A3. The conjecture in A3 is based on our experience.It appears to be true for most selection procedures, but we arenot aware of a general method of verifying that it is indeed truefor a particular procedure. An extensive set of simulations wasused to assess the conjecture’s validity for the constants in Ta-bles 3–5. A total of 18 sets of constants were selected from thetables. For each set, t was varied from 1 to m − 1, and five re-alizations of µ∗ ∈ ϒt were chosen for each t. Then the EER orIER (depending on which error rate was being controlled) wasestimated via simulation for various values of ν. In every case,the results were consistent with the supremum coinciding witheither the limit as ν → 0 or the limit as ν → ∞. On the basisof these simulations, we are very confident that the constants inTables 3–5 provide the stated levels of error control.
The following algorithm is used to produce the constants:
Step 1. Set t = m − 1. Note that only models Mm−1 andMm need be considered, and that to get limν→∞ EERt(µ
∗, ν) =EERT , it is necessary to find cm−1,m such that Pr(RSS[Mm−1/
Mm] < cm−1,m) = 1 − EERT . Values of RSS[Mm−1/Mm] canbe simulated by generating sets of k − m + 1 pseudorandomnumbers from a N(0, 1) distribution and taking the sum of thesquared values divided by the sum of the k−m smallest squaredvalues. The value of cm−1,m can be set to the 1 − EERT per-centile of these values.
TECHNOMETRICS, FEBRUARY 2005, VOL. 47, NO. 1
Dow
nloa
ded
by [
Uni
vers
ity o
f C
hica
go L
ibra
ry]
at 1
2:03
19
Aug
ust 2
013
UNREPLICATED FACTORIAL EXPERIMENTS 63
Step 2. Set t = m − 2. For this step, only models Mm−2,Mm−1, and Mm need be considered. To get limν→∞ EERt(µ
∗,ν) = EERT , we must have the probability of RSS[Mm−2/
Mm−1] < cm−2,m−1 and RSS[Mm−2/Mm] < cm−2,m both occur-ring equal to 1 − EERT . Because cm−2,m = cm−2,m−1 × cm−1,m
and cm−1,m has been determined in Step 1, this means identi-fying the value of cm−2,m−1. Sets of k − m + 2N(0,1) pseudo-random numbers can be generated and used to find values ofRSS[Mm−2], RSS[Mm−1], and RSS[Mm] by taking the sum ofthe squared values, the sum of the k − m + 1 smallest squaredvalues, and the sum of the k − m smallest squared values. Thevalue of max(RSS[Mm−2/Mm−1],RSS[Mm−2/Mm]/cm−1,m)
can then be found for each set and cm−2,m−1 set to the 1−EERT
percentile of these values.Steps 3 through m. Each subsequent step represents the ob-
vious extension of the previous step. Thus values of cm−3,m−2,cm−4,m−3, . . . , c0,1 are generated in that order.
Step m + 1. Values for c10, c20, . . . , cm0 are then found usingthe relationships c10 = c−1
0,1, c20 = (c01 × c12)−1, and so on.
[Received June 2002. Revised July 2004.]
REFERENCES
Aboukalam, M. A. F., and Al-Shiha, A. A. (2001), “A Robust Analysis for Un-replicated Factorial Experiments,” Computational Statistics and Data Analy-sis, 36, 31–46.
Al-Shiha, A. A., and Yang, S. (1999), “A Multistage Procedure for AnalyzingUnreplicated Factorial Experiments,” Biometrical Journal, 41, 659–670.
(2000), “Critical Values and Some Properties of a New Test Statisticfor Analyzing Unreplicated Factorial Experiments,” Biometrical Journal, 42,605–616.
Birnbaum, A. (1959), “On the Analysis of Fractional Factorial ExperimentsWithout Replication,” Technometrics, 1, 343–357.
Benski, H. C. (1989), “Use of a Normality Test to Identify Significant Effectsin Factorial Designs,” Journal of Quality Technology, 21, 174–178.
Berk, K. N., and Picard, R. R. (1991), “Significance Tests for Saturated Orthog-onal Arrays,” Journal of Quality Technology, 23, 79–89.
Box, G. E. P., Hunter, W. G., and Hunter, J. S. (1978), Statistics for Experi-menters, New York: Wiley.
Box, G. E. P., and Meyer, R. D. (1986), “An Analysis for Unreplicated Frac-tional Factorials,” Technometrics, 28, 11–18.
Chipman, H. (1996), “Bayesian Variable Selection With Related Predictors,”Canadian Journal of Statistics, 24, 17–36.
Daniel, C. (1959), “Use of Half-Normal Plots in Interpreting Factorial Two-Level Experiments,” Technometrics, 1, 311–341.
Dong, F. (1993), “On the Identification of Active Contrasts in UnreplicatedFractional Factorials,” Statistica Sinica, 3, 209–217.
Fang, K.-T., and Wang, Y. (1994), Number-Theoretic Methods in Statistics,London: Chapman & Hall.
Haaland, P. D., and O’Connell, M. A. (1995), “Inference for Effect-SaturatedFractional Factorials,” Technometrics, 37, 82–93.
Hamada, M., and Balakrishnan, N. (1998), “Analyzing Unreplicated FactorialExperiments: A Review With Some New Proposals,” Statistica Sinica, 8,1–41.
Holms, A. G., and Berrettoni, J. N. (1969), “Chain-Pooling ANOVA for Two-Level Factorial Replication-Free Experiments,” Technometrics, 11, 725–746.
Langsrud, O., and Næs, T. (1998), “A Unified Framework for Significance Test-ing in Fractional Factorials,” Computational Statistics and Data Analysis, 28,413–431.
Le, N. D., and Zamar, R. H. (1992), “A Global Test for Effects in 2k FactorialDesign Without Replicates,” Journal of Statistical Computation and Simula-tion, 41, 41–54.
Lenth, R. V. (1989), “Quick and Easy Analysis of Unreplicated Factorials,”Technometrics, 31, 469–473.
Miller, A. E. (2003), “The Analysis of Unreplicated Factorial ExperimentsFrom a Geometric Perspective,” The Canadian Journal of Statistics, 31,311–327.
(2004), “Limits of Error Rates for the Analysis of Unreplicated Ex-periments,” Communications in Statistics, Part A—Theory and Methods, 33,1587–1606.
Saville, D. J., and Wood, G. R. (1991), Statistical Methods: The GeometricApproach, New York: Springer-Verlag.
(1996), Statistical Methods: A Geometric Primer, New York: Springer-Verlag.
Small, C. G. (1996), The Statistical Theory of Shape, New York: Springer-Verlag.
Venter, J. H., and Steel, S. J. (1996), “A Hypothesis-Testing Approach TowardIdentifying Active Contrasts,” Technometrics, 38, 161–169.
(1998), “Identifying Active Contrasts by Stepwise Testing,” Techno-metrics, 40, 304–313.
Voss, D. T. (1988), “Generalized Modulus-Ratio Tests for Analysis of Facto-rial Designs With Zero Degrees of Freedom for Error,” Communications inStatistics, Part A—Theory and Methods, 17, 3345–3359.
Ye, K. Q., Hamada, M., and Wu, C. F. J. (2001), “A Step-Down Method forAnalyzing Unreplicated Factorial Designs,” Journal of Quality Technology,33, 140–152.
Wu, C. F. J., and Hamada, M. (2000), Experiments Planning, Analysis, andParametric Design Optimization, New York: Wiley.
