Contents lists available at SciVerse ScienceDirect
Journal of Statistical Planning and Inference
Journal of Statistical Planning and Inference 142 (2012) 878–895
0378-37
doi:10.1
n Corr
E-m
journal homepage: www.elsevier.com/locate/jspi
A fiducial approach to multiple comparisons
Damian V. Wandler a,n, Jan Hannig b
a Department of Statistics, Colorado State University, Fort Collins, CO 80523, United Statesb Department of Statistics and Operations Research, The University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States
a r t i c l e i n f o
Article history:
Received 16 May 2010
Received in revised form
26 May 2011
Accepted 21 October 2011Available online 6 November 2011
Keywords:
Fiducial inference
Multiple comparisons
Importance sampling
Fiducial probability
Model selection
58/$ - see front matter & 2011 Elsevier B.V. A
016/j.jspi.2011.10.011
esponding author.
ail address: [email protected] (D.V. Wan
a b s t r a c t
Comparing treatment means from populations that follow independent normal dis-
tributions is a common statistical problem. Many frequentist solutions exist to test for
significant differences amongst the treatment means. A different approach would be to
determine how likely it is that particular means are grouped as equal. We developed a
fiducial framework for this situation. Our method provides fiducial probabilities that any
number of means are equal based on the data and the assumed normal distributions.
This methodology was developed when there is constant and non-constant variance
across populations. Simulations suggest that our method selects the correct grouping of
means at a relatively high rate for small sample sizes and asymptotic calculations
demonstrate good properties. Additionally, we have demonstrated the flexibility in the
methods ability to calculate the fiducial probability for any number of equal means. This
was done by analyzing a simulated data set and a data set measuring the nitrogen levels
of red clover plants that were inoculated with different treatments.
& 2011 Elsevier B.V. All rights reserved.
1. Introduction
Treatment means are commonly compared to each other to determine their relationship. A variety of problemscompare treatment means. For example, comparing the effectiveness of multiple drugs in a pharmaceutical setting is acommon practice. Other areas of application include agriculture, finance, production industries, etc.
In this scenario there are observations Xi ¼ ðXi1, . . .Xini Þ for populations i¼ 1, . . . ,k. The k populations followindependent normal distributions with means l¼ ðm1, . . . ,mkÞ
T and variance g. The multiple comparison problem (MCP)attempts to perform inference on the groupings of the individual means within l from the observations X1,X2, . . . ,Xk.
There are several frequentist solutions for multiple comparison problems. Using frequentist methods, analysis ofvariance (ANOVA) is used to test for a significant treatment effect. There are several tests for differences amongtreatments. Some are, Fisher’s least significant difference (LSD), Tukey’s honest significant difference (HSD), Sheffe’spairwise differences, Duncan’s multiple range test, etc. These solutions control the comparisonwise or experimentwiseerror rate for some a. However, these solutions cannot determine a likelihood that particular means are equal or unequal.
A Bayesian procedure for MCP has been developed in Gopalan and Berry (1998). This method uses a Dirichlet processprior to decide between competing groupings of l. The final posterior probabilities are used to discern amongst thegroupings for different priors.
We have developed methodology for this scenario using an extension of R.A. Fisher’s fiducial inference. We usegeneralized fiducial inference as developed in Hannig (2009b) to determine the likelihood of grouping particular means as
ll rights reserved.
dler).
www.elsevier.com/locate/jspiwww.elsevier.com/locate/jspidx.doi.org/10.1016/j.jspi.2011.10.011mailto:[email protected]/10.1016/j.jspi.2011.10.011
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 879
equal or unequal. A model selection technique was used to determine, based on the data, the likely model(s). This isdeveloped for g¼ ðZ, . . . ,ZÞ (constant variance) and g¼ ðZ1,Z2, . . . ,ZkÞ (non-constant variance). Simulation results suggestthat our method selects the correct grouping at a high rate for small sample sizes. We have also proven that our methodwill asymptotically select the correct grouping of means.
In addition to simulation results and theoretical calculations, we analyzed a simulated data set and a data setmeasuring nitrogen levels of red clover plants that were inoculated with different treatments. The analyses wereconducted assuming both constant and non-constant variance; the results from the red clover data set were comparedwith those of the Bayesian method (which assumes constant variance). Both the fiducial and Bayesian methods producesomething of a posterior probability for each possible grouping.
2. Generalized fiducial inference
2.1. Overview
Fisher (1930) did not support the Bayesian idea of assuming a prior distribution on the parameters when there islimited information available. As a result, he developed fiducial inference to offset this perceived shortcoming. Fiducialinference did not garner approval when some of Fisher’s claims were found to be untrue in Lindley (1958) and Zabell(1992). More recently, Weeranhandi (1993) has developed generalized inference and the work of Hannig et al. (2006)established a link between fiducial and generalized inference. Hannig (2009b) and references within provide a thoroughbackground on fiducial inference and its properties.
The principle idea of generalized fiducial inference is similar to the likelihood function and ‘‘switches’’ the role of thedata, X, and the parameter(s) x. To formally introduce fiducial inference we assume that a relationship, called the structuralequation, between the data, X, and the parameter(s), x, exists in the form
X¼ Gðx,UÞ, ð1Þ
where U is a random vector with a completely known distribution and independent of any parameters. After observing Xwe use the known distribution of U and the relationship from the structural equation to infer a distribution on x. Thisallows us to define a probability measure on the parameter space, X. If (1) can be inverted the inverse will be written asG�1ð�,�Þ. For an observed x and u we can calculate x from
x¼ G�1ðx,uÞ: ð2Þ
From this inverse relationship we can generate a random sample of u01,u02, . . . ,u
0M and obtain a random sample for
x : x01 ¼ G�1ðx,u01Þ,x02 ¼ G
�1ðx,u02Þ, . . .x0M ¼ G�1ðx,u0MÞ. This sample is called a fiducial sample and can be used to calculate
estimates and confidences intervals for the true parameter(s), x0.Hannig and Lee (2009) address two potential times that G�1ð�,�Þ may not exist. They are when (i) there is no x that
satisfies (2) or (ii) there is more than one x that satisfies (2). From Hannig (2009b) we will handle situation (i) byeliminating such u’s and re-normalizing the sampling probabilities. This is reasonable because we know our data wasgenerated using x0 and u0 so at least one solution for (2) exists. We will only consider the u’s that allow for G
�1ð�,�Þ to exist.Hannig (2009b) suggests that situation (ii) is handled by selecting an x by some, possibly random, rule that satisfies theinverse in (2).
A more rigorous definition of the inverse is the set valued function of
Q ðx,uÞ ¼ fx : x¼ Gðx,uÞg: ð3Þ
We know that our observed data was generated using some x0 and u0. We also know the distribution of U and thatQ ðx,u0Þa|. Coupling these facts we can compute the generalized fiducial distribution from
VðQ ðx,U%ÞÞ9fQ ðx,U%Þa|g, ð4Þ
where U% is an independent copy of U and V(S) is a random element for any measurable set, S, with support on the closureof S, S. Essentially, Vð�Þ is the random rule for picking the possible x’s. We will refer to a the random element that comesfrom (4) as Rx. For a more detailed discussion of the derivation of the generalized fiducial distribution see Hannig (2009b).
From the structural equation the generalized fiducial density is calculated as proposed in Hannig (2009b) and justifiedtheoretically in Hannig (2009a). Let G¼ ðg1, . . . ,gnÞ such that Xi ¼ giðx,UÞ for i¼ 1, . . . ,n. x is a p� 1 vector and denoteXi ¼ G0,iðx,UiÞ where Xi ¼ ðXi1 , . . . ,Xip Þ and Ui ¼ ðUi1 , . . . ,Uip Þ for all possible combinations of the indexes i¼ ði1, . . . ,ipÞ.Furthermore, assume that the functions G0,i are one-to-one and differentiable. Under some technical assumptions inHannig (2009a) this will produce the generalized fiducial density of
fRx ðxÞ ¼f Xðx9xÞJðx,xÞR
Xf Xðx9x0ÞJðx,x0Þ dx0
, ð5Þ
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895880
where
Jðx,xÞ ¼n
p
!�1 Xi ¼ ði1 ,...,ipÞ
det ddxG�10,i ðxi,xÞ
� �det ddxiG
�10,i ðxi,xÞ
� �������
������ ð6Þis the average of all subsets where 1r i1o � � �o iprn and the determinants in (6) are the appropriate Jacobians.
3. Main results
3.1. Structural equation with constant variance
In a multiple comparison problem we have k populations with means l¼ ðm1, . . . ,mkÞ. Data, which follows anindependent normal distribution, is of the form Xi ¼ ðXi1, . . .Xini Þ for all i¼ 1, . . . ,k where Xi is independent of Xj for all iand j. We are interested in the k treatment means. We would like to make some judgement on the equality or inequality ofthe means within competing models.
For example if Xi ¼ ðXi1, . . .Xini Þ is an independent random sample from a Nðmi,ZÞ distribution for i¼1,2 then theappropriate models would either assume m1 ¼ m2 or m1am2. The structural equations in this case could be
X1j ¼ ðm1þffiffiffiZp
Z1jÞIm1 ¼ m2þðm1þffiffiffiZp
Z1jÞIm1am2 ,
X2j ¼ ðm2þffiffiffiZp Z2jÞIm1 ¼ m2þðm2þ ffiffiffiZp Z2jÞIm1am2 ,
where Zij are independent random variables from the Nð0;1Þ distribution. From these structural equations the generalizedfiducial density in (5) can be calculated for each model (m1 ¼ m2 and m1am2Þ.
To simplify notation we will use J¼U19U29 . . . 9UtJ where Ui is a collection of indexes of the means that are equal. Themeans indexed by Ui and Uj separated by a vertical bar ‘‘9’’ are unequal. For example when k¼3, if J¼ 123 then U1 ¼ 123signifies m1 ¼ m2 ¼ m3 ¼ mn1. If J¼ 1 293 then U1 ¼ 1 2 and U2 ¼ 3 signify m1 ¼ m2 ¼ mn1 and m3 ¼ mn2 where mn1amn2. Note thatthere are ui equal means in group Ui, tJ total groupings in J, and the unique means are ðmn1,mn2, . . . ,mnt Þ.
In general, if Xi1, . . . ,Xini is an independent random sample from a Nðmi,ZÞ distribution for i¼ 1, . . . ,k then a structuralequation is
Xij ¼X
J2fJ1 ,...,JHgðmiþ
ffiffiffiZp
ZijÞIJ , ð7Þ
where IJ ¼ 1 if grouping J is selected and 0 otherwise, the equality of mi ¼ mj follow the grouping in J for all possiblegroupings fJ1, . . . ,JHg, and Zij are independent random variables from the Nð0;1Þ distribution.
As explained below, the fiducial distribution in (4) based on the structural equations above, will favor the model withthe most free means (all unequal means). To compensate for this we need to introduce additional structural equations thatare independent of those in (7). These structural equations will allow us to introduce a weight function that down-weightsthe models with many free means.
From Eq. (4) we can see that the generalized fiducial distribution is calculated by taking p (number of parameters)structural equations and conditioning on the fact that the remaining equations occurred. As a result, when there are moreparameters there are less equations that will be part of the conditioning or, equivalently, less conditions have to besatisfied. In this case we have N structural equations ðN¼
Pki ¼ 1 niÞ. If all of the means are different (J¼ 192939 . . . 9kÞ then
p¼ kþ1 (x¼ ðm1, . . . ,mk,ZÞ) and we condition on N�ðkþ1Þ events. If all of the means are equal ðJ¼ 123 . . . kÞ thenp¼ 2ðx¼ ðm,ZÞÞ and we condition on N�2 events. Clearly as more means are grouped together there are more conditionsthat need to be satisfied. In order to offset this unbalanced conditioning we will introduce additional structural equationsthat are independent of our original structural equations as proposed in Hannig and Lee (2009). These additional structuralequations will balance out the number of conditions that need to be met for each selected J.
As noted, adding additional structural equations allows us to down-weight the models with more free means toincrease the likelihood of grouping several means together. Additionally, we used the weight function introduced by theadditional structural equations to make the fiducial distribution more scale invariant. Attempting to make the methodscale invariant in this fashion is rather ad hoc but seemed to work well in simulations and we can show that our method isasymptotically scale invariant.
The additional structural equations are:
WMSXN
2p
!¼ biþPi if iZtJ ,
WMSXN
2p
!¼ Pi if iotJ , ð8Þ
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 881
where MSX ¼ k�1Pk
i ¼ 1 MSXi, MSXi is the maximum likelihood estimate of the variance for group i, Pi is an independent
w2ð1Þ random variable for all i, WðzÞ is the Lambert W function, and tJ is the number of groupings in a given J. Because ofthe independence these structural equations will not affect the distribution of X but they will affect the conditionaldistribution in (4). When inverting the structural equations in (8), if iZtJ we can choose a bi for any Pi so that the equationis satisfied. Thus, conditioning on this equation will not effect the conditional distribution. If iotJ then Pi ¼WððMSXNÞ=ð2pÞÞwhich creates an additional condition to be met. Combining the additional conditions with the original structural equationsthere will always be N�2 conditions regardless of the grouping of the means. This will define the weight function as
wJðxÞ ¼Yio tJ
f ðPiÞ ¼1
MSXN
� �ðtJ�1Þ=2,
where f is the density of the w2ð1Þ distribution.Using the original structural equations, combined with the additional structural equations, the generalized fiducial
distribution (4) has a density given by
f ðxÞpX
J2fJ1 ,...,JHg
~f JðxÞwJðxÞIJ ,
where ~f JðxÞ is the numerator in (5) and will be computed for all groupings. This numerator for a grouping, J, is
~f JðxÞ ¼Vx,JZ
1
ð2pÞn1=2Zn1=2exp � 1
2ZXn1j ¼ 1ðx1j�m1Þ
2
8<:
9=; � � � � 1ð2pÞnk=2Znk=2 exp � 12Z
Xnkj ¼ 1ðxkj�mkÞ
2
8<:
9=;
¼ Vx,JZ�N=2�1
ð2pÞN=2exp � 1
2ZXki ¼ 1
Xnij ¼ 1ðxij�miÞ
2
8<:
9=;¼ Vx,J Z
�N=2�1
ð2pÞN=2exp � 1
2ZXtJi ¼ 1
n0iðmn
i �x0iÞ
2
( )exp � 1
2ZXtJi ¼ 1
n0iMSX0i
( ), ð9Þ
where
JJðx,xÞ ¼ C�1N,J
PtJl ¼ 1
Pi1 ,i22Ul ,i1 o i2
Pji1
,ji19xi1 ,j1�xi2 ,j2 9
2Z , tJ ok,Pkl ¼ 1
Pj1 o ,j2 9xl,j1�xl,j2 9
2Z, tJ ¼ k,
8>>>>><>>>>>:
¼Vx,JZ ,
n0i ¼Xl2Ui
nl,x0i ¼
Pl2Ui
Pnlj ¼ 1 xlj
n0i,
MSX0i ¼P
l2UiPnl
j ¼ 1ðxlj�x0iÞ
2
n0i, N¼
Xki ¼ 1
ni
and CN,J is the number of Jacobian terms to average over.If we recognize that mni 9Z follows a normal distribution for all i and Z follows an inverse gamma distribution then we
can integrate ~f JðxÞ over the parameter space of each grouping XJ . Thus,
pJ ¼ZXJ
f JðxÞwJðxÞ dx¼Vx,JwJðxÞ2N=2ptJ=2G
N�tJ2
� �ð2pÞN=2ð
PtJi ¼ 1 n
0iMSX
0iÞðN�tJ Þ=2QtJ
i ¼ 1ffiffiffiffin0i
p :We can find the probability that any J is correctly grouping the means by
PðJÞ ¼pJP~J p~J
: ð10Þ
Clearly, when a particular J is correctly grouping the means we would like P(J) to be large.
3.2. Structural equation with non-constant variance
Similar to the previous setup, if Xi1, . . . ,Xini is an independent random sample from a Nðmi,ZiÞ distribution for i¼ 1, . . . ,kthen a structural equation is
Xij ¼X
J2fJ1 ,...,JHgðmiþ
ffiffiffiffiffiZi
pZijÞIJ
for groupings fJ1, . . . ,JHg where Zij are independent random variables from the Nð0;1Þ distribution.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895882
Like the previous section, the numerator of (5) is
~f JðxÞ ¼Vx,JQki ¼ 1 Zi
1
ð2pÞn1=2Zn1=21exp � 1
2Z1
Xn1j ¼ 1ðx1j�m1Þ
2
8<:
9=; � � � � 1ð2pÞnk=2Znk=2k exp �
1
2Zk
Xnkj ¼ 1ðxkj�mkÞ
2
8<:
9=;
¼ Vx,JwJðxÞQk
i ¼ 1 Z�ni=2�1i
ð2pÞN=2exp �1
2
Xki ¼ 1
niððmi�xiÞ
2þMSXiÞZi
( ), ð11Þ
where
JJðx,xÞ ¼ C�1N,J
Pkz ¼ 1
Pj1,z o j2,z rnz
Pði1 ...,itJ Þ
Pj ¼ ðj1,z ,j2,zÞ9T9
2kQk
i ¼ 1 Zi¼
Vx,JQki ¼ 1 Zi
,
il ¼ fi1, . . . ,iul�1g � Ul is the set of 1r i1o i2o � � �o iul�1rul, CN,J is the number Jacobian terms to average over,
T ¼YtJl ¼ 1
Yi2il
ðxi,j�mnl Þ
24
35ðxiul ,j1,z�xiul ,j2,z Þ,
xi ¼Pni
j ¼ 1 xij
niand MSXi ¼
Pnij ¼ 1ðxlj�xiÞ
2
ni:
As an example of the Jacobian, if J¼ 19293 then we average over
9ðx1,j1;1�x1,j2;1 Þðx2,j1;2�x2,j2;2 Þðx3,j1;3�x3,j2;3 Þ9
2kQk
i ¼ 1 Zi
for all j1,zo j2,zonz combinations (z¼ 1;2,3Þ. If J¼ 1293 then we average over
9ðx1,j1;1�mn
1Þðx2,j1;2�x2,j2;2 Þðx3,j1;3�x3,j2;3 Þ9
2kQk
i ¼ 1 Ziþ
9ðx1,j1;1�x1,j2;1 Þðx2,j1;2�mn
1Þðx3,j1;3�x3,j2;3 Þ9
2kQk
i ¼ 1 Zi
for all of the appropriate j1,z and j2,z combinations.The weight function is derived akin to the previous explanation. Again, the weight function needed to be incorporated
to offset the lack of scale invariance and to down-weight the models with many free means. The additional structuralequations for each J are
W
Pki ¼ 1
biMSXi
� �1=ðtJ�1ÞNQtJ
j ¼ 1
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj
biMSXi
q� �2=ðtJ�1Þ[email protected]
1CA¼ biþPi if iZtJ ,
W
Pki ¼ 1
biMSXi
� �1=ðtJ�1ÞNQtJ
j ¼ 1
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj
biMSXi
q� �2=ðtJ�1Þ[email protected]
1CA¼ Pi if iotJ
and the weight function is
wJðxÞ ¼Yio tJ
f ðPiÞ ¼
QtJj ¼ 1
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj
biMSXi
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPk
i ¼ 1bi
MSXi
qNðtJ�1Þ=2
,
where bi ¼ ni=maxjðnjÞ, MSXi is the maximum likelihood estimate of the variance for group i, Pi is an independent w2ð1Þrandom variable for all i, WðzÞ is the Lambert W function, tJ is the number of groupings in a given J, and f is the density ofthe w2ð1Þ distribution. We can find the probability that J is correctly grouping the means, PðJÞ, using (10). However, in thiscase pJ cannot be calculated in closed form.
3.3. Simulations
Ideally we would like this inference method to identify the correct model at a high rate. When we assume constantvariance for all of the k groups we can calculate the probabilities directly. When the variance is not assumed to be constantwe used a Monte Carlo approach to generate a sample from the generalized fiducial density. We used the importance
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 883
sampling algorithm in Appendix A to sample from (11) and calculate P(J) for all possible groupings. Our simulation used1000 data sets and an effective sample size of 5000 when the variance was not assumed to be constant.
3.3.1. Constant variance
Looking at a few interesting cases will help us assess the validity of the method. Fig. 1 illustrates that the correctgrouping, J¼ 123, is selected at a high rate. Also, the magnitude of the variance does not effect the selection probability.
Difficulties arise when the true means are relatively close together. For instance, when l0 ¼ ð1,1:5,1:5Þ or l0 ¼ ð1,1:5,2Þthe correct model is selected at a higher rate as the sample size increases. As expected, at small samples sizes our methodattempts to incorrectly group means as equal. Figs. 2 and 3 reflect this.
The easiest case occurs when the means are very different. Fig. 4 demonstrates P(J) when l0 ¼ ð1;3,5Þ and Z0 ¼ 1.
n = 10
J
P (J
)
1|2|3
1 2|3
1 3|2
1|2 3
1 2 3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50 n = 100
n = 10 n = 50 n = 100
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)1.0
0.8
0.6
0.4
0.2
0.0
Fig. 1. P(J) for l0 ¼ ð1;1,1Þ and Z0 ¼ 1 and 100 for top and bottom rows respectively.
n = 10
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50 n = 100
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 2. P(J) for l0 ¼ ð1,1:5,1:5Þ and Z0 ¼ 1.
n = 10
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50 n = 100
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 3. P(J) for l0 ¼ ð1,1:5,2Þ and Z0 ¼ 1.
n = 10
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 4. P(J) for l0 ¼ ð1;3,5Þ and Z0 ¼ 1.
n = 10
J1|2
|3|4
1 2|3|
4
1|2|3
4
1 2|3
4
1 2 3|
4
1 2 4|
3
1 3 4|
2
1|2 3
4
1 2 3
4
n = 50
J1 2
|3|4
1|2|3
4
1 2|3
4
n = 100
J1 2
|3|4
1|2|3
4
1 2|3
4
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 5. P(J) for l0 ¼ ð1;1,2;2Þ and Z0 ¼ 1.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895884
Similar analysis can be done at higher dimensions. Again, when k¼4, l0 ¼ ð1;1,2;2Þ, and Z0 ¼ 1 our method is selectingthe correct model at a high rate as the sample size increases. Fig. 5 reflects this. The omitted groupings in the figures hadmedian probability, PðJÞ, of less than 0.02.
n = 10
J
P (J
)
1|2|3
1 2|3
1 3|2
1|2 3
1 2 3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50 n = 100
n = 10 n = 50 n = 100
P (J
)
P (J
)P
(J)
P (J
)P
(J)
J
1|2|3
1 2|3
1 3|2
1|2 3
1 2 3
J
1|2|3
1 2|3
1 3|2
1|2 3
1 2 3
J
1|2|3
1 2|3
1 3|2
1|2 3
1 2 3
n = 10 n = 50 n = 100
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 6. P(J) for l0 ¼ ð1;1,1Þ and g0 ¼ ð1;1,1Þ, ð1;2,3Þ, and ð100;100,100Þ for top, middle, and bottom rows respectively.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 885
3.3.2. Non-constant variance
When variance is not assumed to be constant similar results follow. Highlighting a few we can see that the variancedoes not effect the probability of selecting the correct model. This is reflected in Fig. 6.
Again the easy case is when the means are very different from each other. Fig. 7 is reflective of this.In the four dimensional simulation we can see that the correct model is being selected at a relatively high rate for all of
the sample sizes. This is illustrated in Fig. 8 for all J where the median probability is greater than 0.02.
4. Asymptotic results
As defined in Eq. (10) we can calculate the probability that each J is the correct grouping. In this section we will provethat our method will asymptotically select the correct model.
Assumption 1. Xij is an independent random variable from a Nðmi,ZiÞ distribution.
Assumption 2. There exists 0obio1 such that ni ¼ bin for all i¼ 1, . . . ,k.
n = 10
J1|2
|31 2
|31 3
|21|2
31 2
3
J1|2
|31 2
|31 3
|21|2
31 2
3
n = 50
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 7. P(J) for l0 ¼ ð1;3,5Þ and g0 ¼ ð1;1,1Þ.
n = 10 n = 50 n = 100
J J J1|2
|3|4
1 2|3|
4
1|2|3
4
1 2|3
4
1 2 3|
4
1 2 4|
3
1 3 4|
2
1|2 3
4
1 2 3
4
1 2|3|
4
1|2|3
4
1 2|3
4
1 2|3|
4
1|2|3
4
1 2|3
4
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0
P (J
)
1.0
0.8
0.6
0.4
0.2
0.0P
(J)
1.0
0.8
0.6
0.4
0.2
0.0
Fig. 8. P(J) for l0 ¼ ð1;1,2;2Þ and Z0 ¼ ð1;1,1;1Þ.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895886
Theorem 1. If J correctly groups the means then PðJÞ-1 almost surely.
To prove this we will show that p~J=pJ-0 for any~JaJ. There are two cases that will be observed. First, when ~J incorrectly
groups means as equal. In this case p~J=pJ will converge to zero exponentially as n-1. The second case is when ~J does notincorrectly group the means but there are too many groups. This will result in p~J=pJ converging to zero polynomially asn-1. The proof was done assuming both constant and non-constant variance. The details are relegated to Appendix B.
5. Examples
5.1. Simulated data
To further demonstrate the ability of our method we analyzed a simulated data set. This allows us to know what thetrue treatment means are. The sample mean and variance of the data is
x ¼ ð0:69,1:65,1:80,1:84Þ
and
s2 ¼ ð1:56,1:35,1:61,2:13Þ:
This data set was generated from independent normal distributions with l0 ¼ ð1;2,2;2Þ, g0 ¼ ð2;2,2;2Þ, and a simple size ofn¼20 for each treatment. Table 1 reflects grouping probabilities when PðJÞ40:03. Both the constant and non-constant
Table 1Multiple comparison P(J) for the simulated example.
J P(J)
Constant variance
192394 0.049
192493 0.051
192934 0.062
12934 0.044
19234 0.663
1 2 3 4 0.060
Non-constant variance
192394 0.061
192493 0.058
192934 0.065
12934 0.041
19234 0.604
1 2 3 4 0.071
0
0.5
0.75
0.9
0.95
1
1
2
3
4
Constant variance
i1 2 3 4
i1 2 3 4
j
P (µ i
= µ j
)
0
0.5
0.75
0.9
0.95
1
1
2
3
4
Non−constant variancej
P (µ i
= µ i+
1=...=µ i+
r) for
r>i
P (µ i
= µ j
)
P (µ i
= µ i+
1=...=µ i+
r) for
r>i
Fig. 9. Pðmi ¼ mjÞ and Pðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ with constant and non-constant variance for the simulated example.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 887
variance methods select the correct grouping at a high rate (PðJÞ ¼ 0:663 and PðJÞ ¼ 0:604 for J¼ 19234 when the variance isassumed to be constant and non-constant respectively).
In addition to finding the probability for each grouping the fiducial method can also find the fiducial probability of anynumber of means being equal. For instance, we can find the fiducial probability that any two means are equal ðmi ¼ mjÞ orthe probability that any sequence of means are equal ðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ. This is done by adding up probabilities forthe models that mi ¼ mj or mi ¼ miþ1 ¼ � � � ¼ miþ r ,
Pðmi ¼ mjÞ ¼X
J2fJ1 ,...,JHgPðJÞIfJ:mi ¼ mjg ð12Þ
and
Pðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ ¼X
J2fJ1 ,...,JHgPðJÞIfJ:mi ¼ miþ 1 ¼ ��� ¼ miþ rg: ð13Þ
Fig. 9 pictorially represent these probabilities for the simulated example. As the pictures show it is very reasonable thatm1am2 ¼ m3 ¼ m4.
In comparison to a common frequentist method, Tukey’s HSD test could not find significant differences in the means (1, 2)and (2, 3, 4) controlling the experimentwise error rate at a¼ 0:05. Tukey’s HSD is commonly known to be rather conservativewhich makes it difficult to detect differences. A method described in Abdel-Karim (2005) uses a similar Tukey approach butallows for unequal variance across the treatments. This method could not find significant differences between the means (1, 2),(2, 3, 4), and (1, 4).
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895888
5.2. Clover plant data
A data set from Steele and Torrie (1980) measured the nitrogen content (in mg) of red clover plants inoculated withcultures of Rhizobium trifolli and the addition of Rhizobium meliloti strains. As discussed in Gopalan and Berry (1998), theR. trifolli was tested with a composite of five alpha strains (3DOk1, 3DOk4, 3DOk5, 3DOk7, 3DOk13), R. meliloti, and acomposite of the alpha strains. There were six treatments in all. The goal of the experiment was to measure the nitrogenlevels for the different treatments. The data can be seen in Table 2.
We analyzed this data set using both the constant and non-constant variance methods. The grouping probabilities areseen in Table 3 when PðJÞ40:03. If we assume that the variance is constant J¼ 129349596 is the most likely scenario. If wedo not assume that the variance is constant the most likely grouping is J¼ 12934956. Looking at the sample means andstandard deviations both of these results seem very reasonable.
The Bayesian method described in Gopalan and Berry (1998) analyzed this data set with the constant varianceassumption. Prior distributions were selected for the parameters using various distributions; the groupings used a Dirichletprocess prior. Table 4 illustrates a few highlighted posterior probabilities. They claim, if the posterior probabilities are largein comparison to the prior probabilities for all values of M (Dirichlet process prior parameter) then these are likelygroupings of the means. The resulting groupings in Table 4 are their recommended groupings.
Similarities between our analysis and theirs exist. J¼ 129349596 and 12934956 are common to all of the methods aslikely groupings of the means.
Table 2Clover plant data.
Treatments
1 2 3 4 5 6
3DOk13 3DOk4 Composite 3DOk7 3DOk5 3DOk1
14.3 17.0 17.3 20.7 17.7 19.4
14.4 19.4 19.4 21.0 24.8 32.6
11.8 9.1 19.1 20.5 27.9 27.0
11.6 11.9 16.9 18.8 25.2 32.1
14.2 15.8 20.8 18.6 24.3 33.0
Mean 13.26 14.64 18.70 19.92 23.98 28.82
SD 1.43 4.12 1.60 1.13 3.78 5.80
Table 3Multiple comparison P(J) for the red clover example.
J P(J)
Constant variance
19293949596 0.037
1293949596 0.100
1929349596 0.052
1929394596 0.030
129349596 0.196
129394596 0.063
129394956 0.043
192934956 0.036
12934956 0.078
12934596 0.051
Non-constant variance
19293949596 0.041
1293949596 0.097
1929349596 0.052
1929394956 0.050
129349596 0.115
129394596 0.049
129394956 0.102
192934956 0.058
12934956 0.139
12934596 0.042
Table 4Posterior probabilities of select J for the red clover example.
J M
0.334 0.733 1.373 1.956 2.605 3.462 4.909 9.13 19.88
Posterior probabilities
12349596 0.032 0.059 0.057 0.046 0.041 0.034 0.026 0.013 0.005
12934596 0.186 0.211 0.202 0.189 0.171 0.148 0.112 0.061 0.020
12934956 0.144 0.178 0.167 0.149 0.137 0.116 0.094 0.049 0.016
129349596 0.037 0.086 0.152 0.199 0.229 0.257 0.276 0.281 0.199
0
0.5
0.75
0.9
0.95
1
1
2
3
4
5
6
Constant variance
i1 2 3 4 5 6
i1 2 3 4 5 6
j
0
0.5
0.75
0.9
0.95
1
1
2
3
4
5
6
Non−constant variance
jP (µ i
= µ j
)
P (µ i
= µ i+
1=...=µ i+
r) for
r>i
P (µ i
= µ j
)
P (µ i
= µ i+
1=...=µ i+
r) for
r>i
Fig. 10. Pðmi ¼mjÞ and Pðmi ¼ miþ1 ¼ � � � ¼miþ rÞ with constant and non-constant variance for the red clover example.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 889
Fig. 10 pictorially represent the probabilities in Eqs. (12) and (13) for the red clover plant example. As the pictures showit is very reasonable that m1 ¼ m2, m3 ¼ m4, and possibly m5 ¼ m6. Tukey’s HSD test could not find significant differencesin the means (1, 2, 3, 4), (3, 4, 5), or (5, 6) and the method in Abdel-Karim (2005) could not detect differences in (1, 2) or(2, 3, 4, 5, 6) using an experimentwise error rate of a¼ 0:05.
6. Conclusion
Frequentist solutions for multiple comparison problems can test for a treatment effect or find differences amongtreatments. However, they cannot make a determination as to how reasonable it is that particular means are groupedtogether as equal.
Using a fiducial inference approach we have developed a method to determine the likelihood of grouping meanstogether. Based on simulation results, our method selects the correct grouping at a relatively high rate for smallsample sizes.
We analyzed a simulated data set and a data set that measured the nitrogen levels of red clover plants that wereinoculated with six different treatments. The analysis of the simulated data set yielded a high probability for the correctmodel ðm1am2 ¼ m3 ¼ m4Þ regardless of the variance assumptions. When analyzing the red clover example under theassumption of constant variance we found that J¼ 129349596 was the most likely grouping of the means ðPðJÞ ¼ 0:196Þ. TheBayesian solution also found that grouping to be reasonable, however, no discernible probability could be assigned to it.Additionally, our method found that J¼ 12934956 was the most likely grouping if the variance was not assumed to beconstant ðPðJÞ ¼ 0:139Þ.
The fiducial method is an interesting solution to the multiple comparison problem. The intuitive feel of the fiducialprobability for each model makes the interpretation very straightforward and the asymptotic properties and simulationresults assure high confidence in the analysis.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895890
Acknowledgments
The authors are thankful to the anonymous referees whose thoughtful comments have led to a great improvement inthe exposition of the manuscript. The author’s research was supported in part by the National Science Foundation grantsDMS 0707037 and DMS 1007543.
Appendix A. Importance sampling algorithm
The following steps were implemented in order to obtain a fiducial sample for x.
1.
For a particular J, start by generating mni ¼ x iþffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiminj2Ui MSXj=ðni�1ÞqTdf where Tdf � tðni�1Þ, x i ¼ u�1i ð
Puij ¼ 1 xjÞ, and
ni ¼ u�1i ðPui
j ¼ 1 njÞ for all i¼ 1, . . . ,tJ .
2.
Note, that J¼U19U29 . . . 9UtJ where Ui ¼ ri1 ri2 . . . riui are the indexes of equal meansl¼ ðmr11 , . . .mr1u1 ,mr21 , . . .mr2u2 , . . . ,mrt1 , . . .mr2utJÞ ¼ mn1, . . .m
n
1|fflfflfflfflfflffl{zfflfflfflfflfflffl}u1 replictes
,mn2, . . .mn
2|fflfflfflfflfflffl{zfflfflfflfflfflffl}u2 replictes
, . . . ,mntJ , . . .mn
tJ|fflfflfflfflfflffl{zfflfflfflfflfflffl}utJ replictes
1CCCA:
Generate ðZl9mlÞ ¼W where W � Inv�Gammaðnl=2,ðnlððml�xlÞ2þMSXlÞÞ=2Þ for l¼ 1, . . . ,k.
3.
Calculate weights of each generated sample with,wJ ¼f JðxÞ
ðQtJ
i ¼ 1 giðmn
i ÞhiðZiÞÞ,
where f JðxÞ is the generalized fiducial density for the model with groupings J and giðmiÞ and hiðZiÞ are the densities fromdistributions described in steps 1, and 2.
4.
This process was repeated until we achieved the effective sample size calculated by ESSJ ¼ nJð1þðs2wJ Þw�2J Þ�1 where nJ isthe sample size for model J, s2wJ is the sample variance of the weights, and wJ is the sample mean of the weights.
5.
Lastly the weights were divided by the ESSJ.
6.
This process was repeated for all J.Appendix B. Proof of Theorem 1
With constant variance: This proof will be done with the assumption of constant variance (i.e. Zi ¼ Zj for all i and j) andwithout using the weight function, wJðXÞ: To prove Theorem 1 we will show that p~J=pJ-0 for any ~JaJ. We will observe twocases. First, when ~J incorrectly groups means as equal. Second, when ~J does not incorrectly group the means but there aretoo many groups.
For the first case let J2 incorrectly group the means and J1 is the correct grouping. Thus, there are t1 groups in J1 and t2groups in J2. At least one of the means in J2 is incorrectly grouped. The subscript in the following calculations note theassociation with J1 or J2.
pJ2pJ1
pG N�t22� �
ðPt1
i ¼ 1 n01iMSX
01iÞðN�t1Þ=2Qt1
i ¼ 1ffiffiffiffiffiffin01i
pG N�t12� �
ðPt2
i ¼ 1 n02iMSX
02iÞðN�t2Þ=2Qt2
i ¼ 1ffiffiffiffiffiffin02i
p using Stirling’s formula
r ð2eÞðt2�t1Þ=2Nðt1�t2Þ=2
Qt1i ¼ 1
ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p ðPt1i ¼ 1 n01iMSX01iÞðN�t1Þ=2ðPt2
i ¼ 1 n02iMSX
02iÞðN�t2Þ=2
WLOG assume U1 2 J2 is an incorrect grouping
r ð2eÞðt2�t1Þ=2Nðt1�t2Þ=2
Qt1i ¼ 1
ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p Zðt2�t1Þ=20Pt1
i ¼ 1 n01iZ0ð1þOð1ÞÞ
Z0
� �ðN�t1Þ=2n021
Znð1þOð1ÞÞZ0
þPt2
i ¼ 2 n02iZ0ð1þOð1ÞÞ
Z0
� �ðN�t2Þ=2where Zn4Z because of the incorrect grouping
r ð2eÞðt2�t1Þ=2Qt1
i ¼ 1ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p Zðt2�t1Þ=20 1þ r4 Zn
Z0�1
� �� �ðN�t1Þ=21þr Z
n
Z0�1
� �h ic
� �ðN�t2Þ=2 Eventually a:s:-0 a:s:
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 891
for
c¼1þ 1þr Z
n
Z0�1
� �� �2 1þr Z
n
Z0�1
� �� �and 0oroPi2U1 biðPki ¼ 1 biÞ�1o1.
The second case when J2 is a valid model with too many groups and J1 is the correct grouping. Thus, there are t1 groupsin J1, t2 groups in J2 and t24t1. Let
J1 ¼U119U129 � � � 9U1t1 ,
J2 ¼U219U229 � � � 9U2t2 ,
where
U1i ¼[
k2Ki
U2k
and Ki � f1, . . . ,t2g for at least one U1i.
pJ2pJ1
pG N�t22� �
ðPt1
i ¼ 1 n01iMSX
01iÞðN�t1Þ=2Qt1
i ¼ 1ffiffiffiffiffiffin01i
pG N�t12� �
ðPt2
i ¼ 1 n02iMSX
02iÞðN�t2Þ=2Qt2
i ¼ 1ffiffiffiffiffiffin02i
pr ð2eÞ
ðt2�t1Þ=2Nðt1�t2Þ=2Qt1
i ¼ 1ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p ðPt1i ¼ 1 SSX01iÞðN�t1Þ=2ðPt2
i ¼ 1 SSX02iÞðN�t2Þ=2
r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2
Qt1i ¼ 1
ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p 1�Pt1
i ¼ 1P
k2Ki n2k16Z log log N
n2k
� �NZ
1A�N=2
eventually a:s: using the law of iterated logarithms
r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2
Qt1i ¼ 1
ffiffiffiffiffiffin01i
pQt2i ¼ 1
ffiffiffiffiffiffin02i
p 1�R log log NN
� ��N=2WLOG assume that U1t1 ¼U2ðt2�1Þ [ U2t2 and U1i ¼U2i for all other i
r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2bffiffiffiffiffiffiffiffi
n02t2
q 1�R loglog NN
� ��N=2-0 a:s:
for some R41 and b40.Therefore, we have shown that p~J=pJ-0 for any
~JaJ where J is the correct grouping. This completes the proof. &With non-constant variance: This proof will not assume constant variance. Additionally, the proof will be done without
the use of the weight function. The generalized fiducial density for any J without the weight function is
~f JðxÞ ¼Vx,JQki ¼ 1 Zi
1
ð2pÞn1=2Zn1=21exp � 1
2Z1
Xn1j ¼ 1ðx1j�m1Þ
2
8<:
9=; � � � � 1ð2pÞnk=2Znk=2k exp �
1
2Zk
Xnkj ¼ 1ðxkj�mkÞ
2
8<:
9=;
¼ Vx,JQk
i ¼ 1 Z�ni=2�1i
ð2pÞN=2exp �1
2
Xki ¼ 1
niððmi�xiÞ
2þMSXiÞZi
( ):
If we could integrate this function we could calculate the probabilities, P(J), directly. However, we cannot fully integrate itso we will apply different techniques. Note, that J¼U19U29 � � � 9UtJ where Ui ¼ ri1 ri2 � � � riui are the indexes of equal means
l¼ ðmr11 , . . .mr1u1 ,mr21 , . . .mr2u2 , . . . ,mrt1 , . . .mr2utJÞ ¼ mn1, . . .m
n
1|fflfflfflfflfflffl{zfflfflfflfflfflffl}u1 replictes
,mn2, . . .mn
2|fflfflfflfflfflffl{zfflfflfflfflfflffl}u2 replictes
, . . . ,mntJ , . . .mn
tJ|fflfflfflfflfflffl{zfflfflfflfflfflffl}utJ replictes
1CCCA:
Without loss of generality we will assume l¼ ðm1, . . . ,mkÞ ¼ ðln1, . . . ,lntJ Þ ¼ ln. Notice that
pJ ¼ZX
f JðxÞ dx¼ p�N=2Yki ¼ 1
n�ni=2i Gni2
� �h iZR
tJ
Vx,JYki ¼ 1ððmi�xiÞ
2þMSXiÞ�ni=2 dln:
Because Vx,J is dependent on mni we will bound this value. It is clear that Vx,J 4c1 for some c140. We could re-write Vx,J as
Vx,J ¼YtJi ¼ 1
mnðui�1Þi
!V1;1þ
XtJj ¼ 1
mnðuj�2ÞjYtJ
i ¼ 1,iajmnðui�1Þi
1AV2,jþ � � � þVz,1
������������,
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895892
where Vi,j are averages over a function of the data. If ui is even then 9mnðui�1Þi 9rmnuii þ1 and if ui is odd then
9mnðui�1Þi 9¼ mnðui�1Þi . Regardless of the ui the same technique will be used. Thus, without loss of generality we will assume
that ui is odd for all i:
Vx,J rYtJi ¼ 1
mnðui�1Þi
!9V1;19þ
XtJj ¼ 1
ðmnðuj�1Þj þ1ÞYtJ
i ¼ 1,iajmnðui�1Þi
1A9V2,j9þ � � � þ9Vz,19
rYki ¼ 1ððmi�xiÞ
2þMSXiÞðui�1Þ=2 !
9V ð1Þ9þ9V ð2Þ9,
where V ð1Þ and V ð2Þ are averages over the data, xi, and MSXi. Thus Vð1Þ and V ð2Þ will converge to some constant almost surely
by the strong law of large numbers.A lower bound for pJ is
pJ Zp.J ¼ c1p
�N=2Yki ¼ 1
n�ni=2i Gni2
� �h iZR
tJ
Yki ¼ 1ððmi�xiÞ
2þMSXiÞ�ni=2dln:
An upper bound for pJ is
pJ rp�N=2Yki ¼ 1
n�ni=2i Gni2
� �h iZR
tJ
Qki ¼ 1ððmi�xiÞ
2þMSXiÞðui�1Þ=2� �
9V ð1Þ9þ9V ð2Þ9Qki ¼ 1ððmi�xiÞ
2þMSXiÞni=2dln
rp�N=2Yki ¼ 1
n�ni=2i Gni2
� �h iZR
tJ
9V ð1Þ9Qki ¼ 1ððmi�xiÞ
2þMSXiÞðni�ui�1Þ=2þ
9V ð2Þ9Qki ¼ 1ððmi�xiÞ
2þMSXiÞni=2dln
rp�N=2Yki ¼ 1
n�ni=2i Gni2
� �h iZR
tJ
c29Vð1Þ9Qk
i ¼ 1ððmi�xiÞ2þMSXiÞðni�ui�1Þ=2
dln ¼ pmJ
for some c240.Because we cannot integrate with respect to ln we observe
gJðlnÞ ¼Yki ¼ 1ððmi�xiÞ
2þMSXiÞ�ni=2
with the transformations of
mni ¼mniffiffiffi
np þmni0 for i¼ 1, . . . ,tJ
and the substitutions of
xi ¼ mi0þZi1ffiffiffiffi
nip for i¼ 1, . . . ,k
and
MSXi ¼ Zi0þZi2ffiffiffiffi
nip for i¼ 1, . . . ,k,
where mi0 and Z10 are the true mean and variance for treatment i and ðZi1,Zi2Þ �Nð0,SÞ. Thus,
gJðmnÞ ¼ n�tJ=2Yki ¼ 1
miffiffiffinp þDi�
Zi1ffiffiffiffinip
� �2þZi0þ
Zi2ffiffiffiffinip
!�ni=2,
where m and mn follows the same structure as l and ln previously stated and Di ¼ mnj0�mi0 for i 2 Uj. We will see that mn
i
converges point-wise to a normal distribution.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 893
Taylor expanding logðgJðmnÞÞ we will get
logðgJðmnÞÞ ¼�tJ2
logðnÞþXki ¼ 1
� bin logðZi0þD2i Þ
2�
biffiffiffinp
2Di mi� Zi1ffiffiffibip
� �þ Zi2ffiffiffi
bip
� �2ðZi0þD
2i Þ
2664
þbi 2Di mi� Zi1ffiffiffi
bip
� �þ Zi2ffiffiffi
bip
� �24ðZi0þD
2i Þ
2�
bi mi� Zi1ffiffiffibip
� �2ð2Zi0þD
2i ÞþOðn�1=2Þ
37775:
Clearly if J is correctly grouping the means then Di ¼ 0. Otherwise we will select mnj0 such thatXi2Uj
biDiðZi0þD
2i Þ¼ 0
for all j¼ 1, . . . ,tJ . Thus,
logðgJðmnÞÞ ¼�tJ2
logðnÞþXki ¼ 1
� bin logðZi0þD2i Þ
2þ
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
26664
þbi 2Di mi� Zi1ffiffiffi
bip
� �þ Zi2ffiffiffi
bip
� �24ðZi0þD
2i Þ
2�
bit mi� Zi1ffiffiffibip
� �2ð2Zi0þD
2i ÞþOðn�1=2Þ
37775:
Or
gJðmnÞ ¼1
ntJ=2Qk
i ¼ 1ðZi0þD2i Þ
bin=2exp
Xki ¼ 1
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
( )exp �
XtJi ¼ 1
1
2s2Z,iðmni �zZ,iÞ
2þCZþOðn�1=2Þ( )
, ðB:1Þ
where zZ,i and s2Z,i are the appropriate mean and variance of mn
i dependent on the Zij values and CZ is the constant used incompleting the square. Clearly gJðmnÞCn converges to a normal density for the appropriate normalizing constant, Cn.
Lemma 1. Let
hJ,nðmnÞ ¼ CngJðmnÞ
for the previously described Cn, then hJ,nðmnÞrkJðmnÞ where k is integrable.
Proof. First, square the function, g, without the n power:
Yki ¼ 1
miffiffiffinp þDi�
Zi1ffiffiffiffinip
� �2þZi0þ
Zi2ffiffiffiffinip
!�bi:
For each Uj we are looking at a function in mnj that has at most uj peaks (local maximum) and at most uj�1 valleys (localminimum). For instance, for U1 we are looking at the function
Yi2U1
mn1ffiffiffinp þDi�
Zi1ffiffiffiffinip
� �2þZi0þ
Zi2ffiffiffiffinip
!�bi:
For large enough n this function has a unique global maximum with probability 1. Our mnj0 is close to this maximum(within cj0n
�1=2 where the cj0 depends on the Z’s). Next, we re-scale the function so that the global maximum is 1.The other local maximums will be cjln
1=2 away where l¼ 1, . . . ,ðuj�1Þ. Here cjl depends on the distances betweenmaximums.
The fraction between the value of the function’s local and global maximums is either a constant ðo1 if Zs0aZr0 for alls,r 2 Uj) or 1�cjln�1=2 if Zs ¼ Zr for all s,r 2 Uj, in which case the difference comes from the Z’s.
Finally, if we raise the function to the power n. The global maximum goes to 1. At the local maximum we have a height ofexpf�cjln1=2g, i.e., the local maximum is located at the point ðcjln1=2,expf�cjln1=2gÞ which is well below the Cauchy densityof ðcjln1=2,cð1þc2jlnÞ
�1Þ for some constant c.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895894
Finally, notice that if there was a point for which our function was larger than a bounding Cauchy it would be at the localmaximum. This is because the function decays from its local and global maxima faster than the Cauchy distribution.
From Eq. (B.1) we can see mni converges point-wise to a normal distribution and Lemma 1 allows us to bound gJðmnÞ for
all n. Therefore, we can calculate the asymptotic behavior of pmJ and p.J . Observe,
p.J ¼p�N=2
Qki ¼ 1 n
�ni=2i G
ni2
h ic1
ntJ=2ðZi0þD2i Þ
bin=2exp
Xki ¼ 1
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
( )
�ZR
tJ
exp
bi 2Di mi� Zi1ffiffiffibip
� �þ Zi2ffiffiffi
bip
� �24ðZi0þD
2i Þ
2�
bi mi� Zi1ffiffiffibip
� �2ð2Zi0þD
2i ÞþOðn�1=2Þ
8>>><>>>:
9>>>=>>>;dm
n
¼p�N=2
Qki ¼ 1 n
�ni=2i G
ni2
h ic1
ntJ=2ðZi0þD2i Þ
bin=2exp
Xki ¼ 1
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
( )B1,n
and a similar calculation produces
pmJ ¼p�N=2
Qki ¼ 1 n
�ni=2i G
ni2
h ic2V
ð1Þ
ntJ=2ðZi0þD2i Þðbin�ui�1Þ=2
expXki ¼ 1
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
( )
�ZR
tJ
exp
bi 2Di mi� Zi1ffiffiffibip
� �þ Zi2ffiffiffi
bip
� �24ðZi0þD
2i Þ
2�
bi mi� Zi1ffiffiffibip
� �2ð2Zi0þD
2i ÞþOðn�1=2Þ
8>>><>>>:
9>>>=>>>;dm
n
¼p�N=2
Qki ¼ 1 n
�ni=2i G
ni2
h ic2V
ð1Þ
ntJ=2ðZi0þD2i Þðbin�ui�1Þ=2
expXki ¼ 1
ffiffiffiffiffiffiffibin
pð2DiZi1�Zi2Þ
2ðZi0þD2i Þ
( )B2,n,
where Bi,n is the constant that comes from integration of the normal density and Bi,n-Bi by Lemma 1.To prove that PðJÞ-1 as n-1 we will observe p~J=pJ rpm~J =p
.J -0 for any
~JaJ: Like the previous proof there are twocases. First, when ~J incorrectly groups means as equal. Second, when ~J does not incorrectly group the means but there aretoo many groups.
For the first case let J2 incorrectly group the means and J1 is the correct grouping. Thus, there are t1 groups in J1 and t2groups in J2. At least one of the means in J2 is incorrectly grouped and at least one of the Dia0. Equivalently Di ¼ 0 for thegrouping in J1.
pmJ2p.J1¼ c2V
ð1ÞðZi0Þbin=2
c1ðZi0þD2i Þðbin�ui�1Þ=2
expPk
i ¼ 1
ffiffiffiffiffibinp
ð2DiZi1�Zi2Þ2ðZi0þD
2i Þ
� �B2,n
expPk
i ¼ 1�ffiffiffiffiffibinp
Zi22Zi0
� �B1,n
-0 a:s:
The second case when J2 is a valid model with too many groups and J1 is the correct grouping. Thus, there are t1 groupsin J1, t2 groups in J2 and t24t1.
pmJ2p.J1¼ n
t1=2c2Vð1ÞðZi0Þ
bin=2
nt2=2c1ðZi0Þðbin�ui�1Þ=2
expPk
i ¼ 1�ffiffiffiffiffibinp
Zi22ðZÞ
� �B2,n
expPk
i ¼ 1�ffiffiffiffiffibinp
Zi22Zi0
� �B1,n
¼ c2Vð1ÞðZi0Þ
bin=2
nðt2�t1Þ=2c1ðZi0Þðbin�ui�1Þ=2
B2,nB1,n
-0 a:s:
Thus we have shown that PðJÞ-1.The same convergence results for both constant a non-constant variance hold if the weight function is included. &
References
Abdel-Karim, A.H., 2005. Applications of Generalized Inference.Fisher, R.A., 1930. Inverse probability. Proceedings of the Cambridge Philosophical Society xxvi, 528–535.Gopalan, R., Berry, D.A., 1998. Bayesian multiple comparisons using Dirichlet process priors. Journal of American Statistical Association 93, 1130–1139.Hannig, J., 2009a On Asymptotic Properties of Generalized Fiducial Inference for Discretized Data. Technical Report UNC/STOR/09/02, Department of
Statistics and Operations Research, The University of North Carolina at Chapel Hill.Hannig, J., 2009b. On generalized fiducial inference. Statistica Sinica 19, 491–544.Hannig, J., Iyer, H.K., Patterson, P., 2006. Fiducial generalized confidence intervals. Journal of the American Statistical Association 101, 254–269.
D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 895
Hannig, J., Lee, T.C., 2009. Generalized fiducial inference for wavelet regression. Biometrika 96, 847–860.Lindley, D.V., 1958. Fiducial distributions and Bayes’ theorem. Journal of the Royal Statistical Society Series B 20, 102–107.Steele, R.D., Torrie, J.H., 1980. Principles and Procedures of Statistics—A Biomedical Approach, second ed. McGraw Hill, New York.Weeranhandi, S., 1993. Generalized confidence intervals. Journal of the American Statistical Association 88, 899–905.Zabell, S.L., 1992. R.A. Fisher and the fiducial argument. Statistical Science 7, 369–387.
A fiducial approach to multiple comparisonsIntroductionGeneralized fiducial inferenceOverview
Main resultsStructural equation with constant varianceStructural equation with non-constant varianceSimulationsConstant varianceNon-constant variance
Asymptotic resultsExamplesSimulated dataClover plant data
ConclusionAcknowledgmentsImportance sampling algorithmProof of Theorem 1References