Contents lists available at SciVerse ScienceDirect

Journal of Statistical Planning and Inference

Journal of Statistical Planning and Inference 142 (2012) 878–895

0378-37

doi:10.1

n Corr

E-m

journal homepage: www.elsevier.com/locate/jspi

A fiducial approach to multiple comparisons

Damian V. Wandler a,n, Jan Hannig b

a Department of Statistics, Colorado State University, Fort Collins, CO 80523, United Statesb Department of Statistics and Operations Research, The University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States

a r t i c l e i n f o

Article history:

Received 16 May 2010

Received in revised form

26 May 2011

Accepted 21 October 2011Available online 6 November 2011

Keywords:

Fiducial inference

Multiple comparisons

Importance sampling

Fiducial probability

Model selection

58/$ - see front matter & 2011 Elsevier B.V. A

016/j.jspi.2011.10.011

esponding author.

ail address: dvwandler@gmail.com (D.V. Wan

a b s t r a c t

Comparing treatment means from populations that follow independent normal dis-

tributions is a common statistical problem. Many frequentist solutions exist to test for

significant differences amongst the treatment means. A different approach would be to

determine how likely it is that particular means are grouped as equal. We developed a

fiducial framework for this situation. Our method provides fiducial probabilities that any

number of means are equal based on the data and the assumed normal distributions.

This methodology was developed when there is constant and non-constant variance

across populations. Simulations suggest that our method selects the correct grouping of

means at a relatively high rate for small sample sizes and asymptotic calculations

demonstrate good properties. Additionally, we have demonstrated the flexibility in the

methods ability to calculate the fiducial probability for any number of equal means. This

was done by analyzing a simulated data set and a data set measuring the nitrogen levels

of red clover plants that were inoculated with different treatments.

& 2011 Elsevier B.V. All rights reserved.

1. Introduction

Treatment means are commonly compared to each other to determine their relationship. A variety of problemscompare treatment means. For example, comparing the effectiveness of multiple drugs in a pharmaceutical setting is acommon practice. Other areas of application include agriculture, finance, production industries, etc.

In this scenario there are observations Xi ¼ ðXi1, . . .Xini Þ for populations i¼ 1, . . . ,k. The k populations followindependent normal distributions with means l¼ ðm1, . . . ,mkÞ

T and variance g. The multiple comparison problem (MCP)attempts to perform inference on the groupings of the individual means within l from the observations X1,X2, . . . ,Xk.

There are several frequentist solutions for multiple comparison problems. Using frequentist methods, analysis ofvariance (ANOVA) is used to test for a significant treatment effect. There are several tests for differences amongtreatments. Some are, Fisher’s least significant difference (LSD), Tukey’s honest significant difference (HSD), Sheffe’spairwise differences, Duncan’s multiple range test, etc. These solutions control the comparisonwise or experimentwiseerror rate for some a. However, these solutions cannot determine a likelihood that particular means are equal or unequal.

A Bayesian procedure for MCP has been developed in Gopalan and Berry (1998). This method uses a Dirichlet processprior to decide between competing groupings of l. The final posterior probabilities are used to discern amongst thegroupings for different priors.

We have developed methodology for this scenario using an extension of R.A. Fisher’s fiducial inference. We usegeneralized fiducial inference as developed in Hannig (2009b) to determine the likelihood of grouping particular means as

ll rights reserved.

dler).

www.elsevier.com/locate/jspiwww.elsevier.com/locate/jspidx.doi.org/10.1016/j.jspi.2011.10.011mailto:dvwandler@gmail.comdx.doi.org/10.1016/j.jspi.2011.10.011

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 879

equal or unequal. A model selection technique was used to determine, based on the data, the likely model(s). This isdeveloped for g¼ ðZ, . . . ,ZÞ (constant variance) and g¼ ðZ1,Z2, . . . ,ZkÞ (non-constant variance). Simulation results suggestthat our method selects the correct grouping at a high rate for small sample sizes. We have also proven that our methodwill asymptotically select the correct grouping of means.

In addition to simulation results and theoretical calculations, we analyzed a simulated data set and a data setmeasuring nitrogen levels of red clover plants that were inoculated with different treatments. The analyses wereconducted assuming both constant and non-constant variance; the results from the red clover data set were comparedwith those of the Bayesian method (which assumes constant variance). Both the fiducial and Bayesian methods producesomething of a posterior probability for each possible grouping.

2. Generalized fiducial inference

2.1. Overview

Fisher (1930) did not support the Bayesian idea of assuming a prior distribution on the parameters when there islimited information available. As a result, he developed fiducial inference to offset this perceived shortcoming. Fiducialinference did not garner approval when some of Fisher’s claims were found to be untrue in Lindley (1958) and Zabell(1992). More recently, Weeranhandi (1993) has developed generalized inference and the work of Hannig et al. (2006)established a link between fiducial and generalized inference. Hannig (2009b) and references within provide a thoroughbackground on fiducial inference and its properties.

The principle idea of generalized fiducial inference is similar to the likelihood function and ‘‘switches’’ the role of thedata, X, and the parameter(s) x. To formally introduce fiducial inference we assume that a relationship, called the structuralequation, between the data, X, and the parameter(s), x, exists in the form

X¼ Gðx,UÞ, ð1Þ

where U is a random vector with a completely known distribution and independent of any parameters. After observing Xwe use the known distribution of U and the relationship from the structural equation to infer a distribution on x. Thisallows us to define a probability measure on the parameter space, X. If (1) can be inverted the inverse will be written asG�1ð�,�Þ. For an observed x and u we can calculate x from

x¼ G�1ðx,uÞ: ð2Þ

From this inverse relationship we can generate a random sample of u01,u02, . . . ,u

0M and obtain a random sample for

x : x01 ¼ G�1ðx,u01Þ,x02 ¼ G

�1ðx,u02Þ, . . .x0M ¼ G�1ðx,u0MÞ. This sample is called a fiducial sample and can be used to calculate

estimates and confidences intervals for the true parameter(s), x0.Hannig and Lee (2009) address two potential times that G�1ð�,�Þ may not exist. They are when (i) there is no x that

satisfies (2) or (ii) there is more than one x that satisfies (2). From Hannig (2009b) we will handle situation (i) byeliminating such u’s and re-normalizing the sampling probabilities. This is reasonable because we know our data wasgenerated using x0 and u0 so at least one solution for (2) exists. We will only consider the u’s that allow for G

�1ð�,�Þ to exist.Hannig (2009b) suggests that situation (ii) is handled by selecting an x by some, possibly random, rule that satisfies theinverse in (2).

A more rigorous definition of the inverse is the set valued function of

Q ðx,uÞ ¼ fx : x¼ Gðx,uÞg: ð3Þ

We know that our observed data was generated using some x0 and u0. We also know the distribution of U and thatQ ðx,u0Þa|. Coupling these facts we can compute the generalized fiducial distribution from

VðQ ðx,U%ÞÞ9fQ ðx,U%Þa|g, ð4Þ

where U% is an independent copy of U and V(S) is a random element for any measurable set, S, with support on the closureof S, S. Essentially, Vð�Þ is the random rule for picking the possible x’s. We will refer to a the random element that comesfrom (4) as Rx. For a more detailed discussion of the derivation of the generalized fiducial distribution see Hannig (2009b).

From the structural equation the generalized fiducial density is calculated as proposed in Hannig (2009b) and justifiedtheoretically in Hannig (2009a). Let G¼ ðg1, . . . ,gnÞ such that Xi ¼ giðx,UÞ for i¼ 1, . . . ,n. x is a p� 1 vector and denoteXi ¼ G0,iðx,UiÞ where Xi ¼ ðXi1 , . . . ,Xip Þ and Ui ¼ ðUi1 , . . . ,Uip Þ for all possible combinations of the indexes i¼ ði1, . . . ,ipÞ.Furthermore, assume that the functions G0,i are one-to-one and differentiable. Under some technical assumptions inHannig (2009a) this will produce the generalized fiducial density of

fRx ðxÞ ¼f Xðx9xÞJðx,xÞR

Xf Xðx9x0ÞJðx,x0Þ dx0

, ð5Þ

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895880

where

Jðx,xÞ ¼n

p

!�1 Xi ¼ ði1 ,...,ipÞ

det ddxG�10,i ðxi,xÞ

� �det ddxiG

�10,i ðxi,xÞ

� �������

������ ð6Þis the average of all subsets where 1r i1o � � �o iprn and the determinants in (6) are the appropriate Jacobians.

3. Main results

3.1. Structural equation with constant variance

In a multiple comparison problem we have k populations with means l¼ ðm1, . . . ,mkÞ. Data, which follows anindependent normal distribution, is of the form Xi ¼ ðXi1, . . .Xini Þ for all i¼ 1, . . . ,k where Xi is independent of Xj for all iand j. We are interested in the k treatment means. We would like to make some judgement on the equality or inequality ofthe means within competing models.

For example if Xi ¼ ðXi1, . . .Xini Þ is an independent random sample from a Nðmi,ZÞ distribution for i¼1,2 then theappropriate models would either assume m1 ¼ m2 or m1am2. The structural equations in this case could be

X1j ¼ ðm1þffiffiffiZp

Z1jÞIm1 ¼ m2þðm1þffiffiffiZp

Z1jÞIm1am2 ,

X2j ¼ ðm2þffiffiffiZp Z2jÞIm1 ¼ m2þðm2þ ffiffiffiZp Z2jÞIm1am2 ,

where Zij are independent random variables from the Nð0;1Þ distribution. From these structural equations the generalizedfiducial density in (5) can be calculated for each model (m1 ¼ m2 and m1am2Þ.

To simplify notation we will use J¼U19U29 . . . 9UtJ where Ui is a collection of indexes of the means that are equal. Themeans indexed by Ui and Uj separated by a vertical bar ‘‘9’’ are unequal. For example when k¼3, if J¼ 123 then U1 ¼ 123signifies m1 ¼ m2 ¼ m3 ¼ mn1. If J¼ 1 293 then U1 ¼ 1 2 and U2 ¼ 3 signify m1 ¼ m2 ¼ mn1 and m3 ¼ mn2 where mn1amn2. Note thatthere are ui equal means in group Ui, tJ total groupings in J, and the unique means are ðmn1,mn2, . . . ,mnt Þ.

In general, if Xi1, . . . ,Xini is an independent random sample from a Nðmi,ZÞ distribution for i¼ 1, . . . ,k then a structuralequation is

Xij ¼X

J2fJ1 ,...,JHgðmiþ

ffiffiffiZp

ZijÞIJ , ð7Þ

where IJ ¼ 1 if grouping J is selected and 0 otherwise, the equality of mi ¼ mj follow the grouping in J for all possiblegroupings fJ1, . . . ,JHg, and Zij are independent random variables from the Nð0;1Þ distribution.

As explained below, the fiducial distribution in (4) based on the structural equations above, will favor the model withthe most free means (all unequal means). To compensate for this we need to introduce additional structural equations thatare independent of those in (7). These structural equations will allow us to introduce a weight function that down-weightsthe models with many free means.

From Eq. (4) we can see that the generalized fiducial distribution is calculated by taking p (number of parameters)structural equations and conditioning on the fact that the remaining equations occurred. As a result, when there are moreparameters there are less equations that will be part of the conditioning or, equivalently, less conditions have to besatisfied. In this case we have N structural equations ðN¼

Pki ¼ 1 niÞ. If all of the means are different (J¼ 192939 . . . 9kÞ then

p¼ kþ1 (x¼ ðm1, . . . ,mk,ZÞ) and we condition on N�ðkþ1Þ events. If all of the means are equal ðJ¼ 123 . . . kÞ thenp¼ 2ðx¼ ðm,ZÞÞ and we condition on N�2 events. Clearly as more means are grouped together there are more conditionsthat need to be satisfied. In order to offset this unbalanced conditioning we will introduce additional structural equationsthat are independent of our original structural equations as proposed in Hannig and Lee (2009). These additional structuralequations will balance out the number of conditions that need to be met for each selected J.

As noted, adding additional structural equations allows us to down-weight the models with more free means toincrease the likelihood of grouping several means together. Additionally, we used the weight function introduced by theadditional structural equations to make the fiducial distribution more scale invariant. Attempting to make the methodscale invariant in this fashion is rather ad hoc but seemed to work well in simulations and we can show that our method isasymptotically scale invariant.

The additional structural equations are:

WMSXN

2p

!¼ biþPi if iZtJ ,

WMSXN

2p

!¼ Pi if iotJ , ð8Þ

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 881

where MSX ¼ k�1Pk

i ¼ 1 MSXi, MSXi is the maximum likelihood estimate of the variance for group i, Pi is an independent

w2ð1Þ random variable for all i, WðzÞ is the Lambert W function, and tJ is the number of groupings in a given J. Because ofthe independence these structural equations will not affect the distribution of X but they will affect the conditionaldistribution in (4). When inverting the structural equations in (8), if iZtJ we can choose a bi for any Pi so that the equationis satisfied. Thus, conditioning on this equation will not effect the conditional distribution. If iotJ then Pi ¼WððMSXNÞ=ð2pÞÞwhich creates an additional condition to be met. Combining the additional conditions with the original structural equationsthere will always be N�2 conditions regardless of the grouping of the means. This will define the weight function as

wJðxÞ ¼Yio tJ

f ðPiÞ ¼1

MSXN

� �ðtJ�1Þ=2,

where f is the density of the w2ð1Þ distribution.Using the original structural equations, combined with the additional structural equations, the generalized fiducial

distribution (4) has a density given by

f ðxÞpX

J2fJ1 ,...,JHg

~f JðxÞwJðxÞIJ ,

where ~f JðxÞ is the numerator in (5) and will be computed for all groupings. This numerator for a grouping, J, is

~f JðxÞ ¼Vx,JZ

1

ð2pÞn1=2Zn1=2exp � 1

2ZXn1j ¼ 1ðx1j�m1Þ

2

8<:

9=; � � � � 1ð2pÞnk=2Znk=2 exp � 12Z

Xnkj ¼ 1ðxkj�mkÞ

2

8<:

9=;

¼ Vx,JZ�N=2�1

ð2pÞN=2exp � 1

2ZXki ¼ 1

Xnij ¼ 1ðxij�miÞ

2

8<:

9=;¼ Vx,J Z

�N=2�1

ð2pÞN=2exp � 1

2ZXtJi ¼ 1

n0iðmn

i �x0iÞ

2

( )exp � 1

2ZXtJi ¼ 1

n0iMSX0i

( ), ð9Þ

where

JJðx,xÞ ¼ C�1N,J

PtJl ¼ 1

Pi1 ,i22Ul ,i1 o i2

Pji1

,ji19xi1 ,j1�xi2 ,j2 9

2Z , tJ ok,Pkl ¼ 1

Pj1 o ,j2 9xl,j1�xl,j2 9

2Z, tJ ¼ k,

8>>>>><>>>>>:

¼Vx,JZ ,

n0i ¼Xl2Ui

nl,x0i ¼

Pl2Ui

Pnlj ¼ 1 xlj

n0i,

MSX0i ¼P

l2UiPnl

j ¼ 1ðxlj�x0iÞ

2

n0i, N¼

Xki ¼ 1

ni

and CN,J is the number of Jacobian terms to average over.If we recognize that mni 9Z follows a normal distribution for all i and Z follows an inverse gamma distribution then we

can integrate ~f JðxÞ over the parameter space of each grouping XJ . Thus,

pJ ¼ZXJ

f JðxÞwJðxÞ dx¼Vx,JwJðxÞ2N=2ptJ=2G

N�tJ2

� �ð2pÞN=2ð

PtJi ¼ 1 n

0iMSX

0iÞðN�tJ Þ=2QtJ

i ¼ 1ffiffiffiffin0i

p :We can find the probability that any J is correctly grouping the means by

PðJÞ ¼pJP~J p~J

: ð10Þ

Clearly, when a particular J is correctly grouping the means we would like P(J) to be large.

3.2. Structural equation with non-constant variance

Similar to the previous setup, if Xi1, . . . ,Xini is an independent random sample from a Nðmi,ZiÞ distribution for i¼ 1, . . . ,kthen a structural equation is

Xij ¼X

J2fJ1 ,...,JHgðmiþ

ffiffiffiffiffiZi

pZijÞIJ

for groupings fJ1, . . . ,JHg where Zij are independent random variables from the Nð0;1Þ distribution.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895882

Like the previous section, the numerator of (5) is

~f JðxÞ ¼Vx,JQki ¼ 1 Zi

1

ð2pÞn1=2Zn1=21exp � 1

2Z1

Xn1j ¼ 1ðx1j�m1Þ

2

8<:

9=; � � � � 1ð2pÞnk=2Znk=2k exp �

1

2Zk

Xnkj ¼ 1ðxkj�mkÞ

2

8<:

9=;

¼ Vx,JwJðxÞQk

i ¼ 1 Z�ni=2�1i

ð2pÞN=2exp �1

2

Xki ¼ 1

niððmi�xiÞ

2þMSXiÞZi

( ), ð11Þ

where

JJðx,xÞ ¼ C�1N,J

Pkz ¼ 1

Pj1,z o j2,z rnz

Pði1 ...,itJ Þ

Pj ¼ ðj1,z ,j2,zÞ9T9

2kQk

i ¼ 1 Zi¼

Vx,JQki ¼ 1 Zi

,

il ¼ fi1, . . . ,iul�1g � Ul is the set of 1r i1o i2o � � �o iul�1rul, CN,J is the number Jacobian terms to average over,

T ¼YtJl ¼ 1

Yi2il

ðxi,j�mnl Þ

24

35ðxiul ,j1,z�xiul ,j2,z Þ,

xi ¼Pni

j ¼ 1 xij

niand MSXi ¼

Pnij ¼ 1ðxlj�xiÞ

2

ni:

As an example of the Jacobian, if J¼ 19293 then we average over

9ðx1,j1;1�x1,j2;1 Þðx2,j1;2�x2,j2;2 Þðx3,j1;3�x3,j2;3 Þ9

2kQk

i ¼ 1 Zi

for all j1,zo j2,zonz combinations (z¼ 1;2,3Þ. If J¼ 1293 then we average over

9ðx1,j1;1�mn

1Þðx2,j1;2�x2,j2;2 Þðx3,j1;3�x3,j2;3 Þ9

2kQk

i ¼ 1 Ziþ

9ðx1,j1;1�x1,j2;1 Þðx2,j1;2�mn

1Þðx3,j1;3�x3,j2;3 Þ9

2kQk

i ¼ 1 Zi

for all of the appropriate j1,z and j2,z combinations.The weight function is derived akin to the previous explanation. Again, the weight function needed to be incorporated

to offset the lack of scale invariance and to down-weight the models with many free means. The additional structuralequations for each J are

W

Pki ¼ 1

biMSXi

� �1=ðtJ�1ÞNQtJ

j ¼ 1

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj

biMSXi

q� �2=ðtJ�1Þ0B@

1CA¼ biþPi if iZtJ ,

W

Pki ¼ 1

biMSXi

� �1=ðtJ�1ÞNQtJ

j ¼ 1

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj

biMSXi

q� �2=ðtJ�1Þ0B@

1CA¼ Pi if iotJ

and the weight function is

wJðxÞ ¼Yio tJ

f ðPiÞ ¼

QtJj ¼ 1

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPi2Uj

biMSXi

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPk

i ¼ 1bi

MSXi

qNðtJ�1Þ=2

,

where bi ¼ ni=maxjðnjÞ, MSXi is the maximum likelihood estimate of the variance for group i, Pi is an independent w2ð1Þrandom variable for all i, WðzÞ is the Lambert W function, tJ is the number of groupings in a given J, and f is the density ofthe w2ð1Þ distribution. We can find the probability that J is correctly grouping the means, PðJÞ, using (10). However, in thiscase pJ cannot be calculated in closed form.

3.3. Simulations

Ideally we would like this inference method to identify the correct model at a high rate. When we assume constantvariance for all of the k groups we can calculate the probabilities directly. When the variance is not assumed to be constantwe used a Monte Carlo approach to generate a sample from the generalized fiducial density. We used the importance

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 883

sampling algorithm in Appendix A to sample from (11) and calculate P(J) for all possible groupings. Our simulation used1000 data sets and an effective sample size of 5000 when the variance was not assumed to be constant.

3.3.1. Constant variance

Looking at a few interesting cases will help us assess the validity of the method. Fig. 1 illustrates that the correctgrouping, J¼ 123, is selected at a high rate. Also, the magnitude of the variance does not effect the selection probability.

Difficulties arise when the true means are relatively close together. For instance, when l0 ¼ ð1,1:5,1:5Þ or l0 ¼ ð1,1:5,2Þthe correct model is selected at a higher rate as the sample size increases. As expected, at small samples sizes our methodattempts to incorrectly group means as equal. Figs. 2 and 3 reflect this.

The easiest case occurs when the means are very different. Fig. 4 demonstrates P(J) when l0 ¼ ð1;3,5Þ and Z0 ¼ 1.

n = 10

J

P (J

)

1|2|3

1 2|3

1 3|2

1|2 3

1 2 3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50 n = 100

n = 10 n = 50 n = 100

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)1.0

0.8

0.6

0.4

0.2

0.0

Fig. 1. P(J) for l0 ¼ ð1;1,1Þ and Z0 ¼ 1 and 100 for top and bottom rows respectively.

n = 10

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50 n = 100

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 2. P(J) for l0 ¼ ð1,1:5,1:5Þ and Z0 ¼ 1.

n = 10

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50 n = 100

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 3. P(J) for l0 ¼ ð1,1:5,2Þ and Z0 ¼ 1.

n = 10

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 4. P(J) for l0 ¼ ð1;3,5Þ and Z0 ¼ 1.

n = 10

J1|2

|3|4

1 2|3|

4

1|2|3

4

1 2|3

4

1 2 3|

4

1 2 4|

3

1 3 4|

2

1|2 3

4

1 2 3

4

n = 50

J1 2

|3|4

1|2|3

4

1 2|3

4

n = 100

J1 2

|3|4

1|2|3

4

1 2|3

4

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 5. P(J) for l0 ¼ ð1;1,2;2Þ and Z0 ¼ 1.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895884

Similar analysis can be done at higher dimensions. Again, when k¼4, l0 ¼ ð1;1,2;2Þ, and Z0 ¼ 1 our method is selectingthe correct model at a high rate as the sample size increases. Fig. 5 reflects this. The omitted groupings in the figures hadmedian probability, PðJÞ, of less than 0.02.

n = 10

J

P (J

)

1|2|3

1 2|3

1 3|2

1|2 3

1 2 3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50 n = 100

n = 10 n = 50 n = 100

P (J

)

P (J

)P

(J)

P (J

)P

(J)

J

1|2|3

1 2|3

1 3|2

1|2 3

1 2 3

J

1|2|3

1 2|3

1 3|2

1|2 3

1 2 3

J

1|2|3

1 2|3

1 3|2

1|2 3

1 2 3

n = 10 n = 50 n = 100

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 6. P(J) for l0 ¼ ð1;1,1Þ and g0 ¼ ð1;1,1Þ, ð1;2,3Þ, and ð100;100,100Þ for top, middle, and bottom rows respectively.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 885

3.3.2. Non-constant variance

When variance is not assumed to be constant similar results follow. Highlighting a few we can see that the variancedoes not effect the probability of selecting the correct model. This is reflected in Fig. 6.

Again the easy case is when the means are very different from each other. Fig. 7 is reflective of this.In the four dimensional simulation we can see that the correct model is being selected at a relatively high rate for all of

the sample sizes. This is illustrated in Fig. 8 for all J where the median probability is greater than 0.02.

4. Asymptotic results

As defined in Eq. (10) we can calculate the probability that each J is the correct grouping. In this section we will provethat our method will asymptotically select the correct model.

Assumption 1. Xij is an independent random variable from a Nðmi,ZiÞ distribution.

Assumption 2. There exists 0obio1 such that ni ¼ bin for all i¼ 1, . . . ,k.

n = 10

J1|2

|31 2

|31 3

|21|2

31 2

3

J1|2

|31 2

|31 3

|21|2

31 2

3

n = 50

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 7. P(J) for l0 ¼ ð1;3,5Þ and g0 ¼ ð1;1,1Þ.

n = 10 n = 50 n = 100

J J J1|2

|3|4

1 2|3|

4

1|2|3

4

1 2|3

4

1 2 3|

4

1 2 4|

3

1 3 4|

2

1|2 3

4

1 2 3

4

1 2|3|

4

1|2|3

4

1 2|3

4

1 2|3|

4

1|2|3

4

1 2|3

4

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0

P (J

)

1.0

0.8

0.6

0.4

0.2

0.0P

(J)

1.0

0.8

0.6

0.4

0.2

0.0

Fig. 8. P(J) for l0 ¼ ð1;1,2;2Þ and Z0 ¼ ð1;1,1;1Þ.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895886

Theorem 1. If J correctly groups the means then PðJÞ-1 almost surely.

To prove this we will show that p~J=pJ-0 for any~JaJ. There are two cases that will be observed. First, when ~J incorrectly

groups means as equal. In this case p~J=pJ will converge to zero exponentially as n-1. The second case is when ~J does notincorrectly group the means but there are too many groups. This will result in p~J=pJ converging to zero polynomially asn-1. The proof was done assuming both constant and non-constant variance. The details are relegated to Appendix B.

5. Examples

5.1. Simulated data

To further demonstrate the ability of our method we analyzed a simulated data set. This allows us to know what thetrue treatment means are. The sample mean and variance of the data is

x ¼ ð0:69,1:65,1:80,1:84Þ

and

s2 ¼ ð1:56,1:35,1:61,2:13Þ:

This data set was generated from independent normal distributions with l0 ¼ ð1;2,2;2Þ, g0 ¼ ð2;2,2;2Þ, and a simple size ofn¼20 for each treatment. Table 1 reflects grouping probabilities when PðJÞ40:03. Both the constant and non-constant

Table 1Multiple comparison P(J) for the simulated example.

J P(J)

Constant variance

192394 0.049

192493 0.051

192934 0.062

12934 0.044

19234 0.663

1 2 3 4 0.060

Non-constant variance

192394 0.061

192493 0.058

192934 0.065

12934 0.041

19234 0.604

1 2 3 4 0.071

0

0.5

0.75

0.9

0.95

1

1

2

3

4

Constant variance

i1 2 3 4

i1 2 3 4

j

P (µ i

= µ j

)

0

0.5

0.75

0.9

0.95

1

1

2

3

4

Non−constant variancej

P (µ i

= µ i+

1=...=µ i+

r) for

r>i

P (µ i

= µ j

)

P (µ i

= µ i+

1=...=µ i+

r) for

r>i

Fig. 9. Pðmi ¼ mjÞ and Pðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ with constant and non-constant variance for the simulated example.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 887

variance methods select the correct grouping at a high rate (PðJÞ ¼ 0:663 and PðJÞ ¼ 0:604 for J¼ 19234 when the variance isassumed to be constant and non-constant respectively).

In addition to finding the probability for each grouping the fiducial method can also find the fiducial probability of anynumber of means being equal. For instance, we can find the fiducial probability that any two means are equal ðmi ¼ mjÞ orthe probability that any sequence of means are equal ðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ. This is done by adding up probabilities forthe models that mi ¼ mj or mi ¼ miþ1 ¼ � � � ¼ miþ r ,

Pðmi ¼ mjÞ ¼X

J2fJ1 ,...,JHgPðJÞIfJ:mi ¼ mjg ð12Þ

and

Pðmi ¼ miþ1 ¼ � � � ¼ miþ rÞ ¼X

J2fJ1 ,...,JHgPðJÞIfJ:mi ¼ miþ 1 ¼ ��� ¼ miþ rg: ð13Þ

Fig. 9 pictorially represent these probabilities for the simulated example. As the pictures show it is very reasonable thatm1am2 ¼ m3 ¼ m4.

In comparison to a common frequentist method, Tukey’s HSD test could not find significant differences in the means (1, 2)and (2, 3, 4) controlling the experimentwise error rate at a¼ 0:05. Tukey’s HSD is commonly known to be rather conservativewhich makes it difficult to detect differences. A method described in Abdel-Karim (2005) uses a similar Tukey approach butallows for unequal variance across the treatments. This method could not find significant differences between the means (1, 2),(2, 3, 4), and (1, 4).

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895888

5.2. Clover plant data

A data set from Steele and Torrie (1980) measured the nitrogen content (in mg) of red clover plants inoculated withcultures of Rhizobium trifolli and the addition of Rhizobium meliloti strains. As discussed in Gopalan and Berry (1998), theR. trifolli was tested with a composite of five alpha strains (3DOk1, 3DOk4, 3DOk5, 3DOk7, 3DOk13), R. meliloti, and acomposite of the alpha strains. There were six treatments in all. The goal of the experiment was to measure the nitrogenlevels for the different treatments. The data can be seen in Table 2.

We analyzed this data set using both the constant and non-constant variance methods. The grouping probabilities areseen in Table 3 when PðJÞ40:03. If we assume that the variance is constant J¼ 129349596 is the most likely scenario. If wedo not assume that the variance is constant the most likely grouping is J¼ 12934956. Looking at the sample means andstandard deviations both of these results seem very reasonable.

The Bayesian method described in Gopalan and Berry (1998) analyzed this data set with the constant varianceassumption. Prior distributions were selected for the parameters using various distributions; the groupings used a Dirichletprocess prior. Table 4 illustrates a few highlighted posterior probabilities. They claim, if the posterior probabilities are largein comparison to the prior probabilities for all values of M (Dirichlet process prior parameter) then these are likelygroupings of the means. The resulting groupings in Table 4 are their recommended groupings.

Similarities between our analysis and theirs exist. J¼ 129349596 and 12934956 are common to all of the methods aslikely groupings of the means.

Table 2Clover plant data.

Treatments

1 2 3 4 5 6

3DOk13 3DOk4 Composite 3DOk7 3DOk5 3DOk1

14.3 17.0 17.3 20.7 17.7 19.4

14.4 19.4 19.4 21.0 24.8 32.6

11.8 9.1 19.1 20.5 27.9 27.0

11.6 11.9 16.9 18.8 25.2 32.1

14.2 15.8 20.8 18.6 24.3 33.0

Mean 13.26 14.64 18.70 19.92 23.98 28.82

SD 1.43 4.12 1.60 1.13 3.78 5.80

Table 3Multiple comparison P(J) for the red clover example.

J P(J)

Constant variance

19293949596 0.037

1293949596 0.100

1929349596 0.052

1929394596 0.030

129349596 0.196

129394596 0.063

129394956 0.043

192934956 0.036

12934956 0.078

12934596 0.051

Non-constant variance

19293949596 0.041

1293949596 0.097

1929349596 0.052

1929394956 0.050

129349596 0.115

129394596 0.049

129394956 0.102

192934956 0.058

12934956 0.139

12934596 0.042

Table 4Posterior probabilities of select J for the red clover example.

J M

0.334 0.733 1.373 1.956 2.605 3.462 4.909 9.13 19.88

Posterior probabilities

12349596 0.032 0.059 0.057 0.046 0.041 0.034 0.026 0.013 0.005

12934596 0.186 0.211 0.202 0.189 0.171 0.148 0.112 0.061 0.020

12934956 0.144 0.178 0.167 0.149 0.137 0.116 0.094 0.049 0.016

129349596 0.037 0.086 0.152 0.199 0.229 0.257 0.276 0.281 0.199

0

0.5

0.75

0.9

0.95

1

1

2

3

4

5

6

Constant variance

i1 2 3 4 5 6

i1 2 3 4 5 6

j

0

0.5

0.75

0.9

0.95

1

1

2

3

4

5

6

Non−constant variance

jP (µ i

= µ j

)

P (µ i

= µ i+

1=...=µ i+

r) for

r>i

P (µ i

= µ j

)

P (µ i

= µ i+

1=...=µ i+

r) for

r>i

Fig. 10. Pðmi ¼mjÞ and Pðmi ¼ miþ1 ¼ � � � ¼miþ rÞ with constant and non-constant variance for the red clover example.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 889

Fig. 10 pictorially represent the probabilities in Eqs. (12) and (13) for the red clover plant example. As the pictures showit is very reasonable that m1 ¼ m2, m3 ¼ m4, and possibly m5 ¼ m6. Tukey’s HSD test could not find significant differencesin the means (1, 2, 3, 4), (3, 4, 5), or (5, 6) and the method in Abdel-Karim (2005) could not detect differences in (1, 2) or(2, 3, 4, 5, 6) using an experimentwise error rate of a¼ 0:05.

6. Conclusion

Frequentist solutions for multiple comparison problems can test for a treatment effect or find differences amongtreatments. However, they cannot make a determination as to how reasonable it is that particular means are groupedtogether as equal.

Using a fiducial inference approach we have developed a method to determine the likelihood of grouping meanstogether. Based on simulation results, our method selects the correct grouping at a relatively high rate for smallsample sizes.

We analyzed a simulated data set and a data set that measured the nitrogen levels of red clover plants that wereinoculated with six different treatments. The analysis of the simulated data set yielded a high probability for the correctmodel ðm1am2 ¼ m3 ¼ m4Þ regardless of the variance assumptions. When analyzing the red clover example under theassumption of constant variance we found that J¼ 129349596 was the most likely grouping of the means ðPðJÞ ¼ 0:196Þ. TheBayesian solution also found that grouping to be reasonable, however, no discernible probability could be assigned to it.Additionally, our method found that J¼ 12934956 was the most likely grouping if the variance was not assumed to beconstant ðPðJÞ ¼ 0:139Þ.

The fiducial method is an interesting solution to the multiple comparison problem. The intuitive feel of the fiducialprobability for each model makes the interpretation very straightforward and the asymptotic properties and simulationresults assure high confidence in the analysis.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895890

Acknowledgments

The authors are thankful to the anonymous referees whose thoughtful comments have led to a great improvement inthe exposition of the manuscript. The author’s research was supported in part by the National Science Foundation grantsDMS 0707037 and DMS 1007543.

Appendix A. Importance sampling algorithm

The following steps were implemented in order to obtain a fiducial sample for x.

1.

For a particular J, start by generating mni ¼ x iþffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiminj2Ui MSXj=ðni�1Þ

qTdf where Tdf � tðni�1Þ, x i ¼ u�1i ð

Puij ¼ 1 xjÞ, and

ni ¼ u�1i ðPui

j ¼ 1 njÞ for all i¼ 1, . . . ,tJ .

2.

Note, that J¼U19U29 . . . 9UtJ where Ui ¼ ri1 ri2 . . . riui are the indexes of equal means

l¼ ðmr11 , . . .mr1u1 ,mr21 , . . .mr2u2 , . . . ,mrt1 , . . .mr2utJÞ ¼ mn1, . . .m

n

1|fflfflfflfflfflffl{zfflfflfflfflfflffl}u1 replictes

,mn2, . . .mn

2|fflfflfflfflfflffl{zfflfflfflfflfflffl}u2 replictes

, . . . ,mntJ , . . .mn

tJ|fflfflfflfflfflffl{zfflfflfflfflfflffl}utJ replictes

0BBB@

1CCCA:

Generate ðZl9mlÞ ¼W where W � Inv�Gammaðnl=2,ðnlððml�xlÞ2þMSXlÞÞ=2Þ for l¼ 1, . . . ,k.

3.

Calculate weights of each generated sample with,

wJ ¼f JðxÞ

ðQtJ

i ¼ 1 giðmn

i ÞhiðZiÞÞ,

where f JðxÞ is the generalized fiducial density for the model with groupings J and giðmiÞ and hiðZiÞ are the densities fromdistributions described in steps 1, and 2.

4.

This process was repeated until we achieved the effective sample size calculated by ESSJ ¼ nJð1þðs2wJ Þw�2J Þ�1 where nJ is

the sample size for model J, s2wJ is the sample variance of the weights, and wJ is the sample mean of the weights.

5.

Lastly the weights were divided by the ESSJ.

6.

This process was repeated for all J.

Appendix B. Proof of Theorem 1

With constant variance: This proof will be done with the assumption of constant variance (i.e. Zi ¼ Zj for all i and j) andwithout using the weight function, wJðXÞ: To prove Theorem 1 we will show that p~J=pJ-0 for any ~JaJ. We will observe twocases. First, when ~J incorrectly groups means as equal. Second, when ~J does not incorrectly group the means but there aretoo many groups.

For the first case let J2 incorrectly group the means and J1 is the correct grouping. Thus, there are t1 groups in J1 and t2groups in J2. At least one of the means in J2 is incorrectly grouped. The subscript in the following calculations note theassociation with J1 or J2.

pJ2pJ1

pG N�t22� �

ðPt1

i ¼ 1 n01iMSX

01iÞðN�t1Þ=2Qt1

i ¼ 1ffiffiffiffiffiffin01i

pG N�t12� �

ðPt2

i ¼ 1 n02iMSX

02iÞðN�t2Þ=2Qt2

i ¼ 1ffiffiffiffiffiffin02i

p using Stirling’s formula

r ð2eÞðt2�t1Þ=2Nðt1�t2Þ=2

Qt1i ¼ 1

ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p ðPt1i ¼ 1 n01iMSX01iÞðN�t1Þ=2ðPt2

i ¼ 1 n02iMSX

02iÞðN�t2Þ=2

WLOG assume U1 2 J2 is an incorrect grouping

r ð2eÞðt2�t1Þ=2Nðt1�t2Þ=2

Qt1i ¼ 1

ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p Zðt2�t1Þ=20Pt1

i ¼ 1 n01iZ0ð1þOð1ÞÞ

Z0

� �ðN�t1Þ=2n021

Znð1þOð1ÞÞZ0

þPt2

i ¼ 2 n02iZ0ð1þOð1ÞÞ

Z0

� �ðN�t2Þ=2where Zn4Z because of the incorrect grouping

r ð2eÞðt2�t1Þ=2Qt1

i ¼ 1ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p Zðt2�t1Þ=20 1þ r4 Zn

Z0�1

� �� �ðN�t1Þ=21þr Z

n

Z0�1

� �h ic

� �ðN�t2Þ=2 Eventually a:s:-0 a:s:

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 891

for

c¼1þ 1þr Z

n

Z0�1

� �� �2 1þr Z

n

Z0�1

� �� �and 0oroPi2U1 biðPki ¼ 1 biÞ�1o1.

The second case when J2 is a valid model with too many groups and J1 is the correct grouping. Thus, there are t1 groupsin J1, t2 groups in J2 and t24t1. Let

J1 ¼U119U129 � � � 9U1t1 ,

J2 ¼U219U229 � � � 9U2t2 ,

where

U1i ¼[

k2Ki

U2k

and Ki � f1, . . . ,t2g for at least one U1i.

pJ2pJ1

pG N�t22� �

ðPt1

i ¼ 1 n01iMSX

01iÞðN�t1Þ=2Qt1

i ¼ 1ffiffiffiffiffiffin01i

pG N�t12� �

ðPt2

i ¼ 1 n02iMSX

02iÞðN�t2Þ=2Qt2

i ¼ 1ffiffiffiffiffiffin02i

pr ð2eÞ

ðt2�t1Þ=2Nðt1�t2Þ=2Qt1

i ¼ 1ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p ðPt1i ¼ 1 SSX01iÞðN�t1Þ=2ðPt2

i ¼ 1 SSX02iÞðN�t2Þ=2

r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2

Qt1i ¼ 1

ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p 1�Pt1

i ¼ 1P

k2Ki n2k16Z log log N

n2k

� �NZ

0@

1A�N=2

eventually a:s: using the law of iterated logarithms

r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2

Qt1i ¼ 1

ffiffiffiffiffiffin01i

pQt2i ¼ 1

ffiffiffiffiffiffin02i

p 1�R log log NN

� ��N=2WLOG assume that U1t1 ¼U2ðt2�1Þ [ U2t2 and U1i ¼U2i for all other i

r ð2eÞðt2�t1Þ=2ð2ZÞð�t1þ t2Þ=2bffiffiffiffiffiffiffiffi

n02t2

q 1�R loglog NN

� ��N=2-0 a:s:

for some R41 and b40.Therefore, we have shown that p~J=pJ-0 for any

~JaJ where J is the correct grouping. This completes the proof. &With non-constant variance: This proof will not assume constant variance. Additionally, the proof will be done without

the use of the weight function. The generalized fiducial density for any J without the weight function is

~f JðxÞ ¼Vx,JQki ¼ 1 Zi

1

ð2pÞn1=2Zn1=21exp � 1

2Z1

Xn1j ¼ 1ðx1j�m1Þ

2

8<:

9=; � � � � 1ð2pÞnk=2Znk=2k exp �

1

2Zk

Xnkj ¼ 1ðxkj�mkÞ

2

8<:

9=;

¼ Vx,JQk

i ¼ 1 Z�ni=2�1i

ð2pÞN=2exp �1

2

Xki ¼ 1

niððmi�xiÞ

2þMSXiÞZi

( ):

If we could integrate this function we could calculate the probabilities, P(J), directly. However, we cannot fully integrate itso we will apply different techniques. Note, that J¼U19U29 � � � 9UtJ where Ui ¼ ri1 ri2 � � � riui are the indexes of equal means

l¼ ðmr11 , . . .mr1u1 ,mr21 , . . .mr2u2 , . . . ,mrt1 , . . .mr2utJÞ ¼ mn1, . . .m

n

1|fflfflfflfflfflffl{zfflfflfflfflfflffl}u1 replictes

,mn2, . . .mn

2|fflfflfflfflfflffl{zfflfflfflfflfflffl}u2 replictes

, . . . ,mntJ , . . .mn

tJ|fflfflfflfflfflffl{zfflfflfflfflfflffl}utJ replictes

0BBB@

1CCCA:

Without loss of generality we will assume l¼ ðm1, . . . ,mkÞ ¼ ðln1, . . . ,lntJ Þ ¼ ln. Notice that

pJ ¼ZX

f JðxÞ dx¼ p�N=2Yki ¼ 1

n�ni=2i Gni2

� �h iZR

tJ

Vx,JYki ¼ 1ððmi�xiÞ

2þMSXiÞ�ni=2 dln:

Because Vx,J is dependent on mni we will bound this value. It is clear that Vx,J 4c1 for some c140. We could re-write Vx,J as

Vx,J ¼YtJi ¼ 1

mnðui�1Þi

!V1;1þ

XtJj ¼ 1

mnðuj�2ÞjYtJ

i ¼ 1,iajmnðui�1Þi

0@

1AV2,jþ � � � þVz,1

������������,

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895892

where Vi,j are averages over a function of the data. If ui is even then 9mnðui�1Þi 9rmnuii þ1 and if ui is odd then

9mnðui�1Þi 9¼ mnðui�1Þi . Regardless of the ui the same technique will be used. Thus, without loss of generality we will assume

that ui is odd for all i:

Vx,J rYtJi ¼ 1

mnðui�1Þi

!9V1;19þ

XtJj ¼ 1

ðmnðuj�1Þj þ1ÞYtJ

i ¼ 1,iajmnðui�1Þi

0@

1A9V2,j9þ � � � þ9Vz,19

rYki ¼ 1ððmi�xiÞ

2þMSXiÞðui�1Þ=2 !

9V ð1Þ9þ9V ð2Þ9,

where V ð1Þ and V ð2Þ are averages over the data, xi, and MSXi. Thus Vð1Þ and V ð2Þ will converge to some constant almost surely

by the strong law of large numbers.A lower bound for pJ is

pJ Zp.J ¼ c1p

�N=2Yki ¼ 1

n�ni=2i Gni2

� �h iZR

tJ

Yki ¼ 1ððmi�xiÞ

2þMSXiÞ�ni=2dln:

An upper bound for pJ is

pJ rp�N=2Yki ¼ 1

n�ni=2i Gni2

� �h iZR

tJ

Qki ¼ 1ððmi�xiÞ

2þMSXiÞðui�1Þ=2� �

9V ð1Þ9þ9V ð2Þ9Qki ¼ 1ððmi�xiÞ

2þMSXiÞni=2dln

rp�N=2Yki ¼ 1

n�ni=2i Gni2

� �h iZR

tJ

9V ð1Þ9Qki ¼ 1ððmi�xiÞ

2þMSXiÞðni�ui�1Þ=2þ

9V ð2Þ9Qki ¼ 1ððmi�xiÞ

2þMSXiÞni=2dln

rp�N=2Yki ¼ 1

n�ni=2i Gni2

� �h iZR

tJ

c29Vð1Þ9Qk

i ¼ 1ððmi�xiÞ2þMSXiÞðni�ui�1Þ=2

dln ¼ pmJ

for some c240.Because we cannot integrate with respect to ln we observe

gJðlnÞ ¼Yki ¼ 1ððmi�xiÞ

2þMSXiÞ�ni=2

with the transformations of

mni ¼mniffiffiffi

np þmni0 for i¼ 1, . . . ,tJ

and the substitutions of

xi ¼ mi0þZi1ffiffiffiffi

nip for i¼ 1, . . . ,k

and

MSXi ¼ Zi0þZi2ffiffiffiffi

nip for i¼ 1, . . . ,k,

where mi0 and Z10 are the true mean and variance for treatment i and ðZi1,Zi2Þ �Nð0,SÞ. Thus,

gJðmnÞ ¼ n�tJ=2Yki ¼ 1

miffiffiffinp þDi�

Zi1ffiffiffiffinip

� �2þZi0þ

Zi2ffiffiffiffinip

!�ni=2,

where m and mn follows the same structure as l and ln previously stated and Di ¼ mnj0�mi0 for i 2 Uj. We will see that mn

i

converges point-wise to a normal distribution.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 893

Taylor expanding logðgJðmnÞÞ we will get

logðgJðmnÞÞ ¼�tJ2

logðnÞþXki ¼ 1

� bin logðZi0þD2i Þ

2�

biffiffiffinp

2Di mi� Zi1ffiffiffibip

� �þ Zi2ffiffiffi

bip

� �2ðZi0þD

2i Þ

2664

þbi 2Di mi� Zi1ffiffiffi

bip

� �þ Zi2ffiffiffi

bip

� �24ðZi0þD

2i Þ

2�

bi mi� Zi1ffiffiffibip

� �2ð2Zi0þD

2i ÞþOðn�1=2Þ

37775:

Clearly if J is correctly grouping the means then Di ¼ 0. Otherwise we will select mnj0 such thatXi2Uj

biDiðZi0þD

2i Þ¼ 0

for all j¼ 1, . . . ,tJ . Thus,

logðgJðmnÞÞ ¼�tJ2

logðnÞþXki ¼ 1

� bin logðZi0þD2i Þ

2þ

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

26664

þbi 2Di mi� Zi1ffiffiffi

bip

� �þ Zi2ffiffiffi

bip

� �24ðZi0þD

2i Þ

2�

bit mi� Zi1ffiffiffibip

� �2ð2Zi0þD

2i ÞþOðn�1=2Þ

37775:

Or

gJðmnÞ ¼1

ntJ=2Qk

i ¼ 1ðZi0þD2i Þ

bin=2exp

Xki ¼ 1

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

( )exp �

XtJi ¼ 1

1

2s2Z,iðmni �zZ,iÞ

2þCZþOðn�1=2Þ( )

, ðB:1Þ

where zZ,i and s2Z,i are the appropriate mean and variance of mn

i dependent on the Zij values and CZ is the constant used incompleting the square. Clearly gJðmnÞCn converges to a normal density for the appropriate normalizing constant, Cn.

Lemma 1. Let

hJ,nðmnÞ ¼ CngJðmnÞ

for the previously described Cn, then hJ,nðmnÞrkJðmnÞ where k is integrable.

Proof. First, square the function, g, without the n power:

Yki ¼ 1

miffiffiffinp þDi�

Zi1ffiffiffiffinip

� �2þZi0þ

Zi2ffiffiffiffinip

!�bi:

For each Uj we are looking at a function in mnj that has at most uj peaks (local maximum) and at most uj�1 valleys (localminimum). For instance, for U1 we are looking at the function

Yi2U1

mn1ffiffiffinp þDi�

Zi1ffiffiffiffinip

� �2þZi0þ

Zi2ffiffiffiffinip

!�bi:

For large enough n this function has a unique global maximum with probability 1. Our mnj0 is close to this maximum(within cj0n

�1=2 where the cj0 depends on the Z’s). Next, we re-scale the function so that the global maximum is 1.The other local maximums will be cjln

1=2 away where l¼ 1, . . . ,ðuj�1Þ. Here cjl depends on the distances betweenmaximums.

The fraction between the value of the function’s local and global maximums is either a constant ðo1 if Zs0aZr0 for alls,r 2 Uj) or 1�cjln�1=2 if Zs ¼ Zr for all s,r 2 Uj, in which case the difference comes from the Z’s.

Finally, if we raise the function to the power n. The global maximum goes to 1. At the local maximum we have a height ofexpf�cjln1=2g, i.e., the local maximum is located at the point ðcjln1=2,expf�cjln1=2gÞ which is well below the Cauchy densityof ðcjln1=2,cð1þc2jlnÞ

�1Þ for some constant c.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895894

Finally, notice that if there was a point for which our function was larger than a bounding Cauchy it would be at the localmaximum. This is because the function decays from its local and global maxima faster than the Cauchy distribution.

From Eq. (B.1) we can see mni converges point-wise to a normal distribution and Lemma 1 allows us to bound gJðmnÞ for

all n. Therefore, we can calculate the asymptotic behavior of pmJ and p.J . Observe,

p.J ¼p�N=2

Qki ¼ 1 n

�ni=2i G

ni2

h ic1

ntJ=2ðZi0þD2i Þ

bin=2exp

Xki ¼ 1

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

( )

�ZR

tJ

exp

bi 2Di mi� Zi1ffiffiffibip

� �þ Zi2ffiffiffi

bip

� �24ðZi0þD

2i Þ

2�

bi mi� Zi1ffiffiffibip

� �2ð2Zi0þD

2i ÞþOðn�1=2Þ

8>>><>>>:

9>>>=>>>;dm

n

¼p�N=2

Qki ¼ 1 n

�ni=2i G

ni2

h ic1

ntJ=2ðZi0þD2i Þ

bin=2exp

Xki ¼ 1

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

( )B1,n

and a similar calculation produces

pmJ ¼p�N=2

Qki ¼ 1 n

�ni=2i G

ni2

h ic2V

ð1Þ

ntJ=2ðZi0þD2i Þðbin�ui�1Þ=2

expXki ¼ 1

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

( )

�ZR

tJ

exp

bi 2Di mi� Zi1ffiffiffibip

� �þ Zi2ffiffiffi

bip

� �24ðZi0þD

2i Þ

2�

bi mi� Zi1ffiffiffibip

� �2ð2Zi0þD

2i ÞþOðn�1=2Þ

8>>><>>>:

9>>>=>>>;dm

n

¼p�N=2

Qki ¼ 1 n

�ni=2i G

ni2

h ic2V

ð1Þ

ntJ=2ðZi0þD2i Þðbin�ui�1Þ=2

expXki ¼ 1

ffiffiffiffiffiffiffibin

pð2DiZi1�Zi2Þ

2ðZi0þD2i Þ

( )B2,n,

where Bi,n is the constant that comes from integration of the normal density and Bi,n-Bi by Lemma 1.To prove that PðJÞ-1 as n-1 we will observe p~J=pJ rpm~J =p

.J -0 for any

~JaJ: Like the previous proof there are twocases. First, when ~J incorrectly groups means as equal. Second, when ~J does not incorrectly group the means but there aretoo many groups.

For the first case let J2 incorrectly group the means and J1 is the correct grouping. Thus, there are t1 groups in J1 and t2groups in J2. At least one of the means in J2 is incorrectly grouped and at least one of the Dia0. Equivalently Di ¼ 0 for thegrouping in J1.

pmJ2p.J1¼ c2V

ð1ÞðZi0Þbin=2

c1ðZi0þD2i Þðbin�ui�1Þ=2

expPk

i ¼ 1

ffiffiffiffiffibinp

ð2DiZi1�Zi2Þ2ðZi0þD

2i Þ

� �B2,n

expPk

i ¼ 1�ffiffiffiffiffibinp

Zi22Zi0

� �B1,n

-0 a:s:

The second case when J2 is a valid model with too many groups and J1 is the correct grouping. Thus, there are t1 groupsin J1, t2 groups in J2 and t24t1.

pmJ2p.J1¼ n

t1=2c2Vð1ÞðZi0Þ

bin=2

nt2=2c1ðZi0Þðbin�ui�1Þ=2

expPk

i ¼ 1�ffiffiffiffiffibinp

Zi22ðZÞ

� �B2,n

expPk

i ¼ 1�ffiffiffiffiffibinp

Zi22Zi0

� �B1,n

¼ c2Vð1ÞðZi0Þ

bin=2

nðt2�t1Þ=2c1ðZi0Þðbin�ui�1Þ=2

B2,nB1,n

-0 a:s:

Thus we have shown that PðJÞ-1.The same convergence results for both constant a non-constant variance hold if the weight function is included. &

References

Abdel-Karim, A.H., 2005. Applications of Generalized Inference.Fisher, R.A., 1930. Inverse probability. Proceedings of the Cambridge Philosophical Society xxvi, 528–535.Gopalan, R., Berry, D.A., 1998. Bayesian multiple comparisons using Dirichlet process priors. Journal of American Statistical Association 93, 1130–1139.Hannig, J., 2009a On Asymptotic Properties of Generalized Fiducial Inference for Discretized Data. Technical Report UNC/STOR/09/02, Department of

Statistics and Operations Research, The University of North Carolina at Chapel Hill.Hannig, J., 2009b. On generalized fiducial inference. Statistica Sinica 19, 491–544.Hannig, J., Iyer, H.K., Patterson, P., 2006. Fiducial generalized confidence intervals. Journal of the American Statistical Association 101, 254–269.

D.V. Wandler, J. Hannig / Journal of Statistical Planning and Inference 142 (2012) 878–895 895

Hannig, J., Lee, T.C., 2009. Generalized fiducial inference for wavelet regression. Biometrika 96, 847–860.Lindley, D.V., 1958. Fiducial distributions and Bayes’ theorem. Journal of the Royal Statistical Society Series B 20, 102–107.Steele, R.D., Torrie, J.H., 1980. Principles and Procedures of Statistics—A Biomedical Approach, second ed. McGraw Hill, New York.Weeranhandi, S., 1993. Generalized confidence intervals. Journal of the American Statistical Association 88, 899–905.Zabell, S.L., 1992. R.A. Fisher and the fiducial argument. Statistical Science 7, 369–387.

A fiducial approach to multiple comparisonsIntroductionGeneralized fiducial inferenceOverview

Main resultsStructural equation with constant varianceStructural equation with non-constant varianceSimulationsConstant varianceNon-constant variance

Asymptotic resultsExamplesSimulated dataClover plant data

ConclusionAcknowledgmentsImportance sampling algorithmProof of Theorem 1References