1
Multiple ComparisonProcedures
Once we reject H0: ==...c in favor of H1: NOT all ’s are equal, we don’t yet know the way in which they’re not all equal, but simply that they’re not all the same. If there are 4 columns, are all 4 ’s different? Are 3 the same and one different? If so, which one? etc.
2
These “more detailed” inquiries into the process are called MULTIPLE COMPARISON PROCEDURES.
Errors (Type I):We set up “” as the significance level for a hypothesis test. Suppose we test 3 independent hypotheses, each at = .05; each test has type I error (rej H0 when it’s true) of .05. However, P(at least one type I error in the 3 tests) = 1-P( accept all ) = 1 - (.95)3 .14 3, given true
3
In other words, Probability is .14 that at least one type one error is made. For 5 tests, prob = .23.Question - Should we choose = .05, and suffer (for 5 tests) a .23 OVERALL Error rate (or “a” or experimentwise)?
OR
Should we choose/control the overall error rate, “a”, to be .05, and find the individual test by 1 - (1-)5 = .05, (which gives us = .011)?
4
The formula 1 - (1-)5 = .05
would be valid only if the tests are independent; often they’re not.
[ e.g., 1=22= 3, 1= 3
IF accepted & rejected, isn’t it more likely that rejected? ]
1 2
21
3
3
5
When the tests are not independent, it’s usually very difficult to arrive at
the correct for an individual test so that a specified value results for the
overall error rate.
6
Categories of multiple comparison tests
- “Planned”/ “a priori” comparisons (stated in advance, usually a linear combination of the column means equal to zero.)
- “Pairwise” comparisons (every column mean compared with each other column mean)
- “Post hoc”/ “a posteriori” comparisons (decided after a look at the data - which comparisons “look interesting”)
7
(Pairwise comparisons are traditionally considered as “post hoc” and not “a priori”, if one needs to categorize all comparisons into one of the two groups)
There are many multiple comparison procedures. We’ll cover only a few.
Method 1: Do a series of pairwise t-tests, each with specified value (for individual test).
This is called “Fisher’s LEAST SIGNIFICANT DIFFERENCE” (LSD).
8
Example: Broker StudyExample: Broker Study
A financial firm would like to determine if brokers they use to execute trades differ with respect to their ability to provide a stock purchase for the firm at a low buying price per share. To measure cost, an index, Y, is used.
Y=1000(A-P)/AwhereP=per share price paid for the stock;A=average of high price and low price per share, for the day.
“The higher Y is the better the trade is.”
9
}1
1235-112
5 6
27
1713117
17 12
381743
7 5
524131418141917
R=6
CoL: broker
421101512206
14
Five brokers were in the study and six trades were randomly assigned to each broker.
10
= .05, FTV = 2.76
(reject equal column MEANS)
Source SSQ df MSQ FCol
Error
640.8
530
4
25
160.2
21.2
7.56
“MSW”
11
0
For any comparison of 2 columns,
/2/2
CL Cu
Yi -Yj
AR: 0+ t/2 x MSW x 1 + 1
ninj
25 df(ni = nj = 6, here)
SQ Root of Pooled Variance, “s2”, perhaps, in earlier class in basic statistics
:p
12
In our example, with=.05
0 2.060 (21.2 x 1 + 1 )0 5.48
6 6
This value, 5.48 is called the Least Significant Difference (LSD).
When same number of data points, R, in each column, LSD = t/2 x 2xMSW
.R
13
Col: 3 1 2 4 5 5 6 12 14 17
Now, rank order and compare:
Underline Diagram
14
Step 1: identify difference > 5.48, and mark accordingly:
5 6 12 14 173 1 2 4 5
2: compare the pair of means within each subset:Comparison difference vs. LSD
3 vs. 12 vs. 42 vs. 54 vs. 5
**
*
<<<<
* Contiguous; no need to detail
5
15
Conclusion : 3, 1 2, 4, 5
Can get “inconsistency”: Suppose col 5 were 18:
3 1 2 4 5 5 6 12 14 18
Now: Comparison |difference| vs. LSD3 vs. 12 vs. 42 vs. 54 vs. 5
* *
*
<<><
Conclusion : 3, 1 2 4 5 ???
6
16
• Broker 1 and 3 are not significantly different but they are significantly different to the other 3 brokers.
Conclusion : 3, 1 2 4 5
• Broker 2 and 4 are not significantly different, and broker 4 and 5 are not significantly different, but broker 2 is different to (smaller than) broker 5 significantly.
17
MULTIPLE COMPARISON TESTING
AFS BROKER STUDYBROKER ----> 1 2 3 4 5TRADE 1 12 7 8 21 24 2 3 17 1 10 13 3 5 13 7 15 14 4 -1 11 4 12 18 5 12 7 3 20 14 6 5 17 7 6 19
COLUMN MEAN 6 12 5 14 17
ANOVA TABLE
SOURCE SSQ DF MS Fcalc
BROKER 640.8 4 160.2 7.56
ERROR 530 25 21.2
18
Using SPSS Variable Score By Variable Broker
Analysis of Variance
Sum of Mean F F Source D.F. Squares Squares Ratio Prob.
Between Groups 4 640.8000 160.2000 7.5566 .0004Within Groups 25 530.0000 21.2000Total 29 1170.8000
19
Fisher’s LSD USING SPSS 5.0 - MAC Variable Score By Variable Broker
Multiple Range Tests: LSD test with significance level .05
The difference between two means is significant ifMEAN(J)-MEAN(I) >= 3.2558 * RANGE * SQRT(1/ N(I) + 1/ N(J))
with the following value(s) for RANGE: 2.91
(*) Indicates significant differences which are shown in the lower triangle
G G G G G r r r r r p p p p p
3 1 2 4 5 Mean Broker
5.0000 Grp 3 6.0000 Grp 1 12.0000 Grp 2 * * 14.0000 Grp 4 * * 17.0000 Grp 5 * *
Subset 1Group Grp 3 Grp 1Mean 5.0000 6.0000- - - - - - - - - - - - - - - - -Subset 2Group Grp 2 Grp 4 Grp 5Mean 12.0000 14.0000 17.0000- - - - - - - - - - - - - - - - - - - - - - - -
20
USING WINDOWS 8.0
1=column of interest, 2=compared column, 3=difference, 4=std. error, 5=p-value
6, 7 = 95 confidence limits
(1) (2) (3) (4) (5) (6) (7)LSD 1 2 -6.00* 2.658 .033 -11.47 -.53
3 1.00 2.658 .710 -4.47 6.474 -8.00* 2.658 .006 -13.47 -2.535 -11.00* 2.658 .000 -16.47 -5.53
2 1 6.00* 2.658 .033 .53 11.473 7.00* 2.658 .014 1.53 12.474 -2.00 2.658 .459 -7.47 3.475 -5.00 2.658 .072 -10.47 .47
3 1 -1.00 2.658 .710 -6.47 4.472 -7.00* 2.658 .014 -12.47 -1.534 -9.00* 2.658 .002 -14.47 -3.535 -12.00* 2.658 .000 -17.47 -6.53
4 1 8.00* 2.658 .006 2.53 13.472 2.00 2.658 .459 -3.47 7.473 9.00* 2.658 .002 3.53 14.475 -3.00 2.658 .270 -8.47 2.47
5 1 11.00* 2.658 .000 5.53 16.472 5.00 2.658 .072 -.47 10.473 12.00* 2.658 .000 6.53 17.474 3.00 2.658 .270 -2.47 8.47
21
Fisher's pairwise comparisons (Minitab)
Family error rate = 0.268
Individual error rate = 0.0500
Critical value = 2.060 t_(/2)
Intervals for (column level mean) - (row level mean)
1 2 3 4
2 -11.476
-0.524
3 -4.476 1.524
6.476 12.476
4 -13.476 -7.476 -14.476
-2.524 3.476 -3.524
5 -16.476 -10.476 -17.476 -8.476
-5.524 0.476 -6.524 2.476
Minitab: Stat>>ANOVA>>one way anova then click “comparisons”.
22
In the previous procedure, each individual comparison has error rate =.05. The overall error rate is, were
the comparisons independent, 1- (.95)10= .401.
However, they’re not independent.
Method 2: A procedure which takes this into account and pre-sets the overall error rate is “TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE TEST ”.
23
Tukey’s method works in a similar way to Fisher’s LSD, except that the “LSD” counterpart (“HSD”) is not
t/2 x MSW x 1 + 1ni nj
t/2 x 2xMSWR
=or, for equal number of data points/col( ) ,
but tuk X 2xMSW ,R
where tuk has been computed to take into account all the inter-dependencies of the different comparisons.
/2
24
HSD = tuk/2x2MSW R
________________________________________
A more general approach is to write
HSD = q/2xMSW R
where q/2 = tuk/2 x2
--- q = (Ylargest - Ysmallest) / MSW R
---- probability distribution of q is called the
“Studentized Range Distribution”.
--- q = q(c, df), where c =number of columns,
and df = df of MSW
25
q table
26
With c = 5 and df = 25,from table:
q = 4.16 (between 4.10 and 4.17)tuk = 4.16/1.414 = 2.94
Then,
HSD = 4.16
alsox
27
In our earlier example:
Rank order:
3 1 2 4 5
5 6 12 14 17
(No differences [contiguous] > 7.82)
28
Comparison |difference| >or< 7.823 vs. 13 vs. 23 vs. 43 vs. 51 vs. 21 vs. 41 vs. 52 vs. 42 vs. 54 vs. 5
* <<>><>><<<
912*8
11*5*
(contiguous)
7
3, 1, 2 4, 52 is “same as 1 and 3, but also same as 4 and 5.”
29
Tukey’s HSD (“LSD”) Mac
Variable Score By Variable Broker
Multiple Range Tests: Tukey-HSD test with significance level .05
The difference between two means is significant ifMEAN(J)-MEAN(I) >= 3.2558 * RANGE * SQRT(1/ N(I) + 1/ N(J))with the following value(s) for RANGE: 4.15
(*) Indicates significant differences which are shown in the lower triangle G G G G G r r r r r p p p p p
3 1 2 4 5 Mean Broker 5.0000 Grp 3 6.0000 Grp 1 12.0000 Grp 2 14.0000 Grp 4 * * 17.0000 Grp 5 * *
Subset 1Group Grp 3 Grp 1 Grp 2
Mean 5.0000 6.0000 12.0000- - - - - - - - - - - - - - - - - - - - - - - -Subset 2Group Grp 2 Grp 4 Grp 5
Mean 12.0000 14.0000 17.0000
30
Windows 8.0 format, with the same column meanings:
(1) (2) (3) (4) (5) (6) (7)Tukey HSD 1 2 -6.00 2.658 .192 -13.81 1.81
3 1.00 2.658 .995 -6.81 8.814 -8.00* 2.658 .043 -15.81 -.195 -11.00* 2.658 .003 -18.81 -3.19
2 1 6.00 2.658 .192 -1.81 13.813 7.00 2.658 .094 -.81 14.814 -2.00 2.658 .942 -9.81 5.815 -5.00 2.658 .353 -12.81 2.81
3 1 -1.00 2.658 .995 -8.81 6.812 -7.00 2.658 .094 -14.81 .814 -9.00* 2.658 .018 -16.81 -1.195 -12.00* 2.658 .001 -19.81 -4.19
4 1 8.00* 2.658 .043 .19 15.812 2.00 2.658 .942 -5.81 9.813 9.00* 2.658 .018 1.19 16.815 -3.00 2.658 .790 -10.81 4.81
5 1 11.00* 2.658 .003 3.19 18.812 5.00 2.658 .353 -2.81 12.813 12.00* 2.658 .001 4.19 19.814 3.00 2.658 .790 -4.81 10.81
For Tukey’s HSD, the Windows SPSS output also provides another format, called
“homogeneous Subsets” (it doesn’t provide it for Fisher’s LSD):
Broker N Subset 1 Subset 2 Subset 3Tukey HSD 3 6 5.00
1 6 6.002 6 12.00 12.004 6 14.005 6 17.00Sig. .094 .353
31
Tukey's pairwise comparisons (Minitab)
Family error rate = 0.0500
Individual error rate = 0.00706
Critical value = 4.15 q_(1-/2)
Intervals for (column level mean) - (row level mean)
1 2 3 4 2 -13.801 1.801 3 -6.801 -0.801 8.801 14.801 4 -15.801 -9.801 -16.801 -0.199 5.801 -1.199 5 -18.801 -12.801 -19.801 -10.801 -3.199 2.801 -4.199 4.801
32
Exercise: Drug StudyExercise: Drug StudyA drug company are developing two new drug formulations for treating flu, denoted as drug A and drug B. Two groups of 10 volunteers were taken drug A and drug B, respectively, and after three days, their responses (Y) were recorded. A placebo group was added to check the effectiveness of drugs. The larger the Y value is, the more effective the drug is. Here is the data: (MSE=1)
Drug A Drug B Placebo
Index i 1 2 3
Column mean -5.3 -6.1 -12.3
Sample size 10 10 10
33
LSD = t97.5%;27 df 2/10 = 2.052 2/10 = 0.9177
HSD = q97.5%;27 df 1/10 = 3.51 1/10 = 1.110
Placebo Drug B Drug A
Placebo Drug B Drug A
34
Method 3: Dunnett’s testDesigned specifically for (and incorporating the interdependenciesof) comparing several “treatments” to a “control.”
Example: 1 2 3 4 5
6 12 5 14 17
Col
} R=6CONTROL
Analog of LSD (=t1-/2 x 2 MSW )R = Dut1-/2 x 2 MSW
R
35
D tablep. 107
36
Dut1-/2 x 2 MSW/R
= 2.61 (2(21.2) )= 6.94
- Cols 4 and 5 differ from the control [ 1 ].- Cols 2 and 3 are not significantly differentfrom control.
6
In our example: 1 2 3 4 5 6 12 5 14 17
CONTROL
Comparison |difference| >or< 6.941 vs. 21 vs. 31 vs. 41 vs. 5
618
11
<< > >
37
DUNNETTDependent Variable: SCOREDunnett t (2-sided)
MeanDifference
(I-J)
Std.Error
Sig. 95% ConfidenceInterval
(I)BROKER
(J)BROKER
LowerBound
UpperBound
2 1 6.00 2.658 .103 -.93 12.933 1 -1.00 2.658 .987 -7.93 5.934 1 8.00* 2.658 .020 1.07 14.935 1 11.00* 2.658 .001 4.07 17.93
* The mean difference is significant at the .05 level.
38
Dunnett's comparisons with a control (Minitab)
Family error rate = 0.0500Individual error rate = 0.0152
Critical value = 2.61 Dut_1-/2
Control = level (1) of broker
Intervals for treatment mean minus control mean
Level Lower Center Upper --+---------+---------+---------+-----2 -0.930 6.000 12.930 (---------*--------) 3 -7.930 -1.000 5.930 (---------*--------) 4 1.070 8.000 14.930 (--------*---------) 5 4.070 11.000 17.930 (---------*---------) --+---------+---------+---------+----- -7.0 0.0 7.0 14.0
39
This procedure provides a subset of treatments that cannot distinguished from the best. The probability of that the “best” treatment is included in this subset is controlled at 1-.
Method 4: MCB Procedure (Compare to the best)
*Assume that larger is better.
40
STEP 1: Calculate the following for all index i
)11
(),1li
ilnn
MSEvcDM
)(max and .. jiji yy
where l (not i) is the group of which mean reaches )(max .jij y
41
STEP 2: Conduct tests
The treatment i is included in the best subset if
.)](max[ .. iljijii MyyD
42
Drug A Drug B Placebo
Index i 1 2 3
Column mean -5.3 -6.1 -12.3
-6.1 -5.3 -5.3.max jij y
Di 0.8 -0.8 -7
894.0)10
1
10
1(12
)10
1
10
1()27,2(%5
MSEDMil
What drugs are in the best subset?
(Given MSE = 1.)
43
Identify the subset of the best brokers
Hsu's MCB (Multiple Comparisons with the Best)
Family error rate = 0.0500
Critical value = 2.27
Intervals for level mean minus largest of other level means
Level Lower Center Upper ---+---------+---------+---------+----1 -17.046 -11.000 0.000 (------*-------------) 2 -11.046 -5.000 1.046 (-------*------) 3 -18.046 -12.000 0.000 (-------*--------------) 4 -9.046 -3.000 3.046 (------*-------) 5 -3.046 3.000 9.046 (-------*------) ---+---------+---------+---------+---- -16.0 -8.0 0.0 8.0
Brokers 2, 4, 5
44
----Post Hoc comparisons*F test for contrast (in “Orthogonality”)
*Scheffe test (p.108; skipped)
To test all linear combinations at once. Very conservative; not to be used for pairwise
comparisons.
----A Priori comparisons* covered later in chapter on
“Orthogonality”