Post Hoc Tests

What is a Post Hoc Test?

Review:– Adjusting Alpha Level– Multiple A Priori Comparisons

What makes a test Post Hoc?– Many tests could be Post Hoc… But, there are set

Post Hoc tests

Studentized Range Statistic q

2tq

Studentized Range Statistic q

nMS

yyq

error

SLr

Independent Groups

Smallest mean

Largest meanLy

Sy

Example

8.2 8.2 11.8

1y 2y 3y

5n

12errordf

12errorMS

1# stepsr

06.275.1

6.3

54.15

2.88.113

q

)05.,12,3( dfrqcritical value

= 3.77

Fail to RejectNote: arrange means in ascending order!

Studentized Range Statistic q

q’s can tell us where differences are (more specific than F)

Solving q’s is just like solving t’s

But we solve a lot of q’s… can we speed things up?

Solving for the Smallest Significant Difference

n

MS

yyq

error

SLr

n

MSqyy errorrSL

5

4.1577.3 61.6

Example

8.2 8.2 11.8

1y 2y 3y

)05.,12,3( dfrqcritical value

= 3.77

5n

12errorMS

Solving for the Smallest Significant Difference

Solving for the smallest significant difference will help us make quicker comparisons

But we still need a way to organize things nicely…

Newman-Kewls

T1 T2 T3 r

T1 0 3.6 3 6.61

T2 3.6 2 5.41

T3

r

smallest difference required was 6.61

If 2 steps

5

4.1508.32 SL yy 41.5

r smallest significant difference

61.63

41.52

8.2 8.2 11.8

1y 2y 3y

Example

A Better Newman-Kewls Example

Example

2 3 3 9 10

1y 2y 3y 4y 5y

9MSerror 9n

40df

)1(86.22 )1(44.33

)1(79.34 )1(04.45

A Better Newman-Kewls Example

T1 T2 T3 T4 T5 r

T1 1 1 7 8 5 4.04

T2 0 6 7 4 3.79

T3 6 7 3 3.44

T4 1 2 2.86

T5

r T1 T2 T3 T4 T5

T1 * *

T2 * *

T3 * *

T4

T5

Read Right to Left UNTIL

1. The row is completed2. A nonsignificant difference is found

3. Reaching a column which was nonsignificant on the previous row

Newman-Kewls Summarized

Newman-Kewls tables help organize your q’s

When doing a set of post hoc comparisons it’s best to use a Newman-Kewls table

Unequal N’s

n

MS

yyq

error

SLr

n

MSqyy errorrSL

Tukey-Kramer

n

MSerror

2S

error

L

error

nMS

nMS

Replace

with

ny larger L

ny smaller S

Unequal N’s

Behrens-Fisher

),(05.0 fdrSLr qyy 2

22

S

S

L

L

nS

nS

11

2222

222

S

S

S

L

L

L

S

S

L

L

n

nS

n

nS

nS

nS

fd*

Each particular pairing of means must be examined with a different critical value and their own

q2S

Thus, the smallest significant difference will vary even for a given

*

r

A Problem with q So Far…

Why are we doing q’s anyway? Why not do t’s instead?

But is q really controlling our alpha level?

NO!

Using q’s will give use more Type I Errors

Trying to fix q

Tukey's HSD

N-K except

(largest) always rHSD qq

If there are 4 means, all differences are treated as 4 steps.

2rk

WSD

qqq

means of # kr = # of steps between the two

means to be compared.

Tukey's WSD

What Happens to Alpha Level? Power?

Tukey’s HSD and WSD

T1 T2 T3 T4 T5 r

T1 1 1 7 8 5 4.04

T2 0 6 7 4 3.79

T3 6 7 3 3.44

T4 1 2 2.86

T5

r

Use Tukey’s WSD, not normal method for q

Back to Post Hoc in General

What is a post hoc test again?– What are the real issues with Post Hoc tests?

Alpha and Power…

q is just one type of post hoc (one way to balance alpha and power), what are others?

Dunnett’s Control vs. Treatment

run standard t and use )(MS Table edt or, solve for critical difference (CV)

n

MStCV edyy

jTc

2 )(

11 , 30 , 4 , 8 , 1021

nMSyyy eTTc

Go to Table for dt ),( edfk

32.2dt

11

)30(232.2CV

42.5)34.2(32.2

21 Tc yy

62 Tc yy 05.0p

ns

*

Example

Pros…? Cons…?

Sheffé’s Test

Linear contrast

MS(contrast)

MS(error)

F =

To evaluate

1) consult F table and find critical value F.05 (k-1, dferror) (CV)

2) multiply CV by (k-1). (new CV)

It sets the family-wise Type-I Error rate ( in our case) for ALL possible linear contrasts, not merely the pair-wise comparisons.

05.

Don’t use when only doing pair-wise, because it will be overly conservative.

Post Hoc Summary

When to use what…– q in most situations… but use Tukey’s WSD for critical

value Put things in a Newman-Kewls table when N’s are unequal, use Tukey’s correction

– Dunnett’s when you have one control and multiple treatments

– Sheffé’s ONLY when you are doing complex comparisons (i.e., contrasts)

Post Hoc Summary

Be aware of the alpha level and power issues…– Why can’t we have a perfect test (i.e., low alpha

level and high power)?

– How does Tukey’s WSD and HSD relate to this?– How does Dunnett’s relate to this?– How does Sheffé’s relate to this?