Multiple comparisons

- multiple pairwise tests

- orthogonal contrasts

- independent tests

- labelling conventions

Card example number 1

Multiple tests

Problem:

Because we examine the same data in multiple comparisons, the result of the first comparison affects our expectation of the next comparison.

Multiple tests

ANOVA shows at least one different, but which one(s)?

significant

Not significant

significant

•T-tests of all pairwise combinations

Multiple tests

T-test: <5% chance that this difference was a fluke…

affects likelihood of finding a difference in this pair!

Multiple tests

Solution:Make alpha your overall “experiment-wise” error rate

affects likelihood (alpha) of finding a difference in this pair!

T-test: <5% chance that this difference was a fluke…

Multiple tests

Solution:Make alpha your overall “experiment-wise” error rate

e.g. simple Bonferroni:Divide alpha by number of tests

Alpha / 3 = 0.0167

Alpha / 3 =0.0167

Alpha / 3 = 0.0167

Card example 2

Orthogonal contrastsOrthogonal = perpendicular = independent

Contrast = comparison

Example. We compare the growth of three types of plants: Legumes, graminoids, and asters.

These 2 contrasts are orthogonal:

1. Legumes vs. non-legumes (graminoids, asters) 2. Graminoids vs. asters

Trick for determining if contrasts are orthogonal:

1. In the first contrast, label all treatments in one group with “+” and all treatments in the other group with “-” (doesn’t matter which way round).

Legumes Graminoids Asters + - -

Trick for determining if contrasts are orthogonal:

1. In the first contrast, label all treatments in one group with “+” and all treatments in the other group with “-” (doesn’t matter which way round).

2. In each group composed of t treatments, put the number 1/t as the coefficient. If treatment not in contrast, give it the value “0”.

Legumes Graminoids Asters +1 - 1/2 -1/2

Trick for determining if contrasts are orthogonal:

1. In the first contrast, label all treatments in one group with “+” and all treatments in the other group with “-” (doesn’t matter which way round).

2. In each group composed of t treatments, put the number 1/t as the coefficient. If treatment not in contrast, give it the value “0”.

3. Repeat for all other contrasts.

Legumes Graminoids Asters +1 - 1/2 -1/2 0 +1 -1

Trick for determining if contrasts are orthogonal:

4. Multiply each column, then sum these products.

Legumes Graminoids Asters +1 - 1/2 -1/2 0 +1 -1

0 - 1/2 +1/2

Sum of products = 0

Trick for determining if contrasts are orthogonal:

4. Multiply each column, then sum these products.

5. If this sum = 0 then the contrasts were orthogonal!

Legumes Graminoids Asters +1 - 1/2 -1/2 0 +1 -1

0 - 1/2 +1/2

Sum of products = 0

What about these contrasts?

1. Monocots (graminoids) vs. dicots (legumes and asters).

2. Legumes vs. non-legumes

Important!

You need to assess orthogonality in each pairwise combination of contrasts.

So if 4 contrasts:

Contrast 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, 3 and 4.

How do you program contrasts in JMP (etc.)?

Treatment SS

}Contrast 2

}Contrast 1

How do you program contrasts in JMP (etc.)?

Normal treatments

Legume 1 1Legume 1 1Graminoid 2 2Graminoid 2 2Aster 3 2Aster 3 2

SStreat 122 67Df treat 2 1MStreat 60

MSerror 10Df error 20

Legumesvs. non-legumes “There was a significant

treatment effect (F…). About 53% of the variation between treatments was due to differences between legumes and non-legumes (F1,20 = 6.7).”

F1,20 = (67)/1 = 6.7 10

From full model!

Even different statistical tests may not be independent !

Example. We examined effects of fertilizer on growth of dandelions in a pasture using an ANOVA. We then repeated the test for growth of grass in the same plots.

Problem?

Multiple tests

Not significantsignificant

Not significant

a a,b

bConvention:Treatments with a common letter are not significantly different

0

5

10

15

20

25

30

35

Farm fa

llow

Ebouf

ek co

mplete

Ebogo

comple

te

Ebogo

mec

hanic

al

Ebogo

man

ual

Ebouf

ek un

clear

ed

Ebogo

uncle

ared

Bilik u

nclea

red

Mea

n no

. but

terf

lies/

5 m

in.w

alk

a a

abbc bc

dd

cd