Multiple Comparisons & Factorial...

Multiple Comparisons & Factorial Designs

EdPsych 496

C.J. Anderson

Multiple Comparisons & Factorial Designs – p.1/38

Multiple Comparisons (part 2)

• When there is no interaction.. . . basically whatwe did with 1-Factor ANOVA• Example• Contrasts

• When there is an interaction.• Example• Contrasts


MC Procedures We’ll Cover

• Pairwise (simple):• Fisher LSD.• Tukey HSD.

• Simple or Complex:• Bonferroni• Scheffé


When There is NO Interaction

• Only need to perform multiple comparisonsfor main effects that were significant.

• Do not need to do them if a factor only has 2levels.

• Multiple comparison procedures are the sameas those used in 1-Factor ANOVA, exceptneed to make sure you’re using the correct n.

• To illustrate, need data with at least 1 maineffect and no interaction. . .


NELS88National Education Longitudinal Study:

• Conducted by National Center for Education Statisticsof the US department of Education. These dataconstitute the first in a series of longitudinalmeasurements of students starting in 8th grade. Thesedata were collected Spring 1988.

• We’ll just use data from a sample of N = 225 studentsfrom 23 schools.

• Response variable: math achievement test score.

• Factors: SES (Low, Middle, High) and Location ofschool (rural, urban, sub-urban).


NELS88: Mean Plots


NELS88: Summary StatisticsMeans and (standard deviations). njk = 25

Location

SES Rural Suburban Urban

Low 44.00 44.60 46.48 45.03

(9.36) (6.84) (9.32) (8.54)

Mid 46.52 49.64 51.20 49.45

(8.85) (10.71) (9.17) (9.60)

High 57.00 53.32 57.96 56.10

(9.17) (11.66) (8.86) (10.05)

49.51 49.19 51.88 50.19

(10.57) (10.46) (10.17) (10.42)Multiple Comparisons & Factorial Designs – p.7/38

NELS88: ANOVA Summary Table

Source df SS MS F p > F

SES 2 4653.90 2326.95 26.27 < .001

Urban 2 324.73 162.36 1.83 .16

SES*Urban 4 229.51 57.38 0.65 .63

Error 216 19130.78 88.57

Corrected Total 224 24338.78

• SES is significant.

• Urban and Interaction are not significant.


NELS88: Pairwise

• Since we have balanced design, we’ll useminimum significant differences.

•

Null Observed Differences

hypothesis between sample means Conclusion

µhigh = µmid 56.10 − 49.45 = 6.64

µmid = µlow 49.45 − 45.03 = 4.42

µhigh = µlow 56.10 − 45.03 = 11.07


NELS88: Fisher’s LSD• Controls type I error rate per comparison, α.

• Least significant difference is

LSD =(1−α/2) tνesy−y

• For α = .05 and our example, .975t216 = 1.971 and

sy−y =

√

MSe

(

1

nK+

1

nK

)

=

√

88.56778

(

2

25(3)

)

= 1.54

• So LSD = 1.971(1.54) = 3.03 −→ Reject all Ho’s.Multiple Comparisons & Factorial Designs – p.10/38

NELS88: Tukey’s HSD• Controls familywise type I error rate per, ασ — the

family being pairs of means for SES.

• Honest significant difference is HSD = qνe,ασ ,Jsy−y.

• For α = .05, νe = 216 and J = 3, q = 3.338.

• The standard error

sy−y =

√

MSe

2

(

1

nK+

1

nK

)

=

√

88.56778

2

(

2

25(3)

)

= 1.09

• So HSD = 3.338(1.087) = 3.63 −→ Reject all Ho’s.Multiple Comparisons & Factorial Designs – p.11/38

Complex Comparisons• For Complex (and simple) comparisons.• For a few planned comparisons −→

Bonferroni.• For post hoc and/or more than (J − 1) −→

Scheffé.• Consider the following two comparisons for

NELS data:

ψ1 = µlow + µmid − 2µhigh

ψ2 = µhigh + µmid − 2µlow


Complex Comparisons• The sums of squares for comparisons:

SSψ =nK(

∑Jj=1 cjYj·)

2

∑Jj=1 c2

j

=nK(ψ)2

∑Jj=1 c2

j

• For comparison 1:

ψ21 = (45.027 + 49.453 − 2(56.093))2 = 313.502

SSψ1=

25(3)(313.502)

12 + 12 + (−2)2=

25(3)(313.502)

6= 3, 919.07

• For comparison 2:

ψ22 = (−2(45.027) + 49.453 + 56.093)2 = 240.002

SSψ2=

25(3)(240.002)

6= 3, 000.03


Complex Comparisons• Test statistics:

Contrast df SS MS F

(low & middle) vs high 1 3,919.07 3919.07 44.25

low vs (middle & high) 1 3,000.03 3,000.03 33.87

Error 216 19,130.64 88.568

• For Bonferroni, αΣ = .05, (.975/2)t216 = 2.413, and for theF ratio, the critical value is (2.413)2 = 5.82

• For Scheffé, (J − 1)(1−αΣ)F(J−1),νe= 2(3.0376) = 6.08.

• For this NELS example, both comparisons aresignificant.


Complex Comparisons• For the complex comparison, we implicitly

tested (for example)Ho(1) : ψ1 = 0 vs Ha(1) : ψ1 < 0

• For directional alternatives, e.g.,Ha(1) : ψ1 < 0, use t-ratio,

t =ψ

sψ

=

∑

j cjYj·√

MSe

(

∑

j c2j/(nK)

)

• The critical value is the square root of theBonferroni or Scheffé (depending on whichyou’re doing).


Complex Comparisons• For our comparison 1 with Ha(1) : ψ1 6= 0

t =45.027 + 49.453 − 2(56.093)

√

88.568(

625(3)

)

=−17.705

2.662= −6.65

• Note: observed test statistic is

t2 = (−6.65)2 = 44.25.

• The (negative) square root of critical values

given previously.

• Bonferroni: tcrit = −2.41.

• Scheffé: −2.47Multiple Comparisons & Factorial Designs – p.16/38

When There is an Interaction

• The basic idea and methodology isessentially the same as multiple comparisonsfor main effects.

• Types of multiple comparisons:• Simple Effect• Mini-Factorials• Specific Cell Means

• We need another data set — a factorial withan interaction.. . .


When There is an InteractionModified hypothetical example from Tatsuoka & Lohnes(1988, p273-279):10 subjects were randomly assigned to one of twomethods (A and B) of teaching keyboarding to one of thefollowing 3 conditions:

• C1: 2 hours of instruction per day for 6 weeks.



At the end of the course, each of the 60 subjects was given

a standardized test measuring accuracy.


The DataThe “raw” data:

Method

Condition A B

C1 26, 22, 21, 23, 21, 25, 24, 29, 25, 26,

19, 25, 20, 21, 20 28, 25, 24, 24, 26

C2 17, 21, 17, 18, 23, 18, 19, 17, 17, 19,

19, 16, 19, 15, 14 19, 17, 21, 18, 20

C3 14, 14, 16, 16, 13, 11, 10, 10, 12, 11,

13, 12, 15, 14, 14 12, 14, 13, 12, 11


Summary Statistics

Condition

C1 C2 C3 Over Conditions

n11 = 10 n12 = 10 n13 = 10 n1• = 30

Method Y11 = 21.8 Y12 = 17.9 Y13 = 14.1 Y1· = 17.93

A s2

11= 5.07 s2

12= 7.43 s2

13= 1.66 s2

A= 14.62

n21 = 10 n22 = 10 n23 = 10 n2• = 30

Method Y21 = 25.6 Y22 = 18.5 Y23 = 11.6 Y2· = 18.57

B s2

21= 2.93 s2

22= 1.83 s2

23= 1.60 s2

B= 35.77

Over Methods n•1 = 20 n•2 = 20 n•2 = 20 n•• = 60

Y·1 = 23.7 Y·2 = 18.2 Y·3 = 12.85 Y·· = 18.25

s2

C1= 7.59 s2

C2= 4.48 s2

C3= 3.19 s2 = 24.87


Computing Sums of Squares• Total Sum of Squares: (Corrected for total mean)

SStot =J

∑

j=1

K∑

k=1

n∑

i=1

(Yijk − Y··)2 = (nJK − 1)s2

= (60 − 1)(24.87) = 1467.25

• Sum of Squares Within (error):

SSe =J

∑

j=1

K∑

k=1

n∑

i=1

(Yijk − Yjk)2 = (n − 1)

2∑

j=1

3∑

k=1

s2jk

= (10 − 1)(5.07 + 7.43 + 1.66 + 2.93 + 1.83 + 1.60)

= 184.70


Computing Sums of Squares• Sum of Squares Between Methods:

SSM = nK

J∑

j=1

(Yj· − Y··)2

= (10)(3)[(17.93 − 18.25)2 + (18.57 − 18.25)2]

= 6.02

• Sum of Squares Between Conditions:

SSC = nJK

∑

k=1

(Y·k − Y

··)2

= (10)(2)[(23.7 − 18.25)2 + (18.2 − 18.25)2 + (12.85 − 18.25)2]

= 1, 177.3


Computing Sums of Squares• Sum of Squares for Method × Condition Interaction:

SSMC = n

J∑

j=1

K∑

k=1

(Yjk − Yj· − Y·k + Y

··)2

= 10[(21.80 − 17.93 − 23.70 + 18.25)2 + . . .

+(11.60 − 18.57 − 12.85 + 18.25)2]

= 99.23

• or much easier

SSMC = SStot − SSM − SSC − SSe

= 1467.25 − 6.02 − 1177.30 − 184.70 = 99.23


ANOVA Summary Table

Source df SS MS F p–value

Method 1 6.02 6.02 1.76 .1903

Condition 2 1177.30 588.65 172.12 .0001

M × C Interaction 2 99.23 49.62 14.51 .0001

Within (error) 54 184.70 3.420

(corrected) total 59 1467.25

R2 =SSmodel

SStot

= (6.02 + 1177.30 + 99.23)/1467.25 = .8741


Let’s Take a Look. . .


MC of Simple Effects. . . by Example

Our design can be represented as

Practice

Conditions

C1 C2 C3

Teaching A

Method B


MC of Simple Effects (continued)

• To investigate simple effects of practice given teachingmethod A, test contrasts involving means in thesecells:

C1 C2 C3

A

• To investigate simple effects of practice given teachingmethod is B, test contrasts involving means in thesecells:

C1 C2 C3

B


Alternatively (or Additionally)

Investigate the simple effects of teaching method givenconditions are test comparisons of means from thefollowing cells:

C1 C2 C3

A A A

B B B

Ho : µAC1= µBC1

Ho : µAC2= µBC2

Ho : µAC3= µBC3


Mini-Factorials: (2 × 2) Sub-TablesOne possible way to break our 2 × 3 design into mini 2 × 2

factorials is as follows:

C1 C2 C2 C3

A A

B B

Sub-design on the left: Test whether there is an interactionfor teaching method and practice conditions C1 and C2 bytesting the hypothesis

Ho : (µAC1− µBC1

) = (µAC2− µBC2

)

or equivalently,

Ho : (µAC1− µAC2

) = (µBC1− µBC2

)

Recall that if there is no interaction, then “curves” shouldMultiple Comparisons & Factorial Designs – p.29/38

Mini-Factorials: (2 × 2) Sub-TablesGiven that pattern of the cell means. . .

C1 C3

A

B


) = (µAC3− µBC3

)

or equivalently,

Ho : (µAC1− µAC3

) = (µBC1− µBC3

)

Recall that if there is no interaction, then “curves” shouldbe parallel; that is, the differences between pairs of meansshould be the same.


Mini-Factorials: (2 × 2) Sub-TablesWhat is the contrast for


) = (µAC3− µBC3

)?

• Contrast

Ho : ψ = 1µAC1+ 0µAC2

− 1µAC3− 1µBC1

+ 1µBC3= 0

• Estimated value

ψ2 = (21.8 − 14.1 − 25.6 + 11.6)2 = (−6.3)2 = 39.69

• Needed for the sum of squares for the contrast∑

j

∑

k

c2jk/njk = 1/10 + 1/10 + 1/10 + 1/10 = .4


Mini-Factorials: (2 × 2) Sub-TablesTesting (continued)


) = (µAC3− µBC3

)?

• Sum of squares,

SSψ =ψ2

∑

j

∑

k c2jk/njk

=39.69

.4= 99.225

• Test statistic,

F =SSψ/1

MSerror

=99.225

3.420= 29.01

• What should the critical value be?Multiple Comparisons & Factorial Designs – p.32/38

Mini-Factorials: (2 × 2) Sub-TablesTesting (continued)

• The critical value for our test. . .• Post hoc −→ Complex −→ Scheffé.• Scheffé critical value is (νAC)(1−αΣ)FνAC ,νerror

;that is,

(J − 1)(K − 1).95F(J−1)(K−1),N−JK

• Our example,Fcrit = 2(.95F4,54) = 2(2.54) = 5.08

• Since the test statistic F = 29.01, reject Ho.Multiple Comparisons & Factorial Designs – p.33/38

Testing Specific Means

• Contrasts and comparisons between any setof means that makes sense can beperformed.

• In our example, from the interaction plot itlooks like• µAC2

= µBC2

• µBC1> µAC1

• µAC3> µBC3


Testing Specific Means• Our example, our null and alternative

hypotheses would be• Ho(1) : µAC2

= µBC2versus Ha(1) : µAC2

6= µBC2

• Ho(2) : µBC1= µAC1

versus Ha(2) : µBC1> µAC1

• Ho(3) : µAC3= µBC3

versus Ha(3) : µAC3> µBC3

• We need t-ratios for Ho(2) and Ho(3) to test

directional hypotheses.

• For simplicity & a bit conservative, I’ll take the

alternatives to be non-directional.• Post hoc −→ Pairwise −→ Tukey HSD.


Testing Specific MeansResults for Tukey’s Studentized Range (HSD) Test for Y,αΣ = .05, νerror = 54, MSerror = 3.42, qcrit = 4.178 andMinimum significant difference = 2.44Means with the same letter are not significantly different.

Tukey Grouping Mean N group

A 25.6000 10 2 (B,C1)

B 21.8000 10 1 (A,C1)

C 18.5000 10 4 (B,C2)

C

C 17.9000 10 3 (A,C2)

D 14.1000 10 5 (A,C3)

E 11.6000 10 6 (B,C3)


Testing Specific Means• For Tukey, the HSD for a main effect

(marginal means), say factor A, use

HSDA = qα,J,N−JK

√

MSerror

nK

• For Tukey, the HSD for interaction(cells means) use

HSDAB = qα,JK,N−JK

√

MSerror

n


SAS & Testing Specific Means• SAS will compute pairwise tests for main effects by the

command

MEANS factorA / tukey;

• This doesn’t work for for cell means with factorialdesigns.

• Solutions:

• Re-run ANOVA but treat cells as a 1-Factor ANOVAand use the MEANS command.

• Run as factorial, but specify pairwise contrasts forall interaction effects using the CONSTRAST command.You need to then find the appropriate critical value.


Date post:	03-Feb-2018
Category:	Documents
Upload:	lamquynh
View:	220 times
Download:	0 times

Multiple Comparisons & Factorial...

Documents