Factorial Designs & Managing Violated Statistical Assumptions

Dr James Betts

Developing Study Skills and Research Methods (HL20107)

Lecture Outline:

•Factorial Research Designs Revisited

•Mixed Model 2-way ANOVA

•Fully independent/repeated measures 2-way ANOVA

•Statistical Assumptions of ANOVA.

Last Week Recap• In last week’s lecture we saw two worked

examples of 1-way analyses of variance

• However, many experimental designs have more than one independent variable (i.e. factorial design)

Factorial Designs: Technical Terms• Factor

• Levels

• Main Effect

• Interaction Effect

Factorial Designs: Multiple IV’s• Hypothesis:

– The HR response to exercise is mediated by gender

• We now have three questions to answer:

1)

2)

3)

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

Main Effect of Exercise

Not significant

Main Effect of Gender

Not significant

Exercise*Gender Interaction

Not significant

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

Main Effect of Exercise

Significant

Main Effect of Gender

Not significant

Exercise*Gender Interaction

Not significant

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

Main Effect of Exercise

Not significant

Main Effect of Gender

Significant

Exercise*Gender Interaction

Not significant

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

Main Effect of Exercise

Not significant

Main Effect of Gender

Not Significant

Exercise*Gender Interaction

Significant

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

?

Post Run 1 Pre Run 2 Post Run 2

Mu

scle

Gly

cogen

(mm

ol

glu

cosy

l u

nit

s. k

g d

m-1)

0

50

100

150

200

250

300

350

CHO CHO-PRO

Factorial Designs: Interpretation210

180

150

120

90

60

30

0

Hea

rt R

ate

(bea

tsm

in-1)

Resting Exercise

?

SystematicVariance

(resting vs exercise)

ErrorVariance

(between subjects)

Systematic Variance

(male vs female)

Systematic Variance

(Interaction)

2-way mixed model ANOVA: Partitioning

= variance between means due to exercise

= variance between means due to gender

= variance between means due IV interaction

= uncontrolled factors and within group differences for males vs females.

ErrorVariance

(within subjects)= uncontrolled factors plus random changes within individuals for rest vs exercise

Procedure for computing 2-way mixed model ANOVA

• Step 1: Complete the table

i.e.-square each raw score

-total the raw scores for each subject (e.g. XT)

-square the total score for each subject (e.g. (XT)2)

-Total both columns for each group

-Total all raw scores and squared scores (e.g. X & X2).

• Step 2: Calculate the Grand Total correction factor

GT =

(X)2

N

Procedure for computing 2-way mixed model ANOVA

• Step 3: Compute total Sum of Squares

SStotal= X2 - GT

Procedure for computing 2-way mixed model ANOVA

• Step 4: Compute Exercise Effect Sum of Squares

SSex= - GT

= + - GT

(Xex)2

nex

(XRmale+XRfemale)2

nRmale+fem

Procedure for computing 2-way mixed model ANOVA

(XEmale+XEfemale)2

nEmale+fem

• Step 5: Compute Gender Effect Sum of Squares

SSgen= - GT

= + - GT

(Xgen)2

ngen

(XRmale+XEmale)2

nmaleR+E

Procedure for computing 2-way mixed model ANOVA

(XRfem+XEfem)2

nfemR+E

• Step 6: Compute Interaction Effect Sum of Squares

SSint= - GT

= + + + - (SSex+SSgen) - GT

(Xex+gen)2

nex+gen

nRmale

Procedure for computing 2-way mixed model ANOVA

nRfem nEmale nEfem

(XRmale)2 (XRfem)2 (XEmale)2 (XEfem)2

• Step 7: Compute between subjects Sum of Squares

SSbet= -SSgen- GT

= -SSgen- GT

(XS)2

nk

Procedure for computing 2-way mixed model ANOVA

(XT)2+(XD)2+(XH)2+(XJ)2+(XK)2+(XA)2+(XS)2+(XL)2

nk

• Step 8: Compute within subjects Sum of Squares

SSwit= SStotal - (SSex+SSgen+SSint+SSbet)

Procedure for computing 2-way mixed model ANOVA

• Step 9: Determine the d.f. for each sum of squares

dftotal = (N - 1)

dfex = (k - 1)

dfgen = (r - 1)

dfint = (k - 1)(r - 1)

dfbet = r(n - 1)

dfwit = r(n - 1)(k - 1)

Procedure for computing 2-way mixed model ANOVA

• Step 10: Estimate the Variances Procedure for computing 2-way mixed model ANOVA

SystematicVariance

(resting vs exercise)

ErrorVariance

(between subjects)

Systematic Variance

(male vs female)Systematic

Variance(Interaction)

ErrorVariance

(within subjects)

SSex

dfex

SSgen

dfgen

SSint

dfint

SSwit

dfwit

SSbet

dfbet

=

=

=

=

=

• Step 11: Compute F values Procedure for computing 2-way mixed model ANOVA

SystematicVariance

(resting vs exercise)

ErrorVariance

(between subjects)

Systematic Variance

(male vs female)Systematic

Variance(Interaction)

ErrorVariance

(within subjects)

SSex

dfex

SSgen

dfgen

SSint

dfint

SSwit

dfwit

SSbet

dfbet

=

=

=

=

=

• Step 12: Consult F distribution table as before Exercise

Gender

Procedure for computing 2-way mixed model ANOVA

Interaction

Tests of Within-Subjects Effects

Measure: MEASURE_1

61504.000 1 61504.000 2396.260 .000

61504.000 1.000 61504.000 2396.260 .000

61504.000 1.000 61504.000 2396.260 .000

61504.000 1.000 61504.000 2396.260 .000

121.000 1 121.000 4.714 .073

121.000 1.000 121.000 4.714 .073

121.000 1.000 121.000 4.714 .073

121.000 1.000 121.000 4.714 .073

154.000 6 25.667

154.000 6.000 25.667

154.000 6.000 25.667

154.000 6.000 25.667

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sourceexercise

exercise * Gender

Error(exercise)

Type III Sumof Squares df Mean Square F Sig.

2-way mixed model ANOVA: SPSS OutputCalculated FexSSex dfex

Systematic Varianceex

SSwit dfwitError

Variancewit

Tests of Between-Subjects Effects

Measure: MEASURE_1

Transformed Variable: Average

227529.000 1 227529.000 17064.675 .000

100.000 1 100.000 7.500 .034

80.000 6 13.333

SourceIntercept

Gender

Error

Type III Sumof Squares df Mean Square F Sig.

2-way mixed model ANOVA: SPSS Output

Calculated FgenSSgen

SSbet

dfgen

dfbet

Error Variancebet

GT

Systematic Variancegen

SystematicVariance

(resting vs exercise)

ErrorVariance

(between subjects)

Systematic Variance

(male vs female)Systematic

Variance(Interaction)

ErrorVariance

(within subjects)

2-way mixed model ANOVA

The previous calculation and associated partitioning is an example of a 2-way mixed model ANOVA– i.e. exercise = repeated measures

gender = independent measures

SystematicVariance

(resting vs exercise)

Systematic Variance

(male vs female)Systematic

Variance(Interaction)

ErrorVariance

(within subjects)

2-way Independent Measures ANOVA

So for a fully unpaired design

– e.g. males vs females

&

rest group vs exercise group

ErrorVariance

(between subjects)

Tests of Between-Subjects Effects

Dependent Variable: HeartRate

58386.000a 3 19462.000 2994.154 .000

234256.000 1 234256.000 36039.385 .000

16.000 1 16.000 2.462 .143

58081.000 1 58081.000 8935.538 .000

289.000 1 289.000 44.462 .000

78.000 12 6.500

292720.000 16

58464.000 15

SourceCorrected Model

Intercept

Gender

CoachCyclist

Gender * CoachCyclist

Error

Total

Corrected Total

Type III Sumof Squares df Mean Square F Sig.

R Squared = .999 (Adjusted R Squared = .998)a.

SystematicVariance

(resting vs exercise)

Systematic Variance

(male vs female)Systematic

Variance(Interaction)

ErrorVariance

(within subjects)

2-way Independent Measures ANOVA

ErrorVariance

(between subjects)

SystematicVariance

(resting vs exercise)

Systematic Variance

(am vs pm)Systematic

Variance(Interaction)

Error Variance

(within subjectsexercise)

2-way Repeated Measures ANOVA

…but for a fully paired design

– e.g. morning vs evening

&

rest vs exercise

Error Variance

(within subjectstime)Error Variance

(within subjectsinteract)

Tests of Within-Subjects Effects

Measure: MEASURE_1

.000 1 .000 .000 1.000

.000 1.000 .000 .000 1.000

.000 1.000 .000 .000 1.000

.000 1.000 .000 .000 1.000

1.000 3 .333

1.000 3.000 .333

1.000 3.000 .333

1.000 3.000 .333

67081.000 1 67081.000 10062.150 .000

67081.000 1.000 67081.000 10062.150 .000

67081.000 1.000 67081.000 10062.150 .000

67081.000 1.000 67081.000 10062.150 .000

20.000 3 6.667

20.000 3.000 6.667

20.000 3.000 6.667

20.000 3.000 6.667

.000 1 .000 .000 1.000

.000 1.000 .000 .000 1.000

.000 1.000 .000 .000 1.000

.000 1.000 .000 .000 1.000

3.000 3 1.000

3.000 3.000 1.000

3.000 3.000 1.000

3.000 3.000 1.000

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

SourceTimeofDay

Error(TimeofDay)

Exercise

Error(Exercise)

TimeofDay * Exercise

Error(TimeofDay*Exercise)

Type III Sumof Squares df Mean Square F Sig.

SystematicVariance

(resting vs exercise)

Systematic Variance

(am vs pm)Systematic

Variance(Interaction)

Error Variance

(within subjectsexercise)

2-way Repeated Measures ANOVA

Error Variance

(within subjectstime)Error Variance

(within subjectsinteract)

Summary: 2-way ANOVA• 2-way (factorial) ANOVA may be appropriate

whenever there are multiple IV’s to compare

• We have worked through a mixed model but you should familiarise yourself with paired/unpaired procedures

• You should also ensure you are aware what these effects actually look like graphically.

Statistical Assumptions• As with other parametric tests, ANOVA is

associated with a number of statistical assumptions

• When these assumptions are violated we often find that an inferential test performs poorly

• We therefore need to determine not only whether an assumption has been violated but also whether that violation is sufficient to produce statistical errors.

Energy Intake (calories per day)

1500 2500 3500 4500 5500

Nu

mb

er o

f P

eo

ple

0

20

40

60

80

100

120

140

160

16 17 18 19 20

Sustained Isometric Torque (seconds)

e.g. ND assumption from last year

Independent Samples Test

7.842 .012 -2.333 18 .031 -1.69600 .72710 -3.22358 -.16842

-2.333 15.447 .034 -1.69600 .72710 -3.24188 -.15012

Equal variancesassumed

Equal variancesnot assumed

SwimTime50mF Sig.

Levene's Test forEquality of Variances

t df Sig. (2-tailed)Mean

DifferenceStd. ErrorDifference Lower Upper

95% ConfidenceInterval of the

Difference

t-test for Equality of Means

Tests of Normality

.161 10 .200* .968 10 .872

.350 10 .001 .747 10 .003

GroupControl

Visualisation

SwimTime50mStatistic df Sig. Statistic df Sig.

Kolmogorov-Smirnova

Shapiro-Wilk

This is a lower bound of the true significance.*.

Lilliefors Significance Correctiona.

e.g. ND assumption from last year

Assumptions of ANOVA• N acquired through random sampling • Data must be of at least the interval LOM (continuous)

• Independence of observations

• Homogeneity of variance

• All data is normally distributed

“ANOVA is generally robust to violations of the normality assumption, in that even when the data

are non-normal, the actual Type I error rate is usually close to the nominal (i.e., desired) value.”

Maxwell & Delaney (1990) Designing Experiments & Analyzing Data: A Model Comparison Perspective, p. 109

“If the data analysis produces a statistically significant finding when no test of sphericity is

conducted…you should disregard the inferential claims made by the researcher.”

Huck & Cormier (1996) Reading Statistics & Research, p. 432

Group BGroup C

Group ASupplement 1Supplement 2

Placebo

Plac. Supp. 1 Supp. 2 Plac.-Supp. 1 Supp. 1-Supp. 2 Plac.-Supp. 2

Tom 2.4 3.0 3.3 -0.6 -0.3 -0.9

Dick 2.2 2.5 2.4 -0.3 0.1 -0.2

Harry 1.8 1.9 2.2 -0.1 -0.3 -0.4

James 1.6 1.1 1.2 0.5 -0.1 0.4

Mean 2.0 2.1 2.3 -0.1 -0.2 -0.3

SD2 0.1 0.7 0.7 0.2 0.04 0.3

Mauchly's Test of Sphericityb

Measure: MEASURE_1

.310 2.343 2 .310 .592 .752 .500Within Subjects EffectTrial

Mauchly's WApprox.

Chi-Square df Sig.Greenhouse-Geisser Huynh-Feldt Lower-bound

Epsilona

Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables isproportional to an identity matrix.

May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed inthe Tests of Within-Subjects Effects table.

a.

Design: Intercept Within Subjects Design: Trial

b.

• Many texts recommend ‘Mauchley’s Test of Sphericity’– A χ2 test which, if significant, indicates a violation to sphericity

• However, this is not advisable on four counts:1.)

2.)

3.)

4.)

• How should we analyse aspherical data?

• Option 1

• Option 2

• Option 3

Managing Violations to Sphericity

Paired 1-way MANOVA: SPSS Output

Multivariate Testsb

.455 .834a 2.000 2.000 .545

.545 .834a 2.000 2.000 .545

.834 .834a 2.000 2.000 .545

.834 .834a 2.000 2.000 .545

Pillai's Trace

Wilks' Lambda

Hotelling's Trace

Roy's Largest Root

EffectTrial

Value F Hypothesis df Error df Sig.

Exact statistica.

Design: Intercept Within Subjects Design: Trial

b.

Mauchly's Test of Sphericityb

Measure: MEASURE_1

.310 2.343 2 .310 .592 .752 .500Within Subjects EffectTrial

Mauchly's WApprox.

Chi-Square df Sig.Greenhouse-Geisser Huynh-Feldt Lower-bound

Epsilona

Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables isproportional to an identity matrix.

May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed inthe Tests of Within-Subjects Effects table.

a.

Design: Intercept Within Subjects Design: Trial

b.

Paired 1-way ANOVA: SPSS Output

Tests of Within-Subjects Effects

Measure: MEASURE_1

.152 2 .076 .840 .477

.152 1.183 .128 .840 .439

.152 1.505 .101 .840 .457

.152 1.000 .152 .840 .427

.542 6 .090

.542 3.550 .153

.542 4.514 .120

.542 3.000 .181

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

SourceTrial

Error(Trial)

Type III Sumof Squares df Mean Square F Sig.

Summary ANOVA and Sphericity• ANOVA is a generally robust inferential test

• Unpaired data are susceptible to heterogeneity of variance only if group sizes are unequal

• Paired data are susceptible to asphericity only if multiple comparisons are made

• Suggested solutions for the latter include either MANOVA or epsilon corrected df depending on sample size relative to number of levels.