Lecture 13

• Multiple comparisons for one-way ANOVA (Chapter 15.7)

• Analysis of Variance Experimental Designs (Chapter 15.3)

15.7 Multiple Comparisons

• When the null hypothesis is rejected, it may be desirable to find which mean(s) is (are) different, and how they rank.

• Three statistical inference procedures, geared at doing this, are presented:– Fisher’s least significant difference (LSD) method

– Bonferroni adjustment to Fisher’s LSD

– Tukey’s multiple comparison method

Example 15.1

• Sample means:

Does the quality strategy have a higher mean sales than the other two

strategies? Do the quality and price strategies have a higher mean than the

convenience strategy? Does the price strategy have a smaller mean sales than quality but a higher

mean than convenience? • Pairwise comparison: Are two population means different?

00.653

65.608

55.577

quality

price

econvenienc

x

x

x

Fisher Least Significant Different (LSD) Method

• This method builds on the equal variances t-test of the difference between two means.

• The test statistic is improved by using MSE rather than sp

2.• We conclude that i and j differ (at % significance

level if > LSD, where

)11

(,2ji

kn nnMSEtLSD

|| ji xx

Multiple Comparisons Problem

• A hypothetical study of the effect of birth control pills is done.• Two groups of women (one taking birth controls, the other

not) are followed and 20 variables are recorded for each subject such as blood pressure, psychological and medical problems.

• After the study, two-sample t-tests are performed for each variable and it is found that one null hypothesis is rejected. Women taking birth pills have higher incidences of depression at the 5% significance level (the p-value equals .02).

• Does this provide strong evidence that women taking birth control pills are more likely to be depressed?

Experimentwise Type I error rate (E) versus Comparisonwise Type I error rate

• The comparisonwise Type I error rate is the probability of committing a Type I error for one pairwise comparison.

• The experimentwise Type I error rate ( ) is the probability of committing at least one Type I error when C tests are done and all null hypotheses are true.

• For a one-way ANOVA, there are k(k-1)/2 pairwise comparisons (k=number of populations)

• If the comparisons are not planned in advance and chosen after looking at the data, the experimentwise Type I error rate is the more appropriate one to look at.

E

Experimentwise Error Rate

• The expected number of Type I errors if C tests are done at significance level each is

• If C independent tests are done,

E = 1-(1 – )C

• The Bonferroni adjustment determines the required Type I error probability per test () , to secure a pre-determined overall E

C

Bonferroni Adjustment

• Suppose we carry out C tests at significance level

• If the null hypothesis for each test is true, the probability that we will falsely reject at least one hypothesis is at most

• Thus, if we carry out C tests at significance level , the experimentwise Type I error rate is at most

C

C/ )/( CC

• The procedure:– Compute the number of pairwise comparisons (C)

[all: C=k(k-1)/2], where k is the number of populations.

– Set = E/C, where E is the true probability of making at least one Type I error (called experimentwise Type I error).

– We conclude that i and j differ at /C% significance level (experimentwise error rate at most ) if

)11

(),2(ji

knCji nnMSEtxx

Bonferroni Adjustment for ANOVA

35.4465.6080.653xx

10.3165.60855.577xx

45.750.65355.577xx

32

31

21

• Example 15.1 - continued

– Rank the effectiveness of the marketing strategies(based on mean weekly sales).

– Use the Fisher’s method, and the Bonferroni adjustment method

• Solution (the Fisher’s method)

– The sample mean sales were 577.55, 653.0, 608.65.

– Then,

71.59)20/1()20/1(8894

)11

(

57,2/05.

,2

t

nnMSEt

jikn

Fisher and Bonferroni Methods

• Solution (the Bonferroni adjustment)– We calculate C=k(k-1)/2 to be 3(2)/2 = 3.

– We set = .05/3 = .0167, thus t.01672, 60-3 = 2.467 (Excel).

54.73)20/1()20/1(8894467.2

)n1

n1

(MSEtji

2

Again, the significant difference is between 1 and 2.

35.4465.6080.653xx

10.3165.60855.577xx

45.750.65355.577xx

32

31

21

Fisher and Bonferroni Methods

• The test procedure: – Assumes equal number of obs. per populations.– Find a critical number as follows:

gnMSE

),k(q

k = the number of populations =degrees of freedom = n - kng = number of observations per population = significance levelq(k,) = a critical value obtained from the studentized range table (app. B17/18)

Tukey Multiple Comparisons

If the sample sizes are not extremely different, we can use the above procedure with ng calculated as the harmonic mean ofthe sample sizes. k21 n1...n1n1

kgn

• Repeat this procedure for each pair of samples. Rank the means if possible.

• Select a pair of means. Calculate the difference between the larger and the smaller mean.

• If there is sufficient evidence to conclude that max > min .

minmax xx

minmax xx

Tukey Multiple Comparisons

City 1 vs. City 2: 653 - 577.55 = 75.45City 1 vs. City 3: 608.65 - 577.55 = 31.1City 2 vs. City 3: 653 - 608.65 = 44.35

• Example 15.1 - continued We had three populations (three marketing strategies).K = 3,

Sample sizes were equal. n1 = n2 = n3 = 20,= n-k = 60-3 = 57,MSE = 8894.

minmax xx

70.7120

8894)57,3(.q

nMSE

),k(q 05g

Take q.05(3,60) from the table: 3.40.

Population

Sales - City 1Sales - City 2Sales - City 3

Mean

577.55653698.65

minmax xx

Tukey Multiple Comparisons

15.3 Analysis of Variance Experimental Designs

• Several elements may distinguish between one experimental design and another:– The number of factors (1-way, 2-way, 3-way,…

ANOVA).– The number of factor levels.– Independent samples vs. randomized blocks– Fixed vs. random effects

These concepts will be explained in this lecture.

Number of factors, levels• Example: 15.1, modified

– Methods of marketing: price, convenience, quality => first factor with 3 levels

– Medium: advertise on TV vs. in newspapers => second factor with 2 levels

• This is a factorial experiment with two “crossed factors” if all 6 possibilities are sampled or experimented with.

• It will be analyzed with a “2-way ANOVA”. (The book got this term wrong.)

Factor ALevel 1Level2

Level 1

Factor B

Level 3

Two - way ANOVATwo factors

Level2

One - way ANOVASingle factor

Treatment 3 (level 1)

Response

Response

Treatment 1 (level 3)

Treatment 2 (level 2)

• This is something between 1-way and 2-way ANOVA: a generalization of matched pairs when there are more than 2 levels.

• Groups of matched observations are collected in blocks, in order to remove the effects of unwanted variability. => We improve the chances of detecting the variability of interest.

• Blocks are like a second factor => 2-way ANOVA is used for analysis

• Ideally, assignment to levels within blocks is randomized, to permit causal inference.

Randomized blocks

Randomized blocks (cont.)• Example: expand 13.03

– Starting salaries of marketing and finance MBAs: add accounting MBAs to the investigation.

– If 3 independent samples of each specialty are collected (samples possibly of different sizes), we have a 1-way ANOVA situation with 3 levels.

– If GPA brackets are formed, and if one samples 3 MBAs per bracket, one from each specialty, then one has a blocked design. (Note: the 3 samples will be of equal size due to blocking.)

– Randomization is not possible here: one can’t assign each student to a specialty, and one doesn’t know the GPA beforehand for matching. => No causal inference.

• Fixed effects– If all possible levels of a factor are included in our

analysis or the levels are chosen in a nonrandom way, we have a fixed effect ANOVA.

– The conclusion of a fixed effect ANOVA applies only to the levels studied.

• Random effects– If the levels included in our analysis represent a random

sample of all the possible levels, we have a random-effect ANOVA.

– The conclusion of the random-effect ANOVA applies to all the levels (not only those studied).

Models of fixed and random effects

Fixed and random effects - examples– Fixed effects - The advertisement Example (15.1): All

the levels of the marketing strategies considered were included. Inferences don’t apply to other possible strategies such as emphasizing nutritional value.

– Random effects - To determine if there is a difference in the production rate of 50 machines in a large factory, four machines are randomly selected and the number of units each produces per day for 10 days is recorded.

Models of fixed and random effects (cont.)

15.4 Randomized Blocks Analysis of Variance

• The purpose of designing a randomized block experiment is to reduce the within-treatments variation, thus increasing the relative amount of between treatment variation.

• This helps in detecting differences between the treatment means more easily.

Examples of Randomized Block Designs

Factor Response Units Block

Varieties of Corn

Yield Plots of Land

Adjoining plots

Blood pressure Drugs

Hypertension

Patient Same age, sex, overall condition

Management style

Worker productivity

Amount produced by worker

Shifts

Treatment 4

Treatment 3

Treatment 2

Treatment 1

Block 1Block3 Block2

Block all the observations with some commonality across treatments

Randomized Blocks

TreatmentBlock 1 2 k Block mean

1 X11 X12 . . . X1k2 X21 X22 X2k...b Xb1 Xb2 Xbk

Treatment mean

1]B[x

2]B[x

b]B[x

1]T[x 2]T[x k]T[x

Block all the observations with some commonality across treatments

Randomized Blocks

• The sum of square total is partitioned into three sources of variation– Treatments– Blocks– Within samples (Error)

SS(Total) = SST + SSB + SSESS(Total) = SST + SSB + SSE

Sum of square for treatments Sum of square for blocks Sum of square for error

Recall. For the independent samples design we have: SS(Total) = SST + SSE

Partitioning the total variability

Sums of Squares Decomposition

• = observation in ith block, jth treatment

• = mean of ith block

• = mean of jth treatment

•

ijX

iX

jX

k

j

b

ijiij

b

ii

k

jj

k

j

b

iij

XXXXSSBSSTSSTotSSE

XXkSSB

XXbSST

XXSSTot

1 1

2

2

1

1

2

1 1

2

)(

)(

)(

)(

Calculating the sums of squares• Formulas for the calculation of the sums of squares

TreatmentBlock 1 2 k Block mean

1 X11 X12 . . . X1k2 X21 X22 X2k...b Xb1 Xb2 Xbk

Treatment mean

1]B[x

2]B[x

1]T[x 2]T[x k]T[x x2

1 X)]T[x(b

...X)]T[x(b

2

2

2

k X)]T[x(b

SST =

2

1 X)]B[x(k

2

2 X)]B[x(k

2

k X)]B[x(k

SSB=

...)()(...

)()(...)()()(

22

21

222

212

221

211

XxXX

XxXxXxXxTotalSS

kk

Calculating the sums of squares• Formulas for the calculation of the sums of squares

TreatmentBlock 1 2 k Block mean

1 X11 X12 . . . X1k2 X21 X22 X2k...b Xb1 Xb2 Xbk

Treatment mean

1]B[x

2]B[x

1]T[x 2]T[x k]T[x x2

1 X)]T[x(b

...X)]T[x(b

2

2

2

k X)]T[x(b

SST =

2

1 X)]B[x(k

2

2 X)]B[x(k

2

k X)]B[x(k

SSB=

...)X]B[x]T[xx()X]B[x]T[xx(

...)X]B[x]T[xx()X]B[x]T[xx(

...)X]B[x]T[xx()X]B[x]T[xx(SSE

22kk2

21kk1

22222

21212

22121

21111

To perform hypothesis tests for treatments and blocks we need

• Mean square for treatments• Mean square for blocks• Mean square for error

Mean Squares

1kSST

MST

1bSSB

MSB

1bknSSE

MSE

Test statistics for the randomized block design ANOVA

MSEMST

F

MSEMSB

F

Test statistic for treatments

Test statistic for blocks

df-T: k-1 df-B: b-1 df-E: n-k-b+1

• Testing the mean responses for treatments

F > F,k-1,n-k-b+1

• Testing the mean response for blocks

F> F,b-1,n-k-b+1

The F test rejection regions

• Example 15.2– Are there differences in the effectiveness of

cholesterol reduction drugs? – To answer this question the following experiment

was organized:• 25 groups of men with high cholesterol were matched by

age and weight. Each group consisted of 4 men.• Each person in a group received a different drug.• The cholesterol level reduction in two months was

recorded.

– Can we infer from the data in Xm15-02 that there are differences in mean cholesterol reduction among the four drugs?

Randomized Blocks ANOVA - Example

• Solution– Each drug can be considered a treatment.

– Each 4 records (per group) can be blocked, because they are matched by age and weight.

– This procedure eliminates the variability in cholesterol reduction related to different combinations of age and weight.

– This helps detect differences in the mean cholesterol reduction attributed to the different drugs.

Randomized Blocks ANOVA - Example

BlocksTreatments b-1 MST / MSE MSB / MSE

Conclusion: At 5% significance level there is sufficient evidence to infer that the mean “cholesterol reduction” gained by at least two drugs are different.

K-1

Randomized Blocks ANOVA - Example

ANOVASource of Variation SS df MS F P-value F critRows 3848.7 24 160.36 10.11 0.0000 1.67Columns 196.0 3 65.32 4.12 0.0094 2.73Error 1142.6 72 15.87

Total 5187.2 99