Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 212 times |
Download: | 0 times |
Statistics for the Social Sciences
Psychology 340Spring 2005
Effect sizes & Statistical Power
Statistics for the Social Sciences
Outline
• Effect size: Cohen’s d• Error types• Statistical Power Analysis
Statistics for the Social Sciences
Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
Experimenter’s conclusions
Reject H0
Fail to Reject
H0
There really isn’t an effect
There really isan effect
Statistics for the Social Sciences
Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
Statistics for the Social Sciences
Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
So there is only one distribution
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
So there are two distributionsThe original
(null) distribution
The new (treatment) distribution
The original (null) distribution
Statistics for the Social Sciences
Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
So there is only one distribution
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
So there are two distributionsThe original
(null) distribution
The new (treatment) distribution
The original (null) distribution
Statistics for the Social Sciences
Effect Size
H0 is wrong
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
So there are two distributionsThe original
(null) distribution
The new (treatment) distribution
• Hypothesis test tells us whether the observed difference is probably due to chance or not
• It does not tell us how big the difference is– Effect size tells us how much the two populations don’t overlap
Statistics for the Social Sciences
Effect Size
The original (null) distribution
The new (treatment) distribution
• Figuring effect size
– Effect size tells us how much the two populations don’t overlap
μ1 − μ 2
μ2 μ1
But this is tied to the particular units of measurement
But this is tied to the particular units of measurement
Statistics for the Social Sciences
Effect Size
The original (null) distribution
The new (treatment) distribution
d =μ1 −μ2
σ
• Standardized effect size
– Effect size tells us how much the two populations don’t overlap
μ2 μ1
– Puts into neutral units for comparison (same logic as z-scores)
Cohen’s d
Statistics for the Social Sciences
Effect Size
The original (null) distribution
The new (treatment) distribution
• Effect size conventions– small d = .2– medium d = .5– large d = .8
– Effect size tells us how much the two populations don’t overlap
sm-m
= 21d
μ2 μ1
Statistics for the Social Sciences
Error types
Real world (‘truth’)
H0 is correct
H0 is wrong
Experimenter’s conclusions
Reject H0
Fail to Reject H0
I conclude that there is an effect
I can’t detect an effect
There really isn’t an effect
There really isan effect
Statistics for the Social Sciences
Error types
Real world (‘truth’)
H0 is correct
H0 is wrong
Experimenter’s conclusions
Reject H0
Fail to Reject H0
Type I error Type
II error
α
β
Type I error (): concluding that there is a difference between groups (“an effect”) when there really isn’t.
Type I error (): concluding that there is a difference between groups (“an effect”) when there really isn’t.
Type II error (): concluding that there isn’t an effect, when there really is.
Type II error (): concluding that there isn’t an effect, when there really is.
Statistics for the Social Sciences
Statistical Power
• The probability of making a Type II error is related to Statistical Power– Statistical Power: The probability that the study will produce a statistically significant results if the research hypothesis is true (there is an effect)
Power =1−
• So how do we compute this?
Statistics for the Social Sciences
Statistical Power
H0: is true (is no treatment effect)Real world (‘truth’)
Fail to reject H0
Reject H0
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
= 0.05
The original (null) distribution
Statistics for the Social Sciences
Statistical Power
H0: is false (is a treatment effect)
= 0.05
Reject H0
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β The original (null) distribution
Real world (‘truth’)
The new (treatment) distributionThe new (treatment) distribution
Fail to reject H0
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
= probability of a Type II error
The new (treatment) distribution
H0: is false (is a treatment effect)
The original (null) distribution
Real world (‘truth’)
= 0.05
Failing to Reject H0, even though there is
a treatment effect
Failing to Reject H0, even though there is
a treatment effect
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
α
β
Power = 1 -
The new (treatment) distribution
H0: is false (is a treatment effect)
The original (null) distribution
Real world (‘truth’)
= probability of a Type II error
= 0.05
Failing to Reject H0, even though there is
a treatment effect
Failing to Reject H0, even though there is
a treatment effectProbability of
(correctly) Rejecting H0
Probability of (correctly)
Rejecting H0
Statistics for the Social Sciences
Statistical Power
1) Gather the needed information: mean and standard deviation of the Null Population and the predicted mean of Treatment Population
• Steps for figuring power
μ2 μ1
μ2 = 55;σ = 2.5 μ1 = 60;σ = 2.5
Statistics for the Social Sciences
Statistical Power
2) Figure the raw-score cutoff point on the comparison distribution to reject the null hypothesis
• Steps for figuring power
= 0.05From the unit normal table: Z = -1.645
Transform this z-score to a raw score
raw score =μ1 +σ(Z) =60 + (2.5)(−1.645) = 55.89
μ1
μ1 = 60;σ = 2.5
Statistics for the Social Sciences
Statistical Power
3) Figure the Z score for this same point, but on the distribution of means for treatment Population
• Steps for figuring power
55.89Transform this raw score to a z-score
Z =X−μσ
=55.88 − 55
2.5
=0.355
μ2 = 55;σ = 2.5
Remember to use the properties of the
treatment population!
Remember to use the properties of the
treatment population!
Statistics for the Social Sciences
Statistical Power
4) Use the normal curve table to figure the probability of getting a score more extreme than that Z score
• Steps for figuring power
= probability of a Type II error =0.355
From the unit normal table: Z(0.355) = 0.3594
Power = 1 -
Power =1−0.3594 =0.64The probability of detecting this an effect of this size from these populations is 64%
55.89
Statistics for the Social Sciences
Statistical Power
-level
– Sample size
– Population standard deviation σ
– Effect size
– 1-tail vs. 2-tailed
Factors that affect Power:
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
= 0.01
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
= 0.01
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
= 0.01
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
= 0.01
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
-levelChange from = 0.05 to 0.01
= 0.01
So as the level gets smaller, so does the
Power of the test
So as the level gets smaller, so does the
Power of the test
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Sample sizeChange from n = 25 to 100
€
σX
=σ
n
Recall that sample size is related to the spread of the distribution
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Sample sizeChange from n = 25 to 100
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Sample sizeChange from n = 25 to 100
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Sample sizeChange from n = 25 to 100
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Sample sizeChange from n = 25 to 100
As the sample gets bigger, the standard error gets smaller and the Power gets larger
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Population standard deviationChange from σ = 25 to 20
€
σX
=σ
n
Recall that standard error is related tothe spread of the distribution
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Population standard deviationChange from σ = 25 to 20
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Population standard deviationChange from σ = 25 to 20
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Population standard deviationChange from σ = 25 to 20
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
As the σ gets smaller, the standard error gets smaller and the Power gets larger
Population standard deviationChange from σ = 25 to 20
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
μtreatment μno treatment
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
μtreatment μno treatment
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
Effect sizeCompare a small effect (difference) to a big effect
As the effect gets bigger, the Power gets larger
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
Statistics for the Social Sciences
Statistical Power
= 0.05
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
p = 0.025
p = 0.025
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
p = 0.025
p = 0.025
= 0.05
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
p = 0.025
p = 0.025
= 0.05
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
p = 0.025
p = 0.025
= 0.05
Statistics for the Social Sciences
Statistical Power
Fail to reject H0
Reject H0
Power = 1 -
Factors that affect Power:
1-tail vs. 2-tailedChange from = 0.05 two-tailed
to = 0.05 two-tailed
p = 0.025
Two tailed functionally cuts the -level in half, which decreases the power.p = 0.025
= 0.05
Statistics for the Social Sciences
Statistical Power
Factors that affect Power: -level: So as the level gets smaller, so does the Power of the test
– Sample size: As the sample gets bigger, the standard error gets smaller and the Power gets larger
– Population standard deviation: As the population standard deviation gets smaller, the standard error gets smaller and the Power gets larger
– Effect size: As the effect gets bigger, the Power gets larger
– 1-tail vs. 2-tailed: Two tailed functionally cuts the -level in half, which decreases the power
Statistics for the Social Sciences
Why care about Power?
• Determining your sample size– Using an estimate of effect size, and population standard deviation, you can determine how many participants need to achieve a particular level of power
• When a result if not statistically significant– Is is because there is no effect, or not enough power
• When a result is significant– Statistical significance versus practical significance
Statistics for the Social Sciences
Ways of Increasing Power