Lecture 15:Hypothesis testing II:The two-sample case
Ernesto F. L. Amaral
October 19, 2017Advanced Methods of Social Research (SOCI 420)
Source: Healey, Joseph F. 2015. ”Statistics: A Tool for Social Research.” Stamford: Cengage Learning. 10th edition. Chapter 9 (pp. 216–246).
Chapter learning objectives• Identify and cite examples of situations in which the two-
sample test of hypothesis is appropriate• Explain the logic of hypothesis testing, as applied to the
two-sample case• Explain what an independent random sample is• Perform a test of hypothesis for two sample means or
two sample proportions, following the five-step model and correctly interpret the results
• List and explain each of the factors (especially sample size) that affect the probability of rejecting the null hypothesis
• Explain the differences between statistical significance and importance
2
Basic logic• We analyze a difference between two sample
statistics– We compare means or proportions of two samples
from specific sub-groups of the population
• This is the question under consideration– “Is the difference between the samples large enough
to allow us to conclude (with a known probability of error) that the populations represented by the samples are different?”
3Source: Healey 2015, p.217.
Null hypothesis• The H0 indicates that the populations are the
same– Assuming that the H0 is true, there is no difference
between the parameters of the two populations
• On the other hand, we reject the H0 and say there is a difference between the populations– If the difference between the sample statistics is large
enough– Or if the size of the estimated difference is unlikely
4
H0, α, Z score, p-value• The H0 is a statement of “no difference”• The 0.05 level (α) will continue to be our
indicator of a significant difference• We change the sample statistics to a Z score
– Place the Z(obtained) on the sampling distribution• Estimate probability (p-value) above Z(obtained)
– p-value is the probability of failing to reject the null hypothesis
– Compare the p-value to the α– If p<α, we reject H0
– If p>α, we fail to reject H0
5
Test of hypothesisfor two sample means
6Source: Healey 2015, p.217.
The five-step model1. Make assumptions and meet test requirements
2. Define the null hypothesis (H0)
3. Select the sampling distribution and establish the critical region
4. Compute the test statistic
5. Make a decision and interpret the test results
7
Changes from one-sample case• Step 1
– In addition to samples selected according to EPSEM principles
– Samples must be selected independently of each other: independent random sampling
• Step 2– Null hypothesis statement will state that the two
populations are not different• Step 3
– Sampling distribution refers to difference between the sample statistics
8
Two-sample test of means(large samples)
• Do men and women significantly differ on their support of gun control?
• For men (sample 1)– Mean = 6.2– Standard deviation = 1.3– Sample size = 324
• For women (sample 2)– Mean = 6.5– Standard deviation = 1.4– Sample size = 317
9
Step 1: Assumptions,requirements• Independent random sampling
– The samples must be independent of each other
• Level of measurement is interval-ratio– Support of gun control is assessed with an interval-
ratio level scale, so the mean is an appropriate statistic
• Sampling distribution is normal in shape– Total N ≥ 100 (N1 + N2 = 324 + 317 = 641)– So the Central Limit Theorem applies and we can
assume a standard normal distribution (Z)
10
Step 2: Null hypothesis• Null hypothesis, H0: μ1 = μ2
– The null hypothesis asserts there is no difference between the populations
• Alternative hypothesis, H1: μ1 ≠ μ2– The research hypothesis contradicts the H0 and
asserts there is a difference between the populations
11
Step 3: Distribution, critical region• Sampling distribution
– Standard normal distribution (Z)
• Significance level– Alpha (α) = 0.05 (two-tailed)– The decision to reject the null hypothesis has only a
0.05 probability of being incorrect
• Z(critical) = ±1.96– If the probability (p-value) is less than 0.05– Z(obtained) will be beyond Z(critical)
12
Step 4: Test statistic• Sample outcomes for support of gun control
• Pooled estimate of the standard error
𝜎"#$"# =𝑠'(
𝑁' − 1+
𝑠((
𝑁( − 1�
=1.3 (
324 − 1 +1.4 (
317 − 1�
= 0.107
• Obtained Z score
𝑍 𝑜𝑏𝑡𝑎𝑖𝑛𝑒𝑑 =𝑋#' − 𝑋#(𝜎"#$"#
=6.2 − 6.50.107 = −2.80
13
Sample 1 (men) Sample 2 (women)𝑋#' = 6.2 𝑋#( = 6.5s1 = 1.3 s2 = 1.4N1 = 324 N2 = 317
Step 5: Decision, interpret
14
• Z(obtained) = –2.80– This is beyond Z(critical) = ±1.96– The obtained Z score falls in the critical region, so we
reject the H0
– Therefore, the H0 is false and must be rejected
• The difference between men’s and women’s support of gun control is statistically significant– The difference between the sample means is so large
that we can conclude (at α = 0.05) that a difference exists between the populations represented by the samples
Two-sample test of means(small samples)
• Do families that reside in the center-city have more children than families that reside in the suburbs?
• For suburbs (sample 1)– Mean = 2.37– Standard deviation = 0.63– Sample size = 42
• For center-city (sample 2)– Mean = 2.78– Standard deviation = 0.95– Sample size = 37
15
Step 1: Assumptions,requirements• Independent random sampling
– The samples must be independent of each other• Level of measurement is interval-ratio
– Number of children can be treated as interval-ratio• Population variances are equal
– As long as the two samples are approximately the same size, we can make this assumption
• Sampling distribution is normal in shape– Because we have two small samples (N < 100), we
have to add the previous assumption in order to meet this assumption
16
Step 2: Null hypothesis• Null hypothesis, H0: μ1 = μ2
– The null hypothesis asserts there is no difference between the populations
• Alternative hypothesis, H1: μ1 < μ2– The research hypothesis contradicts the H0 and
asserts there is a difference between the populations
17
Step 3: Distribution, critical region• Sampling distribution
– Student’s t distribution
• Significance level– Alpha (α) = 0.05 (one-tailed)
• Degrees of freedom– N1 + N2 – 2 = 42 + 37 – 2 = 77
• Critical t– t(critical) = –1.671
18
Step 4: Test statistic• Sample outcomes for number of children
• Pooled estimate of the standard error𝜎"#$"# =
𝑁'𝑠'( + 𝑁(𝑠((
𝑁' + 𝑁( − 2� 𝑁' + 𝑁(
𝑁'𝑁(
�=
42 0.63 ( + 37 0.95 (
42 + 37 − 2� 42 + 37
42 37�
= 0.18
• Obtained t
𝑡 𝑜𝑏𝑡𝑎𝑖𝑛𝑒𝑑 =𝑋#' − 𝑋#(𝜎"#$"#
=2.37 − 2.78
0.18 = −2.28
19
Sample 1 (suburban) Sample 2 (center-city)𝑋#' = 2.37 𝑋#( = 2.78s1 = 0.63 s2 = 0.95N1 = 42 N2 = 37
t(obtained) & t(critical)
20Source: Healey 2015, p.226.
• Sampling distribution with critical region and test statistic displayed
Step 5: Decision, interpret
21
• t(obtained) = –2.28– This is beyond t(critical) = –1.671– The obtained test statistic falls in the critical region, so
we reject the H0
• The difference between the number of children in center-city families and the suburban families is statistically significant– The difference between the sample means is so large
that we can conclude (at α = 0.05) that a difference exists between the populations represented by the samples
• We know the average income by sex from the 2016 GSS
• What causes the difference between male income of $41,583.53 and female income of $28,353.35?
• Real difference? Or difference due to random chance?
Example from GSS: t-test
22
female 28353.34628 male 41583.52814 ts sex mean(conrinc)responden
. table sex, c(mean conrinc)
Pr(T < t) = 1.0000 Pr(|T| > |t|) = 0.0000 Pr(T > t) = 0.0000 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Ho: diff = 0 degrees of freedom = 1630 diff = mean(male) - mean(female) t = 7.4918 diff 13230.18 1765.955 9766.402 16693.96 combined 1,632 34822.52 897.5571 36259.53 33062.03 36583 female 834 28353.35 1049.496 30308.45 26293.38 30413.31 male 798 41583.53 1433.963 40507.87 38768.74 44398.32 Group Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] Two-sample t test with equal variances
. ttest conrinc, by(sex)
Example from GSS: Result• Men have an average income that is significantly higher
than the female average income– The difference between male income ($41,583.53) and female
income ($28,353.35) was large and unlikely to have occurred by random chance (p<0.05) in 2016
23
Edited table
24
Table 1. Two-sample t-test of individual average income of the U.S. adult population by sex, 2004, 2010, and 2016
Sex 2004 2010 2016Male 45,741.48 37,864.34 41,583.53
(1,343.92) (1,359.39) (1,433.96)
Female 29,264.54 26,141.60 28,353.35
(972.15) (972.97) (1,049.50)
Difference 16,476.94*** 11,722.74*** 13,230.18***
(1,665.71) (1,643.94) (1,765.96)
Sample size 1,688 1,202 1,632Note: Standard errors are reported in parentheses. *Significant at p<0.10; **Significant at p<0.05; ***Significant at p<0.01.Source: 2004, 2010, 2016 General Social Surveys.
Two-sample test of proportions(large samples)
• Do Black and White senior citizens differ in their number of memberships in clubs and organizations?– Using the proportion of each group classified as
having a “high” level of membership• For Black senior citizens (sample 1)
– Proportion = 0.34– Sample size = 83
• For White senior citizens (sample 2)– Proportion = 0.25– Sample size = 103
25
Step 1: Assumptions,requirements• Independent random sampling
– The samples must be independent of each other• Level of measurement is nominal
– We have measured the proportion of each group classified as having a “high” level of membership
• Population variances are equal– As long as the two samples are approximately the
same size, we can make this assumption• Sampling distribution is normal in shape
– Total N ≥ 100 (N1 + N2 = 83 + 103 = 186)– So the Central Limit Theorem applies and we can
assume a standard normal distribution26
Step 2: Null hypothesis• Null hypothesis, H0: Pu1 = Pu2
– The null hypothesis asserts there is no difference between the populations
• Alternative hypothesis, H1: Pu1 ≠ Pu2– The research hypothesis contradicts the H0 and
asserts there is a difference between the populations
27
Step 3: Distribution, critical region• Sampling distribution
– Standard normal distribution (Z)
• Significance level– Alpha (α) = 0.05 (two-tailed)– The decision to reject the null hypothesis has only a
0.05 probability of being incorrect
• Z(critical) = ±1.96– If the probability (p-value) is less than 0.05– Z(obtained) will be beyond Z(critical)
28
Step 4: Test statistic• Sample outcomes for club memberships
• Population proportion𝑃C =
𝑁'𝑃D' + 𝑁(𝑃D(𝑁' + 𝑁(
=83 0.34 + 103 0.25
83 + 103 = 0.29
• Pooled estimate of the standard error
𝜎E$E = 𝑃C 1 − 𝑃C� 𝑁' + 𝑁(
𝑁'𝑁(
�= 0.29 0.71� 83 + 103
83 103�
= 0.07
• Obtained Z score𝑍 𝑜𝑏𝑡𝑎𝑖𝑛𝑒𝑑 =
𝑃D' − 𝑃D(𝜎E$E
=0.34 − 0.25
0.07 = 1.29
29
Sample 1 (Black senior citizens) Sample 2 (White senior citizens)Ps1 = 0.34 Ps2 = 0.25N1 = 83 N2 = 103
Step 5: Decision, interpret
30
• Z(obtained) = 1.29– This is below the Z(critical) = 1.96– The obtained test statistic does not fall in the critical
region, so we fail to reject the H0
• The difference between the memberships of Black and White senior citizens is not significant– The difference between the sample means is small
enough that we can conclude (at α = 0.05) that no difference exists between the populations represented by the samples
Democrats .4559471Republicans .117096 party mean(proimmig)Political
. table democrat, c(mean proimmig)
• We know the proportion of pro-immigrants by political party from the 2016 GSS
• What causes the difference between the percentage of Republicans who a pro-immigration (11.7%) and the percentage of Democrats who are pro-immigration (45.6%)?– Real difference? Or difference due to random chance?
Example from GSS: proportion
31
Pr(Z < z) = 0.0000 Pr(|Z| > |z|) = 0.0000 Pr(Z > z) = 1.0000 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
Ho: diff = 0 diff = prop(Republicans) - prop(Democrats) z = -11.0581 under Ho: .0306428 -11.06 0.000 diff -.3388511 .0280803 -.3938875 -.2838147 Democrats .4559471 .0233749 .4101332 .5017611 Republicans .117096 .0155602 .0865987 .1475934 Variable Mean Std. Err. z P>|z| [95% Conf. Interval] Democrats: Number of obs = 454Two-sample test of proportions Republicans: Number of obs = 427
. prtest proimmig, by(democrat)
Example from GSS: Result• Republicans are less pro-immigration than Democrats
– The difference between the percentage of Republicans who are pro-immigration (11.7%) and the percentage of Democrats who are pro-immigration (45.6%) was large and unlikely to have occurred by random chance (p<0.05) in 2016
32
Edited table
33
Table 2. Test of proportions of pro-immigrants among the U.S. adult population by political party, 2004, 2010, and 2016
Political Party 2004 2010 2016Republican 0.0911 0.1429 0.1171
(0.0124) (0.0193) (0.0156)
Democratic 0.2164 0.2761 0.4559
(0.0178) (0.0223) (0.0234)
Difference –0.1253*** –0.1333*** –0.3389***
(0.0217) (0.0295) (0.0281)
Sample size 1,074 731 881Note: Standard errors are reported in parentheses. *Significant at p<0.10; **Significant at p<0.05; ***Significant at p<0.01.Source: 2004, 2010, 2016 General Social Surveys.
Statistical significancevs. importance (magnitude)
34
• As long as we work with random samples, we must conduct a test of significance
• Statistical significance is not the same thing as importance– Importance is also known as magnitude of the effect
• Differences that are otherwise trivial or uninteresting may be significant
Influence of sample size
35
• When working with large samples, even small differences may be statistically significant
• The larger the N– The greater the value of the test statistic– The more likely it will fall in the critical region and be
declared statistically significant
• In general, when working with random samples, statistical significance is a necessary but not a sufficient condition for importance
Sample size & test statistic
36Source: Healey 2015, p.234.
Outcomes of hypothesis testing
37
• Result of a specific analysis could be
– Statistically significant and• Important (large magnitude)
– Statistically significant, but• Unimportant (small magnitude)
– Not statistically significant, but• Important (large magnitude)
– Not statistically significant and• Unimportant (small magnitude)
Factors influencing the decision
38
1. The size of the observed difference– For larger differences, we are more likely to reject H0
2. The value of alpha– Usually the decision to reject the null hypothesis has
only a 0.05 probability of being incorrect– The higher the alpha
• The more likely we are to reject the H0
• But we would have a higher chance of being incorrect
3. The use of one- vs. two-tailed tests– We are more likely to reject H0 with a one-tailed test
4. The size of the sample (N)– For larger samples, we are more likely to reject H0
