Day 10: Statistical Tests - AnIntroduction

Daniel J. Mallinson

School of Public AffairsPenn State Harrisburgmallinson@psu.edu

PADM-HADM 503

Mallinson Day 10 October 16, 2017 1 / 50

Road map

Overview

Steps in testing statisical significance

Chi-square

T-test

Mallinson Day 10 October 16, 2017 2 / 50

Overview

Remember our basic model:

?X −−−−−−−−→ Y

Independent DependentVariable Variable

We want ot find out if there is a statistical relationship between Xand Y

Mallinson Day 10 October 16, 2017 3 / 50

Overview

To understand the basics of statistical tests, we should rememberthat in statistical testing we actually ask three questions:

1 Is the relationship between the variables significant?

2 How strong is the relationship?

3 What is the nature of the relationship between the variables?

Mallinson Day 10 October 16, 2017 4 / 50

Overview

To answer the first question, you will conduct a test of statisticalsignificance

Only if you find a significant relationship, should you attemptto answer the second and third questions

Mallinson Day 10 October 16, 2017 5 / 50

Overview

If you find a significant relationship, then:

To answer the second question (strength), you will use ameasure of association

The method you use to answer the third question depends on:

The type of variable you have (levels of measurement)The particular significance test you conduct

The next several classes will illustrate different tests

Mallinson Day 10 October 16, 2017 6 / 50

Is There a Significant Relationship?

Process for answering the first question:

1 State the research (alternative) and null hypotheses

2 Select and alpha level

3 Select and compute a test of statistical significance

4 Make a decision (is the relationship significant?)

Mallinson Day 10 October 16, 2017 7 / 50

1. State Hypotheses

Research hypothesis (H1): There is a relationship between thevariables

Null hypothesis (H0): There is no relationship

Note that statistical tests actually test the null hypothesis

Mallinson Day 10 October 16, 2017 8 / 50

2. Select An Alpha Level

You should select and alpha (α) level before you conduct yoursignificance test

Necessary for making a decision after the test

The alpha level will be your decision criterion

Most commonly used: .05, .01, .001

When SPSS calculates the probability statistic, you will compareit against the level selected to make a decision

Mallinson Day 10 October 16, 2017 9 / 50

2. Select An Alpha Level

In more technical terms:

Alpha level is the probability of a Type 1 error

Errors:

Type I Error (α): Rejecting a true null hypothesisType II Error (β): Failing to reject a false null hypothesis

We generally try to minimize Type I error

Mallinson Day 10 October 16, 2017 10 / 50

Better Understanding Alpha

Connection to sampling

Recall our discussion of sampling

All statistical tests assume you are analyzing sample data from apopulation

That sample is one of many possible samples

Mallinson Day 10 October 16, 2017 11 / 50

Better Understanding AlphaExample of sampling

Figure: Source: LaMorte 2016Mallinson Day 10 October 16, 2017 12 / 50

Better Understanding Alpha

Connection to sampling

Sampling distribution is the distribution of sample statistics weare interested in (e.g., mean)

Normal distribution is most typical, but not only, samplingdistribution

Mathematical properties of the normal distribution help us makesome statistical calculations

Mallinson Day 10 October 16, 2017 13 / 50

Better Understanding Alpha

Figure: https://www.mathsisfun.com/data/standard-normal-distribution.html

Note: Percentages of the area under the curve are fixed, animportant mathematical property

Mallinson Day 10 October 16, 2017 14 / 50

Better Understanding Alpha

Think of the normal as a sampling distribution - the distributionof all possible sample statistics that can be selected from apopulation

The arithmetic mean equals the population parameter

More sample statistics are closer to the center than far away -this is important

Mallinson Day 10 October 16, 2017 15 / 50

Better Understanding Alpha

Your sample could be at any of the positions on the normaldistribution

Chances of being closer to the center (i.e., correctly estimatingthe parameter) are higher than being away from it

Alpha level is a point that indicates how far away from thecenter (population parameter) we can tolerate being

Mallinson Day 10 October 16, 2017 16 / 50

Better Understanding AlphaThe KEY is that null hypothesis testing is examining the probabilitythat your statistic is different than the null

Figure: “p-value” by Chen-Pan Liao, CC BY-SA 3.0

Mallinson Day 10 October 16, 2017 17 / 50

Better Understanding Alpha

To calculate alpha:

Move two standard deviations (σ) away from the mean

Area of curve that remains in the tails is approximately 4.6%

If you move 1.96 σ, area in tails is 5%

This is the basis of calculating an alpha level of .05

Mallinson Day 10 October 16, 2017 18 / 50

Better Understanding Alpha

Normal curve not used for all tests

t-scores (t-tests), ANOVA, chi-square tests have their owndistributions

Distribution of t-test is similar to normal, but flatter

Mallinson Day 10 October 16, 2017 19 / 50

3. Select and Compute a Test of StatisticalSignificance

How to select a particular test:

Selection depends mainly on:

The level of measurementThe nature of the population a sample was drawn from

Mallinson Day 10 October 16, 2017 20 / 50

3. Select and Compute a Test of StatisticalSignificance

Level of Measurement

Level of Measurement Test of Significance Measure of Association Nature of Relationship

Interval or Ratio t-test, ANOVA Pearson’s r Regression EquationEta square (Y = a + βX )

Ordinal Chi-square Gamma Interpret contingencySomer’s d table.

Nominal Chi-square Lambda Interpret contingencyCramer’s V table.

Mallinson Day 10 October 16, 2017 21 / 50

3. Select and Compute a Test of StatisticalSignificance

Nature of Population

Less important, you have some flexibility here

Parametric tests require that the population from which asample was drawn has a certain type of distribution (normal)

Non-parametric tests do not have such a requirement

Some argue that under some conditions a researcher can useparametric tests even when the requirement is not met.

Mallinson Day 10 October 16, 2017 22 / 50

3. Select and Compute a Test of StatisticalSignificance

Nature of Population

Parametric are more powerful: easier to detect a significantrelationship

Both options not available for all levels of measurement

Mallinson Day 10 October 16, 2017 23 / 50

Tests of Statistical Significance

Univariate Tests

Lowest Level of Parametric Non-ParametricMeasurementOne-sample tests One-sample t-testDV: IntervalOne-sample tests Friedman test(repeated measures)DV: IntervalOne-sample tests Chi-square testDV: Nominal Binomial test

Mallinson Day 10 October 16, 2017 24 / 50

Tests of Statistical SignificanceBivariate Tests

Lowest Level of Parametric Non-ParametricMeasurementPaired Samples Paired-samples t-test Wilcoxon testIV: NominalDV: IntervalPaired Samples Sign testIV: NominalDV: OrdinalPaired Samples McNemar’s testIV: NominalDV: NominalTwo Independent Samples T-test for Mann-Whitney testIV: Nominal independent samplesDV: IntervalTwo Independent Samples Chi-squareIV: NominalDV: Nominal3+ Independent samples One-way ANOVA Kruskal-Wallis testIV: NominalDV: IntervalMultiple Independent Observations Linear RegressionIV: IntervalDV: Interval

Mallinson Day 10 October 16, 2017 25 / 50

Tests of Statistical Significance

Multivariate Tests

Lowest Level of Parametric Non-ParametricMeasurementTwo Independent Variables Two-way ANOVAIV1: NominalIV2: NominalDV: IntervalMultiple Independent Variables Multiple regressionIV1: IntervalIV2: Interval...IVn: IntervalDV: Interval

Mallinson Day 10 October 16, 2017 26 / 50

4. Make a Decision

Is there a significant relationship between the variables?

Decisions made according to test statistic calculated (e.g.,chi-square) and associated probability level (p-level)

Compare probability level (p) calculated by SPSS againstpre-selected alpha level

If p is smaller than alpha level, there is a significant relationshipIf p is larger than alpha level, there is no significant relationship

Mallinson Day 10 October 16, 2017 27 / 50

Thoughts on Significance

p-values and significance tests tell us about statisticalsignificance, not substantive significance

Only tells you whether you can reject the null, does not provethe research hypothesis

All kinds of relationships can be found with data (especially big),but are they meaningful?

Also does not tell you the magnitude of the relationship, need tomore on to measures of association

Also, small effects can be very meaningful

Mallinson Day 10 October 16, 2017 28 / 50

Alternatives

Null Hypothesis Testing has its limitations (see text), there arealternatives (or additions)

1 Specify size of difference in the research hypothesis

2 Report confidence intervals (statistic is not as precise as itseems)

3 Report measure of association or effect size

4 Replicate, replicate, replicate

Mallinson Day 10 October 16, 2017 29 / 50

Break!

Figure: Coffee Pause by Gerd Altmann CC0

Mallinson Day 10 October 16, 2017 30 / 50

Association: Chi-Square (χ2) Test

For two nominal-level variables: Only tests the probability thatthe two are un-related in the population

Does not provide direction or strength of the relationship

Mallinson Day 10 October 16, 2017 31 / 50

Calculating χ2

χ2 = Σ(fo − fe)2/fe (1)

χ2 = Chi-square

fo = Observed frequencies

fe = Expected frequencies

Calculating Expected Frequencies

(Row Total x Column Total) / N

Mallinson Day 10 October 16, 2017 32 / 50

Belle County Example

Let’s look at the relationship between familiarity with financialassistance (knowaid) and use of financial assistance (useaid)

Observed Frequencies

Yes No Row TotalNot at all familiar 3 182 185Only a little familiar 9 114 123Somewhat familiar 13 121 134Very familiar 20 33 53Column Total 45 450 495 (N)

Mallinson Day 10 October 16, 2017 33 / 50

Belle County Example

Expected Frequencies

(Row Total x Column Total) / N

Yes No Row TotalNot at all familiar 185Only a little familiar 123Somewhat familiar 134Very familiar 53Column Total 45 450 495 (N)

Mallinson Day 10 October 16, 2017 34 / 50

Belle County Example - SPSS

Let’s look at the relationship between familiarity with financialassistance (knowaid) and use of financial assistance (useaid)

First - Data Adjustments

Need to change missing values (9) to system missing

1 Transform

2 Recode Into Same Variables

3 Choose the variables

4 Click “Old and New Values”

5 Put old value you want to change (9) in “Old Value” column

6 Click “System-Missing” in “New Value” column

7 Click Continue

8 Click OK

Mallinson Day 10 October 16, 2017 35 / 50

T-Tests

Introduced in 1908 paper by “Student”

“Student” was Chemist/statistician William S.Gossett

Used test for quality control at GuinnessBrewery

Figure: Source: Guinness

Mallinson Day 10 October 16, 2017 36 / 50

One-Sample T-Test

Quality control compares a sample to a standard (e.g., acidity ina spoonful of beer)

The question is: Is the particular deviation that is observedstatistically different from the accepted standard?

This is a one-sample T-test

Comparing sample mean to expected value (i.e., populationparameter)

Mallinson Day 10 October 16, 2017 37 / 50

North Carolina Example

Federal reserve considers 5.0 percent unemployment to represent full“employment.” Is North Carolina fully employed?

Steps in Statistical Testing:

1 State the research and null hypotheses

2 Select and alpha level

3 Select and compute test

4 Make a decision

Mallinson Day 10 October 16, 2017 38 / 50

One-Sample T-Test

In SPSS:

1 Analyze

2 Compare Means

3 One-sample test

4 Select variable UnemploymentRatepct

5 Enter 5 for “test value”

6 Under “Options,” select “Confidence Interval Percentage.” Let’suse 95%

Mallinson Day 10 October 16, 2017 39 / 50

One-Sample T-Test

Interpreting the table:

Mean unemployment rate for NC counties is 10.3

Mean difference from 5 is 5.3

Sig. (i.e., p-value) is well below 0.05, so we reject the null

May also want to report confidence intervals to give reader senseof potential range of parameter values

Mallinson Day 10 October 16, 2017 40 / 50

Paired Sample T-Tests

Used for a before and after comparison for the same set ofsubjects

Comparing means of same subjects before and after treatment

Mallinson Day 10 October 16, 2017 41 / 50

Paired Sample T-Tests

An SPSS example:

Data file: endorph.sav (Canvas)

RQ: Does running a marathon make a difference in a runner’slevel of endorphins?

Running the marathon is the IV, endorphin level is the IV(non-directional)

BUT, we want to compare before and after, assuming thatrunning is the intervention

Mallinson Day 10 October 16, 2017 42 / 50

Paired Sample T-Tests

Steps in SPSS:

1 Analyze

2 Compare Means

3 Paired samples t-test

4 Select “before” and “after” variables

Mallinson Day 10 October 16, 2017 43 / 50

Two Independent Samples T-Test

Examining if the difference between two groups is statisticallysignificant

Ideally used for a true experiment, but can be used for after-onlydesigns

To be able to use:

Your IV should be dichotomousYour DV should be scale

Mallinson Day 10 October 16, 2017 44 / 50

Two Independent Samples T-TestSPSS Example:

Do men watch more TV and spend more time on the Internetper day than women?

Data: gssnet.sav

DVs: Hours per day watching TV and hours per day using theInternet

SPSS Commands1 Analyze

2 Compare Means

3 Independent samples t-test

4 Identify “test variables” (DV) and the “grouping variable” (IV)

5 Also click on “define groups” and enter values of the IV carefullythere

Mallinson Day 10 October 16, 2017 45 / 50

Two Independent Samples T-Test

Interpreting SPSS output:1 Interpret the “Levene’s Test for Equality of Variances”

Variances are equal, the probability levels (Sig.) for bothvariables are .220 and .630, which is higher than .05Thus, use equal variances assumed results (first and third rows)

2 Interpret the significance level

.197 and .946 are higher than .05

3 Interpret the group differences

Cannot reject the null hypothesis

Mallinson Day 10 October 16, 2017 46 / 50

Bar Plots

A great way to visualize difference of means

1 Graphs

2 Legacy Dialogs

3 Bar

4 Simple

5 Click “Other statistic” under “Bars Represent”

6 Axis variable is the nominal group indicator

7 Click Options and select “Display error bars,” set desiredconfidence intervals (95 for .05 alpha)

8 Compare confidence intervals to see if they overlap

Mallinson Day 10 October 16, 2017 47 / 50

Example

Mallinson Day 10 October 16, 2017 48 / 50

Lab/Homework

Problem 1

Finish calculating the χ2 by hand for the Belle County Example (seeslides 32 and 33). Show all of your work. Then use Excel or SPSS tocalculate the same thing. Describe explain what the results mean.

Problem 2

Use the appropriate t-test and the gssnet.sav dataset to test whetherInternet users (usenet) watch less television (tvhours) than non-users.Again, make sure you explain the meaning of the results.

Mallinson Day 10 October 16, 2017 49 / 50

Questions?

Figure: Q&A by Libby Levi, CC BY-SA 2.0

Mallinson Day 10 October 16, 2017 50 / 50