Nonparametric tests II

as randomisation tests

Lecture Outline

• Background: Nonparametric tests as randomisation tests– The sign test– The Wilcoxon signed ranks test– The Mann-Whitney test

• General remarks on randomisation tests

• Brief Review of the course so far

after before640.0 1050.0 70.0 84.0 83.0 77.0 64.0 110.0420.0 440.0 6.4 4.8 26.0 48.0 2.2 16.0 75.0 340.0 16.0 430.0

after before change640.0 1050.0 -410.0 70.0 84.0 -14.0 83.0 77.0 6.0 64.0 110.0 -46.0420.0 440.0 -20.0 6.4 4.8 1.6 26.0 48.0 -22.0 2.2 16.0 -13.8 75.0 340.0 -265.0 16.0 430.0 -414.0

after before change640.0 1050.0 -410.0 70.0 84.0 -14.0 83.0 77.0 6.0 64.0 110.0 -46.0420.0 440.0 -20.0 6.4 4.8 1.6 26.0 48.0 -22.0 2.2 16.0 -13.8 75.0 340.0 -265.0 16.0 430.0 -414.0

schange-414.0-410.0-265.0-46.0-22.0-20.0-14.0-13.81.66.0

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Mann-Whitney

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

which added = number of non-zero datapoints (in this case there are no zeroes)

So if we take ten items that might be plus or minus,

So if we take ten items that might be plus or minus,and randomly choose them, we get the set of relevant comparisons for our dataset of 8 minus and 2 plus. This is the randomisation part of the test.

So if we take ten items that might be plus or minus,and randomly choose them, we get the set of relevant comparisons for our dataset of 8 minus and 2 plus. This is the randomisation part of the test.

To decide whether our actual dataset is extreme in the distribution, we calculate the test statistic in each case - just the number of plusses. We count in what fraction of cases, the relevant comparison has a more extreme number of plusses, that is, either 2 or fewer, or 8 or more.

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

109876543210

200

100

0

C3

109876543210

200

100

0

C3

The truth about confidence intervals

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00MTB >

MTB > stest 0 c3

Sign Test for Median: C3

Sign test of median = 0.00000 versus not = 0.00000

N Below Equal Above P MedianC3 10 8 0 2 0.1094 -21.00

MTB > stest 10 c3

Sign Test for Median: C3

Sign test of median = 10.00 versus not = 10.00

N Below Equal Above P MedianC3 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

The green values cannot be rejected at the 5% level, while the red values can.

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

The green values cannot be rejected at the 5% level, while the red values can.

The range of green values is therefore the 95% confidence interval for the median based on the sign test.

The real definition of 95% confidence interval

• is “the set of values of a parameter that cannot be rejected at the 5% level”

• is therefore not “the set of values that the parameter has a 95% chance of belonging to”, as many textbooks claim. (This is called a “fiducial interval”.)

MTB > sinterval 'change'

Sign Confidence Interval

Sign confidence interval for median

ACHIEVED POSI N MEDIAN CONFIDENCE CONFIDENCE INTERVAL TIONchange 10 -21.000 0.8906 (-265.000, -13.800) 3 0.9500 (-314.640, -8.528) NLI 0.9785 (-410.000, 1.600) 2

MTB >

H0median N Below Equal Above P Median

-50 10 3 0 7 0.3438 -21.00-40 10 4 0 6 0.7539 -21.00-35 10 4 0 6 0.7539 -21.00-30 10 4 0 6 0.7539 -21.00-25 10 4 0 6 0.7539 -21.00-20 10 5 1 4 1.0000 -21.00-15 10 6 0 4 0.7539 -21.00-10 10 8 0 2 0.1094 -21.00-5 10 8 0 2 0.1094 -21.000 10 8 0 2 0.1094 -21.005 10 9 0 1 0.0215 -21.0010 10 10 0 0 0.0020 -21.0015 10 10 0 0 0.0020 -21.00

. .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

after before change640.0 1050.0 -410.0 70.0 84.0 -14.0 83.0 77.0 6.0 64.0 110.0 -46.0420.0 440.0 -20.0 6.4 4.8 1.6 26.0 48.0 -22.0 2.2 16.0 -13.8 75.0 340.0 -265.0 16.0 430.0 -414.0

schange-414.0-410.0-265.0-46.0-22.0-20.0-14.0-13.81.66.0

97.85%89.06%

97.85%89.06%

MTB > sinterval 'change'

Sign Confidence Interval

Sign confidence interval for median

ACHIEVED POSI N MEDIAN CONFIDENCE CONFIDENCE INTERVAL TIONchange 10 -21.000 0.8906 (-265.000, -13.800) 3 0.9500 (-314.640, -8.528) NLI 0.9785 (-410.000, 1.600) 2

MTB > . .. . . .: ..---+---------+---------+---------+---------+---------+---change -400 -320 -240 -160 -80 0

Why does Minitab give three confidence intervals for the sign test?

• the p-value for rejecting a value changes in a step function at observed values

• so exact confidence intervals are given between observed values, at whatever level of confidence is attained

• the NLI (Non-Linear Interpolation) confidence interval is a confidence trick

Lecture Outline

• Background: Nonparametric tests as randomisation tests– The sign test– The Wilcoxon signed ranks test– The Mann-Whitney test

• General remarks on randomisation tests

• Brief Review of the course so far

after before change640.0 1050.0 -410.0 70.0 84.0 -14.0 83.0 77.0 6.0 64.0 110.0 -46.0420.0 440.0 -20.0 6.4 4.8 1.6 26.0 48.0 -22.0 2.2 16.0 -13.8 75.0 340.0 -265.0 16.0 430.0 -414.0

schange-414.0-410.0-265.0-46.0-22.0-20.0-14.0-13.81.66.0

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

MTB > wtest 'change'

Wilcoxon Signed Rank Test

TEST OF MEDIAN = 0.000 VERSUS MEDIAN N.E. 0.000

N FOR WILCOXON ESTIMATED N TEST STATISTIC P-VALUE MEDIANchange 10 10 3.0 0.014 -46.00MTB >

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00

MTB > wtest 'change'

Wilcoxon Signed Rank Test

TEST OF MEDIAN = 0.000 VERSUS MEDIAN N.E. 0.000

N FOR WILCOXON ESTIMATED N TEST STATISTIC P-VALUE MEDIANchange 10 10 3.0 0.014 -46.00MTB >

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00

MTB > wtest 'change'

Wilcoxon Signed Rank Test

TEST OF MEDIAN = 0.000 VERSUS MEDIAN N.E. 0.000

N FOR WILCOXON ESTIMATED N TEST STATISTIC P-VALUE MEDIANchange 10 10 3.0 0.014 -46.00MTB >

MTB > stest 'change'

Sign Test for Median

Sign test of median=0.000 versus N.E. 0.000

N BELOW EQUAL ABOVE P-VALUE MEDIANchange 10 8 0 2 0.1094 -21.00

The Wilcoxon test is more powerful than the Sign Test

MTB > sinterval 'change'

Sign Confidence Interval

Sign confidence interval for median ACHIEVED POSI N MEDIAN CONFIDENCE CONFIDENCE INTERVAL TIONchange 10 -21.000 0.8906 (-265.000, -13.800) 3 0.9500 (-314.640, -8.528) NLI 0.9785 (-410.000, 1.600) 2

MTB > winterval 'change'

Wilcoxon Signed Rank Confidence Interval

ESTIMATED ACHIEVED N MEDIAN CONFIDENCE CONFIDENCE INTERVALchange 10 -46 94.7 ( -218, -8)MTB >

MTB > sinterval 'change'

Sign Confidence Interval

Sign confidence interval for median ACHIEVED POSI N MEDIAN CONFIDENCE CONFIDENCE INTERVAL TIONchange 10 -21.000 0.8906 (-265.000, -13.800) 3 0.9500 (-314.640, -8.528) NLI 0.9785 (-410.000, 1.600) 2

MTB > winterval 'change'

Wilcoxon Signed Rank Confidence Interval

ESTIMATED ACHIEVED N MEDIAN CONFIDENCE CONFIDENCE INTERVALchange 10 -46 94.7 ( -218, -8)MTB >

The Wilcoxon confidence interval is narrower

Sign vs Wilcoxon Signed Ranks

Sign vs Wilcoxon Signed Ranks

• Less powerful • More powerful

Sign vs Wilcoxon Signed Ranks

• Less powerful– Less sensitive– Wider confidence

intervals

• More powerful– More sensitive– Narrower confidence

intervals

Sign vs Wilcoxon Signed Ranks

• Less powerful– Less sensitive– Wider confidence

intervals

• Uses less information– only sign of

difference

• More powerful– More sensitive– Narrower confidence

intervals

• Uses more information– also size of

difference

after before change640.0 1050.0 -410.0 70.0 84.0 -14.0 83.0 77.0 6.0 64.0 110.0 -46.0420.0 440.0 -20.0 6.4 4.8 1.6 26.0 48.0 -22.0 2.2 16.0 -13.8 75.0 340.0 -265.0 16.0 430.0 -414.0

schange-414.0-410.0-265.0-46.0-22.0-20.0-14.0-13.81.66.0

Lecture Outline

• Background: Nonparametric tests as randomisation tests– The sign test– The Wilcoxon signed ranks test– The Mann-Whitney test

• General remarks on randomisation tests

• Brief Review of the course so far

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

Two groupsfrom samedistribution

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

Two groupsfrom samedistribution

Set of ranks;numbers ineach group

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

Two groupsfrom samedistribution

Set of ranks;numbers ineach group

Groupmembershipof datapoints

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

Two groupsfrom samedistribution

Set of ranks;numbers ineach group

Groupmembershipof datapoints

Sum of ranks infirst group

H0 FixedInformation

RandomisedInformation

Test Statistic Notes

Sign Prob of +equalsProb of -

Number ofnon-zerodifferences

Sign of eachdifference

Number of +’s BinomialDistributionin Theory

WilcoxonSignedRanks

Symmetryabout zero

Absolutevalues ofdifferences

Sign of eachdifference

Sum of ranks ofnegativedifferences

Numberedcoin tossing

Mann-Whitney

Two groupsfrom samedistribution

Set of ranks;numbers ineach group

Groupmembershipof datapoints

Sum of ranks infirst group

Randomassignmentto groups

Lecture Outline

• Background: Nonparametric tests as randomisation tests– The sign test– The Wilcoxon signed ranks test– The Mann-Whitney test

• General remarks on randomisation tests

• Brief Review of the course so far

In these randomisation tests,

• there is a simple direct connection between the null hypothesis and the randomisation procedure

• there is freedom of choice of test statistic

• estimation relies on scales of measurement and so is not as ‘principled’ as hypothesis tests

Lecture Outline

• Background: Nonparametric tests as randomisation tests– The sign test– The Wilcoxon signed ranks test– The Mann-Whitney test

• General remarks on randomisation tests

• Brief Review of the course so far

Last remarks• Randomisation tests are powerful tools

• All parametric and nonparametric tests can be understood as randomisation tests

• Nowadays they are used when no others can be used.

• NEXT WEEK: Conclusion to course and some exam questions. READ Chapter 14 of textbook.