1

An Introduction to the Statistics of Uncertainty

Tom LaBoneApril 17, 2009SRCHPS Technical Seminar

MJW CorporationUniversity of South Carolina

2

Introduction All physical measurements must be reported with some

quantitative measure of the quality of the measurement needed to decide if the measurement is suitable for a particular

purpose The concept of “uncertainty” was developed in metrology to

partially fill this need The US Guide to the Expression of Uncertainty in

Measurement ANSI/NCSL Z540-2-1997 (R2007) provides guidance on calculating and reporting the uncertainty in a measurement US version of the ISO guide referred to as the “GUM”

3

Overview Illustrate the use of the GUM methodology

using a relatively simple physical systemCombined Standard Uncertainty

Type A uncertainty onlyProbability DistributionsExpanded UncertaintyMonte Carlo MethodsType B UncertaintyStudent’s t Distribution

4

Example

The SRS Health Physics Instrument Calibration Laboratory “sells” its radiation fields as a product

The uncertainty attached to a radiation field helps the customer decide if the “product” is suitable for their application

This is a rather involved case for such a short talk, so let us work with a less complex example

5

What is the density of the cube?

Measure the height, width, length, and mass of the cube

Calculate the density using this formula

1

1 1 1

M

H W L

34.48621.467973 /

1.578 1.568 1.574

gg cm

cm cm cm

Single measurements

6

Measurand The measurands we directly measure (mass

and dimensions) are called input quantities The measurand we calculate (the density) is

called the output quantity In this discussion the input quantities are

assumed to be uncorrelated e.g., the measurement of the height does not

influence the measurement of the length

1

1 1 1

M

H W L

7

Variability

If we repeated the measurements again would we expect to see exactly the same result?

Our measurements of dimension and mass will exhibit variability if we measure the “same thing” repeatedly we are

likely get a range of answers that vary in a seemingly random fashion

31.467973 g/cm

8

Why Do Measurements Vary? Every measurement is influenced by a

multitude of quantities that are not under our control and of which we may not even be aware (influence quantities)Random effects

Measurements also vary because the measurand is not and cannot be specified in infinite detailFor example, I did not specify how the linear

measurements of the cube should be made

9

Errors Using the input and output quantities we

have defined the “true” value of the density The error in a measurement is defined as

The true value and hence the error are unknowable, but errors can be classified by how they influence the measurementrandom and systematic errors

error = measured value of density – “true” value of density

10

Types of Errors Random errors result from random effects in the

measurement the magnitude and sign of a random error changes

from measurement to measurementmeasurements cannot be corrected for random errors

…but random errors can be quantified and reduced Systematic errors results from systematic effects

in the measurement the magnitude and sign of a systematic error is

constant from measurement to measurementmeasurements can be corrected for known systematic

errors …but the correction introduces additional random errors

11

What can we do about random errors?

Law of Large Numbers If you repeat measurements many times and

take the mean, this sample mean is a good estimator of the true population mean and is taken to be the best estimate of the thing we defined as the measurand

Plug the sample means into the equation to obtain the best estimate of

M

H W L

sample means

12

Repeated Measurements

1

N

ii

xx

N

M

H W L

31.46662771 /g cm

Sample Mean

13

Precision of Result

Precision is the number of digits with which a value is expressed The calculations here were performed to the internal

precision of the computer (~16 digits) The density is arbitrarily presented with 9 digits of

precision

In which digit do we lose physical significance?

31.46662771 /g cm 31.5 /g cm

14

Uncertainty “…parameter associated with the result of a

measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand….” an interval that we are reasonably confident contains

the true value of the measurand the terms “random” and “systematic” are used with the

term “error” but not with the term “uncertainty” associated with the measurement, not the measurement

process

15

Evaluation of Uncertainty

Type A evaluation of uncertaintyevaluation of uncertainty by the statistical

analysis of repeated measurementscalled Type A uncertainty

Type B evaluation of uncertaintyevaluation of uncertainty by any other methodcalled Type B uncertainty

16

Repeated Measurements

1

N

ii

xx

N

21

1

N

ii

x xs

N

Sample mean

sample standard deviation

17

Standard Uncertainty of Inputs

The sample standard deviation ss is a term in statistics with a precise meaning

In metrology the analogous term is standard uncertainty uuFor Type A evaluations the standard deviation

is the standard uncertaintyThis may not true for Type B evaluations

0.0023Lu L s cm

18

Significant Digits Report uncertainty to 2 digits

round to even number if the last digit is 5

Round the measurement to agree with the reported uncertainty

19

Uncertainty in Density

We have calculated the standard uncertainty in the input quantities (length, mass, etc)

How do we get the standard uncertainty in the output quantity (density)? the combined standard uncertainty

Propagation of uncertainty

20

Combined Standard Uncertainty

2 2 2 22 2 2 2 2( ) ( ) ( ) ( ) ( )cu u L u W u H u M

L W H M

2

MMLWH

L L LWH

2 2 2 22 2 2 2 2

2 2 2

1( ) ( ) ( ) ( ) ( )c

M M Mu u L u W u H u M

LWH LW H LWH LWH

Given a small change in the length ofthe cube how much does the density change?

Units must match up properly!

sensitivity coefficient (often abbreviated as “c”)

21

Standard Deviation of the Mean

21

1

N

ii

x xs

N

describes how individualmeasurements are scatteredaround their mean

x

ss

N

describes how repeated estimatesof the mean are scattered aroundtheir grand mean (mean of the means)

22

Which Standard Deviation Should We Use?

Sample standard deviation If you want to describe how individual measurements

are scattered about their mean Standard deviation of the mean

If you want to describe how multiple estimates of the mean are scattered about their grand mean

also called the standard error of mean We need to use the standard deviation of the

mean in the error propagation

23

Combined Standard Uncertainty2 2 2 2

2 2 2 22 2 2

1( ) ( ) ( ) ( ) ( )c

M M Mu u L u W u H u M

LWH LW H LWH LWH

-4 38.8 10 /cu g cm

2

22

5.71722g( ) (0.00059cm) ........

1.5764cm 1.5700cm 1.5751cmcu

-4 314666.3 10 / cg m

= 1.46663 g/cm3 with a combined standard uncertainty uc = 8.8 x 10-4 g/cm3

Type A uncertainty only

24

Where We Are

We have calculated the density and its combined standard uncertainty (Type A uncertainty only)

Next, we want to calculate the expanded uncertainty and address the Type B uncertainty

But, we need to discuss probability distributions and other such things first

25

Probability Distributions

Up to this point we have described our data with the mean (central tendency) the standard deviation (dispersion)

The mean and standard deviation do not uniquely specify the data

Use a mathematical model that defines the probability of observing any given result probability density function (pdf)

26

0 2 4 6 8 10

0.0

00

.02

0.0

40

.06

0.0

80

.10

0.1

2

x

f(x)

Uniform (Rectangular) PDF

a=1 b=9

5

2

a b

22 5.3312

b a

1( )f x

b a

a x b

( ) 0f x otherwise

2.31

12

b a

a = 1 b = 9

27

Rectangular PDF Notation f(x) is the rectangular probability density function

the value of the pdf is not the probability the area under the pdf is probability note that f(x) has units – probability has no units

is the population mean is the population standard deviation a is the lower bound of the distribution

a is a parameter in the pdf the probability of observing a value of x less than a is zero

b is the upper bound of the distribution b is a parameter in the pdf the probability of observing a value of x greater than b is zero

28

0 2 4 6 8 10

0.0

00

.02

0.0

40

.06

0.0

80

.10

0.1

2

x

f(x)

0 2 4 6 8 10

0.0

00

.02

0.0

40

.06

0.0

80

.10

0.1

2

x

f(x)

P(x < -1) =0.2113249 P(x < +1) =0.7886751

15 0.125

9 1f x units

( 5) 0P x

( 5) 0.5P x

The area under the curve

29

Normal PDF

22

1( ) exp

22

xf x

x

5 2 5.33

2.31

The population parameters are the parameters in the pdf – this is unusual -2 0 2 4 6 8 10 12

0.0

00

.05

0.1

00

.15

x

f(x)

30

-2 0 2 4 6 8 10 12

0.0

00

.05

0.1

00

.15

x

f(x)

-2 0 2 4 6 8 10 12

0.0

00

.05

0.1

00

.15

x

f(x)

P(x < -1) =0.1586553 P(x < +1) =0.8413447

The area under the pdf curve( 5) 0P x

( 5) 0.5P x

2

2

5 51( 5) exp 1.727

2.31 2 2 2.31f x units

31

5 2.31

P(x < +1) =0.8413447P(x < +1) =0.7886751

5 2.31

-2 0 2 4 6 8 10 120

.00

0.0

50

.10

0.1

5

x

f(x)

0 2 4 6 8 10

0.0

00

.02

0.0

40

.06

0.0

80

.10

0.1

2

x

f(x)

Normal vs Rectangular

same mean andstandard deviation

32

Sample Statistics and Population Parameters

1ˆ

N

ii

xx

N

2 2 1ˆ

1

N

ii

x xs

N

2

No matter what the probability distribution is, the sample mean and standard deviation are the best estimates (based on the observed data) of the population mean and standard deviation

33

Random Numbers

x

Fre

qu

en

cy

0 2 4 6 8 10

02

04

06

08

01

00

12

01

40

x

Fre

qu

en

cy0 2 4 6 8 10

05

01

00

15

02

00

1000 numbers drawn at randomfrom the rectangular distribution

1000 numbers drawn at randomfrom the normal distribution

34

Uses of PDFs

We use the rectangular pdf to describe a random variable that is bounded on both sides and has the equal probability of appearing anywhere between the bounds

The normal distribution has a special place in statistics because of the Central Limit Theorem

35

Central Limit Theorem

As the sample size N gets “large”, the mean of a sample will be normally distributed regardless of how the individual values are distributed

Theorem provides no guidance on what “large” is The standard deviation of the mean (aka the

standard error of the mean) is equal to

X

ss

N

36

Draw two randomnumbers from the

rectangular distribution

Take the mean of thetwo random numbers

repeat 10 5 times

Mean

De

nsi

ty

2 4 6 8

0.0

00

.05

0.1

00

.15

0.2

00

.25

0.3

0

Mean

De

nsi

ty

2 3 4 5 6 7 8

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Draw ten randomnumbers from the

rectangular distribution

Take the mean of theten random numbers

repeat 10 5 times

37

So What? No matter what probability distribution you

start with, if the sample is large enough the means of data drawn from that distribution are normally distributed

What are the practical implications of this?All the input quantities (length, etc) are meansThe input quantities are normally distributedThe output quantity (density) is normally

distributed

38

Normal Probabilities

1.464 1.466 1.468 1.470

01

00

20

03

00

40

0

x

f(x)

1.464 1.466 1.468 1.470

01

00

20

03

00

40

0

xf(

x)

The area under the normal curve between -1 and +1 = 0.6826895

The area under the normal curve between -1.96 and +1.96 = 0.95

-1.96 +1.96-1 +1

39

Expanded Uncertainty

It is often desirable to express the uncertainty as an interval around the measurement result that contains a large fraction of results that might reasonably be observed

This is accomplished by using multiples of the standard uncertainty the multiplier is called the coverage factor

40

Intervals Confidence interval

interval constructed with standard deviations from known probability distributions

the interval has an exact probability of covering the mean value of the measurand

Coverage interval interval constructed with uncertainties the interval does not have an exact probability of

covering the mean value of the measurand only an approximation uses a coverage factor rather than a standard normal

quantile (e.g., the 1.96) coverage factor of 2 (~95%) or 3 (~99%) is typically

used

41

Expanded Uncertainty for Density

1.46663 1.96 0.00088 1.46490,1.46836

1.46663 0.00088 1.46575,1.46751

95% Confidence Interval

68% Confidence Interval

1.46663 2 0.00088 1.46487,1.46839

95% Coverage Interval

expanded uncertainty, Type A uncertainty only

42

Monte Carlo Methods

Evaluation of measurement data – Supplement 1 to the “Guide to the expression of uncertainty in measurement” – Propagation of distributions using a Monte Carlo method OIML G 1-101 (2008)

Run a statistics experiment using random numbers

43

Draw a random lengthfrom the normal

distribution N( L,L)

Draw a random heightfrom the normal

distribution N( H ,H)

repeat 10 6 timesDraw a random width

from the normaldistribution N( W ,W )

Draw a random massfrom the normal

distribution N( M ,M )

Calculate the densityfrom the random inputs

Calculate the sample mean and standard deviations of the 106 densities

Density of Cube

De

nsi

ty (

fre

qu

en

cy)

0.001462 0.001464 0.001466 0.001468 0.0014700

e+

00

1e

+0

52

e+

05

3e

+0

54

e+

05

5e

+0

5

31.466628 / cX g m-4 38.820005 10 /s g cm

1.46490,1.46836 95% empirical CI

44

for (i in 1: (10^6)) { d[i] <- rnorm(1,M,s.M) / (rnorm(1,L,s.L) * rnorm(1,W,s.W) * rnorm(1,H,s.H))}quantile(d,probs=c(0.025,0.975))mean(d)sd(d)

Implementation in R

draw a random massdraw a random length,

width, and height

calculate a density

calculate the empirical 95% confidence interval, mean and standard deviation

1.46490,1.46836

45

Advantages of Monte Carlo

Intuitive Set up the experiment in the computer just like it

occurs in the lab

Able to handle very complex problems asymmetric probability distributions

No need to mess with the t distribution or effective degrees of freedom you will see what I am talking about shortly

46

Type B Uncertainty Assessment

Calipers Used to measure length, height, and width “Accuracy” of ± 0.02 mm (± 0.002 cm) for

measurements <100 mm

Scale Used to measure “mass” “Accuracy” of ± 0.0001 gram

What do they mean by “accuracy” and how do I use this information?

47

Calipers The “accuracy” of ± 0.002 cm is taken to mean

that if I moved the calipers from 1.500 cm to 1.502 cm the reading could be anywhere from 1.500 cm to 1.504 cm

Assume rectangular distribution with an upper limit of X+0.002 cm and a lower limit of X – 0.002 cm

The standard uncertainty of this distribution is

30.002 0.002 0.004( ) 1.155 10

12 12

X Xu l cm

48

Scale The “accuracy” of ± 0.0001 gram is taken to

mean that if the weight increased from 5.0000 grams to 5.0001 grams the reading could be anywhere from 5.0000 grams to 5.0002 grams

Assume rectangular distribution with an upper limit of X+0.0001 grams and a lower limit of X – 0.0001 grams

The standard uncertainty of this distribution is

50.0001 0.0001 0.0002( ) 5.774 10

12 12

X Xu m g

49

Standard Uncertainty for Length

Combine the Type A and Type B uncertainties in quadrature (i.e., add the variances)

2 6 2( ) 1.684 10cu L cm

22 2( ) ( )cu L u L u l

3( ) 1.298 10cu L cm

22 4 2 3( ) (5.920 10 ) 1.155 10cu L cm cm

Includes Type A andType B uncertainties

The notation uc(L) is used here toindicate the uncertainty includesType A and B uncertainties

50

Combined Standard Uncertainty for Density

2 22 22 2 2

2 2

2 22 22 2

2

( ) ( ) ( )

1.... ( ) ( )

c

M Mu u L u l u W u l

LWH LW H

Mu H u l u M u m

LWH LWH

30.0021 /cu g cm

= 1.4666 g/cm3 with a combined standard uncertainty uc = 2.1 x 10-3 g/cm3

Type A and B uncertainty

51

Type B Uncertainties Remember, once Type B uncertainties are

included in the error propagation the equivalence of standard deviations and standard uncertainties is usually lost

The assessment of Type B uncertainties usually requires some degree of professional judgment and experience

Once you decide what the Type B uncertainty is, it is treated the same as a Type A uncertainty

One goal of GUM is to make Type B uncertainties easier to handle by having people report the right information along with the uncertainty itself

52

Summary of Jargon Standard uncertainty

uncertainty of the result of a single type of measurement (e.g., length)

includes Type A and/or Type B uncertainties Combined standard uncertainty

standard uncertainties from multiple types of measurements used to calculate an output quantity (e.g., density)

Expanded uncertainty the standard uncertainty multiplied by a coverage

factor combined standard uncertainty multiplied by a

coverage factor

53

Where We Are

For the density, we have calculated combined standard uncertaintyexpanded uncertainty (including both Type A and Type B

uncertainties) The only remaining issue is to account for

the impact “small” samples have on the expanded uncertainty

54

Student’s t Distribution

X kN

sX k

N

If is given, i.e., not determined from the data, then k is a standard normal quantile, e.g., 1.96 gives a 95% confidence interval

If is not given and we use s, which is determined from the data, then k is a not standard normal quantile, it is a quantile from a Student’s t distribution

55

( 1) / 22

( 1) / 2( )

/ 2 1

f xx

Student’s t PDF

x

= degrees of freedom (df)

-4 -2 0 2 4

0.0

0.1

0.2

0.3

0.4

x

f(x)

The t distribution looks a lot likethe normal distribution but has“fatter” tails

normal

t

56

So What? A 95% coverage interval for a normal distribution

is given by 1.96s A 95% coverage interval for a t distribution might

be more like 3s (depending the df) The t distribution converges to the normal

distribution as the degrees of freedom (the size of the sample) gets large

The t distribution is needed to calculate coverage intervals for small samples taken from normally distributed data for large samples you can use normal quantiles

57

Expanded Uncertainty for Length (Type A uncertainty only)

1.57640L

0.00059L

mean of N = 15 measurements (see slide 21)

standard error of the mean length (see slide 21)

1.57640 0.00059 1.96

0.975,14 2.14t

0.975 1.96z

95% confidence interval

1.57640 0.00059 2.14

standard normal quantile for p = 0.95

95% confidence interval

t quantile for p = 0.95 (14 df)

58

Degrees of Freedom

Data can be used to estimate parameters or estimate variance degrees of freedom is the number of data available to

estimate variance after the parameters are estimated one degree of freedom was used to calculate the

mean which leaves 15 -1 = 14 left to calculate variance

The GUM views degrees of freedom as an indication of the uncertainty in the uncertainty large degrees of freedom = small uncertainty small degrees of freedom = large uncertainty

59

Expanded Uncertainty for Length (Type A and Type B uncertainty)

The Type A uncertainty of the length has 14 degrees of freedom

How many degrees of freedom does the Type B uncertainty have? unless “they” tell you the degrees of freedom, you end

up doing a bit of hand waving on the df for Type B uncertainties

the GUM gives a way to get an approximate df How many degrees of freedom does the

combined uncertainty of Length have? There is no exact solution to this problem An approximate solution is given by the Welch-

Satterthwaite equation

60

Degrees of Freedom for Type B Uncertainty

210.25 8

2

21

2

u

u

210.1 50

2

210

2

Where u/u) is the relative uncertainty in the uncertainty

25% relative uncertainty in the uncertainty

10% relative uncertainty in the uncertainty

Zero uncertainty in the uncertainty

61

Welch Satterthwaite Gives the effective degrees of freedom

associated with the expanded uncertainty in the length

22 2

2 22 2

( ) ( )

( ) ( )

14

eff

u L u l

u L u l

22

1

22

1

n

i ii

eff

n i i

ii

c s

c s

323.1eff

The sensitivity coefficients c are equal to 1 here

Use this to calculate the t quantile

The Type B uncertainty is assumedto have an infinite degrees of freedom,i.e., the uncertainty has no uncertainty

62

31.57640 /L g cm

30.0013 /cu L g cm

mean of N = 15 measurements

standard uncertainty in length (see slide 49)

1.57640 0.0013 1.96

0.975,323 1.97t

0.975 1.96z

95% coverage interval

1.57640 0.0013 1.97

standard normal quantile for p = 0.95

95% coverage interval

t quantile for p = 0.95 (323 df)

Expanded Uncertainty for Length (Type A and Type B uncertainty)

63

Expanded Uncertainty for Density(Type A Uncertainty Only)

22 2 2 22 2 2 2

2 2 2

2 2 2 22 2 2 22 2 2 2

2 2 2

1( ) ( ) ( ) ( )

1( ) ( ) ( ) ( )

14 14 14 14

eff

M M Mu L u W u H u M

LWH LW H LWH LWH

M M Mu L u W u H u M

LWH LW H LWH LWH

22

1

22

1

n

i ii

eff

n i i

ii

c s

c s

41.4eff

64

31.46663g /cm

30.00088 /cu g cm

1.46663 0.00088 1.96

0.975 1.96z

95% confidence interval

1.46663 0.00088 2.02

standard normal quantile for p = 0.95

95% confidence interval

t quantile for p = 0.95 (41 df)

= 1.46663 g/cm3 with a combined standard uncertainty uc = 8.8 x 10-4 g/cm3

0.975,41 2.02t

(see slide 23)

(see slide 23)

65

Expanded Uncertainty for Density(Type A and Type B Uncertainty)

1200eff

222 2

2

2 22 22 2

2 2

( ) ( ) ....

( ) ( )

....14

eff

Mu L u l

LWH

M Mu L u l

LWH LW H

The Type B uncertainty is assumedto have an infinite degrees of freedom,i.e., the uncertainty has no uncertainty

0.975,1200 1.96t Same as the normal distribution

66

31.4666g /cm

30.0021 /cu g cm

1.4666 0.0021 1.96

0.975 1.96z

95% coverage interval

31.4666 0.0021 1.96 1.4625, 1.4707 /g cm

standard normal quantile for p = 0.95

95% coverage interval – the final answer!

t quantile for p = 0.95 (1200 df)0.975,1200 1.96t

(see slide 50)

(see slide 50)

= 1.4666 g/cm3 with a combined standard uncertainty uc = 2.1 x 10-3 g/cm3

67

Why Bother with Student? It is important not to overstate your

confidence in a number if you make an error calculating the coverage

interval, try to make it too big When you start to include all known

sources of uncertainty, some are very likely to have small degrees of freedom the weakest link determines the strength of the

chainsame goes for uncertainty calculations

68

Monte Carlo

for (i in 1:(10^6)) { r.L <- (L + rt(1,14)*s.L) r.L <- runif(1,r.L-0.002,r.L+0.002) r.W <- (W + rt(1,14)*s.W) r.W <- runif(1,r.W-0.002,r.W+0.002) r.H <- (H + rt(1,14)*s.H) r.H <- runif(1,r.H-0.002,r.H+0.002) r.M <- (M + rt(1,14)*s.M) r.M <- runif(1,r.M-0.0001,r.M+0.0001) d[i] <- r.M / (r.L * r.W * r.H)}quantile(d,probs=c(0.025,0.975))mean(d)sd(d)

MC 95% coverage interval = (1.4626, 1.4707)

Density of CubeD

en

sity

(fr

eq

ue

ncy

)1.460 1.465 1.470 1.475

05

01

00

15

02

00

Expanded Uncertainty for Density(Type A and Type B Uncertainty)

R Code

GUM 95% coverage interval = (1.4625, 1.4707) (see slide 66)

69

Summary Combined Standard Uncertainty

Type A uncertainty only Probability Distributions

rectangular, normal Expanded Uncertainty Monte Carlo Methods Type B Uncertainty

uncertainty in calipers and scale Student’s t Distribution

degrees of freedom in Type B uncertainty degrees of freedom in combined uncertainty

Welch-Satterthwaite

70

Recommended Reading The GUM and its Monte Carlo supplement

already cited An Introduction to Uncertainty in

Measurement by Les Kirkup and Bob Frenkel (Cambridge University Press:2006)

An Introduction to Error Analysis by John Taylor (University Science Book: 1982)