1/26/09 1 Community Health Assessment in Small Populations: Tools for Working With “Small...

1/26/09 1

Community Health Assessment in Small Populations: Tools for Working With “Small

Numbers”

Community Health Assessment in Small Populations: Tools for Working With “Small

Numbers”Region 2 Quarterly Meeting

January 26, 2009

1/26/09 New Mexico Department of Health 2

OutlineOutline

Description of the Problem– Random variation– Survey samples versus complete count

datasets– Observed events versus underlying risk

Statistical Tools– Confidence intervals – Combining data– SMR

1/26/09 3

Small Numbers: The Problem

Small Numbers: The Problem


Random VariationRandom Variation

Exercise:– Select a sample– Calculate the median age

State of New Mexico Median age1

– 36.0

Why are they different?

1. 2007 American Community Survey, U.S. Census Bureau. Downloaded on 1/21/09 from http://factfinder.census.gov


Random Variation and Sample Size

Random Variation and Sample Size

What if we had a sample of New Mexico residents that was:– Randomly selected– n=5,000

Would it better match the state Census Bureau estimate?


Size MattersSize Matters

The larger sample helps to “cancel out” the effects of random variation. – Some sample subjects are older than the

median.– Some sample subjects are younger than the

median.– As you increase the number of sample

subjects, the differences cancel out, and you get closer to the median.


Reliability and ValidityReliability and Validity

The term "accuracy" is often used in relation to validity, while the term, "precision" is used to describe reliability.


Numerator vs. DenominatorNumerator vs. Denominator

A large sample size means we have a large denominator, but the numerator also matters.

Some methods use a “Poisson distribution,” which considers ONLY the numerator size when assessing precision.

If we have only 1 event in one year, and 2 the next year, the addition of a single event doubles the rate of occurrence.


Random Variation and Complete Count Datasets

Random Variation and Complete Count Datasets

What are some “complete count” datasets?

How do we use them for community health assessment?


All-cause Deaths by Year, New Mexico, 1999-2006

0

5,000

10,000

15,000

20,000

1999 2000 2001 2002 2003 2004 2005 2006

Nu

mb

er o

f D

eath

s

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

Po

pu

lati

on

(P

erso

n-y

ears

at

Ris

k)

All-cause Deaths Person-years at Risk


All-cause Deaths by Year, Santa Fe County, 1999-2006

0

200

400

600

800

1,000

1999 2000 2001 2002 2003 2004 2005 2006

Nu

mb

er o

f D

eath

s

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

Po

pu

lati

on

(P

erso

n-y

ears

at

Ris

k)



All-cause Deaths by Year, Union County, 1999-2006

0

20

40

60

1999 2000 2001 2002 2003 2004 2005 2006

Nu

mb

er o

f D

eath

s

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

5,000

Po

pu

lati

on

(P

erso

n-y

ears

at

Ris

k)



Diabetes Deaths by Year, Union County, 1999-2006

0

5

10

15

20

1999 2000 2001 2002 2003 2004 2005 2006

Nu

mb

er o

f D

eath

s

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

5,000

Po

pu

lati

on

(P

erso

n-y

ears

at

Ris

k)

Diabetes Deaths Person-years at Risk

Note: Diabetes deaths include deaths w ith an underlying cause coded w ith ICD-10 E10-E14. Data dow nloaded f rom http://ibis.state.health.nm.us on 1/21/09.


Diabetes Deaths by Year, New Mexico, Santa Fe County and Union County, 1999-2006

0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

Dea

ths

per

100

,000

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

Dea

ths

per

100

,000

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

Dea

ths

per

100

,000

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty


Summary of the ProblemSummary of the Problem

Measurements are subject to sampling variability, also known as random error.

Even complete count datasets are subject to random error because we use them as a reflection of the underlying disease risk or rate.


Summary of the ProblemSummary of the Problem

• A larger sample (denominator, population size) helps to “cancel out” the effects of random variation.

• Size matters, in both the numerator and the denominator.

• A measure that is relatively free from the effects of random variation is called “precise,” “reliable,” and “stable.” Those terms are synonymous.

1/26/09 19

Small Numbers: Statistical Tools

Small Numbers: Statistical Tools


Tool #1. Confidence Intervals

Tool #1. Confidence Intervals

Use confidence intervals to help you decide whether the rate is stable.

Won’t solve the problem, but will provide information to help you interpret the rates.

The stability of an observed rate is important when comparing areas or assessing whether disease risk has increased or decreased.



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

Dea

ths

per

100

,000

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

De

ath

s p

er

10

0,0

00

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

De

ath

s p

er

10

0,0

00

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty



0

50

100

150

200

250

1999 2000 2001 2002 2003 2004 2005 2006

De

ath

s p

er

10

0,0

00

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty


Calculation of 95% C.I.Calculation of 95% C.I.

The 95% confidence interval is calculated as 1.96 x Standard Error of the estimate (s.e.).

s.e. is calculated as

So the 95% C.I. is 1.96


The “Normal” DistributionThe “Normal” Distribution


Poisson DistributionPoisson Distribution


Calculation of 95% C.I.Calculation of 95% C.I. p stands for “probability.” It is the rate

without the multiplier (e.g., 100,000 for deaths). q is the complement of the probability (1 minus P).

In Union County, there were 2 diabetes deaths among the 4,470 population, for a probability of 0.00045 (45 in 100,000)


Calculation of 95% C.I.Calculation of 95% C.I. Formula: 1.96

p=0.00045, q=0.99955, n=4,470

(pq)/n = .000447 / 4470 = 0.000000100051

√(pq)/n = ………………… = 0.000316

1.96√(pq)/n = 1.96 x 0.000316 = 0.00062

Then we need to add the multiplier back in, so the confidence interval is:

+ 100,000*0.00062 = + 62


Calculation of 95% C.I.Calculation of 95% C.I. The diabetes death rate was 44.7 per 100,000.

The confidence interval statistic is applied both above and below the rate.

C.I. LL (lower limit) is: 44.7- 62 = -17.3, and since we cannot have a negative rate, we’ll call it “0”

C.I. UL (upper limit) is: 44.7 + 62 = 106.7

The diabetes death rate for Union County in 2006 was 44.7 per 100,000 (95% C.I., 0 to 106.7)



44.7 (0 - 106.7)

0

50

100

150

200

250

1999 2000 2001 2002 2003 2004 2005 2006

De

ath

s p

er

10

0,0

00

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty




Confidence Interval FactoidsConfidence Interval Factoids

The confidence interval may be thought of as the range of probable true values for a statistic.

The confidence interval is an indication of the precision (stability, reliability) of the estimate.

A confidence interval is typically expressed as a symmetric value (e.g., "plus or minus 5%"). But for percentages, when the point estimate is close to 0% or 100%, a confidence interval with an asymmetric shape can be used.


More Confidence Interval Factoids


The 95% confidence interval (calculated as 1.96 times the standard error of a statistic) indicates the range of values within which the statistic would fall 95% of the time if the researcher were to calculate the statistic from an infinite number of samples of the same size drawn from the same base population. Unless otherwise stated, a confidence interval will be the "95% confidence interval."




The 90% confidence interval, also commonly used, is calculated as 1.65 times the standard error of the estimate.

To calculate a confidence interval when the number of health events = 0, you may use 0 as the lower confidence limit, and for the upper confidence limit, assume a count of 3 health events in the same population.


Tool #2. Combine DataTool #2. Combine Data

Combine years

Combine geographic areas (e.g., use the regional estimate rather than the county estimate)

Use a broader age group



0

20

40

60

80

100

120

140

160

180

200

1999 2000 2001 2002 2003 2004 2005 2006

Dea

ths

per

100

,000

Po

pu

lati

on

State

Santa Fe Cnty

Union Cnty


Diabetes Deaths by Year, Union County, 1999-2006

0

50

100

150

200

250

1999 2000 2001 2002 2003 2004 2005 2006 1999-2006

De

ath

s p

er

10

0,0

00

Po

pu

lati

on


Diabetes Deaths, New Mexico, Santa Fe and Union Counties, 1999-2006

41.3

20.4

30.4

0

10

20

30

40

50

60

70

New Mexico Santa Fe Cnty Union Cnty

De

ath

s p

er

10

0,0

00

Po

pu

lati

on


Interpretation of Diabetes Deaths in Union County

Interpretation of Diabetes Deaths in Union County

Union County’s diabetes death rate (1999-2006) was higher than the state, overall rate, but was not statistically significantly higher.

In other words, the Union County rate was “marginally higher” than the New Mexico state rate.

Was it higher than Santa Fe County?


Differences Between Two Rates

Differences Between Two Rates

Statistical significance of a change in a rate from time1 to time2

Statistical significance of the difference between two rates in one time period (e.g., Union County versus Santa Fe County).

**Test of Proportions**


Test of ProportionsTest of Proportions

Proportion1: 1999-2006 Union County diabetes death rate: 41.3/100,000 = .000413

Proportion 2: 1999-2006 Santa Fe County diabetes death rate: 20.4/100,000 = .000204

Difference between the two proportions:

.000413 - .000204 = .000209


Test of Proportions (cont’d)Test of Proportions (cont’d) The difference between the two rates (0.00026)

must be considered in the context of the standard error of the difference between two rates (pooled standard error), computed as:

If the difference between the two rates, 0.000209, is greater than 1.96 x s.e.diff, then the difference is considered statistically significant.

Bruning, J.L., and Kintz, B.L. (1977) Computational Handbook of Statistics. Scott, Foresman and Company: London.


Calculation of s.e.diffCalculation of s.e.diff

Union County:– p1=0.000413– q1=0.999587– n1=33,929

Santa Fe County: – p2=0.000204 – q2=0.999796 – n2=1,092,565

p=proportion, q=(1-p), n is the person-years at risk, or the sum of the population counts across all eight years.


Calculation of s.e.diffCalculation of s.e.diff


Evaluation of the DifferenceEvaluation of the Difference Union County: 41.3/100,000 = .000413

Santa Fe County: 20.4/100,000 = .000204

Difference: .000413 - .000204 = .000209

s.e.diff = .0001111

1.96 * s.e.diff = .000218

Is .000209 greater than .000218?

No. Union County’s rate is greater than Santa Fe County’s rate, but the difference is NOT statistically significant.


Tool #3. SMR and ISRTool #3. SMR and ISR

Standardized Mortality (or Morbidity) Ratio (SMR)– Estimates the number of deaths (or health

events) one would EXPECT, based on …• The age- and sex-specific rates in a standard

population (e.g., New Mexico rate)• The age and sex distribution of the index area.

Indirectly Standardized Rates– Use SMR to perform age adjustment when

the number of cases is less than 20.


Standardized Mortality RatioStandardized Mortality Ratio

The all-cause death rate in New Mexico in 2006 was 757.5 deaths per 100,000 population.

All other things being equal, we should expect the same death rate in Union County.


Standardized Mortality RatioStandardized Mortality Ratio BUT all other things are NOT equal.

– 2006, % of population over age 65 was• 18.9% in Union County, compared with• 12.3% statewide.

In an older population, we would expect a higher death rate.


Standardized Mortality RatioStandardized Mortality Ratio And Union County’s death rate is higher:

1364.6 deaths per 100,000.

IF we adjust the New Mexico death rate to account for Union County’s older population, THAT is how many deaths we should EXPECT.


Sex Age Group

New Mexico, 2006 Age- & Sex-

specific All-Cause Mortality Rates

Union County Population Estimates

EXPECTED Number of

Union County Deaths

OBSERVED Number of

Union County Deaths

Standardized Mortality Ratio

(Observed/ Expected)

Male Less than 1 year 656.9 21 0.1Male 1-4 years 32.3 104 0.0Male 5-14 years 22.5 279 0.1Male 15-24 years 140.2 328 0.5Male 25-34 years 228.2 184 0.4Male 35-44 years 350.9 247 0.9Male 45-54 years 571.7 400 2.3Male 55-64 years 1035.5 282 2.9Male 65-74 years 2224.3 200 4.4Male 75-84 years 5796.5 125 7.2Male 85+ years 13167.3 41 5.4Female Less than 1 year 505.8 21 0.1Female 1-4 years 23.9 105 0.0Female 5-14 years 16.4 269 0.0Female 15-24 years 59.9 295 0.2Female 25-34 years 89.7 167 0.1Female 35-44 years 161 235 0.4Female 45-54 years 324.6 383 1.2Female 55-64 years 607.6 304 1.8Female 65-74 years 1466.6 242 3.5Female 75-84 years 4184.2 170 7.1Female 85+ years 12976 68 8.8

47.7 61 1.28

SMR (Observed/Expected)

(Rate x Pop) / 100,000

Standardized Mortality Ratio for 2006 Union County, All-cause Mortality

Standardized Mortality Ratio for 2006 Union County, All-cause Mortality


SMR, Union CountySMR, Union County An SMR <1.0 indicates less-than-expected

mortality.

An SMR >1.0 indicates greater-than-expected mortality (also known as “excess mortality).

Union County’s SMR was 1.28, so the county had excess mortality in 2006.

Was it significantly more than expected?


Indirect Age-StandardizationIndirect Age-Standardization You should not use direct age adjustment when

there are fewer than 20 (some say 25) health events. If you multiply the New Mexico crude rate by the Union County SMR, you get the indirectly age-adjusted rate for Union County.– Union Co. crude all-cause death rate: 1364.6– NM crude all-cause death rate: 757.5– Union County SMR: 1.28– Union County indirectly age-standardized rate: 969.6

(still higher than the state rate, but the effects of Union County’s age distribution have been removed).


Confidence Interval for SMRConfidence Interval for SMR Observed deaths: 61 (# deaths from Vital Records data)

Expected deaths: 47.7 (# expected from SMR calculation)

SMR: 1.28 (observed / expected)

StdErr for SMR: 0.16 (SQRT(observed)) / expected

95% Confidence Interval: 0.32 (1.96 x StdErr)

Significance Test Does the 95% confidence interval include 1.0?– If "yes" -> not significant– If "no" -> statistically significant


Summary: Statistical ToolsSummary: Statistical Tools

Use confidence intervals assess the stability of a rate.

Use C.I. to see if your local rate is significantly different from the state rate.

A statistic called a “Test of Proportions” uses the “pooled standard error” to test whether two local rates are significantly different.



Combine data to improve the stability of your rate.– Combine persons (e.g., broader age group)– Combine place (larger area)– Combine time (more years)



Use the Standardized Mortality (Morbidity) Ratio (SMR) to compare a local rate to a standard population (e.g., state overall).

The SMR “expected” can be used for indirect age-adjustment when the number of health events is fewer than 20, or if the age-specific death rates are not known.

1/26/09 59

Thanks!Thanks!

Lois M. Haggard, PhD

Community Health Assessment Program, NMDOH

[email protected]

http://ibis.health.state.nm.us

mailto:[email protected]

http://ibis.health.state.nm.us/

Date post:	28-Dec-2015
Category:	Documents
Upload:	erin-ferguson
View:	219 times
Download:	4 times

1/26/09 1 Community Health Assessment in Small Populations: Tools for Working With “Small...

Documents