Random digit dialing cell phone surveys and surveillance systems: data quality, data weighting...

transcript

Random digit dialing cell phone surveys and ill t d t lit d tsurveillance systems: data quality, data weighting strategies, and bias

Cristine D. Delnevo, PhD, MPH & Daniel A. Gundersen, MA,

UMDNJ‐ School of Public Health

Randal S ZuWallack MS & Frederica R Conrey PhDRandal S. ZuWallack, MS & Frederica R. Conrey, PhD

ICF Macro International

P t d t 137th A l M ti & E itiPresented at 137th Annual Meeting & ExpositionPhiladelphia, PA

November 7‐11, 2009

Work supported in part by the National Cancer Institute (R21CA129474 ) and a contract from the New Jersey Department of Health and Senior Services, through funding from the Cigarette Tax

Wireless substitution (US), 2005–2008Wireless substitution (US), 2005 2008

Source: Blumberg & Luke Wireless substitution: Early release of estimates from the National Health Interview Survey, July‐December 2008. National Center for Health Statistics. May 2009.

Demographic differencesDemographic differences• Gender: men more likely than

b i l lWireless substitution by age (and time)

women to be wireless only

• SES: Adults living in (30.9%) and near poverty (23.8%) more likely h hi h i d l

Wireless substitution by age (and time)

than higher income adults (16.0%) to be wireless only

• Region: Wireless substitution hi h t i S th (21 3%) dhighest in South (21.3%) and Midwest (20.8%) vs. Northeast (11.4%) or West (17.2 %)

R /Eth i it Wi l• Race/Ethnicity: Wireless substitution highest among black (21.4%) and Hispanic (25 0%) adults vs Non Hispanic(25.0%) adults vs. Non‐Hispanic white adults (16.6%)

Biased health estimates?Biased health estimates?

• Potential for biased health estimates due to samplePotential for biased health estimates due to sample under‐coverage remains a real, growing threat to RDD health surveys

• Cell‐phone only also differs with respect to health behaviors and the validity of some health estimates based on traditional RDD surveys are increasingly questionable

Health estimates by phone statusHealth estimates by phone status

Has a landline Wireless-only No telephonetelephone

NHIS July – December 2007

5+ alcoholic drinks in 1 day 17.7 37.3 27.1Current smoker 18.0 30.6 38.6Uninsured 13 7 28 7 44 1Uninsured 13.7 28.7 44.1Has a usual place for care 87.5 68.0 61.8

Flu vaccination 32.7 16.6 20.9

Ever tested for HIV 34.7 47.6 45.8

Source: Blumberg SJ, Luke JV. Coverage bias in traditional telephone surveys of low‐income and young adults. Public Opin Q. 2007;71:734–749

Cigarette smoking among young adults, 2003 2005 NHIS & BRFSS2003‐2005, NHIS & BRFSS

Source: Delnevo, Gundersen & Hagman (2008) Declining prevalence of alcohol and smoking estimates among young adults nationally: artifacts of sample under‐coverage? Am J of Epidemiology

New challenges: Wireless Mostly?

Telephone Status, NHIS July‐December 2008

• The percentage of adults living in wireless‐mostly households has been increasing

• Who are they?• Who are they? (demographic & health behaviors?)

• Will they respond to landline surveys?

Reactions?Reactions?

• Starting in 2009, the Center f Di C l dfor Disease Control and Prevention (CDC) is requiring states to incorporate cell phoneincorporate cell phone interviews in their regular BRFSS sample

• Yet there is no widely yaccepted methods of evaluating data quality or data weighting, particularly for state and local areafor state and local area surveys.

• AAPOR Cell Phone Task Force reportForce report

This special session:This special session:

• Analysis of data quality of cell phone• Analysis of data quality of cell phone surveys

• Demonstrate weighting procedures for merging cell phone samples with landline samples

• An assessment of bias in landline only• An assessment of bias in landline only surveys, and

Assessing Data Quality ofAssessing Data Quality of Cell Phone Random Digit g

Dial Surveys

Frederica R ConreyRandy Zuwallack

Data Quality

Q lit f R• Quality of Responses• Quality of Sample

2008 New Jersey Adult Tobacco Survey

R d Di it Di l L d (RR 19%) d• Random Digit Dial Land (RR=19%) and Cell Phone (RR=16%)

• Short version– 49 Questions (min=37; max=70)– 534 Cell completes– 468 Landline completes

Survival Analysis

Diff t l t diff t• Different people get different surveys because of skip patterns

• Survival analysis– Measures the impact of survey mode on non-

response– Controls for differences in survey length

Quality of Responses

Item Non-Response

M M di Std

Failure Rates by Phone ModeMean Median Std

Cell 3.3% 2.2% 7.4%

Landline 3.4% 2.0% 5.5%

Survival predicted by phone mode:p y pHazard=1.00, p>.95

Open Ended Response

O d t f thTotal Responses per Open

Open end reports of the events of 2 recalled commercials

Responses / OE DK / OE

p p pEnd by Phone Mode

commercials– 251 respondents were

asked at least one

/ OE DK / OE

Cell .52 .49

Landline .48 .55

open ended question P .56 .33

Quality of Sample

Unit Non-Response

D t lit i th t d ifData quality is threatened if– Response rates are low AND– The people who DO NOT respond are

different from those who DO.If cell phone respondents are less likely to

respond, then there is non-response bias.

Survival Analysis

C ll d l dli d t t• Cell and landline respondents may get different surveys

• Response rates alone don’t tell the story• Survival analysis tells whether cell y

respondents are more likely to break off given the same survey lengthg y g

Survival Model

Landline

0%0 20 40 60 80

Survey Questions 001Survey Questions p<.001

What does a difference in survey survival mean?

C ll d t it th l dli• Cell respondents quit sooner than landline respondents.

• The sample under-represents cell phones• The longer the survey, the worse the g y

nonresponse bias• The solution?The solution?

– Careful weighting– Short surveys– Short surveys

In a population study of tobacco use

Mi i l diff S b t ti l diff

behavior…

Minimal difference between cell and landline in response

Substantial difference between cell and landline in responselandline in response

quality• No difference emerged in

landline in response rate

• Sample quality may beNo difference emerged in item-nonresponse

• No difference emerged in i h f d

Sample quality may be threatened if cell phone surveys are too long or weighted incorrectlyrichness of open end

responses.weighted incorrectly

Weighting Cell Phone SurveysWeighting Cell Phone Surveys

Randal Zuwallack

Frederica R ConreyFrederica R Conrey

Thanks

• Cris DelnevoCris Delnevo

• Dan Gundersen

C ( 2 C 29 ) f• NCI (R21CA129474), New Jersey Dept of Health and Senior Services

Dual Frame

A Adults in landline households with no cell phone

A. Adults in landline households with no cell phone,

B. Adults in landline households with a cell phone, and

C. Adults in non‐landline households with a cell phone (cell only).

Dual Frame

Common designs:

Dual frame w/ no overlap: Landline (A+B) + Cell (C)

Dual frame w/ overlap: Landline (A+B) + Cell (B+C)

Uncommon design:

Dual frame w/ no overlap: Landline (A) + Cell (B+C)Dual frame w/ no overlap: Landline (A) + Cell (B+C)

Weighting Challenges

• Challenge 1: How do we put the dual framesChallenge 1: How do we put the dual frames together?

• Challenge 2: Differential Nonresponse

Challenge 1

• How do we put the dual frames together?How do we put the dual frames together?• No overlap

– Estimate of cell‐only population size?Estimate of cell only population size?• Internal estimate• External estimate: NHIS (Blumberg et al.)

• With overlap– Must determine group membership– Adjust for multiple selection probabilities– Estimate of phone group population sizes?

Cell Survey

• “In addition to your cell phone is there atIn addition to your cell phone, is there at least one telephone inside your home that is currently working and is not a cell phone? Docurrently working and is not a cell phone? Do not include telephones only used for business or telephones only used for computers or faxor telephones only used for computers or fax machines.” – ‘yes’ = dual user while those who responded– yes = dual user, while those who responded

– ‘no’ = cell‐only

Landline Survey

• “In addition to your residential landlineIn addition to your residential landline telephone, do you also use one or more cell phone numbers?”phone numbers? – ‘yes’ = dual user

‘no’ = landline only– no = landline only.

Example 1‐‐Colorado

• Combine with BRFSSCombine with BRFSS

• Group membership K f ll– Known for cell

– Unknown for landline

• Limited to dual frame w/ no overlap– Used 15% (NHIS state estimates) for merging landline and cell

– Poststratified dual sample to age and sex.

Example 2

• You are midway through a landline survey andYou are midway through a landline survey and want to add cell phones. You don’t know who has a cell phone and who doesn’t What arehas a cell phone and who doesn t. What are your options?1) Add cell only1) Add cell only

2) Add cell and dual‐users

Challenge 2

• Differential NonresponseDifferential Nonresponse

• Cell‐only overrepresented when conducting cell phone surveyscell phone surveys. – Contact rate

C i– Cooperation rate

• Those who rely more on their cell phone will be easier to reach.

Telephone Reliance

Landline Frame

PLandline householdsne

Landline sample

Cell Phone Frame

PCell Phone Usersne

Cell sampleCell sample

Dual Frames

Landline sample Landline sample

Cell sample Cell sample

Dual Frame sample Dual Frame sample

Ideal Realistic

Our Goal

R b lRebalance on cell reliancereliance

Rebalance on landline reliance

Measuring Telephone Reliance

• Cell only landline onlyCell only, landline only

• Classify Dual users“Of ll th t l h ll th t i ”– “Of all the telephone calls that you receive, are…”

• All or almost all calls received on a cell phone? (cell‐mostly)mostly)

• Some received on a cell phone and some on a regular landline phone? (true‐dual)

• Very few or none received on a cell phone? (landline‐mostly)

Telephone Reliance

LandlineMostly(1)

TrueDual(2)

CellMostly

LandlineOnly(0)

CellOnly

(1) (2) (3)(0) (4)

Response Propensity

• Adjust for differential nonresponse byAdjust for differential nonresponse by benchmarking against NHIS

• Logistic regression model• Logistic regression model– Dependent: Survey type

1 b ll i ti l ll h• 1 = observe cell user in national cell phone survey

• 0 = observe cell user in NHIS

Independent: Cell phone reliance (1 4) age race– Independent: Cell phone reliance (1‐4), age, race

Data sources

National cell sample NHISNational cell samplen=500

Landline mostly 8% 23%

True dual 27% 42%

Cell mostly 23% 17%

Cell only 42% 18%

Cell users 100% 100%

Applying the model

• Applied to same data—poststatificationApplied to same data poststatification

• Applied to independent dataA ti ti l ll l th– Assumption: national cell sample measures the odds ratio for observing a cell‐only respondent in a cell sample relative to a dual‐usera cell sample relative to a dual user.

S l l• State, local surveys

Applying the model

ColoradoColoradoCell Sample

w/o NR adj w/ NR adj

Landline mostly (1) 9% 26%

True dual (2) 26% 37%

C ll tl (3) 19% 15%Cell mostly (3) 19% 15%

Cell only (4) 46% 21%

Total cell sample 100% 100%

Applying the model

• Assume landline only is 20% (we don’t know)Assume landline only is 20% (we don t know)

Landline only 20%

Landline mostly 21%

True dual 30%

Cell mostly 12%NHIS

Cell only 17%

Total population 100%

NHIS state

estimate = 15%p p

City Sample

LandlineSample

Cell Sample Combined Samples

w/o NR w/ NR w/o NR w/ NRw/o NR adj

w/ NR adj

w/o NR adj

w/ NR adj

Cell‐only ‐ 43.5 18.9 35.5 13.4

Cell‐mostly 12.4 25.1 19.8 13.8 12.2

True Dual 30.1 23.6 35.5 18.7 25.1

Landline‐mostly 19.1 7.9 25.8 9.9 17.2Landline mostly 19.1 7.9 25.8 9.9 17.2

Landline‐only 38.4 ‐ ‐ 23.9 32.1

Conclusions

• Dual‐frameDual frame– There are ways to combine the data, even when we don’t have a full picture of group membershipwe don t have a full picture of group membership.

• Differential nonresponseResponse propensity model rebalances the cell– Response propensity model rebalances the cell sample based on cell reliance.

Can be applied at state and local levels when no– Can be applied at state and local levels when no benchmarks exist.

– Next steps: explore a response propensity modelNext steps: explore a response propensity model for landline.

Thank you

Randal Zuwallack@macrointernational comRandal.Zuwallack@macrointernational.com

Frederica.Conrey@macrointernational.com

Examining the bias in landline only g ysurveys: How does the cell phone only population differ from the landlinepopulation differ from the landline

population on health indicators, and are estimates from landline surveys biased?

Daniel A. Gundersen, MA, UMDNJ‐SPH

Cristine D. Delnevo, PhD, MPH, UMDNJ‐SPH

Randy S. ZuWallack, MS, ICF Macro

Cell Phone Substitution and RDD surveysCell Phone Substitution and RDD surveys

• RDD surveys (e.g. BRFSS) have traditionally only sampled household telephones (i.e. landlines)

• Up until early 2000s, rate of cell phone only households was small

• From mid 2000s, rate of substitution has grown substantially– 6 7% of adults in 2005 to 18 4% in 2008 nationally16.7% of adults in 2005 to 18.4% in 2008 nationally

• Higher among certain demographic groups1– Young adults– Hispanics and Blacks– Poor and near poor

What is bias due to coverage error in the li f ?sampling frame?

• Non‐covered population is different fromNon covered population is different from covered population on some variable of interestinterest

– If proportion of non‐covered ( ) is small, bias will be small

– If difference between the covered and

noncovered is small, bias will be small,

Previous ResearchPrevious Research• Data from Jan 2004‐June 2005 NHIS found2

– Greater than 1 percentage point bias in binge drinking, smoking prevalence, usual place for medical care, receiving influenza vaccine

• Data from 2007 NHIS found3Data from 2007 NHIS found– Bias increased slightly for past year binge drinking and receiving influenza vaccineThese biases were larger among young adults and low income– These biases were larger among young adults and low income persons

• Data from 2001‐2005 BRFSS on 18‐24 year olds found4– Prevalence of binge drinking, heavy drinking, and cigarette smoking declined during 2003‐2005; coincided with large increase in wireless substitution among young adults

– NHIS and NSDUH did not observe similar declines during this period

Our Study:Our Study:

• Objective:Objective: – Assess the presence of bias in landline RDD due to exclusion of cell phone only on select healthexclusion of cell phone only on select health indicators

• Data Source and Instrument:Data Source and Instrument:– Cell phone RDD of adults in Colorado (n=501)

• May to September 2008May to September 2008

• Instrument was shortened version of BRFSS

– BRFSS from same data collection period (n=4,527)BRFSS from same data collection period (n 4,527)

MethodologyMethodology

• Cell Phone sample:Ce o e sa p e:– Design weights account for probability of selection

• BRFSS– Standard BRFSS design weight accounts for strata, number of landlines and adults in the household

f d b ( )* *– Postratified by age(7)*sex*race

• Merged dataD i i ht l d t t h f– Design weights scaled to represent share of population by phone status

– Postratified by age(7)*sex*racey g ( )

Statistical AnalysesStatistical Analyses• Comparisons of BRFSS landline and Cell Only based on design

weights• Comparison of BRFSS landline and merged data are

postratified to demographic makeup of CO

• We assume the merged data to be unbiased (i.e. no coverage error due to cell only exclusion)error due to cell only exclusion)

• All analyses conducted in STATA v.10.1 to account for complex sampling design

Table. CO BRFSS vs. CO Cell Only, May‐September 2008 (n=5,028)y, y p ( , )

BRFSS landline Cell only Differencey

Health Indicator % (95%CI) % (95%CI) % (95%CI)

* ( ) ( ) ( )Smoking* 15.29 (±1.14) 28.14 (±4.46) ‐12.85 (±4.60)

Ever had HIV test* 36.64 (±1.85) 52.51 (±5.09) ‐15.87 (±5.41)

Having health insurance* 88.36 (±1.11) 72.46 (±4.38) 15.9 (±4.51)

Having primary care provider* 85.91 (±1.16) 60.39 (±4.85) 25.52 (±5.00)

Not affording care due to cost* 12 27 (±1 08) 20 42 (±3 94) ‐8 15 (±4 08)Not affording care due to cost 12.27 (±1.08) 20.42 (±3.94) ‐8.15 (±4.08)

*p<.05; data weighted to correct for sampling design

Figure 1. Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008

‐2.5

‐7.5

Figure 2. Relative Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Only RDD (n=501) May‐September 2008

‐20%

‐10%

‐40%

‐30%

‐50%

Figure 3. Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008

‐2.5

‐7.5

Figure 4. Relative Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008

‐20%

‐10%

‐40%

‐30%

‐50%

Figure 5. Bias ‐ Having health insurance, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008

‐2.5

‐7 5

‐7.5

Figure 6. Relative Bias Having Health Insurance, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008

‐20%

‐10%

‐40%

‐30%

‐50%

Figure 7. Bias Has primary care provider, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008RDD (n=501) May‐September 2008

‐2.5

‐7.5

Figure 8. Relative Bias Has Primary Care Provider, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008Cell Only RDD (n=501) May‐September 2008

‐20%

‐10%

‐40%

‐30%

‐50%

Figure 9. Bias ‐ could not afford health care due to cost, CO BRFSS (n=4,527) and CO Cell Only RDD (n 501) May September 2008and CO Cell Only RDD (n=501) May‐September 2008

‐2.5

‐7.5

Figure 10. Relative Bias ‐ could not afford health care due to cost, CO BRFSS (n 4 527) and CO Cell Only RDD (n 501) May September 2008BRFSS (n=4,527) and CO Cell Only RDD (n=501) May‐September 2008

‐20%

‐10%

‐40%

‐30%

‐50%

Summary of FindingsSummary of Findings

• Bias is present not only among those with high wireless substitution rates

• Smoking prevalence underestimated among those with higher wireless substitution rateshigher wireless substitution rates

• Ever had an HIV test substantially underestimated among all groups

R l ti bi l th ith hi h i l– Relative bias large among those with high wireless substitution rates (young adults, non‐whites, low SES)

• Bias for health care insurance and having primary care id i d i d hi bprovider is underestimated among non‐whites, but

overestimated among other groups– Relative bias is small

Implications for study design and lanalysis

• When possible, include an RDD of cell phone only population (BRFSS now does this)– If you can’t, be aware of the potential for bias and interpret

findings accordingly• If you’re analyzing landline RDD data from past years

– Interpret findings with potential bias in mind– Historical trend may observe artificial changes due to coverage

error– Wireless substitution rates differ by geographic region so

problem may be less in certain areasA bi d b h hi i ll i• A bias present today may not be the same historically or in the future– Characteristics of the early adopters may not be the same as the

current cell only population today or laggardscurrent cell only population today or laggards

LimitationsLimitations

• Unable to assess bias in some subpopulationsUnable to assess bias in some subpopulations due to small sample size

• Study does not account for cell phone mostly• Study does not account for cell phone mostly population

ReferencesReferences

1. Blumberg SJ & Julian V. Luke. (2009). Wireless Substitution: Early l f ti t f th N ti l H lth I t i Srelease of estimates from the National Health Interview Survey,

July‐December 2008.2. Blumberg SJ, Luke JV & Marcie L. Cynamon. (2006). Telephone

coverage and health survey estimates: evaluating concern aboutcoverage and health survey estimates: evaluating concern about wireless substitution. American Journal of Public Health. 96(5): 926‐931.

3. Blumberg SJ & Julian V. Luke. (2009). Reevaluating the need for concern regarding noncoverage bias in landline surveys. American Journal of Public Health. 99(10): 1806‐1810.

4. Delnevo CD, Gundersen DA & Brett T. Hagman. (2008). Declining estimated prevalence of alcohol drinking and smoking amongestimated prevalence of alcohol drinking and smoking among young adults nationally: artifacts of sample undercoverage? American Journal of Epidemiology. 167(1): 15‐19.

Contact InfoContact Info

Cristine Delnevo PhD MPH delnevo@umdnj eduCristine Delnevo, PhD, MPH delnevo@umdnj.eduDaniel A. Gundersen, MA gunderda@umdnj.edu

Randal ZuWallack Randal.Zuwallack@macrointernational.comRiki Conrey Frederica.Conrey@macrointernational.com

Random digit dialing cell phone surveys and surveillance systems: data quality, data weighting...

Technology