Investigating Internet Opt-in Panels for Behavioral ... · Investigating Internet Opt -in Panels...

Invest igat ing Internet Opt-in Panels for Behavioral Surveillance

Carol A. Gotway Crawford, PhD Chief, Population Health Surveillance Branch

Federal Committee on Statistical Methodology Research Conference November 4, 2013

Washington, DC

National Center for Chronic Disease Prevention and Health Promotion Division of Population Health

Catherine Okoro, PhD Epidemiologist

Satvinder Dhingra, MPH Epidemiologist

Northrop Grumman

What Are Internet Opt-in Panels?

Potent ial panelists are recruited via the Internet Banner ads, email lists, promotions, and offers Double opt-in process to become a panel member

Panelists become the pool for sample select ion

Panel may or may not be representat ive of the

populat ion Coverage is limited to Internet users (~ 80% of the population) Respondent selection and motivation

Why Use Internet Opt-in Panels?

Lower cost than probability-based sampling

Shorter collect ion and prep t ime for data release to the public than current methods (RDD, face-to-face)

Expands the surveillance and study tool-kit

Permits longitudinal and in-depth follow-up studies

Increases administrat ive and design flexibility and efficiency

Pilot Study

4 States Cooperative agreements in GA, IL, NY, and TX

3 Vendors Different sampling methodologies Cooperating and collaborating

De-duplication of respondents Nearly identical questionnaire format

3 Levels of Geography National State Metropolitan Statistical Area

Pilot Object ives

Compare sampling methodologies Sample matching, source blending, and quota

Assess feasibility and accuracy for public health

Compare est imates with those from other surveys

Evaluate across a range of parameters: Cost, geographic granularity, and timeliness

Sampling Methodologies

Sample Matching Different modes of recruitment are used to ensure representativeness for

hard-to-reach populations Potential respondents are selected by matching to a random sample from

the American Community Survey Final responses are weighted to known characteristics in the U.S. using

propensity score weighting

Sample Blending Uses population segments designed to reflect behavioral differences but

based on Census data Apply the segmentation structure locally to balance, weight, and blend

sample

Quota Sampling A non-probability sample in which respondents take the survey on a first-

come, first-served basis according to a fixed quota

Quest ionnaire Development

Survey consists of ~80 questions (20 minutes) Questions drawn from:

• CDC: BRFSS, NHANES, & NHIS • NIH: PROMIS • SAMHSA: NSDUH • ONC: Consumer Survey of Attitudes Toward the Privacy and

Security Aspects of EHR and HIE • NPWF (National Partnership for Women and Families) • NSF supported Cooperative Congressional Election Study

Benchmarking

National: Demographics (Unweighted)

NHIS 2012 HH CAPI

BRFSS 2012 DF-RDD CATI

YouGov 2013 IPS Matched

CPS 2012

0.0 20.0 40.0 60.0 80.0

White nH

Black nH

Hispanic

Other

0.0 10.0 20.0 30.0 40.0 50.0 60.0

Male

Female

0.0 10.0 20.0 30.0 40.0

18 -29

30 - 44

45 - 64

65+

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0

<HS

=HS

>HS

National: Demographics (Weighted)

NHIS 2012 HH CAPI



0.0 10.0 20.0 30.0 40.0

18 -29

30 - 44

45 - 64

65+

0.0 10.0 20.0 30.0 40.0 50.0 60.0

Male

Female

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0

<HS

=HS

>HS

0.0 20.0 40.0 60.0

White nH

Black nH

Hispanic

Other

State: Age

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

GA

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

IL

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

NY

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

TX

Mktg Inc. 2013 Blended

NHIS 2011 HH CAPI


uSamp 2013 IPS Quota


MSA: Age




SMART BRFSS 2012 DF-RDD CATI

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Houston

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

New York

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Chicago

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Atlanta

State: Race/Ethnicity

0

20

40

60

80

White nH Black nH Hispanic Other

TX

0

20

40

60

80


NY

0

20

40

60

80


IL


NHIS 2011 HH CAPI




0

20

40

60

80


GA

MSA: Race/Ethnicity





010203040506070


Atlanta

010203040506070


Chicago

010203040506070


New York

010203040506070


Houston

State: Education

010203040506070

<HS =HS >HS

TX

010203040506070

<HS =HS >HS

NY

010203040506070

<HS =HS >HS

IL

010203040506070

<HS =HS >HS

GA


NHIS 2011 HH CAPI




MSA: Education





010203040506070

<HS =HS >HS

Houston

010203040506070

<HS =HS >HS

New York

010203040506070

<HS =HS >HS

Chicago

010203040506070

<HS =HS >HS

Atlanta

National: Outcomes

NHIS 2012 HH CAPI



0.0 10.0 20.0 30.0 40.0

Obesity (BMI ≥30)

NHANES (MEC) NHANES (SR) NHIS 2012

BRFSS 2012 YouGov 2013

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Diabetes

NHANES (MEC) NHANES (SR) NHIS 2012

BRFSS 2012 YouGov 2013

0.0 20.0 40.0 60.0 80.0 100.0

Ex/VG/GD

Fair/Poor

NHIS 2012 BRFSS 2012 YouGov 2013

State: Obesity (BMI ≥30)

0

10

20

30

40

50

Yes

TX

0

10

20

30

40

50

Yes

NY

0

10

20

30

40

50

Yes

IL

0

10

20

30

40

50

Yes

GA


NHIS 2011 HH CAPI




MSA: Obesity (BMI ≥30)





0

10

20

30

40

50

Atlanta

0

10

20

30

40

50

Chicago

0

10

20

30

40

50

New York

0

10

20

30

40

50

Houston

State: Diabetes

0

5

10

15

Yes

TX

0

5

10

15

Yes

NY

0

5

10

15

Yes

IL

0

5

10

15

Yes

GA


NHIS 2011 HH CAPI




MSA: Diabetes





-2

3

8

13

18

Houston

-2

3

8

13

18

New York

-2

3

8

13

18

Chicago

-2

3

8

13

18

Atlanta

State: Disability

0

10

20

30

40

Activity or Eqpt

TX

0

10

20

30

40

Activity or Eqpt

NY

0

10

20

30

40

Activity or Eqpt

IL

0

10

20

30

40

Activity or Eqpt

GA


NHIS 2011 HH CAPI

BRFSS 2013 Preliminary DF-RDD CATI



MSA: Disability

05

101520253035

Activity or Eqpt

Houston

05

101520253035

Activity or Eqpt

New York

05

101520253035

Activity or Eqpt

Chicago

05

101520253035

Activity or Eqpt

Atlanta





National: Health Care Access, Utilization, Behaviors & Outcomes

NHIS 2012 HH CAPI



0.0 20.0 40.0 60.0 80.0

No Pap

No Mam

No Colorectal

No HIV Test

No Flu

0.0 10.0 20.0 30.0 40.0 50.0

No Insurance

No Usual Source

Cost Barrier

No Past year Checkup

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Current Smoker

Former Smoker

Heavy Drinker

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0

Diabetes

Hypertension

Any Cancer

Asthma

Arthritis

CHD

State: Health Insurance

05

101520253035

No

TX

05

101520253035

No

NY

05

101520253035

No

IL

05

101520253035

No

GA


NHIS 2011 HH CAPI




State: Primary Care Provider

05

101520253035

No

TX

05

101520253035

No

NY

05

101520253035

No

IL

05

101520253035

No

GA


NHIS 2011 HH CAPI




State: Cost Barrier

0

10

20

30

40

Yes

IL

0

10

20

30

40

Yes

GA


NHIS 2011 HH CAPI




0

10

20

30

40

Yes

TX

0

10

20

30

40

Yes

NY

State: Current Smoker

0

10

20

30

40

100 + Ev/Some

TX

0

10

20

30

40

100 + Ev/Some

NY

0

10

20

30

40

100 + Ev/Some

IL

0

10

20

30

40

100 + Ev/Some

GA


NHIS 2011 HH CAPI




State: Heavy Drinker

0

5

10

15

20

Yes

TX

0

5

10

15

20

Yes

NY

0

5

10

15

20

Yes

IL

0

5

10

15

20

Yes

GA


NHIS 2011 HH CAPI




Quantifying Uncertainty

The use of Frequentist confidence intervals with data from a non-probability sample is theoretically inappropriate

Bayesian credible intervals are a more appropriate way to quantify uncertainty when analyzing data from a non-probability sample

In our pilot studies, however, both methods yielded highly similar, if not ident ical, results

Uncertainty Comparison

Variable Confidence

Interval Credible Interval

Obesity 29.22 32.61 29.12 32.56

Diabetes 9.88 11.95 9.85 11.94

High BP 26.64 29.62 26.56 29.58

Major Benefits

• Time (samples constructed to be representative): < 15 days for a national survey ~ 4,000 interviews ~ 30 days for most states ~3,000 interviews ~ 30 days for large (5+ million) MSAs ~2,000 interviews

• Cost: – Internet opt-in panels: $5-$15 per completed interview

• Costs include editing and weighting

– Dual-frame RDD State direct costs average ~$70/CI • Considerable additional costs for editing and weighting

Preliminary Results • Great deal of similarity

– Results of sample matching comparable with BRFSS and NHIS – Variation among surveys consistent across states – Internet opt-in panels fairly accurate at lower levels of geography – Quota sampling not as accurate

• Differences can be attributed to: – Coverage effects (sample selection*outcome interaction) – Use of different control totals and weighting methods – Mode effects (face-to-face, telephone, Internet) – Question differences and order effects – Temporal changes (2013 vs. 2011) – Sample size differences – Cross-sectional differences

Acknowledgements

Stephen Ansolabehere, Harvard University & CCES Steven Gittelman, Mktg. Inc. Douglas Rivers, Stanford University & YouGov Meena Khare, NCHS Georgia, Illinois, New York, and Texas Departments of Health, BRFSS

Rana Bayakly, Madhavi Vajani, Francis Annor, GA Bruce Steiner, IL Bethany Hawke, NY Rebecca Wood, TX

Haci Akcin, CDC Derek Ford, CDC Guixiang Zhao, CDC Soyoun Park, CDC

Date post:	11-Jul-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Investigating Internet Opt-in Panels for Behavioral ... · Investigating Internet Opt -in Panels...

Documents