Ipek BilgenDavid SterrettMichael J. Stern
Exploring Nonresponse andCoverage in a Web Study
AAPOR 2015
Ipek Bilgen, David Sterrett, and Michael J. Stern
• Multi-mode surveys: Increasingly include a webcomponent (concurrently or subsequently)
• Web-surveying became a viable data collection mode– 86% of all U.S. adults now go online
– Compared to 50% in 2000
– The current national broadband adoption rate = 70% (Pew 2014)
• Rise in web enabled mobile devices– The national smartphone adoption rate = 55% (Pew 2014)
• Though increase in RRs to web surveys has notaccelerated as it was expected.
Background
Source: http://www.pewinternet.org 2
• Coverage: Lack of internet access, sufficientinternet speed/proficiency
• Nonresponse: Lack of time/interest/willingness/trust(despite access)
Why people do not respond to web?
3
• Research gap: Disentangle coverage from non-response inweb surveys
• Studies conducted a decade ago looking into this issueindicate that non-coverage is more concerning than non-response for web surveys (Couper et al. 2007, Lee 2006)
• Significant increase in internet use in the last decade, whichlikely means an increase in web coverage.
– Given the dynamic nature of this issue, there is a possibilitythat web coverage is less of an issue now.
Introduction
4
• Disentangle web-specific non-coverage and non-response.
• Examine reasons for nonresponse to web surveys
• Rationale: Understand the reasons of web nonresponseamong individuals with internet access
• Prior to data collection: Tailor data collection strategies todrive more households to the less-expensive web mode
• Post data collection: Tailor post data collection adjustmentstrategies to obtain less biased estimates
Study Aim
5
• Is non-response becoming more of a concernthan non-coverage?
• Which demographic groups are more likely to be notcovered in internet surveys?
• (Controlling for internet access/use) Whichdemographic groups are more reluctant to participatein internet surveys?
– What are the potential reasons for non-response?
Research Questions
6
Healthy Illinois Study (HIL)
• Questions on behaviors and opinions related tothe searching for and transferring of online orelectronic health records
• Sample Size (Web Survey) = 4,000• Sequential Multi-mode Study
• WEB, SAQ, CATI• Match physical addresses to phone numbers (landlines).
Data & Methods (I)
7
• Sequential Multi-mode: Followed up with non-respondents to the HIL WEB survey
• NR Sample: where a phone number (landline) is matched to aphysical address (n=1795)
• Step 1: Mailed out a self-administrated questionnaire (SAQ)booklet (including $2 incentive)
• Step 2: Administered the questionnaire via telephoneinterviews (CATI) for four weeks approximately three weeksafter the SAQ mailings.
• In both modes, we asked sample members a series of questions:– Their electronic device ownership; Mobile internet access; Whether
respondents remember receiving the web invitation letter or e-mail;Their reasons for not responding if they remember such invitations
Data & Methods (II)
8
MODE Samplesize
# ofresponse
Adjusted response rates(%)
WEB(Initial mode) 4,000 487 12.2%
SAQ(NR follow-up) 1,795 352 19.6%
CATI(NR follow-up) 1,455 168 11.5%
TOTAL 1007 37.7% (AAPOR RR2)
Response Rates (per mode)
9
• Based on three questions:• Number of devices own that could access internet
– Do you personally own…[Desktop computer; Laptop computer; Cell phone;Blackberry or iPhone or other handheld device that is also a cell phone]?
– Coded from 0 (own no devices) to 4.(own all four devices).
• Mobile access– Do you have access to the Internet through a mobile device (such as a
smart phone, personal digital assistant (PDA), iPhone, or Blackberry?”– Coded from 0 (no) to 1 (yes).
• Frequency access Internet– How often do you access the Internet? Daily, Several times a week,
Several times a month, Several times a year or less often, or Never?Coded from 0 (never) to 4 (daily).
• Internet Access/Use Scale (0-9) = number of devices own (0-4) +mobile access (0-1)+ frequency access internet (0-4)
Internet Access/Use Scale
10
Internet Access/Use Scale Distribution
0
20
40
60
80
100
120
140
160
0 1 2 3 4 5 6 7 8 9
Num
ber o
f res
pond
ents
Internet Access/Use score
Web
SAQ and CATI
Non-Coverage (NC):12% Web and 30% ofSAQ/CATI haveinternet access/usescore of 0-5
Non-Response (NR):88% Web and 70% ofSAQ/CATI haveinternet access/usescore of 6 or more
11
0%
20%
40%
60%
80%
100%
18-30 31-40 41-50 51-60 61+
% o
f Res
pond
ents
Respondent Age
Non-Coverage(Score = 0-5)
Non-Response(Score = 6-9)
NR > NC among middle aged respondents
Please note that,* indicates p < 0.05; ** indicates p< 0.001; *** indicates p< 0.0001
NC vs. NR**
12
MODE = SAQ and CATI Follow-up
0%
20%
40%
60%
80%
100%
HS andbelow
Associate's Bachelor's Graduate
% o
f Res
pond
ents
Respondent Education
Non-Coverage(Score = 0-5)
Non-Response(Score = 6-9)
NR > NC among respondents with more than HS education
Please note that,* indicates p < 0.05; ** indicates p< 0.001; *** indicates p< 0.0001 13
NC vs. NR***
MODE = SAQ and CATI Follow-up
0%
20%
40%
60%
80%
100%
% o
f Res
pond
ents
Respondent Income
Non-Coverage(Score = 0-5)
Non-Response(Score = 6-9)
NR > NC among higher income respondents
14
NC vs. NR***
MODE = SAQ and CATI Follow-up
0%
20%
40%
60%
80%
100%
Excellent Good Fair Poor
% o
fRes
pond
ents
Respondent Health
MODE = SAQ and CATI Follow-up
Non-Coverage(Score = 0-5)
Non-Response(Score = 6-9)
NR > NC among healthier respondents (self-report)
15
NC vs. NR***
Asked if they rememberedreceiving an invitation (26%)
Reasons for not responding
0%10%20%30%40%50%60%70%80%90%
100%
Don't haveaccess to
web/internet
Did nothave time
Wanted tocomplete but
forgot
Did not findinvitation
email/letterto be
informative
% re
spon
ded
YE
S SAQCATI
16
• The majority of the SAQ and CATI respondents couldpotentially be reached via web since they report:
• Having daily access to internet (~60%)• Having mobile internet access (~40%)• Personally own a smart phone (~39%)
• Can be potentially reached via less expensive web mode• Demographic break-down of potential Web NRs• Learn where need to put more emphasis, creativity• Tailor our data collection strategies to drive more households to
the web survey– Decrease burden to enter URL into browser– Eliminate procrastination: Early-bird incentives in web– Increase trust (some groups are reluctant to respond via web
due to security reasons)
Key Findings & Discussion
17
• The SAQ and CATI respondents (with internet access) whoremember receiving the mail/invitation letters
• Majority indicated did not have time (~ 36%)• Wanted to complete but forgot (~ 37%)• Did not find invitation email/letter informative (~15%)
• However, only 26% of these respondentsremember receiving the mail/invitation letters forthe web survey (potential issues with junk mailand/or address & phone match)
Key Findings & Discussion
18
• Time lag between initial survey and follow up• May cause recall issues
– High % of missing item NR for the questions related to receiving theweb invitation letter or e-mail
• Potential Social desirability bias (CATI)• Qs related to why respondents did not complete the web survey
– High % of “did not have time” responses in CATI.
• External validity• The study only looks at residents of one state, which could limit
the ability to make generalizations about the population.– Can be tested by comparing the sample demographics to census
figures and by comparing responses to the substantive questions tothe benchmark survey results.
Limitations
19
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Don'thaveemail
DK Yes No DK
EMAIL LETTER
SAQ
CATI
Received web invitation letter or e-mail about the study?
21
0%10%20%30%40%50%60%70%80%90%
100%
Desktop Laptop CellPhone
SmartPhone
% re
spon
ded
YE
S
Do you personally own?
WEB
SAQ
CATI
Electronic device ownership
22
0%10%20%30%40%50%60%70%80%90%
100%
Never Severaltimes a year
(or less)
Severaltimes amonth
Severaltimes aweek
Daily
How often do you access the internet?
WEBSAQCATI
Internet Usage
23
0%10%20%30%40%50%60%70%80%90%
100%
Yes No
Do you have access to theInternet through a mobile device?
WEBSAQCATI
Mobile internet access
24