Date post: | 10-Dec-2015 |
Category: |
Documents |
Upload: | nicholas-welch |
View: | 216 times |
Download: | 1 times |
ESRC Research Methods Festival, Oxford, 17-20 July 2006
The Hunt for the Last Respondent
How to decrease response rates and increase bias
Ineke Stoop
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Enhance responserates
Minimize nonrespons
e bias
Enhance nonrespons
e rates
Maximize nonrespons
e bias
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Design study
• Field experiences How to decrease response rates?
• Keeping the customer satisfied How to spruce up response rates?
• The sophisticated question How to maximize nonresponse bias?
• Sources International nonresponse literature European Social Survey Dutch experimental study
• Mainly face-to-face surveys
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Preparation
• Short fieldwork period Timely results
• Fieldwork contracting Don’t specify: Survey organization knows best
• Stick to national and organisational traditions
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Interviewers
• Foot soldiers of survey research• Economize on interviewer’ training and
instruction Experienced interviewers know best how to act Turnover rate is high and CAPI questionnaires
provide instructions anyway
• Economize on interviewer remuneration Not a real job
• No close monitoring fieldwork Interviewers prefer organizing their own schedules Too much rules and control will lessen motivation
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Information and incentives
• No advance letters and brochures Nobody reads them Advance letters just scare off target respondents
• No toll-free telephone number In order to prevent respondent’ refusals
• No incentive required A survey is expensive enough as it is
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Contacting sample persons
• Limit the number of contact attempts• Face-to-face calls during office hours only
Interviewers have a family life too Evening visits too dangerous
• Don’t send interviewers to problematic inner city areas
Dangerous and low response rate anyway
• Follow-up on appointments only when you need more respondents
• Stop contact attempts when you have enough respondents
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Contactability ESS R1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
PL IL IT LU AT SI GR BE FI HU IE DE ES GB NL CH PT
weekday morning/afternoon evening weekend
ESRC Research Methods Festival, Oxford, 17-20 July 2006
At home, interviewer calls, contact rate
0
10
20
30
40
50
60
70
80
90
100
6 12 18 24
morning afternoon evening
0
10
20
30
40
50
60
70
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Obtaining cooperation• Use a questionnaire efficiently and include as
many questions as you can (better safe than sorry)
Matrix questions Small print (seems shorter)
• Do not spend too much time and effort on difficult respondents
Elderly people, language problems, rarely at home
• No bargaining about timing interviews• No refusal conversion (a refusal is a refusal)• Stick to a single interview mode
ESRC Research Methods Festival, Oxford, 17-20 July 2006
ESS: share of re-approached refusals that were converted
•
Greece
Poland
Finland
HU
Switzerland
Israel
Slovenia
I
Belgium
Austria
Netherlands
Spain
Great Britain
LUX
Italy
0 10 20 30 40 50 60 70 80 90 100
GR
GB
CH
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Bad practices
What happens if you do not take
the previous recommendations seriously
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Response rates ESS R1 and R2
0
10
20
30
40
50
60
70
80
GR PL PT FI SI NO HU SE NL DK AT BE IE ES GB DE LU CH
Round 1 Round 2
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Dutch experiment
• F2F survey + long drop-offs every household member
67% response rate
• Follow-up survey lite among sample of persistent refusals
+70% responding refusals
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Why did refusers cooperate?
• High quality interviewers (and telling them they are the best), motivation, remuneration
• Extensive briefing• Trust (money for incentives) and support
(newsletter) interviewers• Perceived importance (newspaper article,
involvement management)• Wide range of incentives• F2F, PAPI, CATI and internet• Commitment sponsor and fieldwork
organisation
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Compare ‘easy’, hard to get, initial refusals and final refusals
• Differences small• Initial refusals more mr. and mrs Average than
easy respondents• Hard to reach respondents different• Final refusals: less participation, less religious,
fewer PCs, more popular culture
ESRC Research Methods Festival, Oxford, 17-20 July 2006
What if you can’t get away with low response rates
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Focus on promising groups
• Rural areas• Families with small children• No apartment dwellers• People who can be reached on the phone
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Better: redefine the denominator
• Lower upper age limit (older people take too long)• Raise lower age limit (parental permission or never at
home)• Exclude non-native speakers• Exclude mobile phone only• Exclude ex-directory telephone numbers• Exclude non-internet users• Exclude unused sample units from the net sample• Include only those who have agreed on the telephone to
participate in a f2f survey• Members of access panels (pre-recruirment panels)• Only those panel members that (almost always)
cooperate
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Select conscientious access panel members: 70% or 80% guaranteed
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
Response percentiles access panel
Res
pons
e ra
te
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Still better
• Make nonresponse impossible Substitute nonrespondents by family members or
neighbours Quota sampling Volunteer panels
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Added benefit
maximize bias
if topic of the survey is related to response mechanism or
selection mechanism
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Different types of success
• Do a poor job Low response rates
• Focus on easy respondents and exclude different groups
Adequate response rates, large nonresponse bias
• Pre-select willing panel members Vey high response rates, low nonresponse bias, poor
survey quality
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Are high response rates better?
• Maximum bias smaller• Difficult respondents similar to final
nonrespondents?• Hard to reach respondents similar to
noncontacts?• Converted refusals similar to final refusals?
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Assumptions valid?
• High response surveys not necessarily better than low response surveys
Possibly because efforts are directed at ‘easy’ nonrespondents (situational refusals)
And survey-related or topic related refusal is left alone
• Low response rate especially harmful if nonparticipation is caused by single factor related to survey
Political interest (Groves, Presser, Dipko, 2004)
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Increased fieldwork efforts
• Groves (1989)Devoting limited resources to increasing response rates with little or no impact on survey error is not money well spent
• Krosnick (1999)Prevailing wisdom that higher response rates are necessary for sample representiveness is being challenged
• Curtin et al. (2000)Large differences in response rates have only minor effects
• Keeter et al. (2000)Similar results Standard and Rigorous survey, significant differences between demographics
• Merkle and Edelman (2002)Response rates in exit pools not related to error
• Teitler et al. (2003)Diminishing returns at higher effort levels
• Loosveldt et al. (2003)ESS: converted refusals do differ in some countries (with high refusal conversion rate)
ESRC Research Methods Festival, Oxford, 17-20 July 2006
What to do?
• If obtaining high response rates is difficult• If high response rates not necessarily imply
high quality• If interviewing difficult respondents does not
always give better insight in final nonrespondents
• If enhancing response rates means getting more of the same
• If you want to minimize bias
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Aim for high response rates
• Close monitoring• High contact rate in all areas among all groups• Obtain cooperation through multiple means
Respect respondents Provide information Internal and external incentives Motivated interviewers Organisational support Mixed mode interviewing
ESRC Research Methods Festival, Oxford, 17-20 July 2006
But maybe do not bother about the final few %
• Collect and code Sample information Observational data Fieldwork data (contact and cooperation) Information on respondent and nonrespondents
related to the topic of the survey Other sources Central Question procedure Survey among nonrespondents
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Go for bias
• Surveys are very expensive: Not enhance response rates indiscriminately Not high response rates combined with non-
probability sampling and high coverage errors Not high response rates within specific groups Collect and use auxiliary information to assess bias
and to adjust for bias
ESRC Research Methods Festival, Oxford, 17-20 July 2006
Treat data collection
with a scientific, controlled approach
as used in sampling and data analysis