Post on 04-Jan-2016
description
transcript
Data Collection Mode Effects Controlling for Sample Origins in a
Panel Survey: Telephone and Internet
Presented at the 2005 Annual Meeting fo the American Association of Public Opinion
Research
Mike Dennis, Knowledge Networks
Cindy Chatt, University of Nebraska-Lincoln
Alicia Motta-Stanko, Knowledge Networks
Paul Pulliam, RTI International
2
Acknowledgements
The survey designed by:
RTI International
The Odum Institute at the University of North Carolina
The authors wish to thank RTI International for permission to analyze the data used in this paper and disseminate the findings.
3
The Purpose of the Research
What matters more in accounting for variance in survey responses?
Mode of Data Collection (Telephone versus Internet)
or
Sample Origin (KN panel sample versus KN panel rejecters/non-cooperators)
This is a contest of data collection mode versus sample quality.
The main purpose of this research is to help further our understanding of the respective contributions of mode of data collection (phone vs. web) and sample origin (cooperators vs. non-cooperators) in explaining variance in survey response data.
4
Background of Study
The analysis is based on the Survey on Civic Attitudes and Behaviors After 9/11.
Study was created with a non-response follow-up study and implemented by Knowledge Networks from January to March 2002.
The study is important because: It is the first mode-effects study using KN data that
effectively controls for sample type (persons who joined the KN panel and participate in KN studies, versus those who refused to join the panel or who refused to participate in the panel study).
5
Questionnaire: Public Policy & Civic Attitudes & Behaviors
The survey questionnaire had several modules of interests: ratings of Bush and Gore (early 2002!), attitudes toward terrorism, the adequacy of governmental response to bioterrorism and terrorism more generally, attitudes toward what the government should be doing and would do in the event of a terrorist attack, civic participation and civic values questions, and background questions on religious faith and other aspects.
The survey was approximately 25 minutes in length.
6
Main Finding
The mode of data collection is more often a significant factor in accounting for variance in survey responses than is the origin of the sample. When controlling for panel experience and demographics, for every one question where sample origin was a significant factor, mode of data collection was significant for six questions.
MODESample
7
Sample Design for the Study
RDD Sample
Panel Acceptors
Panel Rejecters
Internet Sample
Telephone Sample (Controls)
Non-Response Follow-Up Sample(NRFUS)
Random sample for each of three groups.
8
Sample Composition, Mode, & Sample Size
Sample Group/ModeSample Size
FieldedCompleted Interviews
Completion Rate
KN Panel by Web 3627 2979 82%
KN Panel by Telephone (Controls)
477 300 63%
Nonresponse1 (NRFUS) by Telephone
2730 600 22%
1Nonresponse sample consists of ‘panel rejecters’ and those panelists that did not respond to the Internet survey.
9
Sample Representativeness: Demographic Comparison
Each interviewed group was compared to the January 2002 Current Population Survey (Census)
Main Findings: Mean Error (Unweighted):
– Panel by Web: 2.8 percentage points
– Panel by Phone: 4.1 percentage points
– NRFUS by Phone: 3.6 percentage points
10
Summary of Multivariate Analyses: Count of Statistically Significant Predictors of Answers by Mode and NRFUS (p < .05)
N items
N significant for mode
N significant for NRFUS
Grade Bush's performance (attitudinal) 2 1
Worried about terrorism (attitudinal) 2 2
Information expected during bioterrorist event (attitudinal) 6 5
Sources from which anthrax information sought (behavioral) 6 6 2
Trusted source during bioterrorism event (attitudinal) 1
Feeling thermometers for Bush and Gore (attitudinal) 2 2
Important issues, politics, current events (behavioral) 3 3
Neighborhood statements (combination - attitudinal and behavioral) 13 9 1
Self-perception statements (attitudinal) 5 5 2
Volunteerism and/or donating behavior (behavioral) 4 2
44 34 6
100% 77% 14%
11
Results of Multivariate Analyses
Phone respondents more likely to: Disagree that bioterrorism is the most important problem Seek info on anthrax from …web, hotlines, national TV, own doctor, local
government, other Rate Pres. Bush and Al Gore higher on feeling thermometer Discuss politics Discuss community issues Help neighbors Be happy about their neighborhood Have pride in their neighborhood Have a sense of belonging in neighborhood Have a neighborhood that gives them pleasure Rely on neighbors Trust others Enjoy mixing socially with others
12
Freq Distributions by Group (Unweighted)
Question Panel by Internet
Panel by Telephone
NRFUS by Telephone
Rate George Bush for job done as President (% ‘A’ grade)
42 40 40
Grade Bush on dealing with terrorism
(% ‘A’ grade)
57 57 51
No large differences across the groups for grading political job performance.
13
Freq Distributions by Group (Unweighted)
Question: When a bioterrorism event happens, what info do you expect CDC to provide the public?
Panel by Internet
Panel by Telephone
NRFUS by Telephone
Type of GERM (% Yes) 93 94 96
Name CRIMINAL Suspects (% Yes) 38 60 60
No. of people dead, ill, etc. (% Yes) 71 85 88
Explanation of steps people should take to minimize risk (% Yes)
91 97 98
Recommendations for treating illness caused by germ (% Yes)
88 97 97
Travel advisories/restrictions (% Yes) 70 89 89
Telephone mode: Higher expectations for information from CDC.
14
Freq Distributions by Group (Unweighted)
Question:
Sought info on ANTHRAX or other SEVERE ILLNESSES from…
Panel by Internet
Panel by Telephone
NRFUS by Telephone
Local radio and TV 46 43 40
WEB health and news sites 12 26 17
Phone HOTLINES 1 5 6
Cable 24-hour News and Network News 26 46 43
Own Physician or Health Care Professional 4 10 12
Local or State Health Department 3 10 11
Telephone mode: Higher report of information-seeking behavior.
15
Freq Distributions by Group (Unweighted)
Question:
Within the past year, how often have you…
Panel by Internet
Panel by Telephone
NRFUS by Telephone
Borrowed, loaned, etc, small things with neighbors like sugar
(% Sometimes & Often)
39 43 46
Helped people in this neighborhood or they helped you with small favors
(% Sometimes & Often)
60 71 70
Telephone mode: Higher report of ‘good neighbor’ behavior.
16
Freq Distributions by Group (Unweighted)
Civic Self-Perception Statements
(% Completely Agree)
Panel by Internet
Panel by Telephone
NRFUS by Telephone
I am happy to live in this neighborhood 35 49 57
See myself as part of neighborhood 21 37 40
Feel sense of belonging to neighborhood
22 38 42
Being in neighborhood gives me a lot of pleasure
23 39 43
Would get my neighbors to fix problems
9 23 22
Telephone mode: Higher report of positive feelings toward own neighborhood.
17
Freq Distributions by Group (No weight)
Sociability Self-Perception Statements
(% Completely Agree)
Panel by Internet
Panel by Telephone
NRFUS by Telephone
I am trusting of others 9 23 22
I easily fit into groups 9 28 32
I like to mix with others 13 31 39
I tend to be a happy person 13 31 39
I enjoy helping others 23 41 46
Telephone mode: Higher report of positive sociability self-perceptions.
Discussion
19
Major Differences in Response Caused by Mode?
Major differences existing between Internet and telephone modes of data collection include:
Presence or absence of interviewer
Dependence on visual or aural communication
Interviewer or respondent control of interview pace or information sequence
Each difference can lead to multiple mode effects seen in the data.
(Dillman, Sangster, Tarnai, and Rockwood, 1996)
20
Presence or Absence of Interviewer
Can lead to: Tendency to answer on extreme ends of response
scales Non-differentiationSocial desirability / self-presentation bias
21
1. I am happy to live in this neighborhood.
2. I really see myself as a part of this neighborhood
3. I feel a sense of belonging to this neighborhood.
4. Being in this neighborhood gives me a lot of pleasure.
5. If there are things in my neighborhood that need to be fixed or improved, I would be able to get my neighbors to do something about it.
Presence or Absence of Interviewer:Neighborhood Statements
22
% Average Response by Sample Group:Neighborhood Questions
66.2%
79.6% 76.9%
16.9%
9.9% 10.6%16.5%
10.3% 11.6%
0.4% 0.2% 0.8%0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Internet Telephone NRFUS
Sam ple Group
% o
f A
vera
ge
Res
po
nse
Positive
Negative
Neither
Don't Know /Refused
Very similar results for self-perception items (e.g., “I am trusting.”)
23
Evidence of Non-Differentiation and Recency Effects on Phone (Neighborhood Statements)
Response: Completely Agree (+5)
Internet Telephone NRFUS
All same Answer 4.9% 16%* 13.8%*
4 out 5 Same 10.2% 14.8%* 17.3%*
Response: Neutral (0)
Internet Telephone NRFUS
All same Answer 2.1% 0.1% 1.4%
4 out 5 Same 2.5% 1.0% 2.4%
Response: Completely Disagree (-5)
Internet Telephone NRFUS
All same Answer 2.0% 1.1% 1.0%
4 out 5 Same 1.6% 0.4% 0.9%
*Significant differences at p<.05 between Internet and
Telephone and Internet and NRFUS for Completely Agree
24
Conclusions
Sample origins do not appear to be having much of an effect on results compared to mode of data collection
Differences in the data are consistent with inherent differences between Internet and telephone data collection, leading to patterns in the phone-collected data reflecting:
Social desirability bias Extreme positive responses on answer scales Answer distribution differences due to communication type differences Item Non-differentiation
Study limitation/Buyer Beware: Questionnaire potentially laden with more questions (compared to many surveys) susceptible to social desirability bias.
Thank you!mdennis@knowledgenetworks.com
(650) 289-2160