RTI International
RTI International is a trade name of Research Triangle Institute. www.rti.org
Cognitive Interviewing in Online Modes: a Comparison
of Data Collected in Second Life & Skype
Jodi E. Swicegood
Elizabeth F. Dean
Brian F. Head
Michael D. Keating
RTI International
1
RTI International
Study Goals
Pilot test of cognitive interviewing in Skype and Second
Life (SL)
Cognitive testing of questionnaire about SL avatars &
identity
Recruitment – Evaluate recruitment potential of Facebook and Craigslist at recruiting
respondents to virtual cognitive interviews
– Assess feasibility of recruiting users of the virtual world SL
2
RTI International
Cognitive Interviewing
A pretesting technique used to identify questionnaire
problems, such as:
– Comprehension
– Vague or inconsistent instructions
– Logic or flow
– Participant burden
Often conducted in a lab setting, sometimes on-site
Challenges:
– Recruitment
– Identifying appropriate sample
– Selecting appropriate mode
3
RTI International
Virtual Cognitive Interviewing with New Technologies
Shared online spaces introduce a range of options for
cognitive interviewing methodology.
– Skype video calling enables face to face, private cognitive
interviews in any location
– Virtual worlds like Second Life allow users to conduct and
participate in cognitive interviews through avatars
Pros
– Access to wider population without travel
– Convenience for researcher & participant (e.g. scheduling)
– Reduced time and costs
Considerations
– Technical requirements; technical knowledge
4
RTI International
Research Questions
1). How do these virtual interviewing modes compare in
terms of...?
Number
Type
Severity of errors – total # of problems observed across
measures
2). Does data quality differ across modes?
5
RTI International
Data Collection Process
Collected data for 2 ½ months
Conducted 40 interviews w/ SL users
– 21 in SL
– 16 Skype
– 3 In-Person
All participants were contacted in SL 6
RTI International
Second Life Interviews
7
RTI International
Skype Interviews
8
RTI International
Measures of Data Quality
Audio Quality (1=low; 2=medium; 3=high)
Video Quality (1=low; 2=medium; 3=high)
Participant Disengagement (1=disengagement)
Nonverbal Cues (Count of the # of nonverbal cues)
Types of Problems (1=problem identified)
– Comprehension
– Retrieval
– Judgment
– Response
Total Number of Problems (Sum)
9
RTI International
Participant Disengagement
Mode
Total
Observations of
Participant
Disengagement
Mean Number of
Participant Disengagement
(SD)
Second Life (n = 17) 14 0.82 (1.67)
Skype (n = 16) 18 1.13 (1.75)
In Person (n = 2) 0 0 (0)
Total Interviews
(n = 35)
32 0.91 (1.65)
10
RTI International
Nonverbal Cues
Mode
Total Observations
of Nonverbal Cues
Mean Number
Nonverbal Cues (SD)
Second Life (n = 17) 9 0.53 (0.80)
Skype (n = 16) 34 2.13 (3.50)
In Person (n = 2) 19 9.50 (13.44)
Total Interviews
(n = 35)
62 1.77 (3.92)
11
RTI International
Problems in the 4 Stages of the Response Process
30
8 5
42 43
4 8
66
13
1 3 8
0
10
20
30
40
50
60
70
# O
F O
BS
ER
VA
TIO
NS
MEASURE
Question Problems per Mode
SL (n=17)
Skype (n=16)
In Person (n=2)
12
RTI International
Discussion
Relationship between presence and number of problems
observed
– In person (most presence): total problems: 29; mean: 14.5
– Skype : total problems: 114; mean 7.13
– SL (least presence): total problems: 94; mean: 5.53
We believe cognitive interviews are feasible
– Skype: used with global or otherwise widely dispersed
populations; testing questions for an online panel
– SL: testing survey that contains sensitive topics (i.e. online role
playing & gaming; online relationships).
13
RTI International
Conclusion
– Use of virtual cognitive interviewing should be
weighed according to total costs & potential benefits
– Rapid growth in web camera & software technology
may decrease the current gap in presence between
face-to-face & virtual modes
– Future research should continue testing these
platforms (and the ones that will be created in time for
our next study) adding greater controls (i.e. random
mode assignments; larger sample sizes)
14
RTI International
Questions?
15
RTI International
Contact Me
Jodi Swicegood
Research Methodologist
SL: Katherine Cunningham
Skype: jodiswicegood
SurveyPost blogs.rti.org/surveypost
@SurveyPost