Date post: | 18-Dec-2015 |
Category: |
Documents |
Upload: | samson-cox |
View: | 214 times |
Download: | 0 times |
Use of eye-tracking for studying survey response processes
Mirta GalesicRoger Tourangeau
Fred ConradMick Couper
September 10, 2009
Why eye-tracking?
To get another perspective on results of our web experiments To resolve some ambiguities in the data To test some emerging hypotheses To gain additional insight in the answering process
In Fall 05: small pretest (N=24)
This year: larger study (N=117) These are the first results
Acknowledgments
Acknowledgments: We are grateful to Scott Fricker, Duane Gilbert, Ting Yan, and Cong Ye for their help in conducting this study.
This research was supported by a grant from the National Institute for Child Health and Human Development (R01 HD041386-01A1) to Roger Tourangeau, Mick Couper, Fred Conrad, and Reg Baker. The National Institute for Child Health and Human Development is not responsible for the conclusions presented here.
Eye-tracking technology
TOBII: ClearView analysis software + hardware Unobtrusive eye-tracking
Uses near-infrared beams and video
images to capture one’s eye movements No need for helmets, lenses...
Easy calibration Good accuracy
Frame rate 50Hz Margin of error +/- 3 ms (time),
+/- 0.5-1 degree (position)
Data: fixations and durations
Procedure
Lab experiment Sample
N=117 – recruited through advertisements on campus, ads in local newspapers, flyers in libraries, on bus stations, etc.
Age: 48% 18-24, 34% 25-34, 17% 35-64 Sex: 50% male, 50% female Education: most have at least some college Most (80%) use the Internet every day, 59% consider
themselves advanced or expert users Most (77%) already participated in at least one Web survey
Questionnaire A combination of previously used web experiments on
visual context effects, response order, question format, and definitions
1. Response order effects
Rationale
Order of response options can affect the results Possible underlying mechanisms...
Diminishing attention to later options—may not even read all the options
Low threshold for acceptance—consider each option in turn for acceptability; stop when answer is good enough (“satisficing” in the original sense)
Either way, time spent on reading the first options should be longer than the time spent reading the last options on the list
This study: several questions with varying order of response options
Desirable qualities of a child
Which one of these qualities is the most desirable for a child to have?
That he has good manners That he tries hard to succeed That he is honest That he is neat and clean That he has good sense and sound judgment That he has self-control That he acts like a boy or she acts like a girl That he gets along well with other children That he obeys his parents well That he is responsible That he is considerate of others That he is interested in how and why things happen
A B
Other questions: Crime, Police, Morality
Q7. Some say individuals are more to blame than social conditions for crime and lawlessness in this country. Others say the contrary—social conditions are more to blame than individuals for crime and lawlessness in this country. Which one of these two statements comes closest to your opinion on this issue?
Individuals are more to blame. Social conditions are more to blame.
Q8. Next, we would like you to think about the amount of trust you have that the police officers in your area will always do what is right. Would you say you have—
A great deal of trust A moderate amount of trust Equal amounts of trust and distrust A moderate amount of distrust A great deal of distrust
Q9A. In your opinion, should government (federal, state, or local) have some responsibility for preventing the breakdown of morality, or should private organizations and individuals be entirely responsible for preventing the breakdown of morality?
Government is responsible Private organizations and individuals are responsible
Different response styles
Considering all options and choosing the best answer: web14
Selecting the first option, then going through the list and updating the response: web03
Reading only part of the list, then selecting the answer: web01
Hot spot analysis: answer in the first half
Hot spot analysis: answer in the second half
Fixations at response options – Top vs. Bottom half
21.8 4.83.4
3.9
26.3 6.56.3
5.1
0%
50%
100%
q6 - child (12 options),t(106)=4.04, p<.01
q7 - crime (2 options),t(107)=3.62, p<.01
q8 - trust (5 options),t(107)=5.72, p<.01
q9 - morality (2 options),t(107)=2.4, p=0.02
Bottom half Top half
For all questions: more fixations in the top part…
Time spent looking – Top vs. Bottom half
Notes: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001). T-tests calculated on log-transformed data.
6317 ms 1309 ms 911 ms1025 ms
7648 ms 1851 ms 1608 ms1266 ms
0%
50%
100%
q6 - child (12 options),t(106)=3.38, p<.01
q7 - crime (2 options),t(107)=3.07, p<.01
q8 - trust (5 options),t(107)=6.01, p<.01
q9 - morality (2 options),t(107)=1.04, p=0.30
Bottom half Top half
...and more time spent looking at the options in the top part
Proportion of time and answers in the first half
Notes: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001). T-tests calculated on log-transformed data.
55%
59%
66%
53%
60%
55%
69%
51%
40%
50%
60%
70%
80%
q6 - child (12options)
q7 - crime (2options)
q8 - trust (5options)
q9 - morality (2options)
% of time spent looking at the first half
% of answers in the first half
Relationship between gazing time and answers
22%
60% 57%
77%86%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
po
rtio
n o
f an
swer
s in
th
e fi
rst
hal
f
q6 - child (12 options)
2(4)=22.59, p<.01
Note: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001).
Respondents who spend more time looking at the top part are more likely to choose an answer from that part:
Percentage of answers in the first half, by time spent looking at the first half
Relationship between gazing time and answers
Note: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001).
Percentage of answers in the first half, by time spent looking at the first half
27% 29%
45%
71%
91%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
po
rtio
n o
f an
swer
s in
th
e fi
rst
hal
f
q7 - crime (2 options)
2(4)=26.79, p<.01
Relationship between gazing time and answers
Note: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001).
Percentage of answers in the first half, by time spent looking at the first half
35% 31%
87%79%
96%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
po
rtio
n o
f an
swer
s in
th
e fi
rst
hal
f
q8 - trust (5 options)
2(4)=33.66, p<.01
Relationship between gazing time and answers
Note: Times corrected for the time needed to click on an answer (200 msec) (c.f. Kieras, 2001).
30%43%
59% 58%70%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
po
rtio
n o
f an
swer
s in
th
e fi
rst
hal
f
q9 - morality (2 options)
2(4)=8.27, p<.01
Percentage of answers in the first half, by time spent looking at the first half
2. Visual format of response scale: Radio buttons vs. Drop down lists
Rationale
Previous studies showed that people more often choose response options that are initially visible, than the options they need to uncover by additional mouse clicks
For example, in some drop-down lists, several options are initially visible; the others appear after an additional click
Rationale
Typical result: options on top chose more often
Experimental design
3 different question formats: radio buttons, drop-down list with 5 options initially visible, and drop-down list with 0 options initially visible
Two questions: on breakfast cereal and automobiles; 10 response options each
Order of response options also systematically varied
Question formats
1. RADIO BUTTONS
2. DROP DOWN LIST – 5 OPTIONS INITIALLY VISIBLE
3. DROP DOWN LIST – NO OPTIONS INITIALLY VISIBLE
Effects of question format: % of time spent on top part
Overall F(2,105)=10.76, p<.01.
Drop box 5 significantly different from others; and significantly different from 50%.
If top options are the only ones shown initially, Rs look at them much longer
53%
74%
50%53%
72%
53%
40%
50%
60%
70%
80%
90%
Radio buttons Drop box, 5 optionsvisible
Drop box, no optionsvisible
q3 - Breakfast cereal q5 - Automobile
** **
Effects of question format: % of answers in the top part
Top options (especially when they are the only ones shown initially) are chosen somewhat more often.
55%
63%
48%
66%
78%
57%
40%
50%
60%
70%
80%
90%
Radio buttons Drop box, 5 optionsvisible
Drop box, no optionsvisible
q3 - Breakfast cereal q5 - Automobile
**
For both questions, overall 2 n.s.
Relationship between gazing time and answers
13%
32%
57%
75%90%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
port
ion
of a
nsw
ers
in th
e fir
st h
alf
q3 - Breakfast cereal
2(4)=29.28, p<.01
Percentage of answers in the top half, by time spent looking at the top half
Relationship between gazing time and answers
2(4)=18.70, p<.01
Percentage of answers in the top half, by time spent looking at the top half
41% 43%
70%76%
95%
0%
20%
40%
60%
80%
100%
1 2 3 4 5
Proportion of time spent looking at the first half: Quintiles
Pro
po
rtio
n o
f an
swer
s in
th
e fi
rst
hal
f
q5 - Automobile
3. Definitions
Why definitions
Survey concepts are not always understood as intended – definitions can be useful
But – do people read the definitions? Rarely: only 14% - 22% clicked on a link to definition
in recent experiments (Conrad et al, 2005) Substantially more likely to do so for technical than for
ordinary concepts More likely when the definition more accessible Reading the definition may change answers
Conclusions of those studies based on indirect data: number of requests for definitions, response times
This study
Questions about consumption of 8 food items:
Page 1: Fat, Dietary supplements, Grain products, Poultry
Page 2: Vegetables, Diary products, Cholesterol, Calcium
Each item accompanied with a definition
Presentation of the definitions
1) Always on
2) On mouse roll-over
We measured the time spent looking at each definition
Example of “Always on” definitions:
Example of “Mouse roll-over” definitions:
Some respondents appeared to read the definitions…
… and some didn’t
Reading definitions on the first page
Reading definitions on the second page (the same respondent)
Do they read?
In silent reading, a typical fixation lasts about 225 msec, covering about 8 letters (Rayner, 1998)
Only 54% of the respondents read at least 16 letters (about two words) of the definitions i.e. spent at least 2x225 msec looking at the definitions
Big difference between definitions that were always on and those that were opened by mouse-roll-over: Only 20% of the Rs in the mouse roll-over condition
opened at least one definition; but only 9% opened it long enough to fixate at least two words
Vs. 96% in the always-on condition who fixated at least two words in any definition
Total gaze time, by format
t(63)=7.93, p<.01, for log-transformed data
22,482
5420
5,000
10,000
15,000
20,000
25,000
Always on Mouse roll-over(those who
opened)
Gaz
e tim
e in
mse
c Total gaze time (for all definitions together) much longer
in always-on condition
Gaze time per definition, by format
t(63)=5.25, p<.01, for log-transformed data
Gaze time per definition is also much longer in always-on condition
2,810
208
0500
1,0001,500
2,0002,500
3,000
Always on Mouse roll-over(those who
opened)
Gaz
e tim
e in
mse
c
Reading time and estimates of consumption: Dietary supplements
DEFINITION:
A multivitamin supplement taken daily is recommended to help insure adequate levels of necessary vitamins and micronutrients. In addition, dietary supplements help protect cells against aging, improve sexual performance and reduce stress, among other benefits.
r = - 0.19, p<.01
Much less than I should ------------- Much more than I should
Log
read
ing
time
(log
mse
c)
12
10
8
6
4
2
0
Reading time and estimates of consumption: Grain products
DEFINITION:
Bread and foods made with bread, including muffins, French toast, stuffing, popcorn, and pre-sweetened cereals. Include also pasta, rice, and drinks such as beer.
r = 0.20, p<.01
Much less than I should ------------- Much more than I should
Lo
g r
ea
din
g t
ime
(lo
g m
se
c)
10
8
6
4
2
0
Summary
RESPONSE OPTIONS People read from top to bottom; answer depends on
how low they go; Different answering styles More time spent on looking at the top part The more time respondents spend on reading the
top part, the more likely is that they will choose an answer from that part
Summary
QUESTION FORMAT Radio buttons vs. Drop-down lists When only some options are initially shown, respondents
read them longer and are more likely to select their answer among them
DEFINITIONS Mouse roll-over definitions are rarely read When read, can affect results
The End