Date post: | 13-Apr-2017 |
Category: |
Design |
Upload: | caroline-jarrett |
View: | 643 times |
Download: | 1 times |
The Survey OctopusGetting valid data from surveys
Caroline Jarrett@cjforms2016 #uxinthecity
Caroline Jarrett @cjforms
What would you do for a dollar?
2
Caroline Jarrett @cjforms (CC) BY SA-4.03
$1 in the envelope beats $10 (guaranteed) later
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms (CC) BY SA-4.04
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on effort, reward, and trust
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
5
Caroline Jarrett @cjforms (CC) BY SA-4.0
I’m a forms specialist
Image credit: Flickr, taxrebate.org.uk
6
Caroline Jarrett @cjforms (CC) BY SA-4.0
Why do people answer questions?
Caroline Jarrett @cjforms (CC) BY SA-4.07
Caroline Jarrett @cjforms
“Please have a look at this survey”
8
Caroline Jarrett @cjforms
The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
Caroline Jarrett @cjforms
The survey is a processfor gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
Caroline Jarrett @cjforms
The survey is a processfor getting answers to questions from(a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
Caroline Jarrett @cjforms
The survey is a processfor getting answers to questions from(a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
Caroline Jarrett @cjforms
The survey is a processfor getting answers to questions from(a sample of) people for the purpose of getting numbersof the attributes of the larger population of which the entities are members.
Caroline Jarrett @cjforms
The survey is a processfor getting answers to questions from(a sample of) people for the purpose of getting numbersthat you can use to make decisions
Caroline Jarrett @cjforms
To make decisions People
The survey is a process for getting answers to questions
getting numbers
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim of a survey is to get a number that helps you to make a decision
16
Caroline Jarrett @cjforms
“The numbers” or “The number”?
17
Caroline Jarrett @cjforms (CC) BY SA-4.0
Fieldwork used to be expensiveso a survey was a rare event.
Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/
18
Caroline Jarrett @cjforms (CC) BY SA-4.0
1950s mindset: “Ask Everything”
Survey = Big Honkin’ Survey
19
Caroline Jarrett @cjforms (CC) BY SA-4.0
2016 mindset: the Light Touch survey• Choose ONE question• Find ONE person• Ask the question, face-to-face• See if you can make ONE decision• Improve, iterate, increase
20
21
Tip Interview first
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms
One way toiterate, improve,increase
Time for new question
Caroline Jarrett @cjforms (CC) BY SA4.022
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim of a survey is to get a number that helps you to make a decision
23
Caroline Jarrett @cjforms
24
Caroline Jarrett @cjforms
Asking the right question
Asking the right people
25
Caroline Jarrett @cjforms
26
Caroline Jarrett @cjformshttp://www.bbc.com/news/10506482 27
Caroline Jarrett @cjforms (CC) BY SA-4.0
Asking the right people is better than asking lots of people Sample:
the list you sample from
Caroline Jarrett @cjforms (CC) BY SA-4.028
Caroline Jarrett @cjforms
29
Caroline Jarrett @cjforms (CC) BY SA-4.0
There’s often a ‘zone of indifference’
Hate it Love it
30
Caroline Jarrett @cjforms (CC) BY SA-4.0
Feeling special is rewarding
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
31
Caroline Jarrett @cjforms (CC) BY SA-4.0
Ask fewer people to get better response
Sample: the number of people to ask
Caroline Jarrett @cjforms (CC) BY SA-4.032
Caroline Jarrett @cjforms (CC) BY SA-4.0
Try mail, phone or face-to-face for better response rates
Fieldwork:Who answers?
Caroline Jarrett @cjforms (CC) BY SA-4.033
34
Did we get answers from the right people?
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
35
Population of assorted birds
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t just want answers, we want representative answers
Response
Caroline Jarrett @cjforms (CC) BY SA-4.036
Caroline Jarrett @cjforms
Asking the right question
Asking the right people
37
Caroline Jarrett @cjforms
Asking one person the right question
is better thanAsking 10,000 people the wrong question
38
Caroline Jarrett @cjforms
39
Caroline Jarrett @cjforms
In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.040
Caroline Jarrett @cjforms (CC) BY SA-4.0
Helps a lot if you ask good questions
Questions:What are you asking about?How many questions?
Caroline Jarrett @cjforms (CC) BY SA-4.041
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on effort, reward and trust
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
42
Caroline Jarrett @cjforms (CC) BY SA-4.0
A good question works in three ways
Obvious Interesting
Appropriate
43
Caroline Jarrett @cjforms (CC) BY SA-4.0
Why did you visit our website today?
Obvious Interesting
Appropriate
44
Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to a friend or family member?
In a shop, buying a baby buggy
In a hospital,having a miscarriage
Obvious YesInteresting YesAppropriate Yes Cruelly
inappropriate
45
Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to a friend or family member?
In a shop, buying a baby buggy
In a hospital,having a miscarriage
Obvious YesInteresting YesAppropriate Yes Cruelly
inappropriate
46
47
TipTest your questions by interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t want any old answers. We want useful answers.
Response
Caroline Jarrett @cjforms (CC) BY SA-4.048
Caroline Jarrett @cjforms
In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.049
Caroline Jarrett @cjforms
On your most recent working day,what percentage of time did you spend using the Internet?
%
Caroline Jarrett @cjforms (CC) BY SA-4.050
Caroline Jarrett @cjforms (CC) BY SA-4.051
The approximate curve of forgetting
Major life event
Noticeable, occasionalUnremarkable,repetitive
Perfectrecall
DayGone
Week Month YearHour
Caroline Jarrett @cjforms (CC) BY SA-4.0
Can’t remember = can’t answer
Response
Caroline Jarrett @cjforms (CC) BY SA-4.052
Caroline Jarrett @cjforms (CC) BY SA-4.0
Insights are the answers that you use to make the decisions
Insights Caroline Jarrett @cjforms (CC) BY SA-4.053
Caroline Jarrett @cjformsCaroline Jarrett @cjforms (CC) BY SA-4.054
Caroline Jarrett @cjforms
"Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Phone_photography.jpg#/media/File:Phone_photography.jpgModified by Caroline Jarrett
55
56
TipAlways allow for ‘other’
Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk
Caroline Jarrett @cjforms
Can we make the decision yet?
Can we make the decision yet?
Can we make the decision yet?
Caroline Jarrett @cjforms (CC) BY SA4.057
58
TipTry writing your presentation first
Caroline Jarrett @cjforms (CC) BY SA-4.0
We’ve met the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.059
What could possibly go wrong?
Caroline Jarrett @cjforms (CC) BY SA-4.060
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline JarrettTwitter @cjforms
http://www.slideshare.net/cjforms
61