Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London) Copyright 2010 element54 Inc. All rights reserved. Permission to copy and distribute this document is granted subject to the following: The materials provided herein are for internal, non-commercial use only, and any copies or portions thereof must include the copyright notice, and reference the author, element54. bernie.malinoff@element54.com Sexy Questions, Dangerous Results?
Transcript
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Copyright 2010 element54 Inc. All rights reserved. Permission to copy and distribute this document is granted subject to the following: The materials provided herein are for internal, non-commercial use only, and any copies or portions thereof must include the copyright notice, and reference the author, element54.
bernie.malinoff@element-‐54.com
Sexy Questions, Dangerous Results?
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Origin of online research: ‘Radio Button’ question types
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Typical respondent experience
Response PaMerns/ Flatlining "Engagement
& Enjoyment"
Onscreen Layout Respondent
ExpectaVons
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Divergence/Variety of New Question Types
Sources: Spyglass Solutions/2x4 Research & Consulting
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
How we got here
Door-‐to-‐Door
Mail CATI/Phone
Online
Does "Usability" MaMer?
No YES No YES
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Major R&D Initiatives Data Consistency Respondent Usability
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
‘Respondent Engagement’ Converging Issues
Data Consistency
Respondent Usability
(no, it's not) (yes, it maMers)
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Balanced 5 Cell Design 2000 total interviews
Radio BuMon
Anchor @ ‘1’
Anchor @ ‘5’
Anchor @ ’10’
Experimental
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Study Highlights
36% variance how oden brush teeth -‐ slider vs. radio buMon
+8% overstated past 7 day consumpVon -‐ slider vs. radio buMon
-‐8% understated issue importance -‐ slider vs. radio buMon
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
eg -‐ NetPromoter Score (NPS)
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Survey Completion Time & Respondent Enjoyment
Rich Media vs. Radio Button Questions
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Eye Tracking
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Using Eye Tracking for Survey Usability
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Eye Tracking -‐ 'Areas of Interest'
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Inference vs. Instruction
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Inference vs. Instruction 42% did not read the quesVon
33% did not read the "tradiVonal"
quesVon
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Error Prevention & Recovery
This respondent needed 5 minutes to complete this question and encountered 3 error messages.
The question is - how do we treat the data which emerges from someone who tried hitting "next", and then added or changed data to escape?
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Error Prevention & Recovery
Respondent interpreted the error message location as a guide to where to put the answer. Recommendation - highlight the entire row, avoiding any visual cues or biases.
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Minimum Standards -‐ eg: Horizontal Layout
Horizontal layout ensures consistent viewing, regardless of screen size
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Equal Ratings for ‘Engagement’ Metrics
1.0 8.20
8.86
7.02
2.0 8.25
8.85
6.56
Survey Enjoyment
Ease of Use
Desire to Repeat
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
The Perfect Storm
Clients' expectaVon;
suppliers know what they're doing ... of course.
"Full Serve" Suppliers;
programming is a black box cost center; 1.0 vs 2.0 adopVon
rate discrepancy
Programmers; task complexity has increased exponenVally (training? pressure?)
"DIY" sodware is in a space race; widgets and pulling
ahead of best pracVces
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Where do we go? 1. There is no BeMer, Faster, Cheaper -‐ you can only pick 2.
2. We must return to the usability principles which guided paper survey design, not CATI design, to maximize confidence in our data, and business decisions.
3. There are ways to make all quesVons/surveys more appealing, intuiVve and future-‐relevant.
4. 'Responsible InnovaVon' is key.
5. As an industry, we must devise and reinforce best pracVces for online research interfaces, like we have for so many elements of the research process.
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Q & A
Bernie Malinoff and Ray Poynter
Bernie Malinoff, element54, Canada Part 2: Session 3, Convenor Sue York, Chair Ray Poynter, schedule = 2:22pm to 2:46pm (GMT/London)
Thank you
Copyright 2010 distribute this document is granted subject to the following: The materials provided herein are for internal, non-commercial use only, and any copies or portions thereof must include the copyright notice, and reference the author, element54.
