Post on 15-Jan-2017
transcript
Research to Inform Development DecisionsSteve Fadden, Ph.D.
Director, Analytics UX Research, Salesforce
Lecturer, Professional Faculty, UC Berkeley School of Information
Presented at the UserTesting Everyday Insights Roadshow, San Francisco, CA
August 23, 2016
Image: https://pixabay.com/en/photos/spiral%20staircase/
Image: https://www.flickr.com/photos/jiuguangw/5134925627
Agenda1. Introductions
2. Process
3. Formative Methods
4. Case study
About me
Steve Fadden
@sfadden
linkedin.com/in/stevefadden
About youFocus
● Research
● Design
● Engineering
● Marketing
● Management
● Other?
Research experience
● Over 5 years
● 2-5 years
● Up to 2 years
● None
About the research processGenerative: Creation
Formative: Evaluation
Summative: Assessment
Images: https://commons.wikimedia.org/wiki/File:Idea.png,
https://pixabay.com/en/selection-pick-chose-choice-select-443127/,
https://pixabay.com/en/graph-statistics-chart-infochart-29709/
About the research processGenerative: Creation
Formative: Evaluation
Summative: Assessment
Images: https://commons.wikimedia.org/wiki/File:Idea.png,
https://pixabay.com/en/selection-pick-chose-choice-select-443127/,
https://pixabay.com/en/graph-statistics-chart-infochart-29709/
Formative methods inform progressFormative
During development
Lower fidelity
Focused on improvement
Aid decision-making
Faster & lighter
Summative
Ready to launch
High fidelity
Focused on measurement
Understand current state
More comprehensive
Technique 1: Understand the problem
Image: https://pixabay.com/en/detective-magnifying-glass-viewing-788592
Gain insights:
● Evidence of problems
● Opportunities to solve
Based on:
● Recent events
● Specific details
● Feelings and perceptions
● Future behaviors, responses
Critical incident questions
Process1. Confirm user profile
2. Identify time since last experience
3. Gather details
a. Description
b. Actions taken
c. Feelings
d. Outcome
e. Future actions and responses
Reference: http://www.usabilitynet.org/tools/criticalincidents.htm
Applying your findings1. Validate problems
2. Identify opportunities
3. Understand expectations
4. Identify scenarios
5. Build empathy
Image: http://www.freestockphotos.biz/stockphoto/17339
“I needed to share a PDF with a friend, and we use Dropbox, but I hadn’t used it in a while. I logged in through my browser, dragged the PDF to Files, and then saw the PDF open. I expected Dropbox to start uploading. I hit back, created a folder in Dropbox, uploaded the PDF to it, clicked share to add my friend, and sent her the link.”
Technique 2: Assess expectations
Expectation testsGain insights:
● Mental model
● Categorization, labels
● Confusion areas
With:
● Nonworking interface
● No text/labels
● Basic task information
Image: https://pixabay.com/en/photos/man%20thinking/
“Greeking” technique1. Identify important tasks
(e.g. frequent, critical)
2. Write scenario
3. Write “first step”
questions
4. Gather feedback
5. Ask for group names
Example
Scenario: “You are exploring colleges, and need to make sure the courses look interesting and financial aid is available.”
Instruction: “How would you expect to start to apply for financial aid?”
Reference: Thomas S. Tullis, 1998. A method for evaluating web page design concepts. CHI 98 conference summary on human factors in computing systems
“Greeking” activityScenario: You are exploring colleges,
and need to make sure the courses look
interesting and financial aid is available.
How would you expect to:
1. Find out more about financial aid?
2. View information about courses
offered?
3. Contact the department in charge
of Athletics?
“Greeking” resultsHow would you expect to:
1. Find out more about financial aid?
2. View information about courses
offered?
3. Contact the department in charge
of Athletics?
What would you call each group?
12
3
“Academics” “Administrative”
“Connections”
Technique 3: Prioritize needs
Kano model
Absent Present
Dissatisfaction
Satisfaction
Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
Kano model
Absent Present
Dissatisfaction
Satisfaction
Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
Exciting
(truly unexpected)
Perform
ance
(m
ore is better) Basic
(required)
Kano model (automobile example)
Absent Present
Dissatisfaction
Satisfaction
Seat-warmers
Bluetooth
Intuitive GPS
H
orsepow
er
Fuel econom
y
Airbags
Adjustable seats
Kano question processProcess
● Assemble list of features
● Pose realistic scenarios for context
Assess for each:
● Importance of feature
● Satisfaction if available
● Satisfaction if not available
Scenario: “Imagine you are using our mobile app to find a restaurant.”
Question 1: “How would you feel if the app allowed you to filter results by distance?”
Question 2: “How would you feel if the app did not allow you to filter results by distance?”
Question 3: “How important is this function to you?”
Kano question results
If present :( If present :)
If absent :)
If absent :(
Map answers to “product
allowed you to...”
questions
Map answers to
“product did not allow
you to...” questions
Kano question results
If present :( If present :)
If absent :)
If absent :(
Providing
this feature is
dissatisfying
Providing
this feature
is satisfying
Not providing this feature is satisfying
Not providing this feature is dissatisfying
Kano question results pattern
If present :( If present :)
If absent :)
If absent :(
Providing
this feature is
dissatisfying
Providing
this feature
is satisfying
Not providing this feature is satisfying
Not providing this feature is dissatisfying
Technique 4: Evaluate the solution
Usability studyGoals
● Explore concepts
● Assess effectiveness
● Compare competitors
● Validate performance
Measures
● Learnability
● Efficiency
● Memorability
● Errors
● Satisfaction
Process
1. Plan (goals, participant criteria)
2. Prepare (recruit, test materials)
3. Conduct (active observers)
4. Analyze (issues, insights)
5. Report (fixes, future opportunities)
Reference: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
Usability studyRITE - Rapid Iterative
Testing & Evaluation
(address problems)
Discount (find problems)
Traditional (benchmark
performance)
Aim to:
● Test representative users● Follow script with actual tasks● Use high-fidelity interface
Key elements:
● Observation by key decision makers (PM, design, development)
● Agree on critical problems and make changes to interface after each session
References: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2nd Ed.), Rubin, J. & Chisnell, D. (2008)
Jakob Nielsen, 1989. Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using
human-computer interfaces and knowledge based systems (2nd ed.); Image: The RITE Way to Prototype, J. Shirey et al., UX Magazine, 2013
Using the RITE method to improve products; a definition and a case study, M. Medlock et al., Presented at UPA, 2002.
Example: Understanding data export
Approach used● Background
○ Urgently-needed integration with
a 3rd party product
○ Very visible launch event was
already scheduled
○ Small team, no dedicated research
support
○ “Boil the ocean” effort: Was
concept and functionality
understood, desired, and usable?
Process
Critical incident
Expectation
Prioritization
Usability evaluation
Critical incident elicited rich feedback“Consider the last time you had to export data. Describe why you
needed to export data, and list the steps you remember from that
process. (If you haven’t exported data before, or don’t remember
the last time, just skip to the next question).”
“I’m pretty old school, so I export my credit card transaction data about every quarter. My credit card site has a button to export to CSV, so I just click that and it downloads to my computer.”
“We have our marketing, sales, and inventory data in different systems. I have to export data from each system in order to combine it into a spreadsheet for my stakeholders. The export process is easy. Combining the data is more involved.”
Expectations gathered from storyboard“Consider the concept presented on the next 4 slides. As you
read through the concept, comment on anything you find to be
confusing, problematic, useful, or appealing about the concept.”
1. 2. 3. 4.
100%
Initial expectations highlight concerns1.
2.
3.
4.
100%
“Doing this would require a lot of clicks, even for a small number of columns.”
“You should embed best practices for naming here. Otherwise, the result could be messy.”
“Will we be able to save the mappings? That could save time in the future.”
Directed feedback emphasized value and priorities
“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”
“It’s useful, but only if I can save it and reuse it in the future. Wouldn’t use it for a one-time export.”
“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”
Usability sessions identified more issues
● RITE with 4 participants over 2 days
● Key changes: Terminology, embedded instructions, icons
● Stakeholder advocacy (and gratitude)
“It says ‘save’ but I’m just modifying the settings. Unless I can re-use this later?”
“I haven’t selected the columns yet! Why can’t I go back?”
Some formative evaluation results● Evidence of problems
● Value of overall solution
● Prioritizations and tradeoffs
● Issues with flow, terminology, design elements
● Importance of incorporating formative evaluation into
process
Thanks!
Image: https://en.wikipedia.org/wiki/File:TulipStair_QueensHouse_Greenwich.jpg