Toolkit Support for Usability Evaluation
05-830 Spring 2013 – Karin Tsai
1
Overview
• Motivation• Definitions• Background from Literature• Examples of Modern Tools
2
Motivation
• To improve or validate usability• Comparison between products, AB tests, etc.• Measuring progress• Verify adherence to guidelines or standards• Discover features of human cognition
3
Usability Attributes
• Learnability – easy to learn• Efficiency – efficient to use• Memorability – easy to remember how to use• Errors – low error rate; easy to recover• Satisfaction – pleasant to use and likable
4
Evaluation Categories
• Predictive– psychological modeling techniques– design reviews
• Observational– observations of users interacting with the system
• Participative– questionnaires– interviews– “think aloud” user-testing
5
Challenges and Tradeoffs
• Quality vs. Quantity– “Quality” defined as abstraction, interpretability, etc.– User testing – high quality; low quantity– Counting mouse clicks – low quality; high quantity
• Observing Context • Abstraction
– Event reporting in applications places burden on developers
– Complicates software evolution
6
• Evaluation Type: Predictive• Description: Uses a predictive human
performance model (“cognitive crash dummy”) to evaluate designs.
7
CogTool
8
CogToolPros ConsFree Limited in “realisticness”Good for getting a baseline evaluation of prototypes
Quite confusing at first (extremely high learning curve)
Instantly accessible (not limited by participant availability or completion of the system’s functionality)
Documentation is “daunting”
Neat concept and insight into human cognition Limited usefulness
Overall Score: 6.5/10
• Evaluation Type: Observational• Description: Aggregates developer-defined
event data in useful ways.
9
Mixpanel
10
MixpanelPros ConsVery powerful built-in analysis tools High learning curveGood API for automated scripting Expensive
Scalable Application events = developer burden/maintainability issues
Flexible to fit needs of developers Rate-limited (one request at a time)
Overall Score: 9.5/10
• Evaluation Type: Observational• Description: Real-time data visualization.
11
Chartbeat
12
ChartbeatPros Cons
Data is real-time Does not scale well (financially) with huge sites
Captures data that is hard to obtain via events (reading, writing, idling, active time, referrals, social integration, etc.)
Limited in the data it captures (have to “hack” it if you want event-like data)
Great for site monitoring Only records “page-level” interactionsReally awesome visualization Limited historical data accessEasy to use Not built for usability evaluation
Overall Score: 7/10
• Evaluation Type: Participative• Description: Watch a user complete a task on
your system while thinking aloud.
13
User Testing
14
User TestingPros ConsProbably best method for catching usability issues Small sample size (hit or miss)
Most thorough recording of user interaction with the system Not easily scalable (expensive)
“Think aloud” allows data insights not otherwise attainable from just user interactions
Limited user availability
Can observe certain demographics without requesting personal information in the system itself
Sometimes, it’s painful to watch…
Overall Score: 8.5/10
15
Questions?