Post on 21-Jan-2016
transcript
Evaluation Methods - Summary
How to chose a method?
Stage of study– formative, iterative, summative
Pros & cons Metrics
– depends on what you want to measure Qualitative vs. quantitative Research perspective
– CS vs. psychology vs. sociology
Pros & Cons
Realism Precision Generalizability Time & cost Researcher expertise
Methods
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Experimental simulation GOMS Computer simulation Formal theory
Survey
Pros– Easy to get a large number of responses.– Quick and easy to conduct.– Highly generalizable.
Cons– Self-selection.– Participants often only offer enough information to answer the
question.– Can miss details.– Low in realism and precision.
Interviews
Pros– Quick and easy to conduct.– Gives designer quick feedback on a range of ideas.– Can get a person’s initial reaction to an idea.– Can get detailed information from a person.
Cons– Often takes place away from natural setting.– Question wording or interviewer “body language” can bias
answers.– High probability of false positives and missed problems (e.g., users
may not have a clear idea of how an app will be used).– Can miss details if interviewer doesn’t know what issues to draw
out.
Controlled Lab Experiment
Pros– Provides precise, quantifiable data.– Easier to draw inferences from data.– Relatively quick.– Can get a medium-sized number of participants.
Cons– Short duration of a lab experiment may not be enough to allow
users to become accustomed to an app.– Not a natural setting – interaction may not be normal.
In-lab Observation
Pros– Relatively quick.– Can get a medium-sized number of participants.
Cons– Observations are subjective and error prone.– Short duration of lab observation is not enough time
for user to get accustomed to using the interface.– Not a natural setting – interaction may not be
normal.
Controlled Field Experiment
Pros– Less intrusive than most other evaluation methods.– Provides more precise data than field observation.– Can observe natural behavior of user (though some
part of the system will be controlled/unnatural).
Cons– More intrusive than field observation.– Less natural than field observation.
Field Observation Study
Pros– Only way to observe natural behavior of user & interaction
between user & tools.
Cons– Difficult and time consuming.– Hard to get permission to observe people.– Observations are subjective and error prone.– Cannot make strong interpretations from observations.– Not very generalizable.
Heuristic Evaluation
Pros– Quick and easy.
Cons– Nielson’s heuristics may not be as relevant to non-
GUIs.– Results in false positives in missed problems,
especially when experts are not part of target audience.
Cognitive Walkthrough
Pros– Quick and easy.
Cons– Results in false positives and missed problems
when evaluator is different from target audience.
Experimental Simulation
Pros– Still fairly precise.– More realistic than in-lab experiment.
Cons (same as lab exp.)– Short duration of a lab experiment may not be
enough to allow users to become accustomed to an app.
– Not a natural setting – interaction may not be normal.
GOMS
Pros– Predict human performance before committing to a
specific design in code or running user studies– Many studies have validated the model (it works)
Cons– Assumes error-free, skilled user behavior– No formal recipe for how to perform analysis– Significant time investment
Computer Simulation
Creating a complete & closed system that models the operation of the concrete system without users.
Example:– geophysical process going on in connection with the
eruption of Mount St. Helens– Simulation of pilot during plane crash
Computer Simulation
Pros– Supposedly high in realism (depends on accuracy of
data/system replication)
Cons– Low in precision & generalizability
Formal Theory– This is a dissertation
Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest.
Pros– Relatively generalizable
Cons– Not realistic or precise
How to chose a method?
Stage of study Pros & cons
– Realism– Precision– Generalizability– Time & cost
Researcher expertise Metrics Qualitative vs. quantitative Research perspective
Methods
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Early Stage – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Iterative & Summative Stages – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Realism – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Precision – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Generalizability – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Time & Cost – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Researcher Perspective – Whoever is doing will choose whatever they want
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
Metrics: examples
Traditional GUIs:– efficiency (time to complete task)– accuracy (# of errors)– simplicity
Peripheral Displays:– awareness (recall)– distraction (dual-task behavior)– aesthetics