35
Formative Evaluation
Formative Evaluation
Agenda Evaluation overview
Formative evaluation
Observing Users
Asking Users
Summary
A simple product design model
Evaluate
(Re)Design
Identify needs/ establish
requirements
Build a prototype
Final product
Two main types of evaluation
Formative evaluation is done at different stages of development to check that the product meets users’ needs.
Summative evaluation assesses the quality of a finished product.
Development Stages
Early Design Brainstorm different representations
Choose a representation Rough out interface style
Task centered walkthrough and redesign
Fine tune interface, screen design Heuristic evaluation and redesign
Usability testing and redesign
Limited field test
Alpha/Beta tests
Low Fidelity paper prototypes Medium fidelity prototypes High fidelity prototypes Working systems
Late Design
Formative Evaluation
Summative Evaluation
Evaluation paradigm
Any kind of evaluation is guided explicitly or implicitly by a set of beliefs, which are often underpinned by theory. These beliefs and the methods associated with them are known as an ‘evaluation paradigm’
Four evaluation paradigms
‘quick and dirty’ field studies predictive evaluation usability testing
Overview of techniques Observing users Asking users opinions
Asking experts opinions
Testing users’ performance
Modeling users’ task performance
Evaluation approaches and methods
Method Predictive Evaluation
Field Study Usability Testing
Observing users X X
Asking users X X
Asking experts X X
Testing users X
Modeling users X
Observing Users
What is user observation? A usability method used to document implicit user
feedback to optimize a user interface design and learn from participants who undergo a Usability Test
Need to be conducted as early as and as often as possible
Types of observation Direct observation Direct User Observation: with the observer present in the
room taking notes throughout the process in order to maximize its effectiveness.
Indirect observation Indirect User Observation: audio or video recording with
digital prototypes or wireframes.
Direct observation Direct User Observation allows the observer to view
users' behavior up close and to pay more attention to key areas of interest.
Continuous Monitoring Observing subjects and recording (either manually,
electronically, or both) as much of their behavior as possible.
Often used in organizational settings, such as evaluating performance.
Hawthorne Effect states that workers react to the attention they are getting from the researchers and in turn, productivity increases.
Examples education: watching teacher-student interactions nutrition: researchers record how much an individual eats
CM is relatively easy but a time consuming endeavor.
Think Aloud User observed performing task User asked to describe what he is doing and why, what he
thinks is happening etc. Advantages Simplicity- requires litter expertise Can provide useful insight Can show how system is actually use
Disadvantages Subjective Selective Act of describing may alter task performance
Post-task walkthroughs Transcript played back to participant for comment Immediately : fresh in mind Delayed : evaluator has time to identify questions
Useful to identify reasons for actions and alternatives considered
Necessary in cases where think aloud is not possible
Indirect observation Indirect User Observation allows for nuances to be
recorded and remarked upon which might have been missed during direct user observation
Behavior Trace Studies Behavior trace studies involve findings things people leave
behind and interpreting what they mean. The University of Arizona Garbage Project Anthropologists and students dug through household garbage
to find out about such things as food preferences, waste behavior, and alcohol consumption.
in unobtrusive research individuals do not know they are being studied. How would you feel about someone going through your garbage?
Surprisingly, Tucson residents supported the research as long as their identities were kept confidential.
Trace studies may yield enormous data.
Disguised field observations Researcher pretends to join or actually is a member of a group and records
data secretly. Roles:
Complete-participant: for instance, if you are a member of a sorority and study female conflict within sororities
Observer-participant: only participate casually in the group while collecting observations. In this case, any contact with group members is by acquaintance only.
Participant-observer: develop an identity with the group members but do not engage in important group activities e.g. joining a cult but not participating in any of their important rituals (such as
sacraficing animals).
Ethical problem Deception: participant-observers have the most problems The sensitivity of the topic and the degree of confidentiality are important issues
Disguised field experiments are likely to yield reliable data but the ethical dilemmas are a trade-off.
Eye tracking Head or desk mounted equipment tracks the position of
the eye Eye movement reflects the amount of cognitive
processing in a display requires Measurements includes Fixation: eye maintains stable position. Number and duration
indicate level of difficulty with display Saccades: rapid eye movement from one point of interest to
another Scan paths: moving straight to a target with a short fixation at
the target is optimal
Physiological measurements Emotional responses linked to physical changes These may help determine a user’s reaction to an
interface Measurements include: Heart activity, including blood pressure, volume and pulse Activity of sweat glands: galvanic skin response(GSR) Electrical activity in muscle: electromyogram (EMG) Electrical activity in brain: electroencephalogram (EEG)
Some difficulty in interpreting these physiological responses
Pros & Cons: Observations
Advantages Disadvantages
Provides actual behavior rather than reports of what users say
Can avoid biasing factors caused by interviewer
Can avoid biasing factors caused by structured questionnaire
Examine only behavior and physical personal characteristics
Can be time consuming and costly
Interpretation of data may be a problem
Possible invasion of privacy
Asking Users Interviewer-administrated Face to face Telephone
Self-administrated By post E-mail/Internet
P McKeown/ V Bremer/V.Prikazsky
Interview-administered questionnaire Advantages: Participation by illiterate people Clarification of ambiguity Quick answers
Disadvantages: Interviewer bias Needs more resources Only short questionnaires possible
Especially on telephone
Difficult for sensitive issues
P McKeown/ V Bremer/V.Prikazsky
Self-administered questionnaire Advantages: Cheap and easy to administer Preserves confidentiality Completed at respondent's convenience No influence by interviewer
Disadvantages: Low response rate Questions can be misunderstood No control by interviewer
P McKeown/ V Bremer/V.Prikazsky
Interviews
Source: www.id-book.com
Unstructured Not directed by a script. Rich but not replicable
Structured Tightly scripted, often like a questionnaire. Have predetermined questions No exploration of individual attitudes Replicable but may lack richness
Semi-structured Guided by a script but Interesting issues can be explored in more depth. Provides good balance between richness and replicability.
Things to avoid!
Source: www.id-book.com
Long questions
Compound sentences - split into two
Jargon & language that the interviewee may not understand Use user’s word
Leading questions that make assumptions e.g., why do you like …?
Unconscious biases e.g., gender stereotypes
What is a questionnaire? An instrument (form) to collect answers to questions collect factual data
gathers information or measures
A series of written questions/items in a fixed, rational
order
Once delivered, there are no chances to change!
P McKeown/ V Bremer/V.Prikazsky
Before starting to design a questionnaire
Risk factors for being bitten by stray dogs
Study protocol
Prepared by Snoopy
Menorca, 20 September 2005
Write a study
protocol!
P McKeown/ V Bremer/V.Prikazsky
Stages in designing a questionnaire (1) Planning the study: Decide on goals Identify risk factors for getting bitten by dogs
Know the subject Literature, experts on dog bites
Formulate a hypothesis Postmen more likely to get bitten by dogs than the normal
population
Define information needed to test hypothesis Occupation, owning dog, outdoor activities, attitude towards
dogs
P McKeown/ V Bremer/V.Prikazsky
Stages in designing a questionnaire (2) Determine study population:
Know the respondents Occupation Special sensitivities Education Ethnic Language
⇒Questionnaire needs to be adapted to your population, not the opposite!
P McKeown/ V Bremer/V.Prikazsky
Stages in designing a questionnaire (3) Design questions: Content of the questions
Format of the questions
Presentation and layout
Coding schedule (if appropriate)
Pilot and refine questionnaire
P McKeown/ V Bremer/V.Prikazsky
Examples of Questionnaire
Source: www.id-book.com
SUMI: Software Usability Measurement Inventory
MUMMS: Measuring the Usability of Multi-Media Systems
QUIS: Questionnaire for User Interaction Satisfaction
PUTQ: Purdue Usability Testing Questionnaire
Pros & Cons: Interview & Survey
Advantages Disadvantages
Addresses users’ opinions and understanding of interfaces
Can be applied to users and designers
Questions can be tailored to the individual
Can be used for large group of users
Rating scales lead to quantitative results
Low response rate
Irresponsible responses
Possible interviewer bias
Possible response bias
Analysis can be complicated and lengthy
Interviews can be time-consuming
Which one to choose? Mix according to the situation and objective
Observation – Interview - Questionnaire Interview - Questionnaire – Focus Group Interview Interview - Focus Group Interview – Questionnaire Questionnaire – Interview – Questionnaire etc.
