Find out what they like, and how they like it,
And let ‘em have it just that way.
Fats WallerUsability Testing 101
Gina Pearson, Web Manager
January 29, 2003
Today’s Goals
Primary purpose is to learn how ERS is integrating usability testing into its Web design & development process
Review why user-centered design and testing (usability engineering) are important
Explain the usability testing process at ERS
Today’s Goals
You will learn how to:
Assemble your test team
Recruit & select test participants
Formulate a test plan
Create testing tasks & scenarios
Facilitate a test session
Test without a lab
Record, compile & analyze results
Communicate results effectively
Why Is Usability Important?
Ballot Usability
What should have been addressed?
– Font size
– Lighting
– Text vs. background color
– Instructions
– Layout
– Act of voting (punching ballot card)
Ballot Usability
Costs
$20,000 to run usability tests addressing
Layout
Multiple votes
Dimpled ballots
About $1 billion spent by both candidates to try to get elected
Failure of Web Sites
Users could find information only 42% of time (15 large commercial sites - Spool study)
Sixty-two percent of web shoppers gave up looking for an item (Zona study)
Only 51% of sites complied with simple Web usability principles (Forrester study of 20 major sites)
User-Centered Design
Philosophical & Practical Approach
User-centric - not developer, designer nor organization-centric
Who are the users?
What information do they want?
Based on research and data not opinions
Testable & verifiable
The Traditional Development Process
User-Centered Design
A methodical approach to producing a web site or any user interface. It is a practical and systematic way to deliver a product that works for users.
Test early & often
Build a little, test a little, fix a little
At ERS, we are incorporating usability testing & other user assessment techniques as a regular part of our web design & development process
User-Centered Design
Types of user assessment techniquesInterviews
Ethnographic observation / field research
Focus groups
User personas
Expert (heuristic) reviews
Usability testing
Customer satisfaction surveys
Customer email trend analysis
Search engine query analysis
What else?
Myths About Usability Testing
Usability testing:
Is time consuming, expensive & otherwise resource intensive
Typically involves hiring a specialized consulting company
Requires a formal testing lab
Must be videotaped
Always involves writing a detailed analysis and report of results
Diagnostic User Testing Find out what is and is not working well on the site. In a diagnostic usability test, you usually want to answer questions like these:
Do users complete a task successfully?
If so, how fast do they do each task? Is that fast enough to satisfy them?
What paths do they take in trying? Do those paths seem efficient enough to them?
Where do they stumble?— What problems do they have?— Where do they get confused?
What words or paths are they looking for that are not now on the site?
Methods
Assemble test team Administrator Facilitator Observers / recorders Video recordings operator (optional)
Methods
Define goals & scope of what you are testing
Recruit & compensate participants Develop testing tasks & script Set up testing environment Draft participation agreement Rehearse & refine test protocol Conduct tests Analyze & report out results
Methods
Test Environment
Methods
How many participants are needed for a usability test? It depends! Typical range is from 5 -12 users in each test
Our approach: 3 users per test but conduct several rounds of testing throughout an application or feature’s development
If you have different potential user groups (for example, small business owners, unemployed, blue-collar workers), try to include representatives of all these groups
Methods
Participant recruitment & compensation Typically time and labor intensive - consider
outsourcing (average $200 per participant) Go through professional networks & contacts
for volunteers Conduct testing at conferences & other
events with volunteer representatives from your target audiences
Always give volunteers a small token of your appreciation (t-shirt, mug, etc)
Methods
Task construction: where do ideas for tasks come from? Concerns about information architecture,
design, terminology, functionality, etc that have surfaced among the product development team
Trend analysis of emails from site customers Search engine query analysis Trend analysis of call center inquiries What else?
Methods
Constructing usability testing tasksFind or scavenger hunt tasksVerb-based tasksScenario-based tasks
Methods
Conducting the Test (about 2 hours) Pre-test briefing for observers Greet participant & administer participation
agreement Introduce recorder & observers Explain testing process to participant Administer pre-test interview & activities (if any) Facilitate testing tasks Administer post-test interview questions Involve observers in general Q&A session at the end After participant leaves, debrief recorder &
observers
Methods
Developing the testing script & facilitating the test
3 parts of a usability test: Pre-test interview Testing tasks (“think out loud” methodology) Post-test interview
See another sample script at: http://www.sensible.com/Downloads/script.
doc
Methods
Characteristics of a Good Facilitator Quick learner Ability to establish instant rapport Empathic “people person” Good listening skills Patient Flexible & adaptable Long & focused attention span Impartial: not wedded to a particular design
Methods
Analyzing & reporting the test results using the KJ method Assemble your testing team immediately after
the last test for a brainstorming session. Ask each participant to write down each major
observed problem on an index card or sticky note.
Put all index cards or sticky notes on a large wall or board, no discussions!
Read each others cards silently, add additional problems.
Methods
Analyzing & reporting the test results using the KJ method Sort problems by area. Eliminate duplicates if
total agreement. Name each group of problem areas. Vote for most important problems, each
participant has ten votes. Assign one team member the task of writing up
the results of brainstorming session. Disseminate results of brainstorming session to
development team via email
For Further Information
Contact: Gina Pearson, Web Manager
694-5119 Brenda Powell, Web Analytics Manager
694-5134
For Further Information
Don't Make Me Think! A Common Sense Approach to Web Usability
Steve KrugThe book every manager should read!
Handbook of Usability Testing: How to Plan, Design & Conduct Effective Tests
By Jeffrey RubinPractical, step-by-step guidelines, requires no engineering or
human factors training.
A Practical Guide to Usability TestingBy Joseph Dumas & Janice RedishPremier text in the field.
For Further Information
Jakob Nielsen (www.useit.com)Why You Only Need to Test With 5 Users
http://www.useit.com/alertbox/20000319.html
Success Rate: The Simplest Usability Metric
http://www.useit.com/alertbox/20010218.html
Usability Metrics
http://www.useit.com/alertbox/20010121.html
For Further Information
Jakob Nielsen (www.useit.com)Cost of User Testing a Web Site
http://www.useit.com/alertbox/980503.html
When to Outsource the Recruiting of Test Users
http://www.useit.com/alertbox/outsource_recruiting.html
For Further Information
Jared Spool (www.uie.com)Using Paper Prototyping to Manage Risks
http://world.std.com/~uieweb/paper.htm
Usability Labs: Our Take
http://world.std.com/~uieweb/labs.htm
Eight Is More Than Enough
http://world.std.com/%7Euieweb/eight.htm
For Further Information
Interaction Architect13 Common Objections Against User
Requirements Analysis & Why You Should Not Believe Themhttp://www.interactionarchitect.com/articles/
article20000609b.htm
Observation Methods & Tips for Usability Testinghttp://www.interactionarchitect.com/knowledge/
article19991212shd.htm
For Further Information
DialogDesign (www.dialogdesign.dk)
Ethics in Human Computer Interaction
http://www.dialogdesign.dk/chi2001.html
Usability Test of www.TowerRecords.com
http://www.dialogdesign.dk/utestreports.html#sample
For Further Information
Usability.gov Why Should I Do Iterative Usability Testing? How Can I Encourage People in My
Organization to Conduct Usability Engineering and Testing?
Do I Need a Lab to Do Usability Testing? How Many Participants Are Needed for a
Usability Test? How Much Does It Cost to Do Usability
Testing?
For Further Information
Training Opportunities National Cancer Institute
Usability Seminar Series & Education Training Program, http://usability.gov/events/seminars.html
Society for Technical Communication WDC Chapter, www.stcwdc.org
User Interface Engineering Human Factors International User Experience Conferences (Nielsen Norman
Group)
