Usability TestingI. Omaima Al-Matrafi

Why Should You Care About Usability?•Have you ever…

▫ gotten lost in a Web site?▫ left a site without finding the information

you wanted?▫ waited too long for a page to download?▫ gone to a site you can’t view or read?▫ visited a site with outdated information?

• Do you want people to visit and return to your site?

What is usability?•“A function of particular users performing

particular tasks in a particular environment” (Smith et al. 68)

•The “people who use the product can do so quickly and easily to accomplish their own tasks” (Dumas and Redish 4)

ISO Definition (9241-11) for Usability

•“...the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”

Definitions (cont.) Usability is a combination of factors that

affect the user's experience with the product or system, including: ▫Ease of learning

How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?

▫Efficiency of use Once an experienced user has learned to use the

system, how fast can he or she accomplish tasks?

Definitions (cont.)• Memorability

▫ If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?

• Error frequency and severity ▫ How often do users make errors while using the system,

how serious are these errors, and how do users recover from these errors?

• Subjective satisfaction▫ How much does the user like using the system?▫ Confusing, frustration and satisfaction

Usability•Is not functionality

▫Functionality is how the system works•Is not beta testing

▫Beta testing is unfocused and at end of process

•Is not market research▫Market research is about customer demands

Types of Usability Problems• Product doesn’t match job or task• Poor organization/layout• Unexpected occurrence of events• Product not self-evident• Requires recall rather than recognition• Inconsistent screens, messages, terminology• Design is inefficient• Cluttered or unattractive design• No feedback or poor feedback about status or

errors• No exit or undo• Help or documentation is not helpful

Usability Problem Example: Inconsistent

Usability Problem Example: Poor Organization/Layout

Usability Problem Example: Product Not Self-evident

What is Usability Testing?•An empirical study of a product’s usability

by observing actual users do real tasks with the product

•The primary goal is to improve the usability of the product

•5 characteristic:Real usersReal tasksSpecific usability goals/concernsObserving and recording the testingData analysis

Planning: Define the Purpose•Look at goals of your organization.• Which goals is the Web suited to meet?• Prioritize the goals for your site.

▫ Have one primary goal.▫ Optionally have secondary goals.

• Example goals:▫ Provide a positive image of NASA / your

organization▫ Provide information/function that the user

needs

Five Steps for test process•Step 1: User Analysis & Profiles•Step 2: Decide what to Test•Step 3: Preparing for the Testing•Step 4: Conducting the test•Step 5: Analyzing the Data

Step 1: User Analysis & Profiles Knowing Your Users

•What are your users’ characteristics?▫ Age, computer literacy, domain knowledge,

access methods, browsers, work environments, handicaps, etc.

• Collect this information through▫ Surveys / Questionnaires▫ Visits to their environment

▫ A typical test includes 6-12 participants

Step 2: Decide what to Test1. Choose an overall purpose

Example: How useable is our new website?

2. Determine objectives or what you are testing for. Examples:

Does our search engine provide usable results in the first 5 links returned?

Are search results clear to the users?3. Choose type of test:

1. Performance: Can they do it?2. Understandability: Can they understand

it?3. Read-and-locate: Can they find it?

Step 2: Decide what to Test con.4. Select tasks:

▫ Consider tasks with a high chance of user failure (complex tasks, one-of-a-kind tasks, highly abstract or technical tasks)

▫ Consider tasks with a high cost of user failure (tasks that require support, like help or support calls, to complete; tasks where data could be damaged or lost)

▫ Consider: First impressions (look and feel) First tasks Tasks most performed Critical tasks Specific problem areas New task for the product

Step 2: Decide what to Test con.

5. Select performance objectives (should be individualized for each task)▫ Time: How long to complete tasks, to find things, to performance

procedures▫ Error/Success: user errors, attempts to do/find something,

numbers of times section re-read, if the task was completed successfully

• Five attributes of usability (Nielsen, 1994): Learnability: system is easy to learn so users can get started quickly Efficiency: system should be easy to use, resulting in high

productivity Memorability: system should be easy to remember Errors: system should have low error rate and allow error recovery Satisfaction: system should be pleasant to use

The goal and the concerns of the testExample:• General goal: users can use this program to read the documents

easily and quickly.• Quantitative usability goals: users will be able to find the menu

choice they want in less than 30 second with no more than two wrong choices and Know the function of each icon from the first time they see it.

• General concern for this test: Can users find the correct menu choices and icons as they need them (ease of learn)

• Specific concern of this test:1-unrestanding how can open a PDF file to read it.2- Accessing particular information easy and quickly by searching

the document. 3- Increase the ability of enhancing the vision by zooming the page.4- The ability of copying a specific part from this document.5- The ability of making a snapshot for any part of document

Planning: User Tasks•Why will users come to your site?

▫ What’s in it for them?▫ Best way to find out is to talk to them!▫ Create use scenarios to drive your design.

• Example user tasks: Find information about a particular product Learn about newest projects Find out about future presentations Download copy of a document Contact organization employees Provide feedback

Planning: Example ScenarioScenarios try to capture a moment in time

for the user. A sample scenario might read:

John is working at XCompto make their systems more usable. He remembers that someone at MyComp is involved with usability engineering and wants to contact that person, either by phone or email, for more information on doing usability engineering at a government agency.

Step 3: Preparing for the Testing1. Choose order of tasks: start easy, go sequential, or be

random2. Create written test materials:

Task list for users Written welcome speech/ Intro to be read to user Consent forms Pre-task and post task questionnaires & interview questions Other materials

3. Recruit participants & determine “payment”4. Define team member’s roles:

Facilitator/Briefer (necessary): Often only team member to interact with users

Observation recorder/note taker (necessary) Camera operator (optional) Help desk operator (optional) Test administer (optional)

Step 3: Preparing for the Testing Cont.5. Create written test plan6. Practice: conduct walkthroughs of the

testing and if possible pilot test (the pilot test users could even be a team member)

7. Prepare test environment (day of test)

Pre-test questionnaire

•Ensure that participant meets user profile•Gather additional detail about the

participants’ knowledge and experience

Post-task questionnaire

•Asks questions about a task as soon as it is completed

•Captures feedback when it is fresh•Can help measure change in attitude

toward the task•Keep it short

Description of the test• Tasks and their scenarios• The path for each task• Measuring usability• Typical Criteria for measuring usability

• Consent Form• Methods: Details how you will run the test

▫ Pre-Test ( user profile ), Post-Task questionnaires, Post test questioner, and Observation.• Task resources and Procedure:

▫ time, HW, SW and procedure• Preparing the test• Test Team Members: define what each person will do

Measure Excellent Acceptable UnacceptableTask1: open fileTime for task <15 sec 15-20 sec More than 20 secS = Selection error 0 1 More than 1C= confusing 0 0 1 or more

Step 4: Conducting the test1. Greet & Brief participant:

a. Read/say welcomeb. Emphasize that you are not testing them, but the

product and that they should act as natural as possible

c. Explain think-aloud protocol (if using)d. Emphasize how user tells you she has completed a

taske. Stress that the testing is “anonymous”

2. Be unbiased (especially the Facilitator/Briefer)

3. Intervene carefully (avoid as much as possible)

4. Observe and record data5. Debrief user

Step 5: Analyzing the Data1. Collate data into findings:

a. Choose an approach: Top-down approach: predetermine categories of

findings (like navigation, design, terminology) and go through data looking for “hits”

Bottom-up approach: put each observation on a sticky note/note card, sort into categories and label categories

b. Determine time and errors/success Examine findings for each user, user profile, and

task Use analysis techniques such as statistics (even

averages help)

Your opinion on the taskChoices Very easy

1 2 3 4Very difficult

5No. answers 1 1 1 1 1Percentage .20 .2

0.20

.20

.20

Did you finish the taskChoices Yes No

No. answers 4 1Percentage .80 .20

Do you face any problem during executing the taskChoices Yes No

No. answers 2 3Percentage .40 .60

What is the kind of difficultiesChoices Difficult in

navigating between menus

Ambiguity icons

The path to

complete the task is long

others

No. answers 0 1 0 1Percentage .0 .5 .0 .5

Your opinion about the program

Choices Very bad1 2 3 4

Very good5

No. answers

0 0 2 2 1

Percentage 0 0 0.40 0.40 0.20

Screen arrangement

Choices Not logical1 2 3 4

logically5

No. answers

0 0 2 3 0

Percentage 0 0 0.40 0.60 0

Icons designChoices ambiguity

1 2 3 4understandab

le5

No. answers

0 1 2 2 0

Percentage 0 0.20 0.40 .40 0

Easy of finishing taskChoices Very easy

1 2 3 4Very difficult

5No.

answers1 3 1 0 0

Percentage 0.20 0.60 0.20 0 0

Learn of the programChoices Very easy

1 2 3 4Very difficult

5No.

answers0 2 3 0 0

Percentage 0 0.40 0.60 0 0

Step 5: Analyzing the Data con.

1. Analyze data: a. Determine cause of problemsb. Determine scope/severity of problemsc. Make recommendations/changes

2. Report Findings

Usability Related Web Siteshttp://usability.gsfc.nasa.gov

Goddard Usability Engineering Centerhttp://www.usability.gov

Usability resources by National Cancer Institutehttp://www.uie.com

User Interface Engineering (Jared Spool)http://www.UPAssoc.org

The Usability Professionals’ Associationhttp://www.useit.com/alertbox

The Alertbox: Current Issues in Web Usability (Jakob Nielsen)

Book: A Practical Guide to Usability Testing (Dumas & Redish)