Introduction to Team-based Usability Testing

“As companies design more for usability and understanding, they will discover a competitive edge, for these principles save customers time

and money while increasing morale.” - Donald Norman

Objectives

Usability overview Setting objectives Preparing the test plan Conducting the test

Usability

Usability “means that the people who use the product can do so quickly and easily to accomplish their own tasks.”

- Redish and Dumas, A Practical Guide to Usability Testing

Usability

Is not functionality- Functionality is how the system works

Is not beta testing- Beta testing is unfocused and at end of process

Is not market research- Market research is about customer demands

When to test

Design phase- surveys and focus groups- low fidelity prototype testing- high fidelity prototype testing- heuristic evaluation

Development phase- usability evaluation

Advantages of a usability lab

Lab is helpful for tightly controlling the variables

Lab helps capture low-level details - keyboard monitoring- videotaping for later analysis- eye movement tracking

Labs allow less intrusive observations

Disadvantages of a usability lab

Labs are intimidating to users Labs are rarely portable so they cannot

go into the users’ environment Sometimes low-level information is not

as important as making high-level design decisions

Labs are expensive

How long will the test take?

How much usability testing has already occurred?- Have users been profiled?- Has a task analysis occurred?- Have usability objectives been set?

How complex is the product? How much of the product are you going

to test?

How long will the test take?

How many test participants do you need to get the information you want?

How much training or prior experience does the test team have with the product?

How many different usability objectives do you need to test?

Who should run the test?

The most successful approach to usability testing is a cross-functional, team-based approach- People bring varying expertise to the team- All groups benefit from experience with

users- Interpretation of challenges and design

decisions are more creative with teams

Who should run the test?

Team-based testing provides a great way to get support involved early

Developers learn to “hear” user questions when they design

Writers and trainers can help design communication into the product

Human factors experts and user interface designers can provide helpful principles about users and design

Who should run the test?

More people can be involved in planning the test than in conducting the test

Target 3-5 people for conducting the test Sometimes best if the developers are not

actively involved in running the test, although they should observe if possible

How do you ensure success?

You need the buy-in of upper management if the results are to be applied

You need the cooperation of development to ensure they don’t feel attacked

You need the support of marketing, sales, and product management for funding

You need the support of your immediate manager to spend time testing

How do you ensure success?

Pick a manageable portion of the product to test

Select a high-profile project Select an area that will produce

significant results Advertise your successes (become part

of the sales story)

Stages of a usability test

Setting objectives Creating the test plan Creating questionnaires Selecting test participants Training the team Conducting the pilot test Conducting the tests

Usability objectives

Created during user/task analysis and product scoping

Must be measurable Should indicate:

- Type of user- Task to be performed- Specific performance criteria

Defining usability objectives

Five attributes of usability (Nielsen, 1994):- Learnability: system is easy to learn so users can

get started quickly- Efficiency: system should be easy to use, resulting

in high productivity- Memorability: system should be easy to remember- Errors: system should have low error rate and allow

error recovery- Satisfaction: system should be pleasant to use

Examples

Novice PC users can change the ink cartridge in their color inkjet printer in less than 5 minutes using the user manual

Users will be able to install and configure the default application in less than 15 minutes

Users should rate the product as either “easy to use” or “very easy to use” (4 or higher on a 5 point scale)

Considerations

A single usability objective may result in multiple design decisions

Emphasizes importance of having cross-functional team

Focus on objective that are most important to the success of the product

Test plan

Explains what you are testing and how you will conduct the test

Do not skip this step- Important to have a plan that everyone can

agree to and understand- Helps communicate your intentions to

others in the organization Do not overdo this step

Test plan

Purpose- Why are you conducting the test?

Problem Statement/Test Objectives- What usability objectives are you testing?

User Profile- Who are your users and their defining

characteristics?

Test plan

Method- Details how you will run the test

Task List- List of the tasks to be completed, projected

times for completion, required state of system to test tasks

Test Environment/Equipment- List what you need to conduct the test

Test plan

Test Team Roles- Define what each person will do

Evaluation Measures- What data you will collect through the test

Conclusion- What will happen after the test - reports,

design meetings, presentations

Pre-test questionnaire

Ensure that participant meets user profile Gather additional detail about the

participants’ knowledge and experience

Post-task questionnaire

Asks questions about a task as soon as it is completed

Captures feedback when it is fresh Can help measure change in attitude

toward the task Keep it short

Post-test questionnaire

Measure satisfaction and reaction to the product and the overall experience

Ask participants to elaborate on a few critical areas/issues

Training the team

Make sure everyone understands the usability objectives and the testing procedure

Define everyone’s roles and discuss how they should behave during the test

Conducting a pilot test

Conduct a practice test using the complete team, all prepared materials, and the actual facilities

Make sure at least one person observes the test to make recommendations

Refine any of the materials or facilities as necessary

Selecting test participants

Number of participants depends on the number of user groups (differentiating characteristics)

Target 3-5 users from each user group Best if users come from the real user

population rather than internally

Observers

Not actively involved in conducting the test, but can benefit from observing

Must not comment or react to the participant’s actions or comments

Should be separated from the actual room where the testing is going on

Should be removed if they cannot avoid feeling defensive or being intrusive

Data logger

Be familiar with any data logging forms Establish standard codes ahead of time

for recording events and comments Remain open to unforeseen directions Record user comments that summarize

or emphasize good and bad points Keep track of time

Facilitator

Main contact with the participants Conducts the briefing and debriefing Only person to talk to the participants

during the test

Briefing

Thank them for participating in the product evaluation

Assure users that they are not being tested; instead, they are helping you evaluate the product

Explain to them what will happen during the test; they will be especially interested in any recording devices and observers

Briefing

Ask them to sign a consent form and a non-disclosure agreement

Encourage them to think out loud so you can record their actions and the reasons behind their actions

Make sure you explain what they should do if they have questions or experience problems

During the test

Be sensitive to excessive frustration Be ready to handle unplanned situations Remind participants to talk out loud Look and listen for the unexpected Avoid intervening unless necessary; try

using questions to redirect rather than take over as the expert

Debriefing

Get users to elaborate on significant events or comments

Ask for reasons to specific behaviors from the test

Watch the video tape with them and ask them to explain what was going on in their mind

Summary

Build a cross-functional team to:- take advantage of different skills- get “buy in” from different areas- build awareness of usability testing benefits

When you test:- Set measurable objectives- Test with the right people- Be prepared for the test- Share your results

Questions?

Scott DeLoach

Founding Partner, User First Services, Inc.

Certified RoboHELP Instructor and Consultant

404.520.0003

scott@userfirst.net

For more information about this presentation and other training seminars, visit www.userfirst.net. This presentation is at www.userfirst.net/teamusability.ppt.