Usability Testing Bootcamp

How to Plan and Moderate a Usability Test

Dr David Travis, Userfocus ltd

#uxlx #ut

Who am I?

?

I currently run a consultancy firm called Userfocus. We work with clients like Yahoo!, the World Health Organisation and the UK Tax Office. Yes, even they are using these techniques to make sure you pay what you owe.

You can still buy my book on amazon!

What you’ll learn

Design and plan usability tests.

Recruit the right test participants.

Apply usability testing checklists to make sure your test runs without a hitch.

Develop test tasks that are relevant to your customers and your business.

Organise and prepare usability testing sessions.

Moderate 'thinking aloud' usability tests in an unbiased and balanced manner.

Code, collect and interpret behavioural data.

Interview your partner

Where do you work and what do you do?

What one thing do you already know about usability testing?

What do you want to be able to do differently as a result of the training?


What so special about a usability test?

Johansson, P., Halls, L., Sikström, S & Olsson, A. (2005) Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task. Science, 310: 116-119.

When people are asked to justify their choices, they do so readily — even when they never made the choice. We need to be aware of this confabulation when running any kind of user research.

Lichfield Cherokee 3 Man Tent Red

Lichfield Crow 2 Man Tent Denim/Navy

Lichfield Dakota 3 Man Tent Denim/SkyBlue

Confidence Holiday 6 Man 2 Room Tent









Felfernig, A. et al. (2007) Persuasive Recommendation: Serial Position Effects in Knowledge-Based Recommender Systems. Lecture Notes in Computer Science, 4744: 283-294.

The first tent presented was chosen 2.5 times more often than any other. Yet people ‘explained’ their choices using logic and rationality.

“The accuracy of subjective reports is so poor as to suggest that any introspective access that may exist is not sufficient to produce generally correct or reliable reports.”

- Nisbett, R.E., and Wilson, T.D. “Telling more than we can know: Verbal reports on mental processes.” Psychological Review, 84 (1977), pp. 231-259.

What is a usability test?

Behaviour is observed — what users do (rather than what users say they do).

Participants perform real tasks — tasks that are meaningful to them and to the business.

Involves real users — not just someone at the next desk.

Observations and data are recorded and the data are analysed.

Quick Activity: Usability Blankety Blank

If we’re going to test for usability, we better define it first.

Listen to the definitions of usability and call out the missing words.

“Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks. This definition rests on four points: (1) Usability means focusing on USERS; (2) people use products to be productive; (3) users are busy people trying to accomplish tasks; and (4) users decide when a product is easy to use.”

- Janice (Ginny) Redish and Joseph Dumas (1999). A Practical Guide to Usability Testing.

“Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks. This definition rests on four points: (1) Usability means focusing on USERS; (2) people use products to be productive; (3) users are busy people trying to accomplish tasks; and (4) users decide when a product is easy to use.”

- Janice (Ginny) Redish and Joseph Dumas (1999). A Practical Guide to Usability Testing.

“Usability is a term used to denote the ease with which people can employ a particular tool or other human-made object in order to achieve a particular GOAL.”

- Wikipedia.

“Usability is a term used to denote the ease with which people can employ a particular tool or other human-made object in order to achieve a particular GOAL.”

- Wikipedia.

“The capability in human functional terms to be used easily and EFFECTIVELY by the specified range of users, given specified training and user support, to fulfil the specified range of tasks, within the specified range of environmental scenarios.”

- Brian Shackel (1991). ‘Usability—context, framework, definition, design and evaluation’. In B. Shackel and S. J. Richardson (eds) Human factors for informatics usability.

“The capability in human functional terms to be used easily and EFFECTIVELY by the specified range of users, given specified training and user support, to fulfil the specified range of tasks, within the specified range of environmental scenarios.”

- Brian Shackel (1991). ‘Usability—context, framework, definition, design and evaluation’. In B. Shackel and S. J. Richardson (eds) Human factors for informatics usability.

“At first, using words that all started with “E” was a word game, but I was also looking for a way to make the dimensions of usability easy to remember and the 5Es were born. I decided on: Effective; EFFICIENT; Engaging; Error Tolerant; Easy to Learn.”

- Whitney Quesenbery (2004) ‘Balancing the 5Es: Usability’. Cutter IT Journal, Feb, 2004.

“At first, using words that all started with “E” was a word game, but I was also looking for a way to make the dimensions of usability easy to remember and the 5Es were born. I decided on: Effective; EFFICIENT; Engaging; Error Tolerant; Easy to Learn.”

- Whitney Quesenbery (2004) ‘Balancing the 5Es: Usability’. Cutter IT Journal, Feb, 2004.

“It is important to realize that usability is not a single, one-dimensional property of a user interface. Usability has multiple components and is traditionally associated with these five usability attributes: learnability, efficiency, memorability, errors, SATISFACTION.”

- Jakob Nielsen (1993), Usability Engineering.

“It is important to realize that usability is not a single, one-dimensional property of a user interface. Usability has multiple components and is traditionally associated with these five usability attributes: learnability, efficiency, memorability, errors, SATISFACTION.”

- Jakob Nielsen (1993), Usability Engineering.

“Usability is a function of the ease of use (including learnability when relevant) and the acceptability of the product and will determine the actual usage by a particular user for a particular task in a particular CONTEXT.”

- Nigel Bevan, Jurek Kirakowski and Jonathan Maissel (1991) ‘What is Usability?’ Proceedings of the 4th International Conference on HCI, Stuttgart, September 1991.

“Usability is a function of the ease of use (including learnability when relevant) and the acceptability of the product and will determine the actual usage by a particular user for a particular task in a particular CONTEXT.”

- Nigel Bevan, Jurek Kirakowski and Jonathan Maissel (1991) ‘What is Usability?’ Proceedings of the 4th International Conference on HCI, Stuttgart, September 1991.

usersgoals

effectiveness efficiencysatisfactioncontext

So based on these quotations, it seems that usability has something to do with:

• USERS• GOALS• EFFECTIVENESS• EFFICIENCY• SATISFACTION

and

• CONTEXT

tHere’s a definition we can use that brings all of these together…

ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)

“Extent to which a product, system or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” - Effectiveness

The accuracy and completeness with which users achieve specified goals.

- Efficiency

The accuracy and completeness of goals achieved in relation to resources.

- Satisfaction

Freedom from discomfort, and positive attitudes towards the use of the product, system or service.

ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)

“Extent to which a product, system or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” - Effectiveness

The accuracy and completeness with which users achieve specified goals.

- Efficiency

The accuracy and completeness of goals achieved in relation to resources.

- Satisfaction

Freedom from discomfort, and positive attitudes towards the use of the product, system or service.

THIS M

EANS THAT USABIL

ITY IS

NOT JUST

ABOUT MAKIN

G THINGS EASY TO U

SE

1234

With usability, we ask if people can actually use a product.

You don’t need many people to identify a problem that needs to be fixed.

For example, which knob would you use to light the gas under the back right hob?

1234

answer: 4

Usability testing then gives you these kinds of insights to improve designs.

“Testing one user early in the project is better than testing 50 near the end.”

- Steve Krug

So what defines a usability test?

Usability tests don’t require:- A usability lab

- Observers

- A moderator

- Screen recording

- Video recording

All these are ‘nice to have’.

Focusing on these alone puts you at risk of running a ‘cargo cult’ usability test.


A live test

Deconstruct a usability test

Try to deconstruct the usability test.

How would you describe what happened when?

Are there different phases?

Q. Would you like to take part in a usability evaluation?

As a thank-you, you’ll get a free copy of E-Commerce Usability!

YES: Remain standing

NO: Sit down

Q. In the last 6 months, have you bought any electronic goods online, such as a computer, a camera or a mobile phone?


NO: Sit down

Q. Do you use a colour printer at home?


NO: Sit down

Q. Is your printer a laser printer or an ink-jet printer?

LASER: Sit down

INK JET: Remain standing

DON’T KNOW: Remain standing

Q. Have you done any of these activities in the last 2 weeks? Give yourself 1 point for each ‘YES’ answer.- Edited the html code on a web page

- Sold something on eBay

- Created or commented on a blog

- Installed software on your computer

- Signed into your online bank account

- Played an online game

- Downloaded and listened to podcasts

- Shared photographs or videos on web sites like flickr, Twitpic or YouTube

- Added a page to a social bookmarking site like del.icio.us, reddit or Digg

- Subscribed to a syndicated ("RSS") web feed

7 or more: Sit down

6 or fewer: Remain standing

Oh no! Your home printer is broken.

You can spend up to £100 on a new one.

The printer must be wi-fi enabled and it must work with a Mac.

Find the best printer within your budget.

Participant Recruitment ? ? ?

???

Deconstruct the usability test

Identify the test goals

Screen and recruit participants

Identify the test tasks

Greet the participant

Get the video consent form / NDA signed

Demonstrate and practice thinking aloud

Probe with open questions


Planning a usability test

The Five W’s (and one H)

Why are you running the test?

Where will it take place?

When will it take place?

Who will be the test participants?

What system (and what functionality) will you be testing?

How will you collect and analyse the data?

Usability Test Plan Toolkit

The test plan toolkit contains guidance and checklists for running a usability test.

You get an editable version in Microsoft Word when you sign up for refresher training (free!)

!"#$%&'()*+,-*.&/00'01*/*2(/3'&')4*)#()5*6##%(*3#+,-#*)"#*)#()*

! "#$%%!&%'&!()*%+&,-%'!! "#$%%!(.!&/%!0%1&2$%!'%&!&/1&!3,44!)%!&%'&%5!! "#$%%!61$&,+,61.&!6$(0,4%'!! 7$%1&%!61$&,+,61.&!'+$%%.%$!! 8.#1#%!$%+$2,&9%.&!0,$9!! :((;!(2&!1.<!%=2,69%.&!<(2>44!.%%5!?%@#@!$((9A!+(962&%$A!-,5%(!+19%$1B!

!

7*6##%(*3#+,-#*)"#*)#()*

!C$,&%!&1';!'+%.1$,('!!C$,&%!&%'&!641.!1.5!'%.5!&(!'&1;%/(45%$'!0($!$%-,%3!! D.-,&%!()'%$-%$'!

!

8*6##%*3#+,-#*)"#*)#()*

! 7/%+;!$%+$2,&9%.&!,'!(.!'+/%524%!! E,.14,'%!1.5!,''2%!&%'&!641.!!! F%.5!(2&!166(,.&9%.&!&,9%A!916!1.5!-,5%(!+(.'%.&!0($9!&(!61$&,+,61.&!! 7/%+;!&/%!6$(&(&<6%!'266($&'!&/%!'+%.1$,('!G!9(5,0<!'+%.1$,('!,0!.%+%''1$<!! D0!1.<&/,.#!<(2>$%!2',.#!.%%5'!)1&&%$,%'A!#%&!'61$%'!

!

8*9/4*3#+,-#*)"#*)#()*

! H$,.&!'+%.1$,('!(.!,.5%I!+1$5'!0($!%1',%$!/1.54,.#!! H$,.&!&%'&!'+$,6&!! H$,.&!61$&,+,61.&!=2%'&,(..1,$%'!?J!6%$!61$&,+,61.&B!! H$,.&!-,5%(!+(.'%.&!0($9'!?61$&,+,61.&'!91<!0($#%&!&(!)$,.#!&/%!(.%!<(2!'%.&B!! H$,.&!KD.'&$2+&,(.'!0($!()'%$-%$'L!! H$,.&!&/%!'+%.1$,('!(.!1!',.#4%!'/%%&!! M2.!6,4(&!&%'&!3,&/!1!0$,%.54<!4(+14!G!9(5,0<!&%'&!'+$,6&!1.5!'+%.1$,('!1'!.%+%''1$<!!N)&1,.!+1'/!0($!61$&,+,61.&!,.+%.&,-%'!! H$,.&!$%+%,6&'!0($!61$&,+,61.&!,.+%.&,-%'!! 7$%1&%!1!&,9%&1)4%!'/(3,.#!61$&,+,61.&!'+/%524%!G!,.+425%!%1+/!61$&,+,61.&>'!6/(.%!.29)%$!! O%4%6/(.%!&(9($$(3>'!61$&,+,61.&'!1.5!+(.0,$9!1&&%.51.+%!! 891,4!()'%$-%$'!1.5!+(.0,$9!1&&%.51.+%P!1&&1+/!&/%!&%'&!641.!! Q,'1)4%!1.<!'(0&31$%!&/1&!9,#/&!,.&%$$26&!&/%!&%'&!?4,;%!+14%.51$!$%9,.5%$'A!%91,4!14%$&'A!,.'&1.&!9%''1#,.#B!! :((;91$;!&/%!',&%!&(!91;%!,&!%1',%$!0($!<(2!&(!0,.5!52$,.#!&/%!&%'&!! 8.'2$%!<(2!/1-%!1!+(.&1+&!.29)%$!0($!1.<!&%+/.,+14!'266($&!<(2!91<!.%%5!

:"#*;,-0'01*,+*)"#*)#()*

! H2&!KD.'&$2+&,(.'!0($!()'%$-%$'LA!&/%!4,'&!(0!'+%.1$,('!1.5!1!+(6<!(0!&/%!&%'&!641.!,.!&/%!()'%$-1&,(.!$((9!! 7/%+;!&/%!6$(&(&<6%!3($;'!! 7/%+;!&/%!$%+($5,.#!%=2,69%.&!R!'+$%%.!$%+($5,.#!'(0&31$%!3($;'!?125,(!1.5!-,5%(B!! 7/%+;!&/%!-,5%(!1.5!125,(!0%%5'!,.!&/%!()'%$-1&,(.!$((9!

!

<#+,-#*#/$"*./-)'$'./0)*/--'=#(*

! M%S'%&!&/%!3%)!',&%!&(!,&'!5%0124&!'&1&%T!+4%1$!&/%!)$(3'%$!/,'&($<A!5%4%&%!+((;,%'A!4(15!1!)41.;!)$(3'%$!61#%!! O2$.!(00!&/%!'+$%%.!?)2&!;%%6!&/%!+(962&%$!$2..,.#B!! F&1$&!&/%!$%+($5,.#!%=2,69%.&!! M1.5(9,'%!&/%!'+%.1$,(!+1$5'!

!

>"#0*)"#*./-)'$'./0)*/--'=#(*

!U1.1#%!%I6%+&1&,(.'!G!%I641,.!&/%!'%&&,.#!! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!-,5%(!+(.'%.&!0($9!

!

>"#0*)"#*./-)'$'./0)*'(*+'0'("#9*

! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!6('&S&%'&!=2%'&,(..1,$%!! 7/%+;!,0!()'%$-%$'!/1-%!1.<!0(44(3S26!=2%'&,(.'!! H1<!&/%!,.+%.&,-%!1.5!#%&!1!',#.%5!$%+%,6&!! F&(6!&/%!$%+($5,.#!G!.19%!&/%!$%+($5,.#!166$(6$,1&%4<!! H$%61$%!0($!&/%!.%I&!61$&,+,61.&!

!

?)*)"#*#09*,+*#/$"*9/4@(*)#()'01*

! Q,'+2''!&/%!0,.5,.#'!3,&/!()'%$-%$'!G!1#$%%!(.!&/%!&(6!,''2%'!! :1+;26!1.5!'%+2$%!144!$%+($5,.#'!

!

8*9/4*/+)#-*)"#*)#()*

! F%.5!1!&/1.;S<(2!.(&%!&(!%1+/!61$&,+,61.&!!

>')"'0*8*6##%*,+*)"#*)#()*

! H$%'%.&!0,.5,.#'!1.5!1#$%%!.%+%''1$<!+/1.#%'!!

Contents of the Test Plan

Purpose of usability test

Product under test

Participants & responsibilities

Evaluation procedure

Data collection

Appendices- Screener, consent form, checklists, etc.

Recruiting participants

Step 1: Use personas- If you don’t have any, create assumption

personas based on low / high technological

expertise and low / high domain knowledge

Step 2: Go where your users congregate- Friends & family

- Customer lists

- Professional societies

- LinkedIn, Twitter

- Recruitment agencies

Step 3: Filter with an online screener

Step 4: Follow up by phone

A taxonomy of tests

Portable test lab

The moderator takes the test to the participant and sits at the participant’s desk or cubicle. If observers are present, they squeeze in and sit where they can.

A taxonomy of tests

Single room setup

The moderator and participant sit at a desk in a small office. Observers sit at the back of the room and either crane their necks or watch the test on a separate computer running screen sharing software. Observers are close enough to hear what the moderator and participant say to each other.

A taxonomy of tests

Classic testing lab setup

The moderator and participant sit in one room. Observers can see the moderator and participant through a one-way mirror and hear their conversation through the lab’s built-in audio system. Observers get a close-up view of the participant’s screen via screen sharing software.

A taxonomy of tests

Classic benchmark test

The participant sits in one room and works alone. The moderator and observers can see the participant through a one-way mirror and get a close-up view of the participant’s screen via screen sharing software.

A taxonomy of tests

Multi-room setup

The moderator and participant sit in one room. Observers sit in another room elsewhere in the building. Observers watch the participant’s screen via screen sharing software projected on the wall and hear the participant through the computer’s speakers.

A taxonomy of tests

Remote moderated test

The participant carries out the test from his or her home or office desk. The moderator uses screen sharing software to see the participant’s screen and presents the tasks to the participant over the phone. Observers use conferencing software to view the test over the Internet.

A taxonomy of tests

Remote unmoderated test

The participant carries out the test from his or her home or office desk. The computer presents the tasks to the participant and the participant signals when he or she has completed the task. Nobody observes or moderates the test.


Writing Test Tasks

Red routes

Roads in London with red lines on them are known as red routes: these are the key road arteries in London. Transport for London does everything it can to keep these routes clear.

Your product or web site has “red routes” too. They are the critical “user journeys” with your product.

Red Route Examples (from Gerry McGovern (2010), The Stranger’s Long Neck).

What are the Red Routes for a university web site?

- Can I study [subject]?

- What does it cost?

- What qualification will I earn?

- How will this help me in my job or career?

- Where do I go to take this course?

- Can I study part time, while working?


What are the Red Routes for a government web site that aims to help small business?- What grants are available to my

company?

- How do I apply for funding?

- What funding is available to a start-up?

- How do I apply for funding?

- How can I reduce my tax liability?


What are the Red Routes for a health web site?

- Based on my symptoms, what disease/condition do I have?

- What are the basic fact about [condition / disease]?

- How do I book an appointment online?

- How can I get advice from a health professional?

- Based on my symptoms, what should I do next?

Brainstorm red routes

You have 300 seconds.

Think about a product, system or service that you are working on at the moment.

Brainstorm a handful of red routes — the ‘headline’ tasks.

The context of use

People approach tasks differently based on the context of use.- Imagine you need to book a flight and hotel

in Paris for a 2-day trip with your partner.

- Imagine you need to book a flight and hotel in Paris for a 2-day business trip.

The red routes — finding a hotel, booking a flight — are the same but the context of use alters the way you would carry out these tasks.

You build context into your test tasks with scenarios.

“Search for jobs by geographic

area”

Red route

“You’re fed up with the long journey to

work. Find all the suitable job

openings within a reasonable

commute of your home.”

Scenario

Expand your red routes into scenarios

What are the implications of this scenario for the design?

We can offer a better experience if we show “commuting time” as an option in the search results. And note that this may not be the same as distance, it depends on transport connections.

You wouldn’t get this from the original red route which doesn’t have the contextual information.

“Buy a PSP”

Red route

“Darren wants to buy a Sony PlayStation

Portable and a suitable game for his 12

year-old son for his birthday this Monday.

Because of the short notice, he won’t buy

the product unless he can collect it from a

store before his son’s birthday.”

Scenario

Expand your red routes into scenarios

What are the implications of this scenario?

• We need to offer cross-channel support (buy online, pick up in store)

• We need to be able to show shop stock levels• We need to show store locations and directions

relative to the customer’s location• We need to offer a reservation system so the PSP can

be put aside• We need to cross sell suitable games by age group, or

at least classify games according to age group and gender

Six-step scenario checklist

Is it really a red route?Is it specific and

measurable?

Does it describe a complete activity

(integrated, not simple tasks)?

Does it describe what the user wants to do (not how the user will

do it)?

Does it include enough information to complete the task yet avoid hidden clues?

Is the task “portable” to competitor

products?

Create a scenario

You have 300 seconds.

Take one of the red routes you created earlier.

Use it to create a scenario — build in the ‘context’ information.


How to moderate a usability test

Hats moderators wear in a usability test (based on Carolyn Snyder’s Paper Prototyping)

Flight Attendant- Safeguard the physical, psychological and

emotional wellbeing of the participant.

• Greet participants, hang up their coats, offer a drink, engage in small-talk

• Monitor participants for signs of stress

• Provide reassurance and assistance as needed

• Obtain informed consent

How to be a good moderator

Before starting — let participants know they are going to be listened to and that their opinions are valued.

Observe and read the participant’s non-verbal behaviour — movement, facial expressions, etc.

Reinforce the participant by nodding or paraphrasing

Take some notes, but don’t write all the time

Reflect back what you think you heard and check your understanding.

Ask open-ended questions

Pace your talking speed to match the participant’s

Listen to inconsistencies that may need to be challenged

To be an effective listener, listen to yourself

To what degree does my non-verbal behaviour show that I value the participant (e.g. am I shuffling papers)?

What attitudes am I expressing non-verbally (e.g. am I focusing on the recording technology)?

What attitudes am I expressing verbally (e.g. am I making the person feel like a “subject” rather than a “participant”)?

What is stopping me from giving my full attention to this participant (e.g. noise, tiredness, boredom)?

To be an effective listener, listen to yourself

What am I doing to handle these distractions?

How intently am I listening to what the participant is saying?

What am I doing to demonstrate my willingness to listen to the participant?

How well am I reading the participant’s non-verbal behaviours?

How careful am I not to over-interpret non-verbal behaviour?

What NOT to say to participants (slightly adapted from Rubin & Chisnell’s Handbook of Usability Testing)

- Saying, “Remember, we’re not testing you,” more than three times.

- “Don’t worry, the last participant couldn’t do it either.”

- “No-one’s ever done that before.”

- “HA! HA! HA!”

- “That’s amazing! I didn’t even know it could go in upside down!”

- “Could we stop for a moment? Watching you struggle is making me tired.”

- “I didn’t really mean you should press any button.”

- “Yes, it’s normal for observers to laugh during a test.”

- “Don’t feel bad, many people take 15 or 16 tries.”

- “Are you sure you’ve used a computer before?”


Sports Commentator- Ensure that observers get as much useful

information from the test as possible.

Getting participants to think aloud


Phase 1 — Give instructions



Phase 2 — Demonstrate and let them practice

Demonstrate and let them practice

Demonstrate thinking aloud with an example:

Turn your phone to silent

Get the participant to practice with a different example:

Adjust the height of his or her chair

Make sure your practice task is easy: you want the user to practice the technique and feel successful!

Project Activity: Quick activity

Practise welcoming a participant- Set expectations and reassure your

participant.

- Ask for your participant’s consent to record the session.

- Demonstrate thinking aloud.

- Get your participant to practise thinking aloud.



Phase 3 — Probe with open questions


Keep

talking…

Tell me

more

about that What are you

thinking

right now?

it was

easy!what was

easy?

is that how

it's supposed

to work?

is that how

you expected

it to work?

PHASE 3: probe with open questions


Keep

talking…

Tell me

more


thinking

right now?

it was

easy!what was

easy?

is that how

it's supposed

to work?

is that how

you expected

it to work?



Keep

talking…

Tell me

more


thinking

right now?

it was

easy!what was

easy?

is that how

it's supposed

to work?

is that how

you expected

it to work?


is this a useful

feature?

Mmm, do I think that someone, somewhere

will find some use for this feature?

would this feature, as it's currently presented, be

valuable to the way you choose products today?

Mmm, there are some good things and some bad things, so what do

i think overall?

was there anything you particularly

liked or disliked about the

prototype?what specifically?

Is this a good idea?

don't ask users to extrapolate…

…instead, focus on the present

…focus on the specifics

instead of generic questions…

is this a useful

feature?

Mmm, do I think that someone, somewhere

will find some use for this feature?

would this feature, as it's currently presented, be

valuable to the way you choose products today?

Mmm, there are some good things and some bad things, so what do

i think overall?

was there anything you particularly

liked or disliked about the

prototype?what specifically?

Is this a good idea?

don't ask users to extrapolate…

…instead, focus on the present

…focus on the specifics

instead of generic questions…

crikey, he thinks this option should appear on the home page and he won't like it if i say

otherwise.

don't you think this option would be better if it was available on the

home page?

is there any other place you'd like to see

a feature like this?

do i have my 'surprised' face on again? i better

think of something surprising now.

was that what you expected? what did you

expect?

you seem surprised, were you?

don't be judgemental

a better question would be…

don'T encourage confabulation

but keep the participant talking

crikey, he thinks this option should appear on the home page and he won't like it if i say

otherwise.

don't you think this option would be better if it was available on the

home page?

is there any other place you'd like to see

a feature like this?

do i have my 'surprised' face on again? i better

think of something surprising now.

was that what you expected? what did you

expect?

you seem surprised, were you?

don't be judgemental

a better question would be…

don'T encourage confabulation

but keep the participant talking


Scientist- Responsible for avoiding test bias and

recording the data.

• Representative users.• Appropriate task

scenarios.• Balanced design• Neutral questioning. • Where bias can’t be

avoided, make a note. • Decide when to intervene

to help a participant• Log the data• Record audio and video of

the session. • Collect accurate

measures of task completion, time on task and satisfaction.

Usability Test Plan Toolkit

Use a checklist to remind you how to phrase unbiassed questions.

You’ll receive a copy of this with your second refresher training e-mail.

!"#$%&'()*+,-*.,/#-0)'12*0*3(04'&')5*)#()*6#&$,.'12*)"#*70-)'$'701)*

! "#$!#%&#'$($)*+,!

!"#$"%&'()'*+"&*,&-"+-.&)*/"+&*,&0*1")2-*).&3*-"&-24")&231&'2)-$5$'23-.&"+-$02-"1&1()2-$*36&

! -#(,,./#!$0#!&(/$)')&(+$!

7*-&2&-"+-&*,&-8"&'2)-$5$'23-.&%23-&-*&/"2)3&-8"&+-)"39-8+&231&%"243"++"+&*,&-8"&1"+$93.&0*1")2-*)&$+&3*-&-8"&1"+$93")6&

8)0-)'12*0*1#9*)0(%*

! 10#'2!3*/!.+4#/,$(+4)+5!

:+4&-8"&'2)-$5$'23-&-*&)"'"2-&-8"&-2+4&$3&8$+&*)&8")&*%3&%*)1+6&

:3-'12*)"#*)0(%*

! 6#$!$0#!&(/$)')&(+$!$(72!

;2(+"&-*&/"-&-8"0&,$3$+8&2&-8*(98-&<&1*3=-&-2/4&*#")&-8"06&

! -#('$!$*!8*49!7(+5.(5#!

;$54&('&*3&'2)-$5$'23-&>*1?&/239(29"&*,&,)(+-)2-$*3&*)&233*?235"&231&21@(+-&$3-")25-$*3&255*)1$39/?6&

! :*+;$!4#3#+4!$0#!4#,)5+!

:5-&2+&2&3"(-)2/&'2)-?&*(-+$1"&*,&-8"&1"+$93&-"20&A&?*(&2)"&-8")"&,*)&)"+"2)58.&3*-&-*&1","31&1"+$93+6&

!<($'0!3*/!,#73='#+,*/)+5!

;)*>"&,()-8")&%8"3&'2)-$5$'23-+&5"3+*)&-8"$)&%*)1+&*)&>/20"&-8"0+"/#"+6&

! :#>#7*&!3/)#+479!/(&&*/$!

B"&%"/5*0$39&231&8"/',(/&-*&-8"&'2)-$5$'23-&%$-8*(-&>"$39&-**&"0'2-8"-$5&$,&'2)-$5$'23-&$+&3"92-$#"&-*%2)1&2&1"+$936&

! ?/*>)4#!+#.$/(7!#+'*./(5#@#+$!

C)"2-&'*+$-$#"&231&3"92-$#"&5*00"3-+&-8"&+20"&%2?6&DC8$+&$+&#")?&8"/',(/6&E*(=)"&1*$39&@(+-&%82-&%"&3""1&?*(&-*&1*F6&

! -#&#($!

!"$-")2-"&5*00"3-+&#")>2-$0&-*&"35*()29"&"/2>*)2-$*36&DE*(&+2$1&?*(&(+(2//?&+-2)-&-8$+&2''/$52-$*3&,$)+-GF&

! 10#'2!9*./!)+$#/&/#$($)*+!

!"'8)2+"&-8"&'2)-$5$'23-=+&+-2-"0"3-&$3&?*()&*%3&%*)1+&-*&"3+()"&255()25?6&DH&20&8"2)$39&?*(&+2?IF&

!A)+)@)B#!@*4#/($*/!4),$/('$)*+,!

B"&2%2)"&*,&23?&233*?$39&*)&)"'"-$-$#"&9"+-()"+&231&")21$52-"&-8"06&

! C+'*./(5#!$0)+2=(7*.4!

!"0$31&-8"&'2)-$5$'23-&-*&-8$34J2/*(16&DK82-=+&9*$39&-8)*(98&?*()&8"21&2-&-8"&0*0"3-GF&

! :*+;$!3*/'#!*&)+)*+,!

:+4&2>*(-&'2)-$5$'23-&*'$3$*3+&%8")"&2'')*')$2-".&>(-&1*3=-&,*)5"&23&*'$3$*3&$,&-8"&'2)-$5$'23-&1*"+3=-&,""/&+-)*39/?6&

! 17(/)39!(!$#/@!*/!'*+'#&$!

DK82-&1*"+&L+?+-"0=&0"23&8")"GF&DK82-&1*&?*(&52//&-8*+"&>(--*3+GF&

;(%'12*<3#()',1(*

! D>*)4!7#(4)+5!E.#,$)*+,!

B21M&DN*%&1$1&?*(&/$4"&-8"&/*9$3&+5)""3GF&O**1M&DK82-&1$1&?*(&-8$34&2>*(-&-8"&/*9$3&+5)""3GF&

! D>*)4!'7*,#4!E.#,$)*+,!

B21M&DP*"+&-8$+&024"&+"3+"GF&O**1M&DK82-&2)"&?*()&$0')"++$*3+&*,&-8$+GF&

! D>*)4!8)(,#4!E.#,$)*+,!

B21M&DH+&-8$+&,"2-()"&8"/',(/&-*&?*(GF&O**1M&H+&-8$+&,"2-()"&8"/',(/&-*&?*(&*)&$+&-8$+&,"2-()"&3*-&8"/',(/&-*&?*(G&K8?GF&

! F*'.,!*+!('$.(7!8#0(>)*./!(+4!(>*)4!5#+#/(7),($)*+!

B21M&DH+&-8$+&2&9**1&$1"2GF&O**1M&DN*%&#2/(2>/"&%*(/1&-8$+&>"&-*&?*(&$3&?*()&@*>GF&

! :*+;$!(,,)5+!87(@#!

B21M&DK8?&1$1&?*(&1*&-82-GF&O**1M&DH&3*-$5"1&?*(&1$1&Q6&C2/4&0"&-8)*(98&?*()&-8*(98-&')*5"++6F&

! D,2!3*/!,&#')3)'!#%(@&7#,!G!#+'*./(5#!,$*/9$#77)+5!

B21M&DP*&?*(&82#"&-)*(>/"&%$-8&+"2)58GF&O**1M&DC"//&0"&2>*(-&2&-$0"&%8"3&?*(&5*(/13=-&,$31&%82-&?*(&%")"&+"2)58$39&,*)6F&

! D>*)4!7*+5!(+4!*>#/79!'*@&7#%!E.#,$)*+,!

B21M&DK8"3&-8$+&+$-"&%2+&1"+$93"1.&-8")"&%2+&1$+5(++$*3&%8"-8")IF&O**1M&DN*%&5*(/1&-8$+&+$-"&>"+-&0""-&?*()&3""1+GF&

! :*+;$!(,2!&(/$)')&(+$,!$*!/#=4#,)5+!$0#!)+$#/3('#!

B21M&DK82-&%*(/1&>"&-8"&>"+-&%2?&-*&1"+$93&-8$+GF&O**1M&DH3&?*()&"R'")$"35".&%82-&*-8")&%">&+$-"+&1*&-8$+&%"//GF&

! D,2!'./)*.,!E.#,$)*+,!$*!#%&7*/#H!8.$!(>*)4!

'*+3/*+$($)*+!

B21M&DP*&?*(&82#"&23?-8$39&"/+"&-*&+2?&2>*(-IGF&O**1M&DS23&?*(&-"//&0"&0*)"&2>*(-IGF&

! I#$!8#0)+4!$0#!8#0(>)*./J!

B21M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"GF&O**1M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"G&K8?GF&

! ".@@(/),#!(+4!4/(K!(!'*+'7.,)*+!

DT*&%82-&H&20&8"2)$39&?*(&+2?&$+&-82-&-8"&%2?&-8"&LB**4&2&,25$/$-?=&$+&$0'/"0"3-"1&8")"&$+&5*3,(+$39.&>(-&-8"&$1"2&$+&+$0'/"6F&

6-077'12*37*

! D,2!3*/!(+!*>#/(77!(,,#,,@#+$!

DS23&?*(&+(002)$U"&?*()&"R'")$"35"&%$-8&Q&-*12?GF&*)&DK82-&%*(/1&?*(&+2?&%")"&-8"&-*'&-%*&+-)"39-8+&231&%"243"++"+6F&

! -#E.#,$!'*@@#+$,!/#5(/4)+5!$*&)',!+*$!/(),#4!89!

@*4#/($*/!

DK82-&5*00"3-+&1*&?*(&82#"&*3&-*'$5+&%"&1$13=-&5*#")&-*12?GF&*)&DK82-&"/+"&+8*(/1&H&82#"&2+4"1GF&

Quick Exercise: Improve these closed questions

For example:

- Closed question: Don’t you think you should try the Back button?

- Open question: How else might you return to the home page?

Are you looking for search?

Did you just click on the Submit button?

Didn’t you see the link to the site map?

Don’t you think you should look in the pull-down menu?

Do the radio buttons make your options clear?

Do you think the design of this page is good?

Do you think people will find this easy to use?

Getting clients and the development team to log usability issues

Ask the people observing the test to write a short description of each usability problem they see on a sticky note.

One problem per sticky note.

Identify the participant associated with the observation (e.g. P5).

At the end of the day’s testing, work with the observers to group, prioritise and label the usability issues.

Why data logging is important

As you watch the test, you should note down the participant’s behaviours and code your observations.

Good note taking saves hours in analysis time.


- Data logging ensures you note

all behaviours, not just the ones

that stand out (this helps reduce bias in your observations).

- It helps you concentrate and

focus on the session.

- When lots of observations come at

once, you will be able to just note

the observation code — you can

then review it later on the videotape.


- When scanning your notes, the observation codes make it easy to

distinguish one class of

observation (e.g. the usability issues) from other observations.

- It’s one of those things you’ll be glad you did when there are

problems with the video

recording (e.g. when the sound is

poor or when the recording is corrupted).

S Start task

E End task

X Usability problem

* Video highlight — an “Ah-ha!” moment

G Comment from P (general)

P Positive opinion

N Negative opinion

B Bug

F Facial reaction (e.g. surprise)

A Assist from moderator

Q Gives up or wrongly thinks finished

H Help or documentation accessed

M Misc (general observation by logger)

C Clickstream (optional)

How to classify your observation

How to log usability problems

As a rule of thumb, you should average about one observation per minute.

But remember this is an average- Observations are a bit like buses (none for

ages, then three come along at once).

What you should write down:- The time

- The class of observation

- A short description

S Start task B Bug

E End task F Facial reaction (e.g. surprise)

G General comment A Assist from moderator

P Positive opinion Q Gives up or wrongly thinks finished

N Negative opinion H Help or documentation accessed

X Usability problem M Misc (general observation by logger)

* Video highlight — an “Ah-ha!” moment C Clickstream

Time Code Description

Thank You!

Dr David TravisManaging Director

180 Piccadilly, London, W1J 9HFUSERFOCUS

Work 020 7917 9535Mobile 07747 016132Email [email protected] @userfocus

Date post:	27-Jan-2015
Category:	Design
Upload:	david-travis
View:	103 times
Download:	0 times