+ All Categories
Home > Technology > Usability Testing Fundamentals

Usability Testing Fundamentals

Date post: 16-Apr-2017
Category:
Upload: debcook
View: 760 times
Download: 0 times
Share this document with a friend
82
Usability Testing Fundamentals Everything You Need to Know to Get Started Debbie Cook and Michelle Erickson 1
Transcript

Usability Testing Fundamentals

Usability Testing FundamentalsEverything You Need to Know to Get StartedDebbie Cook and Michelle Erickson1

For MathWorks Training:Explain purposeUPA workshop on MondayTried to condense a semester long class, 2 books and years of experience into 3 hour tutorialYoure helping us practice, while at the same time, learning the basics on running your own usability testGoal is that this also becomes MathWorks official training for Usability Testing

INTRODUCE OURSELVESMy brief bioMichelle brief bioLogistics:Feel free to ask questions as we go through the materialsPlease turn phones and other devices onto silent

Print-outs Decide what to doThe content in the presentation is similar but the order of some of the printed material may be slightly different.We wont cover everything in the printed materials.

We feel that the best way to learn is to DO so we want to focus on practicing the most essential concepts.

1

IntroductionsWho we are:Debbie CookMichelle Erickson

Who you are:NameCompanyRoleGoals for this course2

ObjectivesIn this workshop, you will learn to:Plan, design, and prepare for a diagnostic usability testDecide which user(s) to recruitCreate effective tasks and scenariosRun and effectively moderate a usability testEmploy a low-cost, low-effort approach for capturing and analyzing results

3

?Ask them what they hope to get out of classSet expectations

In this workshop, you will learn:How to plan for a diagnostic usability test, includingWhich users to focus test onAnd the fundamentals of writing tasksHow to moderate a usability test and interacting with the participantHow to capture and synthesize the data using a low-cost, low-effort approach we leverage at MathWorks.

Our goal is that youll know enough when you leave here today to get started testing

3

Why Test?

Kegworth Air Disaster - 19894

Why is usability testing important?One reason is to avoid disasters like this one1989 Plane bound for Belfast crashed along the side of the M147 people were killed and 74 people were injured

What happened?One of the engines failedPilot mistakenly turned off the wrong engineInvestigation of the incident revealed that one contributing factor to this horrible accidenthad to do with design

BBC: http://news.bbc.co.uk/2/hi/uk_news/northern_ireland/8363595.stmhttp://www.aaib.gov.uk/cms_resources.cfm?file=/4-1990%20G-OBME%20Append.pdfo with design and usability of gauges4

Why Test?

Updated gauge design

Previous gauge design

5

Gauge design had an impact on pilots ability to determine engine performance at a glance

Do you think this is something that could have been discovered during usability testing?

Its our goal, that at the end of this class, youd be able to plan a test that might uncover this type of design flaw.5

AgendaDefinition of usability testingPlanning, designing, and preparing for a testBREAKModerating and running a testAnalyzing and summarizing results

Were going to start off the tutorial by explaining what we mean by the phrase usability testing, the two common types of usability tests.Then were going to go into detail about how you would:Plan and prepare for a testModerate and conduct a testCapture and analyze the results

The bulk of our time will be spent in the middle two sectionsAgain, because we believe youll learn a lot more by doing some of these things than you will by listening to us talk about these thingswere going to try to give you the opportunity to try your hand at many of these things.

Weve tried to structure the class so that its as hands-on and interactive as we can be given the timeframe and the amount of info were trying to cover.

Lets jump in.6

What is Usability Testing?A systematic way of observing actual and potential users of a product as they work with it under controlled conditions.Joe Dumas & Beth LoringModerating Usability Tests

ParticipantObserverModerator

7

Two Types of Usability TestsFormative testGoal: Uncover and diagnose issues; understand users workflow and/or mental modelFocus is on qualitative assessment, identifying and fixing issues with the designSummative testGoal: Measure usability of a product or provide a baseline for comparison between products or versionsFocus is on quantitative assessmentThis tutorial will focus on Formative Testing.8

Why only covering formative?HarderMore commonYou learn more from the tests8

Formative Testing Can Help YouLearn about users requirementsDiagnose problems early and reduce wasteEnsure users success and improve their satisfaction with the productAdvocate for a change to the designEducate stakeholders about how users interact with the product

Encourage audience participation.9

Planning, Designing and preparing for your test

10

Better graphicsNow that were all on the same page about what a usability test is and what the benefits are, lets talk about how youd plan for one.When I say plan, I mean plan, design and prepare for5 Ps Prior Planning Prevents Poor Performance10

Planning RoadmapCreate a plan: the whatInclude the right people: the whoParticipantsObservers

ObjectivesScenarioTasksProtocolPrototypePilot test11

Get rid of this roadmap and just use an overall roadmap?

To run an effective test youll need all these things and youll need to ensure that the right people are involved. Today, for the sake of time, were going to focus on:ObjectivesScenarioTasksProtocolParticipantsObservers11

Test Plan Template

12

Michelle is going to hand out this template that well refer to throughout this tutorial.In fact, were going to use it to plan a test of our own.12

Case Study: DisneyParks.com

Pass out the Workshop Aid template

Because we think working through an example is a better way to learn than listening to us talk at you for 3 hours, weve selected a Web site to use as a case study throughout the workshop.

Our goal is to plan a test using a case study: Disneyparks.com

Normally, youd probably be testing a product youre familiar with. At the very least, youd likely have had a chance to look at briefly and acquaint yourself with it.

For the purposes of todays workshop, were going to rely primarily on instinct and assumptions because we also want some of you to act as participants?

CASE STUDY SETUP: Were going to pretend that we work for Walt Disney Company.

The company has noticed an increase in the number of telephone calls theyve received from people trying to plan vacations. They want to reduce expenses associated with this service by enabling people to successfully plan a vacation on the Web site, without having to call the support line for additional assistance.

Youre working on the Web site design team.

Well work together to start filling out a Usability Test Plan of the DisneyParks.com site.

13

Deciding on Test ObjectivesWhat are your goals for the test?What do you hope to learn?Are there aspects of the design youre concerned about?Are there particular user groups that youre concerned about?

14

One of the first things you want to think about are test goalsWhy are you running the test? What are you trying to learn from having users work with the product?

Do you have questions about how users will react to certain aspects of the designHave previous tests or heuristic reviews revealed potential flaws in the design that you want to exploreDo you simply want to understand how users are really using the product?

Purpose and focus of test will likely be influenced by:Where you are in development cycle.New product or existing productThink about user groups initially what types of people use this site?Are there groups of users you think may experience difficult or that you want to better understand their workflow.14

Thinking About Your UsersIdentify target user profiles of interestThink about the target users of the productPrimary usersSecondary usersDetermine which user characteristics are most likely to impact resultsReview with team to ensure buy-in

15

Who: How do you decide which user(s) to test?Wont be able to test all user profiles.Think about the different users of your product and the characteristics that differentiate or impact how they interact with the product.Determine which user characteristics are mostly likely to impact test performance, focus your plan and your recruiting efforts on people with those characteristics.

Example: smart phones Do you think teenagers use smart phones differently than their parents or grandparents do?What types of things might differ? Do you think people who travel a lot might use smart phones differently than people who dont travel as much?People with young children?Why is it important to review with team?Dont want them to write off data or downplay feedback because the participant was atypical, not representative, wrong profile

What characteristics might you consider for products youre interested in testing?Demographics?AgeIncome levelComputer experience?Use the computer for internet or e-mail vs. use the computer every day for workMobile usage?

15

DisneyParks.com ActivityBrainstorm Objectives and Users

As a group, lets brainstorm together

What goals or objectives might you have for a usability test of this Web site?

Who are target users of this Web site?

16

The company has noticed an increase in the number of telephone support calls theyve received from people trying to plan vacations. They want to reduce expenses associated with this service by enabling people to successfully plan a vacation on the Web site, without having to call the support line for additional assistance.

Youre working on the Web site design team.

Target time: 10 minutesLarge group activity: Write ideas on whiteboard or large post-it easel paper

Show site while people brainstorm

Sometimes the goals of test are impacted by whether or not you think users are successful with their goals while using the product16

DisneyParks.com ExampleTest Objectives

17

SIMPLIFY? LARGER FONT?Here are some example test objectives we came up withAre there similarities?Differences?17

DisneyParks.com ExampleUsers

18

Consider adding pictures18

Creating a ScenarioWhat is a test scenario?A short description of the participants current state used to set the context for the test or task.A well-written scenarioSupports the goal(s) of your testIs realisticSpeaks the users language, not the systems

19

I actually gave you a scenario to set the stage for our tasks today.

We work for Walt Disney Company.

The company has noticed an increase in the number of telephone calls theyve received from people trying to plan vacations. They want to reduce expenses associated with this service by enabling people to successfully plan a vacation on the Web site, without having to call the support line for additional assistance.

It gave you all a frame of reference for the activities that were doing / going to do.

Youll want to do the same for the people youll test

EXAMPLE OF USERS LANGUAGE VS SYSTEM LANGUAGE?

GET class involved here do a pop quiz!19

Developing Your TasksWhat is a test task?A brief description of what you want the participant to do during the test (not how to do it)A well-written taskSupports the goals of your testIs realistic; it reflects a real task user will performProvides relevant details needed Speaks the users language, not the systemsProbes potential usability issuesIs typically independent of other test tasks20

Scenarios and tasks are the two parts of your test that users will actually see.

Task describes what the user needs to do, not how they should do it (not step by step instructions)

Simple example: for cell phone some sample tasks would be:Make a call toAnswer a callAdd your best friends contact information to your list of frequently called

Include the relevant details participants will need to complete the tasks.For example, for the make a call task, youll want to provide details about the person or phone number to call.

Support the goals of your testTasks should be realisticProbe potential usability issues major and minor (e.g., touch screens for portable devices environmental considerations like glare or gloves)Speaks users language, not the systems example:

Independent tasks help preserve participants self esteem dont want the participant to feel like theyve failed

Example of dependent tasks:Configuring the phoneMaking a call

There may be exceptions to the independent tasks rule

20

Tips for Developing TasksThink about tasks that are:New to the user or require the user to change the way s/he performs the taskPerformed frequentlyCritical to successLikely to reveal a potential usability issue or concernUnique to a particular environment or interaction

21

ANY OR ALL OF THESE!Brainstorm tasks, then refine, scope and finalizeTips for brainstorming: think aboutNew tasks (things the user has to do now that they didnt have to do before)Common users tasks (frequently performed)Critical user tasksTasks where users have or might run into issues with the designEnvironmental considerations and types of devices and interaction behaviors

In the Disney case studywho might be a great source of information about where people have problemspeople providing phone support21

DisneyParks.com Activity Brainstorm Scenario and Tasks

As a group,Brainstorm (15 minutes):Test scenario - how would you set the stage for the user at the beginning of the test?Tasks - what task(s) might you have the users perform to support your test objectives?Report out (5 minutes):

22

Have small groups break to write 3 tasks.Target 15 minutes to brainstormReport out 5 10 minutes

We give feedback while they are working on tasks Catch language users not disney-speak

TELL CLASS TO USE OUR USERS New slide or handout with those users

22

DisneyParks.com Example Scenario and Tasks

23

When showing them the example tasks, explain that the first task is an example of a more exploratory task.

What types of things were we careful of

23

Refining Your TasksRefine and scope tasks based on:PriorityRelevancePrototype capabilitiesTimeEstimated time to completeStakeholder availabilityPossibility of completionNEVER give users an impossible task!24

CONSIDER IMPOSSIBLE controversial?After youve brainstormed the tasks, then refine, scope and finalize themMay combine tasks, eliminate tasks

Typically you do this based on:Priority how important is the task relative to the others the test goalsRelevance how well does the task help achieve test goals. Help you learn what youre trying to learnPrototype capabilities can users be successful at this task given current prototypeWe havent really talked about prototypeprototype can be anything

NEVER GIVE USERS A TASK THAT THEY COULDNT POSSIBLY SUCCEED AT - VERY IMPORTANT THAT YOU DONT INTENTIONALLY SET USER UP FOR FAILURE Time24

Establishing a Test ProtocolAlways start the user off with an easy taskIdentify possible curve balls Create a plan for how you will deal with these if the user encounters themEnsure the team and observers are aware of, and agree to, the protocolDecide how you will provide help if askedIdentify key information for the moderator (you) and the observers to be aware of25

All I mean by protocol is instructions for how youre going to handle certain situations.Observers are the members of your development team that youre going to invite to the test (more on that later)

Always start them off with an easy taskFirst task is about setting them at easeLetting them get comfortable with the concept of usability test.Sometimes you can start with an exploratory task which is easy because there are not wrong answers.Example of a curve ball or land mine and strategy for dealing with it?Part of the prototype isnt thereThey ask for documentation / help what will you do?Go down the wrong path, how long will you let them explore?Example of GUIDE test where we wanted to get feedback on a particular new featureProtocol stated that if users hadnt discovered the feature by the 3rd task wed lead them to it since wed already learned it wasnt discoverable but wanted to learn if they thought the feature added value.

25

Identifying Key Information For each task in your plan, capture:The goal(s) for the taskThe correct answerEstimated time on taskAny special protocol for the taskAreas of interest you want your observers to take note of

This information is only for you and your observers.26

Your goal what you hope to learn from the taskCorrect answer so you know the path to success and when the user has completed the taskAre there any things you observers should be on the lookout for?For example: Where did users look for the feature? Did they go to the right menu or toolbar button?

26

Example Test Protocol

27

http://sharepoint/dev/usability/teams/ltc/Documents/Specialist%20Files/Michelle/UPA%202012/Scrubbed%20GUI%20Building%20test%20tasks.docx27

DisneyParks.com Example Key Task InformationShow me how you would get started planning your trip.

Max time on task: 10 minutesTask goals:Learn how the user navigates to the siteLearn about how the user begins his/her search for information

Protocol Notes: Start participant out with a clear desktop, i.e., no windows open, not even a browser.This is an exploratory task to see how they get started, once the user starts down a particular path the facilitator will guide him/her to an appropriate follow up task.

Observers take note of:How does the user get to the Disney site? (e.g., Google search, typing in direct URL, etc.)How does the user navigate to the DisneyParks.com site?What does the user click on once s/he arrives at the DisneyParks.com site?What information does s/he pay attention to first? What information does s/he start looking for?Does the user browse for information or search?Does the user notice the Vacation Planning option on the DisneyParks.com home page28

MAY NOT INCLUDEHave them brainstorm first - ?CONSIDER ANIMATING THISCUT OFF SOME OF THE TEXTHARD TO SEE

Dont always have to do this, but the more prepared you are, the better the test will go.28

Choosing ParticipantsIdentify user profiles of greatest interestDetermine which user characteristics are most likely to impact resultsReview criteria with team to ensure buy-in

29

ParticipantObserverModerator

This is really about who you will recruit.For our exercises, we focused on vacation planners.

Who: How do you decide which user(s) to test?Wont be able to test all user profiles.Determine which user characteristics are mostly likely to impact test performance, focus your plan and your recruiting efforts on people with those characteristics. What characteristics might you consider for products youre interested in testing?Demographics?AgeIncome levelComputer experience?Use the computer for internet or e-mail vs. use the computer every day for workMobile usage?Why is it important to review with team?Dont want them to write off data or downplay feedback because the participant was atypical, not representative, wrong profile29

Deciding on Number of ParticipantsIt depends onTimeMoneyAvailability of participants that fit the profileAttitude and availability of key stakeholdersGenerally accepted practice = 6 12 participantsThree (3) to five (5) participants per profile80% of problems identified with 4 5 participants

30

KEY TAKEAWAY ACKNOWLEDGE SAMPLE OF CONVENIENCE

Obviously testing takes time, the more people you test, the longer it can take.Often times, teams have a specific window for testingMoneyParticipantsParticipants are usually paid for their time and effort. (Need to think about this as part of your testing budget). How will they get paid and how much?CashGift cardsProductNot a tax expertthere are tax implications for participants so you want to think about thatMore participants = more moneySpecial expertise = more money (doctors, finance experts)Development teamsIf stakeholders are participating in the tests, time they spend testing is time away from other aspects of their jobAvailability of participants - Sometimes difficult to find people to participate due toDesired user characteristics, especially if specific expertise neededPaper prototype (limits to local area)Acknowledge convenience sample - we often take what we can getAttitude and availability of key stakeholdersHow open is the team to feedback? How likely are they to make changes? (If no, why waste your time testing?)If observing, did they plan for this in their estimations?

Some data is better than no data

30

Recruiting ParticipantsDevelop a recruiting screenerTake advantage of internal and/or external resourcesInternal databasesAgency / service providersAdvertisingCast a wide netCommunicate the benefits of participation

31

DONT COVERParticipant background / recruiting screener based on the profile / characteristics of interestLeverage leadsPlace where people do similar tasks nowPeople youve talked to in the pastBenefits you need to be prepared to explain the value they get from testingOpportunity to provide early feedback that could influence design / directionAdvance exposure to a new feature or productChance to interact with developers of a product they might be using (be careful here)

Recruiting participantsInternal Customer databasesLeads from customer facing employeesUsability dbExternalService providers like Alpha Buzz, Usability Works and ones in AppendixAdvertisingOn your own websiteSocial networking sites (e.g., craigslist, professional organization sites or newsgroups)Craigslist

31

Inviting ObserversInvite people to observe and take notesTeam membersDevelopersDesignersWritersQuality EngineersProduct managersKey stakeholdersClientsChampionsInvite anyone who might benefit from attending.32

ParticipantObserverModerator

Tie in importance of taking notes for analysis method well discuss laterMention that # of observers may be influenced by test location (user site fewer people, lab more)If youre running a Wizard of Oz test where someone has to act as the computer, theyre obviously a required attendee!If youre new, may want to get your feet wet before inviting tons of peopleYou may be constrained by space limitations, if youre running a test in the field only want to bring essential personnel.If you have your own lab, highly recommend you invite anyone who might benefit from watching

Invite anyone who might benefit or who we might want to influence.

32

DisneyParks.com Activity Brainstorm Participants and Observers

In a group,Brainstorm (10 minutes):ParticipantsWhich user group(s) will the test focus on?Who will you target as possible participants?Observers which stakeholders might you invite to observe the tests?Report out (5 minutes):

33

Essentially, thinking about the questions youd want to ask on your screener to decide which users are appropriate for your testHave small groups break to write 3 tasks.Target 15 minutes to brainstormReport out 5 10 minutes

We give feedback while they are working on tasks Catch language users not disney-speak

33

DisneyParks.com Example Participants and Observers

34The project team

Label the picture The Team

34

Planning RoadmapCreate a plan: the whatInclude the right people: the whoParticipantsObservers

ObjectivesScenarioTasksProtocolPrototypePilot test35

Hand-out completed version of the DisneyParks.com Test plan

Wrap-up and summarize Touch on idea that you dont need a fully functioning prototype and a tripped out usability lab to test. You dont just test.Pilot test practice run to work out kinksWe didnt cover the additional two sections, filling them out can be helpful. We just dont have time.35

The PrototypeMyth: You need a fully functioning product to conduct a usability test.Reality: You can test withPaper prototypeInteractive mockupsThrowaway prototypeFully functioning productWhen it comes to developing your prototypebe creative.36

May be cutPrototype, can the prototype support tasks, do you need to modify tasks

Deciding what to test, - fidelity of prototype, which features or aspects of application or product to testWhat: What level of fidelity prototype do I need for an effective usability test?Paper prototypes, software prototypes, hybrid paper and software, balsamiq mocks and click throughs, APIs - Instant messaging test, final software PaperInteractive Balsamiq, Axure, PowerPoint, ExcelThrowaway prototype semi-functional product built just to facilitate testing (depends on time and money)Fully functioning product can test with fully functional, released software. Great way to learn about enhancement opportunitiesBe creative.Examples: We work on command line interface We have used IM client to simulate the experience of typing a command at the prompt (Wizard of Oz).Weve paired interactive prototypes with paper

36

Running a Pilot TestWhat is a pilot test?A run through focused on identifying and fixing any issues with the test itselfA pilot can provide valuable feedback on:Task timesTask goals (i.e., are the tasks testing what theyre intended to test?)Understandability of tasksPrototype readiness

37

DONT COVER37

Testing Methods to be Aware OfRemote UnmoderatedComparativeCo-discoveryRapid Iterative Testing and Evaluation (RITE)38

DONT COVERMethods and how they differ, when to do each, pros and cons

38

BREAK

39

Michelle 20-39

Moderating and Running a Test

40

Hello, thanks Debbie!!Tell whale joke?VIDEO HERE show bad video What did you see, what could have been better?

40

Moderator ResponsibilitiesSetting up the environmentPreparing observersCaring for and briefing participantsConducting the test

41

ParticipantObserverModerator

Setting up lab equipment for viewing and recording you may delegate this to a lab tech or AV personBriefing and setting ground-rules for observers, instructing them on how to take notes and ask questionsGreeting and taking care of the participant- Making participant feel comfortable, welcome and appreciatedParticipant briefing - Setting expectations and explaining ground rules and legal issuesAll aspects of running test

41

Setting Up the EnvironmentPrototypePaper prototype ComputerCamerasAudio and video recorders

Myth: You need a state of the art usability lab to run tests.Reality: You can test a paper prototype anywhere!

42

Whether you are testing in a lab or a persons site.You will be responsible for making sure everything is working and set up to test.

Based on the practices at your company, you may be recording the test, you will likely be responsible for setting up all recording and viewing equipment.Depending on the type of prototype and location, it will impact the equipment needed.42

Moderator ResponsibilitiesSetting up the environmentPreparing observersCaring for and briefing participantsConducting the test

43

ParticipantObserverModerator

Setting up lab equipment for viewing and recording you may delegate this to a lab tech or AV personBriefing and setting ground-rules for observers, instructing them on how to take notes and ask questionsGreeting and taking care of the participant- Making participant feel comfortable, welcome and appreciatedParticipant briefing - Setting expectations and explaining ground rules and legal issuesAll aspects of running test

43

Briefing ObserversSet ground rules for etiquetteTeach observers how to take notes Establish a protocol for observers questions during or after studyHave observers write notes on post-its and pass to moderator, or Instant Message during testAllow observers to ask questions directly

44

Reinforce rules for observers such as no sidebars or participant interruptions, also keep volume down even in observation rooms. Suggest that observers are in separate room or at a different table if possible.

Some people allow observers to pass questions to you to clarify any confusion about test.Agree on a method for flagging questionsSafest to Have observers write notes on post-its and pass to moderatorBut if you trust your observers to not bias and to interact with participant appropriately, then Allow observers to ask questions directly

44

Observer Groundrules

45

Add pic of our observers ground rules form UX lab!

45

Collecting Observational DataTraining observers on taking objective notesIndicate direct quotes vs. observationIndicate if it is an observer thought or ideaRecord task and context Indicate time on tasks, when relevantOne observation per post-it

46

Indicate if its a solution or design idea for a problem they observed how to include this in slide??We at MathWorks find it extremely valuable to have observers take notes.This ties into the low-cost method for analyzing test data Ill talk about later.

Showing observer idea rather than an observation lightbulb, i?Note taking how much should the moderator take notes? There are a variety of methods, Morae, heres where you can learn more.

Many different techniques But our focus is using post itsEspecially important for the low cost analysis method we will discuss later

Observations should include things you see the participant doing

46

Whats a Good Observation?Anything that you think is interesting, remarkable, noteworthy.What you see the participant doingWhat you hear the participant sayingWhat you notice the system doing in response to the participants actions

47

What to write down not EVERYTHIGN YOU SEE THEM DOINGanything surprising, relevant to design, problems, expectations, decisions

Show pics of post its with notes on them here47

Example Observation Notes

48

48

Moderator ResponsibilitiesSetting up the environmentPreparing observersBriefing participantsConducting the test

49

ParticipantObserverModerator

Setting up lab equipment for viewing and recording you may delegate this to a lab tech or AV personBriefing and setting ground-rules for observers, instructing them on how to take notes and ask questionsGreeting and taking care of the participant- Making participant feel comfortable, welcome and appreciatedParticipant briefing - Setting expectations and explaining ground rules and legal issuesAll aspects of running test

49

Briefing Participants50

50

OverviewCaring for the participantInformed consent and non-disclosure agreementsUsing a test script Introducing the test

51

51

Caring for ParticipantsReduce anxiety and create comfortable environmentOffer beverages, restrooms, and ensure their physical comfortAnswer questions when possible52

Usability Studies can be stressful - Reduce anxiety and set at easeAnticipate that participants may show up early have a place for them to wait, get comfortable Offer restrooms, beverages, engage in neutral conversation practice think aloudAnswer any questions they may have52

Informed Consent and Non-disclosureInform participants about:Informed consentParticipant rightsConcept of minimal riskNondisclosure and confidentiality agreements

Sample available on STC Website: Usability Toolkit: http://www.stcsig.org/usability/resources/toolkit/toolkit.html53

Be sure to consider these issues and discuss with your company.RESEARCH: Nondisclosure vs confidentiality

Must protect legal rights of organization and the participantMust inform your participants of the following items and have them sign proper legal docs in your presence.

Use boilerplate forms or create a form and have your legal dept. review statement of rightsEnsure participants have read, understand, and have signed forms in your presenceFollow proper procedures to care for participants

Participants rights withdraw at any time, be informed what test is about, what data will be used for

Minimal risk probability and magnitude of harm or discomfort anticipated in the test are not greater than those ordinarily encounteredin dialy life or during routine physical or psychological exams or test.

Informed consent key to protecting participant rights:Information procedures, purpose, risks, opp to ask questions, break, or quit. Importance of test and their feedbackComprehension they completely and clearly understand test procedures and purposeVoluntariness not coerced or unduly influenced

Forms to have signed:Nondisclosure and confidentiality participant cannot share with anyone the details of the test or product evaluated Waivers permission to use recordings, feedback, and comments gained during test but you will not record or use their name

53

Example

54

MAYBE NOT SHOW54

Benefits of Using a Test ScriptEach participant hears all critical briefing informationAll participants get the same information

55

Use script and checklist to ensure equal prep for all participants this could impact results!!

Refer to your participant briefing and training planSame info to all participants to minimize affecting different outcomesEnsure that you cover all important aspects (legal issues, procedures, etc before each testIf someone else needs to facilitate for you unlikely event you can be sure they received consistent info

A note on being organized:Easiest to make it a checklistLabel all test materials keep separate folders for each participant, labeled with participant number and test nameHave separate observer notes,

Staying organized will help maintain test quality

55

Example Test Script

56

Make handout maybe show a screen shotShow general script here.Point to the Briefing Script part and explain its often on a checklist with other moderator set up tasks.

56

Introducing the TestImportance of their participationExpected length of testTest environment Procedure (tasks, questions for participant, wrap up)Breaks, stoppingQuestions for the team

57

Introduce environment: Where you will each sitIntroduce anyone else that will be in room with you and explain their roleTest environment - (test room, computer, recording equipment, observation room)Explain why we record how it helps us analyze product and usability issues, who will see it

Questions they can ask as many questions as they like, but we may not answer them at all or maybe some at end of test.But encourage them to ask because it helps us understand their thought process and areas of possible confusion

57

Think Aloud ProtocolParticipants verbalize thoughts and expectationsModerator reminds participants and prompts for their thoughts58

ROADMAP OUTLINEVISUAL _ PERSON WITH THOUGHT BUBBLE AND THEN TALK BUBBLE THE SAME THING!

DEMO the protocol (Michelle and Debbie live demo if time permits) or give example.

Ask users to verbalize their thoughts and questions as much as possibleWhat are they thinking? What do they expect will happen? What did they think when system did something? What are they looking for? What are they planning to do? Where are they clicking or reading on screen?Doesnt come naturally to most peopleWill provide prompts as test goes along

Advantages lots of insight into users workflow, mental model, expectations and goalsDisadvantages slower performance and speed through tasks, unnatural

Demonstrate using a stapler or some other object (bring the object if you plan to do a demo)Ask if they have any questions about think aloud

Viewpoints on whether think aloud affects performance during the test

An alternative to Think Aloud is self reporting after the tasks or entire testPros:More natural task work, less interruption of users workflow thru tasks, less talkingDisadvantages:This is more time consuming

Think Aloud protocol(fidelity and robustness of prototype, interaction guidelines, no wrong answer, ask questions but we wont answer them)

Questions they can ask as many questions as they like, but we may not answer them at all or maybe some at end of test.But encourage them to ask because it helps us understand their thought process and areas of possible confusion

Think Aloud protocol

58

Were Testing the Product!Assure the participant that youre testing the product, not their performance.Most participants blame themselves when they encounter problems.

59

ROADMAP/OUTLINEVery important to state up front and remind them of this!Problems you may encounter are reflective of the design and the prototype, not your abilities or performance

59

DisneyParks.com Activity Participant Briefing Demo

60

10-15 minutesWe can pull up a volunteer to act as participant and have Michelle brief them.

Fix animation to blow out the briefing piece.REVISE CHECKLIST!! Add Before the Test header on checklistPass out the Disney Usability Test Checklist.

Take a few moments to read through this and start to memorize the key points.Can we have a volunteer from the audience to role play the study participant? I will demonstrate a participant briefing.Your briefing does not have to follow the script verbatim, just cover the important parts. Also try not to read from the script directly make sure you maintain rapport and eye contact with your participant.

After the role play, lets debrief. What did you notice? Did they forget any parts? Which parts do you feel are most important?

60

Moderator ResponsibilitiesSetting up the environmentPreparing observersBriefing participantsConducting the test

61

ParticipantObserverModerator

Handling lab equipment for viewing and recording you may delegate this to a lab tech or AV personBriefing and Setting groundrules for observersGreeting and taking care of the participant- Making participant feel comfortable, welcome and appreciatedParticipant briefing - Setting expectations and explaining ground rules and legal issuesAll aspects of running test

61

Conducting the Test62

62

Overview

Task handlingInteracting with participants Providing helpConcluding the testTroubleshooting

63

63

Task HandlingPaceAssists and AbandonsTask completion64

Task HandlingPaceGive tasks one at a timeLet participants set paceAsk clarifying questions after each task (if needed)

65

CONSIDER ROLLING THIS INTO TASK PACE

Give tasks one at a time Have each task on a single page (one task per page)Start with the scenariosUser does not get overwhelmed seeing a stack of 12 tasks and you may not get to all of themSome people dont number tasks so that participant wont know if they skip a taskremoving excluding tasks. Reordering tasksdependent tasks

Let participant announce when they have finished each task.Ask if they would like to continue and give another taskIf providing observer notes, make sure you have a copy without for participants65

Task HandlingAssists and AbandonsBased on test goals, consider:How will you provide assistance when the participant is stuck?When will you advise or allow the participant to abandon a task or test?

In general, let participant struggle.Dont assist if it interferes with an important objective.

66

DEFINE ASSIST AND ABANDONstruggle (if you think participants will struggle in a particular area, discuss ahead of time with team about assists, how long to let them struggle, important objectives, etc.)

Plan in advance. Based on goals you set ahead of time decide when to assist or abandon.Assist means participant failed task.

CONSIDER Using When a participant gets stuck.66

Task HandlingTask CompletionParticipant should indicate when they are done with each task.Ask them whether they are finished, if necessary.Build confirmation into tasks.If they do not know, moderator needs to determine whether the task is complete.If not, ask them to reread and/or redo task.67

MAY NOT PRESENTWho decides when/whether a participant has completed a task?

Building confirmation into task What tickets would you buy? Write down your answer hereModerator ultimately decides when a task is complete, but:Participant should verbalize when they are done with a task.This gives us critical info about their understanding But when they do not know or say so, you need to determine whether to move on or have them go back and redo task. Or you can ask a question to help them determine whether they are done.

Ask them to keep tryingBe consistent among all participants and test 67

Interacting with ParticipantsAvoiding biasing the participantInterruptingUsing active listening skills

68

68

Interacting with ParticipantsHandling QuestionsAsking unbiased questions:Neutral wordsAvoid adjectives and adverbsUse both extremes (i.e. easy and difficult)

Answering participant questions:Dont answer directlyTurn questions around and ask the participant:What do you think will happen? What do you expect should happen?69

Ask audience why its important to avoid bias??Add animation.Show good example and bad example in slide:What did you expect?How awesome was that?

NEW SLIDE?How to act around partcipants? Funny, formal, taciturn, serious? Positivie? Kind.Respectvul and attentive and BE YOURSELF

Exercise caution when answering and asking questions or giving assists or hintsAsking questions: use neutral words, avoid adjectives and adverbs.If you do use them, use both extremes easy and difficult, fast and slow, Responding to questions:In general, dont answer questions directly. That would be giving them as assist.Turn question around and ask them:What do you think it should do?What do you think it will return?69

Interacting with Participants InterruptingIn general, avoid interrupting the participant. But if you must:Remind them to think aloudPrompt them to clarify or reread tasksProbe for information about their actions or expectations (Say more about that)

70

WHAT IF PARTICIPANT GOES OFF TASK??

Let them do the tasks at their own pace and in their own way

But you may need to:Remind them to think aloudPrompt for rereading task or reading task more carefullyProbes would you have used online help? What would you have searched for?What did you just do? Why did you do that? what were you thinking? What were you expecting?70

Interacting with Participants Using Active Listening SkillsUse nonverbal communication and minimal encouragersAllow for silence, struggle, and uncertaintyCheck expectationsValidate your understanding

71

DONT PRESENTActive listening skillsNonverbal communication, minimal encouragers, validation, Allowing for silence, struggle, and uncertaintyChecking expectations

71

Providing Product HelpProvide minimal helpOnly when and if participant requests help

Unless you are testing Help

72

DONT PRESENTInclude in demo?Participants should initiate requests for help, they indicate theyd search or bring up help

Give minimal amount of help no extra info especially if it relates to a later taskDefine ahead and agree with your team - does help = task failure

If you are testing help, then thats another story.72

Concluding the TestClarifying questions from you or observersPost test interviewsWrap up and closure73

Post test questions - invite observers to ask questions or to pass you questions for participant if you have time.

Post test Interview:anything else to say about what theyve seen? ask for overall impression, Likert scales often used here.

Wrap up and closure clear up users confusion, answer questions? Making the participant leave feeling ok not totally confused.Thank them and give them email or business card in case they want to give you additional feedback as they use your product.

73

What do you do when the user..Fails a task:Exceeded time allowedUnexpected or error stateIs not talking or talking too muchIs the wrong user profileTalks too much about a different feature or asks to see other features

74

Do NOT COVERIn case of task failure:Explain that they did not fail, system/product has failed.Guidelines for determining when a task has failed.

did they timeout?Dumas has some guidelines if estimated time is 10 minutes, give 2x allote time.

Error state or wrong results:

if you have to reset something on screen or protototype, have participant look away first

74

DisneyParks.com Activity Moderation Demo

75

Show video HEREHALF ROOM OBSEVE MODERATION AND HALF TASKS20 minutesPass Out Moderation checklist, markers and post-its for those to observe tasksAlso Debbie begin to setup a laptop that she plugs in and then readies to switch display.

This is where they observe moderating and take notes related to the techniques of moderatingAlso, ask them to take notes on Post-Its. Write down anything you notice about the user doing the tasks.Are they struggling? Where? How? What are they saying? Did they accomplish the tasks?show an example of some post its notes next.

75

During a Real TestObservers take notes on the content of the test, not the moderators performance.

Moderators primary responsibility is to facilitate the test and ensure the comfort of the participant not take notes.

76

It varies across moderators how many notes they take during a test.The most important thing is that you maintain a good rapport with the participant.Taking notes can be very distracting and disruptive to the participant.So until and unless you learn to do it smoothly, its best left to the observers.

When I first started out, I took extensive notes because I was afraid wed miss details.but as I tested more, I started to trust that my observers would capture the important details.

If observers and note takers are in the same room as you, you can look around to see if observers are catching a particular note.You can also come to learn how your developers take notes and know what types of info theyll likely capture and what they never seem to notice.

One observation per post-itIndicate direct quotes if it is an observer thought or ideaIndicate questions to ask moderator/participant

76

DisneyParks.com Activity Moderation Demo and DebriefTask 1: Observe moderators techniquesUse the checklist to takes notes.

Task 2: Capture observational data on post-itsWhat you see the participant doingWhat you hear the participant sayingWhat you notice the system doing in response to the participants actions

77

Task 1 8 minutesWhat did you notice me doing?What did I do well?

For second part of video, take notes on both my moderation technique, but also on the test itself.

Best practices to point out & discussAssistRedirecting participant back to taskAnswering question with questionAsking about participants expectationsPrompt think aloudHelping participant know when they are done with a task.Debbie will ask some questions and Michelle will reflect back as question.Michelle to provide an assist. (if we cover that slide)Michelle to ask about expectations Debbie has.Debbie to stop thinking aloud during a part of the task and Michelle has to promptMichelle to use active listening skills - minimal encouragers and check a perception.Michelle to ask Debbie a question using both extremes.

77

Moderation Demo

78

Moderation SummaryYou dont need a fancy usability lab to test!Brief observers on etiquette and note taking.Make participants feel comfortable, inform them of their rights, and assure them you are testing the product, not them!Let participants set the pace. Interrupt only minimally and be careful not to bias the test!

79

What we just covered key takeawaysSee resources later.

Add more detail for each section ie conducting the test.79

Analyzing and Summarizing Results

80

80

Goals of Analysis and SummaryIdentify the problems that were uncovered in the studyDevelop a common understandingImpact and improve designConvince team to make changes

81

81

Analyzing Test DataAnalysis techniques vary depending on many factors: Culture/organizationTest objectives and requirementsYour test goals82

Describes how to deal with the data we collect during usability testing, and what kind of analysis is needed to make recommendations, prioritize changes and deliver effective reports

Work with your team or clients to determine how best to decide and communicate recommendations and prioritiesAgree on criteria no of issues reported on a specific feature, severity of issues found (agree on severity scale),

Analyzing results collaboratively is what we recommend to help influence the design.Otherwise you analyze by yourself, prepare a report, then pitch it to the team. 82

Our Recommendation:Affinity DiagrammingShare and consolidate observation notes.Categorize notes into meaningful groups.Prioritize by voting on top issues.Record and communicate results to team and stakeholders.

83

Getting team members to observe and take notes during usability sessions is incomparably valuable.Taking notes on post-it notes is a simple and effective tool for data collection.Affinity diagramming is a flexible method to process data quickly.Quick and simple data analysis may be all you ever need to do.

83

DisneyParks.com Activity Affinitize the ResultsPost all notes together, uncategorized. Toss duplicates.Categorize notes into meaningful groups.When you see 2 that are related, take both notes and post together.As soon as you put two notes together, title the category on a 3 x 3 post-it note.Prioritize the feedback (by category or individual note).

84

We have a whole training on this method, but for today wed like to give you a few minutes to try it out together with the notes you took earlier.Powerful when done as a group. Observers can share their perceptions.

Small group or large group affinity session.Depending on time Break into small groups and have them affinitize some of the notes they took during the second live demo. 15 20 minutes

Categorize notes into meaningful groups.Find two related and go over to another wall.

You may not criticize anyone elses placement of a note or category name; you MAY quietly move a note to the correct location or rename a categoryBreak up large categories (> 7 items)

There is a whole workshop on this mathed alone. But I encourage you to try it out.

84

Reporting ResultsReporting techniques vary depending on time, culture, and resources. Some examples include:Video clipsFormal detailed reportExcel spreadsheetExecutive summaryPowerPoint presentation

85

Work with your teams to figure out the best techniques for reporting results.This may depend on the type of project size of project, your companyTime, re

What you do will depend on the needs and resources of your organization and projectSometimes just a small summary report is sufficient.Sometimes videos, extensive analysis of every observation with times on task is donesources, 85

Analysis and Reporting SummaryGoals of analysis:Develop a common understandingIdentify the problems Convince team to make changesImprove a designScale reports to fit.Try Affinity diagramming!

86

86

Conclusion

87

may not need the section break.87

Just Test!Show your product to users and get feedback.Use these materials to refine your testing methods.You dont need to be an expert.You dont need a state-of-the-art lab.You dont need elaborate protocols and test scripts.The more you test, the better youll get.

88

Workshop Conclusion message here.

Remember just do itThe more you do it the better you getThere is a lot of info you can use to hone your skills,But just start testing

88

Primary ResourcesA Practical Guide to Usability Testing

Joseph S. Dumas& Janice C. Redish

Moderating Usability Tests

Joseph S. Dumas& Beth Loring

89

So much we didnt cover, here are more resources for you to cover89

Usability TestingHandbook of Usability Testing: How to Plan, Design & Conduct Effective Usability Tests. Jeffrey Rubin & Dana Chisnell Usability 101: Introduction to Usability. Jakob Nielsen http://www.useit.com/alertbox/20030825.htmlRocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. Steve KrugBeyond the Usability Lab: Conducting Large-scale Online User Experience Studies. Thomas Tullis, Donna Tedesco & William AlbertMeasuring the User Experience. Tom Tullis & Bill AlbertPaper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces. Carolyn SnyderUsability Testing Essentials: Ready, SetTest. Carol M. BarnumUsability Testing: http://www.usability.gov/methods/test_refine/learnusa/index.htmlSTC Website: Usability Toolkit: http://www.stcsig.org/usability/resources/toolkit/toolkit.htmlCollection of examples and templates for various phases of testing processUsing the RITE method to improve products; a definition and a case study. Michael C. Medlock, Dennis Wixon, Mark Terrano, Ramon L. Romero, Bill Fulton http://www.computingscience.nl/docs/vakken/musy/RITE.pdf90

90

Testing ToolsRecordingMorae http://www.techsmith.com/morae.htmlCamtasia http://www.techsmith.com/camtasia.htmlEyetrackingSMI http://www.smivision.com/en/gaze-and-eye-tracking-systems/products/overview.htmlTobii http://www.tobii.com/en/eye-tracking-research/global/products/software/Unmoderated testingUser Zoom www.userzoom.comLoop 11 www.loop11.comUsabilla usabilla.com

91

91

RecruitingRecruiting Test Participants for Usability Studies. Jakob Nielsen http://www.useit.com/alertbox/20030120.html

234 Tips and Tricks for Recruiting Users as Participants in Usability Studies. Deborah Hinderer Sova and Jakob Nielsen http://www.nngroup.com/reports/tips/recruiting/234_recruiting_tips.pdf

92

Number of ParticipantsWhy you only need to test with 5 users. Jakob Nielsen http://www.useit.com/alertbox/20000319.htmlQuantitative Studies: How Many Users to Test. Jakob Nielsen http://www.useit.com/alertbox/quantitative_testing.htmlLewis, J.R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors 36, 368-378.Spool, J. & Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. Extended abstracts of CHI 2001, 285-286.Virzi, R. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors 34, 457-486.93

93

Recruiting Service Providers and ToolsSTC Community website: Usability Lab Rentals and Participant Recruiting http://www.stcsig.org/usability/topics/recruiting-firms.html#USEthnio http://ethn.io/Alpha Buzz http://www.alphabuzzgroup.com/services.htmUsability Works, Dana Chisnell http://www.usabilityworks.net/

94

Thank you!Contact us with any questions

[email protected]@mathworks.com

95

95


Recommended