Human-Computer InteractionHuman Computer Interaction

Evaluation Techniques

Human-Computer Interaction

Human-Computer InteractionWhat is evaluation?The role of evaluation:We still need to assess our designs and test our systems to ensure that they actually behave as we expect and meet user requirements.

Human-Computer Interaction

Human-Computer InteractionWhat is evaluation? (Cont.)Evaluation should not be thought of as a single phase in the design process.Evaluation should occur throughout the design life cycle, with the results of the evaluation feeding back into modifications to the design.It is not usually possible to perform extensive experimental testing continuously throughout the design, but analytical( ) and informal techniques can and should be used.

Human-Computer Interaction

Human-Computer InteractionBroad headings on evaluation techniquesWe will consider evaluation techniques under two broad headings( ):

Evaluation Through Expert Analysis.Evaluation Through User Participation.

Human-Computer Interaction

Human-Computer InteractionGoals of EvaluationEvaluation has three main goals:To assess extent( ) and accessibility of the systems functionality.

To assess users experience of the interaction.

To identify specific problems with the system.

Human-Computer Interaction

Human-Computer InteractionGoals of Evaluation (Cont.)To assess extent and accessibility of the systems functionality:The systems functionality is important in that it must accord with the users requirements.

So the design of the system should enable users to performed their intended tasks more easily.

The use of the system must be matching the users expectations of the task.

Human-Computer Interaction

Human-Computer Interaction

To assess users experience of the interaction:This includes considering aspects such as:How easy the system is to learn.Its usability.The users satisfaction with it.The users enjoyment and emotional response.

Goals of Evaluation (Cont.)

Human-Computer Interaction

Human-Computer Interaction

To identify specific problems with the system:Unexpected results -> confusion amongst users.

Related to both the functionality and usability of the design.Goals of Evaluation (Cont.)

Human-Computer Interaction

Human-Computer InteractionObjectives of User Interface Evaluation Key objective of both UI design and evaluation:

Minimize malfunctions

Key reason for focusing on evaluation:

Without it, the designer would be working blindfold.Designers wouldnt really know whether they are solving customers problems in the most productive way.

Human-Computer Interaction

Human-Computer InteractionEvaluation TechniquesEvaluation:Tests usability and functionality of system.Evaluates both design and implementation.Should be considered at all stages in the design life cycle.

But, in order for evaluation to give feedback to designers......we must understand why a malfunction occurs.

Malfunction analysis:Determine why a malfunction occurs.Determine how to eliminate malfunctions.

Human-Computer Interaction

Human-Computer InteractionOverview of Interface Evaluation Methods Three types of methods:Passive evaluation.Active evaluation.Predictive evaluation (usability inspections). All types of methods useful for optimal results.Used in parallel.All attempt to prevent malfunctions.

Human-Computer Interaction

Human-Computer Interaction1. Passive evaluation Performed while prototyping in a test.

Does not actively seek malfunctions.Only finds them when they happen to occur.Infrequent malfunctions may not be found.

Generally requires realistic use of a system.Users become frustrated( ) with malfunctions.

Human-Computer Interaction

Human-Computer Interaction1. Passive evaluation. (Cont.)Gathering Information:

Problem report monitoring:Users should have an easy way to register their frustration / suggestions.

Best if integrated with software.

Human-Computer Interaction

Human-Computer Interaction1. Passive evaluation (Cont.)Automatic software logs.Can gather much data about usage:Command frequency.Error frequency.Undone operations (a sign of malfunctions).

Logs can be taken of:Just keystrokes, mouse clicks.Full details of interaction.

Human-Computer Interaction

Human-Computer Interaction1. Passive evaluation (Cont.)Questionnaires: Useful to obtain statistical data from large numbers of users.Proper statistical means( ) are needed to analyze results.Gathers subjective data about importance of malfunction.Less frequent malfunctions may be more important.Users can prioritize needed improvements.

Human-Computer Interaction

Human-Computer InteractionQuestionnairesSet of fixed questions given to users.

Limit on number of questions:Very hard to phrase questions well.Questions can be closed- or open-ended.

Advantages:Quick and reaches large user group.

Human-Computer Interaction

Open-ended questions Open-ended questions are those questions that will solicit ( ) additional information from the inquirer( ). Examples:How may/can I help you? Where have you looked already? What aspect are you looking for? What kind of information are you looking for? What would you like to know about [topic]? When you say [topic], what do you mean?

*Human-Computer Interaction

Closed ended questionsClosed ended questions are those questions, which can be answered finitely by either yes or no. Examples:a. Can I help you?b. May I help you?c. Can you give me more information?d. Have you searched elsewhere?e. Can you describe the kind of information you want?f. Can you give me an example?g. Could you be more specific?h. Are you looking for [topic]?

*Human-Computer Interaction

Human-Computer Interaction2. Active evaluationActively study specific actions performed by users.Performed when prototyping done.Gathering Information:Experiments & usability engineering:Prove hypothesis about measurable attributes of one or more UIs.e.g. speed/learning/accuracy/frustrationIn usability engineering test against goal of system.Hard to control for all variables.

Observation sessions (Videotaped Evaluation).Also called interpretive evaluation. Study active use on realistic tasks.

Human-Computer Interaction

Human-Computer Interaction3. Predictive evaluationStudies of system by experts rather than users.Performed when UI is specified (useful even before prototype developed).Can eliminate many malfunctions before users ever see software.Also called usability inspections.( )Gathering Information:Heuristic( ) evaluation.Based on a UI design principle document.Analyze whether each guideline is commit to( ) in the context of the task and users.Can also look at commit to standards.Cognitive( ) walkthroughs.Step-by-step analysis of:Steps in task being performed.Goals users form to perform these tasks.How system leads user through tasks.

Human-Computer Interaction

Human-Computer InteractionSummary of evaluation techniques

Human-Computer Interaction

Human-Computer InteractionEvaluating DesignsThe evaluation should occur throughout the design process.

These methods can be used at any stage in the development process from a design specification, through storyboards and prototypes, to full implementations, making them: Flexible evaluation approaches.

Human-Computer Interaction

Human-Computer Interaction1. Videotaped Evaluation A software engineer studies users who are actively using the user interface:To observe what problems they have.The sessions are videotaped.Can be done in users environment.

Activities of the user:Preferably talks to him/her-self as if alone in a room.This process is called co-operative evaluation when the software engineering and user talk to each other.

Human-Computer Interaction

Human-Computer InteractionThe importance of video:With using it, you can see what you want to see from the system.You can repeatedly analyze, looking for different problems.

Tips for using video:Several cameras are useful.Software is available to help analyse video by dividing into segments and labelling the segments.1. Videotaped Evaluation (Cont.)

Human-Computer Interaction

Human-Computer Interaction2. ExperimentsPick a set of subjects (users):A good mix to avoid biases( ).A sufficient number to get statistical significance (avoid random happenings effect results).Pick variables to test:Variables Manipulated to produce different conditions:Should not have too many.They should not affect each other too much.Make sure there are no hidden variables.Develop a hypothesis:A prediction of the outcome.Aim of experiment is to show this is correct.

Human-Computer Interaction

Human-Computer InteractionVariables

Independent variable (IV):Characteristics changed to produce different conditions.e.g. interface style, number of menu items

Dependent variable (DV):Characteristics measured in the experiment.e.g. time taken, number of errors.

Human-Computer Interaction

Human-Computer Interaction3. Heuristic EvaluationDeveloped by Jakob Nielsen & Rolf Molich in the early 1990s.Helps find usability problems in a UI design.a heuristic is based on UI guideline.usability criteria (heuristics) are identified.design examined by experts to see if these are violated

Human-Computer Interaction

Human-Computer InteractionHeuristic Evaluation (cont.)Evaluators goes through UI several times.Inspects() various dialogue elements.Consider and compares with list of usability principles.Usability principles:Nielsens heuristics.Competitive analysis & user testing of existing products.Use violations to redesign/fix problems.

Human-Computer Interaction

Human-Computer Interaction1. Heuristic EvaluationA type of predictive( ) evaluation:Use HCI experts as reviewers instead of users.Benefits of predictive evaluation:The experts know what problems to look for.Can be done before system is built.Experts give prescriptive( ) feedback.Important points about predictive evaluation:Reviewers should be independent of designers.Reviewers should have experience in both the application domain and HCI.Include several experts to avoid bias.Experts must know classes of users.Beware: Novices can do some very bizarre( ) things that experts may not anticipate.

Human-Computer Interaction

Human-Computer Interaction1. Heuristic Evaluation

Example heuristics:System behaviour is predictable.System behaviour is consistent.Feedback is predictable.

Heuristics( , ) being developed for mobile devices, virtual worlds, etc

Human-Computer Interaction

Human-Computer InteractionNielsens ten heuristics are:Visibility of system status.Match between system and the real world.User control and freedom.Consistency and standards.Error prevention.Recognition( ) rather than recall( ).Flexibility and efficiency of use.Aesthetic( ) and minimalist( ) design.Help users recognize, diagnose( ) and recover from errors.Help and documentation.

Human-Computer Interaction

Human-Computer InteractionH-1: Visibility of system statusProvide feedback.Keep users informed about what is going on.example: pay attention to response time.0.1 sec: no special indicators needed.1.0 sec: user tends to lose track of data.10 sec: max. duration if user to stay focused on 1 action.For longer delays, use percent-done progress bars.

Human-Computer Interaction

H-1: Visibility of system status Continuously inform the user about:What it is doing.How it is interpreting the users input.User should always be aware of what is going on. Whats it doing?Time for coffee.

Human-Computer InteractionWhat did I select?What mode am I in now?How is the system interpreting my actions?Microsoft PaintH-1: Visibility of system status

Human-Computer Interaction

Human-Computer InteractionBe as specific as possible, based on users input.

Best within the context of the action.H-1: Visibility of system status

Human-Computer Interaction

Human-Computer InteractionDrawing Board LTMultiple files being copied, but feedback is file by file.H-1: Visibility of system status

Human-Computer Interaction

Dealing with long delays.Cursors.For short transactions. Percent done dialogs.Time left.Estimated time. RandomFor unknown times.H-1: Visibility of system status

Human-Computer InteractionH-2: Match between system and real world

Dragging disk to trash.Should delete it, not eject it.Speak the users language.Follow real world conventions.

Human-Computer Interaction

Human-Computer InteractionH-2: Match between system and real world

Human-Computer Interaction

H-2: Match between system and real worldTerminology based on users language for task.e.g. withdrawing money from a bank machine.

Use meaningful mnemonics, icons & abbreviations.eg File / SaveCtrl + S (abbreviation)Alt FS (mnemonic for menu action) (tooltip icon)

Human-Computer InteractionH-3: User control and freedomWizardsMust respond to 1 Q before going to nextGood for beginnersHave N versionsSketchUp 6

exits for mistaken choices, undo, redo.Dont force down fixed paths.

Human-Computer Interaction

How do I get out of this?H-3: User control and freedom

H-3: User control and freedomUsers dont like to feel trapped by the computer!Should offer an easy way out of as many situations as possible. Strategies:Cancel button (for dialogs waiting for user input).Universal Undo (can get back to previous state).Quit (for leaving the program at any time).Defaults (for restoring a property sheet).

Human-Computer InteractionH-4: Consistency & standards

Human-Computer Interaction

H-4: Consistency & standardsConsistent syntax of input.

Consist language and graphics.Same visual appearance across the system. Same information/controls in same location on all windows. Consist effectsCommands, actions have same effect in equivalent situationsPredictability.

Human-Computer InteractionH-4: Consistency & standardsThese are labels with a raised appearance.

Is it any surprise that people try and click on them?

Human-Computer Interaction

Human-Computer InteractionWhy

Human-Computer Interaction

Human-Computer InteractionFrom Peachpit website

Human-Computer Interaction

Human-Computer InteractionFrom Peachpit website

Human-Computer Interaction

Human-Computer InteractionH-5: Error prevention

Make it difficult to make errors.

Even better than good error message is a careful design that prevents a problem from occurring in the first place.

Human-Computer Interaction

Human-Computer InteractionH-6: Recognition rather than recallMake objects, actions, options, and directions visible.The user should not have to remember information from one part of the dialog to another.

Human-Computer Interaction

H-6: Recognition rather than recallComputers good at remembering, people are not!Promote recognition over recall.Menus, icons, choice dialog boxes vs commands, field formats.Relies on visibility of objects to the user.

Human-Computer InteractionH-6: Recognition rather than recallGives input format, example and default.

Human-Computer Interaction

Human-Computer InteractionH-7: Flexibility and efficiency of use

Accelerators for experts e.g., keyboard shortcutsAllow users to tailor frequent actionse.g., macrosCustomized user profiles on the web

Human-Computer Interaction

Human-Computer InteractionH-8: Aesthetic( ) and minimalist( ) designNo irrelevant information in dialogues.

Human-Computer Interaction

Human-Computer InteractionH-9: Help users recognize, diagnose, and recover from errorsError messages in plain language.Precisely indicate the problem.Constructively suggest a solution.

Human-Computer Interaction

People will make errors! Errors we makeMistakesConscious( ) actions lead to an error instead of correct solution.SlipsUnconscious behaviour gets misdirected in route to satisfy a goal.H-9: Help users recognize, diagnose, and recover from errors

What is error 15762?H2-9: Help users recognize, diagnose, and recover from errors

H2-9: Help users recognize, diagnose, and recover from errorsProvide meaningful error messages:Error messages should be in the users task language.Dont make people feel stupid:-Try again!.Error 25.Cannot open this document.Cannot open chapter 5 because the application Microsoft Word is not on your system.Cannot open chapter 5 because the application Microsoft Word is not on your system. Open it with Teachtext instead?.

H2-9: Help users recognize, diagnose, and recover from errorsPrevent errors:Try to make errors impossible.Modern widgets: can only enter legal data.

Human-Computer InteractionH2-10: Help and documentationEasy to search.Focused on the users task.List concrete steps to carry out.Not too large.

Human-Computer Interaction

H2-10: Help and documentationHelp is not a replacement for bad design! Simple systems:Use minimal instructions.

Most other systems:Simple things should be simple.Learning path for advanced features.

Documentation and how it is usedMany users do not read manuals.Usually used when users are in some kind of panic.paper manuals unavailable in many businesses! e.g. single copy locked away in system administrators office.online documentation better.online help specific to current context.

Sometimes used for quick reference.list of shortcuts ...

Human-Computer Interaction2. Cognitive WalkthroughProposed by Polson and colleagues.Evaluates design on how well it supports user in learning task.Focus on ease of learning.

Usually performed by expert in cognitive psychology.Expert is told the assumptions about user population, context of use, task details.

expert walks though design to identify potential problems using psychological principles.

Human-Computer Interaction

Human-Computer Interaction2. Cognitive WalkthroughWalkthroughs require a detailed review of a sequence of actions.

In the cognitive walkthrough, the sequence of actions refers to the steps that an interface will require a user to perform in order to accomplish some task.

Human-Computer Interaction

Human-Computer InteractionWalkthrough needs four things:

A specification or prototype of the system.It doesn't have to be complete, but it should be fairly detailed.A description of the task the user is to perform on the system.

Written list of the actions needed to complete the task with the proposed system.

An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.

Human-Computer Interaction

Human-Computer InteractionFour QuestionsThe evaluator will answer these questions:

Is the effect of the action the same as the users goal at that point?

Will users see that the action is available?

Once users have found the correct action, will they know it is the one they need?

After the action is taken, will users understand the feedback they get?

Human-Computer Interaction

Human-Computer InteractionCognitive WalkthroughFor each task walkthrough considers:What impact will interaction have on user?What processes are required?What learning problems may occur?

Analysis focuses on goals and knowledge: Does the design lead the user to generate the correct goals?

Human-Computer Interaction

Human-Computer InteractionSteps of a Cognitive WalkthroughDefine inputs.Convene analysts.Step through action sequences for each task.Record important information.Modify UI.

Human-Computer Interaction

Human-Computer InteractionDefine Inputs - ExampleTask: Move an application to a new folder or driveWho: Win 2003 userInterface: Win 2003 desktopFolder containing desired app. is open.Destination folder/drive is visible.Action sequence...

Human-Computer Interaction

Human-Computer InteractionAction SequenceMove mouse to app. icon.Right mouse down on app. icon:Result: App. icon highlights.Failure: The user may not know that the right mouse button is the proper one to use.Success?: Highlighting shows something happened, but was it the right thing?

Human-Computer Interaction

Human-Computer InteractionAction Sequence (contd)Release mouse button:Result: Menu appears: Cut, Copy, Create Shortcut, Cancel.Success: User is prompted for next action.Move mouse to Cut:Result: Selection highlights.Success: Standard GUI menu interaction.

Human-Computer Interaction

Human-Computer InteractionAction Sequence (contd)Move mouse to destination icon:Result:App. icon follows mouse.Destination icon highlights when mouse reaches it.Success: Dragging is intuitive (and common in GUIs) for moving. The feedback is appropriate.

Human-Computer Interaction

Human-Computer InteractionAction Sequence (contd)Click mouse button:Result:App. icon disappears from under the mouse.App. icon disappears from original folder.App. icon appears in destination folder.Success: Standard GUI menu selection. Feedback shows desired goal was accomplished.

Human-Computer Interaction

Cognitive Walkthrough Example: Human-Computer Interactionstep1: identify taskstep 2: identify action sequence for taskuser action: Press the timed record buttonsystem display: Display moves to timer mode. Flashing cursor appears after start.step 3: perform walkthroughfor each action answer the following questionsIs the effect of the action the same as the users goal at that point?Will users see that the action is available?Once users have found the correct action, will they know it is the one they need?After the action is taken, will users understand the feedback they get?Might find a potential usability problem relating to icon on timed record button.

Human-Computer Interaction

Human-Computer InteractionExample:Programming a VCR by remote control1324657098Time 21:45Channel 31324657098 1Start:End:Channel:Date:

Human-Computer Interaction

Human-Computer InteractionExample: VCR (Cont.)Task: Program the video time-record a program starting at 12.00 and finishing at 13.30 on channel 2 on 23 February 2008.Who: Assume user is familiar with VCRs but not with this particular design.Action Sequence:Users action ( UA ).Systems Display ( SD ).

Human-Computer Interaction

Human-Computer InteractionUA 1: Press the timed record buttonSD 1: Display moves to timer mode. Flashing cursor appears after start:UA 2: Press digits 1 2 0 0SD 2: Each digit is displayed as typed and flashing cursor moves to next positionUA 3: Press the timed record buttonSD 3: Flashing cursor moves to end:UA 4: Press digits 1 3 3 0SD 4: Each digit is displayed as typed and flashing cursor moves to next positionUA 5: Press the timed record buttonSD 5: Flashing cursor moves to channelUA 6: Press digit 2SD 6: Digit is displayed as typed and flashing cursor moves to next positionUA 7: Press timed record buttonSD 7: Flashing cursor moves to dateUA 8: Press digits 2 3 0 2 0 8SD 8: Each digit is displayed as typed and flashing cursor moves to next position.UA 9: Press the timed recordSD 9: Stream number in top right-hand corner of display flashesUA 10: Press the transmit buttonSD 10: Details are transmitted to video player and display returns to normal mode

Human-Computer Interaction

Human-Computer InteractionWe must answer the four questions and tell a story about the usability of the system:UA 1: Press the timed record buttonQ1) is the effect of the action the same as the users goal at that point?The timed record button initiates timer programming. It is reasonable to assume that a user familiar with VCRs would be trying to do this as his first goal.Q2) Will users see that the action is available?The timed record button is visible on the remote controller.Example: VCR (Cont.)

Human-Computer Interaction

Human-Computer InteractionQ3) once users have found the correct action, will they know it is the one they need?It is not clear which button is the timed record button. The icon of a clock is a possible candidate but this could be interpreted as a button to change the time. Other possible candidates might be the fourth button down on the left or the filled circle ( associated with record ). In fact, the icon of the clock is the correct choice but it is quite possible that the user would fail at this point. This identify a potential usability problem.Q4) After the action is taken, will users understand the feedback they get?Once the action is taken the display changes to the timed record mode and shows familiar headings ( start, end, channel, date ). It is reasonable to assume that the user would recognize these as indicating successful completion of the first action.Example: VCR (Cont.)

Human-Computer Interaction

Human-Computer InteractionEvaluation through user participationSome of the techniques we have considered so far concentrate on evaluating a design or system through analysis by the designer, or an expert evaluator, rather than testing with actual users.User participation in evaluation tends to occur in the later stages of development when there is at least a working prototype of the system in place.

Human-Computer Interaction

Human-Computer InteractionStyles of evaluationThere are two distinct evaluation styles:Those performed under laboratory conditions.Those conducted in the work environment or in the field.

Human-Computer Interaction

Human-Computer InteractionLaboratory StudiesIn the first type of evaluation studies, users are taken out of their normal work environment to take part in controlled tests, often in a specialist usability laboratory.This approach has a number of benefits and disadvantages.A well equipped usability laboratory may contain sophisticated audio/visual recording and analysis facilities.

Human-Computer Interaction

Human-Computer InteractionLaboratory Studies (Cont.)There are some situations where laboratory observation is the only option.e.g. if the system is to be located in a dangerous or remote location, such as a space station.Some very constrained single user tasks may be adequately performed in a laboratory.Want to manipulate the context in order to uncover problems or observe less used procedures.Want to compare alternative designs within a controlled context.

Human-Computer Interaction

Human-Computer InteractionField StudiesThe second type of evaluation takes the designer or evaluator out into the users work environment in order to observe the system in action.High levels of ambient noise, greater levels of movement and constant interruptions, such as phone calls, all make field observation difficult.

Human-Computer Interaction

Human-Computer InteractionField studies (Cont.)The very open nature of this situation means that you will observe interactions between systems and between individuals that would have been missed in a laboratory study.The context is retained and you are seeing the user in his natural environment .

Human-Computer Interaction

********************An independent variable is the presumed cause, whereas the dependent variable is the presumed effect.

************************************************