Lecture 13

Evaluation techniques Part 1

1

Today’s Lecture Introduction to Evaluation

Definition, concepts, needsGoals of evaluation

Evaluation DesignsCognitive WalkthroughHeuristic EvaluationModel based EvaluationReview-based evaluation

2

What is “evaluation”? The simple answer, dictionary

definition:e + valuer =

to establish the worth or value of.

Evaluation can be described as the systematic investigation of the merit, worth, or significance of any “object”

3

What is Evaluation?“Program evaluation is the systematic

collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development…

4

What is Evaluation? …Program evaluation does not occur in a

vacuum and is influenced by real-world constraints. Evaluation should be practical and feasible and must be conducted within the confines of resources, time, and political context…

5

What is Evaluation?…Moreover, evaluation should serve a

useful purpose, be conducted in an ethical manner, and produce accurate findings. Evaluation findings should be used to make decisions about program implementation and to improve program effectiveness.” (Centers for Disease Control, 2001, p. 6)

6

We need Evaluation.. In order to assess our designs and test our

systems to ensure that they actually behave as we expect and meet user requirements. This is the role of evaluation. It is not a single phase process.

7

When to evaluate? Ideally, evaluation should occur

throughout the design life cycle, with the results of the evaluation feeding back into modifications to the design.

8

When to Evaluate? Clearly, it is not usually possible to

perform extensive experimental testing continuously throughout the design, but analytic and informal techniques can and should be used.

9

Evaluation It is much easier to change a design in the

early stages of development than in the later stages.

Main theme is to evaluate the system for its correctness.

10

Goals of Evaluation Assess extent of system functionality

Assess effect of interface on user

Identify specific problems

11

Evaluation through Expert Analysis

In particular, the first evaluation of a system should ideally be performed before any implementation work has started. If the design itself can be evaluated,

expensive mistakes can be avoided, since the design can be altered prior to any major resource commitments.

12

Evaluation through Expert Analysis

Typically, the later in the design process that an error is discovered, the more costly it is to put right and, therefore, the less likely it is to be rectified. It can be expensive to carry out user testing at

regular intervals during the design process, and it can be difficult to get an accurate assessment of the experience of interaction from incomplete designs and prototypes.

13

Evaluation The basic intention of evaluation is to

identify any areas that are likely to cause difficulties because they violate known cognitive principles, or ignore accepted empirical results.

14

Evaluating Designs

Evaluation designs include Cognitive Walkthrough Heuristic Evaluation Model based Evaluation Review-based evaluation

15

Cognitive Walkthrough

Proposed by Polson et al.Evaluates design on how well it supports user

in learning taskUsually performed by expert in cognitive

psychologyExpert ‘walks through’ design to identify

potential problems using psychological principles

Forms used to guide analysis

16

Cognitive Walkthrough (ctd)

For each task walkthrough considerswhat impact will interaction have on user?what cognitive processes are required?what learning problems may occur?

Analysis focuses on goals and knowledge: does the design lead the user to generate the correct goals?

17

Cognitive walkthrough To do walk through four things are needed.

A specification or prototype of the system. It doesn’t have to be complete, but it should be fairly detailed. Details such as the location and wording for a menu can make a big difference.

A description of the task the user is to perform on the system. This should be a representative task that most users will want to do.

18

Cognitive WalkthroughA complete, written list of the actions needed

to complete the task with the proposed system.

An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.

19

Cognitive Walkthrough The evaluators try to answer the following four questions for each step in

the action sequence. Is the effect of the action the same as the user’s goal at that point?Will users see that the action is available? Once users have found the correct action, will they know it is the one they need?After the action is taken, will users understand the feedback they get?

20

Cognitive Walkthrough Is the effect of the action the same as the

user’s goal at that point? Each user action will have a specific effect

within the system. Is this effect the same as what the user is trying to achieve at this point?

For example, if the effect of the actionis to save a document, is ‘saving a document’ what the user wants to do?

21

Cognitive Walkthrough Will users see that the action is

available? Will users see the button or menu item,

for example, that is used to produce the action? This is not asking whether theywill recognize that the button is the one they want. This is merely asking whether it is visible to them at the time when they will need to use it.

22

Cognitive Walkthrough Once users have found the correct

action, will they know it is the one they need? It is one thing for a button or menu

item to be visible, but will the user recognize that it is the one he is looking for tocomplete his task?

It is about whether its meaning and effect of action is clear.

23

Cognitive Walkthrough After the action is taken, will users

understand the feedback they get? If you now assume that the user did manage

to achieve the correct action, will he know thathe has done so?

Will the feedback given be sufficient confirmation of what has actually happened?

24

Cognitive Walkthrough It is vital to document the cognitive

walkthrough to keep a record of what is good and what needs improvement in the design.

It is therefore a good idea to producesome standard evaluation forms for the walkthrough.

25

Heuristic Evaluation Proposed by Nielsen and Molich.

usability criteria (heuristics) are identified design examined by experts to see if these are violated

Example heuristics system behaviour is predictable system behaviour is consistent feedback is provided

Heuristic evaluation `debugs' design.

26

Heuristic Evaluation Heuristic is a guideline or general principle

or rule of thumb that can guide a designdecision or be used to critique a decision that has already been made.

It is a method for structuring the critique of a system using a set of relatively simple and general heuristics.

27

Heuristic Evaluation Heuristic evaluation can be performed on

a design specification so it is useful for evaluating early design. it can also be used on prototypes, storyboards

and fully functioning systems. it is often considered a discount usability

technique because it is a cheap approach.

28

Heuristic Evaluation The general idea behind heuristic

evaluation is that several evaluators independently critique a system to come up with potential usability problems.

It is important that there be several of these evaluators and that the evaluations be done independently.

29

Nielsen’s Experience Nielsen’s experience indicates that

between three and five evaluators is sufficient, with five usually resulting in about 75% of the overall usability problems being discovered.

30

Evaluators scale Each evaluator prepares the result based upon

a rating scale. For example 0 = I don’t agree that this is a usability problem at all

1 = Cosmetic problem only: need not be fixed unless extra time is available on project

2 = Minor usability problem: fixing this should be given low priority

31

Evaluator Scale 3 = Major usability problem: important to fix,

so should be given high priority 4 = Usability catastrophe: imperative to fix

this before product can be released (Nielsen)

32

Nielsen’s ten heuristics1. Visibility of system status

2. Match between system and the real world

3. User control and freedom

4. Consistency and standards

5. Error prevention

6. Recognition rather than recall

7. Flexibility and efficiency of use

8. Aesthetic and minimalist design

33

Nielsen’s ten heuristics

9. Help users recognize, diagnose and recover from errors.

10. Help and documentation

34

Heuristic 1 Visibility of system status

Always keep users informed about what is going on, through appropriate feedback within reasonable time.

For example, if a system operation will take some time, give an indication of how long and how much is complete.

35

Heuristic 2 Match between system and the real world.

The system should speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in natural and logical order.

36

Heuristic 3 User control and freedom Users

Often choose system functions by mistakeand need a clearly marked ‘emergency exit’ to leave the unwanted state without having to go through an extended dialog. Support undo and redo.

37

Heuristic 4 Consistency and standards

Users should not have to wonder whether words, situations or actions mean the same thing in different contexts.

Follow platform conventions and accepted standards.

38

Heuristic 5 Error prevention

Make it difficult to make errors. Even better than good error messages is a

careful design that prevents a problem from occurring in the first place.

39

Heuristic 6 Recognition rather than recall

Make objects, actions and options visible. Theuser should not have to remember information from one part of the dialog to another.

Instructions for use of the system should be visible or easily retrievable whenever appropriate.

40

Heuristic 7 Flexibility and efficiency of use

Allow users to tailor frequent actions.Accelerators – unseen by the novice user –

may often speed up the interactionfor the expert user to such an extent that the system can cater to both inexperienced and experienced users.

41

Heuristic 8 Aesthetic and minimalist design

Dialogs should not contain information thatis irrelevant or rarely needed.

Every extra unit of information in a dialog competes with the relevant units of information and diminishes their relative visibility.

42

Heuristic 9 Help users recognize, diagnose and

recover from errors. Error messages should be expressed in

plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

43

Heuristic 10 Help and documentation

Few systems can be used with no instructions so it may be necessary to provide help and documentation.

Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

44

Heuristic Evaluation Once each evaluator has completed their

separate assessment, all of the problemsare collected and the mean severity ratings calculated.

The design team will then determine the ones that are the most important and will receive attention first.

45

Model-based evaluation A third expert-based approach is the use

of models. Certain cognitive and design models

provide a means of combining design specification and evaluation into the same framework.

46

Model Based Evaluation For example, the GOMS (goals, operators,

methods and selection) model predicts user performance with a particular interface and can be used to filter particular design options.

Similarly, lower-level modeling techniques such as the keystroke-level model provide predictions of the time users will take to perform low-level physical tasks.

47

Model Based Evaluation Design methodologies, such as design

rationale, also have a role to play in evaluation at the design stage. Design rationale provides a framework in

which design options can be evaluated. By examining the criteria that are associated

with each option in the design, and the evidence that is provided to support these criteria, informed judgments can be made in the design.

48

Model Based Evaluation Dialog models can also be used to

evaluate dialog sequences for problems, such as unreachable states, circular dialogs and complexity.

Models such as state transition networks are useful for evaluating dialog designs prior to implementation.

49

Using previous studies in evaluation

A final approach to expert evaluation exploits this inheritance, using previousresults as evidence to support (or refute) aspects of the design.

It is expensive to repeat experiments continually and an expert review of relevant literature can avoid the need to do so.

50

Using previous studies.. It should be noted that experimental

results cannot be expected to hold arbitrarily across contexts.

The reviewer must therefore select evidence carefully, noting the experimental design chosen, the population of participants used, the analyses performed and the assumptions made.

51

Review based… For example, an experiment testing

the usability of a particular style of help system using novice participants may notprovide accurate evaluation of a help system designed for expert users.

The review should therefore take account of both the similarities and the differences between the experimental context and the design under consideration.

52

Review-based evaluation Results from the literature used to support or refute parts

of design.

Care needed to ensure results are transferable to new design.

Model-based evaluation

Cognitive models used to filter design optionse.g. GOMS prediction of user performance.

Design rationale can also provide useful evaluation information

53

Summary of today’s Lecture We have covered today several aspects of

evaluation including.Definition of evaluation, concepts, needsGoals of evaluation

Evaluation DesignsCognitive WalkthroughHeuristic EvaluationModel based EvaluationReview-based evaluation

54