HCI 2009-2010

Week 3 - Evaluation

Breaking news…

• Brain computer interaction (Radio 4, 6/10/09, 0720)

• Research relating to mind input.

• Seen as of possible use for people with disabilities

More breaking news…

• … seeking student reps with the aim of achieving one rep in each personal tutor group. We did not get a response from the level 6 IT Systems for Business students so their views will be unheard unless someone comes forward before this week’s deadline

Introduction – Why Evaluate?

• Dix et al (2004) argue that evaluation has these main goals:– Assess the extent and accessibility of the

system’s functionality– Assess users’ experience of the interaction GOW

– Identify any specific problems with the system.

– Save money

Some claims in respect of usability evaluations (courtesy of T Renshaw)

– AOL• Improvements to web architecture following usability

testing increased traffic to second level web areas by 120%

– IBM• Improvement in web usability led to 400% increase in sales • Usability evaluation reduced time to complete task by 9.6

minutes leading to internal savings of $6.8million– Ford

• Usability improvements to accounting system for Ford dealerships removed need for user support with savings of $100,000

– Sun Microsystems • $20K invested in usability can yield returns of $152

million - $7500 per dollar of investment

Two main approaches to evaluation

• To try to achieve these goals there are, it can be argued, 2 broad approaches to evaluation of interactive systems:

• Predictive evaluation

“Expert” evaluation

• Usability testing

Evaluation involving users

PREDICTIVE EVALUATIONTypically we’re here dealing with a design or

prototype system.

Possible techniques:a) Task Action LanguageEvaluates a command language for

consistency and simplicityExamples of command language:

DOSUNIX

Considers number of rules in the grammar needed to achieve a particular task

PREDICTIVE EVALUATION

b) GOMS (Goals, Operators, Methods, Selection rules)Analyses a taskProvides time-based predictions of

how a user will carry them outCognitive Complexity Theory builds

on GOMSpredicts complexity and learnability of

the specified user interface

PREDICTIVE EVALUATIONc) Expert walk-through

User interface expert goes through the system

Identifies what s/he sees as potential problems

Can use independent experts GOW

Possible strengths: quick and easyPossible drawbacks:

May be hard to find an expertTechnique relies on expert’s

judgement

PREDICTIVE EVALUATION

d) Cognitive walk-throughSimilar to expert walk-through, but involves

evaluators going through steps of various tasks to consider usability

• See Dix et al (2004) pp 321/2 (at back of handout)

PREDICTIVE EVALUATION

e) Heuristic evaluationSeveral evaluators independently

critique a system. The system is tested against 10

heuristics – see back of lecture handout (Dix et al pp 324-6)

http://www.useit.com/papers/heuristic/heuristic_list.html

USABILITY TESTING

• Here we are evaluating with actual users.• “In theory there is no difference between

theory and practice. In practice there is.”

Yogi Berra

• We could use a design or paper prototype or a running program.

• Approaches:

USABILITY TESTING

a) “Simulation”• Printed or on-screen version of displays

used in pilot tests (“usability lab testing” – see figure 1 below)

USABILITY TESTING

• Wizard of Oz

In effect, a realistic but simulated (“pretend”) version of the system is evaluated.

USABILITY TESTING

b) Questionnaire• Use them to:

– identify features users find hard, and/or– assess users’ attitude to the design

• Johnson (1992):– questionnaires are poor at identifying usability or

reasons why design is unusable

• however– specific user interface have been developed– and Shneiderman and Plaisant (2005) favour them

USABILITY TESTINGc) Observation• video tape users performing tasks at the interface, and

review for errors etc.• can have two users working together• can invite users to “think aloud” (a “user walkthrough”

(Le Peuple and Scane 2003))• similar technique – “co-operative evaluation” [gow]

– Possible problem – “observer effect”• Aka “Hawthorne effect”

– The knowledge that people are being observed may make them act differently

– And clandestine observation is not allowed

USABILITY TESTING

d) Interviews and group discussions• Can use former to pursue specific issues of concern• The latter can establish the universality of comments

– “Focus groups”

e) on-line consultations• e.g.

– suggestion box– trouble reporting– bulletin board– user news letter

USABILITY TESTING

f) Continuous user performance data collection

• (Software keeps a log of user actions (Shneiderman 2002))

• e.g. data concerning– patterns of system usage– rate of errors– specific errors

USABILITY TESTINGg) Controlled experimentation• Use an experiment to evaluate an implemented design• Steps in designing an HCI experiment:

1) Formulate hypothesisadding video to my CBL will aid learning

2) Develop predictions from the hypothesisthe average user will learn more facts with video than without

3) Choose a means to test the predictionseg a test at the end of the experiment

4) Identify all variables that might affect the results of the experimentability of students, nature of domain, type of machine......

5) Decide which are– independent variables– use of video– dependent variables– performance of the students– variables that need to be controlled out– time of day, type of machine, .....

6) Design the experimental task and method

7) Select participants

8) Decide the experimental design

eg between groups, within groups, longitudinaldata collection method

eg observation, written test,....

means of controlling out confounding variables

9) Decide on statistical or other analysis

10) Carry out pilot study

• As is often the case, there are arguments for and against experiments in HCI.

• See extract in tutorial notes from Hobbs and Moore (1998)

• One approach to experimentation = eye tracking (Web and Renshaw 2008) – next week - vital for the assignment

CHOICE OF EVALUATION TECHNIQUE(S)

• Clearly, then, there are a number of evaluation techniques that can be used.

• How to select the appropriate diet of techniques is an interesting and important question.

• “The choice of evaluation methodology … must arise and be appropriate for the actual problem, research question or product under consideration” (Greenberg and Buxton 2008)

CHOICE OF EVALUATION TECHNIQUE(S)

• See extract in tutorial notes from Hobbs and Moore (1998) for some thoughts on this

• A common view – use both predictive evaluation and usability testing

• When to evaluate?

implementation

Task analysisEvaluationPrototyping

Requirements

Specification

Conceptual Design

The STAR Model

SUMMARY

• A variety of evaluation techniques exist, and are being researched and developed

• Should use one or more to inform future design• Johnson (1992) - treat evaluation results as

over-generous rather than as over-critical• Please get this lecture out when working on

the HCI assignment!– And PIPE, probably!