Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | eleanore-palmer |
View: | 218 times |
Download: | 2 times |
Chapter 10
Evaluation
ObjectivesDefine the role of evaluation
To understand the importance of evaluation
Discuss how developers cope with real-world constraints.
Explain the concepts and terms used to discuss evaluation.
Examine how different techniques are used at different stages of development.
What is Evaluation?
Evaluation is a process by which the interface is tested against the needs and practices of the users.
Evaluation should occur throughout the design life cycle. The results of the evaluation will be used into modifications of the design.
This process helps to ensure that any problem occurs during the lifecycle phases shall be solved earlier and easier to do the modification.
What is Evaluation?
• part of the system are simulated and tested.
• prototypes:
• part of the system, mock-ups, storyboards, paper systems.
EvaluationEvaluation
Analysis phaseAnalysis phase
Design phaseDesign phasePre-production phase(Develop/Implement)Pre-production phase(Develop/Implement)
• analyzing work that has been done in similar fields.
• get feedback from users of previous designed systems.
• changes will be the cheapest.
• develop details of the system.
• measure user performance.
Goals of Evaluation
There are 3 main goals:
To assess the extent of the system’s functionality To assess the effect of the interface on the user To identify any specific problems with the system.
Goals of Evaluation
The system’s functionality must accord with the user’s task requirements
In other words, the design of the system should enable the user to perform the tasks more easily.
The system must function and reachable by the user, this involves matching the use of the system to the user’s expectations of the task.
It is also important to measure the impact of the design on the user.
Star Lifecycle
EVALUATION
IMPLEMENTATIONTASK ANALYSIS/
FUNCTIONAL ANALYSIS
PROTOTYPINGREQUIREMENTSPECIFICATION
CONCEPTUAL DESIGN/FORMAL DESIGN
The Star Life Cycle- adapted from Hix and Hartson, 1993
Styles of Evaluation
There are two main styles:
Evaluation performed under laboratory conditions Evaluation performed in the work environment or
‘in the field’
Styles of EvaluationLaboratory studies In some cases , it may not involve any users.
done in the lab which contain a complete equipment for the
evaluation.
Advantage:
suitable for the system that need to be located in a
dangerous or remote location and involved a single-user
tasks.
Disadvantages:
the system is not tested in real environment.the system is not tested in real environment.
users handling the system not in a real way.users handling the system not in a real way.
Styles of Evaluation
Field studies It takes the designer or evaluator out into the
user’s work environment. Advantage:
You will be able to observe interactions between systems and individuals which would have been missed in a laboratory study.
Disadvantage:
• interruptions by noises (phone calls) and
movements.
Why you need to evaluate?
Nielsen Norman Group point out: “User experience” encompasses all aspects of the end-
user’s interaction.. The first requirement for an exemplary user experience is to meet the exact needs of the customer, without fuss or bother. Next comes simplicity and elegance that produce products that are a joy to own, a joy to use.”
Evaluation is needed to check that users can use the product and like it.
Why you need to evaluate?
4 good reasons for investing in user testing/ evaluation: Problems are fixed before the product is delivered,
not after. The team can concentrate on real problems, not
imaginary ones Time to market is sharply reduced
When to evaluate?
The product can be: A brand new product An upgrade product
If the product new ; Time is usually invested in market research Designers may support this process by developing
mockups of the potential product To gain understanding of users’ needs and early
requirements
When to evaluate?
An upgrade product: Focus on improving the overall product The evaluations will compare user performance
and attitudes towards the previous version and the new one.
Evaluations done during design to check the products continues to meet users’ needs are known as formative evaluationsEvaluations done to assess the success of a finished product or to check a standard is upheld, are known as summative evaluation
Evaluating the designEvaluation should occur throughout the design process.
The first evaluation of a system should ideally be performed before any implementation work has started
4 possible approaches: Cognitive walkthrough Heuristic evaluation Review-based evaluation Model-based evaluation
Cognitive Walkthrough
A review technique where expert evaluators construct task scenarios from a specification or early prototype and then role play the part of a user working with that interface--"walking through" the interface.
They act as if the interface was actually built and they (in the role of a typical user) was working through the tasks. This technique evaluate how well the interface supports "exploratory learning," i.e., first-time use without formal training.
It can be performed by the system's designers in the early stages of design, before empirical user testing is possible.
Cognitive Walkthrough
The early versions relied on a detailed series of questions, to be answered on paper or electronic forms.The cognitive walkthrough was developed as an additional tool in usability engineering, to give design teams a chance to evaluate early mockups of designs quickly.It does not require a fully functioning prototype, or the involvement of users. Instead, it helps designers to take on a potential user’s perspective, and therefore to identify some of the problems that might arise in interactions with the system.
Cognitive Walkthrough
To do the cognitive walkthrough , you need: A description of the prototype of the system A description of the task that user needs to
perform on the system A complete, written list of the actions needed to
complete the task An indication of who the users are, and what kind
of knowledge and experience the evaluators can assume about them.
Cognitive Walkthrough: Example
For example, operating a car begins with the goals of opening the door, sitting down in the driver's seat with the controls easily accessible, and starting the car. And we're not even driving yet! This example shows the granularity that some walkthroughs attain. The goal of "opening the door" could be broken down into sub-goals: find the key, orient the key, unlock the door, open the door. Each of these goals requires cognitive (thinking) and physical actions. To open the door, do I orient my hand with the palm up or with the palm down? What affordances are provided for opening the door?
Heuristic Evaluation
A method for quick, cheap, and easy evaluation of a user interface design. It is a guideline or general principle or rule of thumb that can guide a design decision or be used to critique a decision that has already been made.Heuristic evaluation is the most popular of the usability inspection methods.The goal is to find the usability problems in the design so that they can be attended to as part of an iterative design process. Have a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the "heuristics").
Heuristic EvaluationThe heuristics are related to principles and guidelines. The list is as follows: Visibility of system status – system should always keep
users informed about what is going on Match between system and real world – system should
speak the user’s language User control and freedom- support undo and redo.
Make user feels in control Consistency & standards- follow platform convention Error prevention – prevent a problem from occurring in
the first place
Heuristic Evaluation Recognition rather than recall- make objects,
actions and options visible Flexibility & efficiency- allow users to tailor
frequent actions Aesthetic & minimalist design- dialogs should not
contain information which is irrelevant or rarely needed.
Helps user recognize, diagnose & recover from error- error messages should be expressed in plain language, precise and constructively suggest a solution
Help and documentation- provide help and documentation
Recognition rather than recall- make objects, actions and options visible
Flexibility & efficiency- allow users to tailor frequent actions
Aesthetic & minimalist design- dialogs should not contain information which is irrelevant or rarely needed.
Helps user recognize, diagnose & recover from error- error messages should be expressed in plain language, precise and constructively suggest a solution
Help and documentation- provide help and documentation
Review-based evaluation
Experiment is done to a specific domain.
For example, the usability issues: the review is done on the menu designs, the recall of command name and the choice of icons.
May use the output to support the aspects of design.
The reviewer must therefore select evidence carefully, noting the experimental design chosen, the population of subjects used, the analyses and the assumptions made.
Model-based evaluation
May use cognitive or design models to perform the evaluation.
For example the GOMS(Goals, Operators, Methods and Selections) used to predict user performance with a particular interface and also can be used to filter particular design options.
Design methodologies also have a role in evaluation. It may provide a framework in which design options can be evaluated.
Evaluating the implementation
This type of evaluation will involve users and actual implementation of the system.
The approaches are: Experimental methods Observational methods Query techniques
Empirical methods: Experimental evaluationOne of the most powerful methods of evaluating design is to use a controlled experiment.
The evaluator will choose a hypothesis to test, which can be determined by measuring some attribute of subject behavior.
Main components in experimental evaluation are as follows: Subjects- subject should be chosen to match the
expected user population. May involve the actual users. The sample size also must be identified.
Empirical methods: Experimental evaluation Variables – experiments manipulate and measure
variables under controlled conditions, in order to test the hypothesis. Two main type : variable to manipulate, and variable to measure.
Hypothesis – a prediction of the outcome of an experiment. The aim of the experiment is to show that this prediction is correct.
Experimental design – in order to produce reliable and generalizable results, an experiment must be carefully design
Statistical measures – there are 2 rules , look at the data and to save the data. This can be used to interpret the data.
Observational techniques
A popular way to gather information about actual use of a system is to observe users interacting with it.
Usually they are asked to complete a set of predetermined tasks, and may perform it in the users’ place of work.
2 ways to perform observational techniques: Think aloud Cooperative evaluation
Observational: Think aloud
• May be used to observe how the system is actually used
• user observed performing task• user is asked to describe what he is doing and why,
what he thinks is happening etc.Advantages
• simplicity - requires little expertise• can provide useful insight• can show how system is actually use
Disadvantages• subjective• selective depending on the tasks provided
Observational: Cooperative evaluation
A variation on think aloudUser is encouraged to see himself as a collaborator in the evaluation and not simply as an experimental subjectsboth user and evaluator can ask each other questions
Additional advantages• less constrained and easier to use• user is encouraged to criticize system• clarification possible
Query Techniques
• Usually based on prepared questions• Informal, subjective and relatively cheap• Can be useful in eliciting detail of the user’s view of a
system• The best way to find out how a system meets user
requirements is to ask the user.
Advantages• can be varied to suit context• issues can be explored more fully• can elicit user views and identify unanticipated
problems• Simple and cheap
Query Techniques
Disadvantages• Very subjective• Time consuming• Difficult to get accurate feedback
2 ways to perform query techniques:• Interviews• questionnaires
Choosing an Evaluation Method
Factors to consider:
• what style of evaluation is required? laboratory vs
field
• how objective should the technique be? subjective vs
objective
• what type of measures are required? qualitative vs
quantitative
Choosing an Evaluation Method
• what level of information is required? High level vs
low level
• what level of interference? obtrusive vs unobtrusive
• what resources are available? time, subjects,
equipment, expertise
Example
Choose an appropriate evaluation method for each situations in the next page.
In each case identify • The subjects. • The technique used. • Representative tasks to be examined • Measurements that would be appropriate. • An outline plan for carrying out the
evaluation.
Example 1
• You are at an early stage in the design of a spreadsheet package and you wish to test what type of icons will be easiest to learn.
Spread Sheet Package
Subjects Typical users: secretaries, academics, students, accountants,
home users, schoolchildren
Technique Heuristic evaluation
Representative tasksSorting data, printing spreadsheet, formatting cells, adding
functions, producing graphs
Measurements Speed of recognition, accuracy of recognition, user-perceived
clarity
Outline plan Test the subjects with examples of each icon in various styles noting responses
Example 2
• You have developed a group decision support system for a solicitors' office.
Group decision support system
Subjects Solicitors, legal assistants, possibly clients
Technique Cognitive walkthrough
Representative tasksAnything requiring shared decision making:
compensation claims, plea bargaining, complex issues with
a diverse range of expertise needed.
Measurements Accuracy of information presented and accessible, veracity
of audit trail of discussion, screen clutter and confusion,
confusion owing to turn-taking protocols
Outline plan Evaluate by having experts walk through the system
performing tasks, commenting as necessary
Example 3
• You have been asked to develop a system to store and manage student exam results and would like to test two different designs prior to implementation or prototyping.
Exam Management System
Subjects Exams officer, secretaries, academics
Technique Think aloud, questionnaires
Representative tasksStoring marks, altering marks, deleting marks, collating
information, security protection
Measurements Ease of use, levels of security and error correction provided,
accuracy of user
Outline plan Users perform tasks set, with running verbal commentary on
immediate thoughts and considered views gained by
questionnaire at end.
Summary
Evaluation is an important part of the design process and should take place throughout the design life cycle.
The aim is to test the functionality and usability of the design and to identify and rectify any problems.