3461

Midterm Material

Midterm is short answer Everything up through last week’s lecture Things to study:

Slides Programs Javadoc

No need to memorize all methods of Swing classes. Familiarity with the most common ones may be tested

3461

Midterm Material

Here is an example of a possible midterm question (however this particular question is not on your exam!)

[6 marks] What three different types of layouts for multiple windows onscreen were presented in class? Briefly describe each type.

3461

Evaluation Paradigms

“Quick and Dirty” evaluation Usability Testing Field studies Predictive evaluation

3461

“Quick and Dirty” Evaluation

User-centered, highly practical approach Used when quick feedback about a design is

needed Can be conducted in a lab or the user’s natural

environment Users are expected to behave naturally Evaluators take minimum control Sketches, quotes, descriptive reports are fed back

into the design process

3461

Usability Testing

Applied approach based on experimentation Used when a prototype or a product is available Takes place in a lab Users carry out set tasks Evaluators are strongly in control Users’ opinions collected by questionnaire or

interview Reports of performance measures, errors etc. are

fed back into the design process

3461

Field Studies

Often used early in design to check that users’ needs are met or to assess problems or design opportunities

Conducted in the user’s natural environment Evaluators try to develop relationships with users Qualitative descriptions that include quotes,

sketches, anecdotes are produced

3461

Choose paradigm and techniques

Practical and ethical issues might be considered

Factors: Cost Timeframe Available equipment or expertise

Compromises may have to be made

3461

Decide on ethical issues

Studies involving humans must uphold a certain code

Privacy of subjects must be protected Personal records must be kept confidential Exact description of the experiment must be

submitted for approval

3461

Evaluate the data

Should quantitative data be treated statistically? How to analyze qualitative data? Issues to consider:

Reliability (consistency) Validity Biases Scope Ecological validity

3461

We’ll take a closer look at…

Two predictive evaluation techniques: Heuristic evaluation Cognitive walkthroughs

A usability testing technique User testing

3461

Heuristic Evaluation

Heuristic evaluation is a technique in which experts, guided by a set of usability principles known as heuristics, evaluate whether user interface elements conform to the principles.

Developed by Jakob Nielsen Heuristics bear a close resemblance to design

principles and guidelines Interesting article on Heuristic Evaluation:

http://www.useit.com/papers/heuristic/heuristic_evaluation.html

3461

List of Heuristics

Visibility of system status Match between system and the real world User control and freedom Consistency and standards Help users recognize, diagnose, and recover

from errors

3461

List of Heuristics (cont.)

Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help and documentation

3461

Heuristic for Websites

Avoid orphan pages Avoid long pages that force scrolling Provide navigation support, such a site map that is

always present Avoid narrow, deep, hierarchical menus Avoid non-standard link colours Provide consistent look and feel for navigation Avoid complex URLs Avoid long download times

3461

Cognitive walkthroughs

“Cognitive walkthroughs involve simulating a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action” (Nielsen and Mack, 1994).

Focus is on evaluating design for ease of learning

3461

Cognitive walkthrough steps

Characteristics of typical users are identified Designers and evaluators meet, walk through the

action sequences for each task and try to answer the following questions: Will the correct action be evident to the user? Will the user notice that the correct action is available? Will the user know from the feedback whether they made

a correct choice?

3461

Usability Testing

Test and Retest

3461

Usability Testing is NOT...

“What type of feedback did you gather from your usability participants?”

“I showed my program to three different people and they all said it looked really, really good.”

3461

It doesn’t matter how useful a program could be if the interface is unusable.

“Not even the most brilliantly conceived and ingenious computer system can do all that it was designed to do- or even a small part of what it was designed to do- unless the brilliance of its operation and purpose is matched by the cunning simplicity of its user interface. (Hicks and Essigner, 1991)

There is a distinction between useful and usable.

3461

Usability Testing- Definition

Usability testing is a method by which users of a product are asked to perform certain tasks in an effort to measure the product's ease-of-use, task time, and the user's perception of the experience. Changes are made to the application or site based on the findings of the usability tests. Usability test participants are encouraged to think aloud and voice their every opinion. Usability testing is best used in conjunction with a user-centered design process, a method by which a product is designed according to the needs and specifications of users.

Adopted from http://searchwebservices.techtarget.com/sDefinition/0,,sid26_gci214526,00.html

3461

Why is Usability Testing Important? (1 of 3)

3461

Why is Usability Testing Important? (2 of 3) It is impossible to predict usability from

appearance, just like it is impossible to judge a person’s personality on appearance.

Casual “feel good” feedback is inadequate. Formal testing is often the only way problems

are identified pre-release. Problems found once a product is released are usually not fixed unless they are really severe.

3461

Why is Usability Testing Important? (3 of 3) User, Designer, Programmer each have

different models. The designer’s intuition is not always correct. Design standards and guidelines are not

sufficient. Usability testing leads to competitive

advantages and reduced support costs.

3461

The REAL Components of Usability

Relevance - is how well a system serves the users needs.

Efficiency - states how efficiently the users can carry out their tasks using the system.

Attitude - is the users subjective feelings towards the system.

Learnability - is how easy the system is to learn initially and how well the users remember the skills over time

http://www.affectus.se/artiklar/usabilityinsp/

3461

Relevance

number of good and bad features recalled by users

number of available commands not invoked by users

number of available commands invoked by users

number of times users needs to work around a problem

percent of task completed

http://www.affectus.se/artiklar/usabilityinsp/

3461

Efficiency (1 of 2)

time to complete a task percent of task completed percent of task completed per unit time

(speed metric) time spent in errors number of commands used frequency of help and documentation use time spent using help or documentation

http://www.affectus.se/artiklar/usabilityinsp/

3461

Efficiency (2 of 2)

number of repetitions of failed commands number of runs of successes and of failures number of times interface misleads user number of times user needs to work around a

problem number of times the help facilities solve the

users problem

http://www.affectus.se/artiklar/usabilityinsp/

3461

Attitude

percent of favorable/unfavorable user commands number of good and bad features recalled by users number of users preferring the system number of times user loses control over the system number of times the user is disrupted from a work

task number of times user expresses frustration or

satisfaction

http://www.affectus.se/artiklar/usabilityinsp/

3461

Learnability (1 of 2)

ratio of success to failures (over time) time spent in errors percent or number of errors number of commands used frequency of help and documentation use time spent using help or documentation number of repetitions of failed commands number of runs of successes and of failures

http://www.affectus.se/artiklar/usabilityinsp/

3461

Learnability (2 of 2)

number of available commands not invoked by users

numbers of features or commands that can be remembered after a test

the proportion of users using efficient strategies compared to those using less efficient strategies

number of logical errors made

http://www.affectus.se/artiklar/usabilityinsp/

3461

Nielson’s Usability Severity Ratings

0 - Not a usability problem at all

1 - Cosmetic problem only - need not be fixed unless extra time is available on project

2 - Minor usability problem - fixing this should be given low priority

3 - Major usability problem - important to fix, so should be given high priority

4 - Usability catastrophe - imperative to fix this before product can be released

3461

Usability Frequency Ratings

0 - Problem did not occur

1 - This problem occurs rarely- only once/under very unusual circumstances

2 - This problem occurs occasionally - under less commonly occurring conditions

3 - This problem occurs often during common tasks

4 - Problem occurs every time under all tested conditions

3461

What information to provide?

Give a brief explanation that the participant’s involvement is to solicit user feedback. Any problems are the fault of the software.

In real-world situations explain confidentiality agreement, liability legalities, and that participant is free to leave at any time (and still get paid).

Provide instructions as to the user’s task but not explanations of the software.

3461

When should usability testing be conducted?

Ideally, as early as possible (from the prototyping stage) and then repeatedly throughout the development process.

“Test early and often.”

3461

Usability Labs (http://www.ulabs.com/images/lab.gif)

3461

Usability Labs - Dedicated Lab (http://www.sun.com/usability/)

3461

On-site Usability Testing

http://keith.instone.org/howtotest/

3461

Low Fidelity Prototyping

Hand sketches and scenarios (storyboards, scene by scene) where the focus is on the design, not on the interface mechanics.

Usability testing can be conducted using hand sketches before any code has been written.

Can be used early in the development process Fast to modify and iterate Can be used to define requirements

3461

What to User Test?

Possibilities include: Conformance with a requirement Conformance with guidelines for good design Identification of design problems Ease of system learning Retention of learning over time Speed of task completion Error rates Subjective user satisfaction

Galitz, W. O., (2002) The Essential Guide to User Interface Design, 2nd Edition, Wiley Computer Publishing, New York, NY.

3461

What questions to ask during a usability session? Depends on which phase of the development

cycle- what can be changed? Conceptual model? Layout? Fonts?

There should be a list of questions of the major design issues that is prepared in advance. There should be specific questions that the usability testing is designed to answer. Usability testing has specific objectives.

3461

What is the Goal of Usability Testing?

Usability testing should be designed to determine if the software is meeting the

Qualitative Usability Goals Quantitative Usability Goals

Adapted from Mayhew, Deborah J. (199) The Usability Engineering Lifecycle

3461

Qualitative Usability Goals- Examples

The design must support users working in a high-interrupt environment, with lots of context information on screen to remind users where they are when they get distracted.

The design must support very infrequent users of a very complex task. Thus, it must be self-explanatory, easy to learn and remember.

Adapted from Mayhew, Deborah J. (199) The Usability Engineering Lifecycle

3461

Quantitative Usability Goals - Examples

Experienced users (defined as users who have performed the task five times in a training session) should take no longer than 15 seconds minutes on average to address an email.

Novice users (defined as first-time users) should take no longer than three minutes to complete the registration input form.

Adapted from Mayhew, Deborah J. (199) The Usability Engineering Lifecycle

3461

When to ask questions?

If you are worried about interrupting the task flow of your participant, then ask the question after the completion of the task.

If you are more worried about the participant forgetting their current thought process than interrupting, then ask right away.

3461

Heuristic Evaluation

A heuristic evaluation is a detailed evaluation of a system by interface design specialists to identify problems.

Required reading: http://www.useit.com/papers/heuristic/heuristic_evaluation.html

3461

Thinking Aloud Protocol

During a usability test, instruct participants to verbalize their thoughts aloud protocol. The usability testers prompt participants by asking direct questions about the software, in order to understand their mental model of the system and the tasks, and where they have trouble in understanding and using the system.

3461

Co-discovery Method

During a usability test, two participants perform tasks together while being observed. In order to increase the amount of communication to gain insight to their thought process, one participant is assigned the mouse and the other the keyboard. They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product.

3461

Usability Testing Sequence of Events (1 of 2)1. Know your Purpose. There should specific

questions which need answering

2. Find Representative Users. This is a non-trivial task, whose difficulty is often overlooked in texts on usability. The quality of the participants make or break the success of the usability session.

3. Watch & Learn. Don’t lead the user to give the answers you want to hear. Be prepared to learn new things.

3461

Usability Testing Sequence of Events (2 of 2)4. Report the Data. The data collected and

observations made must be communicated to the developers and to management.

5. Back to the Drawing Board. It is not enough just test- time must put aside to correct the problems found by tests.

Adapted from http://keith.instone.org/howtotest/introduction.html