Methodology and ExplanationXX50125

Lecture 1: Part I. Introduction to Evaluation Methods

Part 2. Experiments

Dr. Danaë Stanton Fraser

Methodology and Explanation 2007

Goals of this unit

• To introduce research methods in human computer interaction

• To explore qualitative and quantative methods• To gain experience in putting these methods into

practice• To gain a critical understanding of analytic

techniques

Methodology and Explanation 2007

Unit structure

Lectures each focusing on methods of analysis

Practical sessions putting these methods into practice

Methodology and Explanation 2007

Course text and slides

Rogers, Y., Sharp, H., and Preece, J. 2007. Interaction Design: Beyond Human Computer Interaction. Wiley.

Slides available online at:http://staff.bath.ac.uk/pssds/M&E.html

Methodology and Explanation 2007

Assessment

2 assessed courseworks:

1st coursework (report)

2nd coursework (report and presentation)

Methodology and Explanation 2007

Evaluation Methods

• Controlled experiments • Ethnography• Expert review• Usability testing• Surveys/questionnaires• Data logging• Interviews and focus groups

Methodology and Explanation 2007

Not just desktop computing.

What other kinds of technologies are we

evaluating?

Methodology and Explanation 2007

Direct manipulation & graphical user interfaces

Methodology and Explanation 2007

Collaborative Desktop

Methodology and Explanation 2007

Immersive technologies

Methodology and Explanation 2007

Tangible interfaces

Methodology and Explanation 2007

Wearable, handheld and mobile devices

Methodology and Explanation 2007

Embedded interfaces

Methodology and Explanation 2007

Evaluate: which method to choose• Design or implementation?• Laboratory or field studies?• Subjective or objective?• Qualitative or quantitative?• Information provided?• Immediacy of response?• Intrusiveness?• Resources?

Methodology and Explanation 2007

Part II

Experiments: A brief guide

Methodology and Explanation 2007

Aim

• To answer a question or test an hypothesis that predicts a relationship between two or more events, known as variables.

• E.g. Will spatial knowledge be superior following exploration of a VR simulation of a building or from exploring a model of a building?

Methodology and Explanation 2007

Variables

• Such hypotheses are tested by manipulating one or more of the variables.

• The variable that is manipulated is called the independent variable (the conditions to test this variable are setup independently before the experiment starts). In the study mentioned media type (VR vs model) is the independent measure

• The dependent variable would be accuracy of spatial information (ie time to reach point a from point b) because the hypothesis is that the time to carry out the task depends on the media explored.

Methodology and Explanation 2007

Variables and Conditions

• In order to test an hypothesis the experimenter sets up the experimental conditions e.g.

• Condition 1 Pretest Explore model Post test• Condition 2 Pretest Explore VR Post test

• Control condition – against which to compare the results?

Methodology and Explanation 2007

Participants

• Between participantsTwo drawbacks – no. of participants needed, individual

differences

Advantage – no order effects

• Within participantsCounterbalancing required

• Matched pairs on characteristics such as gender and expertiseDifficult to match across all variables

Methodology and Explanation 2007

Practicalities

• Where will the experiment be carried out?• How will the equipment be set-up?• How does one introduce participants to study?• What scripts are need to standardise the procedure?• Always include pilot study

Methodology and Explanation 2007

Data Collection and Analysis

• Data – performance measures are taken e.g. response times, no. of errors

• Use graphs• The data should be averaged across conditions to

examine any differences• Statistical tests such as t-tests and ANOVAs can

reveal whether the differences are significant• Software packages such as SPSS are often used• If there is no significant difference then the

hypothesis is refuted.

Methodology and Explanation 2007

Data captured

• Record quantitative results

• However also gather process data e.g. dialogue turns, gesture and non verbal behaviour

Methodology and Explanation 2007

Preparation for Practical 1Topic: Examining the use of gestures in collaboration.

In preparation for the practical please read:

1. Kraut, R. E., Fussell, S. R., and Siegel, J. (2003). Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human-Computer Interaction,Vol. 18, pp. 13–49 Lawrence Erlbaum Associates, Inc.

http://www.columbia.edu/itc/psychology/rmk/Readings/Kraut.pdf

2. Kirk, D. and Stanton Fraser, D. (2005). The Effects of Remote Gesturing on Distance Instruction. Computer Supported Collaborative Learning (CSCL 2005). Taiwan. May.