1

Remote TA testing– Geographical, cultural and technological challenges.

By Steen Filskov AndersenUsability Specialist at Snitker & Co.

Copenhagen, Denmarksfa@snitker.com

This study was co-funded by the Danish Council for Independent Research(DCIR) through its support of the Cultural Usability project.

2

• Abstract. This poster outlines the technological conditions for remote TA testing and compares them to those of local TA testing summing up the benefits and drawbacks of remote TA testing across geographical, cultural and technological frontiers. Focus is on the method’s affect on the respondents’ perceived sense of flow during the test course based on Csikszentmihalyi’s[1] conception of the term.

• Keywords: Remote think aloud (TA) testing. Cultural usability. Technology. Flow. Csikszentmihalyi.

[1] Csikszentmihalyi, Mihaly: Flow – The Classic Work on How to Achieve Happiness. Rev. & upd. Rider Books (2002)

3

1. Introduction– definition of remote TA testing

Remote TA testing is made possible via a wide range of soft- and hardware solutions, enabling the moderator and respondent to be communicating over a great distance.

The stimulus is shared between the moderator and respondent and can originate from a local computer or from a third party on the internet.

The moderator can choose to give the respondent total control of the application or webpage or simply just show the respondent the stimulus allowing for both interactive and static prototypes to be tested.

4

2. Technologicalconditions

Communication between test leader and respondent takes place through a phone line or the internet. Requires a phone line or internet connection

Stimulus is shared via a desktop sharing application allowing for both interactive and static (fx pdf) prototypes to be tested. Requirements are an internet connection (min. dsl-speed) and computer terminals at both ends of the line.

Depending on the stability of the internet connection the respondent might experience more or less lag or waiting time when interacting with or viewing the stimulus. This can affect the way the respondents perceive the prototype thus compromising the result.

5

Depending on the software used the respondents may see the cursor as a cross and fx. not as a hand when placed on top of a hyperlink. This can both be a weakness and strength. In a static prototype it does not matter, but if it is a prototype of a website, it does not resemble a real website to the respondent and so control is best left to the test leader asking the respondent where to click etc. It can however give some good clues to where the users expect to be able to click.

6

3. Pros and conswhen using a remote lab

• At a remote testing situation the respondent’s access to the stimulus can be at a computer placed at his or hers private home.

– Pros:• Very convenient for the respondents.• Depending on the prototype the respondent’s

home can be a more natural environment for experiencing the stimulus.

– Cons:• Varying technology (computer, input devices,

screen resolution, internet connection).

7

• More complex screening process incl. tech. requirements.

• No headset, so respondents have to hold the receiver.

• Respondents are more exposed to distractions from their daily life (children, dogs, spouses, laundry, neighbors, etc.)

• Bigger risk of no-shows.

8

• The respondent’s access to the stimulus can be at a computer placed at a suitable facility.

– Pros:• Respondents are being taken care of by

professionals, offered coffee and sweets and can be helped when joining the desktop sharing application.

• Respondents tend to take the test more seriously when situated in a proper office environment.

• Usually a fast internet connection is available.

9

• Respondents can use a headset thus freeing both hands for interacting with the stimulus.

• The test leader can be informed by the staff of any no-shows or delays.

– Cons:• Respondents may have to travel a distance to

participate.

10

3.1 Advantages and disadvantages ofremote testing compared to local testing

and testing with a local moderator

• Advantages are: – Remote TA tests make it possible to test in every part

of the world at a low cost.– Respondents can be recruited from a large

geographical area – also within the same country – at no extra cost.

– The same test leader can be used to perform all of the tests thereby accumulating information at first hand.

– Clients and collaborators can easily witness the tests from any computer with access to the internet.

11

• Disadvantages are:– Language

• Test leader, respondents and stimulus have to be communicating in the same language which may impede the articulateness of the test leader and respondent and the understanding of the stimulus.

– Loss of non-verbal cues• When communicating over the phone visual cues

such as facial expressions and hand gestures are lost. These can constitute an important part of the communication in a TA test situation.

12

– Cultural differences • If test leader and respondent have different cultural

backgrounds several issues arise that raise doubt about the validity of the TA method incl. power distance between test leader and respondent and the general understanding of the culture in which the stimulus is perceived.

13

4. Introductionto flow

Defined by Mihaly Csikszentmihalyi flow describes a state of mind which is achieved when a person is ’submerged’ in an activity. It is a feeling of joy, creativity and total involvement. The activity can be anything from ordinary work to sporting activities to playing computer games. Certain parameters – the eight elements of enjoyment - have to be fulfilled more or less before a person enters such a state of happiness.

In the following I will suggest that the eight elements of enjoyment can be found in a TA testing situation for the respondent as well as the test leader.

14

The benefits of the respondent’s state of flow such as “a high level of concentration, motivation and absorption” have a positive effect on the test course, the loyalty of the respondents and the validity of the results and are therefore highly appreciated.

The state of flow is achieved through multiple parts of the test setup ranging from the difficulty and complexity of the tasks to be solved to the surrounding environment and technical setup at the location of the test. These parts can easily be controlled in a local test situation but can constitute several problems when testing remote.

The success of a TA test is based partly on the moderator’s understanding of the respondent’s conceptual understanding of the situation as well as that of specific details in the stimulus and the tasks.

15

When the interaction between the moderator and the respondent takes place via a purely auditory media such as a phone and possibly over a great distance both geographical and cultural a lot of the information used to judge the respondent’s competences during the test is lost.

If technological limitations such as network congestion add to this situation it can affect the respondent’s state of flow and the test results will not be comparable to those of a local test on the same prototype.

16

4.1 Elements ofenjoyment – eight major components

Csikszentmihalyi lists the following eight elements of enjoyment which have to be present for a person to engage in a state of flow.

1. A challenging activity that requires skills.• Challenges are essential but have to be manageable.• Design the question guide correct and adjust when

testing if necessary.

2. The merging of actions and awareness• Absorption in the activity.• Respondent should be focused on the test and kept

interested.

17

3. Clear goals• Intentions of what to do.• Set clearly defined tasks.

4. Immediate feedback• Feedback from “the system” on the actions required

to reach the goal.• The prototype should be responsive and stable.

5. Concentration on the task at hand.• A complete focusing of attention on the task at hand.• The test should not be interrupted by third party and

the respondent should be concerned only on solving the present task.

18

6. The paradox of control.• Mastering the skills and not facing real life

consequences if failing.• The tasks and use of terms must be adjusted to

comply to the respondents skill level during the progress of the test as not to let the respondents experience a long series of defeats thus losing patience with the stimulus.

7. The loss of self-consciousness.• A feeling of union with the surrounding environment.• The respondent forgets about their physical needs

and leaves coffee, sweets etc. untouched.

8. The transformation of time• Losing track of time. • The respondent is surprised that one or two hours

suddenly has passed.

19

When testing remote several of the parameters are more difficult to control and the respondents’ experience of flow can be limited making it hard to keep them interested and focused on the tasks at hand.

This must be taken into consideration when testing remote and steps should be taken to limit the affect of the distance between the test leader and respondent.

20

F l o w

C h a n n e l

Boredom

Anxiety

Lo

w

Ch

alle

ng

es

Hig

h

Low Skills High

Flow- A narrow path

21

Contact information

Steen Filskov AndersenUsability Specialist

Snitker & Co.Bredgade 21B

DK-1260 København Ksfa@snitker.com

Mob: +45 26 71 56 18

Snitker & Co.

• Coordinates qualitative research, design the studies and/or execute studies.• Often integrates with media planning, project management, content and

functionality providers etc.• Has more than six years of experience with global qualitative research. • Key research venues:

– London, New York, San Francisco, Sao Paolo, Singapore, Shanghai, Hong Kong, Scandinavia, Finland, Germany, Benelux and Italy.

www.snitker.com