REVIEW PAPER

The Easyline+ project: evaluation of a user interface developedto enhance independent living of elderly and disabled people

Rich Picking • Alexia Robinet • John McGinn •

Vic Grout • Roberto Casas • Ruben Blasco

Published online: 25 September 2011

� Springer-Verlag 2011

Abstract This paper reports the usability evaluation of

interfaces developed to enable elderly and disabled people

interact remotely with kitchen appliances in the home to

enhance their independent living. A number of evaluation

exercises were undertaken throughout the project’s devel-

opment, including user-participative workshops and focus

groups. This paper focuses on the summative usability

evaluation exercise, which comprised a laboratory-based

study in a simulated home environment, with a view to

determining the appropriateness of employing this approach

with potentially vulnerable participants. The study involved

27 participants interacting with the user interface. Their

behaviour was observed and recorded, and their interaction

with the system was analysed. They were also given a post-

session questionnaire, where their opinions of the usability

of the interface were solicited. The results of the usability

testing were positive, and insight has been gained into

how products of this nature can be further improved. The

experience of conducting laboratory-based studies with

vulnerable users was positive and led to propose in this

paper a set of guidelines for future work in evaluating

usability for work in this domain.

Keywords Laboratory-based study � Vulnerable users �Assisted living � Ethical issues

1 Introduction and motivation for research

It is now well documented that the number of elderly

people in Western society is rapidly increasing, and birth

rates are declining. This leads to an inevitable future where

fewer younger people will be available to care for more and

more of the elderly population. In Europe, this population

represented around 17% in 2007 but is expected to increase

rapidly. By 2020, the elderly population will represent an

estimated 20%, and by 2050 it will be 30% [7]. Hence,

there is an identified need to support people’s indepen-

dence for as long as possible [16]. In addition to these

demographic and economic factors, it is a fact that most

elderly people would prefer to stay in their own home

rather than move to a care home, despite illnesses or

impairments that could put their well-being at risk [17].

In response to these issues, researchers and developers

of information technology have sought to find ways to

enable elderly people to live independently for longer, for

example by developing smart home environments to meet

the special needs of the ageing population, providing

security, comfort and quality of life. A smart home envi-

ronment can also offer added value, by monitoring long-

term changes that may cause health concerns [3]. Such

systems could alert carers and family of any significant

changes in behaviour, diet, daily tasks or health. Fall

R. Picking (&) � A. Robinet � J. McGinn � V. Grout

Centre for Applied Internet Research, Glyndwr University,

Wrexham, Wales, UK

e-mail: r.picking@glyndwr.ac.uk

A. Robinet

e-mail: a.robinet@glyndwr.ac.uk

J. McGinn

e-mail: j.mcginn@glyndwr.ac.uk

V. Grout

e-mail: v.grout@glyndwr.ac.uk

R. Casas � R. Blasco

Grupo Tecnodiscap, Universidad de Zaragoza,

Zaragoza, Aragon, Spain

e-mail: rcasas@unizar.es

R. Blasco

e-mail: rblasco@unizar.es

123

Univ Access Inf Soc (2012) 11:99–112

DOI 10.1007/s10209-011-0246-8

detectors, smart pill dispensers, medical equipment to test

heart rate and blood pressure, tracking devices, and sen-

sors, all create a potentially safer environment in which to

live for people with sensory, cognitive or physical dis-

abilities. These developments may also serve to provide

younger disabled people with similar benefits, as well as

for the wider population in general, in the context of the

design for all concept.

This paper presents the usability evaluation of a system

developed for a smart home environment, specifically

interfacing with kitchen appliances. The focus of the

research presented here is on the evaluation process, in the

light of the special nature of the participants involved.

Although many usability engineering methods exist and are

well documented [12], a novel methodology has been

developed specifically with vulnerable users in mind,

where ethical issues critically inform design decisions [22].

Within this methodology, a fundamental question is how

best to evaluate for elderly and disabled users? The ratio-

nale discussed in this paper led to select a range of meth-

ods, one of which was the laboratory-based evaluation

presented here. This leads on to the question of whether

this approach can be effective and appropriate for vulner-

able participants. The conclusion is affirmative, but with a

series of caveats which, based on the experiences of the

exercise, are presented in the form of guidelines that can be

used by researchers involved with similar groups of users.

2 Background

The European Framework 6 IST Easyline? project (Low

cost advanced white goods for a longer independent life of

elderly people) is a collaboration between universities and

companies in Germany, Spain and the UK to produce

technologies that simplify an elderly or disabled person’s

interaction with a range of kitchen appliances, in order to

allow them to live a longer independent life in their own

home. The project focuses specifically on kitchen appli-

ances as many accidents occur in this environment. Addi-

tionally, these appliances may pose significant usability

problems for users with failing physical and cognitive

abilities. In the kitchen, for example, washing machines

often have complex programmes, food contents in refrig-

erators and freezers may be difficult to discern (in partic-

ular the information on packaging), and ovens can cause

serious injury if not used with particular caution. One only

has to visit a high street appliances store to become con-

fused by the vast array of different control panels, displays,

and features—many of which sacrifice usability for visual

appeal. To fulfil the requirements of the project, a range

of usable kitchen appliances have been developed.

Such appliances are integrated through an adaptive system

(called the e-servant), and accessible through common user

interfaces situated around the home and/or via mobile

devices. The role of the system is to provide responsive

help, information, advice, notifications, and warnings at an

appropriate level for each user, dependent on their partic-

ular ability. The system aims to adapt to individual’s

changing circumstances, for example if they become more

cognitively impaired as they grow older.

A simple example of such an interaction is the scenario

of a cooker hob being left on either with no pan on it or after

a pan has been removed. An appropriate message has to be

conveyed to the user (wherever they may be in the home).

The precise nature of the interaction and the range of

options available to the user are adaptive, flexible, and

dependent on their level of ability, which can be assessed on

a number of scales. However, the essence of the dialogue in

this case would be that the user could turn off the hob

remotely or respond: ‘‘Yes, I know; leave it on’’ (if their

profile permits it). Other scenarios include ‘‘Food has

expired in the fridge’’, ‘‘The washing cycle has finished’’,

‘‘This food cannot be microwaved’’, and so forth. A range

of European languages is also supported, including English,

German, Spanish, French, and Welsh. A range of sensing

technologies was used to enable the system’s functionality,

including radio frequency identification (RFID), zigbee,

powerline communications, and wi-fi communication.

The kitchen’s user interface can be situated on any

modern programmable device. It has been tested on mobile

devices (for example, personal digital assistants, cell

phones), televisions, and other ambient devices, such as

interactive digital photoframes. The interface is consis-

tently presented on all devices and includes the option of

speech output for visually impaired users. An example

screen is shown in Fig. 1.

Fig. 1 Example screen showing refrigerator settings control. Other

appliances supported are washing machine, microwave, freezer, hob,

and oven

100 Univ Access Inf Soc (2012) 11:99–112

123

The top part of the screen is used to display the status of

each appliance, so the user can check quickly what is going

on in the kitchen. This area of the screen also serves as a

tab to enable the user to step through the appliances to

control them, or to review their status in more detail. The

bottom part displays four coloured buttons, which reflect

the standard colour design of text buttons on a television

remote control. The middle part of the display is the main

information area, where the content is updated dynamically

to show appliance status or give notifications and warnings

to the user when events occur.

A number of evaluation exercises has been undertaken

throughout the project’s development, including user-par-

ticipative workshops, demonstrations, and focus groups.

This paper focuses on the summative usability evaluation

exercise, which comprised a laboratory-based study in a

simulated home environment. The following sections

discuss the methodology of evaluation, the rationale for

laboratory-based studies, and the experimental design. The

results of the evaluation are reported, also providing con-

clusions and recommendations for similar future work in

evaluating usability in this domain.

3 Evaluation methodology

Designing user-interfaces for the elderly, or the physically

or cognitively disabled is a complex challenge [3, 10, 13,

14, 30]. In the early stages of the Easyline? project, a

critical analysis of the most used interface and develop-

ment methodologies was undertaken in order to assess their

appropriateness for the task. For the addressed user popu-

lation in particular, ethical issues must be considered in the

design process [9]. Ethical guidelines may be used to

accompany methodologies in the development and can

provide useful sets of principles and duties; however,

practitioners have often had difficulty in applying them

[29]. Also, most methodologies take little account of

ethical issues, or they might be addressed typically at the

early stages of a project’s lifecycle.

It was decided that it was necessary to take account of

ethical issues throughout the Easyline? development, and

consequently such issues were regarded as the central focus

of the methodological approach. In the light of the lack

of an existing methodology to support the above require-

ments, a new methodology called EDUCATID was

developed as an ethically driven, user-centred approach to

interface development [22]. As the acronym suggests, it is

grounded in being ethically driven, in that ethical issues are

carefully scrutinized at the initial analysis phase, as well as

in iterations of interface prototyping, development, and

evaluation. Similarly, it is user-centred in that users are

involved in participative, narrative workshops in the initial

analysis phase and also in the method’s iterations, and

naturally during the usability evaluation phase. Although

there is a specific evaluation phase, formative evaluations

along with ethical inspections occur at regular intervals.

3.1 Rationale

The focus of this paper is on the evaluation phase, where a

summative evaluation has taken place at the second itera-

tion of the method cycle. The first iteration comprised pilot

studies, and the third and final iteration is an ongoing

longitudinal study situated in the home environment.

It was decided to conduct the evaluation in question in

the relatively controlled setting of a usability laboratory in

order to make sure that the longitudinal study would not be

compromised by usability or reliability problems. This

environment also helped identify potential ethical issues.

In the context of the project development as a whole, a

fundamental criterion for the methodology was that it

should be practical. Heavily participative methods can be

time-consuming and costly, so reliance on the user popu-

lation in the project’s development was relatively eco-

nomical. Following initial user engagement, it was decided

to formulate personas [2, 4] to reduce the reliance on sig-

nificant user consultation. Also, as the target user popula-

tion is vulnerable, to involve them in a more intensive

participatory design approach may have caused ethical and

practical problems. Such issues have been highlighted in

the past, particularly by [18]. Alternative solutions have

been successful, for example where trained performers

role-play elderly adults [19] and where art has been used to

enable elderly people to articulate their vision of future

home environments [27]. However, a recognized drawback

of these approaches is that they too can be expensive and

time-consuming. A total of ten personas were created based

on the user population, as well as from data sources based

on European statistics [8]. The personas were then used to

create a more formal set of user profiles, which formulated

the initial user models used by the Easyline? e-servant.

Characteristics such as age, education, work situation,

impairment, and technology familiarity were assigned.

This approach was not only useful for user modelling, but

also acted as an excellent pro forma for user recruitment for

the laboratory study.

3.2 The laboratory layout

The study was carried out at the Glyndwr University

usability laboratory. The laboratory comprises a control

room with a one-way mirror looking onto an observation

room. The observation room was furnished in the style of a

small living area with a working kitchenette (Fig. 2). The

living area comprised armchairs, a coffee table with

Univ Access Inf Soc (2012) 11:99–112 101

123

newspapers and magazines, a corner table with a television,

bookshelves, and a nest of tables. Participants had access to

the television and its remote control unit for system inter-

action. They also had at their disposal a Nokia Personal

Digital Assistant (PDA), a small touch screen device which

displayed the same interface. Cameras were placed in the

room to capture the participants’ experience during the

studies, and microphones recorded all audio activity.

The kitchenette included a washing machine, an oven, a

cooker hob, and a fridge. A set of two sensors were posi-

tioned to track the door status (open or closed) of the fridge

and the washing machine, and another sensor was used to

track the fridge temperature. The kitchen appliances were

connected to the power line and were equipped with a

Siemens Serve@home interface card. This appliance

management system was used to monitor and control the

kitchen appliances from the observation room via a secured

wireless network. A small zigbee receiver was connected to

the Easyline? computer and communicated wirelessly

with each of the kitchen appliances.

4 Preparation for the study

There were three roles involved in the user evaluation:

Users: the participants evaluating the technology

Test observer: the individual watching the different

situations evaluated without contact with the user, taking

notes about the participants’ performance and reactions

Test moderator: the leader of the usability study

sessions. The moderator was in charge of interacting

with the participants and observing them during the

testing. The moderator introduced the session to the

participants and facilitated the participants’ signing of

consent forms and in filling out a user profile question-

naire. Factors such as age, gender, profession or past

profession, familiarity with computers, and experience

of domestic activities and disabilities were all elicited.

A short demonstration of the system was also given prior

to the usability study

An observation form and usability experience ques-

tionnaire were also prepared for each participant. The

questionnaire and observation form were developed spe-

cifically for the project, due to the unique nature of the

product and its users. The reliability of the questionnaire

was tested during pilot studies by a repeated application of

the survey, although the small sample size could not jus-

tify undertaking tests of a statistical significance, for

example Chronbach’s alpha. The observation form was

filled out during and immediately after each study, and in

detail during later video analysis. The design of the

observation form used a Likert rating scale and is shown

in Table 1.

The usability experience questionnaire was filled out by

participants immediately after the study. Where partici-

pants were physically unable to complete the forms

themselves, help was provided. The questions in the

usability experience questionnaire were categorized into

five areas: usability, design and layout, functionality, sat-

isfaction, and future use/general outcome. All questions,

apart from the final one which invited general suggestions

for improvement to the system, were answered on a 5-point

semantic rating scale. Every question invited further

comment in writing.

The specific questions and possible answers are shown

in Table 2.

5 Experimental design

We undertook a between-groups study with three sets of

heterogeneous users. The first group comprised elderly

users with some disabilities mainly brought on by their

ageing condition. Although elderly users were the primary

focus, it was also decided to evaluate the usability with a

group of younger users who have specific cognitive

learning disabilities. The latter group was of interest as

potential beneficiaries of the Easyline? project, as well as

with respect to their clear identity as a group with measured

and documented low cognitive faculties. Testing with a

Fig. 2 Usability lab at Glyndwr

University

102 Univ Access Inf Soc (2012) 11:99–112

123

control group, which comprised ‘able-bodied’ people of

varying ages, was also undertaken, for two reasons. First, it

has been documented that elderly and vulnerable partici-

pants in usability studies may react differently than they

normally would, for example by being over-positive due to

their involvement in the study [6, 21]. Comparing their

results with the control group would potentially identify

issues of this nature with the conducted study. Second, the

pilot studies suggested that the product might be suitable

for time-impoverished people (for example, stressed

mothers with babies in the home), not just elderly and

disabled people.

It is well documented that observing four or five par-

ticipants enables 80% of an interface’s usability problems

to be discovered [20, 28]. As the number of participants

increases, the new problems identified decreases. Some

researchers recommend around eight participants as a good

sample size [24], although this is still an area for debate.

Given that there are no specific rules for sample size

selection for studies of this nature, it was concluded from

the literature on this subject that a sample size of nine for

each group should suffice for the study. This provided with

27 participants. The pilot studies had revealed that vul-

nerable participants feel very uncomfortable in a usability

laboratory setting when on their own, and that a small

group was a much better approach. As well as settling the

participants, this ‘co-discovery’ design for the test gave a

much richer qualitative analysis, as it allowed to observe

and later analyse their conversation and comparative

opinions which formed as they undertook the process of the

test. The co-discovery group size was three participants per

session. Therefore, three sessions with three participants

gave a nine participant per group sample size. The partic-

ipants were particularly pleased with this arrangement,

especially where physical disabilities restricted user inter-

action for some individuals.

The experimental design considered two hypotheses:

H0 The user interface is fit for purpose, to be tested by

quantitative analysis of the questionnaires, and qualitative

analysis of the observation and comments made on ques-

tionnaires in the post-test interviews.

H1 There is no significant difference between question-

naire results provided by the three groups of participants.

This hypothesis was considered important to consider

whether there might be any bias involved in participants

giving overly positive feedback. We conducted an analysis

of variance (ANOVA), with the three groups of partici-

pants as the independent variables, and with the quantita-

tive results of the questionnaires acting as the dependent

variables.

In all usability testing exercises at Glyndwr University,

strict codes of ethical conduct were applied, in accordance

with the adopted EDUCATID methodology( for example,

non-malificence, beneficence, confidentiality, informed

consent, trust, honesty, and integrity). It was particularly

important to make sure that participants felt supported, as

well as comfortable in the laboratory environment. Con-

sequently, the test moderator accompanied them wherever

necessary during experimentation. Dickinson et al’s [5]

recommendations were used of measuring tasks completed

with no assistance, with minimal assistance, and with sig-

nificant researcher intervention.

Table 1 User observation formThe participant Strongly

disagree

Strongly

agree

1 2 3 4 5

P1. Sees and identify the icons/text

P2. Seems to understand the meaning of written

notifications including symbols and icons

P3. Hears the notifications

P4. Seems to understands the meaning of audible notifications

P5. Uses correctly the TV remote control

P6. Seems to understand how to use the product with the TV remote

control

P7. Uses correctly the touch screen

P8. Seems to understand how to use the product with the touch screen

P9. Seems confident using the system

P10. Seems to understand the system

P11. Has learnt easily to use the product

P12. Can the user start and execute a task by

himself/herself without external help?

Univ Access Inf Soc (2012) 11:99–112 103

123

Each usability test session lasted no more than 1 h, as

elderly people in particular grow tired of lengthy testing

[15]. First, each participant filled in the consent form, and

the nature of the test was explained. Second, they were

given a short training session on how to use the interface.

This lasted no more than a few minutes, as it was also

Table 2 Usability experience questionnaire

Usability

1. Do you think the product was easy to use?

Very difficult Difficult Normal Easy Very easy

2. Do you think it was easy to learn how to use the product?

Very difficult Difficult Normal Easy Very easy

3. Do you think the product adapted to your particular needs and abilities?

Totally does not adapt Does not adapt Adapts Adapts well Totally adapts

4. How do you think the product reacts to the different input devices?

a. Touch screen

Very bad Bad Normal Good Very good

b. TV remote control

Very bad Bad Normal Good Very good

Design and layout

5. Do you consider the product attractive?

Totally unattractive Very unattractive Attractive Very attractive Totally attractive

6. Was the screen easy to understand?

Very difficult Difficult Normal Easy Very easy

7. What do you think about the sounds of the product?

Very bad Bad Normal Good Very good

8. Do you think that the spoken notifications were comprehensible?

Totally incomprehensible Very incomprehensible Comprehensible Very comprehensible Totally comprehensible

9. Did you find the sound level adequate?

Totally inadequate Very inadequate Adequate Very adequate Totally adequate

10. In general, how does it look to you?

Very bad Bad Normal Good Very good

Functionality

11. In general, do you think the product is suitable to accomplish the purposes explained at the beginning of the test?

Totally unsuitable Very unsuitable Suitable Very suitable Totally suitable

12. Do you think the product could help you to carry out daily activities?

Totally unhelpful Very unhelpful Helpful Very helpful Totally helpful

13. Do you think the product could increase your quality of life?

Not at all Not much Some Very much Totally

14. Do you think you will could live more independently using this product?

Not at all Not much Some Very much Totally

Satisfaction

15. Were you comfortable using the product?

Totally uncomfortable Very uncomfortable Comfortable Very comfortable Totally comfortable

16. Did you feel embarrassed using the product?

Totally embarrassed Very embarrassed Embarrassed A bit embarrassed Not embarrassed

17. Overall, are you satisfied with the product?

Totally unsatisfied Very unsatisfied Satisfied Very satisfied Totally satisfied

Future use/outcome

18. Do you think you might become dependent on the product if you use it in the future?

Great dependence Much dependence Some dependence Little dependence No dependence

19. Do you consider that this product may isolate you from your actual social relationships?

Total isolation Much isolation Some isolation Little isolation No isolation

104 Univ Access Inf Soc (2012) 11:99–112

123

specifically of interest assess how intuitive the interface

would be. Next, the test was carried out in the direct

presence of the test moderator and observed by the test

observer in the laboratory control booth (behind the two-

way mirror). Finally, participants filled out the user expe-

rience questionnaire. Figure 3 shows an example of one

test.

The test itself comprised four situations, involving the

refrigerator, the oven, and the washing machine (Table 3).

The test moderator performed the physical interaction with

the kitchen appliances, within view of the participants. This

was done for health and safety reasons, and because the

participants were unfamiliar with the washing machine and

oven control panels. As the usability of the appliances

themselves was not being tested, this was deemed to be

acceptable. Also, the test moderator was able to control the

situations to enable consistency across test sessions, for

example by ‘accidentally’ leaving the fridge door open to

cause a reminder of this fact to be displayed on the user

interface.

Participants were asked to watch the television and

make themselves ‘at home’. They were offered a drink and

were given time to settle into the environment. During this

time, they could also familiarize themselves with the

Easyline? interface. Each participant was encouraged to

use the remote control TV handset throughout. The PDA

displaying the working user interface was also passed

around the group, although elderly users in particular

struggled with this small device.

Each situation comprises a number of system ‘scenar-

ios’, which are invoked and controlled by the e-servant. For

example, the ‘fridge door open’ and ‘fridge door open

reminder’ scenarios are involved in the Coming home with

shopping situation. These scenarios are initiated by an

event (for example, the fridge door has been open for too

long) and include a series of actions and associated con-

ditions. The actions may be initiated by a user or by the

system, and conditions are determined by those actions.

Each scenario comprised a structure of these constructs and

was derived from extensive task analysis of typical kitchen

situations earlier in the project development.

The situations were chosen to test scenarios of varying

natures, as well as being relatively familiar to the average

kitchen user. The sequence of situations acted out in the

test followed the script below.

Participants were asked to imagine that they had just

come home with some shopping and needed to put it in the

fridge. They would then bake some croissants, and put

some washing in the laundry. The clothes were displayed to

the participants prior to placing them in the washing

machine. They were clearly incompatible (a white shirt and

red socks), which in the large caused some humour and

consternation amongst participants. The ‘washing machine

wrong mix of clothes’ scenario interrupted their television

viewing, and once they had decided what to do by

acknowledging the notification, the test moderator began

the washing cycle. Meanwhile, the croissants became

ready, and they were duly notified via the ‘oven food

ready’ scenarios. The test moderator served up the food,

Fig. 3 Participants undertaking the usability study. Faces are blurred to preserve anonymity

Table 3 Summary of situations tested

Situation name Description

Coming home withshopping

Some items of food are to be placed in the

fridge. The fridge door is accidentally left

open and the participants are notified of this

fact

A reminder is sent to them if the fridge door

remains open for a given time

Doing my laundry A clothes washing exercise is undertaken.

However, there is a mix of whites and

coloured clothing

The participants are warned that this might

cause a problem and are asked to deal with

the situation

My laundry is done The washing cycle has just finished

The participants are informed of this event and

are reminded if they do not unload the

washing machine

Making dinner Some food is baked in the oven

Participants are informed when the cooking is

finished and the food is ready

Univ Access Inf Soc (2012) 11:99–112 105

123

and conversation was recorded whilst they ate and watched

television, until the 10 min washing cycle had completed,

invoking the ‘washing machine cycle finished’ scenarios.

6 Participants

The participants were formed into three groups of nine,

which were termed control, elderly, and learning, and

which refer, respectively, to the control group, the elderly

participants, and the group of cognitive disabled people.

Fourteen participants were female, the other thirteen being

male. Of the elderly group, one was aged over 80, four in

their 70’s, one in her 60’s, and three in their late 50’s. All

participants in this group had a range of physical and

sensory impairments relating to conditions associated with

ageing. One participant had mild cognitive impairments

related to Parkinson’s disease, in addition to marked

physical disabilities typical of this disease. Other condi-

tions included arthritis, failing hearing and vision, glau-

coma, amputation, and paraplegia. The learning group was

recruited from a further education college, which special-

izes in teaching people with learning disabilities. All these

participants were below 50 in age.

7 Analysis of results

7.1 Questionnaire analysis (quantitative)

The results of the user experience questionnaire used for

interface usability testing are shown in Table 4. For every

question, participants were asked to provide specific

comments. A semantic rating scale was used for each

question, each of which was multi-choice in nature, and

enabling one of five responses. The responses were con-

verted to a numerical range from zero to four.

Overall mean and standard deviation scores for each

question and for the three group types were calculated.

Each individual session’s set of statistics was also ana-

lysed to consider whether any peer pressure or other group

dynamic may have affected the results. The only obser-

vation of this was for one of the learning groups, where

two participants were frightened by the notification

sounds.

In order to investigate the possibility of bias between the

three main groups, an Analysis of Variance (ANOVA) was

carried out of every question in the user experience ques-

tionnaire. This analysis revealed that there were no sig-

nificant differences in the responses provided by the three

groups (F = 1.52; p \ 0.05), apart from one question (16):

‘‘Did you feel embarrassed using the product?’’

The control group and the elderly group expressed no

embarrassment at using the interface, whereas some

members of the cognitive learning difficulties group were

uncomfortable with it from this point of view. In particular,

two participants were visibly startled when a notification

alarm sounded, and this clearly affected their experience.

It is interesting, however, that overall the control group

provided consistently slightly higher ratings for most of the

questions. This could be explained by this group’s positive

comments on the potential of the product, as opposed to its

current benefits.

It can be seen that all but one of the questions resulted

on average in a positive response (i.e. [ 2/4). The only

question not to achieve this was question 18:

‘‘Do you think you might end up dependent of the

product if you use it in the future?’’

This issue does raise some concern, although it could be

argued that the one of the aims of the project is to create a

certain amount of dependency on this type of product, to

support people’s independent living.

Questions were categorized into five themes: general

usability, interface design, interface functionality, user

satisfaction, and overall outcome/opinions. The mean

scores per category are displayed in Table 5.

Table 4 Mean results for each question

Overall

average

Control

average

Elderly

average

Learning

average

Question 1 2.85 3.33 2.67 2.56

Question 2 3.07 3.44 3.11 2.67

Question 3 2.70 2.89 2.33 2.89

Question 4a 2.86 3.00 2.86 2.67

Question 4b 2.88 3.44 3.00 2.00

Question 5 2.46 2.44 2.38 2.56

Question 6 3.11 3.56 3.00 2.78

Question 7 2.59 2.44 2.89 2.44

Question 8 2.48 2.78 2.67 2.00

Question 9 2.44 2.78 2.44 2.11

Question 10 3.22 3.44 2.89 3.33

Question 11 2.81 3.11 2.89 2.44

Question 12 2.92 3.33 2.88 2.56

Question 13 2.70 3.00 2.67 2.44

Question 14 2.31 2.11 2.50 2.33

Question 15 2.80 3.11 2.78 2.43

Question 16 3.52 4.00 3.78 2.57

Question 17 2.67 2.89 2.67 2.44

Question 18 1.82 1.89 2.00 1.40

Question 19 3.28 3.44 3.56 2.71

Average satisfaction 2.78 3.02 2.80 2.47

106 Univ Access Inf Soc (2012) 11:99–112

123

The aggregated results for every category and for all

groups clearly indicate a positive outcome for the quanti-

tative element of the usability experience questionnaire.

All results are higher than the median value of two. It can

be seen in every category that the control group scored

highest, followed by the elderly group, with the learning

group scoring the lowest in all cases, except in the ‘Future

use/outcome’ category, where the elderly group gave the

most positive response.

7.2 Questionnaire analysis (qualitative)

Participants were invited to comment on their answer to

every question. Where a participant was unable to write, their

carer filled out the comments. Most comments were provided

understandably by the control group, although the elderly

group also made several comments. In the large, comments

were favourable, although there were some themes which

suggest improvements could be made to the interface design:

1. The TV remote control buttons were too small—whilst

this is strictly speaking outside the scope of the

interface design of Easyline?, care should have been

taken to provide a range of remote control handsets to

support differing needs (comments for question 4).

2. The interface reacted slowly to button presses—this is

a current limitation of the technology’s client–server

model (comments for question 4).

3. More graphical indications were suggested, both in

messages and on-screen buttons (comments for ques-

tions 6 and 201).

4. There were some real problems with the alarm

notification, especially as it was considered frightening

by a number of participants (comments for questions 7,

9 and 20).

7.3 Observational analysis

All test sessions were recorded with three video cameras

observing the participants, with a fourth recording fed

through the user interface to enable synchronization

between user behaviour and interface activity. During the

sessions, the test observer also made real-time notes of the

session’s progress. In terms of timing of reaction to events,

there were no discernible problems or issues. Participants

responded quickly and positively.

User misunderstandings and errors when using the

interface were also analysed. The only problem recorded,

and this happened on a number of occasions with different

groups of users, was where there was confusion between

whether to press the HELP or the OK button in response to

a notification. A design decision to provide help and

tutorial support from any point during the user experience

was taken early in the project’s development. However,

this has been re-evaluated, as experience so far has been

that the interface is particularly intuitive and that always

offering help can sometimes confuse the flow of

interaction.

Where users were surprised or frightened by the notifi-

cation sound was again very clear during the observations.

It is a difficult compromise to reach between making sure

users are aware of potentially problematic and dangerous

events, and minimizing fear and panic. It was decided that

a best-fit solution for this would be to use a range of dif-

ferent sounds for varying levels of seriousness, rather than

a universal audible signal.

The learning group was split into three subgroups of

varying ability in consultation with the carers who

accompanied them. The weakest group struggled funda-

mentally with the concept and the interface, whereas the

two stronger groups coped easily. Observing this experi-

ence suggests, in comparison with elderly people with

dementia, that only those with mild forms of dementia

could benefit from the product.

One final observation was that some users expressed

dissatisfaction when their TV programme was interrupted

by an uncritical notification. Perhaps alternative configur-

ations should be considered, for example a pop-up message

overlayed at the bottom of the screen.

The results of the user observation form were collated

on a per-session basis, taking each group of three partici-

pants as a collated observation. Although there was some

variance within the groups, this was minimal. The results

are displayed in Table 6.

The mean results overall and for each of the three group

types are displayed in Table 7.

The overall average results are clearly biased due to the

control group’s ease of interaction. The elderly group’s

difficulties with the input devices are clear to see in P5 and

P7. Aside from these discrepancies, the results of the

observation are positive, with most scores higher than the

median value of three. This set of results correlate with

the results from the usability experience questionnaire,

Table 5 Mean results by questionnaire category

Overall

average

Control

average

Elderly

average

Learning

average

Usability 2.87 3.22 2.79 2.56

Design and layout 2.71 2.90 2.71 2.54

Functionality 2.67 2.89 2.74 2.44

Satisfaction 3.00 3.33 3.08 2.48

Future use/outcome 2.55 2.67 2.78 2.01

1 Question 20, which is not shown in the analysis, asked the

participant the open question: how would you improve the product?

Univ Access Inf Soc (2012) 11:99–112 107

123

with the control group scoring highest, followed by the

elderly group, with the learning group scoring lowest

again.

7.4 Summary of results analysis

Generally, the results of the usability testing of the Easy-

line? interface were positive. The quantitative and quali-

tative analysis confirm the two hypotheses H0—The user

interface is fit for purpose, and H1—There is no significant

difference between questionnaire results provided by the

three groups of participants. The study design gave a rich

and reliable set of results. The utilization of the usability

laboratory helped triangulate the results of the usability

questionnaire with the observational analysis.

Following the study, the development team were able to

identify improvements that could be made to the interface

design, specifically:

• Introducing graphical pictorial indications, both in

messages and on-screen buttons.

• Changing the alarm sounds, and use a range of different

ones for critical, non-critical and reminder notifications.

• Re-appraising the requirement of a ubiquitous HELP

function to avoid confusion with the OK button when a

notification arrives.

• Introducing a configurable notification indication to

avoid interruption of TV programmes (when the

television is used for this functionality).

These design changes have been implemented, tested

and, are being evaluated in the longitudinal study.

8 Conclusion and recommendations

This paper has presented the rationale for, and the process

of, conducting laboratory-based usability studies with

vulnerable users. The essential aim of the project is to

improve the quality of life of elderly and disabled people

and to extend their independent living. It is well known that

individuals value their independence and anything that can

help them to stay in their own homes for as long as possible

will be popular, as well as potentially saving the healthcare

services huge sums of money. Changing population dis-

tributions mean that we have no choice but to find tech-

nological solutions to the problem of an ageing society.

The development of the Easyline? project has presented

with challenges as to how best to evaluate products that

Table 6 User observation

exercise results

C, E, L abbreviations used for

Control, Elderly, and Learning

groups, respectively

The participant Strongly disagree Strongly

agree

1 2 3 4 5

P1. Sees and identify the icons/text L E EEL CCCL

P2. Seems to understand the meaning of written notifications including

symbols and icons

L E EEL CCCL

P3. Hears the notifications L CCCEEELL

P4. Seems to understands the meaning of audible notifications L E EL CCCEL

P5. Uses correctly the TV remote control EL E ELL CCC

P6. Seems to understand how to use the product with the TV remote

control

L CCCEEELL

P7. Uses correctly the touch screen EL E L EL CCC

P8. Seems to understand how to use the product with the touch screen L L L CCCEEE

P9. Seems confident using the system L E EL CCCEL

P10. Seems to understand the system L L CCCEEEL

P11. Has learnt easily to use the product L ELL CCCEE

P12. Can the user start and execute a task by himself/herself without

external help?

L LL CCCEEE

Table 7 Mean results for the user observation exercise

Overall

average

Control

average

Elderly

average

Learning

average

P1 4 5 3.7 3.3

P2 4 5 3.7 3.3

P3 4.9 5 5 4.7

P4 4.1 5 4 3.3

P5 3.6 5 2.7 3

P6 4.6 5 5 3.7

P7 3.3 5 2.3 2.7

P8 4.2 5 5 2.7

P9 4.1 5 4 3.3

P10 4.4 5 5 3.3

P11 4.2 5 4.7 3

P12 4.3 5 5 3

108 Univ Access Inf Soc (2012) 11:99–112

123

will be used by the user population in question. Ethical

issues are key where vulnerable people are concerned. A

methodological approach called EDUCATID was used,

which places ethical considerations at the centre of the

project’s development cycle. A range of evaluation meth-

ods was used throughout, and care was taken to set up a

summative evaluation which would be valid and reliable.

Ultimately, the approach of laboratory-based usability

evaluation was adopted. In order to do this effectively, it

was important to make the environment as comfortable and

as realistic as possible for the participants. It should be

emphasized that the laboratory-based study presented here

concentrated specifically on usability evaluation. Full

product evaluation follows this exercise and takes place as

a longitudinal study in a real home.

The execution of the usability evaluation has been

somewhat of a learning experience, due to the uniqueness

of the participants and the context of the project. Most

guidance in terms of usability evaluation available in the

literature concentrates either on office-based applications

or on website evaluation. Whilst much of this can be

adapted and may be generically useful, the acquired

experience suggests that specific guidelines for usability

evaluation involving vulnerable users are required. As

quality of life is usually a key aim in developing infor-

mation technology (IT) for vulnerable users, it is argued

that this will be an important issue for many health-related

IT projects. Based on the obtained experience, the following

recommendations are put forward for future projects, which

may involve such users in usability laboratory-based

evaluation.

1. Settle the participants—make them comfortable.

This is a generic principle, but here the aim is to

emphasize the need to make vulnerable participants par-

ticularly comfortable. Provide refreshments if possible.

Give the participants time to settle into what might seem a

strange environment. Stay in the laboratory with partici-

pants until they are happy to proceed with the tests.

Cameras and two-way mirrors may be particularly intimi-

dating, so reassure the participants of their anonymity and

emphasize the professionalism of the researchers involved.

In the conducted study, the participants with learning dif-

ficulties seemed especially conscious that their identities

should be protected.

2. The orientation script needs a degree of flexibility.

Use an orientation script for the purposes of reliable

testing, but do not expect it to work when read word-for-

word prior to each session. An orientation script was used

for this study, which is presented in ‘‘Appendix’’, but it

became clear very early in the pilot studies that a degree of

flexibility would be required to account for the diversity of

participants involved. Sensory difficulties, such as partial

deafness and cognitive difficulties, must be accounted for

[1]. Sometimes things need to be read out twice or more or

explained in simpler terms. The diverse age ranges of the

participants also required sensitivity, as terminology is

familiar for some people and not for others. The use of any

jargon should be avoided, but in today’s hi-tech world, that

is not as straightforward as it may seem. Words such as

‘internet’ and even ‘computer’ are still not universally used

household terms, especially when elderly people are con-

cerned, although there does appear to be a growing trend of

computer-literate elderly people [11]. Consequently,

adaptation of the orientation script on the fly is a recom-

mendation. This requires the application of interpersonal

communication skills, which should be practised during the

pilot study.

3. Always demonstrate the product prior to the test.

The orientation script states that a short demonstration

of the product should be given prior to the usability eval-

uation test. This reassures potentially reticent participants

and helps them to relax into the environment. In the con-

ducted evaluation, a situation was demonstrated which did

not use any scenarios used in the test itself, to avoid

obvious bias.

4. Do not put vulnerable people on their own in a

usability test unless absolutely necessary.

In the pilot studies, it was immediately apparent that

participants felt particularly uncomfortable when faced

with the prospect of being tested alone. The experimental

design placed participants in groups of three in a ‘co-dis-

covery’ situation. This proved very valuable for the

observational analysis, as participants’ conversations about

the product were recorded and analysed. On more than one

occasion when the nature of the usability study was

explained prior to the test, participants expressed relief that

they would not have to conduct the tests in isolation. Bias

within a co-discovery group may be a negative factor with

this approach. It is therefore important to be prepared for

this in subsequent analysis. This issue was experienced in

one co-discovery group, but the fact that there was an issue

actually highlighted a usability problem also felt by other

groups in the evaluation (the ‘alarming’ auditory warning

which frightened some of the participants with learning

difficulties).

5. Keep it short.

It is important that participants are given time to settle

into the environment. However, the usability test itself

should be relatively short for vulnerable people who may

get tired, may have attention span problems, and may

become uncomfortable in a laboratory setting. It is

Univ Access Inf Soc (2012) 11:99–112 109

123

recommended that from start to finish (that is, from par-

ticipants entering the laboratory to leaving it) a session

should last no more than 1 h.

6. Provide a realistic, familiar layout where possible.

This is another generically applicable principle but is

especially important for vulnerable participants. Distract-

ing participants and keeping them occupied with familiar

artefacts can help to alleviate concerns over cameras,

microphones, and two-way mirrors. In the conducted

experiment, books, newspapers, and magazines were used,

along with the television set which was conveniently cen-

tral to the study in question. A reassuring hot drink was

also provided.

7. Ensure the laboratory is accessible and flexible.

Elderly and disabled participants may have a range of

disabilities, and the laboratory should be able to accom-

modate for them wherever possible. Wheelchair users in

particular should have adequate access. The laboratory

layout should be flexible enough to enable fixtures to be

easily moved out if necessary, and for extra facilities to be

moved in.

8. Use realistic and familiar scenarios.

Make sure that participants are testing the product, and

not learning a new scenario or situation, as this will

adversely affect the test. Include questions in the user

profile questionnaire that elicit the participants’ experience

of the scenarios to ensure this. Questions about familiarity

with kitchen appliances and domestic activities were asked

prior to initiation of the evaluation in the conducted

experiment.

9. Prioritize ethical and legal issues.

Working with vulnerable people in any way requires

ethical and legal procedures to be followed. Usability

testing at Glyndwr isbound by University’s own strict

codes of ethical conduct, and the EDUCATID helped to

adhere to them (for example, non-malificence, beneficence,

confidentiality, informed consent, trust, honesty, and

integrity). Similar codes of practice may be obtained online

through the British Psychological Society [25] and The

British Sociological Association [26]. Not only do ethical

and legal issues need to be adhered to, they should be

recorded for auditing purposes.

10. Prioritize health and safety issues.

Any usability laboratory environment should conform to

health and safety policies. Where vulnerable people are

involved, their exposure to potentially hazardous situations

must be avoided. Consequently, there may be a require-

ment for intervention by the usability evaluators to

facilitate a test. In the specific case, the test moderator was

responsible for controlling and interacting with the kitchen

appliances. The participants were unfamiliar with the

appliance controls, and training them in these was con-

sidered unnecessary and may have introduced confusion

with respect to learning the user interface being tested,

regardless of the obvious hazards of using an unfamiliar

oven, for example.

11. Do not rely purely on laboratory-based studies.

The laboratory-based evaluation is a very useful tool

amongst many. The obtained results provided confidence

to move to a more intensive longitudinal evaluation.

Participative methods (for example, focus groups and

workshops) have also been useful in the Easyline? pro-

ject, although from a practical point of view these can be

time-consuming and expensive when relied on too heav-

ily. It is recommended to use a range of evaluative tools

in the process of developing products for diverse user

populations, including longer-term studies in the field as

exemplified by Intel’s Global Ageing Experience [23].

Whatever tools are used, the Hawthorne effect must be

carefully considered. The use of a control group in the

laboratory-based evaluation helped overcome concern

over this factor.

Traditionally, laboratory-based experiments have been

used to provide a controlled environment, where factors

which might adversely bias testing can be nullified.

Repeated testing for reliability and rigorous application of

an experimental design is typically expected to enable

hypotheses to be accepted or rejected.

The methodology followed for the usability evaluation

of Easyline? is intentionally not so scientific. Previous

experience with working with vulnerable people has led to

taking a more flexible, pragmatic, and sensitive approach to

the laboratory-based study. Although it cannot be claimed

that the results of the study are statistically watertight,

valuable insights have been gained into the usability of the

products developed through the use of this approach.

Acknowledgments This work was supported by the European

Union funded framework 6 project Easyline?: Low cost advancewhite goods for a longer independent life of elderly people. [INFSO-

IST-045515]. The authors would like to thank all participants

involved in the study, and in particular the staff and students of

Derwen College, Oswestry, and the members of the Outside/In group,

Wrexham.

Appendix: Orientation script used prior to usability

evaluation tests

‘‘First of all, welcome to the usability lab and thank you for

agreeing to participate in this evaluation study.

110 Univ Access Inf Soc (2012) 11:99–112

123

As you can see, we’ve designed this lab to look like a

small living room with a kitchen area. It looks like this

because we want to test a new product which aims to

make life easier for people who are not as mobile as they

once were—perhaps through age or disability. The idea is

that when something needs to be dealt with in the house,

you don’t necessarily have to run around straight away to

sort it out. An example might be that you’ve accidentally

left the oven on. Using this product, you can switch it off

from here, rather than have to go into the kitchen to sort it

out.

We’re going to do that type of thing in this room. When

something needs to be dealt with, it will attract your

attention through the different displays and help you sort it

out. We’ll show you how it works in a little whilst, but in

the meantime, we want you to make yourselves comfort-

able. There’s a television for you to watch, some news-

papers and magazines, and plenty of books on the

bookshelf. We have drinks too. Would you like a hot drink

perhaps?

hmake drinksiToday you will be asked to perform some tasks using

our system. Our goal is to observe your interaction with it.

This will help us to improve the system over time. So, you

will have to search for information and complete scenarios

or tasks using these interfaces.

hShow and explain system/interfaces/appliances and the

various input devices at handiYour reactions and comments of the interfaces you’ll

view will be recorded. In a room nearby, there is someone

who also observe your interaction with the system and

appliances. During this session I will not be able to offer

any suggestions or hints. There may be times, however,

you will be asked to explain why you said or did some-

thing. You also will be asked a series of questions about

your experience at the end of this session.

Here are some things that you should know about your

participation:

• This is not a test of you; you’re testing the system. So

don’t worry about making mistakes.

• There is no right or wrong answer. We really just want

to know if we designed the interfaces well for you.

• If you ever feel that you are lost or cannot complete a

scenario with the information that you have been

given, please let us know. We’ll ask you what you

might do in a real-world setting and then either put

you on the right track or move you on to the next

scenario.

• We will be video recording this session for further study

if needed. Your name will not be associated or reported

with data or findings from this evaluation. Please fill

out this consent form.

• Finally, as you use the system, please behave as you

would at home.

Now, let’s begin, shall we?’’

References

1. Astell, A., Alm, N., Gowans, G., Ellis, M., Dye, R., Vaughan, P.:

Involving older people with dementia and their carers in

designing computer based support systems: some methodological

considerations. Univers. Access Inf. Soc. 8, 49–58 (2009)

2. Blythe, M., Dearden, A.: Representing older people: towards

meaningful images of the user in design scenarios. Univers.

Access Inf. Soc. 8, 21–32 (2009)

3. Cook, D.J.: Providing for older adults using smart environment

technologies. IEEE USA Today’s Engineer Online. http://

www.todaysengineer.org/2007/May/smart_homes.asp 5(7) (2007).

Accessed 16 June 2009

4. Cooper, A.: The Inmates are Running the Asylum, 1st edn. Sams

Publishing, USA (2009)

5. Dickinson, A., Newell, A.F., Smith, M.J., Hill, R.: Introducing the

internet to the over-60’s: developing an email system for older

novice computer users. Interact. Comput. 17(6), 621–642 (2005)

6. Eisma, R., Dickinson, A., Goodman, J., Syme, A., Tiwari, L.,

Newell, A.: Early user involvement in the development of infor-

mation technology-related products for older people. Univers.

Access Inf. Soc. 3(2), 131–140 (2004)

7. Eurostat: Demographic Change: Challenge or Opportunity?

http://epp.eurostat.ec.europa.eu/ (2009). Accessed 16 June 2009

8. Eurostat: Europa—Eurostat. http://epp.eurostat.ec.europa.eu/ (2008).

Accessed 31 Jan 2008

9. Goodman, J., Brewster, S., Gray, P.: Not Just a Matter of Design:

Key Issues Surrounding the Inclusive Design Process. In: Pro-

ceedings of Include. Helen Hamlyn Research Centre, London,

UK, April 2005

10. Goodman, J., Lundell, J.: HCI and the older population. Interact.

Comput. 17(6), 613–620 (2005)

11. Harley, D., Fitzpatrick, G.: YouTube and intergenerational

communication: the case of Geriatric1927. Univers. Access Inf.

Soc. 8, 5–20 (2009)

12. Holzinger, A.: Usability engineering for software developers.

Commun. ACM. 48(1), 71–74 (2005)

13. Holzinger, A., Searle, G., Nischelwitzer, A.: On some aspects of

improving mobile applications for the elderly. In: Stephanidis, C.

(ed.) Universal Access in Human Computer Interaction. Coping

with Diversity: Research and Development Methods in Universal

Access, Programming and Software Engineering. Lecture Notes

in Computer Science (LNCS 4554), pp. 923–932. Springer,

Berlin (2007)

14. Holzinger, A., Searle, G., Kleinberger, T., Seffah, A., Javahery,

H.: Investigating usability metrics for the design and develop-

ment of applications for the elderly. In: Miesenberger, K., Klaus,

J., Zagler, W., Karshmer, A. (eds.) Proceedings of 11th Interna-

tional Conference on Computers Helping People with Special

Needs (ICCHP 2008). Lecture Notes in Computer Science

(LNCS 5105), pp. 98–105. Springer, Berlin (2008)

15. Kayser-Jones, J., Koenig, B.A.: Ethical issues. In: Gubrium, J.F.,

Sanker, A. (eds.) Qualitative Methods in Aging Research,

pp. 15–32. Sage Publications, Thousands Oaks (1994)

16. Kleinberger, T., Becker, M., Ras, E., Holzinger, A., Muller, P.:

Ambient intelligence in assisted living: enable elderly people to

handle future interfaces. In: Stephanidis, C. (ed.) Universal

Univ Access Inf Soc (2012) 11:99–112 111

123

Access in Human-Computer Interaction: Ambient Interaction,

Lecture Notes in Computer Science (LNCS 4555), pp. 103–112.

Springer, Berlin (2007)

17. Miskelly, F.G.: Assistive technology in elderly care. Age Ageing

Oxf. J. 30(6), 455–458 (2001)

18. Newell, A.F., Gregor, P.: User sensitive inclusive design in

search of a new paradigm. In: Proceedings of the First ACM

Conference on Universal Usability. Arlington VA, USA,

pp. 39–44 (2000)

19. Newell, A.F., Carmichael, A., Morgan, M., Dickinson, A.: The

use of theatre in requirements gathering and usability studies.

Interact. Comput. 18, 996–1011 (2006)

20. Nielsen, J., Landauer, T.K.: A mathematical model of the finding

of usability problems. In; Proceedings of ACM INTERCHI’93

Conference, pp. 206–213. ACM Press, Amsterdam (1993)

21. Park, D., Schwarz, N. (eds.): Cognitive Aging: A Primer. Psy-

chology Press, Taylor and Francis Group, Hove. p. 238 (2000)

22. Picking, R., Robinet, A., Grout, V., McGinn, J., Roy, A., Ellis, S.,

Oram, D.: A case study using a methodological approach to

developing user interfaces for elderly and disabled people.

Comput. J. Adv. Access. Published on 23 Oct 2009. doi:10.1093/

comjnl/bxp089

23. Prendergast, D., Roberts, S.: Practice, systems and technology for

seniors. Univers. Access Inf. Soc. 8, 59–61 (2009)

24. Rubin, J., Chisnell, D.: Handbook of Usability Testing: How to

Plan, Design and Conduct Effective Tests. Wiley Technical

Communications Library (2008)

25. The British Psychological Society: Code of conduct and ethical

guidelines. http://www.bps.org.uk/ (2009). Accessed 18 June 2009

26. The British Sociological Association: Statement of ethical prac-

tice. http://www.britsoc.co.uk/ (2009). Accessed 18 June 2009

27. Vickers, P., Banwell, L., Heaford, S., Sainz de Salces, F.J.:

Painting the ideal home: using art to express visions of technologi-

cally supported independent living for older people in north-east

England. Univers. Access Inf. Soc. 8, 33–47 (2009)

28. Virzi, R.A.: Refining the test phase of usability evaluation: how

many subjects is enough? Human Factors. 34, 457–468 (1992)

29. Walsham, G.: Ethical theory, codes of ethics and IS practice. Inf.

Syst. 6, 69–81 (1996)

30. Zajicek, M.: Interface support for elderly people with impaired

sight and memory. In: Proceedings of 6th European Research

Consortium for Informatics and Mathematics (ERCIM) Work-

shop ‘User Interfaces for All’. CNR-IROE, Florence, Italy, 25–26

Oct 2000

112 Univ Access Inf Soc (2012) 11:99–112

123