User Centered Design and Evaluation in MASELTOV London, 17 March 2015

Stephanie Schwarz

AIT Austrian Institute of Technology GmbH

Innovation Systems Department, Technology Experience

Overview

Motivation and Objectives

MASELTOV Users

User Centered Design

Field Trials

Ethical Issues

Data Collection

Challenges

2

Requirement Analysis and Interaction Design

Motivation

Gain a broad knowledge about how ICT can support integration

Understand the according needs and wishes of immigrants

(MASELTOV target group)

Deduce the requirements for the planned mobile assistant

Objectives

Investigate particularities of cross-cultural design aspects

Design the interactive behaviour and the user interface of

MASELTOV

3

Evaluation and Field Trials

Motivation

To evaluate the designed MASELTOV services with immigrants

in three European countries (Austria, United Kingdom, Spain)

To involve users early in the design process

To support design and implementation activities of partners by

presenting the gathered feedback

Objectives

To detect conformance of designed services with users’ needs

and expectations

To measure acceptance and user experience with the

MASELTOV services during long-term usage

4

MASELTOV Users

Focus on specifically targeted problems of user groups having a

similar background in the following areas:

Third Country Nationals (i.e. with non-EU nationality) having

Arabic, Turkish or Spanish as maternal language

Current length of stay in the EU (up to 5 years)

Rather low education but excluding illiterates

In working age (i.e. between 18 and 60 years old)

5

User-Centered Design

Involve immigrants right from the start of the project to make

sure MASELTOV meets immigrants’ needs

6

Understand Users

Aggregated barriers list of predominant problems based on

collected input from NGOs

23 semi-structured interviews to deepening insights on living

circumstances and everyday problems

7

Define Interactions

Discussed service and design ideas within focus groups

Conducted participatory design sessions to benefit from

ideas and cultural background of immigrants

9

Use cases and scenario-driven approach lead to Mobile

Services in MASELTOV

10

On the spot assistance

Connecting with peers

and locals

Language learning

(Playful) learning of

cultural differences

Orientation and security

Access to information

•Help Radar

•Social Network Service

•Language lessons

•Translation tool

•Serious Game

•Navigation and POI

• Information Service

•Recommendations

Design User Interfaces

Created and iteratively evaluated User Interfaces within 3

stages with experts and immigrants:

Sketches

Click-dummies

High fidelity prototypes (incl. visual design)

11

Evaluate User Interfaces in the Lab

12

1st Lab Evaluation

12/12

2 Usability experts

Vienna/AT

2nd Lab Evaluation

04/13

9 long term immigrants

(Arabic, Latin American)

London/UK

in English

3rd Lab Evaluation

09/13

19 immigrants

London/UK

Graz/AT

in users´mother tongue

Many Interfaces

In Field Experience Evaluation

14

First Field Trials

06/14

30 immigrants

(Turkish, Latin-American, Arabic)

Graz/AT, London/UK, Madrid/ES

Final Field Trials

10/14 – 02/15

73 immigrants

(Turkish, Latin-American, Arabic)

Graz/AT, London/UK, Milton

Keynes/UK

Data Collection: objective measures

MApp service usage frequency and duration

Contextual data (location, mode of transportation, visited places)

15

00:00

14:24

28:48

43:12

57:36

72:00

86:24

00:00

288:00

576:00

864:00

1152:00

1440:00

1728:00

1 2 3 4 5 6 7 8 9

Avg

. T

ime

(m

in)

To

tal ti

me

(m

in)

Week

Weekly time spent on the MASELTOV services in total and avg. per group

Total Time (T) Total Time (L) Total Time (A)

Average (T) Average (L) Average (A)

00:00

72:00

144:00

216:00

288:00

360:00

432:00

504:00

00:00

720:00

1440:00

2160:00

2880:00

3600:00

4320:00

pro

file

tra

nsla

te

ped

na

vi

AR

poi

help

rad

foru

m

lan

gu

_l

gam

e

reco

mm

info

Avg

.

Tim

e (

min

) Services

Usage Time of MASELTOV Services in Total and Average

Total time (min) Average

Data Collection: subjective measures

User Experience (Satisfaction, Privacy)

Subjective wellbeing, perceived usability

Goal achievement

16

0,00

0,50

1,00

1,50

2,00

2,50

3,00

3,50

4,00

4,50

5,00

Sa

tis

fac

tio

n

Week 2

Week 5

Week 8

Data Collection: subjective measures

Facilitator and volunteer assistance protocols

Qualitative feedback in workshops

17

Ethical Issues

Personal Data Security and Privacy during user trials

Anonymity

Secure servers

Explicit allowance for logging data collection

Informed Consent

Information about collection of specific data (Profile)

Location data

Disclaimer text at first login

Communication and community services (Help Radar,

Forum)

Volunteer recruitment via NGOs

Hidden Assistance history

Forum administrator

Ethics Code of Practice

18

Methodological challenges, scientific value

Suitability of given research methods and techniques

Coordination work and quality of the translation processes

Language of data collection

Recruiting and matching of samples

Timing of data collection and analysis

Comparability of the research process and instruments

Lessons learned

Collaborating with NGO‘s

Involving facilitators and volunteers from the community

Benefiting from reflections of long-term immigrants

Taking gender issues into account

Involving researchers with immigration background

Any many more

19

Executive Summary

User Centred Design for and with immigrants (Co-creation)

Evaluations in different fidelity stages

MASELTOV as useful and well appreciated toolbox for immigrants with

different cultural origins

Language learning on the move

Social and assistance tools

20

MASELTOV !

Stephanie Schwarz

AIT Austrian Institute of Technology GmbH

Giefinggasse 2 | 1210 Vienna | Austria

T +43 50550-4533 | F +43 50550-4599

stephanie.schwarz@ait.ac.at | http://www.ait.ac.at

21