Evaluating Usability and User Experience
in Non-Traditional Environments
IHC 2008 Porto Alegre, Brasil
21 to 24 October, 2008
Regina Bernhaupt
Think about it ….
*How to evaluate the usability of this presentation …
* And how to evalute the User Experience during this
presentation …
*Stressed
*Not being well
*Critical Situations
*Relax
* Fun
*Operational Situations
Non-Traditional Environments
Evaluating Usability & UX in Non-Traditional Environments
Overview
*Usability and User Experience as Key Concepts
*Current Evaluation Approaches
* Three Case Studies: Adoption and Development of
Evaluation Methods
*How to evaluate in non-traditional environments
Usability and UX as Key Concepts
Definition of Usability
*Usability is concerned with making
systems easy to learn and use.
* A usable system is one which
enables users to perform their job
effectively and efficiently.
*Usability is the „effectiveness, efficiency
and satisfaction with which a specified
set of users can achieve a specified
set of tasks in a particular environment“.
ISO 9241-11
Usability Decomposed
* Flexibility: Dialog Initiative, Multi-threading,
Task migratability, Substitutivity, Customizability, …
* Learnability: Predictability, Synthesizability, Familiarity,
Generalizability, Consistency, ….
*Robustness: Observability, Recoverability,
Responsiveness, Task Conformance, …..
Fun as Factor of Usability (Caroll and Thomas, 1988)
„We realize that many people will readreadreadread this article as a
joke. To this extend, we are the victims of our own
analysis: there are risks in being serious about fun.
Still though, we continue to see, without humor, the
prospect of a decade of research analysis possibly failing
to provide the leverage it could on designing systems
people will really want to use by ignoring what could be a
very potent determinant of subjective judgements of
usability – FUN“ (p. 23)
Is User Experience different from Usability?
For Playing the Game:press any keyCongratulations – you won!!
Defining UX
*Experiences, Feelings and Needs of the user before,
during and after interacting with the product in various
contexts.
*How do we evaluate?
Current Evaluation Approaches
Evaluation: Process Model
Limitations of Evaluation Methods
*Real usage and user experiences differ from lab usage
* Field studies: adoption of methods needed
*UX is difficult to evalute: lots of factors
* UX: when do we evaluate what and how?
Categories of Evaluation: Usability
HeuristicEvaluation
DesignRationale
Model-basedEvaluation
User Testing
Experiments
User Studies
Adoption and Development of New Evaluation Methods
Case Study 1: Satellite Control Room
*Usage of new interaction technique
· Multimodal Interfaces
· Multi-touch including voice
· Usability Problems lead to enormous damage/loss
Case Study 1: Satellite Control Room
*Problems:
· You are not allowed to access the users
· You can not test in real situations
· Usability Problems lead to considerable
damage/loss
*Solution:
· Full modeling of the interaction
· Combine formal modeling state description of
the model to investigate usability problems (*)
· (*) Bernhaupt, Palanque, Winckler, Navarre (2007); Bernhaupt, Palanque (in preparation)
Adapting Usability Evaluation Methods
*Combination of Methods:
· Usability Studies with model-based evaluation (*)
* Adapting Methods:
· Heuristic Evaluation with usage scenarios (videos) (**)
· Cognitive Walkthrough with usage scenarios (*)
· Performing Heuristic Evaluation in the field (**)
*Bringing the „Field“ into the „Lab“
· Sports equipment to simulate running (***)
* In general: Combination and embedding of methods
* (*) Bernhaupt et al. (2007) (**) Po, Howard, Vetere, Skov (2004) (***) Kjeldskov and Stage (2003)
Adopting Usability Evaluation Methods
for Non-Traditional Environments
HeuristicEvaluation
DesignRationale
Model-basedEvaluation
User Testing
Experiments
User Studies
User Experience Evaluation
Categories of Evaluation: User Experience
Focus GroupsQuestionnaire
Probing
Experiments User Studies
Interviews
UX Sampling
Play Testing
Case Study 2: Playful Probing
Cultural Probing and Variations
*Probing Approach (Gaver et al., 1999)
*Several Variations: Mobile Probes, Technology Probes, …
* Insights into daily life e.g. to understand user experiencefactors
Playful Probing
*Problem:
· Privacy
· Real usage situations: long-term usage
· Adressing all users during study
*Solution (*) :
· Make Probing „Playful“
· Develop Games to extend the methods
· Use creative/playful material/questions
* (*) Bernhaupt, Weiss, Obrist, Tscheligi (2007),
Playful Probing: Material
*Games, especially developed to address the research
questions, to increase participation
Playful Probing: Material
Playful Probing: Results (*)
* more involvement
* more insights
* more „material“
* helps investigating UX
(*) e.g. Bernhaupt et al. 2007
·
Case Study 3: Evaluating UX in Games
*Developing/Using Games to evaluate aspects of UX
*Effects of Emotion
*Emotional Flower can be played by a group of people
*We use an ambient display to enable the players to see
the progress of others
* All flowers including the chosen nickname are displayed,
the screen is in a public area
Social and Ambient Factors related to UX
*Need for attentive user interfaces
*Emotions are a major
influence on peoples attitude
*Emotional Interfaces are
dealing with human emotions
*Goal: make UX positive
Investigation aspects of UX: Manipulating Emotions
Results (*)
*Emotion can be influenced (but not on a longer term)
*Social influences are important for UX in games
*UX depends on usage situations, time is an important
factor
*UX design must take into account different characters of
users (e.g. more/less play-oriented users)
Categories of Evaluation: User Experience
Focus GroupsQuestionnaire
Playful Probing
Experiments User Studies
InterviewsEmotionalFlowers
How to Evaluate in Non-Traditional Environments
*Choose the right set of evaluation methods based on the
context the final product will be used in
*Evaluation should take place in real usage context with
real users – but this is not realistic for several domains
(security, airtraffic, cockpits, satellite control rooms,
homes, cars, public places, …)
*Modify and develop methods to allow the evaluation in
non-traditional environments or adapt methods to reflect
the real usage environment
Evaluation in Non-Traditional Environments
*Real usage context vs. Lab/Field Study
* Investigate the real usage problems and the user
experiences
* Add valuable information for the development of products
that are usable and that are perceived with a positive user
experience
Problems for Evaluation in
Non-Traditional Environments
*Users not accessible
*Non-standardized technologies (e.g. Set-top box in living
room)
*Evaluation is typcially expensive
*Sometimes not reproducable insights – especially for UX.
Solutions - Examples
* Adaption of Methods:
· Playful Probing
· Creative cultural probing
· User Experience Sampling
· Field usability studies
*Development of new methods and learning from „related“
disciplines:
· Evaluating User Experience in Games [Bernhaupt, R., Springer
Verlag, Mid 2009]
Future Work
* Identification of factors influencing the evaluation
in non-traditional environments
*Development of new or adapted methodologies and their
validation
* Framework for comparing evaluation methods with
respect to usage in non-traditional environments
Contacts
*Regina Bernhaupt
* IRIT ruwido
*Groupe IHCS
*118, Route de Narbonne Köstendorferstr. 8
*31062 Toulouse 5321 Neumarkt a. W.
Literature
* IF you have DIFFICULTIES in finding any of the
publications I authored, please send me an e-mail to
request the article!
Literature: Playful Probing and Playful Evaluation Methods
(Bernhaupt et al.)
* PLAYFUL PROBING
* Bernhaupt, R., Weiss, A., Obrist, M., Tscheligi, M.(2007). Playful Probing: Making Probing more Fun, INTERACT 2007, 10 -14 Sep 2007, Springer LNCS, 606-619.
* Beck, E., Obrist, M., Bernhaupt, R., Tscheligi, M., (2008). Instant Card Technique: How and Why to apply in User-Centered Design. Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear.
* Weiss, A., Wurhofer. D., Bernhaupt, R., Beck, E., and Tscheligi, M. (2008) “This is a Flying Shopping Trolley”- A Case Study of Participatory Design with Children in a Shopping Context, Conference on Participatory Design (PDC) 2008, Bloomington, Indiana, USA, to appear.
* STUDIES USING PLAYFUL PROBING
* Bernhaupt, R., Obrist. M., Weiss, A., Beck, E. & Tscheligi, M. (2008). Trends in the Living Room and Beyond. Computers in Entertainment, Special Issue, Vol. 6, 1, online.
* Obrist, M., Bernhaupt, R. and Tscheligi, M. (2008). Interactive TV for the Home: An ethnographic study on users’ requirements and experiences. International Journal of Human-Computer Interaction, Volume 24, Issue 2, 174-196.
* Bernhaupt, R., Wilfinger, D., Weiss, A., and Tscheligi, M. (2008) An Ethnographic Study on Recommendations in the Living Room: Implications for Design of iTV Recommender Systems, Proc. of EuroiTV 2008, Springer LNCS 5066, 92-101.
Literature: Evaluation Methods – Introduction/Overviews
* Bernhaupt, R., Mihalic, K. and Obrist M. (2008). Methods for Usability
Evaluation of Mobile Applications. to appear in: Handbook of
Research on User Interface Design and Evaluation for Mobile
Technology, Lumsden, J. et al. (eds.), IGI Global, p. 742 - 755.
* Bernhaupt, R, Palanque, P., Winckler, M. and Navarre, D. ( 2007)
Usability Study of Multi-Modal Interfaces using Eye-Tracking, Interact
2007, 10 – 14 Sep 2007, Springer LNCS, p 412-424.
* Bernhaupt, R., Palanque, P., Winkler, M. and Navarre, D. (2007).
Supporting Usability Evaluation of Multimodal Safety Critical
Interactive Applications using Dialogue and Interaction Models. In
book: Maturing Usability: Quality in Software, Interaction and Value.
Law, E., et al. (eds.) Springer 95-127.
Literature: User Experience
* Hassenzahl, M. and Tractinsky, N. (2006) User Experience – a
research agenda, Behaviour and Infomration Technology, Vol. 25, No.
2, March-April 2006, 91-97.
* Blythe, M., Hassenzahl, M. and Wright, P. C. (2004), More Funology,
Interactions, 11, 36-77.
* Carroll, J.M. and Thomas, J. C. (1988) Fun. SIGCHI Bulletin, 19. 21-24.
* Norman, D. (2004) Emotional Design: Why we love (or hate) everyday
things (New York: Basic Books).
* Blythe, M., Overbeeke, C., Monk, A.f., and Wright, P. C. (Eds), (2003)
Funology: From Usability to Enjoyment (Dordrecht: Kluwer).
* Tractinsky, N., Katz, A.S. and Ikar, D. (2000). What is beautiful is
usable. Interacting with Computers, 13, 127-145.
Literature: User Experience - Emotion/Affect
* Desmet, P.M.A., Overbeeke, C.j. and Tax, S.J.E.T (2001) DesigningProducts with added emotional value: development and application of an approach for research through design. The Design Journal, 4. 32-47.
* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., Tscheligi, M. (2007). Emotional Flower: How Social Community Building is Influenced by an Emotional Interface Game; to AoiR 2007, online version at: http://conferences.aoir.org/papers.php?first_letter=B&cf=6.
* Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., and Tscheligi, M. 2007. Using emotion in games: emotional flowers. In Proceedings of the international Conference on Advances in Computer Entertainment Technology (Salzburg, Austria, June 13 - 15, 2007). ACE '07, vol. 203. ACM Press, New York, NY, 41-48.
Literature: Evaluating Games
*R. Bernhaupt (to appear) Evaluating User Experience in
Games: Methods and Approaches, Springer Verlag, HCI-
Series.
*Bernhaupt, R., Ijsselsteijn, W., Mueller, F., Tscheligi, M.
and Wixon, D. (2008). Evaluating User Experiences in
Games, In CHI '08 Extended Abstracts on Human Factors
in Computing Systems (Florence, Italy, April 05 - 10,
2008). CHI '08. ACM, New York, NY, 3905-3908.
*Position Papers at:
http://workshops.icts.sbg.ac.at/chi2008uxgames/
Literature: Evaluation in Non-Traditional Environments
* Bernhaupt, R. (accepted) Application of Usability and User Experience
Evaluation Methods for the Mass-Customization of Products and
Services. (Eds).Mourlas, C., & Germanakos, P. Mass Customization for
Personalized Communication Environments: Integrating Human
Factors, IGI Global Publication.
* Bernhaupt, R., Obrist, M., Tscheligi, M. (accepted) Methods for
involving users in the development of social interactive TV: Enhancing
usability and user experience in non-traditional environments, 15.
pages, Ed. Caesar, P., Geerts, D., and Chorianopoulous, K. Social
Interactive Television: Immersive Shared Experiences and
Perspectives. IGI Global Publication.