Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | belen-barros-pena |
View: | 110 times |
Download: | 0 times |
Bernard, packet core engineer at NSN
Belén, interaction designer at Intel’s OTC
how to plan and run usability tests with mobile devices in a laboratory setting
assuming ...
1. you have experience planning and running usability tests with desktop computers
2. you are testing software or an interactive prototype running on a mobile device
3. you are testing in a usability laboratory or similar environment
we will not focus on
1. the fundamentals of the usability testing technique
2. how to prototype for mobile devices
3. how to test in the field
these are the golden rules you need to remember
let us begin ...
usability testinga process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
usability testing
watching people try to use what you’re creating/designing/building (...) with the intention of (a)making it easier for people to use or (b)proving that it is easy to use.
Rocket Surgery Made Easy, S. Krug
dut vs mut
Product Desktop software
Mobilesoftware
Device Computer Mobile phone
Context Predictable Unpredictable
Connection DSL / Cable Mobile phone network
which phone?
which context?
which connection?
which phone?
which context?
which connection?
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html
handset usability affects test results
test with participants’ own phones
when not possible, include training and
warm-up tasks
handset becomes a screening criteria when recruiting test subjects
http://goo.gl/hmp8t
mobile phone subscriptions per 100 people
http://data.worldbank.org
http://gs.statcounter.com/
http://gs.statcounter.com/
WP iOS Android Blackberry
http://developer.android.com/resources/dashboard/platform-versions.html
http://us.blackberry.com/developers/choosingtargetos.jsp
do not confuse target platform with target user
which phone would you use for testing?
who are the target users?
handset screening criteria
will you need warm-up tasks? (yes / no)
if yes, which ones?
BNT are considering a mobile version of their website.
Before investing in the project, they would like to know how their existing desktop site performs on mobile devices.
You are working on an Android application for Sofia Airport.
Together with your developers, you have created a first version of the app. This ‘alpha’ version is not available in Google Play, it is not fully tested and might be buggy.
St. Ekaterina University Hospital has developed a Blackberry application that helps nurses and doctors access and update patients records during their daily shift.
They have asked you to help them test the application.
Evernote wants to uncover the main usability issues in their existing mobile applications for iOS, Android, Windows Phone and Blackberry.
Help them organise the necessary testing.
what about emulators?
mas_fsx_ by Massimiliano Silipigni at http://www.flickr.com/photos/silipigni/6055803785/
good for ...
using an emulator enables testers to thoroughly capture user behaviour (...) The captured data are generally informative and useful for analysing user performance
Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
good for ...
emulators are more suitable to be used for improving the interface design of applications such as the layout of menu structures during the development process.
Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
but bad for ...
using emulators alleviates the problems of latency, inefficient input mechanisms, and the changing wireless environment, potentially leading to untruthful user satisfaction
Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)
which phone?
which context?
which connection?
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
field vs. lab
0 0
The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
0 1Field Lab
according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
0 2Field Lab
evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
1 2Field Lab
The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006
The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006
2 2Field Lab
field vs. lab
EXPERTS DISAGREE
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
... but they all agree
testing in the field requires double the time in comparison to the laboratoryUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
... but they all agree
evaluations in the field (are) more complex and time-consumingIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
... but they all agree
field-based usability studies are not easy to conduct. They are time consuming and the added value is questionable.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004
testing in the lab is better than no testing
can you think of an example of mobile software that you must test in the field?
do it late
plan an run pilot tests
be prepared (like the Scouts)
if doing field testing ...
have I chosen an appropriate location?
which external factors could impact results?
is the equipment working as planned?
do I have enough time to run the tests?
does the protocol work? Is it well understood?
if doing field testing ...
If we cannot go to the field, the field must come to us
Simulations
Scenarios
Take the lab out
Bringing the field into the lab
simulations
Use of techniques to “reflect or recreate a mobile use situation” (...) their level of sophistication is measured by the extent to which the environmental and ergonomic factors of the actual use context are recreated.
Emerging Research Methods for Understanding Mobile Technology Use, P. Hagen, T. Robertson, M. Kan and K. Sadler, Proceedings of OZCHI 2005, Canberra (Australia)
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M.B. Skov, B.S. Als and R.T.Høegh, MobileHCI 2004
Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting, E.T. Beck, M.K. Christiansen, J. Kjedskov, N. Kolbe and J. Stage, OZCHI 2003, Brisbane (Australia)
scenarios
First you choose the tasks to test (...) Then you expand these tasks into scenarios (...) A scenario provides some context and supplies information the user needs to know
Rocket Surgery Made Easy, S. Krug
a task:Find a hotel
a desktop scenario:You are planning a surprise weekend trip to Paris. You’ll be leaving on Friday and coming back on Sunday evening. Your partner’s favourite Paris area is Le Marais. Find a place to stay he/she will like.
a task:Find a hotel
a mobile scenario:You were late to Orly airport and have missed your flight home. There are no more flights from Paris today, so you’ll have to spend the night. Since it’s already 10 pm, use your phone to look for a place to stay close to the airport.
portable labs can be deployed anywhere
document camera
laptop
webcam
Jyske Bank in Roskilde by king_david_uk at http://www.flickr.com/photos/kingdavid/2926673466/
Vodafone, Croydon, London CR0 by Kate Pugh at http://www.flickr.com/photos/kake_pugh/6545344321/
write a mobile scenario for your
group’s task
You are testing:
http://www.tfl.gov.uk
Task:
Find a public transport route
A bit of context:
From Heathrow Terminal 5 to the Olympic Stadium. User must arrive on time for the Olympics opening ceremony (27th July at 7:30pm)
You are testing:
http://www.toptable.co.uk
Task:
Find a restaurant
A bit of context:
Lebanese cuisine Around Shoreditch (an East London area)For tomorrow at 8:30pm For 5 people
You are testing:
An Android app called ‘CycleStreets’, which helps you find cycling routes across London
Task:
Calculate a route distance
A bit of context:
From King’s Cross station to Tate ModernRoute with least traffic
You are testing:
A task management app called Remember the Milk
Task:
Create a task
A bit of context:
The task is to buy flowers for a friend (make up a name)Next Tuesday morningIn her favourite flower shop, called Blooming
which phone?
which context?
which connection?
how often are mobile users connecting their devices to WiFi networks instead of mobile networks?
Towering by Rachel Strohm at http://www.flickr.com/photos/rachelstrohm/4693016049/
25% of households worldwide have WiFi networks set up
Globally, smartphones account for 36% of all public WiFi hotspot connections, compared to 48% for laptops and 10% for tablets
Out of 4 billion unique mobile phone users, 1 billion uses WiFi to access the Internet (there are only about 900 million smartphones)
http://www.lukew.com/ff/entry.asp?1537
http://communities-dominate.blogs.com/brands/2012/05/mobile-services-beyond-messaging-excellent-tns-global-survey-reveals-tons.html
http://databank.worldbank.org
do not test over WiFi
cover your participants’ data costs
and on the big day ...
remote?
Remote Mania by Dramagirl at http://www.flickr.com/photos/dramagirl/2268235917/
http://www.usertesting.com/mobile
but ...I’d say remote testing gives you about 80% of the benefits of a live test with about 70% of the effort. You do lose 20% of something not being in the same room as the participant. The in-person experience is just richer, somehow.
Rocket Surgery Made Easy, S. Krug
To be or not to be, that is the question by Mister_Jack at http://www.flickr.com/photos/machineman22/5130901135/
to record or not to record ...
it is difficult ...We have used Morae in the past also for website testing, but could not easily use the system for mobile (...) Obviously it’s difficult to video something like that so we just sit next to them and observe.
http://belenpena.posterous.com/user-research-for-mobile-software-a-short-sur
... but video is both a
memory aid
powerful communication tool
reactions
actions
actions
reactions
which phone?
which context?
which connection?
which recording setup?
Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#
document camera records mobile screen
USB webcam records participant’s face
both cameras connect to a laptop running screen
recording software
document camera range is marked with tape
Mobile Device Camera by Noldus http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#
Ovo Studios device camerahttp://www.ovostudios.com/devicecamera.asp
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing
Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
are blooming!!
screen audio
Skype Skype
Screen sharingTelephone conferencing system
Mirroring
Telephone conferencing system
the observation room
Screen sharing
Mirroring
Telephone conferencing system
pros cons
provides high quality screen recording
participants won’t appreciate you installing stuff on their phones
provides high quality screen recording no application will
support all platforms
provides high quality screen recording
in touchscreen phones, finger interactions are not captured
screen capture applications
Ovo Studios screen capture application for iOShttp://www.ovostudios.com/
Obviously,
think aloud is critical because I cannot see how the participant is interacting
with his fingers on the touch screen.
I can see using the mirroring in a lab setting to get the signal from the iPad
to the observation room though you still won’t see the physical interaction
with the device, like you would with a device mounted camera (e.g. Noldus).
... and that is a big deal
Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011 - http://www.remoteresear.ch/airplay/
Recently, we’ve questioned the value of capturing the device screen. It does
after all end up being a video of the screen changing but with no sense
of the participant interacting with it.
iPad usability testing - our equipment - http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/
... and that is a big deal
pros cons
good recording quality document cameras bring up the price
easy to set up participants must keep within the camera range
phone must lay on a desk or be hold at a flat angle
document cameras
pros cons
allow natural interaction with the phone
ready-made devices will bring up the price
allow natural interaction with the phone
if bulky, they can prevent single-hand use
allow natural interaction with the phone
if heavy, they can become tiring during long tests
mounted devices
the ideal recording setup
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
the ideal recording setup screen capture
applications
document cameras
mounted devices
ready-made
mounted devices DIY
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
no recording setup is perfect: just use the best tool for the job
?
test with participants’ own phones
do not confuse target platform with target user
testing in the lab is better than no testing
do not test over WiFi
cover your participant’s data costs
no recording setup is perfect: just use the best tool for the job
thanks!!!
www.DIYMobileUsabilityTesting.net