CITERS2003 Presentation 5 July 2003CITERS2003 Presentation 5 July 2003 11
A Usability Study of a A Usability Study of a Language Centre Web Site Language Centre Web Site
By Andy MorrallBy Andy Morrall
22
Why, in general?Why, in general?
Teachers write educational web pagesTeachers write educational web pages
Students use them outside the Students use them outside the classroomclassroom
Teachers don’t always see them being Teachers don’t always see them being usedused
Hard to see if the pages work properlyHard to see if the pages work properly
Students rarely give detailed feedbackStudents rarely give detailed feedback
33
Why, specifically?Why, specifically?
I am in charge of an educational I am in charge of an educational websitewebsite
Very little feedback from usersVery little feedback from users
Need to write reports about quality Need to write reports about quality controlcontrol
44
How?How?Need a tried-and-tested, reliable and valid Need a tried-and-tested, reliable and valid way to test the siteway to test the site
Can’t test if it is good for learning, as there Can’t test if it is good for learning, as there are too many other ways users could be are too many other ways users could be learning Englishlearning English
Users opinions of the user-friendliness of Users opinions of the user-friendliness of the site may be too nice and non-specificthe site may be too nice and non-specific
So, a need to watch students using the So, a need to watch students using the site and analyse the results systematicallysite and analyse the results systematically
55
What is Usability?What is Usability?Rubin (1994, p.22) says that usability testing Rubin (1994, p.22) says that usability testing “employs techniques to collect empirical “employs techniques to collect empirical data while observing representative end data while observing representative end users using the product to perform users using the product to perform representative tasks”. He describes an representative tasks”. He describes an approach to usability testing, which approach to usability testing, which “employs an iterative cycle of tests intended “employs an iterative cycle of tests intended to expose usability deficiencies and to expose usability deficiencies and gradually shape or mould the product in gradually shape or mould the product in question.”question.”
66
What?What?
77
Research QuestionsResearch Questions1.1. Do the improvements to the site made Do the improvements to the site made
during the usability study result in during the usability study result in improvements in test participants’ improvements in test participants’ success rate in completing tasks on the success rate in completing tasks on the CILL site? CILL site?
2.2. Do the improvements to the site made Do the improvements to the site made during the usability study result in during the usability study result in participants being able to carry out these participants being able to carry out these tasks in a shorter time? tasks in a shorter time?
3.3. Do the participants say that they believe Do the participants say that they believe that the site is easier to use?that the site is easier to use?
88
Who and what?Who and what?To learn if the site is usable, I needed to know To learn if the site is usable, I needed to know what it was used for.what it was used for.I asked 5,100 CILL members to take part in the I asked 5,100 CILL members to take part in the survey by e-mailsurvey by e-mail215 replied filled in the questionnaire at 215 replied filled in the questionnaire at http://elc.polyu.edu.hk/CILL/research/cill_site_imprhttp://elc.polyu.edu.hk/CILL/research/cill_site_improvement_questionnaire.htmovement_questionnaire.htm
The survey asked users to:The survey asked users to:– Identify the right siteIdentify the right site– Say how long they had been using the InternetSay how long they had been using the Internet– When they last used the CILL siteWhen they last used the CILL site– What did they use the site forWhat did they use the site for– If they had any suggestions for improving the siteIf they had any suggestions for improving the site
99
Survey ResultsSurvey Results78 users correctly identified the right site78 users correctly identified the right site
Analysis of Responses to Question 7 of the Initial Questionnaire
Grammar 22 Pronunciation 12 Listening 9
Vocabulary 9 Exercises 7 Dictionary 6
Reference Machine 4 Job Application Letter
4 Reading 3
Writing 3 Oral 2 Referencing 2
Idioms 2 Newspapers 2 Academic Writing 2
English in the Workplace
2 Reports 2 Interviews 2
Finding Materials 2 Testing Level 2 CILL Information 2
Research 1 Purdue On-line Writing Lab.
1 CILL teachers’ e-mail 1
Example Assignment 1 Staff information 1 Opening Hours 1
Games 1 Resumes/CV 1 Discussion 1
Exit Test 1 Memos 1 Speaking 1
1010
TasksTasks1.1. Find a grammar exercise on the difference Find a grammar exercise on the difference
between ‘staff’ and ‘staffs’.between ‘staff’ and ‘staffs’.2.2. Find out how to pronounce ‘thorough’.Find out how to pronounce ‘thorough’.3.3. Find a page where you can learn vocabulary Find a page where you can learn vocabulary
about the news.about the news.4.4. Find a page where you can learn English by Find a page where you can learn English by
listening to pop music.listening to pop music.5.5. Find a page with some common job interview Find a page with some common job interview
questions and advice and an exercise about how questions and advice and an exercise about how to answer them.to answer them.
6.6. Look up the meaning of the word ‘discharge’.Look up the meaning of the word ‘discharge’.7.7. Make a reference (Details provided)Make a reference (Details provided)8.8. Find a page which helps you to write a job Find a page which helps you to write a job
application letter.application letter.9.9. Find a page where you can read about e-mail Find a page where you can read about e-mail
pen-pals you can write to.pen-pals you can write to.10.10. Find a page that helps you improve your Find a page that helps you improve your
newspaper reading skills.newspaper reading skills.
1111
ParticipantsParticipants2 groups:2 groups:– Re-test group – did the test tasks both before Re-test group – did the test tasks both before
and after the changesand after the changes– New group – only did the tests after the New group – only did the tests after the
changeschanges
Both groups included:Both groups included:– CILL students and teachersCILL students and teachers– native speakers of English and non-nativesnative speakers of English and non-natives– participants with varying degrees of participants with varying degrees of
computer experience and skillscomputer experience and skills
1212
How to do a usability testHow to do a usability test1.1. Welcome the participant and check their identity and Welcome the participant and check their identity and
profileprofile2.2. Explain the purpose and duration of the test (1 hour in Explain the purpose and duration of the test (1 hour in
total, including post-test questionnaire).total, including post-test questionnaire).3.3. Elicit any questions.Elicit any questions.4.4. Ask for consent for the participant to take part.Ask for consent for the participant to take part.5.5. Ask the participant to sign the consent form. Ask the participant to sign the consent form. 6.6. Participant chooses the browser and version they usually Participant chooses the browser and version they usually
use to access the CILL web site.use to access the CILL web site.7.7. The participant attempts to carry out the tasks, with the The participant attempts to carry out the tasks, with the
tester watching and taking notes.tester watching and taking notes.8.8. The participant either completes the final task or runs out The participant either completes the final task or runs out
of time.of time.9.9. Give the participant a short break.Give the participant a short break.10.10. Ask the participant to complete a post-test questionnaire.Ask the participant to complete a post-test questionnaire.11.11. Chat about the test and the siteChat about the test and the site12.12. Thank the participant for taking part.Thank the participant for taking part.13.13. Participant departs.Participant departs.
1313
Task 1Task 1
1.1. Find a grammar exercise on the Find a grammar exercise on the difference between ‘staff’ and difference between ‘staff’ and ‘staffs’.‘staffs’.
Easy task first to give users Easy task first to give users confidence.confidence.
1414
Task 1Task 1
1515
Task 1 ChangedTask 1 Changed
1616
Results of Task 1Results of Task 1Question 1 Statistics
0
50
100
150
200
250
First Second (Re-test)
Second(New)
Test
Sec
on
ds
Mean
Minimum
Maximum
StandardDeviation
1717
Summary of ImprovementsSummary of Improvements Task
NumberTask Success Mean Task Duration
Improvement or remained at maximum
Improvement in percent
Improvement or remained the same
Improvement in seconds
Re-test group
New group
Re-test group
New group
Re-test group
New group
Re-test group New group
1. 20% 20% 52 15
2. 0% 10% 26 10
3. 0% 0% 71 76
4. 0% 10% 37 12
5. 40% -5% 118 111
6. 0% 0% 0 21
7. 0% 5% 139 -12
8. 0% 0% 3 3
9. -20% 0% 11 24
10. 0% 10% 42 40
Total / Mean
Total = 9
Total =9
Mean =
4%
Mean =
5%
Total =10
Total =9
Mean = 50 Mean =30
1818
Results – Research Question 1Results – Research Question 1
1.1. Do the improvements to the site Do the improvements to the site made during the usability study made during the usability study result in improvements in test result in improvements in test participants’ success rate in participants’ success rate in completing tasks on the CILL site? completing tasks on the CILL site?
Re-test Group:Re-test Group: +4%+4% New Group: New Group: +5%+5%
1919
Results – Research Question 2Results – Research Question 2
2. Do the improvements to the site 2. Do the improvements to the site made during the usability study made during the usability study result in participants being able to result in participants being able to carry out these tasks in a shorter carry out these tasks in a shorter time? time?
Re-test Group: 50 seconds fasterRe-test Group: 50 seconds fasterNew Group: New Group: 30 seconds faster 30 seconds faster
2020
Results – Research Question 3Results – Research Question 3
3. Do the participants say that they 3. Do the participants say that they believe that the site is easier to believe that the site is easier to use?use?
Re-test group only:Re-test group only:Strongly agree: 1Strongly agree: 1Agree: 1Agree: 1Neither agree nor disagree: 3Neither agree nor disagree: 3
2121
Limitations of the Study Limitations of the Study The Small Number of Tasks in Relation to The Small Number of Tasks in Relation to the Size of the Site (10 tasks, 234 pages)the Size of the Site (10 tasks, 234 pages)The Small Number of Design Iterations The Small Number of Design Iterations (I did 3, iterative design principles (I did 3, iterative design principles recommend repeated iterations)recommend repeated iterations)Debriefing Participants – it’s embarrassing Debriefing Participants – it’s embarrassing to talk about why they didn’t do or see to talk about why they didn’t do or see things - the worse the performance of a things - the worse the performance of a participant is, the more valuable their participant is, the more valuable their debriefing feedback and further test debriefing feedback and further test participation is, but the more difficult they participation is, but the more difficult they are to get.are to get.
2222
Recommendations for Further Recommendations for Further Study Study
Further study could be done into:Further study could be done into:using more authentic tasks such as a text using more authentic tasks such as a text correction task, with the answers on the sitecorrection task, with the answers on the siteinvestigating the effects of testers and investigating the effects of testers and participants reading the task instructions aloud, participants reading the task instructions aloud, because some participants didn’t read all the because some participants didn’t read all the instructionsinstructionsthe effects of participants’ levels of computing the effects of participants’ levels of computing skills on their level of task success and task skills on their level of task success and task duration duration the optimum number of tests to run between site the optimum number of tests to run between site design iterations design iterations
2323
Usability StudiesUsability StudiesDiscount Usability Testing Discount Usability Testing Nielsen, J. (2000). Nielsen, J. (2000). Designing web usability: The practiceDesigning web usability: The practice
of usability. of usability. Indianapolis: New Riders.Indianapolis: New Riders.
Nielsen, J. (1994). Guerilla HCI: Using discount usability Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In engineering to penetrate the intimidation barrier. In Bias, R.G. & Mayhew, D. (Eds.) Bias, R.G. & Mayhew, D. (Eds.) Cost-justifying Cost-justifying usability. usability. Boston: Academic Press. Boston: Academic Press.
Nielsen, J. (1993). Nielsen, J. (1993). Usability engineeringUsability engineering. Boston: AP . Boston: AP Professional. Professional.
http://www.useit.comhttp://www.useit.com
Rubin, J. (1994). Rubin, J. (1994). Handbook of Usability TestingHandbook of Usability Testing. New . New York, NY. John Wiley & Sons. York, NY. John Wiley & Sons.