Date post: | 04-Apr-2018 |
Category: |
Documents |
Upload: | rachel-zaporowski |
View: | 221 times |
Download: | 0 times |
of 25
7/30/2019 Usability Final Report
1/25
1
Waer.org Usability Testing
Final Report
Maria Sanfeliu
Nneoma Nwaigwe
Rachel Zaporowski
7/30/2019 Usability Final Report
2/25
2
Abstract
This report outlines usability testing of waer.org. We recruited five participants and gave them
each seven identical tasks to indirectly test different aspects of the website, including design,
navigation, memorability, and efficiency. We also directly requested feedback via
questionnaires to measure the level of user satisfaction. Upon analyzing the results, we detected
both weaknesses and strengths in the current website. Participants demonstrated consistent
difficulty navigating the homepage and locating the search bar while expressing overall sub-
optimal satisfaction. We concluded that waer.org would benefit from renovation and thus
formulated recommendations in accordance with our data.
7/30/2019 Usability Final Report
3/25
3
Table of Contents
Executive Summary 4
Purpose of the Test 5
Objectives and Research Questions 5
Target and Actual Participant Profiles 6
Testing Methods 8
Data and Results 10
Discussion of Results 13
Recommendations 15
Appendices 18
7/30/2019 Usability Final Report
4/25
4
Executive Summary
This report evaluates the usability study conducted for the WAER website. The study was
conducted over a five session period in which five participants with different affiliations to the
radio station were tested. Primary participant characteristics included exposure to the site, level
of technological expertise, and familiarity with waer.org.
Each participant completed an identical set of seven tasks on waer.org and were permitted to
comment openly. Performance and subjective observations were recorded by the usability team
while studying live participant interaction as well as video recordings of participant interaction
via Silverback.
Most Significant Usability Findings Reveal:
A. Participants took advantage of computer functions rather than the website function by using
command + F to locate the end result of a task.
B. Participants generally preferred using the search bar on the bottom of the page to complete a
task as opposed to using the menu for site navigation.
C. Participants were distracted by the overwhelming clutter in design, especially the double-
sided advertisements and layout of the homepage.
D. Participants received multiple error messages when using the search bar to complete a task.
After completing the tasks, participants shared opinions of their experiences with the site by
filling out a post-test questionnaire. They mainly expressed dissatisfaction with aesthetic,
homepage design, and search bar location.
Recommendations:
Create a horizontal menu
Place the search bar at the top of page
Reduce site clutter to improve aesthetic Prioritize updates
Shorten the length of homepage
7/30/2019 Usability Final Report
5/25
5
Purpose of the Test
The team detected numerous usability issues with waer.org upon introduction to the site, ranging
from cluttered design to sub-optimal navigation. We conducted a series of usability tests to
discover the ways in which the interface made it difficult for general visitors to accomplish their
goals. Usability testing generally involves measuring how well test subjects respond in areas of
efficiency, accuracy, recall, and emotional response. Although the tests served as a means to
discover error and areas of improvement through by observing participant navigation, it also took
into account what the site executed well. Therefore, the project intended to:
Analyze data from our observations in order to gauge the level of usability the site
afforded participants: how much time and how many steps were required for people to
complete basic tasks.
Measure the accuracy of the site: how many mistakes participants made.
Assess the level of recall: how much the participants remembered after periods of non-
use.
Evaluate the level of stickiness: how much time participants spent on each task.
Ascertain the degree of user satisfaction: how the participants reacted to tasks and overall
site functions.
In the process of analysis, our final and overarching purpose was to make final recommendations
to the client based on the findings derived from the usability test.
Objectives and Research Questions
Before executing the testing sessions, we organized several objectives in order to create a solid
focal point for the data analysis phase:
Determine the level of site usability by observing participant performance of simple
tasks.
Locate navigation and design errors, which includes errors that lead to an inability tolocate task destinations, excessive use of keystrokes to complete tasks and labeling
ambiguities that prohibit the completion of tasks.
Use collected data to identify issues among different types of users and overall site
pitfalls.
Provide the client with valuable information on website improvement.
7/30/2019 Usability Final Report
6/25
6
We also formulated specific research questions, with the goal of answering them by evaluating
participant performance on our tasks and feedback on our post-test evaluations:
How does the homepage affect user performance?
How do the structure of the menu and location of the search bar affect user navigation?
What obstacles are present with the program schedule?
Do users encounter any problems or obstacles users while using/downloading the listen
live application?
How aesthetically appealing is the website to users? How important is this to them?
Target and Actual Participant Profiles
Target Participants
The original test plan was to recruit five users that would fit into one of two profiles. The first
category, labeled producers, consisted of employees and volunteers at the station. Since the
volunteers are mostly comprised of students, they represented the younger generation of the
participants and we expected them to have a different outlook on using the website. The
members of this group are more likely to be familiar with the site, having to visit it on a regular
basis for information about what is being aired, upcoming events, or just for interests.
The second group, labeled consumers, consisted of the stations listeners within its audience.
According to the NPR profile of 2009, the audience is mostly composed of adult males between
the ages of 45 and 64. The members of this group may or may not be familiar with the site butcould potentially show what a member of WAERs leisure audience is interested in and looking
for.
We hoped the employees and volunteers would represent an expert user profile given by their
amount of exposure to the site and its features, while the listeners would have varying degrees of
site aptitude. We also strived to test users that use the website for different purposes (i.e. work
or leisure) to measure efficiency in different contexts.
Producers
Employees 2
Volunteers 1
Consumers
Listeners 2
_________________________________________________
Total Number of Participants 5
7/30/2019 Usability Final Report
7/25
7
Actual Participants
At the beginning of our recruitment process we reached out to employees and volunteers via
email. Minor setbacks occurred due to inconsistent communication leading us to think for a brief
moment access to employees might not be a possibility as participants in our study. Under the
impression we would not be able to reach our target number of participants, we tested two
volunteers instead of just one like initially planned. We later got in contact with two employees
through the use of social media and a meeting with one of them. As a result we succeeded in the
recruitment of both of employees as participants.
We also reached out to listeners through Facebook and Twitter but did not receive any responses.
This lead to the lack of success in recruitment of participants in the consumer category. We
recruited a participant unaffiliated with the site. The final participant did not listen to WAER or
work for the website or station. Since our previous four participants were already familiar with
the website, we anticipated promising results that might lead to insight as to what a new memberof their audience would think.
Employees 2
Student Volunteers 2
Non-affiliated Student 1
_________________________________________________
Total Number of Participants 5
The chart on the next page documents each users distinct profile. Information was collectedduring the pre-test questionnaire at the beginning of the session.
7/30/2019 Usability Final Report
8/25
8
The compiled user profile reveals that all of our participants were familiar with computers and
the internet, and had at least some familiarity with the format of radio websites. All of them
included communication and music as their main online activities, while four of five mentioned
news and social media. Three stated the search bar as preferred method of navigating any given
website, while the other two preferred using a menu. Employees demonstrated the highest
frequency of visiting waer.org. The most relevant site features among our participants werenews and online listening.
Testing Methods
Data Collection
We set out to collect two types of data: performance and preference. Performance data was
based how much time it took a participant to perform each task, or to realize that they wereunable to complete it, by using a timer and allowing them to verbalize their thoughts. When a
participant insinuated a high level of frustration, we asked if he or she would like to move onto
the next task and made note of task failure. Preference data was to be collected through post-test
questionnaire responses, measuring user satisfaction and perception of the site.
7/30/2019 Usability Final Report
9/25
9
The pre-test questionnaire consisted of multiple choice questions about their affiliation to the
station, previous use of the website and computers, etc. The post-test questionnaire consisted of
questions about their opinions and reactions to the website. Both questionnaires are included in
Appendix D.
Environment and Technology
The waer.org usability tests took place in Huntington Beard Crouse Hall at Syracuse University.
Four of the test were conducted in the Writing Major Lounge using a MacBook Pro. The other
test was given in the Digital Editing Studio where the participant used an iMAC. All of the test
sessions were recorded using Silverback for later analysis.
Our original test plan outlined three different roles for the usability test team. The moderator
was given the responsibility of reading the script to each participant, explaining our reasons
behind the study as well as what was to be expected during the session (see Appendix A). Shewas in charge of executing the test by reading the task scenarios, including handing out the
consent form and questionnaires (see Appendicies B, C, and D). She was the sole communicator
of the testing session, giving assistance to participants if needed. The observer was given the
responsibility of making note of the participants gestures, facial expressions, and comments (see
Appendix E). She was not to intervene in the test itself. The timer used a stopwatch to record
how long each task took for the participants, and also had no involvement in the actual test.
During two of the tests there were two observers in the room, one took notes of the participants
reactions while the other timed the tasks. There was only one observer doing both of these
during the last three tests.
Tasks
The actual test included seven tasks which were read to the participants by the moderator. These
tasks were created to address the different potential problems we identified during our initial
analysis of the website. All participants were given the same set of task scenarios, read aloud by
the moderator. We allowed the testers freedom to progress at their own pace as if they were
using the site in their natural environment. We also offered them the opportunity to ask
questions and express their inability to complete a task if needed. Tasks are listed in Appendix
B.
7/30/2019 Usability Final Report
10/25
10
Data and Results
Performance Data
Quantitative data was collected by timing each participant for each of the seven tasks. This was
intended to measure the efficiency of different aspects the website, as well as identify any areas
of weakness among a specific user profile.
The following spreadsheet documents participant times under the notation
hours:minutes:seconds. Values highlighted in red indicate the time elapsed before a task was
failed*. The subsequent graph illustrates the same data.
*For User 1, Task 3: upon our analysis, we noticed the task was not completed in its entirety.
This graph represents average time needed to either complete or give up on each task:
7/30/2019 Usability Final Report
11/25
11
Trends
Task 1, locating a link on the homepage, posed the most difficulty. Two out of five
participants failed.
Task 3, locating and using the search bar, appears to have posed the second highest
amount of all-around difficulty.
Task 5, locating and using the listen live feature, appears to have the lowest average time
among participants. It was thus the most usable of the site functions.
A general trend of decreasing times with the progression of each test session indicates thesites high level of memorability.
Preference Data
We used the post-test questionnaires as an opportunity to collect user opinions and thus gauge
the level of user satisfaction. They were asked to rate different aspects of the site on a ten-point
scale, with ten representing optimal feedback and one representing poorest level of feedback.
7/30/2019 Usability Final Report
12/25
12
We also asked each participant what he or she would improve about the website. The following
graph represents the frequency each aspect was noted. All five found the homepage
troublesome, while the menu, search bar, and aesthetic were problematic for three different
participants.
7/30/2019 Usability Final Report
13/25
13
Discussion of Results
After careful observation, analysis of the data, and consideration of participant feedback, we
have discovered numerous usability issues with the website. Although the test was mainly
intended to identify problematic features, the study uncovered several strengths as well. Our
findings are presented in accordance with the research questions mentioned earlier.
Q: How does the busy homepage affect user performance?
The design of the homepage proved to be overwhelming to users, confirmed by both
performance and preference data. Its vertical extension requires users to spend much of their
time on the site scrolling up and down. The layout of the links caused users to strain as they
read, searching for the concert link. Advertisements along both sides of the main page were
distracting.
Q: How do the structure of the menu and location of the search bar affect user navigation?
The menu revealed minimal problems among participants, who were all able to navigate the site
with ease using this portal. Upon observing users, it was easily visible and comprehensive. The
location of the search bar at the bottom of the homepage made it difficult to find. As a result, the
excessive scrolling made its use unnecessarily time-consuming.
Q: What obstacles are present with the program schedule?
Performance data on our fourth task scenario, which entailed utilizing the program schedule,
revealed a comparatively low average time for completion. Participants were able to find its link
on the menu and scroll down to the specified program without much hesitation.
Q: Do users encounter any problems or obstacles users while using/downloading the listen
live application?
As this task scenario recorded the lowest average time, user performance revealed this to be the
most usable feature of the site. Users were able to access listen live in more than one way; via
the menu, homepage, and on the side as an advertisement. Multiple means of accessing thisoption made it most efficient.
Q: How aesthetically appealing is the website to users? How important is this to them?
7/30/2019 Usability Final Report
14/25
14
Preference data revealed that site aesthetic was an issues of concern for most participants. Given
the amount of difficulty with the cluttered homepage, an issue based heavily on aesthetic, we
conclude that site appearance is a significant aspect to a website for a majority of users.
We uncovered some strengths of the website, listed as follows:
High level of memorability: on average, users quickly learned their way around the
website
Menu proved to be accessible: users revealed minimal problems navigating with this
portal
Good visibility of several important site features, including listen live and program
schedule
Unintended findings
The Usability Team also discovered and decided to make note of other issues with the site for the
executives convenience:
No events present on the event calendar: when asked to find the link to a concert that
already occurred, a participant went to the calendar and found no events on the calendar,
prolonging her recorded task time.
Search bar results lacked consistency and produced error messages: 2 out of 5
participants received error messages multiple times when they used the search bar to
attempt to find the date of a past concert and the latest final score of the lacrosse game.
Final scores for lacrosse game were often not updated: on separate occasions, 2 out of 5test participants could not complete a task asking to find the final score of the latest
lacrosse game because the latest scores and dates were not updated. The last update was
May 3 while the latest game took place on May 5th.
Participants opted for search shortcuts independent of the site: the first participant made
use of the Mac search function (command + F) rather than the sites search bar after she
discovered an empty calendar of events. She only used site functions to actually
complete a task as a last resort for one task. Also, when asked specifically to use the
search bar to find a task, she found the search bar but did not actually make use of it.
Inconsistency with clickability: when asked to find a news program, a participant clicked
on the news tab and nothing happened. Though the news tab expands into subsections,
other main tabs take participants to their main page as a result of one click.
7/30/2019 Usability Final Report
15/25
15
Recommendations
As a conclusion to our study, the usability team offers our recommendations based previously
presented performance and preference data.
Pursue standard website design: WAERs site design should not abstract too far from
standard site designs. The WAER menu is located at the left side of the page and placed
vertically, blending in with the grey page background. A majority of high-traffic
websites, such as Google, have menus placed horizontally on the top of the page. We
recommend keeping your sites navigation functions close to standard in order to
maximize visibility and enable user confidence.
7/30/2019 Usability Final Report
16/25
16
Relocate the search bar: test results suggest that even employees of the site could not
navigate to the search bar because of its location at the bottom of the page and lack of
proximity to the menu. We recommend that you move the search bar to the top of the
page or closer to the menu.
Reduce site clutter: every extra unit of information in a dialogue competes with therelevant units of information and diminishes their relative visibility. We recommend the
removal of excess textual information. For example, the boxes of links on the homepage
appears with a lengthy description. The average internet user tends to dislike reading
more than necessary, and as a result, may overlook a link they are searching for.
Prioritizing information or grouping information under broader heading should decrease
clutter.
7/30/2019 Usability Final Report
17/25
17
Be mindful of updates: we discovered various tasks could not be completed successfully
due to a lack of site updates. For example, the events calendar was empty and scores for
the most recent lacrosse game was not updated.
General improvement of site aesthetic: preference data revealed that site aesthetic was an
issue of concern for most participants. They expressed frustration with the disorder and
the team also took notice of the extreme length of the homepage. The homepage
contained news-related sections (i.e. NPR Hourly News, Daily News Program) despite
the existence of a news tab on the menu. The same issue was noted for the program
schedule. Also, some tabs on the menu expand into subsections that take the user to a
new page, increasing the potential for error. Compiling all of these options into one page
under one tab will give the users enough options for free navigation without
overwhelming their concept of information volume. We recommend that you reduce the
length of the homepage, removing redundant information, and uniting the menu sub-
options into one tab.
7/30/2019 Usability Final Report
18/25
18
Appendix A - Test Script
Before beginning each test, the following script was presented verbatim to the participant:
Hello, my name is Maria. Im a student here at SU. As the final project for our writing class weare conducting a usability test on the WAER website in order to ascertain what works and what
does not work. The purpose of this test will be to provide the client with valuable information on
how to improve the site. You are one of several participants we will be observing. The test will
consist of a few tasks that we have written down for you. We will need you to let us know once
you have completed each task and feel free to also think aloud as you perform these tasks and
articulate any of your responses to the site. We will not be able to assist you in any of the tasks.
If at any point you feel like you cannot complete a task, let me know and well move on to the
next one.
I will be in the room during the test, explaining the different tasks you will perform. Nneoma
and Rachel will also be in the room observing and taking notes of the test session.
Please make yourself comfortable and take your time. Do not hesitate to share your true feelings
and opinions. We did not have anything to do with the design of the website.
Do you have any questions before we begin?
7/30/2019 Usability Final Report
19/25
19
Appendix B - Task List
How long does it take to find the link to La Vida Boheme, Live In Concert: SXSW 2012 on
the home page?
You wanted to attend the La Vida Boheme, Live In Concert: SXSW 2012 last weekend but
missed it and you want to find out what happened during the concert. Search the home page in
order to find the link for this information.
How long does it take to find the SU Basketball tab on the menu?
You want to find out the score of the SU Basketball teams last game and when their next game
is in the NCAA Tournament. Find this information using the menu.
How long does it take to find the search?
Now you want to find that same information, this time using the search bar.
How long does it take to find a program on the schedule?
You want to catch the latest news as you drive home from work. What time is the All Things
Considered program?
How easy is it to use listen live?
You are stuck in the office and cannot listen to the news in your car like you usually do. Can
you find where to listen to the show on the site?
Can you go back and find the ______ page again? How long does it take?
You forgot the time of the SU Basketball teams next game in the tournament. Go back and find
the time.
Can you access WAERs Twitter page, without leaving the website?
You want to follow WAER on Twitter so you can access their updates from your phone. Find
the WAER Twitter page.
7/30/2019 Usability Final Report
20/25
20
Appendix C - Consent Form
WAER Usability Test- Participant Agreement & Consent Form
Thank you for participating in our usability test of WAER.org. We will be recording the test and
tracking your performance. Only the test moderator and observers will be viewing these
recordings for the sole purpose of collecting data. We will not give out your name or any other
personal information that you give us during the surveys we give you. We will only use your
test results and questionnaire answers for this project.
If you agree to these terms, please read the following information and sign:
I understand that I will be recorded for research purposes during the test. I grant permission to
use these recording for the purpose mentioned above.
Signature: _____________________________________ Date: __________________
7/30/2019 Usability Final Report
21/25
21
Appendix D - Questionnaires
Pre-test Questionnaire
1. How often do you use a computer?
__ Daily
___Often
___Sometimes
___Rarely
___Never
2. How often do you use the Internet?
__ Daily
___Often
___Sometimes
___Rarely
___Never
3. What online activities do you partake in most often? Check all that apply.
__Business
__Communication
__Gaming
__Movies
__Music
__News
__Shopping
__Social media
__Sports
__Other (please specify)
4. Are you an employee or volunteer of WAER?
__Employee
__Volunteer
7/30/2019 Usability Final Report
22/25
22
__Neither
5. How often do you listen to WAER?
__Daily
__Often
__Sometimes
__Rarely
__Never
6. Why do you listen to WAER? Check all that apply.
__Employment reasons
__Leisure__I don't listen to WAER
__Other (please specify)
7. How often do you visit radio websites?
__Daily
__Often
__Sometimes
__Rarely
__Never
8. How often do you visit waer.org?
__Daily
__Often
__Sometimes
__Rarely
__Never
9. For what reason(s) have you visited waer.org? Check all that apply.
__Events
__Local Interest
7/30/2019 Usability Final Report
23/25
23
__News
__Online listening
__I have never visited
__Other (please specify)
10. What features of this website do you use most frequently? Check all that
apply.
__Events
__Local Interest
__News
__Online listening
__I have never visited
__Other (please specify)
11. Do you prefer to use a search bar or a menu?
___Search bar
___Menu
Post-test questionnaire
1 On a scale of 1 to 10, with 1 being poor and 10 being excellent, what was your opinion of
this site overall?
2 On a scale of 1 to 10, with 1 being uncomfortable and 10 being comfortable, how at ease
did you feel about navigating this site?
3 On a scale of 1 to 10, with 1 being inefficient and 10 being efficient, how would you rank
this site?
4 What do you think WAER could do to improve its website?
5 Do you have any other thoughts on WAER.org?
7/30/2019 Usability Final Report
24/25
24
Appendix E - Session Observations
Session 1
-For task 1: went to the calendar and there were no events listed on the calendar.
-She looks confused by the side bar, she furrowed her brows which is a sign of confusion or
hesitation.
-Expresses the fact the site is overwhelming
-Checked the band directory and the calendar
-She took advantage of computer function rather than the website function (command + F)
-Used the find function again, still on the first task.
-I really dont know where it is
-Second task - success
-Third task - she never completed the task, she just found the search bar
-She started using the search bar on the bottom to find task 4 and she found it pretty easily -She used an ad to complete task 5 not the actual menu
-Task 6 is successful
-Task 7 is succesful
Session 2
-Task 1 - successful, a little bit of distraction by the page itself but overall it was successful
-Task 2 - successful
-Task 3 - successful
-Task 4- there is no news homepage, the tab was not clickable
-Task 5 - successful
-Task 6 - successful
-Task 7 - successful
Session 3
-Task 3 -is there a search bar, I didnt know there was one- and hes an employee
-He used the search bar and used last years results and then he used the search bar again and got
an error message (Not Found page)
-so the search bar is a fail - It took him a minute and 15 seconds before he gave up.
-Task 6 The score was not updated
Session 4
-hes not very mac savvy
-Task 1 - he couldnt find it on the homepage, hes using the search bar
-he had to bounce around, theres a lot going on with the page
-Task 2 - the scores for the lacrosse games were not updated
7/30/2019 Usability Final Report
25/25
25
-Task 3 - memorability, took a while to find the search bar even though he used it to find the first
task
-Task 4 - hes the only one that used the program schedule on the homepage to find it, everyone
else used the menu or the search bar
-Task 5 - He found the listen live task on the home page,
-Task 6 - He had some trouble using the search menu to complete task 6
-Task 7 - Successful
Session 5
-Problem with the mac, doesnt really know how to scroll,
-He used the search bar, he couldnt find it on the home page, when he tried to find it on the
search bar, he got error messages, 409 page not found but it gave him options, the third time he
tried, the results came up but he didnt see it because it was presented as an ad by google.
-He got an error message multiple times using the search bar.
-Task 2 - successful, happy that he found it-The rest of the tasks were successful.
-this is a terrible website