Post on 08-Apr-2018
transcript
8/7/2019 IPS Usability Testing Report - Report
1/56
USABILITY TESTINGREPORT
INDIANAPOLIS PUBLIC SCHOOLS
WWW.IPS.K12.IN.US
I N F O I 5 4 3 U S A B I L I T Y A N D E V A L U A T I V E M E T H O D S
I N D I A N A U N I V E R S I T Y S C H O O L O F I N F O R M A T I C S
F A L L 2 0 1 0
D R . D A V I D E B O L C H I N I
I N S P E C T E D B Y T E A M 2 :
S T E V E N E N T E Z A R I
H A I D A N H U A N G
J A Y W H E E L E R
8/7/2019 IPS Usability Testing Report - Report
2/56
2
Table of Contents
I. Executive Summary ................................................................................................. 4
II. Instruments and Methods ...................................................................................... 7Participants .................................................................................................................................................... 7Data Collection............................................................................................................................................. 7Environment & Timeframe....................................................................................................................... 8Tools................................................................................................................................................................ 9Tasks................................................................................................................................................................ 9
III. Results ................................................................................................................ 11Task 1 School Board Meetings .......................................................................................................... 15Task 2 Finding the Technology Plan ................................................................................................ 20Task 3 College Resources .................................................................................................................... 24Task 4 Find the Closest School ......................................................................................................... 27
Task 5 Magnet School Application ................................................................................................... 31Task 6 Payroll Contact Information (Teachers Only).................................................................. 34Task 7 Teacher Amendment (Teachers Only) ............................................................................... 34
IV. Synthesis of Results from Inspection ................................................................... 36Content........................................................................................................................................................ 36Information Architecture........................................................................................................................ 37Navigation................................................................................................................................................... 40Presentation................................................................................................................................................ 41
V. Overall Recommendations for Improvement ........................................................ 43Content........................................................................................................................................................ 43
Navigation................................................................................................................................................... 44Information Architecture........................................................................................................................ 45Presentation................................................................................................................................................ 46
VI. Appendixes ......................................................................................................... 48Appendix A: Pre-test Questionnaire .................................................................................................... 48Appendix B: Usability Test Script......................................................................................................... 50Appendix C: Post-Task Questionnaire ................................................................................................ 54Appendix D: Post-Test Questionnaire................................................................................................ 55Appendix E: Usability Testing Videos................................................................................................. 56
8/7/2019 IPS Usability Testing Report - Report
3/56
3
Table of Figures
Figure 1 - Average times it took each participant to finish each task. .......................................... 11
Figure 2 - Full-Success/Partial-Success/Give-Up rates for each participant per task.......... .... 12Figure 3 - Summary of quantitative data showing average time on task, lostness, success rates
and ease of use for each task......................................................................................................... 13Figure 4- Correlation of measured items ............................................................................................. 14
Figure 5 - IPS Homepage ..................................................................................................................... 16
Figure 6 - Main Navigation for IPS Site .......................................................................................... 17
Figure 7 - Task 2 Quantitative Summary ...................................................................................... 20
Figure 8 - Divisions/Departments page with no Technology section .......... .......... ........... 21
Figure 9 - Task 3 Quantitative Summary ...................................................................................... 24
Figure 10 - Long list of state colleges and universities ........................................................... 25
Figure 11 - Task 4 Quantitative Summary ................................................................................... 27
Figure 12 - Task 5 Quantitative Summary ................................................................................... 31
Figure 13 - Task 5 Quantitative Summary ................................................................................... 34
8/7/2019 IPS Usability Testing Report - Report
4/56
4
I. Executive Summary
The Indianapolis Public School System (IPS) Corporation website is an informational
website about the Indianapolis Public School Corporation. The purpose of the site is to
communicate news, events and educational related information to primary groups of users
who include faculty, parents and the community.
This report identifies key findings from the usability study conducted by our team on
the IPS Corporation website. The usability study included 11 participants comprising 5
community members, 4 parents and 2 teachers. All participants performed 5 tasks except for
teachers who performed 7. Each task was recorded to captured data, which was later
analyzed and included in this reported. Tasks were selected for their saliency in providing
information to perspective users based on activities that were performed both routinely such
as finding school board meetings to those that were performed less frequently such as finding
cost associated to a technology plan budget. Teachers performed two additional task which
were designed to help the team identify how recognizable the layout and content was
between the general users understanding of the site and teachers understanding of the site. In
general, the tasks selected had to include at least 3 steps to be considered for the study. In
addition, an earlier usability inspection was performed which helped establish the direction
and objectives for the final usability study.
The following are the key set of findings captured from both of the usability
inspection and usability study, which have been identified as having the most impact to the
usability of the website:
1. Content: Many participants became confused due to vague, non-dated or missinginformation.
8/7/2019 IPS Usability Testing Report - Report
5/56
5
2. Navigation: Many problems were due to inconsistency in the location and style oflinks. Additionally, the page banner located on every page due to its size and
distracting messages causes all the users unnecessary scanning of pages to locate the
appropriate navigation and/or content identified in the set of tasks.
3. Information Architecture: The website needs to align to how the target usersunderstanding a traditional website. Many problems surrounded the organization of
topics within a subject area. Topics that belonged to different subjects were included
into other unrelated subjects, which caused vague or inconsistent relationships
between information that confused participants.
4. Presentation: A lack of consistency in style and organization of page elements (i.e.links, headers, banners) caused additional overhead in the participants ability to
navigate and use the website effectively.
After careful review of both the usability inspection and evaluation study given the top
four impacted areas mentioned above our team recommends the following top seven critical
areas for improvement. Additional recommendations can viewed in the Overall
Recommendations For Improvement section later in this report.
Key Recommendations:
1. Establish a checklist and a content review board to ensure information is up-to-date and correct
2. Ensure dates are captured and identified on items that have relevance around adate or time3. Use proper formatting ensuring consistency between text and links, as well as
banner picture and links.
4. Keep the most necessary navigation and get rid of the ones irrelevant to the page.Make sure the appearance and layout of the navigation are salient and consistent.
5. Provide a sitemap of the website to aid the user gain an overview of the logicalarchitecture of the website, so that they can find the information they need.
8/7/2019 IPS Usability Testing Report - Report
6/56
6
6. Design should be include visual cues in the navigation that highlights their currentlocation. Ensure each page has a title matching the pages content.
7. Make the navigation bar around the banner area more salient. Avoiding usingcolors similar to the banner for the background of navigation bar. Make theappearance of the navigation bar more touchable like a button.
8/7/2019 IPS Usability Testing Report - Report
7/56
7
II. Instruments and Methods
This study employed several tools and techniques for collecting information
pertaining to the usability study. In the following sections an account of participants,
methods and instruments are explained. A complete list of artifacts including the Pre & Post-
Test Questionnaire, Post-Task Questionnaire, Usability Study Script among others used in
the study can be found in Appendix.
Participants
The IPS district website caters to three primary groups of users: Teachers, Parents
and the Community. For our study we focused on these key user group which we solicited
11 participants (5 community members, 4 parents and 2 teachers). All 11 participants did
have at least some computer experience with 7 self-identifying as very experienced. All 11
participants had over 20 hours of Internet usage per week. Additionally, 9 the users had
never visited the IPS website, however, 4 parents and both teachers visited their own school
corporations website.
Data Collection
Due to the complex nature of collecting both quantitative and qualitative data our
team developed and employed a well-crafted process and artifacts, which were used in the
study (see Appendix B). The evaluator first started by traveling to the participants location
with a laptop, which was loaded with recording and capturing tools (see tools below). Once
the evaluator was setup they began by giving the participant a pre-test questionnaire, which
solicited demographic information along with computer and Internet usage information.
Once the participant completed the pre-test questionnaire they were then read a script that
provided an over view of the study which included information regarding the purpose of the
8/7/2019 IPS Usability Testing Report - Report
8/56
8
study along with the intention of the information collected, privacy & confidentiality and
their rights as a participant in the study. After the participant agreed to the terms the
evaluator then gave the participant an introduction to the computer system and to the
website allowing them to complete a nominal task before the official test was initiated.
Once the participant was familiar with the setup of the laptop, browser and site the
evaluator began the test. The evaluator began recording the session along with reading each
task out load. The evaluator recorded both the start and end times along with the success,
partial success or failure for each of the task. In addition the participants we encouraged to
think out load as they worked through each task. This helped provided additional insight and
rich qualitative data, which was noted and later analyzed by the evaluator and team. Also
after each task the participant was given a post-task closed question questionnaire, which
solicited their feedback on how satisfied with the ease of completing the task they were. The
questionnaire ranged from 0 (strongly disagree) to 5 (strong agree).
After all the tasks were completed the evaluator then moved to the last section of the
script, which included the post-test questionnaire and wrap-up. The post-test evaluation
included several open-ended questions to give the participant a chance to give their overall
impressions of the website and to reflect on the site as a whole. Once completed with the
post-test questionnaire the participants were again thanked for their participant and given
contact information of the evaluator in case additional thoughts or concerns arose about the
website or the study in general.
Environment & Timeframe
Due to the short timeframe for this study and the need to emulate an environment
that best represented our key user groups most of the testing was conducted in either the
8/7/2019 IPS Usability Testing Report - Report
9/56
9
participants home or place of work. The testing was conducted between November 1st and
November 8th, 2010 at times that were most convenient for the participants.
Tools
A variety of tools were used to help record and capture data from the participants.
These tools included Clearlefts Silverback 2.0, Camstudio and Syniums Screenium for local
participants and Ciscos Webex for remote participants. These tools allowed our team to
record the participants actions on the screen, the participants verbal comments along with
the participants face, which allowed us to capture both verbal and nonverbal cues.
Tasks
Below are the following tasks used in the study for participants to complete. Tasks 6 and
7 were included in the test if the participant was a teacher. Tasks were selected for their
saliency in providing information to perspective users based on activities that were performed
both routinely such as finding school board meetings and to those that were performed less
frequently such as finding cost associated to a technology plan budget. The usability
inspection was also used to identify steps used within the tasks that would expose
problematic areas found by our experts. The additional tasks for teachers were designed to
help the team identify how recognizable the layout and content was between the general users
understanding of the site and teachers understanding of the site. In general, the candidate
tasks had to include at least 3 steps (or navigation points) to be considered for the study.
1. Find the next school board briefing meeting so that you can attend.As a parent you are growing concerned about the direction the IPS School board ismoving towards concerning year round school. Another parent mentioned that thereare school board meetings where you can go and voice your concerns. Assume todayis December 1st and you want to go to the next briefing session. Please identify thedate for this meeting.
8/7/2019 IPS Usability Testing Report - Report
10/56
10
2. Find the dollar amount IPS has budgeted towards technology in itsTechnology Plan.
As a parent you feel that your childs ability to learn and use technology is very
important. In the midst of growing concerns about budget cuts you are interested inknowing how much money IPS has allocated to implementing new technology. Findthe budgeted amount IPS has allocated in its Technology Plan.
3. Find contact information for Sawyer College in Merryville.As a parent of a high school junior, another parent told you that IPS has resources foryou and your child to start exploring colleges. Find contact information about SawyerCollege in Merrillville.
4. Find the middle school closest to your house.As a parent you want to identify the schools that are nearest to your home where yourchild would attend middle school. Your house is located at 816 N. Audubon Road,Indianapolis, IN 46219. Please name the closest high school to your house and findthe enrollment information for that school.
5. Find the 2011-2012 English Student Application for the Magnet Schools.You noticed that a school located near your house was a magnet school. Pleaselocate the 2011-2012 English student application for this magnet school.
If you are a teacher, please complete the additional two tasks:
6. Find the telephone number and location of IPS payroll.As a new teacher you had an issue with you pay check this week. You contacted thefront office at your school, however, because they do not directly deal with payrollthey directed you to the corporation website to find the appropriate contactinformation and the location were you needed to go to sort everything out. Identifythe telephone number and location of the payroll office
7. Find amendment 5233 of Teacher Contract.As a teacher you wanted to review updates to the 2010 teacher salary informationspecifically Amendment 5233. Locate information about Amendment 5233.
8/7/2019 IPS Usability Testing Report - Report
11/56
11
III. Results
After careful analysis during both live and recorded user-testing review, we identified
trends common to many of our participants. In our analysis, we looked at quantitative data;
specifically with regard to time-on-task, success/partial-success/fail rates per task, post-task
questionnaire answers, and a calculation on the lostness of the participant. Qualitative data
that we focused on include our team observations, questionnaires, demographic data, and
other such items that gave a sense of the users state of mind, the sites pleasurability, and the
overall acceptance of the site by the user. These results were analyzed first individually, then
in comparison with each team-member for inter-rater reliability. They were then cross-
referenced to the heuristic and scenario-based evaluations conducted previously for this site.
Here is a summary of the tasks, mentioned before:
Task 1 Find the next school board briefing meeting so that you can attend.Task 2 Find the dollar amount IPS has budgeted towards technology in its Technology Plan.Task 3 Find contact information for Sawyer College in Merryvile.Task 4 Find the middle school closest to your house.Task 5 Find the 2011-2012 English Student Application for the Magnet SchoolsTask 6 Find the telephone number and location of IPS payroll
Task 7 Find amendment 5233 of Teacher Contract
Figure 1 - Average times it took each participant to finish each task.
0
50
100
150
200
250
300
350
400
1 2 3 4 5 6 7
Time(sec)to
completetask
Time on Task Analysis
Time-On-Task
Perceived Ease
8/7/2019 IPS Usability Testing Report - Report
12/56
12
We measured the time it took each participant to accomplish the tasks. Above you
see the average time on task for the participants per task. Notice the times for tasks two and
four are highest. These two tasks were proven to be the most compelling for our participants;
as you will see in later parts of this paper. This graph also compares the time on task with the
perceived ease for the participant. Notice, on task 2, that the perceived ease is significantly
lower than the other 6 tasks. This indicates that the user found it most difficult andspent the
most time attempting to complete it. One thing that should be mentioned with regards to the
time on task calculations is that the time on task takes into account the attempts by
participants that were unsuccessful and incorrect. Next you will see success rates which will
correlate to these findings.
Figure 2 - Full-Success/Partial-Success/Give-Up rates for each participant per task
For every participant and for every task the participant started, we kept a tally on the
number of successful completions, partial successes, and complete withdraws. A successful
completion occurs when the user completes the task with no input (or input that would not
alter the intended course of the participant) from the proctor. Partial success occurs when the
participant has successfully completed the task, but was, at some point during the scenario,
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7
Analyzing Levels of Success
"Give up/ wrong/ use
outer tool"
Partial success
Complete success
8/7/2019 IPS Usability Testing Report - Report
13/56
13
assisted by the proctor. Give up or withdraw simply means that the participant never
completed the task; even after given help. While task 2 has the lowest success rate of all the
tasks, notice, from before, that it still maintained the highest time on task of all tasks. This is
indicative of the user attempting, by-all-means, to complete the task yet failing to do so.
Figure 3 - Summary of quantitative data showing average time on task, lostness, success ratesand ease of use for each task.
After combining the two items listed before, time on task and success rate, with a
measure of lostness and ease of use, we can see that the most problematic tasks were Task 2
and Task 4. Both of these tasks displayed the highest time-on-task and lostness levels. They
also displayed the lowest success rate and ease-of-use averages. This not only indicates that
these two tasks were quantifiably more difficult than the rest, but it also indicated that the
user could feel this while completing the task; based on the post-task questionnaire regarding
the users ease-of-use.
Another thing to note is that tasks 6 and 7 were performed only when the participant
was a teacher. There were a total of two teachers, who both were parents. All 11 participants
completed tasks 1 through 5. The data shows that these two tasks (task 6 and task 7) were
both completed with complete success, relatively no lostness, and a comparatively low time
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
Task
1
Task
2
Task
3
Task
4
Task
5
Task
6
Task
7
Quantitative Summary
Time on Task
Lostness
Success Rate
Ease of Use
8/7/2019 IPS Usability Testing Report - Report
14/56
14
on task average. This indicates that the teacher-participants may have been experts with the
system as they need to use it for daily-work. While this may imply that increased exposure to
the site generates higher satisfaction with the usability, it is important to note that the user
may become habituated to a poor usability experience and learn the wrong ways efficiently.
After examining tasks 1 through 5, we noticed an interesting phenomenon regarding
task 3. Out of the 5 main tasks, task 3 had the lowest time on task of all of them. The ease of
use was relatively high in comparison to the other tasks. Success rate tended to be higher than
average and the lostness of the task completion seemed to be extremely low. These facts may
imply a conflict between our usability evaluation based on heuristics and scenarios compared
to the usability testing done by our participants. This disconnect will be explored later when
looking at each task individually.
LostnessTime onTask
SuccessRate Ease
Lostness 0.92 -0.93 -0.39
Time onTask -0.95 -0.68
Success Rate 0.66Ease
Figure 4- Correlation of measured items
After running a correlation on the measured items, we found three interesting, and
very strong, correlations. First, for the average participant there was a +0.92 correlation for
time on task and lostness. This indicates that as the participant would spend more time on
the page, they would get more and more lost. This also implies that time was not spent solely
on a few pages for purposes of our time-on-task. Lostness factors in page counts, therefore,
the user spent a great deal of time searching many pages to complete the task. We also found
a -0.93 correlation regarding success rates and lostness. This is very interesting as it indicates
that as the user got more and more lost in the system, they would have a much less chance of
8/7/2019 IPS Usability Testing Report - Report
15/56
15
succeeding. It also implies that the more the users would succeed, the less lost they would be
in the system; meaning they needed less clicks to get to their final destination. The final and
strongest interesting correlation deals with the participants time on task and their success rate.
Interestingly, as the success rate of an individual would go down, their success rate would go
up. This implies that when the participant did succeed, they did so quickly and the items that
the participant was unsuccessful on took more time.
Task 1 School Board Meetings
Task description
Average Time on Task/Average of All
3:49 / 3:54
Success Rates
Full-Success 7Partial-Success 2Fail 2
Lostness0.53
Participants completed task 1 after completing a warm-up task to help avoid the
learning curve of the site. While this does hellp with avoiding some skewed statistics, it does
not take away from the fact that users of any site will be more proficient with general
navigation, logical constructs, and overal usability of any site; good or bad design.
As a parent you are growing concerned about the direction the IPS School boardis moving towards concerning year round school. Another parent mentioned that
there are school board meetings where you can go and voice your concerns. Please
identify the date for the next briefing session.
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
Quantitative Summary
Time on Task
Lostness
Success Rate
Ease of Use
8/7/2019 IPS Usability Testing Report - Report
16/56
16
Given that, however, the time on task for task 1 was right on average for the first five
tasks (the tasks intended for parents). Seven of the participants completed this task without
any help from the proctors while 2 needed assistance to finish successfully and 2 failed or
gave up. The users had a lostness of 0.53. This number does indicate a relatively high lostness
for this task.
Usability Issues encountered by participants:
# Problem 1: Missed Navigation
* Problem Description:
We noticed that participants failed to
recognize the main navigation area as
clickable links; a phenomena Don Norman
calls Banner Blindness. This is especially
harmful on the homepage of IPS site due to
the dashboard-like style. We observed that users tended to spend a large amount of time
specifically looking for their course-of-action
in the homepage content structure. This was
emphasized in our usability testing in a section denoted Homeless Homepage where we
displayed the problems with a homepage containing only links. We saw that this dashboard-
like style lead the participants down a path that reinforced them to think the homepages main
content had allt he answers for their task. This information architecture set up is one of the
reaons contributing towards a high lostness level. It was assumed that if it wasnt in the
homepage area, it was a task that could not be completed.
* Severity | 1 (Show Stopper)
Figure 5 - IPS Homepage
8/7/2019 IPS Usability Testing Report - Report
17/56
17
* Design defect | Navigation
#Problem 2: Inconsistent Labeling
*Problem Description:The users continuously wanted to find
school board calendars via the calendar links on
both the homepage and the individual school sites.
They were surprised and a little frustrated when
browsing the page that appeared as a result of
clicking the calendar link to find multiple different
types of calendar, but none pertaining to the
school board meetings. The content simply didnt
match the participants expectations. It was
interesting to see that when users ended up on the
IPS school board calendar page, they did so after
passing the link, on average, 4 times before. This accounts for the high lostness displayed by a
majority of the participants.
* Severity | 2 Medium
* Design Defect | Navigation
#Problem 3: Logical Constructs Not Adhered To
* Problem Description:
Participants were fixated on the fact that when looking for school board meetings,
you are looking
from the
The access to the school boardmeeting should also be in the Parents
part
The task is about parents school
board meetings, so I think I can find
the meeting time in parents
I dont think it would be in general
information because the title seemed
to be about static information, but the
meeting times seem more dynamic.
Balanced Calendars seems more
relevant.
I wouldprefer it showed general
information once I am on the school
board page.
The members list in school boards isdistracting. It makes me think the
School Board [link] is about personal
introductions with no information
about events
What Is DOE?
Figure 6 - Main Navigation for IPS Site
8/7/2019 IPS Usability Testing Report - Report
18/56
18
perspective of a parent. While logically, school board meetings may seem to belong under the
category of School Board, our participants still felt that it was an activity that Parents attend.
Given this, for the participants, it seemed only logical to navigate to the main parent link and
search the information in that subdirectory. It was only after much confusion in the parent
directory, that many of ths users finally decided to reevaluate their approach and searh other
main links or simply give up at this point. This emphasizes the idea that the participants had
enormous expectations regarding the location of school board meetings. The majority of
participants expected to find school board meetings within the parents link as opposed to the
school boardlink.
* Severity | 2 Medium
* Design Defect | Information Architecture
#Problem 4: Banner Data Inaccessible
* Problem DescriptionOne item that was not prevelent in our usability evaluation before, but became very
evident during our user testing, was the problems with the banner. For a few participants,
when the rotating banner rotated to a particular ad dealing with school borads (although it
was an ad that was not relevent to specific task), the users felt the need to wait for the ad to
come full circle to proceed. This rotation of the banner caused a higher time on task because
of the wait time for it to come back around.
* Severity | 2 Medium
* Design Defect | Navigation
#Problem 5: User Calculation
* Problem Description:
8/7/2019 IPS Usability Testing Report - Report
19/56
19
Once the users navigated to the correct page, there were some characteristics of the
interaction that were quite interesting. Users noted that when navigating through the list of
school board meetings, it was hard to identify the next meeting because of the order of the
list. They expected the next meeting to be displayed first and didnt want to hunt and search
down the page for the every day to commpare it to today. They needed some kind of
distributed cognition to assist with their calculations of today, the displayed date for the
meeting, and the difference between those two dates.
* Severity | 2 Medium
* Design Defect | Navigation
8/7/2019 IPS Usability Testing Report - Report
20/56
20
Task 2 Finding the Technology Plan
Task Description
Average Time on Task/Average of All
5:28 / 3:54
Success RatesFull-Success 3
Partial-Success 4Fail 4
Lostness0.60
As evident from the graph above, the time on task for task 2 was higher than any of
the other tasks in this report. At 328 seconds, participants seemed to take as more time to
complete this than a majority of the other tasks, even though the ideal path for a majority of
these tasks consisted of only 4 pages. Also, there were fewer successes on this task than for
any of the other main tasks (tasks 1 through 5). Both the time on task and success rates
mentioned before alludes to a high lostness level. As shown in the graph above, the lostness
level for this task was higher than any other task in this report.
Usability Issues encountered by participants
#Problem 1: Skip Navigation
* Problem Description
0.000.501.001.502.00
2.503.003.50
0.0050.00
100.00150.00200.00
250.00300.00350.00
Quantitative Summary
Time on Task
Lostness
Success Rate
Ease of Use
As a parent you feel that your childs ability to learn and use technology is very important. In
the midst of growing concerns about budget cuts you are interested in knowing how much
money IPS has allocated to implementing new technology. Find the budgeted amount IPS has
allocated in its Technology Plan.
Figure 7 - Task 2 Quantitative Summary
8/7/2019 IPS Usability Testing Report - Report
21/56
21
Participants consistently skipped the navigation of the page and skimmed through
the pages content in hopes of finding the answer. This contributes a great deal to the average
time on task for this task. We believe the participants felt the need to skim every page for the
answer due to not only the banner blindness explained earlier, but also the mislabeling of
pages. Page titles within the About IPS section are all the same. The participant had trouble
differentiating between which About IPS section they were in. Also, related to this, many
participants were unsure of their current location on the page due to page title and navigation
not retaining their visited or current pages. This issue was addressed before in our evaluation
in a section under the Information Architecture section labeled Where am I.
* Severity | 1 Show Strpper
* Design Defect | Information Architecutre
#Problem 2: Language Disparities
*Problem Description:
There were some semiotic related
problems regarding the language used on
some of the pages. For an example of this,
consider this tasks language of Technology
Department. When attempting to find the
technology department in the
Divisions/Departments page (which was
navigated to quite efficiently by most
participants), they all immediately scrolled
to the Ts to locate Technology, when in
Figure 8 - Divisions/Departments page with no
Technology section
8/7/2019 IPS Usability Testing Report - Report
22/56
22
actuality, the correct page was Information Technology. Four users simply left the page,
assuming they were in the wrong area. A few of them were redirected back to the page and
given the hint of thinking of synonyms of Technology while others found it, simply by
chance. This is a prime example of ensuring that the language written matches the language
understood by users. In this case, the users are parents who may or may be familiar with
technology, but usually not. Ensuring both methods of language are covered will allow users
to find what they are looking for, rather than simply leaving the page as our participants so
eagerly did.
* Severity | 2 Medium
* Design Defect | Semiotics
#Problem 3: Where to Begin
*Problem DescriptionA couple of our participants navigated to the
superintendents blog or some area of news such as recent
performance metrics or school district goals. Some even expected
it to be a form of general information which, they assumed,
would be located in an About Us type of page. Regardless of
where the users thought the budget, or technology plan, would be,
they didnt expect it to be where it is initially. They all got to the
intended page quickly, but only after some hesitation at the
beginning of the encounter.
* Severity | 2 Medium
* Design Defect | Information Architecture
I dont even know where
I am
If Im looking for
something like report or
annual then its notgoing to be connected
with a department; its
for the whole
organization!
The technology budget
plan should be under
About IPS
I think the tech-budget
belongs to the IPS budget
in general.
Honestly, I wouldprobably just end up
calling them at this
8/7/2019 IPS Usability Testing Report - Report
23/56
23
#Problem 4: Navigation of PDF
*Problem Description:When the users found the technology plan, time on task was taken. The goal here was
to see how long it would take our users to loacte and access the technology plan. After the
task was deemed copmlete, we allowed the users to continue without telling them we werent
timing them any more. This proved to be rather revealing about another issue we didnt even
think about. Once the user has accessed the technology plan, finding the actual information
proved to be rather difficult. The navigation and information architecture of the pdf
document was not easily navigable and elicited some frustration in our users. The users, even
when in the document, didnt know where they were or how to get to where they wanted to
go. We recommend some form of hypertextual linkage on either the pdf document or an
html page with the same information as the pdf document. As of now, the table of content
listed an approximate page that the user had to scroll to; if they used the table of contents at
all.
* Severity | 1 Show Stop
* Design Defect | Navigation & Information Architecture
8/7/2019 IPS Usability Testing Report - Report
24/56
24
Task 3 College Resources
Task Description
Average Time on Task/Average of All
2:23 / 3:43
Success RatesFull-Success 7
Partial-Success 3Fail 1
Lostness0.31
Time on task for task 3 was lower than any of the other 5 main tasks. 143 seconds
surpassed all of our expectations and gave some insight on what we thought was a clear
usability problem. Task 3 had more Full and Partial successes than any other task. Lostness,
while still high, was the lowest of all five tasks as well. The one participant that failed to
complete this task or gave up on it does not account for a major shift in the data due to the
fact that the average for the other 10 successful participants average approximately the same
time on task.
As a parent of a high school junior, another parent told you that IPS has resources for you and
your child to start exploring colleges. Find contact information about Sawyer College in
Merrillville.
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
Quantitative Summary
Time on
TaskLostness
Success Rate
Figure 9 - Task 3 Quantitative Summary
8/7/2019 IPS Usability Testing Report - Report
25/56
25
The pages on this site
contained a lot of information in one
location and were a major focus of our
usability testing. Users searching for
the school began to get frustrated at
the number of scrolls and page clicks
they had to go through just to find
information on a state university.
Some didnt think that resources was
the appropriate place for state universities.
Others, similar to the problems outlined in
task 1, believed that state university searches were activities the parents would complete and,
therefore, would be located in the parents section.
Some areas of focus that were identified during this task by
our team dealt mostly with general elements needing correction as
opposed to task-specific items. Some such items include some
elements discussed earlier. Some examples of these include
navigation blindness due to the banner, mislabeled pages, and
inconsistent sub-navigations.
As discussed earlier, this task had a lower time on task than
any of the other tasks in the 5 main tasks for parents. Some other
important discoveries are made regarding the quantitative data as well.
The success rate, on average, was ultimately higher than any other 5
Figure 10 - Long list of state colleges and
universities
Can I just Googleit?
Does thenavigation change
every time I go to
a new page?
Why are there
two home links?
(While browsingwithin the
schools) How can I
get back to that
other page?
I would have just
Googled this
8/7/2019 IPS Usability Testing Report - Report
26/56
26
main tasks and the lostness level was lowest of all of these. Our usability evaluation indicated
that the resources section would be a very usability-error-prone section of the website due to
the information architecture and navigation presented. Our assumptions were proven
false with regards to quantitative metrics. The participants were able to get to where they
needed to go, relatively efficiently. However, getting there is only half of the battle with
regards to usability. The pleasure attained by the user in accomplishing the task is as equally
important as the accomplishment of the task itself.
Usability Issues encountered by participants
#Problem 1: Mislabeling of Pages
* Problem Description:Participants expressed distaste with their experience of the site based upon psycho-
pleasures related to page semiotics. Some mislabeling of pages and multiple links meaning
the same thing caused grief with the participants. The information architecture that the
users experienced to develop their mental map of the site was truly tested at this point. The
users have developed a sense of how things are set up with the first two tasks and now can
use that construct developed to accomplish this task. However, given the two-task prep with
this task, some users still felt like they could not navigate the site appropriately based upon
some of the structure of the sub-pages.
* Severity | 2 Medium
* Design Defect | Semiotics & Information Architecture
8/7/2019 IPS Usability Testing Report - Report
27/56
27
Task 4 Find the Closest School Task Description:
Average Time on Task /Average of All
4:26 / 3:43
Success RatesFull-Success 5
Partial-Success 1Fail 5
Lostness0.59
The time on task for finding the school closest to your home was the second highest
time of all tasks. There were a total of 5 complete successes, 1 partial success (which involved
redirection by the proctor), and 5 complete fails or give ups. The level of lostness is almost
equivalent to the highest lostness (only one hundredth off).
It is important to note that, even though
time on task for this scenario is the second
highest of all tasks, it could be worse than it looks.
Due to the fact that 5 of the participants
completely failed the task, the numbers for this
task may be slightly skewed. If these participants
As a parent you want to identify the schools that are nearest to your home where your child
would attend middle school. Your house is located at 816 N. Audubon Road, Indianapolis, IN
46219. Please name the closest high school to your house and find the enrollment informationfor that school.
0.00
0.50
1.001.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00150.00
200.00
250.00
300.00
350.00
Quantitative Summary
Time on Task
LostnessSuccess Rate
Ease of Use
Figure 11 - Task 4 Quantitative Summary
8/7/2019 IPS Usability Testing Report - Report
28/56
28
gave up too quickly, these numbers could have factored into the time on task we saw by
dragging it lower than the average would actually have been. By having 5 participants fail or
give up, and a lostness score of 0.59, it is evident that the participants, in general, were not
pleased with what it took to accomplish the task. On top of this, the average ease-of-use
index was the second lowest of all of the tasks.
Usability Issues encountered by participants:
#Problem 1: Banner Inconsistencies
* Problem Description:Participants noticed that the banner was static for some of the pages
and dynamic for others. This caused much confusion when users
were attempting to figure out how they ended up in certain points of
their navigation. Upon navigating over a banner, the user was unable
to determine whether it was a link or not. This enhances the previous
concerns regarding banner blindness; if the banner act like text, the
user will be more likely to avoid text and consider it as part of the
banner than ever before.
* Severity | 2 Medium
* Design Defect | Navigation
#Problem 2: Logical Path Undefined
A logical path was selected by a few of our participants
dealing with enrollment. When participants went to the enrollment
section of the site, they expected to be able to locate a school there.
Locating a school and enrolling in it seemed to be common place to
The Boundary Map
doesnt help. Where
can I give my
address?
The Map is too
small to see the
information
Im not familiarwith the school
boundary map
I think I left the
website
My first reaction isto use Google
maps
Can I use GoogleMaps?
8/7/2019 IPS Usability Testing Report - Report
29/56
29
be together for most of our participants. When accessing the enrollment page and not finding
a school-locator, the users either remained confused and randomly browsed, or simply gave
up. Some users even felt the need to search the schools-specific sites to see if there was
something there that could assist them. This lead to some information architecture issues
we explained earlier.
* Severity | 2 Medium
* Design Defect | Information Architecture
#Problem 3: Impossible Task- Does not exist
*Problem Description:
Another logical area the users expected to find the schools location was in the schools section
of the page. Some participants went immediately here and assumed that it was a trick
question because they couldnt believe the developers would not implement something like
this here. The area they were looking for was the boundary map, which was not located in the
school sub-pages. The semiotics used for the boundary map and expectation of it being in
enrollment or schools are understandable expectations and links to a boundary map should
be included in both of these sections.
* Severity | 1 Show Stop
* Design Defect | Semiotics
#Problem 4: Syntax Rules not Obvious
* Problem Description:The actual boundary map itself had some issues with technology and functionality.
We found that when the user types in their address to locate the nearest school, they need to
match the syntax rules of the system, displayed under the map and input field. When they
8/7/2019 IPS Usability Testing Report - Report
30/56
30
enter an address that doesnt match the syntax rules, they are presented with an alternate
address as a Did you mean type of suggestion. However, upon clicking this, the
subsequent page is another page notifying us that our input is incorrect. The suggestion itself
is as incorrect as the users input. Another higher level issue deals with the users
understanding of the semiotics of the page. Many of our participants did not like the idea of
Boundary Map as the area they navigate to in attempts to locate a school close to them.
* Severity | 1 Show Stop
* Design Defect | Semioics
8/7/2019 IPS Usability Testing Report - Report
31/56
31
Task 5 Magnet School Application
Task Description
Average Time on Task/Average of All
3:32 / 3:43
Success RatesFull-Success 7
Partial-Success 2Fail 2
Lostness0.50
Task 5 could be denoted as the Average task for our participants. The time on task
was 212 seconds with seven of our participants completely finishing, 2 finishing with
assistance, and 2 failing or giving up. The lostness scale is right at 0.50 which is about average
for this site, but completely unreasonable for usage in general.
Usability Issues encountered by participants:
#Problem 1: Mismatched User Expectations
* Problem Description:
This task summarized many of the points previously made in other tasks. One point referred
to before for other sections, and now this section, is the labeling of pages and how those
You noticed that a school located near your house was a magnet school. Please locate the
2011-2012 English student application for this magnet school.
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
Quantitative Summary
Time on Task
Lostness
Success Rate
Ease of Use
Figure 12 - Task 5 Quantitative Summary
8/7/2019 IPS Usability Testing Report - Report
32/56
32
semiotics match the users expectations. Users expected the magnet school applications to be
in the general information area; especially an area denoted as Application.
Some participants felt the need to go to the schools list and find a magnet school to
hone in on while searching for the application for that particular school. At this point in the
scenario, some participants felt confused as to some of the semiotics used to describe the
schools. Magnet schools were both located under high school and other schools. This
confused the user into thinking magnet schools had to be one or the other. However, even
when the user found a magnet school, it was to no avail as the individual school page did not
have what the user expected to be there; an application to that school. This poor set up in
information architecture could cause the participant to feel like there is no place to go from
here.
* Severity | 1 Show Stopper
* Design Defect | Information Architecture & Semiotics
#Problem 2: Ignore Navigation
* Problem Description:
Users tended to want to read all of the material before clicking on the magnet
link. This has to do with an area we discussed in our evaluation testing focusing
on the side navigation being ignored by the user. Most times, especially for this task (which is
odd because it was the last task for most people) the side navigation was not referenced as
often as we expected. While time on task was not horrible, it was high enough to elicit
concerns that items like this navigation instance could cause some major pleasure losses for
the site.
Is other magne
8/7/2019 IPS Usability Testing Report - Report
33/56
33
* Severity | 2 Medium
* Design Defect | Navigation
8/7/2019 IPS Usability Testing Report - Report
34/56
34
Task 6 Payroll Contact Information (Teachers Only) Task Description:
Task 7 Teacher Amendment (Teachers Only) Task Description:
Average Time on TaskTask 6 - 60 secondsTask 7 - 120 seconds
Success RatesFull-Success 2
Partial-Success 0Fail 0
Lostness
0.00
In both task 6 and task 7, there were only two participants due to the nature of the
task. The tasks involved items only teachers would need to do, so they were conducted only
by our two teacher participants. Time on task for both of them was 60 and 120, respectively
for tasks 6 and 7. They both succeeded fully within the ideal number of clicks.
Users both found what they needed to efficiently. This may not be due to great user
experience designing, yet, adaptation by the participants to this poor user design. Since the
users were both teachers within the IPS school district, they were our expert users and thus,
As a new teacher you had an issue with you pay check this week. You contacted the front
office at your school, however, because they do not directly deal with payroll they directed you
to the corporation website to find the appropriate contact information and the location wereyou needed to go to sort everything out. Identify the telephone number and location of the
payroll office
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
Quantitative Summary
Time on Task
Lostness
Success Rate
Ease of Use
As a teacher you wanted to review updates to the 2010 teacher salary information specifically
Amendment 5233. Locate information about Amendment 5233.
Figure 13 - Task 5 Quantitative Summary
8/7/2019 IPS Usability Testing Report - Report
35/56
35
have had much experience with this design interface. Being accustom to a poor user design
experience does not mean a poor user design experience doesnt exist.
8/7/2019 IPS Usability Testing Report - Report
36/56
36
IV. Synthesis of Results from Inspection
The team analyzed the qualitative data discovered during the usability testing and
categorized it according to the five design dimensions: Content, Information Architecture,
Navigation, Presentation, and Semiotics. The team then compared the testing results with
the problems discovered during the inspection to reaffirm the initial findings and to uncover
any new findings.
Content
New Problems
Within the Content section, participants encounter three new problems during the
usability testing which was not discovered in the inspection (ID: 5, 6, 7). They included
search problems caused by Unclear input requirement, Unclear boundary map, and the
inconsistency between participants expectations and the Vague enrollment information.
Inspection Problems
Of the four problems the team found in the inspection, the participants encountered
none of them during the usability testing. The problem of Missing and inconsistent
information (ID: 1) was already updated on the website by the time our participants began
testing. The other three (ID: 2,3,4) were not confirmed as they were included in the tasks,
however, participants who did visit those pages during the test did not uncover any problems.
The table below shows the comparison of problem findings in the Inspection and Usability
testing.
8/7/2019 IPS Usability Testing Report - Report
37/56
37
ID Inspection Results Usability Results Conclusion1 Missing and inconsistent
informationThis problem is repaired bythe website. Not a problem
2 Target User Misidentified Participants didnt encounterthis problem. SeverityDecrease.
3 Consistency of informationand representation
Participants didnt encounterthis problem. SeverityDecrease.
4 When was the news new? Participants didnt encounterthis problem. SeverityDecrease.
5 Unclear inputrequirement
Major new problem
6 Unclear boundary map Major new problem7 Vague enrollment
informationMinor new problem
Information Architecture
In the Information Architecture dimension, participants encountered 7 problems,
including 4 problems predicted by the inspection and 3 new ones.
New Problems:
Mismatch information categorization between user and the website (ID: 1) is
a new problem which was found in usability testing and has a very high severity. About 8
participants encountered this problem and it is a main reason causing the highest time-on-
task and lostness, and lowest success rate of Task 2. Users categorize information according
to the semiotic closeness between target information and category, while the website is
8/7/2019 IPS Usability Testing Report - Report
38/56
38
grouped according to location and structure of the physical IPS departments. The users were
not familiar with the organizational layout of the IPS departments. Problem finding items
in a long list (ID: 10) mainly occurred when users were looking for schools from a page
that contained lists. When the users found finally found the school they were looking for the
expected it to be according to the school name sort alphabetically, however the schools were
listed in the order of IPS school number which did not provide much help.
Overlapping Problems:
The Overlapping Problems (ID: 2,4,5,8) were predicted in the inspection and proved
by the usability testing. Users encountered Problem of find school according to distance
(ID: 2). Their first reaction so solve the problem was by using the boundary map, but the
static and separate boundary map of each school provide little help for measuring distance
from home to school or location comparison. Three of them used Google Map to finish the
task. Regarding to the problem ofNo way go back (ID: 4), users were confused on how
to go back to the previous page, but they overcame it by clicking the back button multiple
times. The problem ofWhere am I? (ID: 5) was caused by both mislabeled pages and a
lack of cues for current location. The weak awareness of the current location makes it
difficult for the user to find the page again as well as understand the website structure.
Participants encountered the problem of Homepage without main content (ID: 8).
Most of them wandered on the homepage for a long time, especially in Task 2, seeking the
entrance to the information they were looking for. They found little help with the
dashboard-like homepage with its many links.
Inspection Problems:
The three Inspection Problems (ID: 3, 7, 9) were not discovered by participants
during the usability testing (Too much information accommodating too many people
8/7/2019 IPS Usability Testing Report - Report
39/56
39
(ID: 3) and Archiving (ID9)). Due to the lack of evidence in testing the severity for these
three problems was decreased.
An Interesting phenomena appeared regarding the problem Unrelated
Navigation. When users were finding information within the page, they naturally ignore
the unrelated navigation. The severity for this problem was also decreased.
ID Inspection Results Usability Results Conclusion1 Mismatch information
categorization between userand the website.
New problemencountered by mostusers with high severity
2 Visualization of schoolproximity
Problem of finding schoolaccording to distance
Increase Severity
3 Too much informationaccommodating toomany people
Not discovered byparticipants. Severitydecrease.
4 No way back Users are confused and havedifficulty back from the schoolwebsite
Increase Severity
5 Where am I? No clues of the currentlocation
Increase Severity
6 Unrelated Navigation Ignore by users. Severitydecrease.
7 Homepage without main
content (Homelesshomepage)
Homepage provide weak
navigation assistant into thewebsite.(user fixate on thehomepage, they are sure theinformation can be foundthere)
Increase Severity
8 Archiving Severity Decreased9 Problem of finding items from
a long listNew minor problem
8/7/2019 IPS Usability Testing Report - Report
40/56
40
Navigation
Overlapping Problems:
In Navigation, all problems predicted in Inspection were proved in Usability Testing.
Undistinguishable Links (ID: 1) was a reoccurring problem where users were confused
by the static text, images (including banners) and links. Users also encountered the problem
of Too much navigation (ID: 2) as predicted by the inspection. The users reaction to this
problem was by ignoring the links (even the useful ones). An example would be instead of
clicking the in-page quick link at the top of the page they instead scrolled down to find the
section they were looking for. No indication when leaving the website (#ID: 3), was yet
another problem that confused the users. Many expected to find the information within the
IPS website, however, were surprised when they had to navigate or find the window back to
it. Again, as proved by the inspection, Provide choices without decision information
(ID: 4), was encountered in the testing and continues to be a problem.
New Problems:
The Usability Testing also uncovered a new problem Navigation bar blocks
navigating within the topic (ID: 5). The navigation bar does not consistently including all
links within the topic.
8/7/2019 IPS Usability Testing Report - Report
41/56
41
ID Inspection Results Usability Results Comments1 Undistinguishable links Undistinguishable links
(text, & banner)Increase Severity
2 Too much navigation User missed navigationlinks
Severity Remain
3 No indication when leaving thewebsite Lack external websitealerts lead confusion Severity Remain
4 Provide choices without decisioninformation
School link with vaguedescription
Severity Increase
5 Technique problematicnavigation bar blocknavigating within thetopic
New major problem
Presentation
Overlapping Problems:
Users encountered the problem of Error Recovery (ID: 7). In the search feature,
they have difficulty in recovering from the error when the information they input cannot
meet the required format because there is no instruction telling them how to do. Regarding to
the problem of Mislabeled Pages (ID:8), users are confused by the phenomena that
several pages have the same label.
New Problems:
The Problem of Banner Blindness (ID: 1) was not predicted in the inspection,
however, due to issues uncovered in the usability testing the team classified this as a high
severity. Several users overlooked the navigation bar on the top, especially on the homepage,
which makes it difficult to gain access to the topics within the website. In contrast, the users
can notice the so call deemphasized menus on the right. The other problem The Search
Bar (ID: 10) is too small to be noticed. The third problem, Un-salient link (ID: 11), does
not show links in a consistent format (i.e. no visual cue the text is a link).
Inspection Problems:
8/7/2019 IPS Usability Testing Report - Report
42/56
42
The problem Users dont know where they are (ID: 2) and Mixed
representation of information and linked files (ID: 3) are another aspect of the
overlapping problem, Where am I and Undistinguishable link. Thus, the severity of
above problems should remain. Other problems, ID: 3, 5, 6, 9 were not discovered by users
in the usability testing, thus their severity was decreased.
ID Inspection Results Usability Results Conclusion1 Banner Blindness. High severity2 Users dont know where they are Severity remain3 Inconsistent font sizes, weights &
decoration (Hyperlinks)Severity Decreased
4 Mixed representation ofinformation and linked files
Severity Remain
5 Inconsistent design of sub-domains Severity Decreased
6 Deemphasizing position ofimportant menus
Severity Decreased
7 Error recovery Searching featureprovide not enoughinformation indicatinghow to recover wheninput informationdoesnt meet therequired format
Severity Remain
8 Mislabeled Pages Different pages have asame label
Severity Remain
9 Insufficient Information Severity Decreased10 The Search Bar is too
small to be noticedNew major problem
11 Un-salient links New minor problem
8/7/2019 IPS Usability Testing Report - Report
43/56
43
V. Overall Recommendations for Improvement
The teams recommendations span over four design dimensions; Content, Navigation,
Information Architecture and Presentation. Each dimension, ordered by severity as
discovered in our Usability Inspection, contains our recommendations ordered by severity
based upon our inspection and user feedback. Items highlighted in yellow were identified in
both our inspection and during user testing.
Content
In the content dimension, there are no problems overlapping Inspection and
Usability Testing. Regarding to the severity of the problem, we recommend the website
should supply a section proving the schools enrollment information (ID: 6) as users expected.
More over, they should also improve the Searching feature (ID: 4) by providing clear input
format requirement and error recovery help. Also, the Boundary map (ID: 5) should also be
improved to provide useful information, which is clear enough to read.
The problems with lower severity can also be improved if develop team has enough
time. The relevant recommendation and their reference please refer to table below (ID: 1, 2,
3).
ID Recommendation Severity Reference1 Establish a checklist and a content review board to
ensure information are updated and correct1 Inspection:Missing and
inconsistent information2 Notify the user of missing/ incomplete information
and when it will be updated1 Inspection:Missing and
inconsistent information3 Ensure dates are captured and identified on items
that have relevance around a date/ time1 Inspection:When was the
news new?4 Search feature: Provide a hint of the requirement to
input keyword for the search feature; for example,provide an example of the address in the formatrequired.
2 Testing:Unclear inputrequirement
5 Boundary map: Make sure the boundary mapprovides information that the users need.
2 Testing:Unclear BoundaryMap: The informationprovide on boundary map
8/7/2019 IPS Usability Testing Report - Report
44/56
44
doesn't meet the user'sneed. The boundary maponly provides a static mapof area around the schoolwithout any explanations,while the users want to
know the distance betweenschool and their home.
6 Enrollment Information: Clarify enrollment processby including the link in various subject areas toprovide enrollment information, including how toenroll, current enrollment status, resource of formsand etc.
3 Testing:Vague information:The users expect to knowthe information about howto enroll and the situationof current enrollment, whilethe current enrollmentinformation is a number ofhow many students enrollin this school
Navigation
The priority recommendation for Navigation (ID: 1, 3, 2, 4, 5) as they have a high
severity in inspection and caused great difficulty in users finishing their tasks in the test.
Regarding navigation links, make sure the overall navigation include links to each pages
within the topic. The website should also keep the links consistent and salient in appearance
and layout to make them easy to be notice and distinguishable from the text or static pictures.
Moreover, strengthen users awareness of boundary of the website to improve the confidence
of information by adding an external navigation alert.
It is also necessary to strengthen the users ability to control the navigation, by adding
a controller to the flash banner. (ID: 6)
ID Recommendation Severity Reference
1 The navigation bar to make sure the links cover allthe pages within the topic
3 Inspection, Testing:Technically problematicnavigation bar blocknavigating within the topic
2 Use proper formatting ensuring consistency betweentext and links, as well as banner picture and links
3 Inspection, Testing:Undistinguishable Links(text & banner)
3 Keep the most necessary navigation and get rid of 3 Inspection, Testing:many
8/7/2019 IPS Usability Testing Report - Report
45/56
45
the ones irrelevant to the page. Make sure theappearance and layout of the navigation are salientand consistent for easy noticed. Distinguish thenavigation of different level by differentpresentation.
navigations that usersignore one or two
4 Provide clear awareness of the boundary of the
website, by alerts when the user is navigated to theexternal website while keeping the graphical andnavigation consistency within the main website.
2 Inspection, Testing:No
indication of leaving thewebsite leads to the user'sconfusion as to where IPSended and new site started.
5 In the School List, provide more strategies forschool searching, such as filtering or arranging theschools according to specific school attribute.Provide more information or attributes to theschools (i.e. address, email fax, etc.) A possible wayto keep the list clear with rich information is toredesign the school list as a folding list like GoogleReader. Users can tab the school name to unfold
and show more information about it. A show allbutton is needed to easily unfold all the items.
2 Inspection, Testing:Providechoices without decisioninformation, and users haveproblem with the schoollink with vague description
6 Provide controller for navigating through theflashed banner: There should be some kind of wayfor the user to go back to a previously flashedbanner quickly. Links to each banner should bedisplayed below the banner, allowing the user tonavigate these asynchronously.
2 Testing:The uncontrollablerotation of the flashedbanner is difficult for usersto browse and make use of.
Information Architecture
Regarding that this is a deep and wide website with the severe problem of mismatch
information categorized between users and the website (ID: 1), however, it takes a great
effort to re-architecture the website, we recommend proving a sitemap as solution. The link
to sitemap should be in a salient position for easy access at any time, which will show the
overall website structure to the user to help them understand the conceptual model of the
website and easy locate the information they are seeking. The website should also provide
visual cues indicating users current location (ID: 2), breadcrumbs and highlighted labels are
possible solutions. The Homepage should also be improved to provide clear and overall
introduction to the website and helpful navigation assistance to the inner of the website (ID:
3).
8/7/2019 IPS Usability Testing Report - Report
46/56
46
ID Recommendation Severity Reference1 Provide a sitemap of the website to aid the user gain
an overview of the logical architecture of the website,so that they can find the information they need.
3 Testing:Mismatchinformation categorizationbetween users and thewebsite. (Usability Testing)
Users categorizeinformation according tothe closeness of their name,but the website groupaccording the departmentwhile the users are notfamiliar with theorganization ofdepartments.
2 Design should be include visual cues in thenavigation that highlights their current location.
Possible solutions are breadcrumb providing thetrack and navigate within the hierarchy. Ensure eachpage has a title matching the pages content.
3 Inspection, Testing:Where amI? Users are lost because
there are no clues of thecurrent location
3 Develop an area for main content to the displayed tothe user. This should summarize the site and whatthe site represents.
2 Inspection, Testing:Homepage without maincontent provides weaknavigation assistant into thewebsite.
4 Provide an access to the School Board meetingtimetable in the Parent Section.
1 Testing:Mismatchinformation categorizationbetween user and thewebsite.
Presentation
One of the most critical problems the team feels is important to fix is the problem of
Banner Blindness, especially on the Homepage (ID: 1). Possible solution is making the
navigation bar more salient and keeps it a distance away from the banner. Also, need to keep
the useful widget, including links and search button more salient in visual and layout aspect.
(ID: 2, 4). Clear error recovery instruction should also be provided (ID: 3).
ID Recommendation Severity Reference1 Make the navigation bar around the banner area
more salient. Possible solution is avoiding using the3 Testing:Banner Blindness
8/7/2019 IPS Usability Testing Report - Report
47/56
47
color similar to the banner for the background ofnavigation bar. Make the appearance of thenavigation bar more touchable like a button.
2 Make the links more visually salient. On the sametime, put them in a more outstanding position, suchas on the top of the page and avoid mixing within
the body of the content.
2 Inspection:Mixedrepresentation ofinformation and linked
files, Un-salient links3 Provide clear recovery instruction when the error
appears. Especially to the search feature, providemore options for user to choose to let themrecognize rather than recall. Provide formatrequirement and example near the input area.
2 Inspection, Testing:Errorrecovery & Users dontknow how to recover fromthe search error
4 Make the search button in the banner area moresalient in order to be noticed. Provide an input areato make it more affordable.
3 Testing:The search bar istoo small to be noticed.
8/7/2019 IPS Usability Testing Report - Report
48/56
48
VI. Appendixes
Appendix A: Pre-test Questionnaire
IPS Pre-Test Questionnaire
Name:
Date:
Age:
Gender:
Type of Computer:
Browser Used:
Location:
Race:
Type of User (Parent/Teacher/Community/Staff/etc):
Computers
How familiar are you with computers?
Very Somewhat Not At All
How often do you use a computer?
Very Somewhat Not At All
What do you mostly use the computer for?
Email SocialNetworks
Work School Browsing Other
School and Computers
Have you ever visited a school corporations website?
Yes No
8/7/2019 IPS Usability Testing Report - Report
49/56
49
If yes, was it IPSs?
Yes No N/A
If yes, how often in the past month?
1 3 Times 4 9Times
10-20Times
20-30Times
More than 30
Would you ever access the IPS site if you were/are not a parent, student, or staff member?
Yes No
8/7/2019 IPS Usability Testing Report - Report
50/56
50
Appendix B: Usability Test Script
Introduction
Hi, Im [Name]. Welcome and thank you for coming. How are you? Today we will be conducting anevaluation on the Indianapolis Public School (IPS) Corporation website.Im helping IPS understand how well their website works for the people who use the website. I willbe observing what you are doing today. In the evaluation you will have a chance to use the website tosee what you think of their product: what seems to work for you, what doesnt, and so on.
This evaluation should take about half an hour.
Were going to be videotaping the screen as a record what happens here today. This video is for ouranalysis only. Our conversation will be recorded as soundtrack to capture full information foranalysis. Its primarily so I dont have to scribble notes and can concentrate on talking to you. Themembers of our evaluation team will also review the video and audio. The information obtainedtoday is strictly for the purposes of this study and your information will be kept confidential.
Like I said, wed like you to help us with a product were evaluating. Its designed for people like you,
so wed really like to know what do you think about it and what works and doesnt work for you.You may run into features or functions that will not work right. Please feel free to tell us about yourexperiences as you go through the evaluation.
Procedure:The procedure were going to do today goes like this: were going to start out and talk for a fewminutes about how you use the web, what you like, what kinds of problems you run into, that sort ofthing.
Then Im going to show you a website of Indianapolis Public Schools which we are evaluating andhave you try out a couple of things with it. Then well wrap up, you will talk about your experience ofusing it and Ill ask you a few more questions about it, and were done.
Any questions?
Now I'd like to read you what's called a statement of informed consent. It's a standard thing I read toeveryone I interview. It sets out your rights as a person who is participating in this kind of research.
As a participant in this research
You may stop at any time.
You may ask questions at any time.
You may leave at any time.There is no deception involved.
Your answers are kept confidential.
Any questions before we begin?
Let's start!
8/7/2019 IPS Usability Testing Report - Report
51/56
51
Preliminary Interview
[Soundtrack recorder on]
-Online experience-1. How much time does you normally spent on the web in a given week (or within a day?)2. How much of that is for work use, and how much of that is for personal use?3. Other than email, is there any one thing you do the most online?4. Do you ever use educational system website? What do you usually visit that website for?
How often do you use it?5. Do you ever do research online for schools information?
-Offline Habits-1. (For parents) What kind ofinformation do you want to know about your childs education?
From where do you usually get this information?2. What kind of information do you want to know about the local schools or educational
system? How do you usually get it?
Evaluation Instructions
In a few minutes, I will ask you to begin your evaluation of the IPS website. However, before we getstarted, let's talk a little about what will be happening.
Please try to feel as comfortable as possible while using the interface. We are studying how thewebsite elicits your actions. There is nothing that you can do wrong as a user of this website. Wewant to observe your thoughts and reactions on the site's interface. With that, it is extremely helpfulto us if you could narrate your experience while using the site. We would love for you to have a nicestream of consciousness while interacting with the site. If you don't like something, let us know. Ifyou really, really like something, let us know. If anything stands out, doesn't stand out enough, or ifyou have an idea as to something that should be that isn't, your suggestions will help future users'experiences.
So, here is just a quick summary of what we discussed. We are testing the site; not you. Make sure tolet us know what you are thinking as you are thinking it. Finally, and most of all, be comfortable. Ifthere is anything we can do throughout the process to make your experience as natural as it would beat home, let us know.
Do you have any questions on the process? Does it all make good sense?
Great! Let's go to our Internet browser and go towww.ips.k12.in.us. As the site loads, please feel freeto move the chair, mouse, monitor, and keyboard to a comfortable position.
Tasks
8. As a parent you are growing concerned about the direction the IPS School board is movingtowards concerning year round school. Another parent mentioned that there are schoolboard meetings where you can go and voice your concerns. Assume today is December 1stand you want to go to the next briefing session. Please identify the date for this meeting.
http://www.ips.k12.in.us/http://www.ips.k12.in.us/http://www.ips.k12.in.us/http://www.ips.k12.in.us/8/7/2019 IPS Usability Testing Report - Report
52/56
52
9. As a parent you feel that your childs ability to learn and use technology is very important. Inthe midst of growing concerns about budget cuts you are interested in knowing how muchmoney IPS has allocated to implementing new technology. Find the budgeted amount IPShas allocated in its Technology Plan.
10. As a parent of a high school junior, another parent told you that IPS has resources for youand your child to start exploring colleges. Find contact information about Sawyer College inMerrillville.
11. As a parent you want to identify the schools that are nearest to your home where your childwould attend middle school. Your house is located at 816 N. Audubon Road, Indianapolis,IN 46219. Please name the closest high school to your house and find the enrollmentinformation for that school.
12. You noticed that a school located near your house was a magnet school. Please locate the2011-2012 English student application for this magnet school.
If you are a teacher, please complete the additional two tasks:
13. As a new teacher you had an issue with you pay check this week. You contacted the frontoffice at your school, however, because they do not directly deal with payroll they directedyou to the corporation website to find the appropriate contact information and the locationwere you needed to go to sort everything out. Identify the telephone number and locationof the payroll office
14. As a teacher you wanted to review updates to the 2010 teacher salary information specificallyAmendment 5233. Locate information about Amendment 5233.
Evaluation Wrap-up
Wonderful. Now, if you would, please, simply exit your Internet browser and we'll come together fora quick wrap-up session.
If you could describe this site in one or two sentences to a colleague of friend of yours, how wouldyou do so? What adjectives would you use?
Would you say, by the end of the process, that you were satisfied by the service provided by IPS?
When you first started the process, did you expect the site to contain what it did? More than it did?
Based on your experience with the website, can you give three general pros and three general consabout the site?
Is there anything that you wish this site would provide to people in your same shoes with regards tostake in the system (IPS)?
Thank you so much for your time and participation. Many times, after completing evaluations,people tend to have amazing ideas when they are at home, browsing the Internet and utilizingservices like these in their home environment. If you have anything you could add to this evaluationtoday, tomorrow, or even as far out as next week, please don't hesitate to contact us at
8/7/2019 IPS Usability Testing Report - Report
53/56
53
[email_address].
Thanks again and have a wonderful day!
8/7/2019 IPS Usability Testing Report - Report
54/56
54
Appendix C: Post-Task Questionnaire
IPS Post-Task Questionnaire
Evaluator: ________________ Participant ID: ________________ Task ID: ___________
I was satisfied with the ease of completing this task?
StronglyDisagree
Strongly Agree
8/7/2019 IPS Usability Testing Report - Report
55/56
55
Appendix D: Post-Test Questionnaire
IPS Post-Test Questionnaire
Evaluator: ___________________ Participant ID:___________________
1. How would you explain the set up of the site; Architecture?
2. How would you explain how you are feeling after completing the tasks?
3. Do you feel like you were In Control of navigating around the site?
4. What would you change about the site?
5. Could you understand where you were in relation to the sites structure?
6. What would you change about the site?
7. Is there anything youd like to add?
8/7/2019 IPS Usability Testing Report - Report
56/56
Appendix E: Usability Testing Videos
Please reference the attached CD File submitted with this project for a complete list of videos.