Validation of a Usability Evaluation Protocol based on
the Heuristic Inspection Method: An Experimental Case
Study in the Web Domain
Freddy Paz1, José Antonio Pow-Sang1, César Collazos2
1 Pontificia Universidad Católica del Perú, San Miguel, Lima 32, Perú
[email protected], [email protected] 2 Universidad del Cauca, Popayán, Colombia
Abstract. The usability is nowadays an aspect that can determine the success of
a software product. For this reason, there are methods that allow specialists to
verify if the properties of the system interfaces contribute to achieving the
desired degree of usability. Although the heuristic evaluation is one of the most
recognized methods, there is not a formal process to execute this technique. The
original author only proposes general recommendations. Based on the analysis
of different studies that report the use of this method, we established a new
usability evaluation process that can be used for any professional in Computer
Sciences. This work is focused on the validation of our protocol to conduct
heuristic evaluations. For this purpose, a case study was conducted with the
participation of twenty postgraduate students. The analysis establishes that the
results are more accurate when a formal process is used.
Keywords: Human-Computer Interaction, heuristic evaluation, usability,
experimental case study, assessment protocol.
1 Introduction
Nowadays, one of the most important quality attributes of software applications is the
usability [1]. Given the current market in which there are several alternatives for a
specific type of software, if the system is hard to use, the users will look for another
application that let them to achieve their purpose easily [2]. This fact is even more
relevant for e-Commerce Web sites, where the distance to the competition is as short
that can be located by a simple search on Internet. Companies are progressively
adopting a Web model to reach more customers through the new information and
communication technologies (ICTs). In this scenario, the usability represents a crucial
role that can determine the user’s decision to use certain Web sites instead of others.
The relevance of usability has led to the development of several methods that allow
specialists to evaluate this quality aspect of any technological interface [3]. The
heuristic evaluation is one of the most widely used methods in the context of software
[4]. This recognized technique allows the identification of a high number of usability
problems by the use of minimal resources [5].
Advanced Science and Technology Letters Vol.142 (UNESST 2016), pp.63-68
http://dx.doi.org/10.14257/astl.2016.142.11
ISSN: 2287-1233 ASTL Copyright © 2016 SERSC
Although the heuristic evaluation is a common activity performed by the software
developers that follow a user-centered design, there is no agreement between the
authors on how to carry out this assessment process. The author of the technique only
establishes general recommendations about the procedure [10]. However, there is not
a specific protocol with well-defined activities that can guide step by step the usability
evaluation. In this work, we present the validation of a formal framework to conduct
heuristic evaluations in the context of software products. This usability evaluation
protocol was developed with basis on a previous systematic literature review [6].
2 Usability and Heuristic Evaluation
A general definition of usability is proposed by the standard ISO 9241-11 [7], which
establishes that this feature is related to “the extent to which a product can be used by
specific users to achieve specific goals with effectiveness, efficiency, and satisfaction
in a specified context of use”. This standard establishes the interface must be intuitive
in a way that allows users through the use of the system, to reach the desired goal.
In order to verify if a software system meets an appropriate degree of usability,
several methods were developed. These evaluation techniques allow specialists to
determine systematically if the properties of the system interface contribute to achieve
the desired usability in the software product [8]. The heuristic evaluation is a usability
inspection method that is widely used during the software development process. This
technique involves having a group of specialist who examines the interfaces of the
system to conclude if they were properly designed [5]. This inspection is guided by a
set of principles called “heuristics”. The evaluators verify if there are infringements to
these guidelines, and the results are cataloged as usability problems.
2.1 Usability Evaluation Protocol
The author of the heuristic evaluation method only provides general recommendations
to carry out this technique [10]. For this reason, we proposed a formal process based
on a previous work [6], which involved the conduction of a systematic mapping
review of the usability evaluation methods that were used during the last three years
to develop software products. From these results, only those studies that reported the
use of the heuristic evaluation method were analyzed. According to the papers, the
process can be divided into five phases. This proposal also considers the framework
established by Hurtado et al. [9] for usability inspections:
1. Planning: In this phase, the evaluation manager must choose the specialists
that will examine the system interfaces. This person as well has to decide the number
of evaluators and the profile of the professionals that will participate in the inspection.
Finally, the reviewers must be recruited by these criteria.
2. Training: The phase of training is optional depending on the experience of
the reviewers. If the group of evaluators has performed few usability inspections, it
would be necessary to train them on the basic concepts of the method.
3. Evaluation: After the evaluators have defined the set of heuristics that will
be employed, they proceed to examine the system interfaces individually to identify a
Advanced Science and Technology Letters Vol.142 (UNESST 2016)
64 Copyright © 2016 SERSC
list of problems. During this time, the purpose is to determine infringements to the
principles. These issues will be cataloged as usability problems.
4. Discussion: Once the individual lists are identified, the inspection team has
to reach a consensus and prepare a single list of usability problems. The team must
organize a meeting, in which, each identified issue must be discussed.
The process of discussion to determine a single list of usability problems.
5. Report: Once the evaluators have established the single list, they have to rate
the severity and frequency of each problem individually. For this activity, the
team can use the ranking proposed by Nielsen [2]. Subsequently, the team
4.1. The moderator verbalizes the usability issue.
4.2. The evaluator who identified the issue provides more details about the
problem, even with the help of screenshots.
4.3. The team will discuss if the issue is indeed a usability problem. For this, each
evaluator will give an opinion about the identified problem.
If the group agrees that it is a usability problem then,
4.3.1. The team has to determine if the issue was identified by more than
one evaluator.
If the issue was identified by more than evaluator then,
4.3.1.1. The evaluators who identified the issue must discuss to
determine if they have indeed identified the same incidence or if
they are doing reference to different aspects.
If they are doing reference to the same issue then,
4.3.1.1.1. They must collaboratively prepare a definition and
description of the issue.
Else
4.3.1.1.2. The new incidence should be classified as
usability problem to discuss afterwards it.
4.3.1.2. The team has to determine if the issue, in the way it was
written, express indeed the incidence to which it is doing reference.
If the problem is written properly then
4.3.1.2.1. The problem is included in the final list.
Else
4.3.1.2.2. The team collaboratively must prepare a
definition and description of the identified problem.
Else
4.3.1.3. Go to step 4.3.1.2.
Else
4.3.2. The evaluator who identified the issue should decide after having
heard the different opinions.
If the evaluator persists that it is a usability issue then,
4.3.2.1. Go to step 4.3.1.2.
Else
4.3.2.2. The problem is discarded.
Advanced Science and Technology Letters Vol.142 (UNESST 2016)
Copyright © 2016 SERSC 65
must calculate the average of this rates. Finally, the team must prepare a report
in which they should include possible solutions to each identified problem.
3 Experimental Case Study
The validation of the protocol was performed with the participation of twenty post-
graduate students from Pontifical Catholic University of Peru (PUCP), specifically
from the Master Program of Informatics Engineering. They were selected from a
mandatory course of Human-Computer Interaction. In order to obtain the validation of
the new formal protocol, the students were requested to participate as evaluators in the
execution of a heuristic inspection. Given that one of the goals of the course was the
instruction of this method, they were trained in the process. For this activity, they
were divided into four groups of five students. Two groups were trained according to
the original recommendations proposed by Nielsen [10]. The other two groups were
trained with basis on our new evaluation approach. Two e-commerce Web sites were
selected: latam.com (Web site of an airline) and aliexpress.com (website of a retail
store). Teams A and C performed the heuristic inspection of this Web application
employing the ten usability principles for user interface design [11]. In contrast,
groups B and D organized a meeting to determine the set of usability that they should
use for the evaluation. They concluded that an appropriate set of heuristics was the
fifteen principles proposed by Paz et al. [12] for transactional Web sites. The
distribution for this experimental case study is presented in Table 1.
Table 1. Instrument-Subject Distribution for the Heuristic Evaluations
Evaluation
Team
Web Site that was
assigned Evaluation Process Usability Heuristics
Group A www.latam.com Nielsen’s traditional
approach Nielsen’s heuristics
Group B www.latam.com New Evaluation Protocol Paz’s heuristics
Group C www.aliexpress.com Nielsen’s traditional
approach Nielsen’s heuristics
Group D www.aliexpress.com New Evaluation Protocol Paz’s heuristics
4 Analysis of the Results
Both approaches were compared by the number of usability issues that were identified
in each scenario. The results are presented in Table 2, and establish that when our new
proposal is used to carry out a heuristic evaluation, more issues are detected. In
addition, the severity of the problems is higher, considering that this value varies
between 0 and 4, where the maximum value of four represents a catastrophic problem.
The protocol allows specialists to conduct the assessment in a more structured way, in
which all findings are discussed to the obtain accurate results. The establishment of
the proper heuristics is another factor that can modify the outcomes of the inspection.
Advanced Science and Technology Letters Vol.142 (UNESST 2016)
66 Copyright © 2016 SERSC
The evaluation process has been in a way that it can be used by any professional in
the field of the Computer Science. Table 3 and Table 4 show five of the most severe
problems that were only identified by those teams that used our approach. The results
establish that these issues are relevant and should not be overlooked in a usability
evaluation. However, when an evaluator only follows general recommendations to
perform the assessment, these features are not considered. The quality of the results
increases if a formal protocol is employed to carry out this type of inspection.
Table 2. Results of the heuristic evaluations
Team Number of identified problems Average of Severity of the Problems
Group A 22 2.37
Group B 37 2.65
Group C 17 2.38
Group D 28 2.80
Table 3. Five of the most severe problems that were only identified by the evaluation team B
that used our new approach
ID Problem Definition Severity
P17 The system displays a blank screen when the user logs off. 3.25
P26 The button that allows users to return to the previous step during the
payment process with VISA cancels the entire procedure. 3.25
P13 There is no help system to understand the interface design. 3.00
P35 There is unwell written information. 3.00
P19 The system does not support different Web browsers. 3.00
Table 4. Five of the most severe problems that were only identified by the evaluation team D
that used our new approach
ID Problem Definition Severity
P04 The option to log out is not intuitive. 3.3
P02 The help system displays the information only in English. 2.8
P03 There is not an option to display the prices in the local currency. 2.3
P04 The system interfaces have a different design. 2.0
P10 Many options of the system force the user to register. 2.0
5 Conclusions and Future Works
The heuristic inspection is a recognized method to evaluate the usability of software
products. However, there is not a formal procedure that allows specialists to execute
this technique in a structured way. For this reason, we have proposed a protocol that
can be used by software developers and professionals in the field of Computer
Science to conduct a usability evaluation based on this technique. In this work, we
present the validation of a new process to measure the degree of usability of software
Advanced Science and Technology Letters Vol.142 (UNESST 2016)
Copyright © 2016 SERSC 67
systems. For this purpose, we requested the participation of postgraduate students
from the Master Program of Informatics Engineering of the Pontifical Catholic
University of Peru, who voluntarily agreed to contribute with this research. The
results establish that more usability problems can be identified when a particular
procedure is used. The issues that were detected by the teams who employed the new
approach are relevant. These findings may be due to the fact new usability principles
were selected to carry out the evaluation. After a meeting, the teams agreed that the
Nielsen’s ten traditional heuristics fail to cover new aspects of the current e-
commerce Web sites such as real-time processing and sophisticated designs. For this
reason, they chose to use another heuristic proposal. Although the results are further
accurate when a formal protocol is followed, it is still necessary to perform more case
studies to demonstrate its usefulness and applicability in other contexts and scenarios.
References
1. Dhouib, A., Trabelsi, A., Kolski, C., Neji, M.: A classification and comparison of usability
evaluation methods for interactive adaptive systems. In: Proceedings of the 9th
International Conference on Human System Interactions (HSI 2016), pp. 246-251,
Portsmouth, United Kingdom (2016)
2. Nielsen, J.: Usability Engineering. Academic Press, San Diego, CA, USA (1993)
3. Paz, F., Pow-Sang, J.A.: Current Trends in Usability Evaluation Methods: A Systematic
Review. In: Proceedings of the 7th International Conference on Advanced Software
Engineering and Its Applications (ASEA 2014), pp. 11-15, Hainan, China (2014)
4. Paz, F., Villanueva, D., Rusu, C., Roncagliolo, S., Pow-Sang, J.A.: Experimental
Evaluation of Usability Heuristics. In: Proceedings of the 10th International Conference on
Information Technology: New Generations (ITNG 2013), pp. 119-126, Las Vegas, NV,
USA (2013)
5. Nielsen, J.: Usability inspection methods. In: Proceedings of the Conference Companion
on Human Factors in Computing Systems (CHI '94), pp. 413-414, Boston, Massachusetts,
USA (1994)
6. Paz, F., Pow-Sang, J.A.: A Systematic Mapping Review of Usability Evaluation Methods
for Software Development Process. International Journal of Software Engineering and Its
Applications. 10(1), 165 - 178 (2016)
7. ISO: Ergonomic requirements for office work with visual display terminals (VDTs). ISO
IEC 9241-11, International Organization for Standardization, Geneva, Switzerland (1998)
8. Fernandez, A., Abrahão, S., Insfran, E., Matera, M.: Further analysis on the validation of a
usability inspection method for model-driven web development. In: Proceedings of the 6th
International Symposium on Empirical Software Engineering and Measurement (ESEM
2012), pp. 153-156, Lund, Sweden (2012)
9. Hurtado, N., Ruiz, M., Orta, E., Torres, J.: Using simulation to aid decision making in
managing the usability evaluation process. Information and Software Technology 57, 509-
526 (2015)
10. Nielsen, J.: How to conduct a heuristic evaluation, (1995)
https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
11. Nielsen, J.: 10 Usability Heuristics for User Interface Design, (1995)
https://www.nngroup.com/articles/ten-usability-heuristics/
12. Paz, F., Paz, F. A., Pow-Sang, J.A., Collantes, L.: Usability Heuristics for Transactional
Web Sites. In: Proceedings of the 11th International Conference on Information
Technology: New Generations (ITNG 2014), pp. 627-628, Las Vegas, NV, USA (2014)
Advanced Science and Technology Letters Vol.142 (UNESST 2016)
68 Copyright © 2016 SERSC