Post on 13-Apr-2017
transcript
ABSTRACT
Mayank chadha[Email address]
The aims of this project to investigate and record usability testing of a mobile application that uses dimensions of context to receive electronic updates from the university’s virtual learning environment. Previously,a user study of two RSS based information retrieval widgets to support mobile learning within a higher education environment was implemented and results of which were published in [1]. Created and deployed for implementation on a GPS enabled Nokia devices, the user is presented with arbitrary information from the virtual learning environment based upon either their schedule or location. The study aimed to identify whether integrating location context awareness can override the utility of temporal based mobile electronic alerts. The major function of both the mobile applications is to disseminate information surrounding course updates and deliver these in a context of either time or space, aiming to identify if there is precedence between these two dimensions of context. Results from both quantitative and qualitative data indicated that both applications were evenly helpful for receiving information and supported the way in which the students organized their learning. The other outcome of interest was that both the space and time applications were deemed equally unobtrusive in their personal space, although utilizing location information was deemed a less comfortable form of ambient information.Presently, a third dimension of context (activity) was included and further usability testing is required.
The aim of this project is to carry out a study with particular focus on usability testing of the mobile application developed
Introduction
At this present moment in time mobile devices are the extreme source of delivering information
to its users rather it be simply browsing the internet for multimedia and social networking or for
navigation from one route to the other. The abundance in mobile technologies in the today’s
world has let the user to explore vast amount of facilities from news to shopping, from
entertainment to your important email updates, at fingertips. Coupled with the wireless
connectivity, mobile devices can provide some powerful applications to its user. And the nature
of it being portable and on the move, has the ability of outweighing the capability of present
computer systems. The use of mobile devices for delivering learning tools is an attractive
concept. Termed as mobile learning (m-learning), this new technology allows people to
participate in learning activities without being tied to a fixed location and provides users with
convenient and flexible access to learning resources ‘’anytime, anywhere’’[1].
With the invention of GPS in present day mobile systems, a user’s location can be set as update
points. Context-aware dimensions such as location, schedule and activity mobile application can
now offer automated updates to the users which in case of students could help in their learning
environment. M-learning is fast growing area where most of the learning institutions already
have implied and integrated their learning environment to mobile based platforms. These m-
learning sources can increase the effectiveness and quality of learning in a student friendly space.
In order to provide a better learning source in mobile education the applications have to be
highly focused on collaborative, communicative, constructive and supportive activities which
can enhance a students learning experience. Hence the Usability of the application plays a really
crucial role. By usability of an application means the effectiveness and providing a high quality
system to the learners. The modes and methods on how to test if a system is usable or not will be
explained further in details in coming topics.
1
1.1 Project Aims
The aims of the project is to investigate the usability of context-aware mobile application in the
university’s learning environment. Previously, a study was made using the temporal and spatial
dimensions of context where the users were to receive automated RSS updates over their mobile
devices depending on the location and schedule selected by the user. Recently a third dimension
Activity has been introduced and further usability testing is required.
1.2 Report Structure
The report will be structured in following way:
Chapter 2 will include all the explanations of terminologies used in the development of the
application all the background research and the systems which were related to the development
process in some ways.
Chapter 3 will include the design and implementation of the procedure being followed in the
usability testing of the application and will give an insight to area to look at while performing the
test.
Chapter 4 will include the evaluation process and usability testing results of the study. This will
also include the insights to the feedback from the test users from this study.
Chapter 5 will demonstrate how it is like to use the system from a user’s point of view. This will
include the screenshots of the application working on an Android phone.
Chapter 6 will evaluate the overall findings, conclusions, scope of improvements and will
discuss if the end results meets the aims of the project along with summed up outcomes of the
project.
2
2. Background & Related Work
In this part of the report I will focusing on the previous research been done and all the existing
technologies which are relevant to the project in some ways.
2.1 Enabling Technologies
2.1.1 Android operating system
Android operating system is a mobile operating system that makes the use of java based
programming language knows as Android programming language in the application
development. It is fairly new and one of the top leading mobile operating system in the market
till date. It was first distributed in 2007 and is led by Google Inc. According to Canalys (a
leading business analysis company), Android was the bestselling mobile operating system
platform in the last quarter of 2010 with over 200 million Android devises in use by November
2011 [5]. A research shows that in second quarter of 2012, 105 million units of Android based
smartphones were sold, capturing the 68% of overall market shares till Q2 2012 [6]. Within in
the Android community there are Android software developer which are working on the
development of mobile applications in order to extend the functionality of the devices. Being an
open source programming language (meaning you can actually see the coding bits of the
programs) it serves as the best platform for application developers to add the newly custom made
features to an already available mobile application and tailor them according to their desired
needs. The main advantage of the android programming is that it is free to use, ie you don’t need
to purchase an authority license to build a mobile application. Keeping this in mind and the fact
that how popular it is amongst students, the m-learning application has been designed on
Android operating system and the features, and various other artefacts associated with the
working, usability will be discussed in the upcoming sections.
3
2.1.2 Eclipse IDE
Eclipse is a multilingual Integrated Development Environment (IDE) comprising of a base
workspace and extensible plugin systems to customizing the environment. It is basically written
in java and has a well-established support for programming in Android. It features the drag and
drop GUI’s and is able to use all the packages provided in Android SDK. Eclipse allows it
developers to create a virtual GUI so that they can work and test their applications on desired
screen sizes and variable memory just like any Android smartphone without actually having to
transfer the application to a mobile phone.
Again, Eclipse IDE is an open source project which is run by the community Eclipse foundation
that ‘develop open platforms and products’. Primarily it is a java based platform but with the use
of various optional plug-ins the use can be extended to other programming languages such as
C/C++, Fortran, PHP, COBOL, JavaScript, Python etc [7].
2.1.3 Android SDK
Android Software Development Kit (Android SDK) comprises of a set of tools which are needed
in the development of the application. It includes an emulator, libraries, debugger,
documentation, sample source codes and tutorials. Android SDK is the main source of building
any kind of Android application [8].
2.2 Context Awareness
The understanding and definition of context awareness has been a research field spanning
decades, but has its potential realized with the arrival of mobile devices. The term context-
awareness has commonly been used for two different kinds of application approaches: to capture
context so that it can later be used as a cue for information retrieval or more commonly, to user
context to adapt device behavior to correspond to the manner of its usage (Dourish 2004) [9]. In
addition to these two cases, context-awareness can be used for providing information
4
presentation to its user (Dey 2001) [10]. There haven’t been one specific definition to describe
the term but some attempts are presented.
Schilit et al. (1994) [11] introduced the term context-aware computing which
‘adapts according to its location of use, the collection of nearby people and objects, as well as
changes to those objects over time’.
In addition to this, they proposed three general categories for describing the context, user
context, physical context & computing context. The user context here describes the general
situation from the user’s point of view such as user’s activity and social factors. Physical context
comprises information of the physical environment. Computing context includes the use of
available computing & application resources such as device connectivity.
As stated by Dey and Abowd (2000) [12],
‘Any information that can be used to characterize the situation of an entity. An
Entity is a person, place, or object that is considered relevant to the interaction
between a user and an application, including the user and applications
themselves’.
In regard to this they state that a context-aware may support three kinds of features: 1)
presentation of services and information to its users, 2) automatic execution of a service and, 3)
tagging of context to information for its later retrieval. Thus, according to the selected definition
of context one needs to figure out on how to choose relevant factors, in order to make sure that
they are taken into consideration in the application development process.
Brown et al [13] described the concept of context as ‘the formal or informal setting in which a
situation occurs’, it can include many aspects or dimensions, such as environment, user goals,
tasks and activities. The diversity in the meaning of context can be summarized in Table 1.
5
2.2.1 Using Context-Awareness in Mobile Applications
The use of context-awareness is not only limited to mobile applications, but has a scope and
many other kind of applications, for instance recording of the audio and videos in the lecture
rooms (Abowd 1999) [14], Keeping track of the patients in hospital beds (Bardram 2004) [15].
However in scope of this project the discussion of this is limited and is more focused towards the
use in mobile applications.
Location being the highly usable context-awareness source has its use in many different kind of
mobile applications. With just the user’s point of location one can find really useful information
in sense of time management and social navigation. Location awareness forms a significant area
for research studies as it has a huge potential to offer to the present generation of mobile users.
Discount offers nearby shopping/eating places depending upon the user’s present location,
navigation aid systems, city/museum tour guide systems, lunch menu choices in nearby
6
restaurants, store opening hours and various location based advertising systems works on the use
of location based contexts in mobile application systems.
Apart from Location based context awareness systems, time based context awareness mobile
applications also provides a big range of facilities according to the user’s time domain
information. The user’s time zone, the device time the user is using is used to trigger temporal
based features of the mobile applications. Some of the common examples would be cinema time
applications which tell its users the next available cinema timings depending on their current
device/time zones. Another simple example would be a shopping store official mobile
application which tells it users whether the shop is open/closed depending on the time they
switch to the mobile application.
There are several kinds of mobile sensors within the mobile phones which allow the application
to make use of the user’s spatial and temporal sensitive information in order to deliver these
information. The mobile application will only perform correctly if it has the access to use the
device’s application programming interfaces. The permission is demanded by the application to
make use of the device API which is only asked once. An example of this is shown below with
the help of a mobile application screenshot for asking its user to gain full access to the device’s
API [4].
7
Ofcourse, this permission is asked only once during the installation of the mobile application.
After this the application gains the full access to user and device’s information in order to
maintain a seamless information flow to and fro from the server to its user.
Figure 1 shows the available sensors in a modern mobile device which could be used to
determine the user context.
8
One of our own university’s mobile application for instance, Pocket Lancaster is a perfect
example of how this process works. This mobile application uses both the time based and
location based context-awareness features in order to tell its users the upcoming events in or
around the campus. It also features the nearest point of interest places whether it be a coffee shop
or a lecture room locations based on the user’s current location. Some working screenshots of the
mobile application are shown below to make it more clear how this mobile application uses the
both features within the same module [16].
9
10
2.3 What is Usability?
Usability is an inseparable part and a building block of any type of application development
process. In the hoard of tightening market competition, application developers and competing
amongst each other developing similar featuring applications. How to choose which one amongst
all is better? Usability is the key to provide a shear strategy to outstand one application amongst
others. Usability reflects to the easiness and efficiency of use in order to provide full user
satisfaction with application.
According to ISO 13404 on Human-Centered Design Process for Interactive Systems defines the
Usability as:
‘Extent to which a product can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use’.
Thus an attempt of employing the usability criteria with the use of context awareness is a
relevant approach. Figure 1 presents the usability framework described by ISO/DIS 9241-11,
which describes the relation between usability measures and goals.
11
2.3.1 Why do we need it?
According to Nielson (1994), Usability can be best defined with the five attributes Learnability,
Efficiency, Memorability, Errors, and Satisfaction.
Learnability here emphasizes on the ease of learning of the various aspects the application has to
offer to its user. The system should be easy to learn so that the user can rapidly start learning the
system. Efficiency relates to the system being efficient to use, so that once the user has learnt the
system a high level of productivity can accumulated in less time. Memorability signifies that the
system should be easy to remember, so that the casual user is able to return to the system after
some period of not having used it, without having to learn everything all over again. Errors
12
relates to the system having a low error rate, so that the user makes few errors while using the
system. Even though if the users do end up making some errors, they could easily recover from
them and be sure to not make those errors once again. Satisfaction, being the highly prioritized
one signifies that the user is completely satisfied with the use and ease of the system. In other
words the system should be pleasant to use.
Preece et al (2002) [17] sets a good illustration of the usability goals and user experience goal
putting up together.
13
2.3.2 Usability in M-Learning
A learning environment will be accepted or not from the users, according to its practical
acceptability and usefulness that include its usability and utilization [18]. The usability of a
learning environment is an important attribute as it influences the students by providing them a
good use of learning space available by making full use of it. In order to enhance the students
learning approaches mobile learning environment plays a big role in modern day mobile
generation. Most of the mobile phones in the present time are capable of providing support for
vast amount of application and providing learning through the mobile devices is indeed an
interesting concept. The integration of mobile devices with learning applications provides a
better way of learning for the students by removing the limitations of time and place in learning
activities which increases the student’s interest in learning. Being able to see your lecture notes
on the go over a mobile phone, being able to be up to date with all the classroom announcements,
discussions over the mobile application enhances the student’s experience in learning and the
usability of such kinds of applications provides a higher user satisfaction level and helps them to
remember content of their learning environment by minimizing errors which eventually leads to
an efficient and effective way of learning.
There are some potential benefits of usable learning environment [19] 1) More efficient user
tasks leads to better and more effective mode of learning activities, 2) More effective learning
activities leads to better understanding of the concept and successful learning, 3) Increased
satisfaction level of the students with the environment leads to a better way of learning once
students starts to enjoy the interaction with the environment, 4) Increased satisfaction and
effectiveness leads to the reuse of the learning environment, 5) Reduced errors enhances the user
satisfaction and their interest towards the education and hence provides even better mode of
effectiveness. 6) Higher satisfaction with the learning environment encourages the student
towards learning.
14
3 Design &Implementation
In this section we will be discussing the designed nature of the test, implementation procedure
and the process I would be following for the usability testing of the mobile application developed
for the study. Testing the mobile application on the real world users is the best way to test the
usability of any kind of application. It gives a clear cut view on, how other users for whom the
mobile application is actually have been developed for, finds the system to use.
Several questions are raised in the process of developing any type of application. As the
application is based for all types of individual users irrespective of the fact that they have past
experience of using such type of system or not. 1) How to make an easy to use, a simple straight
forward approach, and more useful and productive type of mobile learning environment? 2) How
to provide users with the experience that fits according to their style of learning and adapts itself
in a way the user want it to be? 3) How to make the mobile learning environment that could fit
according the user’s background knowledge and objectives? 4) How to improve the overall
usability of the system? This goes through the insight of the usability testing approach I
followed.
Figure 1 shows the framework followed for the usability testing process.
15
3.1 Testing Strategy
The general aim for the testing part was to conduct the user testing on a set of users, where users
are a mixed group of students from the university within different courses. The testing strategy
was to aim a specific mix of students where their past experience with the mobile devices and
using the different sort of mobile applications had no part to play for the test. The test was to
focus highly on the qualitative analysis of the application rather than the quantitative analysis
due to the nature of the mobile application. The mobile application was a pretty straight forward
functioning app where the users were given RSS based alerts depending upon the student choice
of context-awareness attribute. Also as suggested by Nielson (1994) ‘think aloud protocol’ might
not work always, as it greatly depends on the testing goals and the nature of application to be
16
tested. The quantitative results may not be always relevant and can be misleading because of the
individual differences amongst the individuals. Quantitative studies are based on statistical
analysis that as Nielson states are often false, biased, misleading or overly narrow. Better to
emphasize on the insights of qualitative research. A group of users with various backgrounds
could offer a qualitative feedback in different approaches which improves the scope of
improvement could be done in the system. Even though quantitative analysis could be
misguiding I did not want to completely ignore this aspect in my project. Prior to the testing I
evaluated the quantitative approach towards the basic functionality of the mobile application, and
results of both quantitative and qualitative criteria will be discussed in the upcoming sections.
3.2 Test Setup
The testing procedure was initially aimed to be performed with 30 students within the university,
but due to the lack of people who volunteered for the testing and the time available I have
evaluated all the testing results on 12 users. The test users for this study were all the students of
Lancaster University, studying different disciplines and variable degree year. The users consisted
of 12 users where 7 were males and 5 were females. Not surprising, all 12 of the 12 users had
some past experience with the mobile devices and applications. 8 out of 12 students had known
about some forms of mobile learning including VLE and iLancaster. The test will be conducted
in a laboratory setup where the users will be given an Android supporting smartphone with the
preinstalled applications designed for this project. Prior to the beginning of the test, users will be
given some certain set of instruction stating the nature, purpose and main goal of the usability
test. During the test the users will be asked to perform certain tasks which in turn will make them
familiar with the application, how it works and all the available features it could avail to the
students. Once the test has finished the users will be given a couple of questionnaires, one asking
them about the satisfaction with the nature of this test and the other one regarding their feedback
on the application, stating their satisfaction level with the use of application, and describing some
positive or negative aspects about the mobile application. The test will generally be about for 20
minutes including the preparation, instructing, conducting the test itself and gathering the
17
following feedback questionnaire. Which will then later be considered for the usability test
evaluation purposes stating different aspects of the mobile application.
3.3 Laboratory test
The test was conducted in the Engineering Building, here at Lancaster University around the
normal hours of the evening after the college hours, it was made sure that none of the users were
in a hurry, or were getting late for something, or were unintentionally there. The test was
arranged at the agreed time and the day available according to the free time of the students. The
users were given 3 Android based Samsung Galaxy S3 phones running the current android
version of 4.2.2 along with a detailed written set of instructions stating that the aim of this test is
to check the usability of the mobile application designed for the m-learning purposes. The users
were told specifically that I haven’t designed this mobile application, so they are free to share
their thoughts about the application, its use, the scope in improvements, and the graphical user
interface of the system. No camera recording systems were used as I didn’t think it was
necessary according to the nature of application being tested. Also, users were allowed to work
on ‘Think aloud protocol’, ie, they could share their gestures in words if they need to. The main
aim for this test was to test their reaction towards the concept of the mobile application and what
it can actually offer them in their study life style.
Following are the tasks which were asked to perform during the test:
1) Switch off the mobile phone.
2) Switch on the mobile phone.
3) Press the Menu button.
4) Navigate through the Menu.
5) Locate the mobile application.
6) Run the Mobile Application by clicking on the Icon.
18
7) Explore through it.
8) Locate the settings menu.
9) Navigate through the settings menu.
10) Explore through it by switching the available setting buttons (on/off).
11) Exit the application.
Once the user has finished all these listed tasks. They were asked to fill the testing procedure
evaluation questionnaire (ASQ). The After Scenario Questionnaire (ASQ) is 3 item questionnaire
which was designed by the IBM to determine the user satisfaction after the completion of the
task. It basically consists of the user satisfaction level from the completion of the tasks they were
asked to perform, the time consumed in the task was appropriate and the support information
provided for the tasks. A screenshot below shows the outlook of the ASQ.
It is designed using a 7 point scale, where 1 denotes the strongly agree and 7 denotes strongly
disagree. 25 & 26 describe the psychometric evaluation can be summed up to a one scale
denoting the overall satisfaction score of the task.
19
An overall evaluation questionnaire was also performed and evaluated after the ASQ, consisted
of application quality and performance based questions. The Computer System Usability
Questionnaire (CSUQ) also developed by IBM consisted of 19 questions which assesses the user
satisfaction on the overall system usability. It was divided into 3 subscales, questions 1 to 8 were
classified to figure out the System Usefulness, questions 9 to 15 were scaled to figure out the
Information Quality of the overall system, and questions 16 to 18 were then scaled to the
Interface Quality. The last question in the CSUQ was aimed for the overall satisfaction. Similar
to the ASQ, CSUQ was based on a 7 point scale with, 1 being the strongly agree and 7 being the
strongly disagree. Additional comments were also designed in order to point out any thoughts
user has to share about any aspect of the mobile application. In the end user were asked to state
some positive and some negative features they think could be improved also helped the
evaluation of the project. A screenshot of the CSUQ questionnaire is also featured on the next
page of the report.
Figure 1. Computer System Usability Questionnaire used for the study
20
4 The System in Operation
This section will feature how the mobile application works from a user’s point of view. As this
project aims at how these context aware mobile application could help a user in their learning
style, the coming parts will demonstrate how the mobile application could be beneficial for a
student to be up to date with their VLE environment.
Figure 1(a) shows the main user interface for the RSS based mobile application where all the
new uploads to the Moodle VLE are displayed. New lecture notes, coursework uploads, any new
added discussions appears in a form of a thread as shown in figure 1(b).
Figure 1(a) & 1(b)
21
The updates are displayed in a descending order of the dates they have been added. The newest
upload will be displayed on the top. These added updates can be further scrolled up or down if a
user wants to look for some specific added content from a past date.
Once the user clicks on the content of their choice, the thread opens in a new tab displaying the
detailed content with the upload URL where the content have been added. Every time an update
is added to the system the user receives a pop-up dialogue saying that there is an update since
you checked the mobile device. When the user clicks on the pop up feature it then redirects the
user to the newly added content displaying the detailed view of the topic along with the URL
address. This feature is showed in figure 2(a) and 2(b).
22
Figure 2(a) & 2(b)
Figure 3(a) and 3(b) displayed below are the layouts of setting menu screen of the activity based
context-aware mobile application and temporal based context-aware mobile application
respectively. The setting option can be found on the top right side corner of the menu screen
denoted by the setting icon in the menu screen and the icon on the top left side corner of the
screen is for the main RSS feed title screen which was discussed earlier in Figure 1(a).
For the Activity based application figure 3(a), user has the options for selecting when they want
these updates to be delivered to them. Three types of these optional features are included in the
application. 1) Permit the alerts when walking, as the application has access to the user’s device
API which was discussed in previous section, so when a user is walking for instance let’s say
around the campus, the inbuilt GPS sensor of the mobile device detects the change in the
23
location of the user and triggers this feature, the application will then check if there are any
updates which are to be delivered. Of course this is when this feature is opted ‘On’ by the user.
Similarly, the second feature when switched on by the user, the application will check for the
updates when a user is on the move (travelling or driving). With the help of suitable third party
mobile applications available on the Android market this feature is feasible with the voice
dictating application. That means, if a user is driving and this feature is opted ‘On’ and there is
an application on voice dictating already installed on the user’s mobile device. He will be
delivered the alert using the voice automated alert system which indeed is a very useful feature
of this mobile application.
Figure 3(a) & 3(b)
24
For the Temporal based Context aware mobile application figure 3(b), A user has choice to input
how many times per day they want the mobile application to check for the updates. For Example
if a user inputs 4 times a day, the application will then automatically check for the available
updates 4 times in a day and then deliver them to the user with help of RSS feeds.
One location based context aware mobile application also designed in the same way which
allows the user to receive the RSS updates depending on the location of the object points such as
learning zone, library etc. The application checks for the updates once the student reach a
specific area point within the campus. So a user can choose amongst the three temporal, activity
and spatial based context aware application whichever matches their study style. Earlier a
comparison between temporal and spatial context aware mobile application has been done and
the results were stated by Crane et al 2011. I will be evaluating the usability analysis and the
results in the next chapter.
25
5 Evaluation and Results
This section of the report discusses the results obtained from the questionnaires which were
performed in the task evaluation process and the overall system usability testing in order to
provide a better view for developing mobile applications. This will help to further state the
usability of the application and will outline the features which could be improved in the system.
5.1 Black Box Testing
This method is used to make sure all the basic features provided by the application work fine
when they are executed. This technique helps to figure out if there is any erroneous outcomes or
unexpected result when a task is performed in the application.
5.1.1 Main Screen
Test No. Test Expected Outcome Result
1 Click the application icon Launches the
applicationPASS
2 Click on an update Displays the content PASS
3 Click on settings icon Displays the available
options
PASS
4 Exit the application Application Exits PASS
26
5.1.2 Settings Screen
Test No. Test Expected outcome Result
1 Change the walking option in
settings
Toggles On/Off PASS
2 Change the driving option in
settings
Toggles On/Off PASS
3 Change the resting option in
settings
Toggles
On/Off
PASS
5.2 Black box testing results
According to the black box testing results the application performs all the basic feature it
provides effectively. That means the application does not crash or perform any unexpected
results. The application was easy to install and uninstall and had no problems so far.
5.3 User Testing Results
The results that I received narrowly indicated that:
1) They have used similar features in the application.
2) It doesn’t take that much of time for them to understand the use of application.
3) They think it could help them in their time management skills.
27
4) The system was really easy to use and straight forward.
5) They are willing to use it in future
6) They are comfortable with the concept that how context aware features uses their
location, device time and personal data. As according to them for most of the application
presently in the market they use these kind of features.
According to the task evaluation survey most of the users were satisfied with the purpose and the
nature of the task. They were able to understand how the application can be useful in their
learning space. The users gave the ASQ rating of 2.66 overall satisfaction from the task
evaluation process. They were confused in the beginning of the test, about the nature of test
being in laboratory based set-up. But they understood by the end of the evaluation process how it
could help the system to improve its existing features.
5.4 Computer system usability questionnaire results
The CSUQ questionnaire was used to evaluate the usability of the system overall. Table 1 shows
the average rating given by the 12 test users on each of the sub categories of the system usability
discussed in earlier chapter. The score ranges from 1 to 7 of the individual users but the average
rating given by the user is summed of in the table below.
Overall Usability 2.24
System Usefulness 2.72
Information Quality 2.77
Interface Quality 2.93
Table 1: Users average usability ratings
28
Overall the application received a positive response from its users. Most of the users have agreed
that the application is useful and easy to use. Table 1 shows that user’s opinion varied somewhat
in terms of the usability but most of the users have given the application a rating between 1 to 3
on a 7 point scale.
Figure 1 User’s overall satisfaction level
5.4.1 System Usefulness
Most of the users think that system is really useful. It has the ability to help the users in their
time management skills as well as being updated with their courses/modules over the mobile
phone. 10 out of 12 students have rated the application that it is easy to use the application. 11
out of 12 students rated the application as they were comfortable using it. 8 out of 12 users think
that they can use this application in their work space. Remaining users said that they couldn’t
navigate to the URL listed in any of the threads. To access those updated content over the phone
29
was not possible as the URL listed in the menu wasn’t hyperlinked. In order to access the newly
uploaded content they had to type the whole URL given in the RSS feed manually over the web
browser of the phone in order to access what the actual content was. Figure 2 shows the overall
rating given by the users.
Figure 2 User’s feedback on System Usefulness
5.4.2 Information Quality
This category shows the quality level of information provided within the application. This
includes the data content within each RSS based message. The users have given the CSUQ rating
of 2.77 out of 7. There were no unexpected and irrelevant data within the application users
commented. However, some users think that the application could have been integrated with the
Lancaster University’s own Moodle environment iLancaster. Some users commented that
application could have provided the download link of the added content instead which I think
30
was an interesting aspect. Most of the users were satisfied with information quality. Figure 3 and
Figure 4 shows the variable rating given by the users under this field. Majority of users think that
application as such did not need any specific help section as it was fairly easy to understand and
really simple to operate.
Figure 3 Information Quality (1)
31
Figure 4 Information Quality (2)
5.4.3 Interface Quality
Under this category users rated the mobile application on an average of 2.93. Most of the users
think the application had a really simplistic user interface, which was really easy to learn, to use
and to remember. They got familiar with the system right after they used the application. They
understood all the features within the mobile application and appreciated the simplicity of the
application. Most of the user said the application did not have any type of unnecessary features
which were not used. Navigation through the interface was fairly simple. However one of the
user didn’t like the user interface. They think the user interface could be improved by adding
more features to it. 9 out of the 12 users somewhat liked interface of the application as they think
adding extra features increases the complexity of the program. Although the distribution of
people finding the user interface pleasing is quite distributed but overall they intend to ask for a
simple structured mobile application.
32
Figure 5 shows the Interface Quality of the application
5.4.4 User Feedback
Apart from the questionnaire some user feedback were also received in the evaluation process.
Majority of the users came up with the most popular positive attributes of the mobile application
1) Useful
2) Simple
3) Helpful
4) Usable
5) Effective
6) Collaborative
33
Some negative aspects according to some users point towards the information content, lack of
integration with the built in web browser of the mobile phone as they could not access the URL
provided with the data content with in the RSS feed, and improving the user interface by adding
more features.
5.5 Summary
This chapter has explored the testing and the evaluation of the mobile learning application. The
usability test conducted earlier in this project deduced the overall usability of the mobile
application for this project. The Black box testing signifies that the application and all the
features included within the application were working appropriately. There were no unnecessary
features in the application which were not used during the testing procedures. The usability
testing performed specifically for this task was focused more on the laboratory setting as user’s
full time and focus were recommended while they share their views on the aspects of the mobile
application. The testing method showed different aspects of the application from a user’s point of
view. Some of the attributes were highlighted during the evaluation procedure. Some of them
were positive and some of them were negative. In order to present a better view on the usability
both the quantitative analysis of the system along with qualitative data are needed. However it
was apparent that there was a scope in improving the user interface and the information content
according to some users, but then it’s the prototype of the application. Relevant features and
additional plugins could be added after the pilot model of application.
34
6 Conclusion
This section of the report we will revisit the initial stated aim and all the work that was done in
the project, unimplemented work that was supposed to be done and the scope of improving the
system as an overall. This section mainly focuses on whether the initial aim of this project
described earlier in the beginning of this report have met.
6.1 Summary
The report has explored the fact that how mobile devices has become a vital part in a student’s
life. Every system is now at the verge of being mobile or it already has almost. Mobile devices
can serve a big difference by exploiting the areas of their learning environment on the go. How a
user can still be updated with all the learning content and lectures and all the important
discussions which cannot be missed now can be available to the students wherever they are. This
project explored the usability of mobile learning application in order to provide a student a good
learning environment on their mobile device. It also explored some similar environments which
made this study possible or are somehow related in the development procedure of the mobile
application. In really simple words the student at whom this study and application were based,
were asked how an environment like this could help them? By sharing their thoughts on the
mobile application, they can create a scenario for the developers who can actually bring their
ideas into the mobile application. This project insights on some of those user point of view based
scenarios in order to provide a better scope of features within the application.
6.2 Review of the Aim
The overall aim of the project which were stated in the introduction to this report.
To investigate the usability of the context-aware application in learning
environment.
35
Three dimensions of the context-awareness were exploited in mobile application designed for
this study. Spatial dimension, temporal dimension, and Activity based context-awareness. All
three came out to be equally important and equally useful for the student based on the student’s
choice and nature of their learning style. The mobile application was very easy to use, simple to
understand and efficient to use. Users who were in fact the university’s own students found that
the application can be used effectively and can provide positive outcomes in their time
management skills. The mobile application was based on the Android platform which was a
good plus point as some of the users were using the same platform based mobile phones in their
life style. Most of the users who were the test users of this study have used M-learning
applications which were similar to the one I used in this study in some ways, which indicates that
the concept of bringing learning to mobile indeed is a great idea. Users showed great interest in
sharing their thoughts on bringing learning content on their mobile. The study shows that they
can come up with ideas which could help in making the system more systematic and more
efficient to use. However there were some features they think could be improved which will be
covered in the coming sections.
6.3 Problem Encountered
In the beginning of this project I did experience some of the problems, which were then fixed
after the discussion with my supervisor. I was given clear guidance on how to look at the whole
scenario and scope of the usability testing.
During the project, gathering the test users for the study was difficult as it was really unlikely for
them to be all free at the same time for the test. So that created a bit of delay in the evaluation
procedure. But then I did eventually was able to collect all users at one place at the same time.
Performing the Test itself was somewhat challenging, as I was the only one conducting,
instructing the users at the same time but then it was a good experience.
36
Running the designed applications on the Eclipse software was not possible to install on various
computers in the Lab. So I decided to gather as many Android phones I could to perform the
usability testing which was then performed 3 mobile phones.
6.4 Unimplemented feature
The test initially was aimed to be performed on 30 users but due to lack of time and gathering 30
test users and conducting test amongst all of them was very difficult for me. Which was then
discussed with my supervisor and we decided to perform the test on 12 users instead.
6.5 Improvements and possible future work
In this section we will be discussing about the scope of all improvements which came into the
picture during this study.
6.5.1 Extension of the mobile app with the mobile inbuilt browser
One improvement that could be made is adding a web browser support in the application. For
example once a user get an automated RSS update, he receives the update as the date, time, the
topic in which the upload has been made and the web moodle URL. But that web URL is not
hyperlinked. In other words If a user can simply click on the given URL he or she should
automatically be able to redirect the URL to the mobile device’s inbuilt browser. A snapshot of
the update received by a user is display below.
37
Figure 1 A snapshot of a RSS Update.
The URL in the update is inaccessible. So if a user who is already mobile (limitation of time and
availability space) get an update about a new lecture note being added he cannot just simply click
on the URL. The user have to type the whole URL manually to a web browser in case he or she
wants to see what the actual content/discussion is.
38
6.5.2 Integration of the application with iLancaster
The university’s m-learning environment application is really popular amongst student. It has
number of features that are available within the mobile application including library, bus
timetable etc etc. According to one newly added feature in the iLancaster mobile app. A student
can now access their moodle virtual learning environment within the app itself. The application
which was designed for this study if possible could be integrated with the iLancaster application.
So that once the user clicks on the URL provided by the RSS feed, simply by clicking on that
link he or she should be able to access the content. A screen shot of the iLancaster’s Moodle
environment is displayed below.
39
6.5.3 Improving User Interface
From the usability test performed for the study, the results shows that there is still scope for
enhancing the user interface of the application. The usable mobile app should be pleasing to look
at as well as satisfying to use. The GUI could be enhanced with the help of custom background
or simply by adding some graphical icons for the application. The user interface could be
categorized is such a way that all the newly alerts about the lecture/course notes should go in one
section and all the discussions should go under the thread of discussion. It will eventually
increase the usability of the overall system. This leaves the space for the future work and gives
an insight to the possible work which could be done to enhance the user’s experience in m-
learning.
40
6.6 Lessons Learned
This Project has had a lot to teach me. The main things which I have learnt while working on this
project is the importance of time management skills. I gained supervising experience while
conducting the usability test and has encouraged me to do something like this in the coming
future. I got to improve my organization skills which was one of the key feature while doing this
project on the whole. I have gained some experience with the basic Android programming as
well in order to carry this study.
The main lesson I have learnt will be key to success in developing any type of system, whether it
be a mobile application or any type of system in general. Learning about the possibilities of
improvement in a mobile application I have gained some really valuable core concepts on how to
make any type of system usable by testing it on the real world users.
6.7 Overall Conclusion
This project has fairly completed its goal to investigate the usability of a mobile application in
learning and its usability amongst student. However I was able to produce some of the features
that could improve the overall usability of the application from the study. This will help in
improving the overall functionality of the mobile application in the future.
This project has been a lot to be learnt from and has given me insights on how to make a system
even more efficient. Testing user’s focused mobile application on the same area of users creates
new scope for feasible environment which could have been there in the application.
41
References
[1] Nielsen J,. Usability 101: Introduction to Usability. (2007)
[2] Nielsen J., & Mack L.R., Usability Inspection Method, John Wiley & Sons Inc. 1994, New
York.
[3] Nielsen J., Usability Engineering 1994.
[4]L.Crane, P. Benachour, P.Coulton, “A User Study of Spatial and
Temporal Context Aware Technologies to support Mobile Virtual
Learning Environments”. [MLearn2011 Conf. Beijing, China,
pp.335-342, 2011]
[5] Google’s Android becomes the worlds leading smart phone platform (Canalys)
http://www.canalys.com/newsroom/google%E2%80%99s-android-becomes-world
%E2%80%99s-leading-smart-phone-platform
[6] IDC Press Release http://www.idc.com/getdoc.jsp?containerId=prUS23638712
[7] Eclipse IDE http://en.wikipedia.org/wiki/Eclipse_(software)
[8] Android SDK Wikipediahttp://en.wikipedia.org/wiki/Android_software_development
[9]Dourish P (2004) What We Talk When We Talk About Context. Personal and Ubiquitous
Computing 8: 19-30.
[10]Dey AK (2001) Understanding and Using Context. Personal and Ubiquitous Computing 5:
4-7.
Springer-Verlag London Ltd
[11] Schilit B, Adams N, Gold R, Tso MM, Want R (1993) The ParcTab Mobile Computing
System. In
42
Proceedings of Fourth Workshop on Workstation Operating Systems, Oct 1993: 34-39.
[12]Dey AK & Abowd GD (2000a) CybreMinder: A Context-Aware System for Supporting
Reminders. In Proceedings of Second International Symposium on Handheld and Ubiquitous
Computing, HUC 2000: 172-186.
[13] Brown LM, Brewster SA & Purchase HC (2006) Multidimensional Tactons for Non-visual
Information Presentation in Mobile Devices. In Proceedings of MobileHCI’06: 231-238.
[14] Abowd G (1999) Classroom 2000: An Experiment with the Instrumentation of a Living
Educational Environment. IBM Systems Journal, Special issue on Pervasive Computing 38(4):
508-530.
[15] Bardram JE (2004) Application of Context-Aware Computing in Hospital Work – Examples
and
Design Principles. In Proceedings of ACM Symposium on Applied Computing: 1574 – 1579.
[16] Pocket Lancaster http://pocket.lusu.co.uk/
[17] Preece J, Rogers Y & Sharp H (2002) Interaction Design. Beyond Human-Computer
Interaction.
John Wiley & Sons Inc.
[18] Evans K M & Finck N., Interview with J. Nielsen, The web professional online magazine.
[19] Fetaji M http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4588461&tag=1
[20] Monahan T. Usability Testing of Collaborative & Interactive Iniversity on a mobile device
43