Improving Usability and Correctness of a Mobile Tool toHelp a Deaf person with Pharmaceutical Instruction

Michael B. Motlhabi1, William D. Tucker1,Mariam B. Parker2

Computer Science1, Pharmacy2

University of the Western CapePrivate Bag X17, Bellville 7535 South Africa

{2706912, btucker,mbparker}@uwc.ac.za

Meryl GlaserDeaf Community of Cape Town

12 Gordon Road, Heathfield 7945 South Africamerylglaser@gmail.com

ABSTRACTThe computing for development community knows how tobuild user interfaces using qualitative methods for text il-literate users, especially on mobile devices. However, lit-tle work has been done specifically targeting Deaf users indeveloping regions who cannot access voice or text. Thispaper describes a multi-disciplinary collaboration towardsiterative development of a mobile communication tool tosupport a Deaf person in understanding usage directionsfor medication dispensed at a pharmacy. We are improv-ing usability and correctness of the user interface. The tooltranslates medicine instruction given in English text to SignLanguage videos, which are relayed to a Deaf user on a mo-bile phone. Communication between pharmacists and Deafpatients were studied to extract relevant exchanges betweenthe two users. We incorporated the common elements ofthese dialogues to represent content in a verifiable mannerto ensure that the mobile tool relays the correct informa-tion to the Deaf user. Instructions are made available for aDeaf patient in signed language videos on a mobile device.A pharmacy setup was created to conduct trials of the toolwith groups of end users, in order to collect usability datawith recorded participant observation, questionnaires andfocus group discussions. Subsequently, pre-recorded signlanguage videos, stored on a phone’s memory card, weretested for correctness. Results of these two activities arepresented and discussed in this paper.

Categories and Subject DescriptorsK.4.2 [Social Issues]: Assistive technologies for personswith disabilities; D.5.2 [HCI]: User Interfaces—Evaluationand methodology ; H.5.2 [HCI]: User Interfaces—User-centereddesign

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions/acm.org.ACM DEV 4 December 6–7, 2013 Cape Town, South AfricaCopyright 2013 ACM DEV 4 978-1-4503-2558-5/13/12 $15.00.DOI: http://dx.doi.org/10.1145/2537052.2537063

General TermsICT4D, Human Factors, Design

KeywordsMobile applications, Sign Language, Community-based co-design

1. INTRODUCTIONMobile devices have found broad application in the pro-

vision of medical services and are being used robustly indeveloping regions [2], not only as communication tools butalso as keys to solving socio-economic challenges. For dis-advantaged end users, mobile devices present a significantchallenge by way of inadequate literacy skills needed to un-derstand the information on the device [4]. This problem isfurther compounded for people with disabilities, as can benoted in the context of a Deaf1 patient visiting a pharmacyto collect medication. Registered sign language interpretersare a possible solution, however they are expensive and thusnot accessible to Deaf people because 70% of Deaf peopleare unemployed thus can not afford the services of an in-terpreter [21]. Deaf patients often leave the pharmacy notunderstanding how to ingest their medication.

Deaf people in South Africa have an average education ofGrade 7 [21] and even those who possess rudimentary textliteracy experience difficulty in communicating with hear-ing people. It is a frequent misunderstanding that a SignedLanguage (SL) is a signed form of a written/spoken language[1]. Thus, most Deaf individuals cannot communicate withinthe same medium as hearing individuals, and are also un-able to read or write adequately. This functional illiteracyrenders medicine labels useless to Deaf patients in develop-ing regions, since reading is not a viable option for manyof them. Instructions for using medicines are given bothverbally, by the pharmacist instructing the patient how touse his/her medicines, and in a written form, by way ofa pharmacy-generated medicine label. Since both optionsare not viable for many Deaf people, a situation may arisewhereby a Deaf person does not know how to use his/herprescribed medicines appropriately (see Figure 1). Thus,the need for concise and clear communication of medicine

1Deaf with a capital ‘D’ is different from deaf or hard ofhearing in that Deaf people primarily use a natural sign lan-guage to communicate and this defines their sense of culture,similar to other groups who use spoken/written languages.

Figure 1: Pharmacist’s interface to dispense medi-cation (left) and Deaf person’s view of informationin SL (right).

instruction between a pharmacist and a Deaf patient is crit-ical for the process of medication dispensing, especially indeveloping regions.

This paper is organized as follows. §2 covers backgroundwork on which this study is built. §3 discusses the motiva-tion for using information and communication technologiesin a developing region, in this case for a disabled subgroup.§4 presents work related to signed language communicationaids and mobile videos for signed language. §5 explains themethodology used to realize the current prototype with thedual aims to improve usability and correctness. §6 illustrateshow SignSupport is used in practice. §7 describes the designand implementation of SignSupport. §8 first recounts datacollected from initial role plays to determine the commu-nication options, then analyzes a follow-up trial with Deafparticipants and final year pharmacy students and finally,describes the verification testing of the SL videos. §9 con-cludes and outlines future work.

2. BACKGROUNDDeaf people in developing regions use services such as

Short Messaging Service (SMS) and WhatsApp to communi-cate with each other and with hearing people. Yet their levelof text literacy is adequate for social purposes rather thanspecific and/or technical discussion [10]. Inadequate text lit-eracy creates a communication barrier, and in a pharmacyenvironment, only SL can precisely convey information in away that Deaf people can clearly understand [6]. For thesereasons, SignSupport was developed to assist communica-tion between the Deaf community and pharmacists. We col-laborate with a non-government organization (NGO), whichhelps Deaf people deal with social issues. This NGO has ap-proximately 2000 members. All members are fluent in SASLand most of them are functionally text illiterate. Since phar-macists are also the intended users of SignSupport, we alsocollaborate with local senior pharmacy students as surrogatepharmacists.

SignSupport is a mobile application that uses a touch-sensitive interface (see Figure 1) and is based on a systemthat employed a mock-up of a mobile phone on a desktopcomputer that allowed a Deaf person and a doctor to com-municate with each other using pre-recorded SL videos [14].The mock-up asked a Deaf person questions in SL. Aftera Deaf person answered the questions, the answers were

presented to a hearing doctor in English. The doctor readthe summary of symptoms and responded using an Englishlookup dictionary. The Deaf person then watched a corre-sponding SL video.

That mock-up was implemented on a Symbian phone witha guided set of web pages with a combination of SL videosand English text [18]. This enabled a Deaf patient to com-municate to a doctor which symptoms they were experienc-ing and for how long they have been experiencing them. Ateach step, the Deaf user responded to a series of questionspresented in SL, and finally enabled the system to conveyhow the Deaf user was feeling into English for a doctor tounderstand, and the doctor could respond. However, thereality of pre-recording all communication topics compris-ing a possible patient-doctor interaction, taking into accountall conceivable symptoms and diseases, presented an infinitenumber of possibilities. Thus a true two way communicationwas not feasible.

A follow-up effort investigated the way Deaf people usemobile phones to communicate in their daily lives with bothhearing and Deaf people. With input from the local Deafcommunity, SignSupport was re-oriented towards a phar-macy scenario [6]. Since the pharmacy exchange is restricted,pre-recording and storing a restricted yet useable communi-cation flow on a mobile phone became possible and a teamdesigned an Android prototype [17]. Machine translationsof SL are currently problematic. These problems are am-plified when there are important structural differences as isthe case with spoken/written languages and SL [12]. Au-tomated sign language systems have a limited translationaccuracy of merely 61% [9, 20] which is unacceptable forpharmaceutical use. Moreover, using these systems requirehigh-end smart phones or expensive computers [7, 8, 16].We wish to minimize the cost of a DEV solution by usingaffordable mid-range technology that will soon become to-morrow’s low-end. Therefore, pre-recording a restricted yetcomplete exchange became feasible.

3. MOTIVATIONDEV is a still a relatively new field, often driven by Com-

puter Scientists, that explores how computing technologiescan be applied to solve the needs of those who reside indisadvantaged regions [4]. DEV has focused on increasingstandards of living in developing countries, often by import-ing or adapting technologies created and used in developedcountries. To make the project a success for Deaf people in adeveloping region, we review a number of factors that couldhinder the acceptance of SignSupport: text literacy beingthe most significant. Mobile phones are heavily text-based.This is a style that was inherited from developed regions,yet many of the people who live in developing regions aretext illiterate [15]. Since the target group intended to useSignSupport are not in a position to afford a high-end smart-phone, the phones running SignSupport will be loaned fromthe NGO. We were conscious of the history of failed ICT4Dprojects [11]. SignSupport does not use text to communi-cate with Deaf people but rather SL. Thus, text illiterateDeaf people interact with SignSupport in SL videos and canunderstand the information being shared with them. Thispaper demonstrates that it is possible to devise interfacesthat can aggregate and accurately represent information forDeaf users in their preferred language, in our case, a signedlanguage.

4. RELATED WORKThis section describes the work related to SignSupport

and links the principles and methods that others in the areaof technologies for Deaf users have used. We study deaf/Deaftechnology and how Deaf-specific challenges have been ad-dressed within ICT. Very little work exists for DEV workfor Deaf users. Thus, we also review the challenges faced bypeople with disabilities in developing regions.

4.1 Base technology for Deaf usersMobileASL uses American Sign Language (ASL) and is an

ongoing video compression project that seeks to enable low-cost low-bandwidth sign language communication with mo-bile phone technology [5]. The goal is to make ASL mobilephone communication possible without the need for equip-ment other than a mobile phone with a front-facing camera.It works on commercial phones that are accessible to Deafpeople. The motivation for MobileASL is to make as cleara sign language video as possible to transmit over a cellu-lar network [5]. Cavendar et al conducted user studies withmembers of a deaf community to determine the intelligibil-ity effects of video compression techniques that exploit thevisual nature of sign language. Preliminary studies stronglysuggested that even today’s best video encoders cannot pro-duce the quality video needed for intelligible ASL in real-time, given the bandwidth and computational constraints ofeven the best video mobile phones. MobileASL concentrateson three major areas when manipulating video for sign lan-guage use; (1) Bitrate, (2) Frame rate and (3) Region ofinterest. They deemed these three variables important forusing sign language videos on mobile phones [5].

4.2 Assistive applications for Deaf usersIn the UK, an experimental system called Text and Sign

Support Assistant (TESSA) was developed to assist withtransactions between a Deaf person and a clerk in a postoffice by translating the clerk’s speech into British Sign Lan-guage (BSL) and the Deaf person’s signs to text/sound [7].To generate the signs needed for the TESSA system, thesigns of the native signer are first captured as motion datavia sensors affixed to the body of the signer. The post officeclerk speaks into a microphone, while the system generatesthe respective signs (currently, not all phrases can be cor-rectly identified) [7, 8, 12], which are represented by virtualsigning avatars resembling humans. TESSA was developedfor the post office scenario because most of the conversationsare predictable and simple to follow. The movements of theavatar are copies of those of a native sign language user.Software specially developed for the project captures theBSL of the signer’s hand, mouth and body movements usinga variety of electronic sensors. These movements are thenstored and used to animate an avatar [7]. The ViSiCASTproject sought to improve the quality of life for Deaf peopleby widening their access to services and facilities enjoyed bythe community at large [8]. The objective of the ViSiCASTproject was to produce adaptable communication tools al-lowing sign language communication where only speech andtext are presently available. The project identified a num-ber of aspects of life where the integration of Deaf individ-uals into society would be improved if sign language com-munication were available, such as access to public services,commercial transactions and entertainment [7, 8]. The ViSi-CAST team started their first prototype with an interpreter

and later changed to using an avatar. ViSiCAST project4now makes sign language communication possible for face-to-face transactions and television broadcasting. One of theaspects of this project transcribes spoken speech to text,which is processed to its equivalent signs in BSL, and conse-quently signed by an avatar [8]. A face-to-face transactionvirtual signing system was tested with TESSA. The biggestdrawback of these types of systems is the current lack of signrecognition; their communication is one way, from the avatarto the Deaf person, thus rendering them to be ineffective forcritical communication exchange.

4.3 Sign Language translationA group in Iran uses Artificial Intelligence (AI) techniques

to research ways of solving communication problems amongDeaf and hard-of-hearing people using mobile phones. Theyproposed a system called ‘ASRAR’, that finds a common fac-tor between Augmented Reality (AR) and Automatic SpeechRecognition (ASR) technology which is stored as a string.They combined AR, ASR and Text-to-Speech (TTS) synthe-sis to develop a system that allows Deaf and hard-of-hearingpeople to communicate with each other and hearing people[16]. ASRAR gives an opportunity to deaf people to controland manage the information and adapt it easily to a desiredform to improve their interactions with hearing people. Inthe ASRAR system scenario, the ASR engine collects thespeech from the detected narrator and uses the ASR engineto recognize the narrator’s speech and convert the speech totext. To get the video and the speech of the narrator, theASRAR uses built-in cameras and microphones on the mo-bile phone. A joiner algorithm is used to combine AR andASR engines to work together and does this by updatinga version of the text file, which means every word must besaved in the text file. The script operation changes everysecond when a word is written in the text file by the engine.A survey with Deaf and hearing people was conducted atthe end of the research process to understand the interestamong Deaf users using different communication methods[16]. Preliminary results showed that the system worked wellin many different environments. Deaf participants showedinterest in using the system as an assistant to communicatewith hearing people.

4.4 Relation to SignSupportLike TESSA, SignSupport is designed to function in a

limited communication domain setting [17]. For reasons ofaccuracy, however, SignSupport differs from TESSA in thatit does not use Artificial Intelligence. Like MobileASL, Sign-Support emphasizes video quality and resolution, albeit lo-cal to the phone and does not yet incur over-the-air mobiledata charges. SignSupport uses a similar method as withASRAR to link English text with a signed language equiv-alent, although we do not perform any form of automatedSL recognition or generation. And like ViSiCAST, we hopeone day to generalize SignSupport to handle multiple com-munication scenarios (see §9).

5. METHODSSignSupport is based on a decade of research and collab-

oration by a multi-disciplinary team of professionals [6, 14,18, 17, 22]. §5.1 introduces multi-disciplinary roles. §5.2 ex-plains a design method employed that employs user-centeredsolutions in a complex design space [3]. §5.3 details the cy-

cles taken to collect requirements and iterate design anddevelopment. §5.4 breaks down the video recording proto-col and §5.5 details the process taken to verify video contentcorrectness.

5.1 Multi-disciplinarityOur multi-disciplinary team comprises a broad range of

expertise. All team members were involved continuouslyfrom design through to development, and from experimen-tation through to verification.

Deaf participants decide what the project is and howthey would like to use it, and most of the user requirementsemanate from them because integrating their perspectivesincreases the chance of developing a successful and acceptedsystem.

Pharmacists play a critical role in the protocol designaspects of the system. Their input brought to bear a numberof important pharmacy-oriented directives, concerning:

• Pharmacists’ code of practice [19, 22]. Pharmacistswere instrumental in designing the application suchthat it follows a known and standardized logic.

• Ethical principles and respect for persons. As health-care professionals, pharmacists are trained to adhereto strict ethical standards at all times when interact-ing with patients. Thus we must ensure correctnessof the medication usage and description informationdisplayed in the user interface (see §5.5).

• Data input. SignSupport provides an autonomous wayfor a Deaf user to acquire medicine instruction, givenproper input from a pharmacist. The application hadto address the following elements: diseases, dosageforms, medicines, instructions, warnings and recom-mendations [19].

Industrial design engineers were responsible for de-termining the appropriate conceptual model of the system.They achieved this goal by involving both Deaf and phar-macy participants, and acquiring requirements by means ofrole playing, questionnaires and focus groups (see Figure2). They presented this information in the form of a de-sign/sketch on paper that best represented the expectationsof both Deaf users and Pharmacists. They designed Sign-Support’s interfaces (Figure 1) based on interactions withend users over several versions starting from the initial mock-up [14], its implementation for medical diagnosis [18] thento the first pharmacy design [6].

A Deaf education specialist was a link between thetechnical team members and the Deaf community members;a bridge between the technical team and Deaf users of ICT.This specialist helped customize the interface and logic ofSignSupport to seamlessly fit Deaf users’ expectations, andhelped analyze the sentences extracted from the role playsbetween Deaf patient and pharmacist, and structured thesentences to make sense in SL.

Computer scientists’ core duty was to bring the soft-ware application to reality, to evolve the human computerinterface and verify that the correct SL videos were shownat the right place and time. They examined how end usersinterface with SignSupport, and helped avoid the problem ofoverloaded menus by using hierarchical menus [13] as shownon the left side of Figure 1.

5.2 Community-based co-designThe requirements gathering process highlighted four dis-

tinct aims. (1) Reliability: since SignSupport is medicalsoftware; it is expected to work reliably and accurately with-out fault at all times (see 5.5 for more details). (2) Usability:both types of users must be able to operate SignSupportwith ease after a simple short training session. There aretwo different interfaces, one for each user: the pharmacistinterface is English-text based and the Deaf interface is SL-video based. (3) Acceptability: users must be willing to useSignSupport. For example, Deaf users need intelligible SLvideos and a vibrating and an SL-based (as opposed to text)reminder system to help them remember what medicine totake and when to take it. (4) Sustainability: this aim ledus to investigate how to easily add more functionality toSignSupport and to design a back-end architecture that willeasily accommodate new functionality (see §7 for details).

To address these aims in holistic and realistic ways, weemployed the strategy of community-based co-design [3].Naturally, traditional human centered design methods werechosen for the community-based co-design process, as thesetechniques facilitate the participation of the target groups.This approach required us at every stage to refer back tothe participants to show how their suggestions had been in-corporated into the SignSupport (see Figure 2). During allinteractions with Deaf participants, an SL interpreter facil-itated the communication process. Aided by an interpreter,and a co-author acceptably fluent in SL, we could mergethe design and usage context by understanding and buildingpositive relationships with the Deaf community. Similarlythe communication between the team and senior pharmacystudents from a local university were facilitated by a se-nior pharmacist to build trust and relationships with a localpharmacy community.

Figure 2: A timeline of events showing how require-ments were gathered. Each event with end users alsoacted as a feedback session showing participants howtheir prior input was factored into the process.

5.3 Design cycles and developmentA cornerstone of the development process was the contin-

uous integration of end-user requirements and feedback intothe design process. The result of end-user contributions ledto iterative re-design of the entire back-end architecture ofSignSupport (see Figures 2 and 8 for more detail). Volun-teers were purposively sampled from the NGO staff becausewe wanted informed views from the Deaf community. Figure2 illustrates nine steps which were followed, each describedin detail below.

Step 1: Focus group interviews with pharmacists. Fromthese discussions, the challenges of dispensing medicines topatients with whom they could not communicate, particu-larly Deaf patients, emerged because pharmacists use spo-ken language which Deaf patients are unable to understand.Pharmacists expressed that these interactions were oftenvery difficult, leaving the pharmacist unsure if the Deaf pa-tient had actually understood how to use their medicines.

Step 2: A paper prototype, as a result of Step 1. Thisprototype exposed basic user expectations for both targetusers.

Steps 3 and 4: Role plays to establish user perspectives forboth Deaf participants and pharmacists. Since SignSupportis intended for pharmacist-patient interaction, it was nec-essary to mimic a typical routine interaction between thetwo users. This was done by studying the patient interac-tion taught to pharmacy students, based on the School ofPharmacy’s Objective Structured Dispensing Examination(OSDE) sheet, a tool used in assessing students on patientcounseling.

Step 5: Conversation mapping. Video footage of the inter-action between the two parties during the role plays (Steps3 and 4) was studied. What was studied was what the phar-macist said to the patient, how it was said, and at whichstage of the interaction it was said. We successfully elicitedthe common dialogues, which occur between pharmacistsand hearing patients at public hospitals. The communica-tion flow at the pharmacy was limited in a similar fashionto TESSA, which covered about 90% of the communicationat the post office [7].

Step 6: Disease/medicine selection. Pharmacists wereasked to help identify one hundred of the most commonillnesses they thought were important to include in the pro-totype. Designations of 47 illnesses were video recorded inSL and stored onto the phone’s memory card. Medicinenames had to include every possible prescribed medicine forthese illnesses.

Step 7: Identification of the prescription and instruction.We studied real-world patient prescriptions scripted by doc-tors and mirrored much of the existing paper prescriptionlayout and content, and optimized it to fit on a mobile de-vice. The prescription text, instructions on the prescriptionand the sequence in which these instructions occur were re-viewed and incorporated into SignSupport. This was doneto ensure that when the pharmacist dispenses, s/he followsan already familiar natural flow. Note that while SignSup-port can act as a virtual prescription, it is not intended toreplace the doctor or pharmacist.

Step 8: SL video recording. A finite set of 180 videos wererecorded to represent the possibilities of diseases, medicinesand instructions determined in Steps 6 and 7 (see Figure8). A conversation script was created and used to guide therecording of the SL videos. An interpreter translated eachmessage together with an informed Deaf staff member of theDeaf NGO.

Step 9: Training and trial. Deaf and pharmacist partic-ipants underwent training, in two separate groups of 8, onhow to use SignSupport. Each session was about three hoursand included a projected presentation followed by hands-onpractice with SignSupport running on 8 phones. Partici-pants were encouraged to “play” with the application andprovide feedback after their hands-on usage. Each train-ing session was video recorded. Subsequently, both groups

Figure 3: Steps showing how the initial SL videorecordings transpired.

then participated together in a test trial at a mock hospitalpharmacy. Pharmacists had to dispense medicines as peractual prescriptions to Deaf patients the simulated dispen-sary. Great care was taken to mimic, as far as possible,the scenario that would occur when a Deaf patient collectsmedicines at a hospital pharmacy, including not being ableto hear their name being called when it was their turn. Deafpatients were asked to present their prescriptions to a phar-macist at the counter and s/he used SignSupport to dis-pense the medication. After participants finished the trial,they were asked to complete a questionnaire individually,and then participated in a combined focus group discussionwhere they could give more detailed and open-ended feed-back.

5.4 Initial verification during recordingSignSupport is heavily dependent on the SL videos that

are pre-loaded onto a phone’s memory card. It is thereforeimperative that the correct video corresponds exactly to theEnglish text instructions. The first level of verification oc-curred when initially recording the videos. A follow up ver-ification procedure is described in §5.5. We devised a set ofrules/procedures as seen in Figure 3 to verify that the videosactually represent the instructions that the pharmacist hascommunicated. The following steps were taken to record,edit and load the videos onto SignSupport, as illustrated inthe figure.

Step 1: Number every screen activity on SignSupport thatwill display a video. Every number corresponds to a sentencethat needs to be recorded on the content script (a scriptcontaining all the sentences that need to be recorded in SL).

Step 2: Label every screen activity on SignSupport witha dummy/placeholder video that explains in English textwhat SL message will be displayed on that screen.

Step 3: Once all the activities are labeled, numbered andappear in the correct place, start recording videos, readingfrom the content script and translating the text to SL withthe help of a SL interpreter. Every video that is recorded iswatermarked with the corresponding number from Step 2.

Step 4: Edit the videos, label them and remove the place-holders within SignSupport, then load the actual (not dummy)videos onto SignSupport one at a time using the watermark-ing numbers from the previous step. There are two ways inwhich we test whether the correct video has being displayed;this is explained in the next section.

5.5 Subsequent verification procedureThis section details the methods used to verify that Sign-

Support communicates the correct sign language informa-

tion for medication that the pharmacist has conveyed intext. Figure 4 below shows the sequence of taken to ver-ify the correctness of videos within SignSupport.

Below is a breakdown of Figure 4. We focus on two pa-rameters. Parameter 1 tests for the content of the video,and parameter 2 checks if the video appears in the correctplace. We know what the content of the video should befrom the conversation script and we know the position ofthe video from the watermarking.

Figure 4: A diagram showing steps taken to verifycorrectness of the videos.

Parameter 1 follows the these steps:Step 1: Transfer all videos from the phone’s memory card

onto a computer’s hard-drive.Step 2: Randomize the videos (arrange them in a way

that it is not possible for the interpreter to predict the nextvideo) and give them unique numerical identities.

Step 3: Present the videos to an interpreter who watchesthem on a different computer monitor on the other side ofthe desk and voices their content in English. Here we areexpecting a general translation of what the video explains.

Step 4: After the interpreter watches the video and anexplanation is given, we look at the conversation script thathas the two variables on it. Variable one is the numberidentity of the video. Variable two is the English equiva-lent of the SL used in the video. So to mark a video ascorrect/satisfactory, we look at the number given to thatvideo, listen to the interpreter’s explanation and compare itwith the one on the script.

Step 5: If a match is found in Step 4 we tick the “passbox”; otherwise the “fail box” if no match is found.

Step 6: Record comments from the interpreter on howsome of the content could have been expressed differently inSL for the next system iteration (see §/refconclusion). Re-peat Steps 3 through to Step 6 until we have viewed andchecked off all the videos that are on the system.

Parameter 2 is a simulation exercise and involves usingSignSupport to verify the position of the videos, and followsthe following steps:

Step 1: Conduct role plays with the interpreter by run-ning through the entire application. At each stage wherewe encounter an SL video, s/he interprets it and we confirmwith the script that indeed the video means what it shouldand appears where it should. We perform this task in thesame way a Deaf user using SignSupport would.

Step 2: Test the medicine instructions from the pharma-cist. We prepared “dummy prescriptions” that covered all ofthe different instructions and permutations that the phar-macist can set. From this comprehensive list we randomlychose the prescriptions that would be entered in SignSup-port. After we entered those, we asked the interpreter to

explain the message on the SL videos. Here we were testingwhether the English text selections made by the pharmacistare consistent with the information on the videos, referringto the conversation script for confirmation.

6. USING SIGNSUPPORT IN PRACTICEThis section summarizes how to use our system. Sign-

Support requires an Android phone running at least version2.3.3. Once installed, SignSupport opens by finger-tap. Fig-ure 1 shows two typical interface screens, one for the phar-macist and one for a Deaf user. Any Deaf user associatedwith the NGO could conceivably borrow the phone whengoing to the hospital pharmacy and return it when medicaltreatment is completed. What follows is a typical scenarioas described more fully by [17].

At the hospital, a doctor diagnoses the Deaf patient andhands him/her a paper prescription as per normal. TheDeaf patient takes this prescription and the smartphone withSignSupport to the pharmacy. While waiting for the pre-scription to be processed at the pharmacy, the Deaf usercan enter background information including medicine aller-gies, gender, access to clean water, pregnancy status andother information that the pharmacist may require to dis-pense medication appropriately. The background informa-tion complements the information about the patient on thepatient card at the pharmacy. This can be viewed as a lim-ited form of communication from Deaf user to pharmacist.

A Deaf patient must unlock SignSupport by entering afour-digit PIN that protects the patient’s medical informa-tion. When the Deaf patient is called upon to retrievehis/her medicines, usually by hand because Deaf patientsare usually shown to a special needs queue, s/he hands thephone to the pharmacist as soon as s/he gets to the dis-pensing counter. At this point, the interaction between thepharmacist and Deaf patient commences using SignSupportas the communication medium. The first screen promptsthe Deaf patient to show a hospital identification card. Thepharmacist is then able to ensure it is the correct patient,and can view the patient’s background history and can checkfor problems like allergies and concomitant medical condi-tions before dispensing the medicine. The pharmacist inter-acts with the application’s interface in order to dispense theprescribed medication, as shown on the left side of Figure1, by tapping information on the phone’s display, selectingfrom the provided options and capturing a photograph ofeach medicine with the phone’s built-in camera.

Since every screen activity is in line with the acceptedpharmacy practice code [22], this process should be easyand natural to follow for a trained pharmacist. Informationabout each prescription is delivered in SL videos for the Deafpatient. This represents communication from pharmacist toDeaf patient and comprises the bulk of SignSupport’s in-formation lookup translation from text to SL. SignSupportallows the Deaf patient to review instructions for any pre-scribed medication in SL at any time. Currently, SignSup-port also reminds the patient when to take their medicinein text (see §9 below), and also warns users when they areabout to run out of medicine for chronic conditions such ashypertension, diabetes or cancer. Note that the text phrasebeneath the video is not meant for the Deaf user, and is akey phrase to help a non-signer follow the application logic.

Figure 5: A representation of the user interface nav-igation used for SignSupport.

7. DESIGN AND IMPLEMENTATIONThis section breaks down technical details of SignSupport,

including its evolution from the first prototype to the cur-rent more robust and flexible user interfaces. The versionof SignSupport presented in this paper has undergone threeiterative cycles of development. Each stage of developmentbrought changes to the SignSupport’s back-end navigationstructure (see Figure 8) which in turn changed the flow ofthe user interface (see Figure 5). The block structures inFigures 5 and 6 represent screen activities within tSignSup-port and the arrows represent the direction of flow for allscreen activities.

7.1 System back-end evolutionIn Figure 5, the downward pointing arrows demonstrate

how the system has developed over successive generations,becoming more robust from Cycle 1 through to Cycle 3, thecurrent development cycle.

Cycle 1: After users had been interviewed via focus groupsand had drawn their own depiction of how the solutionshould look, our resident design engineer sketched a designthat was coded and deployed on an actual device. The firstprototype was monolithic and used a linear approach to sys-tem navigation(see Figure 5). This approach was acceptablefor small applications with about three or four screen activ-ities. However, it was undesirable eventually deemed unac-ceptable as the current prototype contains over fifty screenactivities. Moreover, people made errors and needed to goback to a previous screen. The linear navigation approachgave us an idea of how to organize and structure screen ac-tivities, so that we could start to address them in a moreefficient way.

Cycle 2: Hierarchical navigation was introduced in thesecond iteration (see Figure 5), and came in two forms:when a screen activity had multiple drop-down menus Sign-Support as can be seen in Figure 1, and between screenactivities that appear to the user at different stages. Al-though there were two different implementations for hierar-chical navigation, the application algorithm was the same forboth. The difference was how we aggregated the data thathad been entered into the system. The case of moving fromone video-playing activity to the previous or next proved tobe challenging because of the high frame rate and resolutionof the SL videos on the memory card. Limited processingpower of the mobile device caused the application to inter-mittently fail when loading the videos. We later discovered

that the AndroidMediaPlayer programming interface causedthe error, as it could not play videos back-to-back on differ-ent screen activities. We remedied this problem by creatinga buffer activity that separated the video activities and al-lowed the device to redistribute its resources/memory andprepare it for the next video activity. When collecting datafrom multiple screen activities, the data from the user waswritten to a file (similar to ASRAR) and kept there untilall of the instructions/tokens were collected (Figure 7), andat each stage the user could go to the previous activity andre-enter an instruction (see Figure 5). Should the user makean error, it was not necessary to go to the end and repeatthe entire process, and s/he could go back one screen activ-ity at a time. The former design would not be suitable, asit wastes time and led to user frustration.

Cycle 3: Cross-linked navigation is currently a combina-tion of the two approaches discussed above. The system hasareas where a monolithic navigation style is required. Thisis mostly true for the Deaf user interface, e.g. the one-timeonly set-up of patient background information. Hierarchicalnavigation was used for pharmacists because of the playbackchallenges mentioned in Cycle 2. The system’s back buttoncould not be used for this cycle because it does not call thebuffer activity but “calls” the AndroidMediaPlayer, which isnot suited to our needs. To solve this problem, we createda softkey button inside the application and disabled the de-vice’s “Back” and “Menu” buttons. This means navigatingthe system is only possible from inside the application. Wecan control the direction of flow and also the process thatleads to a specific event. The cross-linked approach alsoincreases the productivity of the pharmacist during dispens-ing as it allows the user flexibility to move back and forth,and make adjustments spontaneously. Thus far, we have ex-plored a number of different designs as seen in Figure 5 withdifferent outcomes. The current design is therefore a cul-mination of three cycles of both technical and user-centereddesign and evaluation.

Figure 6: The Deaf user provides text to the phar-macist via SL interaction and the pharmacist pro-vides SL to the Deaf user via a graphical user inter-face with text

Figure 6 shows a high-level use case of the entire systemand its users. From Figure 6 we can see that the systemforms a closed loop. This is why we define the system as alimited communication device tool. Communication is delib-erately limited to the pharmacy context and is for the mostpart pre-recorded on the mobile device running system.

Figure 7: A diagram showing the Sentence construc-tor on system

7.2 System specificationsEffort was dedicated to the provision of acceptable video

quality to SL users. We edited the video size to be 640x700(width and length) in MPEG-4 format at 30fps. We chose640x700 dimensions because we can programmatically ma-nipulate the video length and width (dimensions) withoutdistorting the signed language video when viewed on dif-ferently sized screens. Thus, the bigger the phone’s dis-play the better the sign language video quality. MobileASLfound that at between 10fps and 15fps users could not dis-tinguish the difference in video quality [5]. While this istrue, the video quality can still be poor at such low rates.A higher frame rate helps with legibility [9, 5] during videocompression from MOV to H.246. We convert from MOVbecause that is the format that our recording camera usesto H.246/MPEG-4 AVC (Advanced Video Coding), whichis a standard of video compression that uses open sourceand also because MPEG-4 is the official video format forAndroid OS. We have coded the system to fit on most cur-rently available Android phones.

The mobile phones we used had at least 2 gigabytes (GB)of external storage space or more, 1GB of RAM and a back-facing camera. The SL videos are an average length of about2:00 minutes and they were all recorded at the same locationwith the same background and lighting conditions over twodays. We removed the sound from the videos to make thevideos smaller and set all videos to black and white as sug-gested by Looijesteijn [14]. This is because pixels have onlyone property, i.e. colour. The colour of a pixel is representedby a fixed number of bits. The more bits, the more subtlevariations of colours can be reproduced and thus the largerthe video. For this reason we chose to make the videos blackand white. This allowed us to essentially give each pixel ei-ther a black or white colour resulting in a smaller video.

The user interface was coded with Extensible MarkupLanguage (XML), a language that allows the definition oftags, while having the qualities of HTML.

7.3 Text-to-SLThe pharmacy backend is coded in Java and has four lay-

ers as seen in Figure 8. Layer 1 contains possible medicalconditions, currently restricted in number to code and trialthe prototype. Layer 2 holds the medicines in the system,similarly restricted. Videos from Layers 1 and 2 appear ondifferent screen activities at different times, so these param-eters are labeled with the same name with which the SL

Figure 8: A layered view of the pharmacy backendthat indicates how a prescription is encoded.

videos are stored. To fetch and play them, we only referencethe name of the disease or medicine to the correct directoryon the memory card. Layer 3 holds combinations of prescrip-tion instructions with different permutations. Videos in thislayer are recorded as complete sentences in SL. A selection ofone item on every axis forms a token. This token is writtento a file that is later accessed and read (similar to ASRAR).We aggregate the data contained in these files to form a to-ken sentence. This sentence matches one of the SL videos onthe memory card. We search the memory card using linearsearch because the videos are fetched randomly dependingon the prescribed medicine. We find a particular video byperforming linear search on the memory card and compar-ing every string token until we find the one that matches.The matching string is the name of the video and that is thevideo to playback for the Deaf user. For videos that appearin a predictable sequence (background and security videos)we play them back by calling their associated text names(no searching/sorting is necessary here). However this isimpossible to achieve with the rest of the videos on the sys-tem because we do not know beforehand what video will beplayed next. This is why a video order procedure is neededand implemented. Layer 4 holds combinations of possiblewarnings and recommendations for the Deaf user and usesthe same simple lookup algorithm as for Layers 1 and 2.

A Deaf patient reviews a prescription in a four-stage se-quence. For videos in Layer 3, we recorded a limited numberof complete sentences instead of stitching together fragmentsbecause they would not make sense in SL. We concentratedon three prescription factors: frequency, quantity and dosageevent. Selected values for the three parameters limit thepharmacist from making selections that are not pre-loadedand thus limiting the communication flow. We can restrictthe domain of communication because we have captured allthe conversations and their flow loaded on the phone [17].The TESSA research group studied restricted domain com-munication and covered about 90% of the exchange [7] thatoccurs at a Post Office, we have used some techniques de-ployed in the TESSA project to also ensure that we reachhigh percentage levels of the communication exchange in thepharmacy context.

We store all SL videos on the phone’s memory card so asnot to incur network charges. This means all communicationis effectively limited to what has been stored on the phone,

and does not cost the Deaf user anything at all to use theapplication.

8. RESULTS AND ANALYSISThis section analyses the results obtained from the most

recent development cycle consisting of two types of testingindicated above: usability of the user interface for both typesof users, Deaf and pharmacist, and the verification of thevideos; that they say what they are supposed to say, and doso correctly according to the application flow.

8.1 UsabilityParticipants (n=16) were present for training and testing:

semi-computer literate Deaf participants (n=8) from a localNGO and senior pharmacy students (n=8). The sessionswere conducted using role plays followed by individual ques-tionnaires and then focus group discussions. We used SLinterpreters to collect data from the Deaf participants.

The focus was on usability testing including monitoringuser interaction with the system, and identifying potentialdesign flaws to be addressed in the next prototype. Re-searchers did not assist any of the participants during therole plays. Deaf participants were asked to input backgroundinformation into the system while they waited for a prescrip-tion to be handed to them. Pharmacists worked at the dis-pensary counter as they normally do, and patients were mo-tioned to the front to collect their medicine. When they gotto the counter, they produced the prescription and phone,and handed both to the pharmacist. The pharmacist (stu-dent) used the phone and the (faked) doctor’s prescription todispense medication, without directly communicating withthe Deaf patient because that was impossible as none of thepharmacists knew SL.

Both sets of participants were asked to answer a question-naire that enquired about the usability of the software andalso what they would like to improve. Pharmacists reportedthat the system was easy to use. They suggested that it wasmuch better to use the SignSupport to dispense medicineto a Deaf patient. The average dispensing time using Sign-Support was 4:23 minutes. In the first run of role plays,pharmacists dispensed medicine without SignSupport andtheir average dispensing time was about 9:55 minutes perpatient. Pharmacists reported that SignSupport was directand to the point about giving explanations and instructions.

Deaf users were happy to use SignSupport for collectingmedicine. They reported that it was easy to use and theywould use it in real life. Deaf participants did not have anychallenges navigating, performing and completing tasks us-ing SignSupport, and this they did after just one trainingsession. All the Deaf participants accepted SignSupport,but they expressed concern that pharmacists would not ac-cept the software at real pharmacies. We explained that thehospital staff would be informed of the technology when itis ready for deployment.

8.2 Video verificationTo establish SL video correctness, we were not testing

SignSupport itself, but rather using it as a tool to verifythe content and position of the SL videos within the proto-type. This section details the results following the procedureoutlined above to test whether the videos on SignSupportgive the correct information about the prescribed medica-tion. The following tools were used; a computer, a monitor,

a video, an interpreter, a conversation script and an Androidphone running SignSupport. The system was divided intotwo: background setup and pharmacy dispensing. Below isan analysis based on evaluating both of them.

The background setup contains 18 sign language videos.These are short sign language videos used to extract per-sonal information from a Deaf user when using SignSupportfor the first time. All videos in this section passed the veri-fication procedure and could be understood by the SL inter-preter, and confirmed with the corresponding informationon the conversation script. All videos found to be in thecorrect position within the prototype.

For pharmacy-dispensing, there are 162 instruction videosand 35 videos were found to be either undecipherable, am-biguous or the semantics did not match the conversationscript. Most of them could be understood at first glance.However, some were unclear because of the signs that wereused and others were discovered to be unusable because theydid not convey the information in the most understand-able/desirable format. However, they were all found to bein the correct position within the prototype. All partici-pants showed a positive response during the training andtesting process and were always eager to try out the exer-cises with which they were presented. The learning curveof the participants was remarkable since none of the partici-pants raised any major questions about the functionality ofthe system. The video verification test showed that out of180 videos in total, only those 35 mentioned above requiredre-recording. This translates to 80.6% usable videos. Wefound that all videos that deal with a patient consuming onetablet/capsule a day were misleading and thus could not beused consistently. For example, the video saying “take onetablet once a day every 24 hours after meals”was interpretedin SL videos as“take one tablet every morning once a day ev-ery 24 hours, every 24 hours”. Furthermore, “take one tabletevery 6 hours four times a day after meals” was interpretedas “take a tablet four times a day after meals, 6 hours after,6 hours after, 6 hours after, 6 hours after (sic)”. One inter-pretation is that the repetition is for emphasis. However,the senior pharmacist insists that it could lead to patientsoverdosing.

9. CONCLUSION AND FUTURE WORKSignSupport has shown promise as an effective commu-

nication bridge between a hearing pharmacist and a Deafpatient. The following elements of the application can beenhanced to ensure its success in a real-world situation. AllSL videos must be verified to be a direct 100% translationof the English instruction. Furthermore, the positioning ofthe videos must be re-established after re-recording in orderto continually ensure the correct dispensing sequence. Mal-functions in either of these components can result in harmto a patient. Since 19.4% of the recorded videos were foundto be either ambiguous or erroneous, these videos will bere-recorded with the presence of a pharmacist, an SL inter-preter and an informed Deaf member from the Deaf NGO.A subsequent verified version of SignSupport with correctedvideos will be assessed by experimentation in an actual hos-pital pharmacy after sufficient medical ethics clearance hasbeen granted.

SignSupport is an“internal sign language translation” sys-tem, internal in the sense that it incorporates a closed loopof limited conversations which typically take place between

pharmacist and patient. SignSupport does however not in-clude some conversations that could possibly arise, for ex-ample a Deaf patient asking a question. SignSupport is de-signed in such a way that it answers most common questionsasked by patients before they actually ask the questions. Forexample, recommendations and warning explanations, whento come back for a refill and so on are some of the commonquestions which SignSupport can answer. In the future tosolve this problem SignSupport might include functionalitythat allows for a video relay break-out, which will allow avideo conference-type communication, whereby an off-siteinterpreter can relay SL to English, for example, and viceversa. This would provide true two-way communication.

Although SignSupport was designed, implemented andtested in a pharmacy context, it could also be modified andapplied to any context, e.g. a Police Station or Home Af-fairs. Furthermore, we also realize that this tool could beadapted to use any signed language, and could also be usedwith audio instead of video to serve the text illiterate in de-veloping regions everywhere. These generalizations wouldentail building an authoring tool to allow SignSupport tobe context and (signed) language independent, and wouldmake an even more valuable contribution to the computingfor development community.

10. ACKNOWLEDGMENTSWe thank the Deaf Community of Cape Town for their

collaboration, and SANPAD for financial support. Thanksalso to Prangnat Chininthorn, Adinda Freudenthal and to allinterpreters who have helped us with this project. We alsothank Telkom, Cisco, Aria Technologies and THRIP (Tech-nology and Human Resources for Industry Partnership) fortheir financial support via the Telkom Center of Excellence(CoE). This work is based on the research supported in partby the National Research Foundation of South Africa (Grantnumber (UID) 75191). Any opinion findings and conclusionor recommendations expressed in this material are those ofthe authors and therefore the NFR does not accept any lia-bility in this regard.

11. REFERENCES[1] D. Aarons and M. Glaser. A Deaf adult literacy

collective. Stellenbosch Papers in Linguistics, 34:1–18,2002.

[2] Y. Anokwa, N. Ribeka, T. Parikh, G. Borriello, andM. Were. Design of a phone-based clinical decisionsupport system for resource-limited settings. In Proc.ICTD, 2012.

[3] E. Blake, W. Tucker, M. Glaser, and A. Freudenthal.Deaf telephony: Community-based co-design. InInteraction Design: Beyond Human-ComputerInteraction, pages 412–413. Wiley, 2011.

[4] E. Brewer, M. Demmer, M. Ho, R. Honicky, J. Pal,M. Plauche, and S. Surana. The challenges oftechnology research for developing regions. PervasiveComputing, pages 15–23, 2006.

[5] A. Cavender, R. Ladner, and E. Riskin. MobileASL:Intelligibility of sign language video as constrained bymobile phone technology. In Proc. SIGACCESS,page 78, 2006.

[6] P. Chininthorn, W. Tucker, A. Freudenthal, andM. Glaser. Mobile communication tools for a South

African Deaf Patient in a Pharmacy Context. In Proc.IST-Africa, 2012.

[7] S. Cox, M. Nakisa, and M. Lincoln. TESSA, a systemto aid communication with deaf people. In Proc.SIGACCESS, pages 205–212, 2002.

[8] R. Elliott, J. Glauert, J. Kennaway, and I. Marshall.The development of language processing support forthe ViSiCAST project. In Proc. SIGACCESS, pages101–108, 2000.

[9] M. Ghaziasgar and J. Connan. Investigating theintelligibility of synthetic sign language visualizationmethods on mobile phones. In Proc. SAICSIT, pages86–92, 2010.

[10] M. Glaser and W. Tucker. Telecommunicationsbridging between Deaf and hearing users in SouthAfrica. In Proc. CVHI, 2004.

[11] R. Heeks. Information systems and developingcountries: Failure, success, and local improvisations.The Information Society, pages 101–112, 2002.

[12] M. Hilzensauer. Information Technology for DeafPeople, Center for Sign Language and DeafCommunication. University of Klagenfurt, 2006.

[13] M. Jones and G. Marsden. Mobile Interaction Design.Wiley, 2006.

[14] K. Looijesteijn. The design of a Deaf-to-hearingcommunication aid for South Africa. UnpublishedMSc thesis, Delft University of Technology,Netherlands, 2009.

[15] I. Medhi and K. Toyama. Full-context videos forfirst-time, non-literate PC users. In Proc. ICTD, pages140–148, 2007.

[16] M. Mirzaei, S. Ghorshi, and M. Mortazavi. HelpingDeaf and hard-of-hearing people by combiningaugmented reality and speech technologies. In Proc.9th Intl Conf. Disability, Virtual Reality & AssociatedTechnologies, pages 149–158, 2012.

[17] M. Motlhabi, M. Glaser, M. Parker, and W. Tucker.SignSupport: A limited communication domainmobile aid for a Deaf patient at the pharmacy. InProc. SATNAC, 2013.

[18] M. Mutemwa and W. Tucker. A mobileDeaf-to-hearing communication aid for medicaldiagnosis. In Proc. Southern AfricanTelecommunication Networks and ApplicationsConference (SATNAC), pages 379–384, 2010.

[19] South African Pharmacy Council. Good PharmacyPractice in South Africa. Printed By S.A. Council,2010.

[20] T. Starner and A. Pentland. Real-time American SignLanguage recognition from video using HMMs inmotion based recognition. In Proc. 2nd Intl. Conf.Computer Vision, pages 222–243, 1997.

[21] STATSSA. StatsOnine. Technical report, StatisticsSouth Africa, 2013. Available online atwww.statssa.gov.za/keyindicators/keyindicators.asp(accessed 04 September 2013).

[22] J. Watermeyer and C. Penn. Working across languageand culture barriers: Communication skills forpharmacists. Technical report, Johannesburg, SouthAfrican, Printed in Cape Town 2009.