Passive Haptic Learning of Typing Skills Facilitated by Wearable Computers

Abstract Passive Haptic Learning (PHL) allows people to learn “muscle memory” through vibration stimuli without devoting attention to the stimulus. PHL can be facilitated by wearable computers such as gloves with an embedded tactile interface. Previous work on PHL taught users rote patterns of finger movements corresponding to piano melodies. Expanding on this research, we are currently exploring the capabilities and limits of Passive Haptic Learning as we investigate whether more complex skills and meaning can be taught through wearable, tactile interfaces. We are creating and studying a system for passively teaching typing skills, with the ultimate goal of passively teaching Braille typing. Our initial studies in perception and learning provide key information for system development including the importance of visual feedback in learning to type; while our pilot study using the current system for Passive Haptic Learning of typing on an unfamiliar keyboard shows passive learning in all participants.

Author Keywords Haptic; tactile; typing; wearable; passive training; learning; PHL. ACM Classification Keywords H.5.2 Information Interfaces and Presentation: Miscellaneous.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s). CHI 2014, Apr 26 - May 01 2014, Toronto, ON, Canada ACM 978-1-4503-2474-8/14/04. http://dx.doi.org/10.1145/2559206.2581329

Caitlyn Seim College of Computing, 85 5th St., TSRB 338 Georgia Institute of Technology Atlanta, GA 30332-0760 ceseim@gatech.edu David Quigley College of Computing, 85 5th St., TSRB 338 Georgia Institute of Technology Atlanta, GA 30332-0760 quigs@gatech.edu Thad Starner College of Computing, 85 5th St., TSRB 338 Georgia Institute of Technology Atlanta, GA 30332-0760 thad@gatech.edu

Introduction Haptic systems can help users learn manual tasks [1, 2, 3]. One convenient method of creating a tactile interface is to insert tactors into a glove. The vibration motors are placed at the bases of the fingers and a computing system controls which fingers receive vibration stimulation. Markow et al. [7] demonstrated that such a wearable, tactile interface system can facilitate Passive Haptic Learning (PHL) and Passive Haptic Rehabilitation (PHR). Passive Haptic Learning is a phenomenon where users can learn through tactile stimulation without devoting active attention to the stimulus. Passive learning is caught, rather than taught, and is typically effortless [4]. Gloves for PHL have been found to be an effective learning tool for manual dexterity skills of the fingers. The previous work on Passive Haptic Learning, the Mobile Music Touch (MMT) project, focused on teaching rote order tasks, such as the order of notes for a piano melody [7]. Now, with the interest of exploring the capabilities and limits of PHL, we are investigating, creating, and studying a system to facilitate Passive Haptic Learning of typing skills. This work has the ultimate goal of lowering barriers to learning Braille typing and will better define the abilities of a wearable, tactile interface in passively teaching users. We are currently researching human perception of vibration stimuli, the importance of visual feedback in active haptic learning, and Passive Haptic Learning of typing on an unfamiliar keyboard. Work conducted thus far defines some important needs of the system and shows successful Passive Haptic Learning in three subjects. Background and Motivation Previous work has established Passive Haptic Learning for rote muscle movement patterns of the fingers on one hand. The Mobile Music Touch (MMT) project focused on passively teaching a pattern of keys for a piano melody using Gloves for PHL. The piano student wore the glove while doing other tasks, such as reading

email, taking a test, or watching a video. The MMT glove played the song to be learned and stimulated the appropriate finger for each note. The student could ignore the glove, even while performing distracting tasks, and yet learning still occurs. Studies showed that participants could learn the first 45 notes of simple melodies, such as Amazing Grace, in 30 minutes using this method [7]. We are now expanding research in Passive Haptic Learning to investigate whether more complex systems and tasks can be taught through PHL. Our group is exploring the use of gloves with embedded vibration motors in the fingers for use in Passive Haptic Learning of typing skills. Though this research has significance in the areas of interface design and passive learning - as we discover what, and how, information can be conveyed through passive haptics - this work is also motivated by a specific problem. Over six million people in the United States alone are blind. We are investigating the use of PHL for learning Braille typing, a chorded text entry system. Learning to type the Braille system is time consuming and a major component of rehabilitation and independence training for individuals who are blind or visually impaired. Braille is especially difficult to learn for those who lose their sight later in life, such as the aging population, wounded veterans, and the increasing number of diabetics. Passive learning of Braille typing would reduce the seven or more months typically spent learning at special Blind Rehabilitation Centers, if the patient is fortunate enough to even have access to one of these instructional facilities. Creating a system that passively teaches Braille typing through vibrations stimuli (PHL) would facilitate a reduction in rehab training time by allowing patients to learn while doing other tasks such as cane training, orientation and mobility or even tasks in their daily life or at home. With knowledge of the dot system that comprises Braille and the Braille alphabet, the system

for PHL of typing skills may better help individuals to learn to read Braille as well. The lightweight and user-conscious system that we are developing also would provide access to Braille instruction for those with financial or geographic constraints that prevent them from getting rehabilitation instruction. This research aims not only to explore the subject of Passive Haptic Learning, but to also create this system for Braille instruction. Initial Work and Findings Expanding upon previous research, we now work to show that PHL can be used to teach typing systems, not just key patterns for piano melodies. The next sections discuss our efforts in determining how best to provide PHL for typing and how to create experiments that are suitably sensitive for detecting the effect of PHL on learning a text entry keyboard. The Current System The current system consists of fingerless gloves with embedded vibration motors (one per finger) controlled by a microcontroller, our typing training software that records user data, and two BAT keyboards from InfoGrip (which form a unique keyboard consisting of only letters A-H, space and enter). Figures 1 and 2 illustrate the system and use. Users learn one phrase at a time and may encounter several different kinds of sessions: Active Practice sessions (used in our feedback study) consist of audio of a phrase (e.g. “he had”), followed by a carefully devised pattern of vibrations that stimulate the pattern of finger presses that type that phrase on our unique keyboard. Timing of vibrations in the sequence was determined to maximize vibration strength and clear perception (400 ms vibrations on average, with 100 ms pauses). The audio is then played again, and the user attempts to type the pattern into the training software which displays typed letters (see feedback study) and records user statistics.

Passive Haptic Learning (PHL) and control sessions consist of users playing a distracting memory card game while they have audio of the phrase they are learning played for them. Those not in the control group also wear the PHL gloves and feel the phrase’s vibration pattern after each audio clip. Users focus on the distraction task and pay no attention to stimuli. During Testing sessions, participants are played the audio of the phrase that they were to learn (without tactile stimulation). They then attempt to type that phrase. The testing software displays an asterisk for each letter typed so as to minimize any additional learning (see feedback study). Following three tries at typing the phrase, various words from that phrase are also presented via audio in a random order to be typed. Phrases in the current system were chosen to be 15-17 characters in length, the same as the number of notes presented in our piano work [6]. In order to be typed on our unfamiliar keyboard, phrases consist of only the first 8 letters of the alphabet, and were chosen to have meaning for easy recall. Each phrase consists of 4 words, with an average of 3 letters each, and they contain almost similar frequency distributions of letters between A-H (i.e. only missing one of these). They also contain simply-spelled words with minimal homophones (i.e. ‘be’ vs. ‘bee’) as found to be ideal in other audio-prompted typing research [8]. System phrases are depicted in Figure 3. Visual Feedback during Active Typing Practice Some studies that we will be conducting on PHL will contain a session of Active Practice, guided by vibration stimuli. To optimize our system for this portion of the research, some aspects must be studied. During any portions of our studies where subjects must type, our specially designed typing tutor software prompts the subjects with what to type (via audio), records what text they enter, and calculates statistics like average error rate. The software also displays a

Figure 1. A Right and Left BAT keyboard comprise our unique keyboard that is unfamiliar to users. Red and blue keys are not used and serial output is decoded to represent letters A‐H, space and enter. 

Figure 2. Current system in use (Active Practice). A. is the microcontroller that controls the attached gloves’ vibrations motors (C.). B. are the keyboards pictured in Figure 4. D. shows the feedback display (letters condition). 

Figure 3. Phrases in the current system. 

blank screen with our choice of visual feedback. We wondered if the type of visual feedback affected the user’s ability to learn to type. Previous research showed that expert typists perform slightly faster and more accurately with limited visual feedback. We hypothesized that hiding the letters typed during Active Practice might aid in learning as the users would be forced to focus on the audio and the vibration pattern as opposed to the screen. This is a desirable effect when we are allowing users to actively practice typing, a time where learning is encouraged. During Testing sessions, though, we do not want any further learning to occur. In order to better understand whether visual feedback during typing practice does affect active learning, we are studying two conditions of feedback in our system. The feedback conditions that we chose to test are full “letter” feedback and asterisks “stars” only feedback. In the letter feedback condition, users see each character as they enter it. This informative condition allows users to see where they make errors (Figure 4, top). We also test an uninformative “stars” condition where users are shown only an asterisk for each character typed (Figure 4, bottom). Our choice of asterisks as the alternative form of (uninformative) visual feedback comes from the idea that users need some type of visual signal that indicates they are interacting with the keyboard correctly to produce input (i.e. pressing keys sufficiently hard to register, yet not producing repeated keys unintentionally: ‘hhhad’ vs. ‘had). In a study, containing 7 participants thus far, four phrases (from Figure 3, in random order) are presented to be typed on an unfamiliar 8-key keyboard. Audio of a phrase is presented, followed by a pattern of vibrations corresponding to the fingers that type that phrase. The training software then plays the audio again, and participants try to type the phrase. When typing, participants either see the letters they enter or

only asterisks presented on a computer screen, depending on the visual feedback condition they are experiencing. Each user starts in one condition, assigned randomly (users switch to the opposite condition for the 2nd and 3rd phrase). Users repeat the audio-vibration-audio-type process for ten attempts for the first phrase presented (this is the main test - to test learning under that condition). Presumably, after learning to type a phrase in this feedback condition, the user may have learned the finger-to-key mapping. A 2nd phrase is then presented in the same manner (with only three attempts given). In this extra test, the feedback condition is opposite to what was used for the first phrase; this allows us to see how well participants learn a phrase when they have some knowledge of the keyboard, and how well they learned the keyboard under the previous condition. Users then experience a second pair of phrases in the same way (main test, extra test), but under the opposite feedback conditions (e.g. 1st and 2nd were stars then letters, so 3rd – letters, 4th - stars). The keyboard mapping (what finger types what letter) is changed for the second pair of phrases (so participants begin learning a totally new keyboard). An image of the randomized keyboard mapping is presented in Figure 5. The second mapping is mirrored to avoid transfer of learning between the two keyboards while still maintaining the same letter distribution on each hand. Orders of what phrase is presented and feedback formats are counterbalanced. Findings for this study thus far indicate that users who are provided feedback on the letters that they typed, being totally unfamiliar with the keyboard and otherwise only guided by vibration stimuli, actually perform better. These users show better improvements in accuracy from beginning to end, better average (as can be seen in Figure 6 for user performance on the first phrase presented) and final-try accuracy, but lower WPM. Users also become more

Figure 6. Errors in typing the first phrase presented, averaged over 10 attempts. Users presented with informative feedback (the letters they typed) show lower errors on average and better final accuracy. 

Figure 5. Randomized keyboard mappings for our  feedback study.  Top is the standard mapping for our system, which is used in PHL work,

Figure 4. Full “letter” feedback (above): what the user types is displayed to them; versus Asterisks “stars” only feedback (below): ‘*’s are displayed when the user enters a key or Space 

comfortable (with fewer errors overall) from the beginning to the end of the study. Results from this study inform us of the effect of visual feedback on data entry and allow us to better create a system of training for PHL. Passive Haptic Learning of Typing After iterative design modifications and study of our PHL system, we have arrived upon our current system which shows evidence of Passive Haptic Learning. In our previous exploration with PHL for typing, we discovered: Visual prompts may not work for PHL typing –

users may need audio prompts like that with which they learn

Teaching letters and words, randomly presented only a couple times, showed no effect – learning time and information “chunk” size may be pivotal

Teaching chords did not work – our users reported difficulty in distinguishing which fingers were being stimulated

Speed is the wrong metric – accuracy is much more indicative of PHL

After consideration of these previous experimentations, we have made typing prompts audio-based to maintain consistency with learning conditions, and altered the system to teach one phrase at a time. This phrase structure and use of a simple keyboard devotes ample time to learning (an entire 30 min. PHL session), “chunks” information for better learning, and improves on our previous research in that it investigates two-handed learning (as opposed to our efforts on one-handed piano melodies). We are now exploring the effectiveness of our improved system. In a three subject experiment, we presented participants with no Active Practice before placing them into a PHL session (while playing a distracting memory

card game) for a different phrase each. During this learning session, users heard each word in the phrase through headphones, then felt the word’s vibration sequence before the following word’s audio was played. After 30 minutes of Passive Haptic Learning of their phrase, users were tested on the phrase they passively learned, as well as a totally new phrase using the same unfamiliar keyboard. All users were able to type the passively learned phrase with less than 20% errors and maintained the same errors throughout (i.e. “ecc” for “egg”), if any at all. This is a significantly lower error rate than participants typically exhibit during their first 3 tries at typing a phrase (typically ~70%), during our feedback study for example (see Figure 7). Having never practiced typing the phrase before, this level of accuracy seems to indicate that participants successfully passively learned typing across both hands during the PHL session. Users were also consistently near 0% errors when tested on individual words in the phrase. In addition, users were able to figure out how to type the new phrase with less than 20% errors, but at a slightly reduced speed (Figure 8). These findings lead us to infer that these users learned not only the PHL phrase’s rote pattern, but also the mapping of our simple keyboard. These results are a promising beginning to our exploration into Passive Haptic Learning for typing skills. Future Work Moving forward, we will expand the current studies to confirm our findings. Studies in human perception of haptic stimuli at multiple points on the fingers and other relevant research will also be conducted in order to best refine our system and inform future research. We will then implement a formal study of our system for PHL of typing on a unique keyboard. Following successful demonstration of robust passive learning of typing skills, we will design and test efficient mappings between haptic stimuli and key combinations on the Braille keyboard.

Figure 7. Typing errors during test after PHL only (solid lines). Contrast to average errors of participants during first three tries of Active Practice (no previous learning) sessions from feedback study, plotted here for visualization purposes (dashed line).  All users were typing the phrase for the first time (and 2ed and 3rd), on the same keyboard mapping, and were all in the stars feedback condition

Figure 8. Typing errors during test on totally new phrase (solid lines). Contrast to average errors of participants during first three tries of Active Practice (no previous learning) sessions from feedback study, plotted for visualization purposes (dashed line).  All users were typing the phrase for the first time (and 2ed and 3rd) and were all in the stars feedback condition

A subsequent goal of this research is to develop a system that aids in learning Stenography, a phoneme-based, chorded text entry technique used for real-time transcription. Learning Stenography is currently an obstacle to those wanting to enter the field; schools report 85%-95% drop out rates, and even experts must practice for hours a week in order to maintain industry completive speeds [5]. Passive Haptic Learning of stenography would lower the barriers to entry into this industry. With these similarities to Braille typing, alteration of our system to passively teach Stenography, once proven feasible via PHL, will be straightforward. Conclusion With the goal of lowering the barriers to learning Braille typing, we are exploring Passive Haptic Learning of typing skills. While previous research in our group, which established the existence of PHL, focused on teaching rote patterns to one hand, our research explores teaching a system with meaning across both hands with the addition of speech stimuli. Research on the need for visual feedback in learning aids us in system design and informs other researchers working in the areas of haptics, text entry and interfaces. Our initial work on using PHL (facilitated by wearable computers) to teach typing suggests positive results in each subject and transferred learning of the unfamiliar keyboard’s mapping.

Acknowledgements This material is based upon work supported, in part, by the National Science Foundation under grant No. 1217473. References [1] Bluteau, J., Coquillart, S., Payan, Y., and Gentaz, E.

Haptic guidance improves the visuo-manual tracking of trajectories. In PLoS ONE (2008), Vol. 3 Issue 3, p1.

[2] Chellali, A. M., Dumas, C., and Milleville-Pennel, I.

Wyfiwif: A haptic communication paradigm for collaborative motor skills learning. In Web Virtual Reality and Three-Dimensional Worlds (2010).

[3] Eid, M. A., Mansour, M., El Saddik, A. H., and

Iglesias, R. A haptic multimedia handwriting learning system. In Proc. the international workshop on Educational multimedia and multimedia education ((EMME), ACM (2007), 103–108.

[4] Krugman, H. E., and Hartley., E. L. Passive Learning

From Television. The Public Opinion Quarterly 34, 2 (1970), 184–190.

[5] Kuhlin, T. Considering a Career Change to Become a

Court Reporter? Suite101. http://suite101.com/article/considering-a-careerchange-to--become-a-court-reporter-a370134#.UWj_c0nft0x.

[6] Markow, T. T. (2012). Mobile music touch: using

haptic stimulation for passive rehabilitation and learning (Doctoral dissertation, Georgia Institute of Technology).

[7] Markow, T., Ramakrishnan, N., Huang, K., Starner,

T., Eicholtz, M., Garrett, S., Profita, H., Scarlata, A., Schooler, C., Tarun, A., and Backus, D. Mobile music touch: Vibration stimulus in hand rehabilitation. In 4th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) (2010).

[8] Southern, C., Clawson, J., Frey, B., Abowd, G. D.,

Romero, M., "An Evaluation of BrailleTouch: Mobile Touchscreen Text Entry for the Visually Impaired." Mobile HCI 2012. San Francisco, September 2012.