+ All Categories
Home > Documents > [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South)...

[IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South)...

Date post: 23-Dec-2016
Category:
Upload: thad
View: 216 times
Download: 4 times
Share this document with a friend
8
Mobile Music Touch: The Effect of Primary Tasks on Passively Learning Piano Sequences Daniel Kohlsdorf TZI University Bremen Bremen, Germany Email: [email protected] Thad Starner GVU Center and School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332 Email: [email protected] Abstract The Mobile Music Touch (MMT) system allows users to learn to reproduce piano note sequences while performing other tasks. The system consists of a mobile Bluetooth- enabled computing device and a fingerless glove with em- bedded vibrators corresponding to each finger and thumb. Melodies to be learned are played over the user’s head- phones repeatedly. As each note is played, the finger cor- responding to the appropriate piano key is stimulated. Past experiments have shown that users could learn simple note sequences even though they were performing a reading comprehension test. Here, we investigate different primary tasks to determine which, if any, interfere with the Passive Haptic Learning (PHL) effect. In a 12 participant within- subject user study, no overall difference was observed in the number of passive sessions required to learn a random note sequence when users viewed a film, played a memory game, or followed a walking path as their primary task. However, individual differences in scores suggest that the type of pri- mary task may have a greater or lesser effect for a given user. 1 Introduction and related work Many people would like to learn an instrument but are too busy to practice. Others played an instrument in the past but quit due to the effort required to maintain their skills. Yet others may not practice due to a repetitive strain injury. Learning an instrument is a very time consuming activity and often does not fit in the schedule of a working adult. Learning to play a song successfully once is not enough. As soon as the song is learned, forgetting begins. During active learning, a multi-modal combination of audio and haptic cues gives the user a richer understand- ing of musical structure and improves performance of the musical piece [3, 4, 9]. Similarly, haptic feedback has proved effective during active learning for motor skill train- ing [1, 2, 10, 11, 12] and for memory [7, 13]. However learning is not always an active process. Most research describes passive learning as learning that is “caught rather then taught.” Krugman and Hartley charac- terise passive learning as “typically effortless, responsive to animated stimuli, amenable to artificial aid to relaxation and characterized by absence of resistance to what is learned” [8]. What if haptic stimulation could be used to train a user passively, instead of using it for reinforcement during active learning? Perhaps a user can be exposed to such passive training while engaged in their daily routines (e.g. reading e-mail, attending a talk, driving, etc.) and can thus rein- force their skills “automatically.” We term the phenomenon of acquiring motor skills without active attention “Passive Haptic Learning” (PHL). To explore this idea in the past, we have focused on the task of learning note sequences on a pi- ano while performing other tasks. In this paper we describe an experiment where the effects of distractions on PHL are compared. We hypothesize that there will be a difference in the number of sessions required to learn a note sequence passively for each of the different distraction conditions. 2 Mobile Music Touch In our previous work, we described a mobile passive mu- sic instruction system called Mobile Music Touch (MMT) [5, 6]. Mobile Music Touch consists of two distinct parts: fingerless gloves and a Bluetooth-enabled mobile comput- ing device. The gloves are equipped with vibration mo- tors (Precision Microdrives’s model #310-101, see Figure 1), mounted on each knuckle (see Figure 3). A micro controller equipped with Bluetooth is connected to the glove via a ribbon cable (for the controller box see Figure 2). This controller handles communication between
Transcript
Page 1: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

Mobile Music Touch: The Effect of Primary Taskson Passively Learning Piano Sequences

Daniel KohlsdorfTZI

University BremenBremen, Germany

Email: [email protected]

Thad StarnerGVU Center and School of Interactive Computing

Georgia Institute of TechnologyAtlanta, GA 30332

Email: [email protected]

Abstract

The Mobile Music Touch (MMT) system allows users tolearn to reproduce piano note sequences while performingother tasks. The system consists of a mobile Bluetooth-enabled computing device and a fingerless glove with em-bedded vibrators corresponding to each finger and thumb.Melodies to be learned are played over the user’s head-phones repeatedly. As each note is played, the finger cor-responding to the appropriate piano key is stimulated. Pastexperiments have shown that users could learn simple notesequences even though they were performing a readingcomprehension test. Here, we investigate different primarytasks to determine which, if any, interfere with the PassiveHaptic Learning (PHL) effect. In a 12 participant within-subject user study, no overall difference was observed in thenumber of passive sessions required to learn a random notesequence when users viewed a film, played a memory game,or followed a walking path as their primary task. However,individual differences in scores suggest that the type of pri-mary task may have a greater or lesser effect for a givenuser.

1 Introduction and related work

Many people would like to learn an instrument but aretoo busy to practice. Others played an instrument in the pastbut quit due to the effort required to maintain their skills.Yet others may not practice due to a repetitive strain injury.Learning an instrument is a very time consuming activityand often does not fit in the schedule of a working adult.Learning to play a song successfully once is not enough.As soon as the song is learned, forgetting begins.

During active learning, a multi-modal combination ofaudio and haptic cues gives the user a richer understand-ing of musical structure and improves performance of the

musical piece [3, 4, 9]. Similarly, haptic feedback hasproved effective during active learning for motor skill train-ing [1, 2, 10, 11, 12] and for memory [7, 13].

However learning is not always an active process. Mostresearch describes passive learning as learning that is“caught rather then taught.” Krugman and Hartley charac-terise passive learning as “typically effortless, responsive toanimated stimuli, amenable to artificial aid to relaxation andcharacterized by absence of resistance to what is learned”[8].

What if haptic stimulation could be used to train a userpassively, instead of using it for reinforcement during activelearning? Perhaps a user can be exposed to such passivetraining while engaged in their daily routines (e.g. readinge-mail, attending a talk, driving, etc.) and can thus rein-force their skills “automatically.” We term the phenomenonof acquiring motor skills without active attention “PassiveHaptic Learning” (PHL). To explore this idea in the past, wehave focused on the task of learning note sequences on a pi-ano while performing other tasks. In this paper we describean experiment where the effects of distractions on PHL arecompared. We hypothesize that there will be a differencein the number of sessions required to learn a note sequencepassively for each of the different distraction conditions.

2 Mobile Music Touch

In our previous work, we described a mobile passive mu-sic instruction system called Mobile Music Touch (MMT)[5, 6]. Mobile Music Touch consists of two distinct parts:fingerless gloves and a Bluetooth-enabled mobile comput-ing device. The gloves are equipped with vibration mo-tors (Precision Microdrives’s model #310-101, see Figure1), mounted on each knuckle (see Figure 3).

A micro controller equipped with Bluetooth is connectedto the glove via a ribbon cable (for the controller box seeFigure 2). This controller handles communication between

Page 2: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

Figure 1. Vibration motors.

the glove and the mobile device. Here, the mobile deviceis an Openmoko Neo Freerunner mobile phone, which actssimilarly to a MP3 player. A MIDI file containing both theaudio and fingering information (which tone is played withwhich finger) is loaded onto the mobile phone. The finger-ing information is encoded in the MIDI lyrics channel. Inthis way each fingering information is also a MIDI eventwhich occurs to the same time as the note event.

Figure 2. The controller box.

As each note is played, the mobile phone sends com-mands (the fingering events) over Bluetooth to the glovewhich stimulates the finger corresponding to the appropri-ate piano key and with the length of the corresponding note.The commands are one byte in which the first 5 bits aremapped to each finger. The first bit indicates if the rightthumb should be vibrating and the fifth bit if the right smallfinger should be vibrating. For example, if the small fin-ger and the middle finger should be vibrating, the commandwould be 00101 = 5.

The phone continuously repeats the song (with fingerstimulation) during passive learning trials. Participantswear an earphone while listening to MMT’s audio.

3 Previous experiments

In a pilot study [5], four subjects learned portions ofAmazing Grace and Jingle Bells (the “dashing through thesnow...” verse, not the chorus) using the MMT system ac-tively. The song sequences involved 44 notes and 45 notes,

Figure 3. Mounting locations of the vibrators.

respectively, and were played with one hand. In active modewith MMT, each tone is played by a MIDI controlled Ca-sio keyboard while each key is lit by an embedded light-emitting diode (LED) and the appropriate finger is stim-ulated by a vibrator. During active mode, MMT wouldplay each passage, and the user would attempt to repeatthe passage. The process continued until each passage wasplayed correctly. Passages were approximately 10-15 notesin length. Thus, four passages were learned per song, and,after learning the last phrase, the subjects were required toplay the whole song correctly. After participants could playthe notes of the song without error once (errors in rhythmwere permitted), they progressed to the passive portion ofthe study. The participants then spent 30 minutes doing adaily task of their choice. During this time, the participantslistened to both songs playing in a loop, but only one of thetwo songs provided tactile stimulation. Afterwards, the par-ticipants attempted to play both songs again. The subjectsmade significantly fewer errors in playing the note sequencewhen playing the song with tactile stimulation (0.750) thanthe one without (5.75). These results suggest that the MMTcan be used for passively rehearsing a song while perform-ing other tasks.

A subsequent 16 participant within-subject study showedthat participants could learn a note sequence passively whileperforming a reading comprehension test [6]. This study fo-cused on internal validity in its design. To avoid any priorknowledge of the experimental song, two 10-note werecomposed. Both songs were played with the right hand.To avoid any complications of learning lateral hand move-ment, the songs were designed using the notes C throughG, so that no movement was required and each finger wasuniquely mapped to a note. Participants heard/saw/felt asong once and then tried to reproduce it. Errors in note se-

Page 3: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

Figure 4. The Mobile Music Touch System: 1)glove. 2) controller box. C-G) tones mappedto fingers.

quence and rhythm were recorded. Next, the participantsspent 30 minutes listening to the song and, if in the tactilecondition, feeling the notes on their fingers. However, thesubjects were required to perform a reading comprehensionexam, designed such that it would not be completed beforethe end of the 30 minutes. The participants then attemptedto play the song again. The procedure was repeated forthe second song, but in the audio-only or audio plus tactilecondition, whichever the participant had not yet completed.Conditions were balanced with respect to which song wasplayed first and which song was assigned to the tactile con-dition. Order was randomized with respect to subject. Thesubjects made significantly fewer errors during the tactilecondition (ave=1.44), with half making no errors. In theaudio-only condition, subjects averaged 9.06 errors. Theparticipants also more closely matched the song’s rhythmin the tactile condition, though the results were not statisti-cally significant. Unlike the pilot study, this study focusedon learning a note sequence instead of rehearsing it after itwas initially learned.

In the same paper [6], a third study attempted to dis-cover if the amount of time a subject required to learn a10-note sequence actively could predict the number of 20-minute passive sessions that would be required to learn asimilar 10-note sequence composed from quarter and eightsnotes. During active training, subjects would watch/feel thepassage played on the piano and then attempt to repeat it.This procedure was repeated until no errors were made inplaying the passage. The same procedure was used duringpassive training, but after each failure the subjects wouldperform a task for 20-minutes while the passage played intheir headset and on their fingers. Both note sequences

were randomly generated and were created so that each fin-ger mapped uniquely to each piano key. No hand move-ment was required. The result was a noticeably amusicaltune. Interestingly, in piloting, participants with a musicalbackground had difficulty in learning the note sequencespassively. We decided to recruit an equal number of sub-jects with a musical background and without — five in eachgroup. The musicians required significantly more passivesessions to learn the passage than the non-musicians. Yetthe musicians required, on average, less active repetitionsto play the passages than the non-musicians. Interestingly,three musicians quit the passive part of the study before re-producing the 10-note sequence correctly, spending an av-erage of 160 minutes in the attempt before quitting. How-ever, non-musicians required an average of 64 minutes ofpassive learning before reproducing the note sequence cor-rectly. For non-musicians there was a weak correlation be-tween the number of active and passive repetitions required.

While performing these studies we began to questionunder what conditions is passive haptic learning inhibited.In this study participants with a musical background wereomitted because we wanted to explore the effects of differ-ent distractions and not the effects of song structures. Inaddition, novice musicians are the obvious audience for acommerical version of MMT.

4 Metrics

When judging a participant’s performance, both theaccuracy of playing the notes in sequence and the rhythmis recorded. The user’s performance is recorded by thestudy software and saved in MIDI format. Following theISO standard for accuracy as used in speech recognition,insertions, deletions, and note substitutions are countedagainst note accuracy. Consider the following example:

O r i g i n a l : B D E C G F D D − F E CPer fo rmed : B E E C − F D D C F E C

In this case there are three errors. There is one substitu-tion (D → E), one deletion (missing G) and an insertion(extra C). In the studies presented later in this paper, sub-jects continued their passive training until their note accu-racy reached 100%.

Rhythm performance is more difficult to judge. UsingDynamic Time Warping (DTW), we compare participants’performances against the reference performance played bythe system before each attempt at reproducing the passage.The cost function for two given sequences A and B whereA[i] indicates the ith note of the sequence A and B[j]indicates the jth note in sequence B, is as follows:

Page 4: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

i Cost of matching A[i] with B[j] is given bycost(i, j) = abs(duration(A[i])− duration(B[j]))

ii Cost of a gap in A = duration(B[j])

iii Cost of a gap in B = duration(A[i])

One can think of DTW as an algorithm which computes theeffort to change a sequence to another one. The durationsare measured in milliseconds, and the error is the sum of allcosts. While rehearsing a song, users often slowly producethe notes or hold one note until they can remember whichnote is next. Thus, rhythm accuracy may be a valuable,but overly harsh, metric when looking at initial learning bynovices.

Figure 5. A user practicing actively with theMMT during the pilot experiment.

5 Experiment examining different primarytasks’ effects on PHL

Here, we describe a new within-subject, counterbalancedstudy designed to examine the effect of different primarytasks on the learning effect during passive piano learning.We selected three primary tasks which might extinguish thepassive haptic learning effect. These tasks involve body mo-tion, active memory and audio-visual stimulation.

The testing methods and the passive learning session arevery similar to the third study described above. The studywas performed by 12 subjects, each learning a randomlygenerated 10-note song while performing each of three pri-mary tasks: watching a film, playing a memory game, orwalking a path. These three conditions seemed good prox-ies for activities in everyday life (relaxing, thinking, andwalking).

We hypothesize that there will be a difference in thenumber of sessions required to learn a note sequence pas-sively for each of the different distraction conditions. More

precisely, we expected the walking path might involve somuch motor and somatosensory activation in the brain thatthe tactile sensations of the glove used in the experimentwould be masked. We also hypothesized that actively exer-cising the user’s working memory might interfere with pas-sive learning. The film condition was expected to be leastdemanding and was included as it is a situation where weexpect users might want to perform passive haptic learning.We expected the PHL effect to be fastest in this condition.

The subjects were selected for little to no musical back-ground and had an age range from 21 - 44. The pas-sage was designed so that each tone mapped directly to aunique finger, and the user did not need to move his handto play the passage. As in the third study, subjects first lis-tened/watched as the keyboard demonstrated the passage torepeat. The subject then tried to repeat the 10-note passage.If unsuccessful, the subject would perform the primary taskfor five minutes while MMT played the passage and stimu-lated the subject’s fingers appropriately. At the end of fiveminutes, the user would repeat the process until he suc-cessfully played all the notes in the correct sequence andwithout any extraneous notes (i.e. 100% accuracy). Aftereach condition was complete, the users completed a NASATask Load Index (TLX) survey to measure relative subjec-tive workload.

The walking path was designed so that one iterationcould be completed in approximately five minutes. The pathwas inside a laboratory building and included two doors theparticipants had to open (with a RFID card) and two setsof stairs (one up and one down). Subjects were followedby a supervisor to navigate them through the building. Thesupervisor used phrases such as

1. Follow the floor and turn to the right.

2. Now open the door and walk downstairs.

3. Let’s walk back to the lab.

There was no performance rewards used in the walking taskas the user had to attend the task to navigate and completethe course in time. However, performance rewards weregiven in the film and game conditions below to focus theparticipants’ attention on their primary task.

Figure 6 shows the memory game. To insure that sub-jects focused on the task, they were told that their paywould be based on the average number of memory gamesthey completed during their five minute sessions. The ba-sic payment was 5 euros. If the subjects completed morethan seven games, they received 7 euros. If they completedover 10 games, they received a maximum payment of 8 eu-ros. However, at the end of the study, all subjects were paidequally (to their surprise). Some declined pay.

During the film watching condition, users watched “StarTrek 9: Insurrection.” The subjects were told their pay

Page 5: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

Figure 6. Memory game.

would be based on their accuracy in answering questionsabout the film. Questions included

1. Who is attacking the village? (Answer: Data)

2. What is the name of the admiral? (Answer:Dougherty)

3. In which system should the Enterprise help using con-flict? (Answer: the Goren System)

After each attempt at playing a given note sequence, noteaccuracy and deviation from rhythm in milliseconds wasrecorded. Memory game performance, film answers, num-ber of five minute sessions required to complete each con-dition, and NASA TLX scores were also recorded.

6 Results

6.1 Sessions required per condition

Our primary metric of interest is the number of passivesessions needed to perform the songs with zero sequenceerrors. The participants’ performances are listed in Ta-ble 1. There was no statistically significant difference be-tween the conditions. However, the data seem to reflectthat some subjects had more difficulty with some conditionsthan others. Perhaps there exist individual differences asto which primary task affects passive haptic learning? Weran paired two-tailed t-tests comparing subjects’ best versustheir worst and middle conditions. All pairs of comparisonswere significant.

#user film game walking1 1 2 12 2 2 23 1 2 54 3 3 35 7 4 36 1 7 37 2 2 48 2 3 39 1 4 510 2 3 311 1 1 112 2 1 3mean 2 2.8 3std 1.6 1.6 1.2

Table 1. Passive sessions required per condi-tion

i Best: mean = 1.66, std = 0.77

ii Middle: mean = 2.5, std = 1

iii Worst: mean = 3.75, std = 1.91

i Best vs middle: p = 0.005

ii Best vs worst: p = 0.001

iii Middle vs worst: p = 0.005

One hypothesis is that this result is due to a learning ef-fect. We performed a Pearson correlation on condition or-der versus sessions required for each subject and checkedfor significance using a standard one-sample t-test. No sig-nificant correlation was found (p = 0.41).

6.2 Rhythm

We measured subjects’ improvement in reproducing therhythm of the trained note passage between the first attemptat playing versus the last. The improvement is calculatedas dtwDistancestart − dtwDistancelast. The results inrhythm are listed in Table 2.

Similar to the evaluation in number of sessions needed,we compare each condition against all others. The p valueswere computed by a paired, two-sided t-test.

i Film vs game: p = 0.012

ii Film vs walk: p = 0.157

iii Walk vs game: p = 0.34

Page 6: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

#user film [ms] game [ms] walking [ms]1 124 1115 -15282 1636 926 13663 40 3921 4784 1145 816 34345 43 619 10066 5 1625 10297 157 -525 16678 625 1608 11019 1230 3263 265010 2029 4539 226111 3746 5855 200112 6 1134 2213mean 898 2074 1469std 1143 1887 1249

Table 2. Improvement in rhythm

After Bonferonni correction, the game condition seems tobe statistically different than the film condition (p = 0.036).Surprisingly, the improvement in rhythm during the gamecondition is significantly higher than in the film condition!Interestingly, rhythm improvement was not correlated withthe number of sessions required to complete a condition.

Again, if the individual subject’s best rhythm improve-ment results are taken as a class against their worst condi-tion, statistically significant differences appear. Perhaps im-provements in rhythm differ from user per condition. Therewas no statistically significant correlation between condi-tion order and rhythm score.

i Best: mean = 472ms, std = 1021ms

ii Middle: mean = 1317ms, std = 1075ms

iii Worst: mean = 12, 662ms, std = 1527ms

i Best vs avg: p = 0.0001

ii Best vs worst: p = 0.000012

iii Avg vs worst: p = 0.0006

6.3 Memory game results

Table 3 shows the number of memory games finished persession. There was little correlation (r2 = 0.078) betweenthe average number of games finished per session and thenumber of sessions required to complete the task.

6.4 Film results

Table 4 shows the participants’ accuracy in answeringthe comprehension questions about the film. Note that ten

#user games sessions avg gamesfinished needed per session

1 6 2 32 4 2 23 3 2 1.54 11 3 3.65 7 4 1.76 16 7 2.27 10 2 58 6 3 29 14 4 3.510 9 3 311 4 1 412 3 1 3

Table 3. Performance in memory game

questions were asked after the participant had completed allsessions for the condition (i.e., they successfully reproducedthe 10-note sequence). The questions were selected basedon how much of the movie had been viewed. There wasmoderate correlation (r2 = 0.49) between quiz accuracyand the number of sessions required to complete the task.

#user sessions correctly % correctneeded answered questions

1 1 6 60 %2 2 7 70 %3 1 4 40 %4 3 8 80 %5 7 10 100 %6 1 5 50 %7 2 8 80 %8 2 6 60 %9 1 2 20 %10 2 5 50 %11 1 3 30 %12 2 7 70 %

Table 4. Performance for comprehensionquestions in film condition

6.5 NASA TLX

As shown in Table 5, there was a small correlation be-tween a subject’s TLX rating for the walking condition andthe number of sessions required for the subject to completethat condition. However, there was almost no correlationseen for this comparison in the other conditions. In addi-tion, no correlation was found across conditions betweenthe TLX scores and the rhythm improvement scores. How-

Page 7: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

user # film game walking[TLX] [TLX] [TLX]

1 23 52 442 27 63 303 50 57 394 71 95 675 80 74 486 41 20 327 61 88 818 27 91 599 25 27 1910 34 22 2811 55 32 2912 44 34 66mean 45 54 45r2 value compared to# sessions required 0.0064 0.04 0.43

Table 5. NASA TLX results

ever, each user again has his own best, average, or worstcondition.

i Best vs Middle: p = 0.018

ii Best vs Worst: p = 0.00043

iii Middle vs Worst: p = 0.0003

7 Discussion and Future Work

Contrary to expectations, all subjects were able to learnthe note sequences in all conditions, and no condition seemssignificantly worse than any other as to its effect on pas-sive haptic learning. In addition, our subjects rated thesubjective workload between the conditions to be similar.While unexpected, these results indicate good news — pas-sive haptic learning may be able occur in a wide variety ofeveryday situations!

Even so, our post-hoc analysis suggests that users mayhave individual differences as to which tasks interfere morewith passive haptic learning and increase their sense ofworkload. Yet these individual differences do not seem tocorrelate between the overall workload scores and the num-ber of passive sessions required. A more focused experi-ment exploring this potential effect is needed.

Subjects seemed to attend the primary tasks well (i.e.,they did not seem to “cheat” by overtly attending the pas-sive task), and there was no correlation between the per-formance metric on the memory game and the number ofsessions required to finish the passive learning task. How-ever, there was moderate correlation between the number ofquestions answered correctly in the film condition and the

number of passive sessions required. Perhaps subjects whoattended the film more carefully required more sessions tocomplete the passive task. Or perhaps the film questions goteasier the more the subjects watched the film, due to greaterexposure to the story or due to the choice of questions usedlater in the film. This result indicates a possibility for a moreprecise study in the future.

The higher improvement in rhythm during the memorygame condition over the film condition may be due to thegreater number of sessions performed in the game condi-tion. However, rhythm improvement was not correlatedwith the number of passive sessions required to complete acondition. Also possibly contradicting this hypothesis, thewalking condition required, on average, even more sessionsthan the game condition and had rhythm improvements onpar with the film condition. An alternative explanation maycome from an observation made during a previous study.One user used a text editor during his passive sessions andremarked that he synchronized his keystrokes to the vibra-tions in the glove. He said that this technique helped himlearn the song and did not distract him from his primarytask. In this experiment, all subjects used their right (MMTglove) hand to interact with the mouse during the mem-ory game. Perhaps they, consciously or not, reinforced theglove’s cues with pressing the mouse button in synchrony.If so, such mixing of the “active” and “passive” tasks maybe a reasonable strategy for learning a manual task whileperforming other duties. Such serendipitous haptic learningdeserves future study.

All of the experiments above focus on exploring the pas-sive haptic learning effect in short-term experiments usingshort sequences. If PHL is to be an effective means forlearning or rehearsing the “muscle memory” of full songs,we need to create gloves for both hands and create a sys-tem where phrases of longer songs are learned and thenconnected to form a whole. Can a complicated melody belearned or rehearsed efficiently with PHL? Can the moresubtle effects of music be trained passively? Will forget-ting happen faster or slower once a song has been learnedpassively versus actively? Can PHL be applied to other do-mains, such as learning to type, sign, or control a compli-cated manual interface? Is “passive learning” an artifact ofsubjects quickly switching between tasks (multiplexing) orare subjects truly ignoring the passive task as they performthe primary (multitasking)? Our previous studies showedthat subjects performed equally well on primary tasks withand without tactile stimulation, but perhaps fMRI scans dur-ing PHL experiments might provide more evidence as tohow PHL occurs. Furthermore, perhaps users could wear arecording mobile EEG and video camera and use the PHLsystem over a normal day. Given a clean enough EEG sig-nal, we could then post-process the data focusing on the ar-eas related to PHL (hopefully already discovered by fMRI)

Page 8: [IEEE 2010 International Symposium on Wearable Computers (ISWC) - Seoul, Korea (South) (2010.10.10-2010.10.13)] International Symposium on Wearable Computers (ISWC) 2010 - Mobile Music

and try to determine which daily activities might interferewith it. Investigations such as these will help determine ifPHL is an intellectual curiousity or an effective new tool fortraining.

8 Conclusion

Twelve participants successfully learned 10-note se-quences through “Passive Haptic Learning” while activelyplaying a memory game, walking through two levels of alaboratory, and watching a film. On average, no activityseemed to affect the participants’ learning rates significantlymore than another, and subjective workload ratings weresimilar across conditions. However, post-hoc analysis sug-gests that further explorations should be performed to exam-ine individual differences as to the effect of primary tasks onPHL. The results presented here and in previous work con-tinue to suggest that PHL may be a useful tool for trainingmanual tasks.

9 Acknowledgements

The authors wish to thank Professors Mark Guzdial andRichard Catrambone for their user study suggestions. Thismaterial is based upon work supported, in part, by the Na-tional Science Foundation under Grant No. 0812281 andthe NIDRR’s Wireless RERC. The Rehabilitation Engineer-ing Research Center for Wireless Technologies is sponsoredby the National Institute on Disability and RehabilitationResearch (NIDRR) of the U.S. Department of Educationunder grant number H133E060061. The opinions containedin this paper are those of the author and do not necessar-ily reflect those of the U.S. Department of Education orNIDRR.

References

[1] R. J. Adams, D. Klowden, and B. Hannaford. Virtual train-ing for a manual assembly task. Haptics-e, 2(2), 2001.

[2] D. Feygin, M. Keehner, and F. Tendick. Haptic guidance:Experimental evaluation of a haptic training. In IEEE Hap-tics Symposium, pages 40–47, 2002.

[3] G. Grindlay. Haptic guidance benefits musical motor learn-ing. In HAPTICS ’08: Proceedings of the 2008 Symposiumon Haptic Interfaces for Virtual Environment and Teleoper-ator Systems, pages 397–404, Washington, DC, USA, 2008.IEEE Computer Society.

[4] G. C. Grindlay. The impact of haptic guidance on musicalmotor learning. Master’s thesis, Massachusetts Institute ofTechnology, 2007.

[5] K. Huang, T. Starner, and E. Do. Mobile music touch: Mo-bile haptic stimulation for passive learning and rehabilita-tion. ISWC, 2008.

[6] K. Huang, T. Starner, E. Do, G. Weiberg, D. Kohlsdorf,C. Ahlrichs, and R. Leibrandt. Mobile music touch: mo-bile tactile stimulation for passive learning. In CHI ’10:Proceedings of the 28th international conference on Humanfactors in computing systems, pages 791–800, New York,NY, USA, 2010. ACM.

[7] M. Jones, A. Bokinsky, T. Tretter, and A. Negishi. Acomparison of learning with haptic and visual modalities.Haptics-e, 3(6), May 2005.

[8] H. E. Krugman and E. L. Hartley. Passive learning fromtelevision. The Public Opinion Quarterly, Oxford UniversityPress, 34(2):pp. 184–190, 1970.

[9] C. E. Lewiston. MaGKeyS: A haptic guidance keyboard sys-tem for facilitating sensorimotor training and rehabilitation.PhD thesis, MIT, February 2009.

[10] D. Morris, H. Tan, F. Barbagli, T. Chang, and K. Salisbury.Haptic feedback enhances force skill learning. In WHC ’07:Proceedings of the Second Joint EuroHaptics Conferenceand Symposium on Haptic Interfaces for Virtual Environ-ment and Teleoperator Systems, pages 21–26, Washington,DC, USA, 2007. IEEE Computer Society.

[11] J. Patton and F. Mussa-Ivaldi. Robot-assisted adaptive train-ing: Custom force fields for teaching movement patterns.IEEE Transactions on Biomedical Engineering, 51(4):636–646, 2004.

[12] G. Srimathveeravalli and K. Thenkurussi. Motor skill train-ing assistance using haptic attributes. In WHC ’05: Proceed-ings of the First Joint Eurohaptics Conference and Sympo-sium on Haptic Interfaces for Virtual Environment and Tele-operator Systems, pages 452–457, Washington, DC, USA,2005. IEEE Computer Society.

[13] X.-D. Yang, W. F. Bischof, and P. Boulanger. Validating theperformance of haptic motor skill training. In HAPTICS ’08:Proceedings of the 2008 Symposium on Haptic Interfaces forVirtual Environment and Teleoperator Systems, pages 129–135, Washington, DC, USA, 2008. IEEE Computer Society.


Recommended