77~72 5en*ry~

f77~72j 5en*ry~

~

4

'4 - K

"4'

7:

K'4 4

"'K' 74 'K'""'.

'V 's"~..

''4 '>44',.KK""

"4

'4 4"' KQ'4K44K."'~""-"

* '44

~ K"K

4 '"'" "~~7" ~K~4'K"

K'> "4' ~4'4'

~4'4"' """

4' ' 'K" '>4' '4

4'

''K,

4''

"'44'

~ ~ '~ K' "K' ,K~ ','4K~' "'4 K" "'K "" , ,,K'4K 'K' "4

~K'' "K"'4'

'4 ~K

'K"' K' "' ' K '4'"'"K

K' 'K''

'4K'" ~ ~K

A K'

K' K

K ' "' ' K

w

348 RLE Progress Report Number 137

Chapter 1. Sensory Communication


Academic and Research Staff

Professor Louis D. Braida, Nathaniel I. Durlach, Professor Richard M. Held, Professor Anuradha M.Annaswamy, Dr. David L. Brock, Dr. Donald K. Eddington, Dr. Susan L. Goldman, Dr. Julie E. Greenberg,Dr. Lynette A. Jones, Dr. Jeng-Feng Lee, Dr. William M. Rabinowitz, Dr. Christine M. Rankovic, Dr. CharlotteM. Reed, Dr. Wendelin L. Sachtler, Dr. J. Kenneth Salisbury, Dr. Barbara G. Shinn-Cunningham, Dr.Mandayam A. Srinivasan, Dr. Anne H. Takeuchi, Dr. Thomas E.v. Wiegand, Dr. David Zeltzer, Dr. Patrick M.Zurek, Lorraine A. Delhorne, Seth M. Hall, Dorrie Hall

Visiting Scientists and Research Affiliates

Dr. Paul Duchnowski, Geoffrey L. Plant, Dr. Matthew Power

Graduate Students

Walter A. Aviles, Gerald L. Beauregard, Maroula Bratakos, Jyh-Shing Chen, Kiran Dandekar, JosephDesloge, Eric M. Foxlin, Joseph A. Frisbie, Rogeve J. Gulati, Rakesh Gupta, Louise Jandura, Jean C.Krause, David S. Lum, Hugh B. Morgenbesser, Philip M. Nadeau, Michael P. O'Connell, Nicholas Pioch,Hong Z. Tan, Daniel P. Welker, Craig B. Zilles

Undergraduate Students

Walter E. Babiec, Stephen V. Baird, James H. Bandy, Susan E. Born, Erika N. Carmel, Gail Denesvich,Ashanthi Gajaweera, Dorrie Hall, Gabrielle Jones, Steingrimur P. Karason, Danielle G. Lemay, David C.Lossos, Jonathan Pfautz, Frederick L. Roby, Tomas Rodriguez-Perez, Jonathan R. Santos, Matthew Sexton,Ranjini Srikantiah, Joseph D. Towles, Lukasz A. Weber, Evan F. Wies

Technical and Support Staff

Ann K. Dix, Eleanora M. Luongo, Mabayoje Tuyo

1.1 Research on Hearing, Hearing Aids,and Tactile Communication of Speech

1.1.1 Hearing Aid Research

SponsorNational Institutes of Health

Contract 2 R01 DC00117

Project StaffProfessor Louis D. Braida, Dr. Paul Duchnowski,Dr. Matthew Power, Dr. Christine M. Rankovic, Dr.Anne H. Takeuchi, Seth M. Hall, Dr. Patrick M.Zurek, Joseph A. Frisbie, Ann K. Dix, Danielle G.Lemay, Matthew Sexton, Ranjini Srikantiah, Jean C.Krause, David S. Lum

Characteristics of Sensorineural HearingImpairment

Although there is no satisfactory model ofsensorineural hearing impairment, several functionalmodels that allow listeners with normal hearing toexperience the perceptual distortions and speech-reception problems of hearing-impaired listeners areemerging. Our past work has documented theeffectiveness of masking noise that elevates thetone-detection thresholds of normal-hearing lis-teners to those of listeners with hearing impair-ments in simulating limitations on speech-receptionassociated with the impairment.' More recently,Duchnowski and Zurek 2 have shown that a type ofmultiband amplitude expansion proposed by

1 P.M. Zurek and L.A. Delhorne, "Consonant Reception in Noise by Listeners With Mild and Moderate Hearing Impairment," J. Acoust.Soc. Am. 82: 1548-1559 (1987).

2 P. Duchnowski and P.M. Zurek, "Villchur Revisited: Another Look at AGC Simulation of Recruiting Hearing Loss," J. Acoust. Soc.Am., forthcoming.

349


Villchur3 is also capable of simulating these limita-tions.

Both the noise and expansion simulations ofhearing loss are addressed nominally at theminimal factors of audibility and abnormal loudnessgrowth. These simulations differ both with respectto phenomenological realism and in the relation ofthe acoustic properties of processed stimuli to thenormal listener's hearing. Current research isdirected towards understanding the ability of thetwo simulation approaches to account for theimpaired perception of nonspeech sounds. To eval-uate the ability of the expansion simulation toreproduce the effects of hearing impairment forsuch sounds, we have developed a version of thesimulator that operates in real time. This simulatoruses a digital signal processor to filter signals intofourteen bands (with widths comparable to criticalbands), estimates the short-term energy in eachband, applies a level-dependent attenuation basedon this estimate, and sums the resulting attenuatedband signals. Measurements4 of simultaneousnarrowband masking patterns and forward-maskedpsychophysical tuning curves indicate that both sim-ulation techniques produce reductions in frequencyselectivity that are similar to those observed in lis-teners with sensorineural hearing impairments (e.g.,Dubno and Schaefer5 ). By contrast, the expansionsimulation preserves the microstructure of thenormal-hearing listener's audiogram, while themasking simulation does not.

Characteristics of the Speech Signal

Token Variability

The goal of this research is to understand the limi-tations on speech intelligibility associated withnatural variations in the utterances of the samespeech sound. Although differences in the acousticrealization of speech elements associated with the

identity of the speaker are typically large enough tohave stable perceptual effects, subtler differencesassociated with context and variability of productionin a fixed context can also be discerned6 and effectthe ability to identify speech sounds, particularlywhen the speech signal is degraded by noise orother distortions. Although the difficulties posed bysuch variation for automatic speech recognitionsystems are increasingly appreciated, their effectson human speech reception have not been studiedsystematically.

To obtain insight into the combined effects of mul-tiple sources of acoustic variability, Takeuchi andBraida 7 measured the ability of listeners to identify/b/-V-/t/ syllables in which the vowel was selectedfrom 9 monophthongs of American English. Tomake the identification task nontrivial, the syllableswere distorted by a multiplicative noise process. Inone stimulus set, the three tokens of each syllablewere produced by a single speaker, but each tokenwas distorted by three different noises. In a secondset, the tokens were produced by three talkers, buteach token was distorted by a single noise. Theeffects of speaker variability were generally greaterthan the effects of token variability or noise vari-ability. Token variability tended to have a greatereffect when the stimulus set varied in distortingnoise than when the stimulus set varied in speaker,suggesting that the degree of variability effectdepends upon the context provided by the otherstimuli in a set. The effects of speaker and tokenvariability, as well as of token and noise variability,were generally additive. Blocking the stimuli byholding one or both sources of variability constantreduced, but did not eliminate, these effects.

Clear Speech

We have initiated a new study of the changes thatoccur in the acoustic speech signal associated withthe attempt to speak clearly. Previous studies (e.g.,

3 E. Villchur, "Electronic Models to Simulate the Effect of Sensory Distortions on Speech Perception by the Deaf," J. Acoust. Soc. Am.62: 665-674 (1977).

4 D.S. Lum, Evaluation of a Hearing Loss Simulator, Advanced Undergraduate Project Report, Dept. of Electr. Eng. and Comput. Sci.,MIT, 1994.

5 J.R. Dubno, and A.B. Schaefer, "Frequency Selectivity for Hearing-Impaired and Broadband-Noise-Masked Normal Listeners," Quart.J. Exp. Psych. 43A: 543-564 (1991).

6 R.M. Uchanski, L.D. Braida, C.M. Reed, and N.I. Durlach, "Speaking Clearly for the Hard of Hearing. IV: Further Studies of the Roleof Speaking Rate," J. Speech Hear. Res., forthcoming.

7 A.H. Takeuchi and L.D. Braida, "Effects of Multiple Sources of Variability on the Accuracy of Vowel Identification," submitted to J.Acoust. Soc. Am.



Uchanski et al.)8 have demonstrated that clearspeech is generally enunciated more slowly thanconversational speech, but the degree to which thereduction in rate is essential to clarity has not beenestablished. To improve our understanding of theimportance of reduced speaking rate to improvedintelligibility, we plan to measure the intelligibility ofspeech produced by "professional" talkers undernine different speaking modes. The talkers will beindividuals, such as debaters, public speakers, andactors who have been trained to adopt speakingstyles that differ from everyday conversationalspeech. The speaking modes will differ with respectto both speaking rate and other speech character-istics. Four of the modes will be at a conversa-tional speaking rate: soft, loud, conversational, andclear speech at conversational rate. Two modeswill be at a quick speaking rate: quick speech andclear speech at quick rates. Three modes will be ata slow rate: slow speech, clear speech, and con-versational speech with pauses between words (asif speaking to a discrete-word automatic speechrecognition system).

Preliminary measurements of the intelligibility ofspeech produced in the traditional conversationaland clear styles suggests that there may be asimple relation between the word score W andaverage speaking rate R (with sentence durationscorrected for pauses between words): W = a - bR,where a but not b depends on the identity of thetalker. Based on this observation we plan to selecttalkers with relatively high values of a and attemptto train them using techniques similar to thosedeveloped by Chen 9 to increase their speaking ratewhile maintaining the intelligibility of clear speechand intelligibility while maintaining the speaking rateof conversational speech.

Models of Speech Intelligibility

Speech Transmission Index

Unlike the Articulation Index (AI), (e.g., Dugal etal. 10) which bases predictions of speech intelligibilityon the long-term power spectrum of speech, theSpeech Transmission Index (STI), (e.g., Houtgast etal. 11) bases predictions on the intensity modulationspectra for individual frequency bands. This dis-tinction is fundamental to the understanding of thedependence of intelligibility on speaking mode.Whereas clear and conversational speech haveroughly the same long-term spectrum, 12 they arelikely to have different modulation spectra becauseof differences in speaking rate and consonant-vowelenergy (CV) ratios. Also the effects ofreverberation and multiband amplitude compressionon the speech signal are likely to be more evidentin the intensity modulation spectra than in thepower spectrum.

Although the STI is typically computed from esti-mates of modulated spectra derived from theoreticalconsiderations or from measurements of modulationtransfer functions made with intensity modulatednoise, we are attempting to base this calculation onmeasurements of real speech waveforms.However, these attempts have been obstructed byartifacts that distort the envelope spectra of speechthat has been degraded by noise and reverberation.Specifically, we observe a "noise floor" in theenvelope spectra if any background noise ispresent. For power spectra of white noise, theexpected power spectral level is proportional to thevariance of the noise. In our case, we computepower spectra of filtered noise envelopes. The vari-ance of the envelopes is inversely proportional tothe bandwidth of the octave filters we use. Wehave confirmed that the noise floors we observevary in level according to the bandwidth of thefilters. We are currently examining various ways toremove or compensate for these noise floor effects.

8 R.M. Uchanski, S. Choi, L.D. Braida, C.M. Reed, and N.I. Durlach, "Speaking Clearly for the Hard of Hearing IV: Further Studies ofthe Role of Speaking Rate," J. Speech Hear. Res., forthcoming.

9 F.R. Chen, Acoustic Characteristics and Intelligibility of Clear and Conversational Speech at the Segmental Level, S.M. thesis, Dept.of Electr. Eng. and Comput. Sci., MIT, 1980.

10 R.L. Dugal, L.D. Braida, and N.I. Durlach, "Implications of Previous Research for the Selection of Frequency-Gain Characteristics," inAcoustical Factors Affecting Hearing Aid Performance and Measurement, eds. G.A. Studebaker and I. Hochberg (New York: Aca-demic Press).

11 T. Houtgast and H.J.M. Steeneken, "A Review of the MTF Concept in Room Acoustics and Its Use for Estimating Speech Intelli-gibility in Auditoria," J. Acoust. Soc. Am. 77: 1069-1077 (1985).

12 M.A. Picheny, N.I. Durlach, and L.D. Braida, "Speaking Clearly for the Hard of Hearing II: Acoustic Characteristics of Clear andConversational Speech," J. Speech Hear. Res. 29(4): 4344-446 (1986).


Even if they work on noise alone, none of the strat-egies tested have resulted in speech in noisespectra as we would expect.

We have also examined some alternative tech-niques to calculate the STI. Ludvigsen et al. 13 havedeveloped a technique in which the squared envel-opes of octave bands of undistorted speech arecorrelated with those of degraded speech using alinear regression analysis. The slope of theregression is used, along with the squaredenvelope means, to compute a transmission indexfor each octave band. When we implemented theiralgorithm, we found that it is extremely sensitive tothe existence of delays between the original andcorrupted speech envelopes and would thus be dif-ficult to apply to reverberant speech. Since, inmany other ways this technique holds promise, wewill continue studying this idea.

Publications

Duchnowski, P., and P.M. Zurek. "Villchur Revis-ited: Another Look at AGC Simulation ofRecruiting Hearing Loss." J. Acoust. Soc. Am.Forthcoming.

Lum, D.S. Evaluation of a Hearing Loss Simulator.Advanced Undergraduate Project Report, Dept.of Electr. Eng. and Comput. Sci., MIT, 1994.

Maxwell, J.A., and P.M. Zurek. "FeedbackReduction in Hearing Aids." IEEE Trans.Speech Audio Proc. Forthcoming.

1.1.2 Enhanced Speechreading

Sponsor

National Institutes of HealthContract 1 R01 DC02032

Project Staff

Professor Louis D. Braida, Dr. Susan L. Goldman,Charlotte M. Reed, Lorraine A. Delhorne, Dr. PaulDuchnowski, Maroula Bratakos, Gabrielle Jones,Mabayoje Tuyo, Philip M. Nadeau, Danielle G.Lemay, Matthew Sexton

Basic Studies of Audiovisual Integration

The McGurk effect 14 demonstrates that visual lipmovements can influence perception of auditoryspeech even when auditory and visual cues are inconflict. For example, when auditory /ma/ isdubbed onto visual lip movements of /ta/, theauditory-visual stimulus is often perceived as "na."Whereas this effect has been found to be robust forAmerican perceivers, it is generally much weakerfor Japanese perceivers when the stimuli areproduced by Japanese speakers.'5 Although thisfinding suggests that linguistic and/or culturalfactors affect the integration process, it may alsoreflect the properties of the utterances of the singleJapanese speaker who produced the utterancesused in these studies.

We conducted an experiment with new stimuli toexplore this issue. These stimuli consisted of utter-ances of the syllables /ba,da,ga,pa,ta,ka,ma,na/produced by two Japanese and two Americanspeakers. Recorded sound and videotaped lipmovements from each speaker were combined tocreate 16 pairs, half of which consisted ofconcordant AV components (e.g., A=/ba/, V=/ba/)the remainder consisted of discrepant components(e.g., A=/ba/, V=/ga/). Since Japanese vowels aretypically shorter than English vowels, the Japanesespeakers were instructed to produce sounds withthe same duration as those produced by theEnglish speakers. This is in contrast to our pre-vious study in which the American speaker wasinstructed to produce shorter vowels. The slowerJapanese articulations were more easily lipreadthan those used in earlier studies.

The stimuli were presented in auditory-visual (AV),visual only (V), and auditory only (A) conditions.Subjects wrote what they heard (AV and A condi-tions), or what they thought the speaker was saying(V condition), with no restrictions on the responseset. The size of the McGurk effect was estimatedas the increase in percentage of auditory place ofarticulation errors (labial versus nonlabial) causedby discrepant visual cues. Results indicated thatalthough the group difference was replicated, theJapanese subjects showed a stronger McGurkeffect than before for auditory labials, although

13 C. Ludvigen, Acta Otol. Suppl 469: 190-195 (1990).

14 H. McGurk and J. MacDonald, "Hearing Lips and Seeing Voices," Nature 264: 746-748 (1976).

15 K. Sekiyama and Y. Tohkura, "McGurk Effect in Non-English Listeners," J. Acoust. Soc. Am., 90: 1797-1805 (1991); K. Sekiyama,"Differences in Auditory-Visual Speech Perception Between Japanese and Americans," J. Acoust. Soc. Japan (E) 15: 143-158(1994).



there were large individual differences. Unlike ourprevious study, 16 there was no difference betweenthe Japanese and English stimuli. The strong effectof the visual stimulus on Japanese subjectsappears to be related to the auditory ambiguity andvisual robustness of the stimuli for these listeners.For the American perceivers, on the other hand, thesize of the McGurk effect was comparable to thatfound previously and relatively unaffected by theambiguity of the auditory stimuli.

Supplements Based on Automatic SpeechRecognition

The goal of this work is to develop systems forproducing and displaying discrete speechreadingsupplements that can be derived from the acousticsignal by speech recognition technology. Currentefforts are focused on supplements related to theManual Cued Speech System used in some educa-tional and communication settings by deaf individ-uals.

Although previous studies of the reception ofManual Cued Speech and of the performance ofASR systems suggests that current technology mayprovide an adequate basis for the design of anautomatic cueing system,17 the effects of speechrecognition errors and delays associated with therecognition process on cue reception and inte-gration are not well understood. To estimate theeffects of such imperfections, we have initiated astudy of the reception of synthetically generatedcues characterized by errors and delays. The visualcues are static handshapes that correspond to thenominal shapes and positions of manual cuedspeech but lack fluid articulation. These cues aredubbed onto video images of a face that wererecorded simultaneously with the acoustic speechsignal. In preliminary tests on one skilled receiver ofManual Cued Speech, scores for words in IEEEsentences were 48 percent for speechreadingalone, 92 percent for Manual Cued Speech, 84

percent for error-free static handshapes, and 72-74percent for cues derived from phones recognizedwith 10-20 percent errors. Although scores forerror-free cues are lower than for manual cuedspeech, this may result from the reduced speakingrate that accompanies cue production (100 versus150 wpm). The scores obtained with cues derivedfrom a simulation of recognition errors are suffi-ciently encouraging to warrant further study of thisapproach.

Supplements Based on Signal Processing

Listeners who have severe to profound hearingimpairments typically rely heavily on speechreadingto communicate and often receive little benefit fromconventional hearing aids in the absence ofspeechreading. We are attempting to develop aidsthat present simplified representations of thespeech waveform acoustically to the impaired ear.These simplified signals consist of tones that areamplitude modulated by the amplitude envelopes offiltered bands of speech. Research in this area isfocused on the effect of background noise on thebenefit provided by the signals, and with perceptualinterference between pairs of modulated tones.

Envelopes Derived from Noisy Speech

The amplitude envelopes of filtered bands ofspeech, when derived from clean speech and pre-sented as amplitude modulations of tones locatedat the centers of the bands, substantially improvethe speechreading ability of listeners with normalhearing. 18 To assess the effectiveness of these sup-plements when the band-envelopes are extractedfrom noisy speech, Lemay and Braida1 9 tested fouryoung adults with normal hearing and superiorspeechreading abilities on word reception in bothhighly contextual (CUNY) and low-context 'sen-tences (IEEE). The interference was additiveGaussian noise filtered to have a speech-like powerspectrum. The noisy speech was filtered into

16 K. Sekiyama, "Differences in Auditory-Visual Speech Perception143-158 (1994).

Between Japanese and Americans," J. Acoust. Soc. Japan E 15:

17 R.M. Uchanski, L.A. Delhorne, A.K. Dix, L.D. Braida, C.M. Reed, and N.I. Durlach, "Automatic Speech Recognition to Aid the HearingImpaired. Prospects for the Automatic Generation of Cued Speech," J. Rehab. Res. and Dev. 31: 20-41 (1994).

18 M. Breeuwer and R. Plomp, "Speechreading Supplemented With Frequency Selective Sound-Pressure Information," J. Acoust. Soc.Am. 76: 686-691 (1984); M. Breeuwer and R. Plomp, "Speechreading Supplemented With Auditorily Presented Speech Parameters,"J. Acoust. Soc. Am. 79: 481-499 (1986); K.W. Grant, L.D. Braida, and R.J. Renn, "Single Band Amplitude Envelope Cues as an Aidto Speechreading," Quart. J. Exp. Psych. 43: 621-645 (1991); K.W. Grant, L.D. Braida, and R.J. Renn, "Auditory Supplements toSpeechreading: Combining Amplitude Envelope Cues From Different Spectral Regions of Speech," J. Acoust. Soc. Am. 95:

1065-1073 (1994).

19 D.G. Lemay and L.D. Braida, "Band Envelope Speechreading Supplements Derived From Noisy Speech," J. Acoust. Soc. Am. 95:

3014 (1994).

353


octave bands centered at 500, 1600, and 3300 Hz,fullwave rectified, and low pass filtered (50 Hzbandwidth) to create band-envelope signals. Theresulting low-bandwidth signals were then used toamplitude-modulate one or more tones centered inthe passbands.

For the CUNY sentences, word intelligibility scoresincreased from 31.4 percent for speechreadingalone to 67.4 percent, 68.3 percent, and 47.8percent when single band supplements derivedfrom noiseless speech were presented at 500,1600, and 3300 Hz respectively, and to 93.9percent when all three envelope signals were pre-sented. For the IEEE sentences, these scores were14.5 percent for speechreading alone; 36.0 percent,26.5 percent, and 18.3 percent for the singleenvelope signals, and 82.0 percent for the multipleenvelope signal. These results are similar to thefindings of Grant, Braida, and Renn (1991).20 Whenthe envelopes are derived from noisy speech,scores in the audiovisual condition are reduced, buthigher than for speechreading alone for S/N > -4dB (CUNY) and S/N > 0 dB (IEEE) for the multipleenvelope supplement. Roughly half of the increasein score (relative to speechreading alone) isretained at S/N = 0 dB (60 percent words correct)for the CUNY sentences. These results suggestthat the band envelope signals can be successfullyadapted for presentation to listeners with severelyrestricted dynamic ranges, since at a S/N of 0 dB,excursions of the component envelope signals arelimited to roughly 12-15 dB by the relatively con-stant background noise. Moreover, all subjectswere able to utilize the envelopes regardless oflipreading ability while the best lipreaders derivedthe greatest benefits.

Supplements Derived from Multiple BandEnvelopes

Although speechreading can be considerablyenhanced when a normal hearing listener receivesa single tone modulated by the amplitude envelopeof a single octave speech band, even larger gainsare possible when additional envelopes are used tomodulate tones at the center frequencies of thebands from which they were derived. To benefit a

listener with high-frequency hearing loss, thesemodulated signals must be presented at low fre-quencies in close spectral proximity. However,when two or more modulated tones are simultane-ously presented, regardless of the frequency sepa-ration between the tones, there may be perceptualinteractions that can enhance or interfere with theperception of the modulation patterns.

In previous work, Takeuchi and Braida 21 reportedthat the ability of normal-hearing listeners tocompare amplitude modulation patterns wasadversely affected by the presence of simultaneousipsilateral amplitude-modulated distractors. Percep-tual interference effects, although generally small inmagnitude, were greatest when the distractorenvelope was uncorrelated with the target patternand was greater when the separation betweencarrier and distractor carrier frequencies was small(200 and 250 Hz) than when it was large (200 and500 Hz). To determine whether this finding resultsfrom the effects of peripheral masking (which typi-cally decreases with spectral separation), Takeuchiand Braida2 2 extended these experiments to includeconditions in which the distractor consisted of anunmodulated tone or was presented dichotically. Ingeneral, the amount of interference found in thepresent experiment was smaller than that found inthe earlier one, but the dependence on correlationand separation was similar for replicated conditions.Dichotic presentation produced roughly the sameamount of interference as diotic presentation, forboth correlated and uncorrelated distractors. Thisindicates that interference is unlikely to be ofperipheral origin and also suggests that listenerscan perceive correlations between envelopesequally well under diotic and dichotic presentation.Unmodulated distractors did not degrade envelopecomparisons when the frequency separation waslarge and reduced performance when the frequencyseparation was small only for two of the four lis-teners tested.

Field Studies

To obtain insight into the benefits provided tohearing impaired listeners by band-envelope sup-plements in everyday situations, we plan to provide

20 K.W. Grant, L.D. Braida, and R.J. Renn, "Single Band Amplitude Envelope Cues as an Aid to Speechreading," Quart. J. Exp. Psych.43: 621-645 (1991); K.W. Grant, L.D. Braida, and R.J. Renn, "Auditory Supplements to Speechreading: Combining AmplitudeEnvelope Cues from Different Spectral Regions of Speech," J. Acoust. Soc. Am. 95: 1065-1073 (1994).

21 A.H. Takeuchi and L.D. Braida, "Recognition of Amplitude Modulation Patterns in the Presence of a Distractor: I. Effects of Corre-lation and Frequency Relation," J. Acoust. Soc. Am., forthcoming.

22 A.H. Takeuchi and L.D. Braida, "Recognition of Amplitude Modulation Patterns in the Presence of a Distractor: II. Effects of DichoticPresentation and Unmodulated Distractors," J. Acoust. Soc. Am., forthcoming.



wearable DSP-based SiVo Aids23 that have beenprogrammed to produce these supplements to lis-teners with severe to profound impairments. In pre-liminary work, digital oscillators and filters havebeen implemented on the TMS320C50 processorused in the SiVo Aid and found capable ofproducing signals similar to those studied previ-ously. Preliminary evaluations of these amplitudeenvelope signals as aids to speechreading havebeen performed by one hearing-impaired listenerusing tests of word intelligibility in (IEEE) sen-tences. For that listener, who is an effective hearingaid user, the increase in intelligibility provided bythe single-band envelope cues was found to beroughly equal to that provided by her own aid.Detailed planning of the field study and recruitmentof participants is now underway.

Publications

Grant, K.W., L.D. Braida, and R.J. Renn. "AuditorySupplements to Speechreading: CombiningAmplitude Envelope Cues From Different Spec-tral Regions of Speech." J. Acoust. Soc. Am.95: 1065-1073 (1994).

Takeuchi, A.H., and L.D. Braida. "Recognition ofAmplitude Modulation Patterns in the Presenceof a Distractor: I. Effects of Correlation and Fre-quency Relation." J. Acoust. Soc. Am. Forth-coming.

Takeuchi, A.H., and L.D. Braida. "Recognition ofAmplitude Modulation Patterns in the Presenceof a Distractor: II. Effects of Dichotic Presenta-tion and Unmodulated Distractors." J. Acoust.Soc. Am. Forthcoming.

Uchanski, R.M., L.A. Delhorne, A.K. Dix, L.D.Braida, C.M. Reed, and N.I. Durlach. "AutomaticSpeech Recognition to Aid the HearingImpaired. Prospects for the Automatic Gener-ation of Cued Speech." J. Rehab. Res. Dev.31: 20-41 (1994).

1.1.3 Cochlear Implants

SponsorNational Institutes of Health

Contract 2 P01 DC00361 24

Contract N01 DC22402

Project StaffProfessor Louis D. Braida, Lorraine A. Delhorne,Dr. Donald K. Eddington, Dr. William M. Rabinowitz

The use of cochlear implants is directed at aidingindividuals with profound sensorineural deafnesswho are unable to derive useful benefits fromacoustic input to the ear. The prosthesis bypassesthe impaired acoustic tranduction mechanism anddirectly stimulates surviving auditory nerve fiberswith currents delivered via an electrode arrayimplanted within the cochlea. The overall goal ofour research is to understand the mechanismsresponsible for the improved hearing provided bythese prostheses and to exploit this understandingfor the development of improved systems. Ourprogram uses a dedicated group of postlinguallydeafened adults implanted with a multichannelcochlear prosthesis (Ineraid, Richards Medical Cor-poration), who participate in intensive multifactorialstudies. The research capitalizes on having directaccessibility to the implanted electrode array via apercutaneous connector.

During the past year, work has involved (1) analysisof cue integration in audiovisual speech reception,(2) studies of interactions among different implantchannels, and (3) alternative speech processing forimproved implant performance. The work in (3) isperformed with Joseph Tierney and Dr. MarcZissman (see also part 5 section 3 chapter 1.7).

The work on audiovisual integration assesses theability of an implantee to combine cues that areavailable from separately using vision and audition.Since most implantees require audiovisual input forreliable communication, analysis of integration is ofparticular significance.

A series of experiments on audiovisual integrationwith speech segments has been essentially com-pleted. Closed-set identification for a set of tenvowels, a set of the 12 most frequently occurringconsonants, and the full set of 24 (initial-position)consonants was tested. Results have been

23 A. Faulkner, V. Ball, S. Rosen, B.C.J. Moore, and A. Fourcin, "Speech Pattern Hearing Aids for the Profoundly Hearing Impaired:Speech Perception and Auditory Abilities," J. Acoust. Soc. Am. 91: 2136-215 (1992).

24 Subcontract from Massachusetts Eye and Ear Infirmary. Dr. Joseph P. Nadol, M.D., Principal Investigator.

355


obtained using vision alone, audio (the implant)alone, and audiovisually. Analyses to determine theefficiency with which the information available sepa-rately from vision and audition is utilized in the com-bined audiovisual condition are underway.25 Initialanalysis of the 12-consonant results suggests that(1) confusion patterns for the vision-alone conditionare more consistent and more highly structuredthan are those for the audition-alone condition, and(2) most subjects make efficient use of theunimodal results in the combined condition.

The work in channel interactions is designed toassess the perceptual independence among dif-ferent channels of an implant. Current spread fromintracochlear electrodes is extensive26 and psycho-physical studies with simple stimuli reveal substan-tial interelectrode interactions.2 7 However, theconsequences of these interactions for speechreception are not clear.

In one set of experiments, consonant identificationwas measured using each of four individualintracochlear electrodes (driven from their corre-sponding individual processor channels), using dif-ferent pairs of electrodes, and using all fourelectrodes (the normal system). Analysis ofmultielectrode integration uses the same modelsbeing applied to study audiovisual intergration.These models quantify the extent to which theresults from multielectrode conditions reflectoptimum use of the information available from theseparate electrode conditions versus interferencefrom electrode interactions. The models indicatethat predicted optimum performance with multipleelectrodes sometimes exceeds observed perfor-mance by substantial amounts (e.g., 85 percent-correct predicted versus 65 percent-correctobserved). In these cases, however, large responsebiases are evident in the results from the separateelectrode conditions. When the integration modelsare constrained to maintain these same biases, thepredicted and observed multichannel results aregenerally more similar.

Another set of experiments assessed multichannelsaliency with the Ineraid prosthesis by altering thenormal tonotopic mapping between the sound-processor's four filter channels and theintracochlear electrode array. 28 A single high-performing subject was tested with (1) the normalfour channel processing system, (2) a "single-channel" processor, formed by summing theoutputs of the four processor filters and deliveringthe resulting current to a single electrode, and (3) atonotopically "reversed" system, formed by con-necting the four processor filter outputs to fourintracochlear electrodes in reversed order. Whenusing the implant in conjunction withspeechreading, all three mappings showed a largeimprovement on the recognition of words in sen-tences over speechreading alone. When using theimplant alone (without speechreading), tests of con-sonant and vowel recognition, and the recognitionof words in isolation and in sentences all showed asubstantial decrease in performance across thethree mappings: normal > single-channel >reversed. The patterns of segmental confusions andthe relations among scores on different tests werehighly consistent.2 9 Overall, the results indicate that,despite the extensive spread of current associatedwith monopolar intracochlear stimulation, the Ineraidelectrode array affords a degree of perceptualselectivity that substantially aids speech reception.

The work on altered speech processing involvesevaluation of the recently developed continuous-interleaved-stimulation (CIS) strategy. Each channelof a CIS processor uses the compressed envelopeof its bandpass filtered output to modulate biphasiccurrent pulses that are delivered to an intracochlearelectrode. Pulses are interleaved across channelsto avoid simultaneous field interactions, and pulserates are high (-2000 pps/channel) to preservetemporal waveform cues. Using up to six monopolarelectrodes directly accessible with the Ineraidimplant, this strategy has shown considerablepromise in acute evaluations conducted in the labo-

25 L.D. Braida, "Crossmodal Integration in the Identification of Consonant Segments," Quart. J. Exp. Psychol. 43A: 647-677 (1991).

26 C. van den Honert and P.H. Stypulkowski, "Single Fiber Mapping of Spatial Excitation Patterns in the Electrically Stimulated AuditoryNerve," Hear. Res. 29: 195-206 (1987).

27 R.V. Shannon, "Multichannel Electrical Stimulation of the Auditory Nerve in Man: II1. Channel Interaction," Hear. Res. 18: 135-143(1983).

28 W.M. Rabinowitz and D.K. Eddington, "Effects of Channel-to-Electrode Mappings on Speech Reception With the Ineraid CochlearImplant," Ear Hear., forthcoming.

29 W.M. Rabinowitz, D.K. Eddington, L.A. Delhorne, and P.A. Cuneo, "Relations Among Different Measures of Speech Reception inSubjects Using a Cochlear Implant," J. Acoust. Soc. Am. 92: 1869-1881 (1992).



ratory.3 0 In collaboration with a group in Innsbruck,a prototype portable real-time system has beendeveloped (based on a DSP56001) that can realizesome CIS implementations. Two subjects with nineyears of experience using the standard Ineraidanalog sound processor are now wearing the CISsystem on a full-time basis. After several weeks,one subject prefers the CIS strategy; however,objective measures of speech reception show nogain (re the standard processor). The secondsubject shows a large gain with CIS and some evi-dence of continuing improvement. On relatively dif-ficult sentences presented in quiet and withoutlipreading cues, the subject scores near perfectlywith CIS versus 65 percent with the standard pro-cessor. For speech reception in noise, he shows adeficit of 7 dB re normal-hearing listeners, whichcompares favorably to deficits of 10 to 15 dBobtained with the standard analog processor.

Publication

Rabinowitz, W.M., and D.K. Eddington. "Effects ofChannel-to-Electrode Mappings on SpeechReception With the Ineraid Cochlear Implant."Ear Hear. Forthcoming.

1.1.4 Binaural HearingSponsorNational Institutes of Health

Grant R01-DC00100131

Project StaffNathaniel I. Durlach, Dr. Patrick M. Zurek

The long-term goal of this program is (1) to developan integrated, quantitative theory of binaural inter-action that is consistent with psychophysical andphysiological data on normal and impaired auditorysystems and (2) to apply our results to the diag-nosis and treatment of hearing impairments.

Recent work has examined the precedence effectthat is observed when leading and lagging soundsoccupy different frequency regions.32 Subjects wereasked to match the intracranial lateral position of anacoustic pointer to that of a test stimulus composed

of two binaural noise bursts with asynchronousonsets, parametrically-varied frequency content,and different interaural delays. The precedenceeffect was measured by the degree to which theinteraural delay of the matching pointer was inde-pendent of the interaural delay of the lagging noiseburst in the test stimulus. Results show an asym-metric frequency effect in which the lateralizationinfluence of a lagging high-frequency burst isalmost completely suppressed by a leading low-frequency burst, whereas a lagging low-frequencyburst is weighted equally with a leading high-frequency burst. This asymmetry is shown to be theresult of an inherent low-frequency dominance thatis seen even with simultaneous bursts. When thisdominance is removed (by attenuating the low-frequency burst), the precedence effect operatesequally effectively both upward and downward infrequency.

1.1.5 Multimicrophone Hearing Aids

Sponsor

National Institutes of HealthGrant R01-DC00270

Project Staff

Joseph Desloge, Nathaniel I. Durlach, Dr. Julie E.Greenberg, Michael P. O'Connell, Dr. William M.Rabinowitz, Daniel P. Welker, Dr. Patrick M. Zurek

The goal of this research is to determine theimprovements that can be provided to hearing aidsthrough the use of multiple microphones. The workis directed toward developing algorithms for proc-essing the signals from a head-worn microphonearray for the primary goal of improving the intelli-gibility of speech (assumed to arise from a knowndirection) in the presence of noise andreverberation. Ideally, this intelligibility enhance-ment would be achieved without compromising thelistener's ability to monitor and localize soundsources from all directions. Array processing algo-rithms are first implemented and evaluated in termsof signal-to-noise improvement in computer simu-lations. The most promising approaches are thenimplemented in real-time with wearable devices(tethered to a computer) for laboratory evaluations

30 B.S. Wilson, C.C. Finley, D.T. Lawson, R.D. Wolford, D.K. Eddington, and W.M. Rabinowitz, "Better Speech Reception With CochlearImplants," Nature 352: 236-238 (1991).

31 Subcontract from Boston University, Professor H. Steven Colburn, Principal Investigator.

32 B.G. Shinn-Cunningham, P.M. Zurek, N.I. Durlach, and R.K. Clifton, "Cross-frequency Interactions in the Precedence Effect," J.Acoust. Soc. Am., forthcoming.

357


in terms of speech reception in noise and soundlocalization by normal and hearing-impaired lis-teners. Some devices will be made portable for fieldstudies.

Work in the past year has focused on algorithmicdesign and analysis. One particular area concernscontrolling the adaptation of time-varying processingschemes to minimize signal cancellation and misad-justment. These problems arise when the array isnot perfectly aligned to the desired source and inthe presence of strong source signals. They areespecially bothersome in the hearing aid applicationbecause: (1) hearing aid users cannot be expectedto hold their heads rigidly pointing at the signalsource; and (2) often the signal is larger than theinterference and so it degrades the precision ofinterference cancellation that can be achieved.These issues were outlined previously, along withad hoc solutions, by Greenberg and Zurek.33

Recent work has thoroughly analyzed and refinedthose ad hoc solutions, producing a modified adap-tive algorithm that includes two additional features.One modification is to adjust the steps taken by theadaptive algorithm in order to reduce misadjustmentwhen the output power is strong, and the other is toinhibit adaptation based on the correlation betweenmicrophone signals. Guidelines have been estab-lished for selecting the relevant processing parame-ters in a variety of acoustic environments.

Another problem with adaptive systems is thatreverberation results in a signal that does not arrivefrom the source direction, but that is correlated withthe source signal. A simple solution to the problemof reverberation-induced signal cancellation hasbeen studied.34 By reducing the amount by whichthe adaptive filter can "look ahead in time" (by abulk processing delay), signal cancellation can beavoided with little loss in noise-reduction perfor-mance.

The result of this design and analysis work is aspecification for a relatively simple and robustbroadside adaptive array.35 Computer simulationshave evaluated this system in a variety of acoustic

33 J.E. Greenberg and P.M. Zurek, "Evaluation of an Adaptive1662-1676 (1992).

environments. Steady-state results show that itprovides very large improvements in relativelyanechoic environments. Substantial benefits areobtained in moderate reverberation, particularly ifrelatively long filters are used. In extremereverberation, performance is comparable to that ofthe underlying nonadaptive microphone array. Tran-sient results indicate that convergence of the adap-tive algorithm is sufficiently rapid for processingspeech signals. In addition, this work has consid-ered the use of directional microphones and thenumber of microphones required in a practicalsystem.

Work to date on multimicrophone arrays for hearingaids has been aimed at processing the microphonesignals to form a single output signal. Suchmonaural-output systems contain none of the infor-mation about the location of sound sources that isnormally conveyed primarily by differences betweenthe signals at the two ears. In addition to soundlocalization ability, binaural hearing provides a sen-sation of auditory space and improved speechreception in noise. In an attempt to provide thesenatural benefits of binaural hearing along withimprovements from multimicrophone array proc-essing, we have been exploring ways of combiningthem. In one project, we have been exploringdesigns of a fixed four-microphone array in which,by choice of a frequency-dependent parameter,directivity can be traded with fidelity of interauraltime delay (the most important binaural cue).36 Atone extreme in this trade maximal directivity isachieved, while at the other extreme the outermostmicrophone signals are simply passed to the twoears. In another project, we are exploring thedesign of a two-microphone adaptive binaural array,with the microphones worn at the ears.3 7

In both projects a very promising approach is a par-ticularly simple one that exploits the psychoacousticfinding that sound localization is dominated by low-frequency interaural delay cues. Using this fact, weallocate the lowpass part of the signal spectrum fortransmitting natural binaural cues by simply passing

Beamforming Method for Hearing Aids," J. Acoust. Soc. Am. 91:

34 J.E. Greenberg and P.M. Zurek, "Preventing Reverberation-induced Target Cancellation in Adaptive-array Hearing Aids," J. Acoust.Soc. Am. 95: 2990-2991 (1994).

35 J.E. Greenberg, Improved Design of Microphone-Array Hearing Aids, Ph.D. diss., Division of Health Sciences and Technology, MIT,1994.

36 J.G. Desloge, A Fixed Microphone Array Hearing Aid with a Binaural Output, M.S. Thesis, Dept. of Electr. Eng. and Comput. Sci.,MIT, 1994.

37 D.P. Welker, A Binaural-Output Adaptive-Beamforming Hearing Aid, M.S. Thesis, Dept. of Electr. Eng. and Comp. Sci., MIT, 1994.



the lowpass-filtered microphone signals to the ears.The highpass part is the output of the array, pre-sented diotically. This approach allows source local-ization via the low frequencies and an enhancedsignal via the high frequencies.

1.1.6 Tactile Communication of Speech

Sponsor

National Institutes of HealthGrant 5 R01 DC00126

Project Staff

Lorraine A. Delhorne, Gail Denesvich, Nathaniel I.Durlach, Ashanthi Gajaweera, Geoffrey L. Plant,William M. Rabinowitz, Dr. Charlotte M. Reed, Dr.Mandayam A. Srinivasan, Hong Z. Tan, JonathanR. Santos

The ultimate goal of this research is to developtactual aids for the deaf and deaf-blind that canserve as substitutes for hearing in speech commu-nication. Research is conducted in three majorareas summarized below.

Basic Study of Encoding and DisplaySchemes

This research is designed to develop methods ofdisplaying acoustic signals to the tactual sense foroptimal information transfer. Research in this areaincludes measurements of basic resolution andinformation transfer for tasks involving active fingermotions and for those involving motional or vibra-tory stimulation of the finger, in addition to work on

the design and construction of a multifinger tactualstimulator. Studies employing active finger motionsinclude measurements of the ability to discriminateand identify joint-angle rotation, measurements ofthe ability to discriminate the thickness of materialswith different bending moduli, 38 and measurementsof manual perception of compliance with work cueseliminated and force cues minimized. 39 Studiesemploying motional or vibratory stimulation includemeasurement of amplitude and frequency discrimi-nation for motional (i.e., low-frequency, high-amplitude) signals40 and a study of the reception ofMorse Code sequences employing motional andvibratory stimulation.41 In addition, work has beenconducted on the design and construction of amultifinger tactual device capable of providing stim-ulation along the frequency continuum from motionto vibration.

Tactual Supplements to Speechreading

This research is designed to lead towards thedevelopment of tactual aids to supplement informa-tion available through speechreading. Recentresearch in this area has been concerned withdescribing and understanding differences in perfor-mance achieved through auditory and tactual pres-entation of a simple acoustic-based supplement tospeechreading. Data comparing the two modes ofpresentation were obtained for various tasks,including sentence, prosodic, and segmental recep-tion.42 In addition, related nonspeech psychophys-ical measurements for tactual and auditorypresentation were obtained to increase our under-standing of the differential results observed in aidedspeechreading tasks.43

38 A. Gajaweera, Tactual Discrimination of Thickness, B.S. thesis, Dept. of Mech. Eng., MIT, 1994.

39 H.Z. Tan and N.I. Durlach, "Manual Discrimination Using Active Finger Motion: Compliance, Force, and Work," Percept. Psychophys.,forthcoming.

40 G.L. Beauregard, W.M. Rabinowitz, H.Z. Tan, and N.I. Durlach, "Amplitude and Frequency Resolution for Motional Stimulation," ThirdInternational Conference on Hearing Aids, Tactile Aids, and Cochlear Implants, Miami, Florida, May 2-5, 1994.

41 H.Z. Tan and N.I. Durlach, "A Study of the Tactual Perception of Motor Input Sequences," Third International Conference on HearingAids, Tactile Aids, and Cochlear Implants, Miami, Florida, May 2-5, 1994; H.Z. Tan, N.I. Durlach, W.M. Rabinowitz, and C.M. Reed,"Tactual Performance with Motional Stimulation of the Index Finger," (Abstract) J. Acoust. Soc. Am. 95: 2986 (1994).

42 J.M. Besing, C.M. Reed, and N.I. Durlach, "A Comparison of Auditory and Tactual Presentation of a Single-band Envelope Cue as aSupplement to Speechreading," submitted to Seminars in Hearing, 1994; M.S. Bratakos, Supplements to Speechreading: A Compar-ison of Auditory and Tactile Presentation of a Single-band Envelope Cue, B.S. thesis, Dept. of Electr. Eng. and Comput. Sci., MIT,1993; C.M. Reed, M.S. Bratakos, L.A. Delhorne, and G. Denesvich, "A Comparison of Auditory and Tactile Presentation of a Single-band Envelope Cue as a Supplement to Speechreading," Third International Conference on Hearing Aids, Tactile Aids, and CochlearImplants, Miami, Florida, May 2-5, 1994.

43 W.M. Rabinowitz, C.M. Reed, L.A. Delhorne, and J.M. Besing, "Tactile and Auditory Measures of Modulation Resolution," (Abstract)J. Acoust. Soc. Am. 95: 2987 (1994); W.M. Rabinowitz, C.M. Reed, and L.A. Delhorne, Tactile and Auditory Measures of ModulationResolution," Third International Conference on Hearing Aids, Tactile Aids, and Cochlear Implants, Miami, Florida, May 2-5, 1994.

359


Evaluation of Practical Aids

This research is designed to evaluate the speech-reception performance of experienced users of port-able, wearable tactual aids and to compare thisperformance to that of users of other types ofauditory prostheses. Research in this project areaincludes (1) laboratory evaluations of tactualdevices with artificially deafened subjects, (2) a fieldstudy of the performance of deaf adult users oftactual aids,44 (3) a study of the effects of trainingon adult performance with a tactile device, and (4)a study of the effects of training on the use of atactile device by deaf children.45 The tactile devicesused in this research include two commerciallyavailable, wearable tactile aids: Tactaid 2 andTactaid 7 (both products of Audiological Engi-neering Corporation), with research focused prima-rily on the Tactaid 7 as a multichannel display ofthe first and second speech formants. The Tactaid2, by comparison, is a two-channel tactile display ofone lowpass frequency region of speech and onehighpass band. In addition, some data have beenobtained on several nonwearable laboratory-basedsystems, including a nine-channel laboratory imple-mentation of the Queen's University vocoder(Brooks and Frost, 1983) and a high-performancesingle-channel vibrator (the Alpha-M AV6Minishaker) used to present speech-envelope sup-plements to speechreading.

Publications

Besing, J.M., C.M. Reed, and N.I. Durlach. "AComparison of Auditory and Tactual Presenta-tion of a Single-band Envelope Cue as a Sup-plement to Speechreading." Submitted toSeminars in Hearing (1994).

Rabinowitz, W.M., D. Henderson, C.M. Reed, L.A.Delhorne, and N.I. Durlach. "Development andEvaluation of an Improved Synthetic TadomaSystem." J. Acoust. Soc. Am. Forthcoming .

Reed, C.M., and L.A. Delhorne. "Current Results ofa Field Study of Adult Users of Tactual Aids."Submitted to Seminars in Hearing (1994).

Reed, C.M., L.A. Delhorne, N.I. Durlach, and S.D.Fischer. "A Study of the Tactual Reception ofSign Language." J. Speech Hear. Res. Forth-coming.

Tan, H.Z., and N.I. Durlach. "Manual DiscriminationUsing Active Finger Motion: Compliance, Force,and Work." Percept. Psychophys. Forthcoming.

Thesis

Gajaweera, A. Tactual Discrimination of Thickness.B.S. thesis, Dept. of Mech. Eng., MIT, 1994.

1.2 Haptics Research

1.2.1 Introduction

In the following subsections, we describe work per-formed during the past year on a wide variety ofprojects concerned with manual sensing and manip-ulation (i.e., haptics). Further work on haptics, ori-ented specifically toward the use of virtualenvironments for training, is discussed in section1.3 below.

1.2.2 Mechanistic Modeling of the PrimateFingerpad

Sponsor

National Institutes of HealthGrant R29-DC00625

Project Staff

Dr. Mandayam A. Srinivasan, Kiran Dandekar,Walter E. Babiec, Rogeve J. Gulati

When we touch an object, the source of all tactileinformation is the spatio-temporal distribution ofmechanical loads on the skin at the contact inter-face. The relationship between these loads and theresulting stresses and strains at the nerve terminalswithin the skin plays a fundamental role in theneural coding of tactile information. Althoughempirical determination of the stress or strain state

44 C.M. Reed, L.A. Delhorne, and N.I. Durlach, "Results obtained with Tactaid 2 and Tactaid 7," Proceedings of the 2nd InternationalConference on Tactile Aids, Hearing Aids, and Cochlear Implants, eds. A. Risberg, S. Felicetti, G. Plant, and K.-E. Spens,(Stockholm, Sweden: Dept. of Speech Comm. Music Acoust., KTH, 1992), pp. 149-155; C.M. Reed and L.A. Delhorne, "CurrentResults of a Field Study of Adult Users of Tactual Aids," submitted to Seminars in Hearing, 1994.

45 G. Plant, "The Development of a Training Program for Profoundly Deaf Children Fitted with Tactaid 7," Third International Conferenceon Hearing Aids, Tactile Aids, and Cochlear Implants, Miami, Florida, May 2-5, 1994.



of a mechanoreceptor is not possible at present,mechanistic models of the skin and subcutaneoustissues enable generation of testable hypotheseson skin deformations and associated peripheralneural responses. The hypotheses can then beverified by comparing the calculated results fromthe models with biomechanical data on the defor-mation of skin and subcutaneous tissues and neu-rophysiological data from recordings of theresponses of single neural fibers. The researchunder this grant is directed towards applying analyt-ical and computational mechanics to analyze thebiomechanical aspects of touch-the mechanics ofcontact, the transmission of the mechanical signalsthrough the skin, and their transduction into neuralimpulses by the mechanoreceptors.

Determination of the Shape andCompressibility of the Primate Fingertip(distal phalanx)

The first step in performing mechanistic analyses ofthe primate fingertip is to determine its geometricand material properties. The three-dimensional(3D) external geometry of primate fingertips wasdetermined from accurate epoxy replicas of humanand monkey fingertips. Using a videomicroscopysetup, we obtained images of orthographicprojections of the epoxy replicas at various knownorientations. The images were then digitized andprocessed to determine the boundary of the fingerat each orientation. By combining the boundarydata for all the different orientations, we were ableto reconstruct the 3D external geometry of thefingertip.46 We have reconstructed several humanand monkey fingertips using this method. Formechanistic modeling of the human fingerpad, thePoisson's ratio, which is a measure of itscompressibility, is required as an input to the math-ematical models. The Poisson's ratio for thehuman fingerpad in vivo is unknown at present. Inprevious noninvasive experiments on human sub-jects, we have measured the change in volume of

the fingerpad under static indentations with differentindentors.4 7 Our results show that increases ineither the depth of indentation or the contact areawith the indentor increased the compressibility ofthe fingertip. The highest change in fingertipvolume was about 5 percent.

We have now developed an experimental setupinvolving a computer controlled linear actuator forfingertip volume change measurements underdynamic conditions.48 We measured volumechanges of the fingerpad under three types ofindentors (point, circular flat and flat plate) imposingconstant velocity ramps (1 to 8 mm/s), severaldepths of indentation (1 to 4 mm) and sawtoothstimulation (1 and 2 mm amplitude; 0.25 to 2Hz fre-quency). The fingerpad, encased in a thin latexfingercot, was placed in a fluid-filled chamber andindented by using a computer controlled steppermotor. The resulting fingertip volume changes werecalculated (resolution - 1/100 ml) by videorecording the fluid level in a small diameter pipette,transcribing the data frame-by-frame, and sub-tracting out the fluid level changes due to indentordisplacement. The results show that reductions infingertip volume are small and in phase with stim-ulus variations, with an increase in their mean valueover time. The volume changes during the rampphase increase linearly with indentor displacementand are independent of velocity; during sawtoothstimulations, however, the nature of the hysteresisloops depend on velocity of indentation. We haveanalyzed such data for four subjects.

Fingertip Models and Finite ElementAnalysis

We have performed linear and nonlinear finiteelement analysis of a series of mechanistic modelsof the fingerpad under a variety of mechanicalstimuli. 49 The models range from a semi-infinitemedium to a three-dimensional (3D) model basedon the actual finger geometry, and composed of a

46 T.R.R. Perez, K. Dandekar, and M.A. Srinivasan, Videomicroscopic Reconstruction of the Human Finger, Project report to the MIT

Minority Summer Science Research Program, 1992.

47 M.A. Srinivasan, R.J. Gulati, and K. Dandekar, "In vivo Compressibility of the Human Fingertip," Advances in Bioengineering, ed.

M.W. Bidez (Chicago: ASME, 1992), vol. 22, pp. 573-576.

48 W.E. Babiec, In vivo Volume Changes of the Human Fingerpad under Indentors, B.S. thesis, Dept. of Mech. Eng., MIT, 1994.

49 M.A. Srinivasan and K. Dandekar, "Role of Fingertip Geometry in the Transmission of Tactile Mechanical Signals," Advanes in

Bioengineering, ed. M.W. Bidez (Chicago: ASME, 1992), vol. 22, pp. 569-572 (1992); M.A. Srinivasan and K. Dandekar, "An Investi-

gation of the Mechanics of Tactile Sense Using Two Dimensional Models of the Primate fingertip," J. Biomech. Eng., forthcoming; K.

Dandekar and M.A. Srinivasan, "Tactile Coding of Object Curvature by Slowly Adapting Mechanoreceptors," in Advances in

Bioengineering, ed. M.J. Askew. Chicago: ASME, 1994, vol. 28, pp. 41-42; K. Dandekar and M.A. Srinivasan, "A Three-dimensional

Finite Element Model of the Monkey Fingertip for Predicting Responses of Slowly Adapting Mechanoreceptors," ASME summer

annual meeting, Beaver Creek, Colorado, 1995.

361


homogeneous elastic material, a thick elastic shellcontaining a fluid or a multilayered medium. Simu-lations of the mechanistic aspects of neurophysio-logical experiments involving mapping of receptivefields with single point loads, determination ofspatial resolution of two-point stimuli, and indenta-tions by single bars as well as periodic andaperiodic gratings have been carried out for thetwo-dimensional (2D) and 3D models. We havealso solved the nonlinear contact problem of inden-tations by cylindrical objects and sinusoidal stepshapes. The large number of numerical calcula-tions needed even for the linear two-dimensionalmodels necessitated the use of the Cray-C90 at theNSF Pittsburgh Supercomputer Center.

The results show that the model geometry has asignificant influence on the spatial distribution of themechanical signals, and that the elastic mediumacts like a low-pass filter in causing blurring of themechanical signals imposed at the surface. Multi-layered 3D models of monkey and human fingertipsaccurately predicted the surface deformations undera line load, experimentally observed bySrinivasan.50 The same models predicted the exper-imentally observed surface deformations undercylindrical indentors as well. These 3D finiteelement models were used to simulate neurophys-iological experiments involving indentation by rec-tangular bars, aperiodic gratings, cylindricalindentors and step shapes. Several strain meas-ures at typical mechanoreceptor locations werematched with previously obtained neurophysio-logical data to determine the relevant mechanicalsignal that causes the receptors to respond. In thesimulations, the strain energy density at thereceptor location was found to be directly related tothe static discharge rate of the slowly adapting (SA)afferents. In addition, strain energy density is ascalar that is invariant with respect to receptor ori-entations and is a direct measure of the distortionof the receptor caused by loads imposed on theskin. We have therefore hypothesized that thestrain energy density at the receptor site is the rele-

vant stimulus to the slowly adapting receptors. Insummary, this research project has resulted inmechanistic models of the primate fingertip whosepredictions match a variety of biomechanical andneurophysiological data. The models can also beused to generate hypotheses to be tested in futurebiomechanical and neurophysiological experiments.

1.2.3 Peripheral Neural Mechanisms ofHaptic Touch

Sponsor

U.S. Navy - Office of Naval ResearchGrant N00014-88-K-0604

Project Staff

Dr. Mandayam A. Srinivasan, Professor AnuradhaM. Annaswamy, Dr. Robert H. LaMotte,5 1

Steingrimur P. Karason

We have been collaborating with Dr. RobertLaMotte, Yale University School of Medicine, inconducting psychophysical and neurophysiologicalstudies on the tactile perception of the microtexture,shape and softness of objects. We have shownthat humans can detect extremly fine textures com-posed of 50 nm-high parallel bars on plane glassplates. 52 Our neurophysiological recordings indicatethat when such fine textures are stroked on thefingerpad skin, the fingerprint ridges vibrate andcause Pacinian Corpuscles to respond, thus ena-bling detection of the microtexure.5 3

In studies of the tactile perception of shape, aseries of two- and three-dimensional objects (e.g.,cylinders, spheres, ellipsoids and wavy surfaces)were pressed or stroked across the fingerpads ofanesthetized monkeys and evoked responses incutaneous mechanoreceptive primary afferent nervefibers were recorded.54 Major geometrical propertiesof the shapes were well represented in the spatio-

50 M.A. Srinivasan, "Surface Deflection of Primate Fingertip Under Line Load," J. Biomech. 22(4): 343-349 (1989).

51 Yale University School of Medicine, New Haven, Connecticut.

52 R.H. LaMotte and M.A. Srinivasan, "Surface Microgeometry: Neural Encoding and Perception," in Information Processing in theSomatosensory System, eds. 0. Franzen and J. Westman, Wenner-Gren International Symposium Series (New York: MacmillanPress, 1991).

53 M.A. Srinivasan, J.M. Whitehouse, and R.H. LaMotte, "Tactile Detection of Slip: Surface Microgeometry and Peripheral NeuralCodes," J. Neurophys. 63(6): 1323-1332 (1990).

54 M.A. Srinivasan and R.H. LaMotte, "Tactual Discrimination of Softness," J. Neurophys. 73(1): 88-101 (1995); R.H. LaMotte and M.A.Srinivasan, "Responses of Cutaneous Mechanoreceptors to the Shape of Objects Applied to the Primate Fingerpad," ActaPsychologica 84: 41-51 (1993); R.H. LaMotte, M.A. Srinivasan, C. Lu, and A. Klusch-Petersen, A., "Cutaneous Neural Codes forShape," Can. J. Physiol. Pharm. 72: 498-505 (1994).



temporal responses of SA and RA afferent fiberpopulations, particularly those of the SAs. Theresults show that the following hypothesis explainsqualitatively the data we have obtained: the depthof indentation and the change in curvature of theskin surface are encoded by the discharge rates ofSAs; in addition, the velocity and the rate of changein skin surface curvature are encoded by the dis-charge rates of both SAs and RAs.

Thus, the intensive parameters of shapes, such asthe magnitude of change in skin curvature producedby contact with the object surface were encoded inthe discharge rates of SAs and RAs, but this neuralcode was also influenced by changes in strokevelocity. Spatial parameters of shapes such as thecurvature width and the changes in contour thatcharacterize a shape as belonging to a particularcategory (such as a sphere as opposed to a cyl-inder) are encoded in the spatially distributed dis-charge rates of the SA population. This spatialresponse profile provides a neural code that isprobably invariant with moderate changes in theparameters that govern contact conditions betweenthe object and the skin, such as the contact force ororientation and velocity of its trajectory. Therefore,among the different possible geometric represen-tations of the shape of objects, the intrinsicdescription, i.e., the surface curvature as a functionof the distance along the surface, seems to be rele-vant for tactile sensing of shape.

Based on a theoretical analysis of the mechanics ofcontact, we have proposed a mechanism by whichshapes of objects within contact regions are per-ceived through the tactile sense. The curvature ofthe skin surface under an object, which we knowfrom differential geometry is approximated by thesecond spatial derivative of surface deflection, iscoded without differentiating (which is a noiseenhancing process), but by exploiting its relation tosurface pressure. Pressure peaks occur where thedepths of indentation and/or changes in the skinsurface curvature are large. The skin effectivelyacts as a low-pass filter in transmitting the mechan-ical signals, and the mechanoreceptors respond tothe blurred versions of the surface pressure dis-

tribution, thus encoding the shape of the object interms of its surface curvatures.55

We have also shown that the human discriminabilityof softness or compliance of objects depends onwhether the object has a deformable or rigidsurface.56 When the surface is deformable, thespatial pressure distribution within the contactregion is dependent on object compliance, andhence information from cutaneous mechanorecep-tors is sufficient for discrimination of subtle differ-ences in compliance. When the surface is rigid,kinesthetic information is necessary for discrimi-nation, and the discriminability is much poorer thanthat for objects with deformable surfaces.

Development of a Computational Theory ofHaptics

Our research on the computational theory of hapticsis focused on developing a theoretical frameworkfor studying the information processing and controlstrategies common to both humans and robots per-forming haptic tasks. For example, although the"hardware" of the tactile apparatus in humans androbots are different, they have the common featureof mechanosensors embedded in a deformablemedium. Therefore the mechanistic analysesneeded to solve the computational problem ofcoding (predicting sensor response for a givenmechanical stimulus at the surface) and decoding(inferring the mechanical stimulus at the surface bysuitably processing the sensor response) are similarfor human and robot tactile sensing systems.

We first developed such a computational theoryusing a simplified 2D half-space model of thehuman or robot finger subjected to arbitrary pres-sure or displacement loading conditions normal tothe surface and then gave explicit formulae forcoding and decoding problems. 5 7 We have .nowexpanded these results to a more general 3D half-space model where the load direction can be com-pletely arbitrary. 8 Explicit solutions for the codingproblem are given and enable the selection of auseful set of relevant stimuli as well as the choiceof sensors appropriate for maximizing the informa-

55 M.A. Srinivasan and R.H. LaMotte, "Encoding of Shape in the Responses of Cutaneous Mechanoreceptors," in Information Proc-essing in the Somatosensory System, eds: O. Franzen and J. Westman, Wenner-Gren International Symposium Series (New York:Macmillan Press, 1991).

56 M.A. Srinivasan and R.H. LaMotte, "Tactual Discrimination of Softness," J. Neurophys. 73(1): 88-101 (1995).

57 M.A. Srinivasan, "Tactile Sensing in Humans and Robots: Computational Theory and Algorithms," Newman Laboratory Technical

Report, Department of Mechanical Engineering, MIT, 1988.

58 S.P. Karason, M.A. Srinivasan, and A.M. Annaswamy, "Tactile Sensing of Shape," Center for Information Driven Mechanical Systems

(CIDMS) Workshop Proceedings, Dept. of Mech. Eng., MIT, 1994.

363


tion about the stimulus on the skin surface. Thesolution of the decoding problem is also given, bothfor the idealized noise-free case and for the realisticcase with measurement noise. For the latter, thesolutions are shown to be numerically stable andoptimal.

In our work during the previous years, we were suc-cessful in answering basic identification and controlissues that arise during manipulation of compliantobjects using compliant fingerpads. 59 In order tounderstand the fundamental aspects of these tasks,we have analyzed the problem of identification ofcompliant objects with a single finger contact, aswell as under a two-finger grasp. Using lumpedparameter models, we have carried out the identifi-cation of human and object parameters, usingeither force or displacement inputs to the rigidbacking of the end-effector. Based on identifiedparameters, control strategies are developed toachieve a desired manipulation of the object in theworkspace. We have also modeled the dynamicinteractions that occur between compliant end-effectors and deformable objects by a class of non-linear systems. It was shown that standardgeometric techniques for exact feedbacklinearization techniques were inadequate. Newalgorithms were developed by using adaptive feed-back techniques which judiciously employed thestability characteristics of the underlying nonlineardynamics. In both theoretical and simulationstudies, it was shown that these adaptive controlalgorithms led to successful manipulation. The the-oretical results can be used to generate testablehypotheses for experiments on human or robothaptics.

1.2.4 Biomechanics of HumanFingerpad-Object Contact

Sponsor

U.S. Navy - Office of Naval ResearchGrant N00014-91-J-1454

Project Staff

Dr. Mandayam A. Srinivasan, Jyh-Shing Chen,Kiran Dandekar, Rogeve J. Gulati, Tomas

Rodriguez-Perez, Frederick L. Roby, Joseph D.Towles

Although physical contact is ubiquitous in our inter-actions with objects in the environment, we do notyet understand the mechanistic phenomena occur-ring at the skin-object interface. As mentionedbefore, the spatio-temporal distribution of mechan-ical loads on the skin at the contact interface is thesource of all tactile information. These loads, spec-ified as pressure, displacements, etc., depend onthe geometrical and material properties of both thecontacting entities, as well as the overall forces ofinteraction.

The goals of this project are (1) determination ofthe growth and motion of contact regions and theassociated force variations over time between thehuman fingerpad and carefully chosen transparenttest objects whose microtexture, shape or softnessis varied in a controlled manner, (2) experimentalmeasurement of the surface deformations of humanfingertips under shaped indentors, and (3) charac-terization of the mechanical properties of the humanfingerpad. The results obtained are being used togain a deeper understanding of the neurophysio-logical and psychophysical data we have alreadyobtained for the same test objects.

To measure the in vivo surface deformations of thefingerpad under various tactile stimuli, we havedesigned a videomicroscopy system together withhigh precision force sensors. The videomicroscopysystem consists of a set of video zoom lensesattached to a high-resolution CCD camera, whoseoutput can either be digitized directly at about 5frames/s, or stored on a laserdisk at real-time framerates (30 frames/s) for off-line digitization. Thezoom lenses allow continuous variation of magni-fication, with the field of view covering the entirefingerpad, or just a few fingerprint ridges. Highcontrast images are achieved with coaxial and otherfiberoptic lighting. In collaboration with our col-leagues at the Artificial Intelligence Laboratory atMIT, we designed and built two 6-axis forcesensors that are customized to our application.These sensors have much higher resolutions (10bit) than commercial sensors operating in compa-rable range of forces (5 Newtons). Transparent testobjects can be attached to these sensors for bothbiomechanical and psychophysical experiments.

59 A.M. Annaswamy and D. Seto, "Object Manipulation Using Compliant Fingerpads: Modeling and Control," ASME J. Dynam. Syst.Measure, Control (1993); A.M. Annaswamy and M.A. Srinivasan, "A Study of Dynamic Interactions between Haptic Interfaces andCompliant Fingerpads," Proceedings of the Motion Control Workshop, Berkeley, California, March 1994; A.M. Annaswamy and M.A.Srinivasan, "The Role of Compliant Fingerpads in Grasping and Manipulation: Identification and Control," Institute of Mathematics(New York: Springer Verlag, 1995).



During the past year, we enhanced the videomicro-scopy system to be able to acquire images whenthe human subject's finger remained passive. Alinear stepper motor with a microstepping drive wasadded to the system. Transparent compliant speci-mens can be mounted on the motor to indent a sta-tionary finger at a given velocity. The motor iscontrolled by a 80386 PC, with a specified indenta-tion velocity commanded by a 80486 PC via adigital link. A strain gage based single degree offreedom force sensor is mounted between thespecimen and the motor to record contact force.With this setup, we will be able to investigate howthe skin-object contact region changes with indenta-tion velocity and force. During the indentation, thecontact force and the video signal generated by theCCD camera focused on the contact region arerecorded. A new video grabber was incorporatedinto the system to allow video data to be digitizedinto the computer system memory in real time. Thismethod allows the force and video data to besynchornized. The video data can be digitized atrates up to 20 frames/sec. For the processing ofcontact images, a computer program which usesmouse cursor input to manually extract the borderof an contact image was developed. Although thismethod is slow, it serves to calibrate more auto-mated image processing methods under develop-ment.

Videomicroscopy of the Fingerpad-objectContact Regions

Using the test facility described above, we haveperformed a set of experiments with human sub-jects to investigate the relationship between thecontact force, contact area and compliance of theobject. The experiments involved active indentationof transparent compliant rubber specimens and aglass plate with the subjects' fingerpads. Staticvideo images of the contact regions were capturedat various force levels and magnifications. In orderto minimize the effects of nonuniform illumination,we implemented homomorphic image processingalgorithms with or without image decimation. Theprocessed images showed that contact regions con-sisted of discontinuous "islands" along each fingerridge, with clear distinction between contact andnoncontact regions over the entire field of view.

Results show that for objects whose compliancesare discriminable, even when the overall contactareas under a given contact force are the same, theactual contact areas can differ by a factor of two ormore. The actual pressure distribution, which actsonly within the discontinuous contact islands on theskin, will therefore be radically different for theobjects. Consequently, a spatio-temporal neuralcode for object compliance emerges with far higherresolution than an intensive code such as theaverage pressure over the overall contact area.These results are in agreement with our hypothesisthat the neural coding of objects with deformablesurfaces (such as rubber) is based on the spatio-temporal pressure distribution on the skin. Thiswas one of the conclusions from our psychophys-ical, biomechanical and neurophysiological exper-iments in a companion project conducted incollaboration with Professor LaMotte of the YaleUniversity School of Medicine.

Measurement of Surface Deformation ofHuman Fingerpads

The finite element models described previouslyneed to be verified by comparing the experimentallyobserved skin surface deformations with those pre-dicted by the finite element models under the samemechanical stimuli. The experimental data wasobtained by indenting human fingerpads withseveral cylindrical and rectangular indentors andacquiring images of the undeformed and deformedfingerpad using the videomicroscopy setup. 60 Finemarkers were placed on the fingerpad and the skinsurface deformation was measured by tracking thedisplacements of the markers in the high resolutionvideo images. The same experiment was simulatedusing the finite element models of the humanfingertip and the displacements of correspondingpoints were compared with the experimental data.The displacements predicted by the multilayered 3Dmodel matched the experimental data quite well.

Force Response of the Human Fingerpad toIndentation

A 2-DOF robot designed by Dr. Howe of theHarvard group was modified to serve as a "TactileStimulator" capable of delivering static and dynamic

60 F.L. Roby, K. Dandekar, and M.A. Srinivasan, Study of Fingertip Deformation Under Indentations by Circular and Rectangular

Indentors, Report to the MIT Summer Research Program, 1994.

365


stimuli to the human fingerpad.61 Three types ofindentors (point, flat circular, and a flat plate)attached to the stimulator imposed a variety of con-stant velocity ramps (1 to 32 mm/s), depths ofindentation (0.5 to 3 mm), and sinusoids (0.25 mmto 0.5 mm amplitude; 0.125 to 16 Hz frequency)under displacement control (resolution- 20 microns). The resulting normal and shearforces were measured by a 2-axis force sensor(resolution ~ 10 mN). The results showed a pro-nounced nonlinear force-indentation depth relation-ship under both static and dynamic conditions,viscoelastic effects of force relaxation under con-stant depth of indentation, and hysteresis undersinusoidal displacements. There was wide vari-ability in the magnitude of response for the five sub-jects who were tested, and their fingertip diameteror volume did not account for the observed vari-ability. A piecewise linear, lumped parametermodel with spring and dashpot elements was devel-oped to identify the mechanical parameters causingthe nonlinear response. The model predictions withonly one set of parameters for each subjectmatched the empirical data very well under a widevariety of stimuli. The model represents a compactdescription of the data and will be used to verifyand tune our finite element models of the fingertip.

Force Response of the Human Fingerpad toShear Displacement

The 2-DOF tactile stimulator was also used todeliver shear displacement ramp (0.5 to 16 mm/secfor a total shear displacement of 7 mm) at variousdepths of indentation of the fingerpad (0.5 to 3 mm)by a flat, smooth aluminum plate. 62 Only onesubject has been tested so far under these stimuli,and the resulting data has been analyzed with aview towards fine tuning the experimental protocoland parameters. The results show that at eachdepth of indentation, the shear displacement initiallycaused increasing skin stretch and shear force, fol-lowed by slipping of the plate across the skinsurface. The shear force-shear displacement wasalmost linear and slip occurred at around 3 mmshear displacement at all velocities. Low velocitiestended to cause stick-slip (as indicated byoscillatory shear force during slip), whereas theshear force decreased smoothly at higher velocities.

At increasing depths of indentation, slip occurred atlarger shear displacements, as is to be expected.The coefficient of static friction was obtained bymeasuring the slope of the normal and shear forcesat the incipience of slip for a given shear velocity.To a first approximation, it was found to be inde-pendent of shear velocity. More experiments ondifferent subjects are being initiated.

1.2.5 Human and Robot Hands: Mechanics,Sensorimotor Functions and Cognition

Sponsor

U.S. Navy - Office of Naval ResearchGrant N00014-92-J-1814

Project Staff

Dr. Mandayam A. Srinivasan, Dr. J. KennethSalisbury, Nathaniel I. Durlach, Dr. Robert H.LaMotte, Dr. Robert D. Howe,63 Jyh-Shing Chen,Kiran Dandekar, Louise Jandura, Steingrimur P.Karason

The premise of this University Research Initiativeproject is that the integrated study of human androbot haptics can provide complementary know-ledge of the processes of prehension and manipu-lation. From the human side, we wish tounderstand the basic mechanical, perceptual andstrategic capabilities that lead to the dexterity anddeftness we observe in human task performance.By studying the underlying competences thathumans bring to bear on task performance, weseek guidelines on how to build better robots.From the robotic side, we wish to understand howmechanism and sensor design choices can best bemade to maximize grasping and manipulative com-petences. By better understanding the mechanicaldemands of task performance, we seek to under-stand the performance demands which underlieskillful human manipulation.

The main components of the research conductedunder this project during the past year are (1)development of new hardware for robotic andhuman studies, (2) processing of robot sensorsignals and task-level control of the devices, (3)

61 R.J. Gulati and M.A. Srinivasan, "Human Fingerpad under Indentation. I: Static and Dynamic Force Response," ASME summerannual meeting, forthcoming.

62 J.D. Towles and M.A. Srinivasan, Frictional Properties of the Human Fingerpad, Report to the MIT Summer Research Program,1994.

63 Harvard University, Cambridge, Massachusetts.



experiments on human perception and control offorces using some of the devices. The subsectionsto follow provide descriptions of the results obtainedin each of the three topics.

Development of New Hardware for Roboticand Human Studies

During the past year, we have completed a high-precision glass smooth actuator (GSA) test bed thatwas used for experiments in human perception anddual actuator control. A force sensing fingertip hasbeen integrated with a 3-degree-of-freedom forceexerting device (the PHANToM) to provide an appa-ratus for conducting force-based touch perceptionexperiments. A passive three-finger hand, knownas the Talon, has been instrumented with a coarsearray of tactile sensors and multiple grasp forcesensors for use in grasp quality assessment exper-iments. A high-performance 8x8 tactile array hasbeen developed and interfaced. Several one- andtwo-axis force sensors were fabricated for exper-iments to determine mechanical properties of thehuman fingerpad as well as ability of humans tocontrol forces. For psychophysical and neurophys-iological experiments on tactile information obtainedthrough tool contact, an instrumented stylus wasconstructed to measure three-axis force signals.An apparatus capable of measuring humanfingerpad volume changes to a resolution of 1/100ml was built to investigate the compressibility of thefingerpads. An "Instrumented Active Object" whichcould be made to expand or collapse rapidly undercomputer control was developed and used in exper-iments on human grasp control.

Robot Sensing, Actuation, Control andPlanning

Sensing and Interpretation of Tactile andContact Force Information

Significant effort in the development of sensors andinterpretation methods has been undertaken at bothHarvard University and at MIT's Artificial Intelli-gence (AI) Laboratory. Dr. Howe's group atHarvard has developed greatly improved tactilearray sensors. Work on the manufacturing processhas created sensors that are easily made with aphotolithography-etching process. The resultingsensors are thin and flexible and easily attached toa wide range of robot finger tip shapes and sizes.In combination with new readout and signal proc-essing electronics, this has resulted in very fast andsensitive sensors: less than 0.1 gram noise perelement with complete readout of an 8x8 array in 5msec.

At the Artificial Intelligence Laboratory, we havefocused on perception of contact states observablefrom contact temporal force data. This work hasprogressed in four areas. Low frequency measuredforce data reflects information about contact con-straints. Estimating the form of these constraintsfrom force and motion measurements is essential tounstructured robot force control. A procedure wasdeveloped for estimating the directions of con-straint, and the geometric parameters defining theconstraint equations, for points in a Cartesian spaceand for the interaction of planar polygons. Theseresults may be applied to three-dimensional prob-lems by taking planar slices of three-dimensionalspace.

A procedure using the force and motion measure-ment residual from the sensor was also developedfor selecting the type of the contact constraint.After each new force and motion measurement, thebest estimate of the geometric parameters can becomputed. These can then be used in conjunctionwith an assumed form of the contact constraints tocompute a measurement error or residual. Bytesting the size of the error for all possible meas-urement models, the model which best fits the datacan be determined.

A theoretical foundation and set of definitions forrepresenting the forces and motions in contacttasks was defined. The forces and motions arerepresented as a network of measurement models(task network). Each node in the network repres-ents a particular measurement model. A set ofpoints in configuration space can be associatedwith each and every node. The points in configura-tion space then define the connections in thenetwork. These definitions provide a starting pointfor learning and computing networks for given tasksfrom models of the geometry and desired actions.

A task state observer was developed that deter-mines the probable current state of a task given atask network for the task. An observer was codedfor a network and set of measurement models forthe energy in the derivative of the strain energy.This signal is a good representation for high fre-quency short duration events. The observer wasimplemented in real-time and achieved 72 percentlabeling accuracy on a test data set. A simpledemonstration event driven robot program was alsowritten. The robot program uses the most probablerobot state, determined by the observer, to selectan action.

High-Precision Actuation

The high precision actuator project has designedand built a prototype glass smooth actuator (GSA).

367


The basic design relies on a dual actuator concept,in which a small actuator runs a high-frequencytorque control loop, and a large actuator creates alow frequency force bias using a large spring in thetransmission. The GSA has been used both as astimulation device for psychophysical experimentsand as a platform for advancing robot actuator con-cepts. It has been used in an initial set of exper-iments in human force control in which the subjectattempts to track a moving endpoint, while exertingconstant force. Another set of experiments isunderway in which the actuator follows a movingenvironment, while acting to maintain constantforce. The current performance of the GSA is verygood. The actuator has a force control dynamicrange of 500:1. It is also capable of controlling forceat 40 Hz bandwidth.

Grasp Quality Assessment

In conjunction with Dr. Salisbury's group, Dr.Pratticizzo from the University of Pisa (while on afive-month fellowship at the MIT Artificial Intelli-gence Laboratory) has developed an approach formeasuring the quality of whole hand grasps. Thework combines a solid theoretical approach tomeasuring the "distance" from losing the grasp witha practical computational approach to real-timemeasurement of grasp quality. The approach uti-lizes contact geometric and friction information todefine the maximum possible grasp quality; with themeasurement of contact forces, it also permitsassessment of the instantaneous quality as a per-centage of the maximum attainable quality. Theapproach provides a means for estimating themaximum quality possible even in the absence offull sensory information, provides a method for opti-mizing internal grasp forces and can be used tospecify limits on dynamic motions to avoid graspfailure. Real-time processing of touch and forcesensor data from the talon hand has been imple-mented to demonstrate online grasp quality assess-ment.

Grasp Gaits

We have developed an approach for planning multi-finger grasp gaits. Truly dexterous manipulation ofobjects with the fingers requires repeatedsequences of moving the object and repositioningof the fingers. Our approach permits planningsequences of repositioning and regrasping anobject with two and three finger grasps. It considersloss of mobility due to figure range limits and lossof constraint due to friction and force closure limita-tions. The enormous search tree has been suffi-

ciently reduced by a series of planning heuristicsthat real-time planning for rotation of an unknownplanar object is possible. A real-time graphic simu-lation permits viewing progress of the plannedactions.

Robot Control

Professor Howe's group at Harvard has imple-mented a Cartesian stiffness controller on the two-fingered Planar Manipulator in the Harvard RoboticsLaboratory. This controller permits accurate controlof the apparent stiffness and the center of compli-ance of a grasped object. The controller will beused in planned experiments on the role of tactileinformation in real-time control of precision manipu-lation tasks.

Human Perception and Control

The Human Haptics Group at MIT has investigated(1) the mechanics of the human fingerpad and itsrole in tactile sensing, and (2) human perceptionand control of forces exerted by the fingerpads.The former, described earlier and partially sup-ported by this grant, includes experiments to deter-mine the mechanical behavior of the fingerpadunder normal and shear forces, as well as thecompressibility of the fingerpad. Nonlinear lumpedparameter models were developed and finiteelement analysis of multilayered 3D fingertipmodels were carried out to match biomechanicaland neurophysiological data.

To quantify human perception and control of forces,experiments were conducted under a wide varietyof conditions: (1) tracking of visual displays of staticand dynamic force traces with a stationaryfingerpad (isometric case), (2) maintaining constantcontact force on a moving robot end-effector(isotonic case), (3) sensing and control of torqueapplied on the shaft of an "Instrumented ScrewDriver", and (4) control of grasp forces on an"Instrumented Active Object". The data from eachof these experiments have been analyzed with theviewpoint of developing engineering specificationsof human haptic performance. The Yale group hasperformed psychophysical and neurophysiologicalexperiments on perception and control of tactileinformation obtained by touching an object with atool. An instrumented stylus was used in bothactive and passive touch experiments on the dis-crimination of softness by human subjects and inrecording cutaneous neural responses to the samestimuli applied to monkey fingerpads. Brief descrip-tions of each of the experiments and modelsdescribed above are given in the sections below.



Isometric Force Tracking Ability of Humans

In the experimental setup, a human subject tracksvisual images of force traces displayed on themonitor by applying appropriate normal forcesthrough a fingerpad that is in contact with a forcesensor. In these experiments, the finger moves byonly fractions of a millimeter, thus approximatingisometric muscular contraction conditions. We hadcompleted one set of experiments on such forcetracking last year, and now we have expanded therange of target forces and their frequency. Duringtracking constant and sinusoidal force targets (threesubjects with three trials per stimulus), the meanabsolute error increased with constant force magni-tude, target sinusoid frequency and amplitude. Theerrors for a sinusoid of a given amplitude are 5 to40 times higher than those for constant forcetargets with the same magnitude. Even at relativelylow frequency of 2 Hz, the errors can be higherthan 50 percent of the sinusoid amplitude at allamplitudes.

Isotonic Force Control Ability of Humans

We employed the high-precision glass smoothactuator (GSA) to measure the ability of humansubjects to maintain a constant force (0.1 to 0.8 N)by pressing their fingerpads on the actuator's endeffector while it was moving sinusoidally (2 to 16degrees in amplitude; 0.5 to 16 Hz frequency).During each trial, the robot maintained a constantposition for the first 10 seconds, and during the first8 seconds, the subjects tracked a constant targetforce displayed as a line on a monitor. The monitorscreen was then blanked out, and after twoseconds, the actuator started moving sinusoidally ata preprogrammed frequency and amplitude, but thesubjects were asked to maintain the same force asbefore. All the subjects were able to perform thetask with very little drift in mean force. However,the deviations from the mean were in phase withthe actuator motion, and various error measureschanged differently with stimulus parameters. Themean absolute error increased with frequency andamplitude almost linearly, but remained constantwith respect to target force magnitude.

Torque Sensing and Control

The humaninvestigatedScrewDriver

ability to sense and control torque wasin experiments with the Instrumented(ISD). 64 The ISD is comprised of a

single shaft, which is supported by low frictionbearings, and is connected to a reaction torquesensor and a magnetic particle brake. Angularposition of the shaft is measured by an incrementaloptical encoder. In all cases the subjects graspedthe handle of the ISD between the thumb and indexfinger of their dominant hand and turned the shaftclockwise for 180 degrees against a constantresistive torque applied by the magnetic particlebrake. The magnitude of this resistive torque wasvaried across different trials. Two types of exper-iments were conducted: discrimination experimentsto determine the human resolution in sensingtorque and control experiments to determine thehuman motor capability in controlling torque.

All torque discrimination experiments used a one-interval, two-alternative, forced-choice paradigmwith no feedback to the subject. The referencetorque value was 60 mN-m and the comparisonvalues were equal to 5 percent, 10 percent, 20percent and 30 percent of the reference torque. Inaddition, training runs were conducted with a com-parison value of 50 percent of the reference torqueuntil the subject response was 90 percent correct.The just noticeable difference for torque was foundto be 12.7 percent for the reference torque of 60mN-m. During some of the trials, in addition torecording the stimulus and the subject's response,the resistive torque, the output of the torque sensorand the angular position of the shaft over time werealso recorded. These data are used to make com-parisons between the motor performance in the dis-crimination task and the control task.

For the control experiments, subjects were asked tomaintain a constant angular velocity while turningagainst the constant resistive torque. The value ofthe angular velocity was up to the subject tochoose, but they were asked to try and use thesame value for each trial. Because of the physics ofthe ISD, attempting to maintain a constant angularvelocity is directly related to attempting to apply andmaintain a constant torque during shaft motion. Theconstant resistive torque values used were thesame as for the discrimination experiments. Asbefore, the resistive torque, the output of the torquesensor, and the angular position of the shaft wererecorded over time. Comparison of the time pro-files of angular velocity indicate that even whensubjects were trying to maintain a constant angularvelocity in the control experiments, their perfor-mance was not significantly better than when theywere trying to discriminate the torques.

64 L. Jandura and M.A. Srinivasan, "Experiments on Human Performance in Torque Discrimination and Control," in Dynamic Systems

and Control, Ed. C.J. Radcliffe (Chicago: ASME, 1994), DSC-Vol. 55-1.

369


A curious phenomena observed rather consistentlyin all of the data is the occurrence of peaks in thevelocity and acceleration profiles at about 0.1second intervals. To further investigate this phe-nomena, the power spectral density of the middlethird of the angular velocity profile was calculated.Although there is some tendency for the discrimi-nation PSDs to be single-peaked while the controlPSDs are double-peaked, this was not observedconsistently across all subjects. However, in allsubjects, most of the frequency content was lessthan about 15 Hz for both the discrimination andcontrol experiments.

Grasp Control of an Instrumented Active Object

Most of the investigations of human grasp controlreported in the literature pertain to passive objects.In order to test the limitations of the human motorsystem in compensating for sudden disturbances,we fabricated an "Instrumented Active Object."65 Itmainly consisted of a pneumatic cylinder whosepiston could expand or contract through the opera-tion of computer controlled valves. A positionsensor monitored the motion of the piston while two2-axis force sensors measured the normal andshear forces applied by a subject's fingerpads ontwo circular plates when the object was held in apinch grasp. The plate surfaces were either pol-ished aluminum or sandpaper depending on theexperiment. A visual display on a computer monitorindicated the force of pinch grasp the subjects wererequired to apply, and when subjects achieved it,after a random time-delay the object contractedsuddenly (170 to 240 mm/s velocity; 19 mm dis-placement). When initial grasp forces were lessthan 1.8 N, the subjects dropped the object all thetime (three subjects; ten trials per grasp force valueper subject), whereas they held it in increasingnumber trials as the grasp forces increased. No sig-nificant difference between aluminum and sandpaper surfaces were observed. In one trial wherethere was a small slip, a perturbation in forceapplied by the thumb is seen at around 100 ms, butthe object did not drop. In another trial in which thesubject dropped the object, oscillations in forceapplied by the middle finger began at 60 ms, andthose oscillations in force applied by the thumbbegan at about 80 ms. Both these kinds of oscil-lations continued until the object was dropped atabout 100 ms. These data are being analyzedfurther for a deeper understanding of the dynamicsof grasping.

Processing of Tactile Information Obtained byTouching an Object with a Tool

The capacity of humans to discriminate the softnessof rubber specimens differing in compliance bymeans of a tool (stylus) was measured under avariety of experimental conditions. Under activetouch (AT), the observer held the stylus in a pinchgrip and either tapped, pressed or bounced itagainst the rubber specimens during tests requiringeither ranking or pairwise discrimination. Underpassive touch (PT), one end of the stylus indentedthe stationary fingerpad with a base force main-tained by a torque motor while the other end wastapped, pressed or bounced (dropped onto) by therubber specimen. Discrimination of softness wassuperior during AT than during PT for both tappingand pressing, suggesting that proprioceptive cuescontribute to optimal tactual performance. The factthat discrimination was more accurate for tappingthan for pressing under PT suggests the importanceof tactile cues generated by the initial impact of thestylus against the surface of the rubber. Bouncingthe specimen (attached to a pivoted arm) againstthe stylus under PT produced characteristicoscillatory contact forces on the skin that subjectscould readily discriminate indicating the usefulnessof vibratory signals generated by the impact of alightly held tool against an object.

The same stimuli used in the PT tests in humanswere applied via the stylus to the fingerpad of theanesthetized monkey. Evoked nerve impulseactivity was recorded electrophysiologically fromslowly-adapting type I and rapidly-adapting type I(RA) and type II (PC) mechanoreceptive peripheralnerve fibers. Increases in the hardness of thespecimen tapped or bounced against the stylus elic-ited nerve-impulse trains with shorter interspikeintervals and shorter durations for all three fibertypes. Fiber responses of SAs during pressingwere related only to the force rate (velocity ofpressing) and did not discriminate between speci-mens. Thus, the psychophysical measures can beaccounted for by the responses of cutaneous mech-anoreceptors. An implication of this research forthe development of robotic hands is the need todevelop sensors and decoding algorithms for trans-ducing and recognizing dynamic changes in contactforce produced by the impact of a tool against anobject.

65 S.P. Karason and M.A. Srinivasan, "Human Grasp Control of an Instrumented Active Object," submitted to the 1995 ASME winterannual meeting, San Francisco, California.



1.2.6 Publications

Annaswamy, A.M., and M.A. Srinivasan. "A Studyof Dynamic Interactions between Haptic Inter-faces and Compliant Fingerpads." Proceedingsof the Motion Control Workshop, Berkley,California, March 1994.

Annaswamy, A.M., and M.A. Srinivasan. "The Roleof Compliant Fingerpads in Grasping and Manip-ulation: Identification and Control." Institute ofMathematics, Springer Verlag. Forthcoming.

Bicchi, A., and D. Prattichizzo. "A Standard Formfor the Dynamics of General ManipulationSystems." Submitted to the IEEE InternationalConference on Robotics and Automation, 1995.

Bicchi, A., P. Petreschi, and D. Prattichizzo."Control of Force Decomposition in EnvelopingGrasping." In Proceedings of the Symposium onRobot Control, 1994.

Bicchi, A., and D. Prattichizzo. "Controllability ofWhole-Arm Manipulation." 33rd CDC, forth-coming.

Dandekar, K., and M.A. Srinivasan. "Tactile Codingof Object Curvature by Slowly Adapting Mech-anoreceptors." In Advances in Bioengineering.Ed. M.J. Askew. Chicago: ASME, 1994,BED-Vol. 28, pp. 41-42.

Dandekar, K., and M.A. Srinivasan. "A three-dimensional Finite Element Model of the MonkeyFingertip for Predicting Responses of SlowlyAdapting Mechanoreceptors." ASME summerannual meeting. Beaver Creek, Colorado. Forth-coming.

Eberman, B., and J.K. Salisbury. "Application ofChange Detection to Dynamic Contact Sensing."Int. J. Robotics Res. 12(5): (1994).

Gulati R.J. and Srinivasan M.A. "Human Fingerpadunder Indentation. I: Static and Dynamic ForceResponse." ASME summer annual meeting.Forthcoming.

Jandura, L., and M.A. Srinivasan. "Experiments onHuman Performance in Torque Discriminationand Control." In Dynamic Systems and Control,Ed: C.J. Radcliffe. Chicago: ASME, 1994,DSC-Vol. 55-1, pp. 369-375.

Karason, S.P., M.A. Srinivasan, and A.M.Annaswamy. "Tactile Sensing of Shape." Centerfor Information Driven Mechanical Systems

(CIDMS) Workshop Proceedings, Dept. of Mech.Eng., MIT, September 1994.

Karason, S.P., and M.A. Srinivasan. "Human GraspControl of an Instrumented Active Object." Sub-mitted to the 1995 ASME winter annual meeting.

LaMotte, R.H., M.A. Srinivasan, C. Lu, and A.Klusch-Petersen. "Cutaneous Neural Codes forShape." Can. J. Physiol. Pharm. 72: 498-505(1994).

LaMotte, R.H., J. Zhang, and J. Yan. "TactileSignals Generated by a Tool Used to Discrimi-nate the Softness of Objects." Soc. Neurosci.Abstr. 20: 1381 (1994).

Peine, W.J., J.S. Son, and R.D. Howe. A PalpationDevice for Artery Localization in LaparoscopicSurgery. Harvard University, Robot Sensor Lab-oratory Technical Report, September 1994.

Petreschi, P., D. Prattichizzo, and A. Bicchi. "Artic-ulated Structures with Tendon Actuation forWhole-Limb Manipulation," Proceedings of the1994 International Conference on Robotics andAutomation, San Diego, California, May 1994.

Roby, F.L., K. Dandekar and M.A. Srinivasan.Study of Fingertip Deformation Under Indenta-tions by Circular and Rectangular Indentors.Report to the MIT Summer Research Program,1994.

Son, J.S., E.A. Monteverde, and R.D. Howe. ATactile Sensor for Localizing Transient Events inManipulation. Harvard University, Robot SensorLaboratory Technical Report, May 1994.

Srinivasan, M.A., and K. Dandekar. "An Investi-gation of the Mechanics of Tactile Sense UsingTwo-dimensional Models of the PrimateFingertip." J. Biomech. Eng. Forthcoming.

Srinivasan, M.A., and R.H. LaMotte. "Tactual Dis-crimination of Softness." J. Neurophys. 73(1):88-101 (1995).

Towles, J.D., and M.A. Srinivasan. Frictional Prop-erties of the Human Fingerpad. Report to theMIT Summer Research Program, 1994.

Thesis

Babiec, W.E. In vivo Volume Changes of theHuman Fingerpad under Indentors. S.B. thesis.Dept. of Mech. Eng., MIT, 1994.

371


1.3 Virtual Environment Technology forTraining

1.3.1 Research Testbed

SponsorU.S. Navy -

SystemsContract

Naval Air Warfare Center TrainingDivisionN61339-94-C-0087

Project Staff

Nathaniel I. Durlach, Dr. David Zeltzer, Dr. Jeng-Feng Lee, Walter A. Aviles, James H. Bandy,Rakesh Gupta, Dorrie Hall, Jonathan Pfautz,Nicholas Pioch

During the second year of the Virtual EnvironmentTechnology for Training (VETT) program, the majorgoal of the Testbed project was to continue devel-opment of the basic facilities and architecture of theTestbed so that meaningful training experimentscould be initiated. The following efforts were com-pleted or are underway during the second year.

Implementation of Research SupportMethods and Tools

Testbed personnel have worked closely with per-sonnel of the Satellite projects in an effort to refinethe software and hardware interfaces that each Sat-ellite will require to conduct experimental researchusing the Testbed. We have recently completedthe Phase-1 implementation of the Experimenters'Interface (El), a distributed software package thatwill allow experimenters-either VETT investigators,or personnel from other military or governmentlaboratories-to make use of the VETT Testbed asa research tool.

Performance Characterization

Test hardware and software for measuring the per-formance of the Testbed have been completed.Using these tools the performance of the Testbed(e.g., graphics, auditory, and haptic update ratesand delays) has been precisely measured. This isimportant to characterize the system itself for devel-opment purposes, but it is also necessary toprovide baseline data for Satellite experimentswhich will be run on the Testbed.

Development of Requirements andSpecifications for Software Architectureand Multimodel VE Database Required toSupport Physically-based Modeling ofObjects and Actions within VirtualEnvironments

In collaboration with the Haptics and Sensorimotorgroups, the design of a common representationalframework is underway, beginning with physicallybased, articulated, rigid bodies. This represent-ational framework will be utilized to model andrender the visual, auditory, and haptic interactionswhich characterize virtual environments.

Upgrade Testbed Graphics Capabilities andPerformance

A RealityEngine2 graphics subsystem was pur-chased and integrated into our SGI Onyx system.This system not only allows improved graphics per-formance but also increases compatibility withNaval Air Warfare Center Training Systems Division(NAWCTSD) systems. The Core TestbedVE-prototyping software, known as 3D, now makesuse of the anti-aliasing capabilities of theRealityEngine2. In addition, work is currentlyunderway to utilize more than one processor withinour Onyx system to support graphics and generalcomputation. As one aspect of this, Core Testbedpersonnel are exploring the substitution of the SGIPerformer software library for the SGI GL graphicslibrary used in the 3D system. The Performerlibrary directly supports the use all RealityEngine2features and the use of multiple processors forgraphics processes. A Virtual Research VR4 head-mounted display was recently purchased and hasbeen integrated into the Testbed. This LCD-basedHMD has a larger horizontal field-of-view (-40'horizontal) than our previous HMD.

Integration and Evaluation of ImprovedHaptic Interfaces, Speech InteractionSystems, and Auditory Displays

The PHANToM haptic display device was incorpo-rated into the Testbed, and several demonstrationapplications have been implemented. A BBN Harkspeaker-independent, connected speech recogni-tion system was purchased and integrated. Theauditory event representation, modeling, and ren-dering process received particular attention duringthe second year. As part of the aforementionedmultimodal object database efforts, initial methodsof representing and modeling acoustic events havebeen developed. In addition, an audio-event serverand sound spatialization system were developed.



The audio-event server allows the playback of pre-recorded audio sounds (i.e., based either uponrecordings or generated by physical models). Hun-dreds of sounds may be simultaneously generated,mixed, and output on eight physical output chan-nels. These output channels in turn are spatializedusing commercial sound spatializers.

A Crystal River Engineering BEACHTRON2-channel sound spatializer has been integratedinto the Core Testbed. Three new spatializationsystems (the Ctronll and the Gargantutron fromCrystal River Engineering and the system fromTucker-Davis Technologies) were acquired underGrant N61339-93-C-0104. The goal of this grantwas to create a "next-generation" system withexpanded capabilities to further the field of soundspatialization for auditory virtual environments.Since no single system appeared capable of deliv-ering all of the envisioned needs of the field (e.g.,enhanced computational throughput, increasedinternal storage, use of off-the-shelf rather thanspecial-purpose hardware, greater prog.ammability,etc.), three solutions were simultaneously devel-oped. The Gargantutron, based on mature hard-ware and software, was developed to ensurecontinuity between existing technology and newapproaches while increasing throughput and effec-tive storage by a factor of four. The Gargantutronwill be used primarily for generation of "natural-sounding" spatial sound, including environmentswith reflective surfaces. The Ctronll was developedusing off-the-shelf DSP cards and frequency-domain filtering techniques. As such, the Ctronllshould prove to be a fairly inexpensive system forthe simulation of relatively simple echoic environ-ments and a proof of concept for frequency-domainrepresentations of HRTFs in real-time environ-ments. Finally, the system developed by Tucker-Davis technologies is highly modular and extremelyflexible.

All three of the new sound spatialization systemswill be tested to ensure that they meet their designspecifications. Following this initial evaluation, thesystems will be employed in various projects underthe VETT program. Both the Gargantutron and theCtronll will be used primarily in generating natural-sounding spatial sound for projects running on theCore Testbed. While developed primarily for thesimulation of natural-sounding acoustic environ-ments, the Tucker-Davis system's flexibility makes itan ideal tool for use in further auditory-motor exper-iments in the Sensorimotor Loop project.

Implementation of the Officer of the DeckTraining Application and Pilot Experiments

The first training system under development on theCore Testbed involves training a submarine officerof the deck (OOD) to perform in-harbor navigationon a surfaced submarine. This is a task thatinvolves a dozen or more personnel, most of whommake up the below-deck piloting team that operatesperiscopes, radar and sonar systems, and performsbasic navigational calculations. The OOD, sta-tioned on the conning tower of the surfaced boatduring this task, is essentially responsible for moni-toring the progress of the boat as it negotiates amarked channel. This officer verifies that the navi-gational suggestions of the piloting team are correctand that the course of the submarine is indeed con-forming to the navigational plan.

The U.S. Navy, in fact, currently has simulatorsystems for the below-deck piloting team, but thereis no corresponding simulator or formal trainingprogram for junior officers who must learn toperform this task- essentially in an informal on-the-job apprenticeship. As part of the VETTprogram, a VE system has been developed thatallows an OOD trainee to view a particular harborand associated waterway through a head-mounteddisplay, receive spoken reports from a simulatedpiloting team, give spoken commands to thehelmsman, and receive verbal confirmation ofcommand execution from the helm. The trainingeffectiveness of the system-including instructionalaids unique to VE systems-will be evaluatedexperimentally with a small number of subjects.

The design of the OOD application has beenaccomplished with the close cooperation of theTraining Satellite at BBN, and the members of thisgroup have designed the experimental program forevaluating the OOD training application.

1.3.2 Sensorimotor Satellite

SponsorU.S. Navy - Naval Air Warfare Center

Training System DivisionContract N61339-93-C-0055Contract N61339-94-C-0087

Project Staff

Nathaniel I. Durlach, Professor Richard M.Thomas E.v. Wiegand, Dr. Wendelin L.Dr. Barbara G. Shinn-Cunningham, EvanDavid C. Lossos, Susan E. Born, StephenLukasz A. Weber, Erika N. Carmel

Held, Dr.Sachtler,F. Wies,V. Baird,

373


Figure 1. The virtual workbench concept in schematicform.

Work on the sensorimotor satellite in the VETTprogram is directed towards evaluating humanresponse to alterations in sensorimotor loops asso-ciated with the use of virtual environment (VE) tech-nology. Even in cases in which the design goal iscomplete realism, alterations in the form of dis-tortions, time delays, and statistical variability("noise") are inevitable (1) in the presentation ofinformation to the human operator (because of limi-tations in displays), and (2) in the sensing of theoperator's responses via the controls in the human-machine interface. Such alterations will occur in allsensorimotor loops involved in the system,including, for example: (1) those associated withvisual and proprioceptive sensing, and motorcontrol of hand position and posture; and (2) thoseassociated with visual and auditory images of com-puter generated scenes and motor control andproprioceptive sensing of head position and orien-tation. Furthermore, in some cases, it appears thatspecific training goals can be achieved more effi-ciently by purposefully introducing distortions tohighlight or magnify certain features or relationshipsin the training situation. In these cases, as well asthose in which the alterations are the result of tech-nological limitations, it is essential to determine thehuman response to the altered characteristics of thesensorimotor loops involved.

Figure 2. The virtual circuit board display andPHANToMTM unit mounted below the mirror.

In the past year, we have addressed these issuesin three areas. First, we have continued our basicpsychophysical experiments in the area of adapta-tion to alterations in the visual-proprioceptivesensorimotor loop (using the apparatus andparadigm described in RLE Progress Report No.136). Second, we have completed a set of exper-iments elucidating a model of adaptation to alter-ations in the loop involving auditory localization.

Third, we have begun a new area of research uti-lizing a VE system that includes an alternative man-machine interface: the virtual workbench. Themultimodal virtual workbench was developed toboth study human sensorimotor capabilities in near-field space and to explore cognitive performanceand training in complex tasks involvingspatialization of information in VEs. The systemcombines state-of-the-art haptic feedback andvisual display technologies with audio capabilities tocreate an alternative "front-end" interface for thecore VETT testbed.


Virtual WorkbenchSchematic View showing Subject

performing manual manipulation taskusing the PHANToM device.


performance (using memory and reasoning tasks).Development efforts are also being directed towardproducing a highly veridical training task involvingmanual interaction with virtual electronic circuitboards and components.

1.3.3 Haptics Satellite

Project Staff

Dr. J. Kenneth Salisbury, Dr.Srinivasan, Nathaniel I. Durlach,Jones, Dr. David L. Brock, GeraldHugh B. Morgenbesser, Hong Z.Zilles

Figure 3. Photograph of subject seated at the virtualworkbench.

Currently available HMDs and their associatedtracking and graphics rendering systems operate atthe limits of current technology. Despite their greatcost, these systems are still insufficient to present avirtual world free of performance-reducing andnauseogenic effects. In the virtual workbench, wecombine visual display and haptic feedback technol-ogies into a form in which the user looks andreaches into a virtual space (as opposed to trying toimmerse the entire body). In this way, we are ableto sidestep many of the technical shortcomings ofHMDs, enabling the virtual workbench to serve as aprecision instrument for measuring human sensoryand cognitive performance. The virtual workbenchwas specifically designed for precise overlay ofvisual and haptic spaces, providing a cohesiveworld percept. The effects (on human perfor-mance) of deviations from precise registration of thespaces, as well as distortions, delays, and othermanipulations (e.g., modification of forces in hapticspace), will be studied using this system. In addi-tion to these sensorimotor effects, we are preparingexperiments designed to examine the influence ofthe manipulations mentioned above on cognitive

Mandayam A.Dr. Lynette A.L. Beauregard,Tan, Craig B.

In the past year, we have developed device hard-ware, interaction software and psychophysicalexperiments pertaining to haptic interactions withvirtual environments. Two major devices for per-forming psychophysical experiments, the Linear andPlanar Graspers, have been fitted with additionalsensors for improved performance. Another hapticdisplay device developed previously, thePHANToM, has been used to prototype a widerange of force-based haptic display primitives. Thethree devices were used to perform psychophysicalexperiments. Copies of the PHANToM, along withdemonstration software, have been delivered to thesponsor and the core testbed. In addition, two full-scale tool-handle haptic interface devices havebeen designed and constructed.

We had previously developed software for the dem-onstration of haptic displays with the PHANToM,such as interactions with virtual pushbuttons, slidingfriction and surface texture displays, virtual screwturning, and virtual grasping as well as two-personhaptic "pong" with two PHANToMs. Additional soft-ware was developed for visual and haptic renderingof deformable toroids and needle biopsy pro-cedures. Different algorithms for renderingCoulomb and static friction in haptic displays weredeveloped. Stroking of surfaces with varying com-pliance in the normal direction was demonstratedand experiments revealed a need for preservingpassivity of interaction. Stable two-finger graspingand stacking of two virtual blocks was demon-strated with two tool-handle haptic interfaces.Network communication between a haptic renderer(Pentium PC) and graphics renderer (SiliconGraphics) was implemented. The development andspecification of a software format to facilitatestandard multimodal representation of convex andconcave polyhedral objects was started. This formatis necessary to allow consistent visual and hapticrendering of objects in virtual environments. Multi-modal rendering of a triangular meshed polyhedron

375


of moderate complexity (200 facets) was demon-strated.

Software was also developed for the conduction ofpsychophysical experiments with the linear grasper,planar grasper and the PHANToM. The lineargrasper is now capable of simulating fundamentalmechanical properties of objects such as compli-ance, viscosity and mass during haptic interactions.Virtual wall and corner software algorithms weredeveloped for the planar grasper, in addition to thesimulation of two springs within its workspace. Avariety of haptic rendering algorithms for displayingthe shape, texture and friction of solid surfaceshave been implemented on the PHANToM. Wecompleted a series of psychophysical experimentswith the Linear Grasper to characterize the ability ofthe human haptic system to discriminate funda-mental mechanical properties of objects.66 Theseincluded several experiments to measure themanual resolution of viscosity and mass. In addi-tion, experiments were also performed to study theinfluence of mechanical work and terminal force onthe perception of compliance.

Utilizing the planar grasper, we have completed aset of multimodal psychophysical experimentsinvestigating the influence of visual information onthe haptic perception of stiffness in virtual work-spaces. Results indicate that the perception ofstiffness for objects like virtual pushbuttons can besignificantly altered by presenting visually skewedpositional information to the subject. We have initi-ated psychophysical experiments using thePHANToM to determine the feasibility of varioushaptic display algorithms for shape, texture and fric-tion of solid surfaces. The results show that evenwhen the user is exploring nominally flat surfaces,he can be made to feel as if he is interacting with ashaped and textured surface by appropriatelyvarying the direction of the force reflected back tothe user. These new algorithms permit themapping of a shape or texture onto a polygon, sothat they may be used in haptic rendering in thesame way that texture mapping and color shadingare used in graphics rendering. Our psychophys-ical experiments are aimed at characterizing theeffectiveness of such computationally efficient simu-lations and rendering algorithms in conveying thedesired object properties to the human user.

Publications

Beauregard, G.L., M.A. Srinivasan, and N.I.Durlach. "Manual Resolution of Viscosity andMass," Submitted for presentation at the ASMEwinter annual meeting, San Francisco,California, 1995.

Srinivasan, M.A. "Haptic Interfaces." In VirtualReality: Scientific and Technical Challenges.Eds. N.I. Durlach and A.S. Mavor. Report of theCommittee on Virtual Reality Research andDevelopment. Washington, D.C.: NationalResearch Council, National Academy Press,1994.

Srinivasan M.A. "Virtual Haptic Environments: FactsBehind the Fiction," Proceedings of the EighthYale Workshop on Adaptive and LearningSystems, Center for Systems Science, NewHaven, Connecticut: Yale University, 1994.

Tan, H.Z., and N.I. Durlach, G.L. Beauregard, andM.A. Srinivasan. "Manual Discrimination of Com-pliance Using Active Pinch Grasp: the Roles ofForce and Work Cues." Percept. Psychophys.Forthcoming.

Tan H.Z., M.A. Srinivasan, B. Eberman, and B.Cheng. "Human Factors for the Design of Force-reflecting Haptic Interfaces." In DynamicSystems and Control, Ed. C.J. Radcliffe.Chicago: ASME, 1994, DSC-Vol.55-1, PP.353-359.

1.4 Research on Improved SonarDisplays: A Human/Machine ProcessingSystem

Sponsor

U.S. Navy - Office of Naval ResearchGrant N00014-93-1-1198

Project Staff

Dr. David Zeltzer, Dr. Jeng--Feng Lee, Nathaniel I.Durlach

The human/machine processing system (HMPS)program began August 1, 1993. The overall objec-

66 H.Z. Tan, N.I. Durlach, G.L. Beauregard, and M.A. Srinivasan, "Manual Discrimination of Compliance Using Active Pinch Grasp: theRoles of Force and Work Cues," Percept. Psychophys., forlhcoming: G.L. Beauregard, M.A. Srinivasan, and N.I. Durlach, "ManualResolution of Viscosity and Mass," submitted to the 1995 ASME winter annual meeting, San Francisco, California.



tives of this research, as discussed in the proposal,are to:

explore general techniquesinteraction with sonar dataacoustic propagation model,

for multisensoryin light of an

* develop a working human-machine processingsystem (HMPS) testbed embodying the mostpromising of these techniques, and

* evaluate the effectiveness of the experimentalHMPS.

Nearly all the first year objectives have been met.A significant effort was required to become familiarwith naval sonar operations; interviews with domainexperts (i.e., sonar technicians) and a task analysisbased on these interviews were required beforesystem design could begin. At present, the user-centered system design of the HMPS is largelyaccomplished, and HMPS implementation isunderway.

An extensive literature review and a survey of avail-able undersea acoustic propagation models wereconducted; this included discussions with underseaacoustics experts at MIT and elsewhere. No extantmodels were deemed appropriate for the HMPSproject. The ray tracing paradigm has been identi-fied as the most appropriate and effective techniquefor computing undersea sound propagation. A mod-eling and simulation package based on modifica-tions to ray tracing algorithms reported in Ziomek67

is currently being implemented.

In order to design a human/machine interface forimproved sonar display, it is necessary to under-stand and analyze the tasks actually performed bysonar crews. To this end, a number of interviewshave been conducted with two domain experts,both active-duty sailors, one of whom has servedas sonar supervisor, and the other as a sonar tech-nician, on a U.S. Navy fast attack submarine.Based on these interviews, we have developed atask analysis of the functions performed by thesonar supervisor, and we have subsequentlydesigned the functional architecture of the HMPS.In addition, we have been in contact with a numberof other submarine officers in the course of theVETT project, and their input has been valuable forthe HMPS work as well.

We have chosen to focus the HMPS on the require-ments of the fast attack submarine, rather than a

ballistic missile submarine or a surface ship, for thefollowing reasons. First, a ballistic missile subma-rine is a strategic asset and does not pursue offen-sive tactical operations. Sonarmen on a fast attacksubmarine, therefore, have a range of offensive tac-tical operations to be supported by an HMPS thatare simply not encountered on a ballistic missilesubmarine. Surface ships, unlike either class ofsubmarines, can often call on support from aircraft,other naval vessels and satellites in support of tac-tical operations. The fast attack submarine,however, must operate as a self-contained sensorand weapons platform in a complex and noisyundersea environment.

Once an acoustic contact is detected on a fastattack sub, it is the responsibility of a singleindividual-the sonar supervisor-to form acoherent picture of the immediate tactical situation,and to make recommendations to the connregarding control of the vessel. In addition toviewing "waterfall" and similar displays and inter-acting with the sonar technicians in the sonarshack, the sonar supervisor must construct a"mental model" of the tactical situation with essen-tially no supporting documentation, hardware, orsoftware. It all must be done "in his head", basedon his knowledge and experience. The purpose ofthe HMPS, then, is to provide tools for enhancingsituational awareness and to support "what if" simu-lations of sensor, vessel, and weapons operations.

We have obtained undersea terrain data fromseveral sources, and we have selected a data setappropriate for initial HMPS implementation. Inaddition, we have acquired several geometricmodels of submarines which have also been inte-grated into the HMPS. Furthermore, we found thatocean eddy data is available from researchers atWoods Hole.

The initial implementation of the HMPS is inprogress. The ray tracing model is being coded; alarge effort has gone into programming an efficientray tracing package and programming efficientgraphics modules to display ray tracing results andacoustic shadow volumes.

Once the initial HMPS implementation is complete,work will begin on a multimodal operator interface.We expect this work to be underway late in thethird quarter of the second year, with subject evalu-ations to begin in the fourth quarter, as called for inthe original proposal.

67 L.J. Ziomek, "The RRA Algorithm: Recursive Ray Acoustics for Three-Dimensional Speeds of Sound, IEEE J. Oceanic Eng. 18(1):

25-30 (1993).

377


We have identified sources for platform and biolog-ical signatures, and we are prepared to incorporatesuch acoustic databases into the HMPS. As timepermits, we may investigate cooperativeantisubmarine warfare (ASW) techniques. In partic-ular, we will be able to simulate remote activesources which serve as acoustic illuminators for thepassive sensors of an attack submarine.

1.5 Development of InertialHead-Tracking Technology

Sponsor

National Aeronautics and Space Administration/Ames Research CenterGrant NCC 2-771

Project Staff

Eric M. Foxlin, Nathaniel I. Durlach

The inertial head-tracker program began in Sep-tember 1992. Our objective is to develop a newhead-tracking technology for HMD applicationswhich offers large working volume, high fidelity, andunencumbered operation.

Commercially available tracking systems suffer froma variety of limitations which restrict their use inhuman-machine interfaces. Mechanical trackersoffer fast response times and good accuracy, butrequire physical attachment to the head which limitsrange of motion and user comfort. Optical methodsare expensive, require a clear line-of-sight, andusually necessitate a trade-off between precisionand working volume. Magnetic trackers (such asthose available through Polhemus or AscensionTechnologies) are reasonably priced, but have asmall working volume and suffer from noise andmagnetic field distortions caused by metallicobjects. Finally, acoustic systems based upon atransmitted signal's time-of-flight are available atreasonable cost and offer a large working volume.However, the speed of these systems is limited bythe speed of sound, and they are sensitive toacoustic interference.

Our system departs from the prior state-of-the-art inhead-tracking by relying on inertial sensors for itsprimary measurements. This offers the potentialadvantage of a self-contained head-mounted meas-urement device that does not rely on signals fromother equipment in the lab, making possible largerange and freedom from interference. The main dif-ficulty with inertial systems is drift, which we correctwith an inclinometer and compass. The orientation

tracker therefore retains its large range and relativeinvulnerability to interference. Position can betracked more accurately than orientation by opticaland acoustic systems, but provides greater difficul-ties for inertial systems. We propose to build asystem which combines an inertial orientationtracker with a hybrid acoustic-inertial positiontracker because this would overcome most of theproblems of either technology used alone.

1.5.1 Orientation Tracker First Prototype

In the first year of the grant, an initial prototype ori-entation tracker was built. The tracker consisted ofthree Systron-Donner GyroChips mounted on analuminum bracket, together with a fluid inclinometerfor drift compensation. It was large and heavy, butprovided a convincing demonstration of the poten-tial usefulness of inertial orientation tracking.Methods were developed for bench-testing the per-formance of the prototype and a simple apparatusbuilt for this purpose. Detailed testing of drift, accu-racy, resolution, noise, and dynamic responsebegan late in the first year.

Testing of the initial prototype inertial head-orientation tracker was completed early in thesecond grant year. Three new projects werestarted to further develop the inertial head-trackingconcept: (1) development of a second prototype ori-entation tracker, (2) construction of a generaltracker testing facility, and (3) six-degrees-of-freedom (6 DOF) inertial tracker concept develop-ment and research on its underlying devicetechnologies.

1.5.2 Orientation Tracker Second Prototype

The second prototype effort is now nearing com-pletion. Its primary purpose is to take the inertialtracker from the proof-of-concept phase through tothe demonstration of a practical tracking device thatcan be used in real applications including auditoryexperiments. Several improvements were needed toaccomplish this goal. The sensor assembly, entirelyredesigned to incorporate a two-axis fluxgatecompass for yaw drift compensation, weighs justfour ounces. The electronics were improved andtransferred to a circuit board which installs in anexpansion slot of a PC. Finally, a variety ofimprovements to the software were made includingthe addition of rs-232 host interface capability, moresystematic sensor error compensation, and a newdemonstration and testing program that runs on aseparate PC connected to the tracker through thers-232 interface.



1.5.3 Calibration and Test Facility

A second new project, which is now well underway,is the design and construction of a facility for cali-brating our inertial trackers and evaluating all typesof head-trackers. We would like to be able to eval-uate the performance of our tracker as it wouldbehave in a real VE system, including the effect ofthe serial interface. In the testing procedure used toevaluate our first prototype, the testing was per-formed by the same program running on the samePC that integrates the sensor signals. Therefore,any delays that would result from the interface werenot included in our measurements. It was alsoimpossible to measure the performance of othercommon trackers such as those manufactured byPolhemus, Ascension Technologies, and ShootingStar Technologies, and thereby perform a fair com-parative evaluation of our inertial tracker versusother tracking systems.

We set out to design an autonomous system whichcould meet several needs: (1) testing any type ofhead tracker by stimulating it with a very well-known mechanical input motion and comparing thetracker output with the known mechanical input; (2)obtaining calibration data for inertial trackers byapplying well-known mechanical input motions andrecording all the sensor outputs simultaneously; and(3) testing the dynamic performance of a completevirtual environment system by applying mechanicalmotions to the head-tracker and measuringresulting changes directly from the surface of thevisual display using photodiodes. For the first task,it is desirable to to provide arbitrary 6 DOFmechanical input motions to identify any nonlinearinteractions between the input axes. However, a full6 DOF rate table was too expensive, so we devisedother simpler means for determining the degree ofnonlinearity. We designed a mechanical motionstimulus generator that is capable of applying con-trolled motion to any one of the 6 degrees offreedom at a time. The 6 DOFs comprise threelinear translation axes for position and three angularrotation axes for orientation; therefore, one linearmotor and one rotary motor, of which one may beoperated at any given time, and means formounting any head tracker in any of threeorthogonal orientations on either motor, weredeemed sufficient. This equipment is also adequatefor the second and third tasks. The motionstimulator package needs to generate significantoutput up to 15 Hz for both linear and rotationalmotion, up to 1000 degrees rotation rate; and the

linear and rotary axes must be provided withencoders accurate to 0.01 mm and 0.005 degreesrespectively.

1.5.4 Six-Degrees-of-Freedom InertialTracker

Investigation has begun into the possibility of aug-menting the inertial orientation tracker with triaxialaccelerometers to track position as well as orien-tation. Background reading has been done in thefield of strapdown inertial navigation to identify anyexisting theory that can be applied to the head-tracking problem. A thorough review of the state-of-the-art in accelerometer technologies has also beenconducted, including both library research andextensive telephone inquiries. A renewed effort toidentify small, more accurate gyros has also beenmade, because the orientation subsystem in the 6DOF inertial tracker must be accurate to 1 mrad.Drawings and clay models of the five most prom-ising combinations of sensors have been preparedin order to balance size, cost and performance. Acombination has been selected to provide threeaxes of acceleration and three axes of angular ratewith navigation-grade performance in a package ofonly 1.6 cubic inches and three ounces weight. Ahybrid 6-DOF tracking system is currently beingdesigned which uses a Kalman filter to achieve effi-cient sensor data fusion of the signals from thethree rate gyroscopes, three linear accelerometers,and an ultrasonic time-of-flight position triangulationsystem. When completed, this system will offer thehighest resolution-range product ever achieved fora human motion-tracking system.

1.5.5 Publications

Foxlin, E. Inertial Orientation Tracker ApparatusHaving Automatic Drift Compensation forTracking Human Head and Other Similarly SizedBody. U.S. Patent Application No. U.S.S.N.261,364, 1994.

Foxlin, E., and N. Durlach. "An Inertial Head-Orientation Tracker with Automatic Drift Com-pensation for Use with HMDs." Virtual RealitySoftware and Technology Conference,Singapore, August 23-26, 1994. In ProceedingsVRST '94. Eds. G. Singh, S.K. Feiner, and D.Thalmann. Singapore: World Scientific, 1994.

379


Date post:	25-Nov-2021
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

77~72 5en*ry~

Documents