Marie Klopfenstein, Ph.D. Elizabeth Paquin, B.S. Jennifer Davis Ariel Hagemann Noel Williams Southern Illinois University Edwardsville

INCREASING THE RELIABILITY OF SPEECH NATURALNESS DATA IN PERCEPTUAL RESEARCH

Disclosure Statement • The authors have no relevant financial or nonfinancial

relationship(s) within the products or services described, reviewed, evaluated or compared in this presentation.

Speech Naturalness •What makes speech natural?

•“Speech is natural if it conforms to the listener’s standard of rate, rhythm, intonation, and stress patterning, and if it conforms to the syntactic structure of the utterance being produced” (Yorkston, Beukelman, Strand, & Hakel, 2010).

Optimizing Naturalness

Background • Klopfenstein (2012) • Investigated acoustic and perceptual correlates of naturalness in

individuals with dysarthria • Naturalness was not defined for raters following Martin et al and

for method/design purposes • Problem of low reliability in naturalness ratings; data from some

raters had to be excluded

How do we measure naturalness? • 7 or 9 point equal-appearing interval (EAI) • Definition not defined for listeners

• Direct magnitude estimation (DME) • Listeners familiarized with the speech dimensions

• Inter and intra-rater reliability are a concern

Research Question Will providing listeners a definition of naturalness and examples of each end of the scale result in higher inter- and intra-rater agreement and reliability?

Method: Speech data to be rated • 4 individuals with hypokinetic dysarthria secondary to

Parkinson’s disease • 2 age and gender matched individuals with no

neurological impairment of history of speech and language disorder

• Age range of participants: 65-80 years old • Subjects read lists of short (5-7 word) sentences, a

short story, and spontaneous speech tasks • 5 sentences from each subject repeated for test-

retest reliability

Method: Naturalness rating • Raters were 19 undergraduate students enrolled in an Intro to

Communication Disorders course at Southern Illinois University Edwardsville

• Randomly assigned into Group A or B • Group A followed methods of EAI 1-9 scale

• 1 most natural • 9 least natural • No definition of speech naturalness given

• Group B also followed EAI scale but were given a modified version of Yorkston and colleagues’ definition of naturalness

• Examples of extreme ends of naturalness scale by Klopfenstein (2012)

Rating Procedure

Highly Unnatural Speech

Highly Natural Speech

Results: Intra-rater reliability • Test-retest using Cronbach’s alpha

Alpha Mean (Scale) SD (Scale)

Group A .896 8.08 5.201

Group B .931 7.60 5.239

Results: Inter-rater agreement & reliability • Looking at ratings across raters by subject

• Intra-class correlation

• Group A .419 • Group B .774

• Krippendorff’s alpha

• Group A .3797 • Group B .6965

Discussion • Preliminary results indicate greater reliability may be

achieved by providing listeners with a definition or naturalness and providing examples of extreme ends spectrum

• Increased reliability also increases the validity of EAI scales as used for naturalness rating, which has implications for future research involving speech naturalness

Thoughts on future variables to investigate • Naïve vs. experienced listeners • Number of samples to rate • Definition of naturalness • Effect of being monitored • Motivation: extra credit • Effect of rating normal and disordered speech

References • Armson, J., & Kiefte, M. (2008). The effect of SpeechEasy on stuttering frequency, speech

rate, and speech naturalness. Journal of Fluency Disorders, 33(2), 120-134. • Boersma, P., & Weenink, D. (2013). Praat: doing phonetics by computer. Retrieved from

http://www.praat.org/ • George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and

reference. 11.0 update (4th ed.). Boston, MA: Allyn & Bacon. • Ingham, R. J., & Onslow, M. (1985). Measurement and modification of speech naturalness

during stuttering therapy. Journal of Speech and Hearing Disorders, 50(3), 261-281. • Kalinowski, J., Noble, S., Armson, J., & Stuart, A. (1994). Pretreatment and posttreatment

speech naturalness ratings of adults with mild and severe stuttering. American Journal of Speech-Language Pathology, 3(2), 61-66.

• Klopfenstein, M. (2012). Prosodic Features and Speech Naturalness in Individuals with Dysarthria. (Unpublished doctoral dissertation). University of Louisiana at Lafayette, Lafayette, LA.

• Martin, R. R., Haroldson, S. K., & Triden, K. A. (1984). Stuttering and speech naturalness. Journal of Speech and Hearing Disorders, 49(1), 53-58.

• Metz, D. E., Schiavetti, N., & Sacco, P. R. (1990). Acoustic and psychophysical dimensions of the perceived speech naturalness of nonstutterers and posttreatment stutterers. Journal of Speech and Hearing Disorders, 55(3), 516-25.

• Onslow, M., & Ingham, R. J. (1987). Speech Quality Measurement and the Management of Stuttering. Journal of Speech and Hearing Disorders, 52(1), 2-17.

References • Onslow, M., Adams, R., & Ingham, R. J. (1992). Reliability of speech naturalness ratings

of stuttered speech during treatment. Journal of Speech and Hearing Research, 35(5), 994-1001.

• Runyan, C. M., Hames, P. E., & Prosek, R. A. (1982). A perceptual comparison between paired stimulus and single stimulus methods of presentation of the fluent utterances of stutterers. Journal of Fluency Disorders, 7(1), 71-77.

• Sacco, P. R., Metz, D. E., & Schiavetti, N. (1992). Speech naturalness of nonstutterers and treated stutterers: Acoustical correlates.

• Schiavetti, N., Whitehead, R., Whitehead, B., & Metz, D. (1998). Effect of fingerspelling task on temporal characteristics and perceived naturalness of speech in simultaneous communication. Journal of Speech, Language & Hearing Research, 41(1), 5-17.

• Southwood, M. H. (1996). Direct magnitude estimation and interval scaling of naturalness and bizarreness of the dysarthria associated with amyotrophic lateral sclerosis. Journal of Medical Speech-Language Pathology, 4, 13–25.

• Southwood, M. H., & Weismer, G. (1993). Listener judgments of the bizarreness, acceptability, naturalness, and normalcy of the dysarthria associated with amyotrophic lateral sclerosis. Journal of Medical Speech-Language Pathology, 1(3), 151-161.

• Yorkston, K. M., Beukelman, D. R., Strand, E. A., & Hakel, M. (2010). Management of Motor Speech Disorders in Children and Adults (3rd ed.). Austin, TX: Pro-Ed.