11

Speech, Reading and the Speech, Reading and the Linguistic Process:Linguistic Process:

A conference in honor ofA conference in honor ofIgnatius G. MattinglyIgnatius G. Mattingly

22

Prosody and ReadingProsody and Reading

Janet Dean FodorJanet Dean Fodor Graduate Center, CUNYGraduate Center, CUNY

33

Collaborative research with Dianne Bradley, Eva Fernández, Yuki Hirose, Yoshihisa Kitagawa, Nenad Lovrić,

Deirdre Quinn, Amit Shaked.

44

Ignatius’ synthesis by rule Ignatius’ synthesis by rule researchresearch

1964 Holmes, J. N., Mattingly I. G., and Shearme, J. N. 1964 Holmes, J. N., Mattingly I. G., and Shearme, J. N. Speech synthesis by rule. Speech synthesis by rule. Language and SpeechLanguage and Speech 7, 127.7, 127.

1966 Mattingly, I. G. Synthesis by rule of prosodic 1966 Mattingly, I. G. Synthesis by rule of prosodic features. features. Language and SpeechLanguage and Speech 9, 1-13. 9, 1-13.

1968 Mattingly, I. G. Synthesis by rule of General 1968 Mattingly, I. G. Synthesis by rule of General American English. American English. Haskins Laboratories Status Haskins Laboratories Status Report on Speech Research.Report on Speech Research. Supplement. Supplement.

55

Ignatius’ synthesis by rule Ignatius’ synthesis by rule researchresearch

1964 Holmes, J. N., Mattingly I. G., and Shearme, J. N. 1964 Holmes, J. N., Mattingly I. G., and Shearme, J. N. Speech synthesis by rule. Speech synthesis by rule. Language and SpeechLanguage and Speech 7, 7, 127.127.

1966 Mattingly, I. G. Synthesis by rule of prosodic 1966 Mattingly, I. G. Synthesis by rule of prosodic features. features. Language and SpeechLanguage and Speech 9, 1-13. 9, 1-13.

1968 Mattingly, I. G. Synthesis by rule of General 1968 Mattingly, I. G. Synthesis by rule of General American English. American English. Haskins Laboratories Status Haskins Laboratories Status Report on Speech Research.Report on Speech Research. Supplement. Supplement.

66

Some input strings for the JSRU Some input strings for the JSRU speech synthesis systemspeech synthesis system

A ‘BERD in DHA ‘HAAND IZ WERTH ‘TUU IN A ‘BERD in DHA ‘HAAND IZ WERTH ‘TUU IN DHA ‘BOOSH. DHA ‘BOOSH.

DID YUU ‘KUM BIE ‘MOATAKAR?DID YUU ‘KUM BIE ‘MOATAKAR?

Prosodic marks: Prosodic marks: .. falling tone, final pause falling tone, final pause ?? rising tone, final pause rising tone, final pause ++ fall-rise tone, non-final pause fall-rise tone, non-final pause

77

Realization of these tonesRealization of these tones

Adjust the F0 (pitch) of the syllable Adjust the F0 (pitch) of the syllable immediately following the stressed immediately following the stressed syllable of the last prominent word in the syllable of the last prominent word in the sentencesentence

8 steps down, for a falling tone8 steps down, for a falling tone 4 steps up, for a rising tone4 steps up, for a rising tone

and extend the fall / rise to end of and extend the fall / rise to end of sentence.sentence.

88

The motor carThe motor car

Did you come by motor car? Did you come by motor car?

(original, (original, synthesized)synthesized)

Thanks to Matthew Mattingly, Gary Chant Thanks to Matthew Mattingly, Gary Chant

99

The original motorcarThe original motorcar

Did you co me by mo tor car?Did you co me by mo tor car?

1010

The available technology – a The available technology – a computercomputer

An excerpt from an interview with Ignatius by An excerpt from an interview with Ignatius by Matthew Mattingly in 1999:Matthew Mattingly in 1999:

““The procedure was to make paper tapes at JSRU…you couldn’t see The procedure was to make paper tapes at JSRU…you couldn’t see what you were typing at the time...you could run the tape through a what you were typing at the time...you could run the tape through a printer and find your mistakes… When you’d made these tapes, that printer and find your mistakes… When you’d made these tapes, that consisted of your program and your data, you shipped the whole consisted of your program and your data, you shipped the whole thing to the installation several miles away, where the computer was. thing to the installation several miles away, where the computer was. They would run the program, and make more paper tapes as output, They would run the program, and make more paper tapes as output, and ship it all back to you. In the case of the speech synthesis, we’d and ship it all back to you. In the case of the speech synthesis, we’d send a paper tape with the synthesis by rule program, and more send a paper tape with the synthesis by rule program, and more paper tapes with the input sentences that we were synthesizing, and paper tapes with the input sentences that we were synthesizing, and they would send back paper tapes that contained groups of values they would send back paper tapes that contained groups of values for each parameter in the synthesizer. The value for the pitch, and for each parameter in the synthesizer. The value for the pitch, and each of the three formants, and the amplitudes and so on, one such each of the three formants, and the amplitudes and so on, one such set for every ten milliseconds. We would take this paper tape and set for every ten milliseconds. We would take this paper tape and mount it on a series of pulleys, and an electric eye read the tape and mount it on a series of pulleys, and an electric eye read the tape and sent the signals to the actual speech synthesizer. As the tape went sent the signals to the actual speech synthesizer. As the tape went round on the pulleys, you heard the output that you’d called for.”round on the pulleys, you heard the output that you’d called for.”

1111

The motor carThe motor car

Did you come by motor car? (original)Did you come by motor car? (original)

Did you come by motor car? (JDF high)Did you come by motor car? (JDF high)

Did you come by motor car? (JDF low)Did you come by motor car? (JDF low)

You came by motor car. (JDF)You came by motor car. (JDF)

????

1212

JDF Did you come by motorcar?JDF Did you come by motorcar?H*H* vs vs L*L* question question

Time (s)0 1.56717

0

500

Time (s)0 1.72222

0

500

1313

JDF Did you come by motorcar? JDF Did you come by motorcar?

QuestionQuestion vs vs statementstatement

Time (s)0 1.57333

0

500

Time (s)0 1.56717

0

500

1414

Group data by Tanya Viger Group data by Tanya Viger 15 Ss, 5 15 Ss, 5 itemsitems

e.g., Su sa nna has an en e my.e.g., Su sa nna has an en e my. (Does) Su sa nna (Does) Su sa nna havehave an en e my? an en e my?

American English: QUESTION vs DECLARATIVE 3-syllable stress-initial final word (e.g. enemy)

150

170

190

210

230

250

270

1 2 3 4 5 6 7 8

Mean DEC

Mean Q

1515

The motor carThe motor car

Did you come by motor car? (original)Did you come by motor car? (original)

Did you come by motor car? (JDF high)Did you come by motor car? (JDF high)

Did you come by motor car? (JDF low)Did you come by motor car? (JDF low)

You came by motor car. (JDF)You came by motor car. (JDF)

????

1616

Relevance for sentence Relevance for sentence processingprocessing

In speech, the prosodic contour can In speech, the prosodic contour can disambiguate some syntactic disambiguate some syntactic ambiguities.ambiguities.

Many studies since Lehiste 1973.Many studies since Lehiste 1973.

They fed her dog biscuits.They fed her dog biscuits.

They fed her dog-biscuits.They fed her dog-biscuits.

Flying planes can be dangerous.Flying planes can be dangerous.

1717

At CUNY we study At CUNY we study silentsilent prosodyprosody

For readers, few prosodic cues in the input.For readers, few prosodic cues in the input.

In reading aloud, the reader computes a In reading aloud, the reader computes a prosodic contour and imposes it on the prosodic contour and imposes it on the word string.word string.

We claim that this happens in silent reading We claim that this happens in silent reading too.too.

How can we know this? Why does it matter?How can we know this? Why does it matter?

1818

Syntactic parsing may be Syntactic parsing may be biased by silent prosodybiased by silent prosody

For an ambiguous sentence, a reader who For an ambiguous sentence, a reader who assigns a certain prosodic contour may treat assigns a certain prosodic contour may treat that prosody that prosody as if it had been part of the as if it had been part of the input stimulusinput stimulus, and use it to resolve the , and use it to resolve the ambiguity.ambiguity.

If so, ambiguity resolution preferences will If so, ambiguity resolution preferences will not be a reliable source of information about not be a reliable source of information about syntacticsyntactic parsing strategies – even in parsing strategies – even in experiments on silent reading.experiments on silent reading.

1919

In fact, a crisis in syntactic parsing In fact, a crisis in syntactic parsing theory: theory: universal parsing principles in universal parsing principles in

jeopardyjeopardy1978 Frazier & Fodor: The Sausage Machine.1978 Frazier & Fodor: The Sausage Machine. Hypothesis: The human sentence parsing Hypothesis: The human sentence parsing

mechanism is mechanism is innate (universal); just plug in a innate (universal); just plug in a grammar.grammar.

The parser’s task: Take in words; build a syntactic The parser’s task: Take in words; build a syntactic tree.tree.

Chop the word string into approx 6-word chunksChop the word string into approx 6-word chunks Minimal AttachmentMinimal Attachment Late Closure ( local / low attachment)Late Closure ( local / low attachment) Minimal Chain PrincipleMinimal Chain Principle

1988 Cuetos & Mitchell: 1988 Cuetos & Mitchell: Late Closure is Late Closure is notnot universal.universal.

Spanish doesn’t obey it.Spanish doesn’t obey it.

2020

The relative clause The relative clause attachment ambiguityattachment ambiguity

Someone shot the servant Someone shot the servant

of the actress of the actress

who who was…was…

WhoWho was on the balcony? was on the balcony?

The servant = HIGH ATTACHMT favored in Spanish The servant = HIGH ATTACHMT favored in Spanish * LC* LC

The actress = LOW ATTACHMT favored in English The actress = LOW ATTACHMT favored in English LC LC

2121

the servant of the actress the servant of the actress who…who…

NPNP NP NP 2 2 3 3 the N’the N’ the the N’ N’ 2 2 3 3

N1N1 PP PP N’ N’ RCRC servant servant 2 22 2 of of NP NP N1N1 PP PP

22 servantservant 22 the N’the N’ of of NP NP

2 2 2 2 N2N2 RCRC the the N2 N2

actressactress actressactress

2222

Proposed explanationProposed explanationProsodic breaks are optimally aligned with Prosodic breaks are optimally aligned with syntactic phrase edges syntactic phrase edges (Selkirk 2000)(Selkirk 2000)

For perceivers, a prosodic discontinuity For perceivers, a prosodic discontinuity favors a syntactic discontinuity. favors a syntactic discontinuity.

Prosody N1 of N2 Prosody N1 of N2 // RC RC HIGH attachment HIGH attachment SPANISHSPANISH

Prosody N1 Prosody N1 // of N2 RC of N2 RC LOW attachment LOW attachmentProsody N1 of N2 RC Prosody N1 of N2 RC LOW attachment LOW attachment ENGLISHENGLISH

2323

Prosody could explain the curious Prosody could explain the curious grouping of languagesgrouping of languages

Hypothesis: Weaker vs stronger tendency to Hypothesis: Weaker vs stronger tendency to break before RC. break before RC.

LOWLOW RC-ATTACHMENT RC-ATTACHMENT HIGHHIGH RC-ATTACHMENT RC-ATTACHMENT TENDENCY TENDENCY TENDENCY TENDENCY

American English American English Afrikaans Afrikaans British English British English Croatian Croatian Egyptian Arabic Egyptian Arabic Dutch Dutch Norwegian Norwegian French French RomanianRomanian German German SwedishSwedish Russian Russian SpanishSpanish

2424

Prosody can explain a Prosody can explain a universaluniversal effect of RC-lengtheffect of RC-length

Short RC Short RC Less likely break before RC Less likely break before RC More low attachment More low attachment

……the servant the servant of the actress of the actress

who smokes.who smokes. who smokes twenty cigarettes a day.who smokes twenty cigarettes a day.

This is reminiscent of the ‘packaging’ effects of the This is reminiscent of the ‘packaging’ effects of the Sausage Machine, which were also length- Sausage Machine, which were also length-dependent. dependent.

HIGH attachment is easier for phrases packaged as a HIGH attachment is easier for phrases packaged as a

separate package. separate package. Package = prosodic phrase?Package = prosodic phrase?

2525

Crisis over.Crisis over.

Why do readers of different languages sometimes Why do readers of different languages sometimes make different ambiguity resolution choices? make different ambiguity resolution choices?

Many interesting and plausible explanations, but Many interesting and plausible explanations, but the evidence now suggests:the evidence now suggests:

The parsing routines obey The parsing routines obey fully universalfully universal principles. principles. Any differences are due to differences Any differences are due to differences in the grammarin the grammar

that is applied to input strings of words.that is applied to input strings of words. The language-specific grammar The language-specific grammar includes prosodic includes prosodic

principlesprinciples – which are applied even in silent reading. – which are applied even in silent reading.

2626

Prosody in silent reading can Prosody in silent reading can explain a variety of other explain a variety of other

parsing factsparsing facts Effect of preposition in Croatian, Hebrew, Effect of preposition in Croatian, Hebrew,

German, Greek.German, Greek. Clause boundary placement in Japanese.Clause boundary placement in Japanese. Effect of focus particles in German.Effect of focus particles in German. Wh-scope interpretation in Japanese.Wh-scope interpretation in Japanese. Not-because scope preference in Not-because scope preference in

English.English. PP-attachment in English questions.PP-attachment in English questions.

2727

Prosody can explain the effect of Prosody can explain the effect of prepositionsprepositions

Croatian Croatian (Lovric 2003) (Lovric 2003) no prepositionno preposition between nouns between nouns prepositionpreposition between between the nounsthe nouns no prosodic break there no prosodic break there prosodic break prosodic break there there prosodic break before RC prosodic break before RC no prosodic break no prosodic break before RCbefore RC highhigh RC-attachment RC-attachment lowlow RC-attachment RC-attachment

Hebrew Hebrew (Shaked 2004)(Shaked 2004)

Similar results but shifted along the scale. Similar results but shifted along the scale. Hebrew “construct state” with no preposition is one Hebrew “construct state” with no preposition is one phonological phonological

word. word. Permits absolutely no prosodic break between the nouns. Permits absolutely no prosodic break between the nouns.

2828

Prosody can explain preferences in Prosody can explain preferences in garden-path reanalysisgarden-path reanalysis

Hirose 1998: Japanese clause Hirose 1998: Japanese clause boundaries are boundaries are highly ambiguous. Readers tends to highly ambiguous. Readers tends to locate locate them where there is a prosodic break for them where there is a prosodic break for reasons of phrase length. reasons of phrase length.

Bader 1998: German focus particlesBader 1998: German focus particles force a particular stress pattern. Garden- force a particular stress pattern. Garden-pathpath recovery is harder if the prosody needs recovery is harder if the prosody needs correcting as well as the syntax. correcting as well as the syntax.

2929

Effects of focus prosody as well Effects of focus prosody as well as prosodic phrasingas prosodic phrasing

Kitagawa & Fodor 2004: Japanese wh-questions Kitagawa & Fodor 2004: Japanese wh-questions are ambiguous if there are 2 clauses, each with a are ambiguous if there are 2 clauses, each with a potential scope-marker for the wh-phrase.potential scope-marker for the wh-phrase.

[[……wh-NP……… ka]………ka][[……wh-NP……… ka]………ka]

Scope can be disambiguated by post-focus de-Scope can be disambiguated by post-focus de-prosodification, which extends from the wh-prosodification, which extends from the wh-phrase (focus) to its (later) scope-marker. phrase (focus) to its (later) scope-marker.

Prosodic pressures (e.g., retain rhythmicity; align Prosodic pressures (e.g., retain rhythmicity; align with syntax) predict which scope interpretation with syntax) predict which scope interpretation readers prefer in different cases.readers prefer in different cases.

3030

Not-because scope ambiguityNot-because scope ambiguity(pilot data only, so far) (pilot data only, so far)

Koizumi 2004: Based on Frazier & Clifton 1996.Koizumi 2004: Based on Frazier & Clifton 1996.

Preferred Preferred BECAUSE > NOTBECAUSE > NOT:: Sue doesn’t cry because she realizes life Sue doesn’t cry because she realizes life is hard.is hard.

Inside an Inside an ifif-clause, preferred -clause, preferred NOT > BECAUSENOT > BECAUSE::

a. a. Sue didn’t cry because she was in public. Was she tearful later?Sue didn’t cry because she was in public. Was she tearful later? b. b. If Sue didn’t cry because she was in public, was she tearful later? If Sue didn’t cry because she was in public, was she tearful later? c. c. Sue didn’t cry because she felt lonely. What else was the matter? Sue didn’t cry because she felt lonely. What else was the matter? d. d. If Sue didn’t cry because she felt lonely, what else was the matter? If Sue didn’t cry because she felt lonely, what else was the matter? The if-clause contextThe if-clause context ▪▪ reduces the prosodic break before reduces the prosodic break before becausebecause ▪▪ induces F0 rise at end of the induces F0 rise at end of the becausebecause--clause clause

This prosody is typical of the This prosody is typical of the NOT>BECAUSENOT>BECAUSE reading ( reading (Hirschberg & Hirschberg & AvesaniAvesani 1997 1997 ))

(But also pragmatics of (But also pragmatics of ifif? How to dissociate prosody & pragmatics?)? How to dissociate prosody & pragmatics?)

3131

More on More on NOT-BECAUSENOT-BECAUSE

Troseth, Fodor, Koizumi & Fernandez 2004:Troseth, Fodor, Koizumi & Fernandez 2004:

Force the Force the NOT > BECAUSENOT > BECAUSE reading by using negative reading by using negative polarity item:polarity item:

John didn’t leave the meeting early because he John didn’t leave the meeting early because he was mad at anyone. was mad at anyone.

Grammaticality judgment task. Readers accepted Grammaticality judgment task. Readers accepted only 14%only 14%

Listeners accepted 49%Listeners accepted 49%- but they regard the sentence as incomplete.- but they regard the sentence as incomplete.

3232

Now – Now – the prosody of questionsthe prosody of questions A current investigation, far from A current investigation, far from

completecomplete PP-attachment in English questions, e.g.,PP-attachment in English questions, e.g.,

a. The nanny seated the cranky little child on the swing in a. The nanny seated the cranky little child on the swing in Oakwood Park. Oakwood Park.

b. The nanny seated the cranky little child on the swing in b. The nanny seated the cranky little child on the swing in his stroller. his stroller. DIFFICULT!DIFFICULT!

c. Did the nanny seat the cranky little child on the swing in c. Did the nanny seat the cranky little child on the swing in his stroller? his stroller? NOT SO DIFFICULT?NOT SO DIFFICULT?

Hypothesis: final rise in questions is a Hypothesis: final rise in questions is a discontinuitydiscontinuity favors a discontinuity in the syntactic tree favors a discontinuity in the syntactic tree structurestructure facilitates high attachment of the final PP. facilitates high attachment of the final PP.

Fodor, Bradley & Shaham Fodor, Bradley & Shaham 20042004

3333

PP1 high PP1 high reanalyze PP1 to reanalyze PP1 to lowlow

VPVP

99

VV NP NP PP1 PP1 PP2PP2

seated child on swingseated child on swing in stroller!in stroller!

VPVP

99

V NP PP2V NP PP2

seatedseated 1 1 in in strollerstroller

NN PP1PP1

child child on swingon swing

3434

Recall the sharp prosodic Recall the sharp prosodic discontinuity discontinuity in questions in questions

e.g., Su sa nna has an en e my.e.g., Su sa nna has an en e my. (Does) Su sa nna (Does) Su sa nna havehave an en e my? an en e my?

American English: QUESTION vs DECLARATIVE 3-syllable stress-initial final word (e.g. enemy)

150

170

190

210

230

250

270

1 2 3 4 5 6 7 8

Mean DEC

Mean Q

3535

In sumIn sum

Ignatius’ research encompassed both Ignatius’ research encompassed both prosody and readingprosody and reading

It seems now that these are even more It seems now that these are even more closely bound together than we knewclosely bound together than we knew

I am happy to be following (a little bit) in I am happy to be following (a little bit) in

his footsteps.his footsteps.