From Sound to Sense and back again: The integration of lexical and speech processes From Sound to...

transcript

From Sound to Sense and From Sound to Sense and back again:back again:

The integration of lexical and The integration of lexical and speech processesspeech processes

The integration of lexical and The integration of lexical and speech processesspeech processesDavid Gow

Massachusetts General HospitalDavid Gow

Massachusetts General HospitalBob McMurray

Dept. of Brain and Cognitive SciencesUniversity of Rochester

Bob McMurrayDept. of Brain and Cognitive Sciences

University of Rochester

Complex computations from sound to sense must be broken up for study.

The Speech Chain

Assume intermediate representations:

Phonemes…Words…Syntactic

Phrases…

The Standard Paradigm

Phonemes

Delimited fields of study.

•Speech Perception

•Spoken Word Recognition

•Phonology

Phonemes* essential

* or other sublexical category

Why? Categorical Perception (CP)

•Sharp identification of tokens on a continuum.

ID (%/pa/)0

100Discrim

inatio

Discrimination

•Discrimination poor within a phonetic category.

Continuous Acoustic Detail => Discrete Categories

Does CAD affect speech categorization?

Categorical Perception (CP)

Defined fundamental computational problems.

CP is output of •Speech perception

Input to •Phonology•Word recognition.

Phonemes

But… • Not all speech contrasts are categorical.

• Lots of tasks show non-categorical perception.

Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)…

Categorical Perception is about phonetic classification.

Why has the Standard Paradigm persisted?

SenseThe minimal computational problem: compute meaning from sound.

CP tasks don’t necessarily tap a stage of this problem.

?CPWords

Lexical activation… seems a good bet.

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.

Example: Word Segmentation

• Vowel Length• Stress/Meter• Coarticulation

Phonemes

Cue extra-segmental process.W

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.

No. Standard Paradigm is fine…

Yes. Hmm…

Does continuous acoustic detail affect interpretation via core word-recognition processes?

Need to use stimuli with:•Precise control over CAD

Need to use tasks that:•reflect only minimal computational problem:

meaning.•are sensitive to acoustic detail.

Sublexical Filter(phonemes)

Visual World Paradigm

•Subjects hear spoken language and manipulate objects in a visual world.

•Visual world includes set of objects with interesting linguistic properties (names)

•Eye-movements to each object are monitored throughout the task.

Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995)Allopenna, Magnuson & Tanenhaus (1998)

•Meaning based, natural task: Subjects must interpret speech to perform task.

•Eye-movements fast and time-locked to speech.

•Fixation probability maps onto dynamics of lexical activation.

•Context is controlled: meaning lexical

activation.

?Does continuous

acoustic detail affect interpretation?

Is lexical activation sensitive to continuous

acoustic detail?

Combine tools of

• speech perception:

9-step VOT continuum.

• spoken word recognition:

visual world paradigm

McMurray, Tanenhaus & Aslin (2003)

A moment to view the items

Methods

500 ms later

Repeat 1080 times…

Target = Bear

Competitor = Pear

Unrelated = Lamp, Ship

200 ms

Trials

Time (ms)

VOT=0 Response=

0 400 800 1200 1600

Systematic effect on competitor dynamics.Fixations to the competitor.

Predictions

Categorical Results Gradient Effect

target

competitor

target

competitor competitorcompetitor

target

What would lexical sensitivity to CAD look like?

Results

0 400 800 1200 16000

0 ms5 ms10 ms15 ms

0 400 800 1200 1600 2000

20 ms25 ms30 ms35 ms40 ms

Time since word onset (ms)

Response= Response=

LPhoneme ID

Not part of minimal computational

problem.

Same stimuli in metalinguistic task…

…more categorical pattern of fixations

Continuous acoustic detail is not helpful in metalinguistic tasks…

Summary

Word recognition shows gradient sensitivity to continuous acoustic detail.

Not extra-segmental: VOT

CAD affects higher-level processes.

Consistent with other studies:

Andruski, Blumstein & Burton (1994)Marslen-Wilson & Warren (1994)Utman, Blumstein & Burton (2000)Dahan, Magnuson, Tanenhaus & Hogan (2001)McMurray, Clayards, Aslin & Tanenhaus (2004)McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)

The Standard Paradigm?

Phonemes

Continuous Acoustic Detail

From other work:

Lexical activation influences sublexical representations.

Samuel & Pitt (2003)Magnuson, McMurray, Tanehaus & Aslin (2003)Samuel (1997)Elman & McClelland (1988)

The Standard Paradigm?

Phonemes

From other work:

Lexical activation influences sublexical representations.

Phonological regularity affectssignal interpretation.

Massaro & Cohen (1983)Halle, Segui, Frauenfelder & Meunier (1998)Pitt (1998)Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)

?Sense

Phonemes

Perhaps interaction and integration make sense.

Do they help solve sticky problems?

The Emerging Paradigm

Integration of work in:• spoken word recognition • speech perception• phonology

New computations simplify old problems and solve new ones.

•Cognitive processes: Lexical activation & competition.

•Perceptual processes: sensitivity to CAD & perceptual grouping.

CAD is helpful in language comprehension.

• Word segmentation

• Coping with lawful variability due to assimilation

Combination of approaches helps solve both problems.

Some lexical processes can’t work

in the Standard Paradigm

Lexical Segmentation

The SWR Solution

active

active department

active departmentact of dip art mint

a partdepart in

arepar

Standard Paradigm: Template matching overgenerates

•Overgeneration resolved through competition in TRACE (McClelland & Elman 1986)

Problem: What if the speaker is trying to say “suck seeds”?

‘ k s I d -

succeed

Frauenfelder & Peeters (1990)

Cues shown to affect segmentation:

•Initial strong syllable•Initial lengthening•Increased aspiration•Increased glottalization

Lehiste, 1960; Garding,1967; Lehiste, 1972; Umeda, 1975; Nakatani & Dukes, 1977; Nakatani & Schaffer,1978; Cutler & Norris, 1988…..

Implied processing model requires separate segmentation process

Phonemes

The Speech Solution

Problem: cues are subtle and varied, extra-segmental processes are inelegant

?Is there a better mechanism?

Phonemes

The proposal had a strange syntax that nobody liked. ^

The proposal had a strange sin tax that nobody liked. ^

• CAD affects interpretation.• does not trigger segmentation.

Gow & Gordon (1995)

GRAMMAR primedSyntax

Tax INCOME inhibited

GRAMMAR primedSyntax

Tax INCOME primed

•Observation: All segmentation cues happen to enhance word-initial features

• Strengthened cues facilitate activation, making intended words stronger competitors

Incorporating CAD:

• Solves overgeneration problem.

•No extra-segmental segmentation process.

Good Start Model

Gow & Gordon (1995)

When continuous acoustic detail affects lexical

activation, speech and SWR models can

be integrated and simplified

Summary

The emerging paradigm reframes

computational problems

Assimilation

English coronal place assimilation

/coronal # labial/ [labial # labial]

/coronal #velar/ [velar # velar]Standard Paradigm: Change is • discrete• phonemically neutralizing

Redefining Computational Problems

[ ]# berries nonword?

right berries?

ripe berries?

[ ]# berries

Standard Paradigm solution: Phonological inference (Gaskell & Marslen-Wilson, 1996; 1998; 2001)

Knowledge driven inference:

If [labial # labial] infer /coronal # labial/

greem beans green (Gaskell & Marslen-Wilson, 1996; Gow, 2001)

ripe berries right (Gaskell & Marslen-Wilson, 2001; Gow, 2002)

Moreover: Assimilation effects dissociated from linguistic knowledge (Gow & Im, in press)

Assimilatory modification is acoustically continuous

This is not discrete feature change!

Assimilation Produces CAD

F2 Transitions in /æC/ Contexts

Pitch Period

coronalassimilatedlabial

F3 Transitions in /æC/ Contexts

Pitch Period

coronalassimilated

labial

SmaSelect thecat

Regressive Context Effects

Subject Hears: Assim_Non-Coronal (cat/p box)

0 400 800 1200 1600Time (ms)

Coronal (cat)

Non-Coronal (cap)

Subject Hears: Assim Non-Coronal (cat/p drawing)

0 400 800 1200 1600Time (ms)

Coronal (cat)Non-Coronal (cap)

Looks to Final Non-coronal (box)

0 400 800 1200 1600

Time (ms)

Assim Non-Coronal

Coronal Non-Coronal

Progressive Context Effects

Progressive effect in the same experiment

Assimilation is resolved through phonological context.

Fully assimilated items show neither* (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003)

Assimilation: Use of CAD

Partially-assimilated items show

regressive context effects (Gow, 2002; 2003)

progressive context effects (Gow, 2001; 2003)

assimilation # context

Infinite regress (eternal ambiguity)…. or something more interesting?

Continuous acoustic detail is subject to basic perceptual

processes

Feature cue parsing (Gow, 2003)

Time (s)0 0.760454

A Perceptual Account

Time (s)0 0.760454

Features encoded by multiple cues that are integrated

Time (s)0 0.760454

Assimilation creates cues consistent with multiple places

Extract feature cues

Group feature cues by similarity and resolve ambiguity

example: eight….

catp # box cat

p # drawing catp

# | | | |

[cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab]

example: eight….

catp # Box cat

p # Drawing catp

# | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab]

Progressive and regressive effects fall out of grouping

SWR problem (eternal ambiguity) replaced by simpler perceptual problem

CAD important in solution: processing obstacle facilitates perception.

Integration of continuous perceptual features facilitates higher-level processes.

Facilitation via core-word recognition mechanisms—no extra-segmental routines required.

Summary

Standard paradigm

•Created artificial boundaries that misframed issues.

•Continous acoustic detail is variability to be conquered..

The basis of the standard paradigm is undercut.

•Meaning-based processes are affected by CAD.

•CAD is an essential component of word recognition.

The emerging paradigm

•Emphasis on methodologies that tap the minimal computational problem: meaning.

•Stresses integration of speech and spoken word recognition, questions methods and theory.

•Continuous acoustic detail is useful signal, not noise.

The Emerging Paradigm

The integration of lexical and The integration of lexical and speech processesspeech processes

The integration of lexical and The integration of lexical and speech processesspeech processesDavid Gow

Massachusetts General HospitalDavid Gow

Massachusetts General HospitalBob McMurray

Dept. of Brain and Cognitive SciencesUniversity of Rochester

Bob McMurrayDept. of Brain and Cognitive Sciences

University of Rochester

From Sound to Sense and back again: The integration of lexical and speech processes From Sound to...

Documents