Louis Goldstein Yale University and Haskins Laboratories PaPI June 21, 2005 1 Syllable structure and modes of coupled dynamical systems Louis Goldstein Department of Linguistics Yale University and Haskins Laboratories Acknowledgments Cathe Browman Dani Byrd Hosung Nam Marianne Pouplier Michael Studdert-Kennedy Elliot Saltzman NIH Phonological and physical descriptions of speech Phonological sequence of discrete, context-independent symbols from a small inventory that can recombine to create the word-forms of language. Physical continuous, context-dependent variation in many articulatory, aerodynamic, acoustic, auditory parameters. Is it possible to model speech with a common structure? To what extent can phonological and physical properties emerge lawfully as macroscopic and microscopic consequences of a common description or representation? Syllable Structure: Example properties Phonological (macroscopic) Onsets and rimes exhibit relatively free combination in most languages. Other combinatorial possibilities are typically more limited: Nuclei and codas Cs wihin onsets and and within codas CV syllables are unmarked. Onset consonants are usually weightless. Physical (microscopic) Relative timing of consonants in an onset cluster is more stable (less variable) than in a coda cluster. Timing of consonants to the vowel varies as additional consonants are added to an onset, but not to a coda (in English). Central Question for talk Is there a formal characterization of syllable structure from which both kinds of properties emerge? Outline of Talk I. Gestures as combinatorial units II. A coupled oscillator model of intergestural timing and cohesion III. Coupled oscillator model and syllable structure a) CV and VC as distinct coupling modes b) Multiple C gestures in onset and coda • clusters • segments c) Re-syllabification and related phenomena d) Language-specific coupling grammars IV. Summary and Prospects
Transcript
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
1
Syllable structure and modes ofcoupled dynamical systems
Louis Goldstein
Department of Linguistics
Yale University
and
Haskins Laboratories
Acknowledgments
Cathe Browman Dani Byrd Hosung Nam Marianne Pouplier Michael Studdert-Kennedy Elliot Saltzman NIH
Phonological and physicaldescriptions of speech
Phonological sequence of discrete, context-independent symbols from a
small inventory that can recombine to create the word-formsof language.
Physical continuous, context-dependent variation in many articulatory,
aerodynamic, acoustic, auditory parameters. Is it possible to model speech with a common structure?
To what extent can phonological and physical propertiesemerge lawfully as macroscopic and microscopicconsequences of a common description or representation?
Syllable Structure:Example properties
Phonological (macroscopic) Onsets and rimes exhibit relatively free combination in most
languages. Other combinatorial possibilities are typically more limited:
Nuclei and codas Cs wihin onsets and and within codas
CV syllables are unmarked. Onset consonants are usually weightless.
Physical (microscopic) Relative timing of consonants in an onset cluster is more stable
(less variable) than in a coda cluster. Timing of consonants to the vowel varies as additional
consonants are added to an onset, but not to a coda (in English).
Central Question for talk
Is there a formal characterization of syllablestructure from which both kinds of propertiesemerge?
Outline of Talk
I. Gestures as combinatorial unitsII. A coupled oscillator model of intergestural timing
and cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Louis GoldsteinYale University and Haskins Laboratories
Act of speaking can be decomposed intoatomic units of vocal tract constriction action,or gestures.
Properties Macroscopic. Gestures are discrete and can
function as units of information (contrast andcombination).
Microscopic. Continuous, context-dependentmotion of articulators and sound unfolds lawfullyfrom gestures.
What makes gestures discrete?
distinct organs within-organ differentiation into discrete modes of
activity abstract (task) dynamical systems
Organ independence
Gestures control independentconstricting devices, or organs.Organs = Articulators of phonological theory(Halle, 1983)
TongueTip (TT)
TongueDorsum (TD)
LIPSGlottis
Velum
TongueRoot (TR)
Gestures of distinct organs count asdiscrete differences.
Even neonates show sensitivity to thepartitioning of the oro-facial systeminto distinct organs (Meltzoff & Moore,1977).
Examples of organ-specificimitation in infants
Virtual neonates can imitateoro-facial gestures.
When initiating an imitation,all organs cease moving,except the one the infantobserved.
Imitation of the organ’s actionis not always accurate (infantwill show improvement acrossattempts), but organ chosen iscorrect.
TongueProtrusion
LipProtrusion
MouthOpening
Meltzoff and Moore, 1977
Discrete differentiation ofwithin-organ action Gestures of a given organ can be differentiated by the degree and
location of the constriction goal.
ticksickthick TTCL
TTCDtongue tip constrict locationtongue tip constrict degree
LPLA
TBCLTBCD
VEL
GLO
lip protrusionlip aperture
tongue body constrict locationtongue body constrict degree
velic aperture
glottal aperture
Constriction Goal Parameters
Possible to model the emergence of discrete regions ofcontinua through self-organization in systems of agents thatattune to one another (e.g. de Boer, 2000; Goldstein, 2003;Oudeyer, 2003) .
Between- vs. within-organdifferentiation
View predicts that systematic differentiation of an organ’sconstriction goals are acquired later than systematic use ofdistinct organs themselves. (Studdert-Kennedy, 2002; Goldstein, 2003).
Infant must attune to the environment. Preliminary confirmation using perception of infant
productions Goldstein (2003), Son (in prep)
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
3
Discreteness in time:Dynamical systems
Articulators move continuously in time during aconstriction action.
Where is the discrete unit in this continuous change?
Give rise tocontinuousmotion overtime Activation intervals
Task Dynamics (Saltzman, 1985; 1995)
Constriction formation can be modeled as a (time)-invariant dynamical system that achieves a goal (task): e.g., LA (distance between the lips) is task goal variable.
Form of continuous motion over time emerges fromthe system definition.
Context-dependence emerges from temporal overlap ofinvariant dynamical units Invariant dynamics at the task level shapes the time-varying,
context-dependent dynamics at lower levels of the system(articulators and muscles).
I. Gestures as combinatorial unitsII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
cloalv
clolab
widephar
From gestures to words: glue
“ban”
VEL
TT
TB
LIPS
GLO
“mad”cloalv
clolab
widephar
wide
Word forms are molecules composed of multiplegestures.
Relative timing of gesture activation is significantinformation and can be displayed in a gestural score.
What is the glue that coordinates them appropriately?
widewide
Hierarchical syllable structureas glue?
p nae
Ons Rime
σ
They encode the macroscopicproperties of syllable structure(e.g. greater dependency betweenV-C than between C-V).
But they do not explain them.
And they cannot account formicroscopic properties in a generaland principled way (timing andstability of timing).
Gestures can be organized intohierarchical segment andsyllable structures.
Planning can be modeled as kind of internal repetition. Each gesture corresponds to an oscillator. Oscillators are coupled pair-wise to one another (according
to a coupling graph) so as to achieve a target relative phase. During (internal) repetition, coupling causes oscillators to
settle at stable relative phases (Saltzman & Byrd, 2000). Final relative phases can be used to trigger gestural
activation (as shown in the gestural score). Coupling graph for an utterance
specifies how pairs of gestures are coupled to one another(target relative phases).
Properties of syllable structure emerge as consequences ofthis graph.
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
5
Rate
Intergestural
INTER-ARTICULATORCOORDINATION
Lexicon
Coupling Graph
Activationvariables(GestualScore)
outputspeech
) ) ))ProsodyGesturalplanningoscillatorvariables
Modelarticulatorvariables
Tract/Constrictionvariables
INTER-GESTURALCOORDINATION
LIP (lab clo) TT (alv clo)
TB (phar wide)
0o 180o
“bad”
Browman & Goldstein (1990)
Saltzman & Munhall (1989)
Saltzman & Byrd (2000)
Nam & Saltzman (2003)
Settling of Coupled Oscillators
Random Initialrelative phase
= -90°
Final relativephase = 0°
LIP (lab clo)
TB (phar wide)
0o
Time
Phase 0o of each oscillator triggers gesture onsetsso relative phase 0o means onsets synchrony
I. GesturesII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Coupling C and V gestures
If a consonant (C) gesture and a vowel (V) gestureare to be coordinated in an intrinsically stable mode,there are just two possibilities: in-phase
hypothesized for C-V (onset relation) most stable anti-phase
hypothesized for V-C (coda relation)
Distinct C-V and V-C modes have been hypothesizedhas far back as Stetson (1951) [more recently, Tuller & Kelso, 1991; DeJong (2001)]
Evidence for C-V and V-Cmodes
C and V gestures (of CV) are in phase
V and C gestures (of VC) are anti-phase
p i p a p
LIPS
TONGUEROOT
Explaining combinatorialproperties of syllables
Hypothesis: Combinatorial freedom ofgestures is possible just where intergesturalcoordination exploits the most stable mode ofcoupling. As long as gestures are coupled in the most stable
mode, any gesture can be combined with anyother.
With less stable (or non-intrinsically stablemodes), specific couplings may have to belearned, so free combination is less likely.
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
6
Predictions
Onset C gestures should combine freely with V gestures,(which can explain free combinatoriality of onsets and rimes).
Coda C gestures are in a less stable mode with Vs, andtherefore there should increased dependency between V andfinal C.
Within-onset and within-coda consonant coordination mayemploy non-intrinisically stable modes. specific couplings must be learned acquired late typically small numbers of combinations
C and V gesture valences
C and V gestures are differentiated by degree of constriction (V is wider) dynamic stiffness (V takes longer to get to target ) activation interval (V still active after C released)
Nature of these differences is such that C and Vgestures can be in phase (at onset) and still be both berecoverable by listeners (Mattingly, 1981).
These gestural properties, together with the stabilityof in-phase coupling gives rise to valence of C and Vgestures -- they combine freely with each other in C-V structures.
Biases in CV combinations
Grammatically, onset C and V combine freely inmany languages (e.g., English).
However, MacNeilage and Davis (2000) have foundthere are statistical biases in C-V combinations in thelexicons in a sample of 10 languages Combinations occurring with greater than chance
frequency: Coronals with front Vs Labials with central Vs Dorsals with back Vs
McNeilage and Davis also find these patterns in the earliest“syllables” produced by infants. They hypothesize that infants are only oscillating their jaws.
McNeilage & DavisJaw Oscillation model
Oscillate Jaw
RetractedTongue
AdvancedTongue
Alternative: gestural synchonyand articultory constraint Some problems with jaw oscillation only theory for
infants: Preferred patterns occur more frequently than expected by chance, but
many other combinations also are produced. Adult languages show similar trends, but we know adults do more than
oscillate the jaw -- C and V can be independent.
Alternative Hypothesis: While gestures in CV are hypotheiszed to be triggered synchronously,
some CV combinations do not afford articulatory synchrony between Cand V gestures, due to intrinsic constraints of the gestures themselves(e.g., Recasens, Solé).
The most frequent combinations are those in which the articulatorysynchrony matches synchrony in gestual triggering.
p a p i p
LA
TBcd
TRx
p a t i p
TT
TBcd
TRx
p i t a p
TT
TBcd
TRx
p i p a p
LA
TBcd
TRx
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
7
Specific model of modes andadditional predictions
A simple potential function has been found tocharacterize qualitative features of coupled oscillatorysystems (Haken, Kelso & Bunz, 1985).
VC
CV
PredictionsShorter planning time for CV than VC syllablesEarlier acquisition of CV than VC syllables
two local minima (0˚, 180˚)V(Φ) = -a cos(Φ) - b cos (2 Φ)modeled results of many experiments on interlimb coordinationin-phase attractor is wider and deeper
Planning Time: Simulations
Because the in-phase attractor has a steeper well,relative phase should settle at its target more quickly.
Test: Initial Φο = 5, 10, 15, 20, 25 degrees from target (0o or 180o) 100 repetitions for each Φο with random choice of
individual oscillator phases (φ1, φ2)
5 10 15 20 25Φο
200
100
cycles
anti-phase
in-phase
Planning time: Pilot BehavioralExperiment (Nam, 2004)
Task: Produce syllable(s) as fast as possible segments of a syllable were vertically
aligned(not a reading task)
Syllable types: CV, VC, CVCV, VCVC C and V randomly chosen
among /p, t, k/ and /a, e, i, o, u/ Measure:
lag from stimulusto acoustic onset of response
Subjects: 2 English speakers 2 Korean speakers
0.3
0.4
0.5
0.6
0.7
CV VC CVCV VCVC
syllable type
RT
(se
c.)
CV VC CVCV
VCVC
.7
.5
.3
RT (secs)
Acquisition of CV vs. VC
Infants develop CV syllables before VC (in alllanguages).
Develop model of phase learning thatincorporates HKB coupling function (Nam).
Planning
phase
den
sity
phase
den
sity
•1̃° ≈ •2̃° ?
randomlychosen •̃1°
intrinsic potential
0° 180°
intended potential
•°̃
randomlychosen •2̃°
+
ADULT
Tuning
CHILDResults
Over time, the child learns distribution ofphases in adult model.
But regardless of proportion of CV vs. VC inthe adult model, the CV mode develops earlierthan the VC mode.
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
8
I. GesturesII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Onsets composed ofmultiple gestures
If onset is defined by an in-phase relation between Cgesture and V, then all onset C gestures should besynchronous with V (and therefore with each other).
Combinations of a clo or crit gesture (stop orfricative) with a wider gesture allow recoverability ofboth gestures even when synchronized. This result is a segment (e.g., nasal or aspirated stop).
Combinations of multiple clo or crit gestures presentrecoverability problems if synchronous. Gestures must be at least partially sequential (cluster). What makes them all part of the onset?
Coupling modes specified in the coupling are abstract andcan compete with one another
C-V coupling All C gestures in an onset are coupled in-phase with the V.
C-C coupling C gestures are also coupled sequentially (anti-phase or ?)
Observed coordination should reveal the presence of bothcouplings.
C CV
Onset Add an additional coordination(C-C phasing)?
But C-V phasing is preserved asglobal c-center
C-center
C-V and C-C phasingsin competition
C-C phasingseparates CC in timingC-V phasing
C
C
V
s
s
p
s
p
l
d
Honorof & Browman (1995) data
“sayed”
“spayed”
“splayed”
Hypothetical Model
d
d
C
C
C-center (Browman & Goldstein, 1988)
CV timing in onset cluster“pea spots”
Lip Aperture
Tongue TipConstriction
Tongue Bodydistancefrompalate
15 mm
Time100 ms
p ea s p o t s
s
a
p
Competitive coupling model
Generalization of phase planning model toaccomodate multiple, competing ψ (Nam &Saltzman, 2003).
Coupling graph hypothesized for onsets:
C C
V
Oscillators still settle into stable patterns (but ψ willnot be all be achieved if they are in competition).
Output phasing consistent with C-center is obtained
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
9
Example: /spœt/
CLO REL
PHAR WIDE
ALV CRIT ALV CLO
WIDE
Lips
TB
TT
GLO
50 ms Time
Clusters in onset vs. coda
Coda clusters do not regularly show C-center cup cuss cusp cups cusps (Honorof & Browman, 1995)
Variable across subjects (Byrd, 1995)
Consistent with coupling mode distinction of onsetand coda: Onset is in-phase relation between C gestures and V Coda is anti-phase relation:
between V and C among C gestures in coda Only weak attraction of multiple Cs to anti-phase relation to V.
Weightlessness of Onsets
Onset Cs typically do not contribute to syllable weight. Coda Cs may contribute to weight depending on the language If weight is related to duration, then proposed coupling
structures can account for the difference between onset andcoda consonants in weight.
With synchronous C-V coupling in onset, adding Cs to onsetdoes not increase syllable duration as much as when suchcoupling is lacking.
For languages in which coda consonants do not contribute toweight, additional competitive coupling is predicted
C CVOnset
C CV Coda
Timing stability in onsetsvs. codas
Onset Coda
Timing between C gestures is more stable inonset clusters than in coda clusters (Byrd, 1996).
/sk/
Stability and couplingstructure Nam and Saltzman (2003) add different amounts of noise to
onset and coda coupling models:
Onsets
Codas
.05 .65.25 .45 .85std. of noise
std. of C-C phase (radians)
1.0
C CV
Resulting variability in C-to-C phasing is greater for codasthan for onsets (because of multiple couplings).
C CV
I. GesturesII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
10
Segments in onset and coda Multiple C gesture segments are (typically)
combinations of a clo or crit gesture (stop or fricative) with a wider gesture (e.g. velum lowering for nasal (N) ) that allow recoverability of both gestures even when
synchronized. Adding a velum gesture to CV or VC:
In onset, velum, clo, V should all be synchronized In coda, clo, V are already anti-phase so velum cannot be
synchronized with all.
C V
OnsetN
N
Coda C
V
Intrasegmental coordinationin English
If velum synchronized with V in coda, then clo-velum coordination should differ in onset and coda.
English (Krakow, 1993):Coordination of velum and clo gestures differqualitatively as a function of syllable position(‘bistability’).
Syllable-initial: C gestures are in-phase. Syllable-final: C gestures are sequential, anti-phase
(consistent with synchronization of clo and V)
This pattern of gestural configuration will produce,as a consequence, more vowel nasalization beforesyllable-final than before syllable-initial nasalconsonants.
Velum-Lip Coordination inNasals (Krakow, 1993)
Ons
etCo
da
Alternative coda patterns
English pattern generalizes to other segments TT and TB gestures for /l/ (Sproat & Fujiumura, 1993)
Other languages exhibit synchronization ofvelum and clo in coda (as well as in onsets): French (Cohn, 1993) Australian languages (Butcher, 1999)
The model predicts that there should be thesetwo possibilities in coda, but not onset.
I. GesturesII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Re-syllabification
Coupling approach predicts a tendency to re-syllabifyintervocalic coda consonants as onsets (in-phase patternis the stronger attractor).
Rate: a shift (phase transition) from anti-phase to in-phase (most stable) pattern at fast rate is predicted (Tuller& Kelso, 1991) and seems to occur: Possibility first raised by Stetson (1951). Shift of laryngeal-oral coupling from coda to onset pattern
(Tuller & Kelso, 1991; deJong, 2001). At fast rate, English coda pattern of nasals begins to shift to
the onset pattern (Krakow, 1999).
Louis GoldsteinYale University and Haskins Laboratories
PaPIJune 21, 2005
11
HKB Phase transition model
f
VCCV
Phase transitions in many types of multiple limbmovements have been successfully modeled in thisway.
I. GesturesII. A coupled oscillator model of intergestural timing and
cohesionIII. Coupled oscillator model and syllable structure
a) CV and VC as distinct coupling modesb) Multiple C gestures in onset and coda
• clusters• segments
c) Re-syllabification and related phenomenad) Language-specific coupling grammars
IV. Summary and Prospects
Language-particularcoupling grammars
Differences in coupling graphs Modes provide preferences, but ultimately, coupling graphs
must be learned. Different V-C coupling in VC light vs. heavy Different coupling of oral constrictions and velum in coda.
Language differences in coupling graphs could bemodeled as resulting from different constraint rankings: Gafos (2002) Nam (2004)
max (zero-coordination) [in-phase] min (NON-zero-coordination) [other phase targets]
Other language diferences
In a competitive model, coupling strengths (potentialwell depth) can differ for different links.
Language differences in relative coupling strength: Georgian initial clusters (Chitoran, Goldstein & Byrd, 2002)
more separation in time than English clusters (C-C > C-V) more separation in back-to-front order than front-to-back.
May yield qualitative differences, depending on nature ofcompetitive model linear vs. non-linear (strict dominance)
Tashlhiyt Berber
Syllabification algorithm (Dell & Elmedlaoui) Makes most sonorous segments nuclei. This maximizes synchronous coordination, because they
afford synchronization and recoverability
Time50 ms
Lip Aperture
Tongue TipConstriction
t b d a
10 mm
Lip Aperture
Tongue TipConstriction
t u d a
10 mm
But what makes syllables like [tb] syllables?Any evidence for the synchronous mode in this case?
Summary and Prospects
A competitive, coupled oscillator model for planningintergestural timing may be able to account for severalmicroscopic and macroscropic properties related tosyllable structure.
Future Directions: Modelling of multisyllabic utterances Development of an explicit model that takes account of
intrisic articulatory constraints in modulating relative timingof gestures needs to be developed.
Extension of model to incorporate pitch gestures and thekinds of alignment facts Ladd and others have unconvered.
Louis GoldsteinYale University and Haskins Laboratories
