The Role of Learning in Typological...

The Role of Learning in Typological Modeling

Joe Pater Robert Staubs Coral Hughto

University of Massachusetts Amherst

Colloquium Nov. 17, 2014,Laboratoire Structures Formelles du Langage, Paris 8

Joe Pater, Robert Staubs, Coral Hughto UMass Amherst SFL Paris

The Role of Learning in Typological Modeling 1 / 47

The standard generative approach to typology, grammar,and learning

A theory of grammar is designed to generate all and onlypossible languages (typological modeling)

A theory of learning is designed to find a correct grammar forall languages in the space defined by the grammatical theory

The learning theory plays no role in the modeling of typology



The ‘Iterated Learning’ approach to typology, grammarand learning

Pioneered by Simon Kirby and colleagues (see e.g. Kirby andHurford 2002), developed in a Bayesian framework by Griffithsand Kalish (2007) and Kirby et al. (2007), see Dediu (2009)for useful overview and further extensions, and Wedel (2007,2011), Mackie and Mielke (2011) and Rafferty et al. (2013)for related work in phonology

Uses agent-based modeling (iterated learning) to study therole of learning and transmission in shaping typology

Uses very simple language models, and often highlightsemergence of putatively innate features of language



Grammatical Agent-Based Modeling of Typology: a ‘thirdway’?

What is the effect of learning on typology with a relativelyrichly specified theory of grammar?

We take no position on the innateness or even universality ofthe grammatical features, but we do take them as given in thework we discuss here



Grammatical Agent-Based Modeling of Typology: a ‘thirdway’?

Today’s talk will be an overview of some results from Pater(2012), Pater and Moreton (2012) and Staubs (2014ab), plusunpublished work by Carroll (2012) and Hughto (2014), andsome more recent joint work (see further Hughto et al. 2014:NECPhon)

See also Boersma and Hamann (2008), as well as Stanton(2014: AMP, NECPhon) for work in this general vein, andHeinz (2009) and colleagues for a quite different approach tolearning and typology (based on formal learning theory, i.e.FSMs) – other references? – and the larger related literatureon modeling linguistic change (overview in Zuraw 2003)



Overview

We’ll show that the incorporation of learning into typologicalexplanation allows standard grammar-based theories oflanguage to deal with phenomena that typically pose deepchallenges: systemic simplicity (Pater 2012), and probabilisticgeneralizations (i.e. typological ‘tendencies’ see esp. Staubs2014ab on stress typology)

We’ll also show that learning can affect predicted typologies inways that bear on current debates in linguistic theory – hereon the choice between ranked and weighted constraints (i.e.Optimality Theory vs. Harmonic Grammar)



The challenge of feature economy

A simple example of feature economy(J. Kingston p.c., based on Madiesson and Precoda 1992)

[b] no [b]

[g] 244 11

no [g] 43 153

χ2 = 260, d .f . = 1, p < 0.01

Languages tend to have either both [b] and [g] or neither: it isespecially unlikely for a language to have [g] without [b].

More generally, a segment is more likely if its feature valuesare shared by other segments.

In other words, languages tend toward feature economy(Martinet 1968; Clements 2003)




First difficulty - feature economy is a property of systems, notof individual representations or derivations

How do we express the dependency of [b] on [g] and viceversa?

Standard phonological theories, be they rule- orconstraint-based, do not provide a formal mechanism toexpress such systemic dependencies




Second difficulty - feature economy is a tendency

Languages with [p k g] or [p k b] are rare, not unattested

Standard phonological theories deal only with typologicalabsolutes, not probabilities

These problems for an analysis of feature economy seem tohave not been discussed prior to Pater (2012), but the issuesare the same as those discussed for the largely parallel case ofcross-linguistic tendencies toward uniform phrasal headednessacross syntactic categories (see e.g. Aristar 1991, Dryer 1992).



General constraints, tendencies to generality andcategoricity

Could we get tendencies toward systemic simplicity by havingconstraints or rules that state a general prohibition against, orpreference for, a structure across contexts?

For example, could we get [voice] economy by includingconstraints on voicing that generalize across place ofarticulation?

And could we get a tendency toward uniform syntacticheadedness with constraints on headedness that generalizeacross category?

Yes, if we include learning in our model of typology (Pater2012, see Pater and Staubs 2013: MFM for more on featureeconomy)




Whether we adopt a parametric framework, or ranked (OptimalityTheory) or weighted constraints (Harmonic Grammar), thegrammar model can represent arbitrarily non-uniform headedness




The interaction of the general Head-R/L and the category-specificconstraints is an example of a vacuous gang effect: the HGtypology is the same as OT, or a parametric theory. In all of theseframeworks, in terms of the set of systems that our theory canrepresent, the general constraint is doing no work for us.




One approach: predicted probability is proportional to thenumber of rankings generating it (Coetzee 2002, Bane andRiggle 2008)

In HG, randomly sampling constraint weights from a uniformdistribution from 0 to 1 gives 0.4 probability consistentheadedness with the general constraint, 0.125 without it

Random sampling is not a model of learning and transmission;adopting it may or may not be a reasonable idealization




Another approach: relate typological attestation to ease oflearning

This will be basically the approach we take, althoughagent-based modeling goes beyond the case of a single fixedteacher-learner relationship

See Pater (2012) for some discussion of issues that arise inattempting to equate simple ease of learning with typologicalattestation for the case of feature economy; see moregenerally Rafferty et al. (2013)




We assume a probabilistic version of HG in which theprobability of a candidate is proportional to the exponential ofits Harmony (MaxEnt grammar, Goldwater and Johnson 2003)

This grammar model can represent arbitrary probability on thechoices across phrase types







Learning: difference between datum produced by teacher andlearner’s expectation scaled by a learning rate (below always 0.1),added to current weights




Interactive learning: a simple way to generate typologies

Two ‘agents’ are started with zero weight on the constraints,and hence equal probability for the two candidates in eachtableau

In each trial, an agent is picked at random as the teacher, atableau is picked at random, and the grammar is used togenerate a learning datum for the learner

This is done 10,000 times to generate a language; I’ll reportresults over 50 languages




There are 4 phrase types each with a binary choice, so thereare 24 = 16 possible headedness combinations across them.Of these, only 2/16 have consistent headedness: 0.125 is abaseline probability of consistent headedness.

Taking the higher probability candidate from each tableau asthe choice of headedness, 40/50 = 0.8 have consistentheadedness




50 languages generated without the general constraints produced5/50 cases of consistent headedness (0.10, similar to 0.125baseline)




Probabilities in each tableau over first 12 runs, averaged overagents




Why are we getting categoricity?

1 Weight changes can push the agents towards either more orless categorical states.

2 As the agents drift into categorical grammars1 They change less and less.2 The effective change to probability from a change in weights

shrinks.

3 The system spends most of its time in categorical states.

(cf. Wedel 2007 on models where a positive feedback loop createssimilar pressures)




The network structure of this agent-based model is the polaropposite of those in Kirby and Hurford (2002) and Griffithsand Kalish (2007), besides the fact that there are only twoagents (at a time) in both

Their models are purely generational: a designated learnerlearns for some number of trials from a designated teacher,and then becomes the teacher for the next generation

This model is purely interactive: agents always have the sameprobability of being the teacher or learner

Reality lies between these two poles; in any case the purelygenerational model also produces the categoricity tendency(Dediu 2009: 555, Hughto et al. 2014)




Take-home message on categoricity: a probabilistic grammar,embedded in a model of learning and transmission, can predictthat languages should tend to be categorical

Aside: why do generative grammarians tend to shunprobabilistic models? Is it partially due to observed tendenciesto categoricity? If so, these results are relevant.

Take-home message on generality: a general constraint can yieldtendencies toward generality if we incorporate learning intotypological modeling



HG and typology: background

The predecessor to Optimality Theory (Prince and Smolensky1993) was a weighted constraint theory of grammar calledHarmonic Grammar (Legendre, Miyaya, and Smolensky 1990,Smolensky and Legendre 2006)

There has been a recent re-emergence of interest in HG, interms of learning (e.g. Wilson 2006, Jaeger 2007, Jesney andTessier 2011, Boersma and Pater to appear, though see Magri2013), and also in terms of typology (e.g. Pater 2009, Pottset al. 2010, Jesney 2012)

We’ll look at perhaps the best case of an attested HG-specificpattern, and two potential problems for HG in light ofGrammatical Agent-Based Modeling




Japanese loanword devoicing (Nishimura 2003, 2006), Kawahara(2006), Pater (2009)










This can be analyzed in OT with a larger constraint set, but is onlypredicted by HG



Stress windows

We can also get gang effects between just two constraints

Two constraint gang effect

3 2X Y H

→A -1 -2B -1 -3

→C -1 -3D -2 -4



Stress windows

Potential problem for HG: stress windows (Legendre, Sorace andSmolensky 2006, Pater 2009)



Stress windows



Stress windows



HG vs. OT

Only 3-syllable windows exist, not 4-syllable or larger, and3-syllable windows are much rarer than 2 (Staubs 2014b: 89)



Stress windows

Prince (1993/2007) points out the potential relevance oflearning to the unattestedness of larger windows, insofar as theweight ratios needed for larger windows are harder to acquire

Staubs (2014b): if we use a generational version of GABM tomodel the relative probabilities of 2- and 3-syllable windows,what do we predict for 4 syllable windows?

‘Although the fitted model predicts a value of one as most likely(32.69% probability), the observed zero count is by no meansunlikely (20.60% probability).’



Neutralization in the ‘elsewhere’ environment

Our second case of an apparent problem for HG relates to anobservation first made by Prince (1998) about OT

Allophony with a marked feature in a specific environment:e.g. ‘vowels are predictably [+nasal] before nasal consonants,[–nasal] elsewhere’

In some versions of OT (Pater 1999) faithfulness constraintscan specify the direction of change (i.e. from ‘+’ to ‘–’ andfrom ‘–’ to ‘+’)

Predicts neutralization in the elsewhere environment: e.g.‘vowels contrast for [+/–nasal] before nasal consonants, andare [–nasal] elsewhere’

Though Prince (1998) doesn’t talk about the attestedtypology, the standard OT prediction seems basically correct:that neutralization in the elsewhere environment does not exist




Carroll (2012) points out standard OT constraints used in HGalso predict ‘neutralization elsewhere’ (gang effect betweenFaith and specific Mark)

He looks at the typology of consonant palatalization, wheremarked palatals are often found before front vowels

There is in fact one language with neutralization elsewhere:Gujarati ‘[s] and [S] before front vowels, [s] elsewhere’

There are many more languages with the expected patterns ofallophony (23 languages, 11 families) and of neutralization topalatalized before front vowels (15, 10)




Carroll (p.c.) notes that with standard OT constraints,uniformly sampling the weights independently from a boundeduniform distribution yields neutralization elsewhere three timesmore often than allophony, or regular neutralization.

This case, in which a pattern predicted only by HG exists, butis rarer than predicted, may well fit with many phonologists’‘seat of the pants’ assessment of the general situation

Carroll (2012) also shows that usual adjustments to the priorof a MaxEnt model don’t change the probabilities in thedesired way (a high-variance log-normal prior could work, butthis essentially just builds in ranking-like behavior)




Hughto’s (2014) attack on this problem is based on Carroll’s(2012) observation that the space of HG-specific patternsoverlaps greatly with the space of patterns yielding(noticeable) variation in a MaxEnt model

Relatedly, Flemming (p.c.) reports that attested gang effectstend to be variable, like Japanese loanword devoicing

Given that the results of agent-based modeling tend towardscategoricity, perhaps they would also tend away fromcumulativity / gang effects?

They do – what follows is a demonstration with a simpletwo-constraint case



Cumulativity tableaux

3 2X Y H

→A -1 -2B -1 -3

→C -1 -3D -2 -4



Cyan and orange: no cumulativity effect. Black: cumulativity.Joe Pater, Robert Staubs, Coral Hughto UMass Amherst SFL Paris


Cumulativity

A simulation like our syntax one, except that the stopping criterionis 0.95 probability on some candidate in each tableau

X Y

A -1B -1

C -1D -2

Language Proportion

A, D 312B, C 688A, C 0



0 500 1000 1500

0.0

0.4

0.8

Avoidance of cumulative pattern, starting at 88% probability

Learning Step

Pro

babi

lity

of [a

]

0 500 1000 1500

0.0

0.4

0.8

Learning Step

Pro

babi

lity

of [c

]



0 1000 2000 3000 4000 5000

0.0

0.4

0.8

Maintaining cumulative pattern, starting at very high probabilities

Learning Step

Pro

babi

lity

of [a

]

0 1000 2000 3000 4000 5000

0.0

0.4

0.8

Learning Step

Pro

babi

lity

of [c

]



HG vs. OT

Take-home message on ranking vs. weighting: a weightedconstraint grammar, embedded in a model of learning andtransmission, predicts that languages should tend to avoid gangeffects

Much remains to be done in terms of studying the generalityof this result, but the potential consequences are striking,since the grammatical model itself might overgenerate, withthe full model remaining sufficiently restrictive



Conclusions

In generative linguistics, we typically make relatively directinferences about the structure of UG from typology

These results suggest that this idealization may not beentirely safe (categoricity, cumulativity) and may result inmissed opportunities (generality from general constraints,accounts of tendencies)



Thank you!

This material is based upon work supported by the NationalScience Foundation under Grant No. S121000000211 to thesecond author, Grants BCS-0813829 and BCS-424077 to theUniversity of Massachusetts Amherst, and by the city of Parisunder a Research in Paris fellowship to the first author.

We would also like to thank Lucien Carroll and Elliott Moreton fordiscussion, as well as John McCarthy and other members of thephonology grant group at UMass. .



Date post:	10-Mar-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

The Role of Learning in Typological...

Documents