Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 206–217Florence, Italy. August 2, 2019 c©2019 Association for Computational Linguistics

Encoder-decoder models for latent phonological representations of words

Cassandra L. JacobsUniversity of California, Davisclxjacobs@ucdavis.edu

Frederic MailhotAutodesk Inc.

fred.mailhot@autodesk.com

AbstractWe use sequence-to-sequence networkstrained on sequential phonetic encoding tasksto construct compositional phonologicalrepresentations of words. We show that theoutput of an encoder network can predictthe phonetic durations of American Englishwords better than a number of alternativeforms. We also show that the model’s learnedrepresentations map onto existing measuresof words’ phonological structure (phonolog-ical neighborhood density and phonotacticprobability).

1 Introduction

The representation of linguistic categories is a fun-damental problem in (psycho)linguistics and nat-ural language processing. The formation of com-plex representations from more basic componentsis relevant at all levels of linguistic representa-tion, semantic, syntactic, and phonological. Find-ing good representations for words’ phonological1

structure is critical in psycholinguistics, where wewish to understand the phonological structure ofthe lexicon, which has been shown to be relevantfor language comprehension and production.

The distributional hypothesis defines a word bythe context in which it occurs (Harris, 1954; Firth,1957). This approach has been extended more re-cently to other types of compositional structures,for example in characterizing the meanings andforms of sentences (Cer et al., 2018; Joulin et al.,2017; Conneau et al., 2017; Devlin et al., 2018).In this paper we explore whether distributional ap-proaches can capture important phonological de-pendencies.

1There are disagreements in the literature about the lo-cation (Hale and Reiss, 2008) and even existence (Ohala,1990b) of the boundary/interface between phonetics andphonology, so we remain as theory-agnostic as pos-sible, freely using “phonological”/“phonetic” and “seg-ment”/“phone” interchangeably.

Specifically, we test the extent to which recur-rent encoder-decoder models (Cho et al., 2014;Sutskever et al., 2014) can learn representationsthat characterize the phonological structure of thelexicon while also having linguistic and psycho-logical validity (Sibley et al., 2008). We pro-pose that this approach can be used to learn viablelexical-level phonological representations. Theoutput of the encoder component of our modelyields promising results in the prediction of pho-netic duration, outperforming a number of alter-nate phonological representations of words.

2 Quantifying a word’s phonology

Given a set of discrete phonetic symbols i.e.graphemes with conventionalized pronunciationssuch as the International Phonetic Alphabet, it istrivial to represent any word’s pronunciation as asequence of such symbols. Conversely, relatingsequences of such symbols (viz. words) to eachother, as well as to the entire lexicon is less obvi-ous. This challenge has led to a proliferation ofmeasurements that characterize a word’s phoneticor phonological relationship with all other wordsin the lexicon. We summarize some salient exam-ples below, and briefly discuss some of their short-comings.

2.1 Metrics insensitive to serial orderPhonological neighborhood density (PND).This measure is defined as the number of wordshaving a Levenshtein edit distance of one froma given word (in terms of phonetic or phonologi-cal symbols) (Luce and Pisoni, 1998; Levenshtein,1966). Under this definition, a word like “cat”has many neighbors, while a word like “molt” hasfewer. This measure is simple to calculate and awide variety of resources exist for obtaining thesemeasures across many languages (Marian et al.,2012; Baayen et al., 1993; Luce and Pisoni, 1998).

206

While conceptually simple, PND is insensitiveto the position of a segment within a word (e.g.word-initial versus word-final substitutions), andso “sat” and “cab” are treated as equally similar to“cat”. Additionally, identifying a word’s phono-logical neighbors using the Levenshtein distancemetric requires specifying how many sounds canbe added, deleted, or substituted, and potentiallythe allowable edit distance2, increasing the num-ber of choice points in determining what a “neigh-borhood” is.

Frequency-weighted phonological neighbor-hood density. An augmented version of PND,which weights phonological neighbors in pro-portion to their lexical frequencies (standardlyestimated from large corpora; Marian et al.,2012). So, a more common word like “hat”would contribute more to the neighborhood den-sity of “cat” than a less common word like “cap”,even though they are at equal string edit dis-tance. Whether and to what extent density mea-sures should be frequency-weighted is an empiri-cal question, though these measures seem to betterreflect psycholinguistic processes than frequency-insensitive measures.

Feature-wise similarity. In the phonologi-cal literature it is standard to represent segmentsas collections of articulatory or acoustic features,e.g. [+voice], [-obstruent] (Chomsky (1968) is thecanonical reference). Some linguists (e.g. Frisch(1996), inter alia) have posited that words like“cat” and “cap”, which differ only in the place ofarticulation of their final segments (alveolar ver-sus labial), should be considered more similar thane.g. “cat” and “can”, which differ in both voicingand manner of articulation. This measure of simi-larity is potentially controversial, as there are the-oretical and empirical questions as to which fea-tures to include, or even whether phonetic fea-tures exist at all (Stevens and Blumstein, 1981;Marslen-Wilson and Warren, 1994).

2.2 Metrics incorporating serial orderAll of the previously described measures effec-tively characterize words as unordered collectionsof segments. These characterizations are incom-plete because they fail to capture the fact thatwords unfold over time in usage. Representing thepositions of phones within a word is critical for ex-

2See e.g. Suarez et al., 2011 who allow edit distancegreater than one, and track the mean distance to a fixed num-ber of neighbors

plaining a number of aspects of language process-ing. For example, the beginnings of words con-tribute more strongly than their ends to psycholin-guistic effects that are attributed to their phono-logical representations (Levelt et al., 1999; Sevaldand Dell, 1994, inter alia), and a word’s phono-logical similarity to the rest of the words in thelexicon has important consequences for speechcomprehension (Buz and Jaeger, 2016; Metsala,1997). Some computational models encode seg-ments as a function of their linear position withina syllable, e.g. in a onset-vowel-coda format (e.g.Dell, 1986; Sevald and Dell, 1994). Other ap-proaches include segment n-grams to encode localaspects of serial order (e.g. Seidenberg and Mc-Clelland, 1989; Davis, 2010) and the oft-lamentedWickelphone (Houghton and Hartley, 1996). Mostclosely related to the present approach, some workhas demonstrated the viability of sequence en-coder models for representing sequences of char-acters or phonetic segments (Sibley et al., 2008).

2.3 Incorporating variability intorepresentations

Psycholinguistic measures that quantify words’phonological properties in the lexicon generallyignore their variability in pronunciation. In usage,segmental context, or lexical factors such as wordfrequency, can significantly influence the phoneticrealization of a given phone, ranging from assimi-latory processes (Ohala, 1990a) to massive reduc-tion and even complete omission (Pitt et al., 2005;Johnson, 2004, inter alia). For example, there areover 200 distinct transcriptions of the word “and”in the Buckeye corpus (Pitt et al., 2005), and itsnormative, dictionary pronunciation (i.e. [ænd])only accounts for 3% of its realizations.

Measures such as PND rely on single, fixedpronunciations (generally normative/dictionary-based) and corpus-derived lexical frequencies toestimate how many similar-sounding words agiven word has, but take no account of variabil-ity in realization. As there is evidence that listen-ers remember and can access/use individual exem-plars of perceived speech (Pierrehumbert, 1980;Goldinger, 1998), it seems natural to model dis-tinct realizations within the lexical network. Thevariability in a word’s realizations may especiallymatter for identifying phonological competitors(Luce and Pisoni, 1998; Marian et al., 2012; Vadenet al., 2009). For example, words like “sand” and

207

“and” may rarely compete during lexical access,given that “and” is rarely pronounced similarly to“sand.” By incorporating the variability availablein naturalistic speech corpora, we hope to providea better characterization of a word’s phonologicalproperties and its relation to the lexicon.

3 Latent phonological representations

Representing arbitrary-length sequences ofphones with a single distributed representationhas a number of potential practical and conceptualadvantages. On the practical side, these repre-sentations have a fixed dimensionality, so findingmeaningful groupings or clusters is computa-tionally more tractable than directly clusteringvariable-length sequences. Moreover, projectingthese sequences into a latent space offers thepotential of discovering hidden relationshipsor variables that affect phonological or lexicalstructure.

Our aim in this paper is to test whether and towhat extent recent approaches to building sentencerepresentations can also be applied to the phono-logical domain. Both simpler and more complexlatent representations can be constructed to char-acterize the phonological forms of words. We firstdiscuss potential “naıve” means of accomplishingthis, and then move into discussion of our pro-posed model.

Principal components on bag-of-n-phonesA number of document classification schemes andinformation retrieval tasks have treated documentsas a product of the vector representations of wordslearned by principal components analysis (PCA;Landauer and Dumais, 1997). We apply this tothe phonetic domain as well. By analogy to abag of words, we refer to bag-of-phones (unigramfeatures) and bag-of-n-phones (higher-order seg-ment co-occurrence categories), which can thenbe fed into a dimensionality reduction algorithmlike principal components analysis (PCA) as anapproximate composition function to produce la-tent phonological representations of words.

doc2vecAnother dimensionality reduction method extendsthe continuous bag-of-words algorithm used tolearn word vectors (Mikolov et al., 2013) to thedocument domain. Specifically, the model learnsto compose (predict) a document (i.e. a word)from its phonological contents. doc2vec (Le and

Mikolov, 2014) has been used in information re-trieval and natural language processing applica-tions (Lau and Baldwin, 2016) and so may be aviable way to obtain lexical phonological repre-sentations. As with bag-of-phones, this model isinsensitive to serial order.

Sequential representationsEncoder-decoder or sequence-to-sequence(seq2seq henceforth) neural network architectureshave shown considerable success in encodingsentences (viz. sequences of words) for taskssuch as machine translation (Sutskever et al.,2014; Cho et al., 2014). These methods may beappropriate as a means of composing segmentalrepresentations, as they are intrinsically sensitiveto ordering, easily take usage frequencies intoaccount (directly from training corpora), and havebeen shown to be effective learners of sequentialdistributional properties of their training data.

4 Seq2seq model

We trained seq2seq models to either reproducetheir input, or to recover (predict) normative (dic-tionary) pronunciations from the phonetic tran-scriptions of words in the Buckeye corpus (Pittet al., 2005), a dataset of monologues provided inresponse to interviewer questions about the talk-ers’ hometown of Columbus, Ohio. The corpuscontains approximately 300,000 words.

Data inclusion criteria. There are some tran-scription errors in the Buckeye corpus, and so weexcluded combinations of phones that did not oc-cur at least ten times. This removes many errors,but a few remain. For example, the segment “h”occurs in some transcriptions but is not part of thecharacter set of the transcription dictionary, and isthus likely an error of omission for actual digraphsfrom the dictionary; “th”, “hh”, etc. Despite thepresence of these remaining errors, we do not cor-rect the transcriptions of any words. In total, 57phone/segment categories are represented. Fulldocumentation of the coding scheme used in thecorpus can be read in Pitt et al. (2005). For bag-of-n-phones features, we add the additional char-acters “w s” and “w e” as word boundary charac-ters, signaling the starts and ends of words, respec-tively.

There are no standard train/dev/test splits for theBuckeye corpus, and so we restricted ourselves torandomly selected 80/20 train/test split (Pitt et al.,2005) for training all models.

208

Figure 1: Encoder-decoder LSTM architecture (Nor-mative decoder; for the Observed decoder, the outputis the observed phonetic sequence).

Model architecture. Methodologically, we ap-proach the problem with an eye to restricting thecomputational power of our model, and to re-stricting the space of hyperparameters to explore.To this end, our models use a basic recurrentencoder-decoder architecture, with an input-sideembedding layer, and single-layer, unidirectional3

LSTMs (Hochreiter and Schmidhuber, 1997) onthe encoder and decoder sides. The encoder takesas input a sequence of phone indices (e.g. “cat”→ [’k’, ’ae’, ’tq’] → [11, 1, 20]), em-beds them, and encodes the sequence in the spacedefined by the LSTM. The encoder LSTM’s finalhidden state is provided as input to the decoder,whose task is to “unroll” this latent representa-tion. The outputs of the decoder LSTM are suc-cessively fed through a softmax, sequentially out-putting class probabilities for each character classin the phone vocabulary, which are then decodedvia simple argmax (see Figure 1).

4.1 Training

Hyperparameters. The number of trainingepochs was empirically determined on the basis ofasymptoting training loss, which we determined tobe 25 epochs. We used a cross-entropy loss func-tion, using the Adam optimizer (Kingma and Ba,2015) with a learning rate of 0.001. Other Adamparameters were at default values in the dynetpython implementation as of this writing (version2.0.3; Neubig et al., 2017). All hyperparameterswere selected on the basis of asymptoting loss ona small subset of the training set. The embedding

3While we do not perform these experiments here, we be-lieve that a Bi-LSTM encoder (Schuster and Paliwal, 1997)will enable further advances in constructing psycholinguisti-cally predictive word representations.

layer had 32 dimensions, and the encoder and de-coder LSTMs were 64-dimensional.

Tasks. We trained two models to performslightly different decoding tasks; the Norma-tive Decoder model, and the Observed Decodermodel. In both tasks, the inputs are transcriptionsof observed realizations of words in the Buck-eye corpus, which include e.g. phonetic changesand omissions. The Normative Decoder’s taskis to output the word’s normative pronunciation(e.g. [k, ae, tq] → [k, ae, t]), whilethe Observed Decoder model is trained as a se-quential autoencoder (e.g. Chung et al., 2016);the task is to reproduce the input sequence exactly.Both are potentially viable approaches to the cre-ation of lexical phonological representations andshow similar performance in the downstream tasksreported on below, which may be useful for re-searchers who only have access to normative pro-nunciations.

We evaluated the performance of the model onthe 20% held-out portion of the corpus.

4.2 Lexical representations

Once the model is trained, any sequence of phonescan be input to the encoder, yielding a latentphonological representation of that sequence. Aswith character-based NLP models, the compara-tively low dimensionality of the input space (57segments) mitigates sparsity issues, consequentlywe can obtain latent phonological representationsnot just of vocabulary words that have been trainedbut also for rare, out-of-vocabulary (OOV) wordsand non-words. We plot some aspects of thelearned representations in Figures 2 and 3. Onepattern that is particularly apparent is that the left-to-right serial nature of the encoder leads to repre-sentations that strongly encode the final segmentin their representations, for both consonants andvowels.

5 Evaluation

As a preliminary investigation of the informa-tion encoded in the learned lexical representations,we assess their ability to model phonetic dura-tion, which is known to be sensitive to phono-tactic probability and phonological overlap (Gahlet al., 2012; Watson et al., 2015; Buz and Jaeger,2016; Yiu and Watson, 2015; Goldrick and Lar-son, 2008; Vitevitch and Luce, 2005), in additionto other factors like contextual predictability (e.g.

209

Figure 2: Topology of word vectors from phonological encoder models learned by t-SNE (Maaten and Hinton,2008). Degree to which word vectors encode vowel information. Clusters largely prioritize word-final information,especially the last segment. Left graph represents the identities of the first segment. Right graph represents theidentities of the final segment. The strong encoding of the final segment may be due to the model architectureusing uni-directional recurrent layers.

Figure 3: Topology of word vectors, t-SNE projection (Maaten and Hinton, 2008). Degree to which word vectorsencode consonant information. Clusters largely prioritize word-final information, especially the last segment. Leftgraph represents the identities of the first segment. Right graph represents the identities of the final segment.

Cohen Priva and Jaeger, 2018; Seyfarth, 2014).We show that the encoder creates sequence repre-sentations that are useful for predicting word du-ration, and compare the success of the encoder toseveral other models, described below.

5.1 Predicting word duration

Ultimately we are interested in whether latentphonological representations have predictive va-lidity for phonetic cues, potentially in conjunc-tion with other phonological and lexical repre-sentations. Word duration has been shown to bestrongly related to phonological structure (Gahlet al., 2012), because duration may reflect the me-chanics of the phonological sequencing process inlanguage production (Yiu and Watson, 2015; Wat-son et al., 2015; Fox et al., 2015) or because speak-ers lengthen words in dense neighborhoods to pro-mote the listener’s understanding (Tily and Kuper-

man, 2012).We built a series of nested statistical models de-

signed to predict whole-word phonetic duration.The durations were obtained by summing up thedurations of each of the annotated phonetic seg-ments for an individual word, which are them-selves derived from time stamps extracted fromthe Buckeye metadata. Whole-word durationswere log transformed due to their positive skew;failing to account for this can make statistical in-ference more difficult (Campbell, 1992). All mod-els were constructed using ridge (L1 norm) re-gression using the scikit-learn package inPython (version 0.2.0; Pedregosa et al., 2011). Wereport goodness of fit measures in all cases byR2 values (the coefficient of determination; pro-vided automatically by the score function withinthe ridge regression model object).

All duration models were trained on the same

210

80-20% split that was used to train the encoder-decoder. Consequently, there were 282,742 obser-vations (words) during training, and 70,686 wordsat test. The vocabulary for the bag-of-words rep-resentations was estimated from the training data.All models are summarized in Table 1.

5.2 Baseline models

Word embeddings. A word’s distributional prop-erties, such as its part of speech and meaning; la-tent part-of-speech; or word-frequency informa-tion may reliably predict a word’s duration (Sey-farth, 2014; Turnbull et al., 2018; Priva, 2015).Consequently, we incorporate 100-dimensionalword embeddings into the regression models. Weobtained these word embeddings from gensim’s(Rehurek and Sojka, 2010) skip-gram implemen-tation trained on the Fisher corpus (Cieri et al.,2004), which we selected due to its size, whichis critical for generating good word embeddings(Antoniak and Mimno, 2018), and because it be-longs to the same domain as the Buckeye corpus(conversational speech).

The skip-gram model used a context window of5 words and a negative sampling size of 5. Weused a zero vector to represent OOV (e.g. Colum-bus, Ohio-specific place names that would not oc-cur in the Fisher corpus). Word embeddings were,on their own, not a strong predictor of word dura-tion (R2 = 0.082) on the test set, but neverthelessaccount for some of the variance in word duration.

Bag-of-phones models. Bag-of-words repre-sentations are a useful and informative baselinein other NLP tasks, especially text classification(Wang and Manning, 2012). We obtained bag-of-phone representations by learning a vocabulary onthe training data and creating sparse count vectorsin which the features represent individual phones.A simple bag-of-uniphones model, which ignoresorder information, has greater predictive powerthan word embeddings on the test set (R2=0.140).This shows that it is possible to at least partly pre-dict the duration of a given word’s realization fromrelatively unstructured phonological information.

Bag-of-n-phones. Unlike bag-of-words repre-sentations, bag-of-ngrams encode localized orderinformation. We constructed n-gram features ofphone combinations (bag-of-n-phones) of lengths2 to 5, using a cutoff frequency of 10 observations.These more complex representations performedsimilarly to the simpler bag-of-phones model on

the test set (R2 = 0.140).We also tested whether incorporating word

boundary information into these models (“w s”and “w e” phones) would induce boundary-sensitive phonotactics, but this also did not pro-vide additional gains over simpler models (R2 =0.138 and R2 = 0.140).

Principal components analysis over bag-of-n-phones. Following from the previous section,we take our bag-of-n-phones representations andfeed them into a truncated singular value de-composition model to obtain latent representa-tions of words (“documents”). This representationexplained a slightly greater amount of variancein word duration than word embeddings (R2 =0.106). However, this method performed far worsethan the bag-of-phones and bag-of-n-phones mod-els described in the previous section, indicatingthat some information is lost in this dimensionalityreduction method.

doc2vec. Our doc2vec model vectors weretrained to predict a word from a phonological rep-resentation. The resulting vectors had the samedimensionality as the PCA vectors and the en-coder output of the seq2seq models. Surprisingly,doc2vec performed the worst of models that weconsidered (R2 = -0.05).

seq2seq. The outputs of the encoders for theObserved and Normative decoder models wereamong the best we considered, both on their ownand in conjunction with other measures. Inter-estingly, the Observed Decoder provides a muchcloser fit to phonetic duration than word embed-dings, bag-of-phones, PCA, doc2vec, and the Nor-mative Decoder representations. When combinedwith bag-of-phones and word embedding infor-mation, the Observed Decoder representations ex-plain the greatest amount of variance in word du-ration (R2 = 0.181), suggesting that these latentphonological representations encode useful infor-mation for characterizing word form.

The disparity between the Observed and Nor-mative decoder models may be a consequenceof the Normative model’s more difficult learningproblem. One potential explanation is that de-spite training the two models for equal lengthsof time (25 epochs), the Normative decoder wasnot trained to the same criterion as the Observeddecoder. Future work should explore whetherthe worse performance of the Normative decodermodel is due to the precision of its representations

211

Simple Test R2 No. features Combined Test R2 No. featuresWord embeddings (WE) 0.082 100 BoP + wb + WE 0.161 159

Bag-of-phones (BoP) 0.140 57 + Observed decoder 0.181 223+ w s + w e (wb) 0.140 59 + Normative decoder 0.177 223

Bag-of-n-phones (BoNP) 0.140 1700 BoNP + wb + WE 0.159 5018+ w s + w e (wb) 0.138 4918 + Observed decoder 0.175 5082

PCA bag-of-n-phones 0.106 64 + Normative decoder 0.173 5082doc2vec -0.05 64 Observed + WE 0.149 164

Observed decoder 0.149 64 Normative + WE 0.141 164Normative decoder 0.140 64

Table 1: Ablation study. Effectiveness of features and combinations of features for predicting (log) phoneticduration.

or due to what is embedded in the representationsthemselves.

6 Probing phonological structure

While it is clear that seq2seq representations ofthe phonological forms of words are partiallypredictive of a phonetic phenomenon (duration),whether the representations encode anything use-ful about the lexicon requires further investiga-tion. In this section, we explore whether charac-terizing the similarity space of these phonologicalword vectors can approximate standard measuresof a word’s phonological properties. The resultsshow that the vectors produce coherent clusters ofwords with different phonological properties. Wealso show that there are correlations between ourmeasures and phonotactic probability.

6.1 Latent phonological neighborhooddensity

While it is not commonly the case that similarityscores follow a normal distribution, in our case,the similarity scores for words are by visual spotinspection roughly symmetric and normally dis-tributed, so we chose to characterize individualwords wi by the mean and standard deviation oftheir similarity scores to every other word in thelexicon. Although not a priori obvious, one possi-bility is that these metrics correlate with other lexi-cal metrics, for example, a wide standard deviationcould mean that a word has a number of differentways it can be similar to other words, whereas anarrow standard deviation suggests that the wordis fairly unique.

6.2 The similarity structure of the lexicon

The distributions of similarity scores show someinteresting properties. Unlike the measurementsof phonological neighborhood density provided inVaden et al. (2009), which follow a quasi-Zipfiandistribution, a histogram of the mean word-lexiconsimilarities across the whole vocabulary shows avery different pattern. In particular, there appearto be three distinct clusters of similarity scores, asshown in Figure 4.

Figure 4: Three clusters of similarity scores from Ob-served Decoder model.

Words in the first cluster, which show negativeaverage similarity scores, were highly frequentwords, typically encompassing function words(e.g. but, about, the). The second cluster ap-peared to include less high-frequency terms (e.g.day, brain, wants). Finally, the rightmost clustertypically had higher similarity scores, represent-ing low frequency and longer words (e.g. devices,widely, element).4 Going forward, a meta-model

4We thank our reviewers for pointing out that all of theseproperties are correlated with word length in segments (e.g.highly frequent words are on average shorter), which is a use-ful baseline that we will explore in future work.

212

will be necessary to determine what factors deter-mine a word’s mean lexicon-similarity value.

6.3 Correlation with existing phonologicalproperties

Ideally, a new measure of phonological formshould relate to measures already known to af-fect speech production. For example, a significantcorrelation with a particular word’s mean or stan-dard deviation similarity to all the other words inthe lexicon would suggest that our measures char-acterize the lexicon in a similar way to existingmeasures. Similarly, because our latent represen-tations encode sequences, we expect them to cor-relate with phonotactic probability (Vitevitch andLuce, 2004). So, as a final set of analyses, wesought to test whether and to what extent the Ob-served decoder learns representations that can tellus about a word’s relationship to the rest of thelexicon.

There are two measures of interest that have re-ceived some attention in the speech production lit-erature. For the present analyses, we referencethe phonological neighborhood density metricsas well as the phonotactic probability scores forwords in Buckeye that are also in the Irvine Phono-tactic Online Dictionary (IPhOD; Vaden et al.,2009). We show that our measures (both mean andstandard deviation) strongly correlate with phono-tactic probability and IPhoD’s additional PNDmeasure. This suggests that the vectors’ useful-ness extends to researchers who wish to explorethe phonological similarity structure of the lexiconfor psycholinguistic research.

Phonological neighborhood density. Giventhe importance of phonological neighborhooddensity (PND) in speech production (Luce andPisoni, 1998; Vitevitch and Luce, 2005; Metsala,1997; Mirman, 2011), we correlated the (log)number of phonological neighbors with our latentdensity scores and phonetic duration. A phono-logical neighbor is a word that differs by a singlesound (either an addition, a substitution, or a dele-tion; Levenshtein, 1966). PND ((log) # of neigh-bors, Figure 5) has a strong negative correlationwith mean word-lexicon similarity (greater meansimilarity translates to fewer neighbors; ρ = -.59)while the standard deviation of word-lexicon sim-ilarity shows a non-linear relationship with neigh-borhood density.

Phonotactic probability. Phonotactic proba-

bility is a measure of the phonological typicalityof a word, computed from product of uni-phoneand bi-phone probabilities of that word pronunci-ation, in the same fashion that sentence probabil-ities are computed in a standard bigram languagemodel (Vitevitch and Luce, 2004, 2005). In ourfinal analysis, we compare the mean and standarddeviation of a word’s similarity to all other wordtypes, including alternate pronunciations of thesame word, to existing measures of phonotacticprobability. As with phonological neighborhooddensity, we see significant positive correlations be-tween our phonological similarity measures (bothmeans and standard deviations; ρ = 0.41 and ρ =0.13, respectively) between phonotactic probabili-ties, which we visualize in Figure 5.

7 Conclusion

The results presented here suggest that encoder-decoder models are a promising frameworkfor composing segment-based representations ofwords. The models also characterize words’phonological forms relative to the rest of the lex-icon. We believe that encoder-decoder models’usefulness extends beyond that of many exist-ing approaches, as they can seamlessly gener-ate gestalt representations for out-of-vocabularywords and even nonce words. Our approach hasa number of potential advantages for the cog-nitive modeling of language processing in bothcomprehension and production tasks, or indeed inany task that can be modeled with phonologicalword representations. Importantly, the encoder-decoder modeling framework is flexible, learn-ing both from observed, quasi-phonetic realiza-tions of words as well as from idealized, normative(dictionary-based) pronunciations, and allows formany variations in expressivity and computationalpower.

The reported correlations between phonologi-cal neighborhood density, phonotactic probability,latent phonological similarity, and phonetic dura-tion motivate a need to better understand the em-bedding representations themselves. We have pre-sented considerable evidence that the models cap-ture some non-trivial dependencies between pho-netic segments that can characterize word forms.Going forward, we believe that our latent phono-logical representations may be useful for design-ing stimuli, or provide an alternative to standardcovariates in psycholinguistic experiments such

213

Figure 5: Correlation between a word’s phonetic duration in Buckeye, phonological neighborhood density, globalword-lexicon similarity (mean and standard deviation), and phonotactic probability.

as phonological neighborhood density and phono-tactic probability. Finally, our results on theNormative-Decoder suggest that low-resource lan-guages with only a pronunciation dictionary arealso a viable means of learning these represen-tations, assuming that there is a correspondingcorpus of conversational data. In sum, we havedemonstrated that our approach is useful for mod-eling of phonological structure.

ReferencesMaria Antoniak and David Mimno. 2018. Evaluating

the stability of embedding-based word similarities.Transactions of the Association of ComputationalLinguistics, 6:107–119.

R Harald Baayen, Richard Piepenbrock, and Rijn vanH. 1993. The {CELEX} lexical data base on {CD-ROM}. Linguistic Data Consortium.

Esteban Buz and T Florian Jaeger. 2016. The (in) de-pendence of articulation and lexical planning duringisolated word production. Language, Cognition andNeuroscience, 31:404–424.

W Nick Campbell. 1992. Syllable-based segmental du-ration. Talking machines: Theories, models, and de-signs, pages 211–224.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua,Nicole Limtiaco, Rhomni St John, Noah Constant,Mario Guajardo-Cespedes, Steve Yuan, Chris Tar,et al. 2018. Universal sentence encoder. arXivpreprint arXiv:1803.11175.

Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder–decoderfor statistical machine translation. Proceedings ofthe 2014 Conference on Empirical Methods in Nat-ural Language Processing (EMNLP), pages 1724–1734.

Noam Chomsky. 1968. The sound pattern of English.Studies in language. Harper & Row, New York.

Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen,Hung-Yi Lee, and Lin-Shan Lee. 2016. Audioword2vec: Unsupervised learning of audio segmentrepresentations using sequence-to-sequence autoen-coder. Interspeech 2016, pages 765–769.

Christopher Cieri, David Miller, and Kevin Walker.2004. The fisher corpus: a resource for the nextgenerations of speech-to-text. In LREC, volume 4,pages 69–71.

Uriel Cohen Priva and T Florian Jaeger. 2018. The in-terdependence of frequency, predictability, and in-formativity in the segmental domain. LinguisticsVanguard, 4.

214

Alexis Conneau, Douwe Kiela, Holger Schwenk, LoıcBarrault, and Antoine Bordes. 2017. Supervisedlearning of universal sentence representations fromnatural language inference data. In Proceedings ofthe 2017 Conference on Empirical Methods in Nat-ural Language Processing, pages 670–680.

Colin J Davis. 2010. The spatial coding model ofvisual word identification. Psychological Review,117:713–758.

Gary S Dell. 1986. A spreading-activation theory ofretrieval in sentence production. Psychological Re-view, 93:283–321.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. Bert: Pre-training of deepbidirectional transformers for language understand-ing. arXiv preprint arXiv:1810.04805.

John R Firth. 1957. A synopsis of linguistic theory,1930-1955. Studies in linguistic analysis.

Neal P Fox, Megan Reilly, and Sheila E Blumstein.2015. Phonological neighborhood competition af-fects spoken word production irrespective of sen-tential context. Journal of Memory and Language,83:97–117.

Stefan Frisch. 1996. Similarity and frequency inphonology. Ph.D. thesis, Northwestern University.

Susanne Gahl, Yao Yao, and Keith Johnson. 2012.Why reduce? phonological neighborhood densityand phonetic reduction in spontaneous speech. Jour-nal of Memory and Language, 66:789–806.

Stephen D Goldinger. 1998. Echoes of echoes? anepisodic theory of lexical access. Psychological Re-view, 105:251–279.

Matthew Goldrick and Meredith Larson. 2008. Phono-tactic probability influences speech production.Cognition, 107:1155–1164.

Mark Hale and Charles Reiss. 2008. The PhonologicalEnterprise. Studies in language. Oxford UniversityPress, New York.

Zellig S Harris. 1954. Distributional structure. Word,10:146–162.

Sepp Hochreiter and Jurgen Schmidhuber. 1997.Long short-term memory. Neural computation,9(8):1735–1780.

George Houghton and Tom Hartley. 1996. Parallelmodels of serial behaviour: Lashley revisited. Psy-che: An Interdisciplinary Journal of Research onConsciousness.

Keith Johnson. 2004. Massive reduction in conver-sational american english. In Spontaneous speech:Data and analysis. Proceedings of the 1st sessionof the 10th international symposium, pages 29–54.Citeseer.

Armand Joulin, Edouard Grave, Piotr Bojanowski, andTomas Mikolov. 2017. Bag of tricks for efficienttext classification. In Proceedings of the 15th Con-ference of the European Chapter of the Associationfor Computational Linguistics: Volume 2, Short Pa-pers, volume 2, pages 427–431.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In Proceed-ings of the 3rd International Conference on Learn-ing Representations (ICLR).

Thomas K Landauer and Susan T Dumais. 1997. A so-lution to plato’s problem: The latent semantic anal-ysis theory of acquisition, induction, and representa-tion of knowledge. Psychological Review, 104:211–240.

Jey Han Lau and Timothy Baldwin. 2016. An empiri-cal evaluation of doc2vec with practical insights intodocument embedding generation. In Proceedingsof the 1st Workshop on Representation Learning forNLP, pages 78–86.

Quoc Le and Tomas Mikolov. 2014. Distributed repre-sentations of sentences and documents. In Interna-tional conference on machine learning, pages 1188–1196.

Willem JM Levelt, Ardi Roelofs, and Antje S Meyer.1999. A theory of lexical access in speech produc-tion. Behavioral and Brain Sciences, 22:1–38.

Vladimir I Levenshtein. 1966. Binary codes capableof correcting deletions, insertions, and reversals. InSoviet Physics Doklady, volume 10, pages 707–710.

Paul A Luce and David B Pisoni. 1998. Recognizingspoken words: The neighborhood activation model.Ear and Hearing, 19:1–36.

Laurens van der Maaten and Geoffrey Hinton. 2008.Visualizing data using t-sne. Journal of MachineLearning Research, 9:2579–2605.

Viorica Marian, James Bartolotti, Sarah Chabal,and Anthony Shook. 2012. Clearpond: Cross-linguistic easy-access resource for phonological andorthographic neighborhood densities. PloS one,7(8):e43230.

William Marslen-Wilson and Paul Warren. 1994. Lev-els of perceptual representation and process in lexi-cal access: words, phonemes, and features. Psycho-logical review, 101(4):653.

Jamie L Metsala. 1997. An examination of word fre-quency and neighborhood density in the develop-ment of spoken-word recognition. Memory & Cog-nition, 25(1):47–56.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-ity. In Advances in Neural Information ProcessingSystems, pages 3111–3119.

215

Daniel Mirman. 2011. Effects of near and distant se-mantic neighbors on word production. Cognitive,Affective, & Behavioral Neuroscience, 11(1):32–43.

Graham Neubig, Chris Dyer, Yoav Goldberg, AustinMatthews, Waleed Ammar, Antonios Anastasopou-los, Miguel Ballesteros, David Chiang, DanielClothiaux, Trevor Cohn, Kevin Duh, ManaalFaruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji,Lingpeng Kong, Adhiguna Kuncoro, Gaurav Ku-mar, Chaitanya Malaviya, Paul Michel, YusukeOda, Matthew Richardson, Naomi Saphra, SwabhaSwayamdipta, and Pengcheng Yin. 2017. Dynet:The dynamic neural network toolkit. arXiv preprintarXiv:1701.03980.

John J Ohala. 1990a. The phonetics and phonologyof aspects of assimilation. Papers in LaboratoryPhonology, 1:258–275.

John J Ohala. 1990b. There is no interface betweenphonology and phonetics: a personal view. Journalof Phonetics, 18:153–171.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,B. Thirion, O. Grisel, M. Blondel, P. Pretten-hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas-sos, D. Cournapeau, M. Brucher, M. Perrot, andE. Duchesnay. 2011. Scikit-learn: Machine learningin Python. Journal of Machine Learning Research,12:2825–2830.

Janet Breckenridge Pierrehumbert. 1980. The phonol-ogy and phonetics of English intonation. Ph.D. the-sis, Massachusetts Institute of Technology.

Mark A Pitt, Keith Johnson, Elizabeth Hume, ScottKiesling, and William Raymond. 2005. The buck-eye corpus of conversational speech: labeling con-ventions and a test of transcriber reliability. SpeechCommunication, 45:89–95.

Uriel Cohen Priva. 2015. Informativity affects con-sonant duration and deletion rates. LaboratoryPhonology, 6(2):243–278.

Radim Rehurek and Petr Sojka. 2010. Software Frame-work for Topic Modelling with Large Corpora. InProceedings of the LREC 2010 Workshop on NewChallenges for NLP Frameworks, pages 45–50, Val-letta, Malta. ELRA. http://is.muni.cz/publication/884893/en.

M. Schuster and K.K. Paliwal. 1997. There is no in-terface between phonology and phonetics: a per-sonal view. IEEE Transactions on Signal Process-ing, 45:2673–2681.

Mark S Seidenberg and James L McClelland. 1989. Adistributed, developmental model of word recogni-tion and naming. Psychological Review, 96:523–568.

Christine A Sevald and Gary S Dell. 1994. The sequen-tial cuing effect in speech production. Cognition,53:91–127.

Scott Seyfarth. 2014. Word informativity influ-ences acoustic duration: Effects of contextual pre-dictability on lexical representation. Cognition,133(1):140–155.

Daragh E Sibley, Christopher T Kello, David C Plaut,and Jeffrey L Elman. 2008. Large-scale modelingof wordform learning and representation. CognitiveScience, 32(4):741–754.

Kenneth N Stevens and Sheila E Blumstein. 1981. Thesearch for invariant acoustic correlates of phoneticfeatures. Perspectives on the study of speech, pages1–38.

Lidia Suarez, Seok Hui Tan, Melvin J Yap, and Win-ston D Goh. 2011. Observing neighborhood effectswithout neighbors. Psychonomic Bulletin & Review,18(3):605–611.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural net-works. In Advances in Neural Information Process-ing Systems, pages 3104–3112.

Harry Tily and Victor Kuperman. 2012. Rationalphonological lengthening in spoken dutch. TheJournal of the Acoustical Society of America,132(6):3935–3940.

Rory Turnbull, Scott Seyfarth, Elizabeth Hume, andT Florian Jaeger. 2018. Nasal place assimilationtrades off inferrability of both target and triggerwords. Laboratory Phonology: Journal of the As-sociation for Laboratory Phonology, 9(1).

Kenneth I Vaden, HR Halpin, and Gregory SHickok. 2009. Irvine phonotactic online dictio-nary, version 2.0. [data file]. Available fromhttp://www.iphod.com.

Michael S Vitevitch and Paul A Luce. 2004. A web-based interface to calculate phonotactic probabil-ity for words and nonwords in english. Behav-ior Research Methods, Instruments, & Computers,36:481–487.

Michael S Vitevitch and Paul A Luce. 2005. In-creases in phonotactic probability facilitate spokennonword repetition. Journal of Memory and Lan-guage, 52:193–204.

Sida Wang and Christopher D. Manning. 2012. Base-lines and bigrams: simple, good sentiment and topicclassification. Proceedings of the 50th Annual Meet-ing of the Association for Computational Linguis-tics: Short Papers, 2:90–94.

Duane G Watson, Andres Buxo-Lugo, and Do-minique C Simmons. 2015. The effect of phono-logical encoding on word duration: Selection takestime. In Explicit and implicit prosody in sentenceprocessing, pages 85–98. Springer.

216

Loretta K Yiu and Duane G Watson. 2015. When over-lap leads to competition: Effects of phonological en-coding on word duration. Psychonomic Bulletin &Review, 22:1701–1708.

217