Strategies for Structuring Story Generation Angela Fan FAIR, Paris LORIA, Nancy [email protected]Mike Lewis FAIR, Seattle [email protected]Yann Dauphin Google AI * [email protected]Abstract Writers often rely on plans or sketches to write long stories, but most current language models generate word by word from left to right. We explore coarse-to-fine models for creating nar- rative texts of several hundred words, and in- troduce new models which decompose stories by abstracting over actions and entities. The model first generates the predicate-argument structure of the text, where different mentions of the same entity are marked with placeholder tokens. It then generates a surface realiza- tion of the predicate-argument structure, and finally replaces the entity placeholders with context-sensitive names and references. Hu- man judges prefer the stories from our models to a wide range of previous approaches to hi- erarchical text generation. Extensive analysis shows that our methods can help improve the diversity and coherence of events and entities in generated stories. 1 Introduction Stories exhibit structure at multiple levels. While existing language models can generate stories with good local coherence, they struggle to coalesce in- dividual phrases into coherent plots or even main- tain character consistency throughout a story. One reason for this failure is that classical language models generate the whole story at the word level, which makes it difficult to capture the high-level interactions between the plot points. To address this, we investigate novel decompo- sitions of the story generation process that break down the problem into a series of easier coarse-to- fine generation problems. These decompositions can offer three advantages: • They allow more abstract representations to be generated first, where challenging long- range dependencies may be more apparent. *Work done while at Facebook AI Research Figure 1: Proposed Model. Conditioned upon the prompt, we generate sequences of predicates and ar- guments. Then, a story is generated with placeholder entities such as ent0. Finally we replace the placehold- ers with specific references. • They allow specialized modelling techniques for the different stages, which exploit the structure of the specific sub-problem. • They are applicable to any textual dataset and require no manual labelling. Several hierarchical models for story generation have recently been proposed (Xu et al., 2018; Yao et al., 2019), but it is not well understood which properties characterize a good decomposition. We therefore implement and evaluate several repre- sentative approaches based on keyword extraction, sentence compression, and summarization. We build on this understanding to devise the proposed decomposition (Figure 1). Our approach breaks down the generation process in three steps: modelling the action sequence, the story narra- tive, and lastly entities such as story characters. To model action sequences, we first generate the predicate-argument structure of the story by gen- erating a sequence of verbs and arguments. This
Writers often rely on plans or sketches to writelong stories, but most current language modelsgenerate word by word from left to right. Weexplore coarse-to-fine models for creating nar-rative texts of several hundred words, and in-troduce new models which decompose storiesby abstracting over actions and entities. Themodel first generates the predicate-argumentstructure of the text, where different mentionsof the same entity are marked with placeholdertokens. It then generates a surface realiza-tion of the predicate-argument structure, andfinally replaces the entity placeholders withcontext-sensitive names and references. Hu-man judges prefer the stories from our modelsto a wide range of previous approaches to hi-erarchical text generation. Extensive analysisshows that our methods can help improve thediversity and coherence of events and entitiesin generated stories.
1 Introduction
Stories exhibit structure at multiple levels. Whileexisting language models can generate stories withgood local coherence, they struggle to coalesce in-dividual phrases into coherent plots or even main-tain character consistency throughout a story. Onereason for this failure is that classical languagemodels generate the whole story at the word level,which makes it difficult to capture the high-levelinteractions between the plot points.
To address this, we investigate novel decompo-sitions of the story generation process that breakdown the problem into a series of easier coarse-to-fine generation problems. These decompositionscan offer three advantages:
• They allow more abstract representations tobe generated first, where challenging long-range dependencies may be more apparent.
*Work done while at Facebook AI Research
Figure 1: Proposed Model. Conditioned upon theprompt, we generate sequences of predicates and ar-guments. Then, a story is generated with placeholderentities such as ent0. Finally we replace the placehold-ers with specific references.
• They allow specialized modelling techniquesfor the different stages, which exploit thestructure of the specific sub-problem.• They are applicable to any textual dataset and
require no manual labelling.
Several hierarchical models for story generationhave recently been proposed (Xu et al., 2018; Yaoet al., 2019), but it is not well understood whichproperties characterize a good decomposition. Wetherefore implement and evaluate several repre-sentative approaches based on keyword extraction,sentence compression, and summarization.
We build on this understanding to devise theproposed decomposition (Figure 1). Our approachbreaks down the generation process in three steps:modelling the action sequence, the story narra-tive, and lastly entities such as story characters.To model action sequences, we first generate thepredicate-argument structure of the story by gen-erating a sequence of verbs and arguments. This
representation is more structured than free text,making it easier for the model learn dependenciesacross events. To model entities, we initially gen-erate a version of the story where different men-tions of the same entity are replaced with place-holder tokens. Finally, we re-write these tokensinto different references for the entity, based onboth its previous mentions and global story con-text.
The models are trained on a large dataset of300k stories, and we evaluate quality both in termsof human judgments and using automatic metrics.We find that our novel approach leads to muchbetter story generation. Specifically, we showthat generating the action sequence first makesthe model less prone to generating generic events,leading to a much greater diversity of verbs. Wealso find that by using sub-word modelling for theentities, our model can produce novel names forlocations and characters that are appropriate giventhe story context.
2 Model Overview
The crucial challenge of long story generation liesin maintaining coherence across a large numberof generated sentences—in terms of both the log-ical flow of the story and the characters and enti-ties. While there has been much recent progressin left-to-right text generation, particularly usingself-attentive architectures (Dai et al., 2018; Liuet al., 2018), we find that models still struggle tomaintain coherence to produce interesting storieson par with human writing. We therefore intro-duce strategies to decompose neural story gener-ation into coarse-to-fine steps to make modellinghigh-level dependencies easier to learn.
2.1 Tractable Decompositions
In general, we can decompose the generation pro-cess by converting a story x into a more abstractrepresentation z. The negative log likelihood ofthe decomposed problem is given by
L = − log∑z
p(x|z)p(z). (1)
We can generate from this model by first samplingfrom p(z) and then sampling from p(x|z). How-ever, the marginalization over z is in general in-tractable, except in special cases where every xcan only be generated by a single z (for exam-ple, if the transformation removed all occurrences
of certain tokens). Instead, we minimize a vari-ational upper bound of the loss by constructing adeterministic posterior q(z|x) = 1z=z∗ , where z∗
can be given by running semantic role labeller orcoreference resolution system on x. Put together,we optimize the following loss:
z∗ = argmaxz
p(z|x) (2)
L ≤ − log p(x|z∗)− log p(z∗) (3)
This approach allows models p(z∗) and p(x|z∗) tobe trained tractably and separately.
2.2 Model Architectures
We build upon the convolutional sequence-to-sequence architecture (Gehring et al., 2017). Deepconvolutional networks are used as both the en-coder and decoder. The networks are connectedwith an attention module (Bahdanau et al., 2015)that performs a weighted sum of the encoder out-put. The decoder uses a gated multi-head self-attention mechanism (Vaswani et al., 2017; Fanet al., 2018) to allow the model to refer to previ-ously generated words and improve the ability tomodel long-range context.
3 Modelling Action Sequences
To decompose a story into a structured form thatemphasizes logical sequences of actions, we useSemantic Role Labeling (SRL). SRL identifiespredicates and arguments in sentences, and as-signs each argument a semantic role. This repre-sentation abstracts over different ways of express-ing the same semantic content. For example, Johnate the cake and the cake that John ate would re-ceive identical semantic representations.
Conditioned upon the prompt, we generate anSRL decomposition of the story by concatenatingthe predicates and arguments identified by a pre-trained model (He et al., 2017; Tan et al., 2018)1
and separating sentences with delimiter tokens.We place the predicate verb first, followed by itsarguments in canonical order. To focus on themain narrative, we retain only core arguments.
Verb Attention Mechanism SRL parses aremore structured than free text, allowing scopefor more structured models. To encourage the
1for predicate identification, we usehttps://github.com/luheng/deep srl, for SRL given pred-icates, we use https://github.com/XMUNLP/Tagger
Figure 2: Verb-Attention. To improve the model’s ability to condition upon past verbs, one head of the decoder’sself-attention mechanism is specialized to only attend to previously generated verbs.
model to consider sequences of verbs, we desig-nate one of the heads of the decoder’s multiheadself-attention to be a verb-attention head (see Fig-ure 2). By masking the self-attention appropri-ately, this verb-attention head can only attend topreviously generated verbs. When the text doesnot yet have a verb, the model attends to a zerovector. We show that focusing on verbs with a spe-cific attention head generates a more diverse arrayof verbs and reduces repetition in generation.
4 Modelling Entities
The challenges of modelling characters through-out a story is twofold: first, entities such as charac-ter names are rare tokens, which make them hardto model for neural language models. Human sto-ries often feature imaginative, novel character orlocation names. Second, maintaining the consis-tency of a specific set of characters is difficult,as the same entity may be referenced by manydifferent strings throughout a story—for exampleBilbo Baggins, he, and the hobbit may refer to thesame entity. It is challenging for existing languagemodels to track which words refer to which entitypurely using a language modelling objective.
We address both problems by first generating aform of the story with different mentions of thesame entity replaced by a placeholder token (e.g.ent0), similar to Hermann et al. (2015). We thenuse a sub-word seq2seq model trained to replaceeach mention with a reference, based on its con-text. The sub-word model is better equipped tomodel rare words and the placeholder tokens makemaintaining consistency easier.
4.1 Generating Entity Anonymized StoriesWe explore two approaches to identifying andclustering entities:
• NER Entity Anonymization: We use anamed entity recognition (NER) model2 to
identify all people, organizations, and loca-tions. We replace these spans with place-holder tokens (e.g. ent0). If any two entitymentions have an identical string, we replacethem with the same placeholder. For exam-ple, all mentions of Bilbo Baggins will be ab-stracted to the same entity token, but Bilbowould be a separate abstract entity.• Coreference-based Entity Anonymization:
The above approach cannot detect differ-ent mentions of an entity that use differentstrings. Instead, we use the Coreference Res-olution model from Lee et al. (2018)3 to iden-tify clusters of mentions. All spans in thesame cluster are then replaced with the sameentity placeholder string. Coreference mod-els do not detect singleton mentions, so wealso replace non-coreferent named entitieswith unique placeholders.
4.2 Generating Entity References in a Story
We train models to replace placeholder entitymentions with the correct surface form, forboth NER-based and coreference-based entityanonymised stories. Both our models use aseq2seq architecture that generates an entity ref-erence based on its placeholder and the story. Tobetter model the specific challenges of entity gen-eration, we also make use of a pointer mechanismand sub-word modelling.
Pointer Mechanism Generating multiple con-sistent mentions of rare entity names is challeng-ing. To make it easier for the model to re-use pre-vious names for an entity, we augment the stan-dard seq2seq decoder with a pointer-copy mecha-nism (Vinyals et al., 2015). To generate an entityreference, the decoder can either generate a newabstract entity token or choose to copy an alreadygenerated abstract entity token, which encourages
en core web lg3https://github.com/kentonl/e2e-coref
Figure 3: Input for Coreferent entity reference generation. The model has a representation of the entity context ina bag of words form, all previous predicted values for the same anonymized entity token, and the full text story.The green circle represents the entity mention the model is attempting to fill.
the model to use consistent naming for the entities.To train the pointer mechanism, the final hidden
state of the model h is used as input to a classifierpcopy(h) = σ(wcopy · h). wcopy is a fixed dimen-sion parameter vector. When the model classifierpredicts to copy, the previously decoded abstractentity token with the maximum attention value iscopied. One head of the decoder multi-head self-attention mechanism is used as the pointer copyattention head, to allow the heads to specialize.
Sub-word Modelling Entities are often rare ornovel words, so word-based vocabularies can beinadequate. We compare entity generation usingword-based, byte-pair encoding (BPE) (Sennrichet al., 2015), and character-level models.
NER-based Entity Reference GenerationHere, each placeholder string should map ontoone (possibly multiword) surface form—e.g. alloccurrences of the placeholder ent0 should maponly a single string, such as Bilbo Baggins. Wetrain a simple model that maps a combinationplaceholder token and story (with anonymizedentities) to the surface form of the placeholder.While the placeholder can appear multiple times,we only make one prediction for each placeholderas they all correspond to the same string.
Coreference-based Entity Reference Genera-tion Generating entities based on coreferenceclusters is more challenging than for our NERentity clusters, because different mentions of thesame entity may use different surface forms. Wegenerate a separate reference for each mention byadding the following inputs to the above model:
• A bag-of-words context window around thespecific entity mention, which allows localcontext to determine if an entity should be aname, pronoun or nominal reference.• Previously generated references for the same
entity placeholder. For example, if the model
is filling in the third instance of ent0, it re-ceives that the previous two generations forent0 were Bilbo, him. Providing the previousentities allows the model to maintain greaterconsistency between generations.
5 Experimental Setup
5.1 Data
We use the WRITINGPROMPTS dataset from (Fanet al., 2018) 4 of 300k story premises paired withlong stories. Stories are on average 734 words,making the generation far longer compared to re-lated work on storyline generation. In this work,we focus on the prompt to story generation aspectof this task. We assume models receive a human-written prompt, as shown in Figure 1. We fol-low the previous preprocessing of limiting storiesto 1000 words and fixing the vocabulary size to19,025 for prompts and 104,960 for stories.
5.2 Baselines
We compare our results to the Fusion model fromFan et al. (2018) which generates the full story di-rectly from the prompt. We also implement vari-ous decomposition strategies as baselines:
• Summarization: We propose a new baselinethat generates a summary conditioned uponthe prompt and then a story conditioned uponthe summary. Story summaries are obtainedwith a multi-sentence summarization model(Wu et al., 2019) trained on the full-text ver-sion of the CNN-Dailymail summarizationcorpus (Hermann et al., 2015; Nallapati et al.,2016; See et al., 2017)5 and applied to stories.• Keyword Extraction: We generate a series of
keywords conditioned upon the prompt and4https://github.com/pytorch/fairseq/
Figure 4: Human evaluations of different decomposed models for story generation. We find that using SRL actionplans and coreference-resolution to build entity clusters generates stories that are preferred by human judges.
Table 1: Negative log likelihood of generating storiesusing different decompositions (lower is easier for themodel). Stage 1 is the generation of the intermediaterepresentation z∗, and Stage 2 is the generation of thestory x conditioned upon z∗. Entity generation is witha word-based vocabulary to be consistent with the othermodels.
then a story conditioned upon the keywords,based on Yao et al. (2019). Following Yao etal, we extract keywords with the RAKE algo-rithm (Rose et al., 2010)6. Yao et al. extractone word per sentence, but we find that ex-tracting n = 10 keyword phrases per storyworked well, as our stories are much longer.
• Sentence Compression: Inspired by Xu et al.(2018), we generate a story with compressedsentences conditioned upon the prompt andthen a story conditioned upon the compres-sion. We use the same deletion-based com-pression data as Xu et al., from Filippova andAltun (2013)7. We train a seq2seq model tocompress all non-dialog story sentences (asthe training data does not contain much spo-ken dialogue). The compressed sentences areconcatenated to form the compressed story.
Figure 5: Average Longest Common Subsequence ofGenerated Stories with human-written stories in thetraining set.
5.3 Training
We implement models using fairseq-py (Ottet al., 2019)8 in PyTorch and train Fan et al.(2018)’s convolutional architecture. We tune allhyperparameters on validation data.
5.4 Generation
We suppress the generation of unknown tokens toease human evaluation. For all evaluations, we re-quire stories to be at least 150 words and cut offthe story at the nearest sentence for stories longerthan 250 words. We generate stories with temper-ature 0.8 and random top-k sampling method pro-posed in (Fan et al., 2018), where next words aresampled from the top k candidates rather than theentire vocabulary distribution. We set k = 10.
6 Experiments
6.1 Comparing Decomposition Strategies
Automated Evaluation We compare the rela-tive difficulty of modelling using each decompo-sition strategy by measuring the log loss of thedifferent stages in Table 1. We observe that gener-ating the SRL structure has a lower negative log-likelihood and so is much easier than generating
Figure 6: Our decomposition can generate more coher-ent stories than previous work.
either summaries, keywords, or compressed sen-tences — a benefit of its more structured form. Wefind keyword generation is especially difficult asthe identified keywords are often the more salient,rare words appearing in the story, which are chal-lenging for neural seq2seq models. This suggeststhat rare words should appear mostly at the lastlevels of the decomposition. Further, we comparemodels with entity-anonymized stories as an in-termediate representation, either with NER-basedor coreference-based entity anonymization. En-tity references are then filled using a word-basedmodel9. The entity fill is the more difficult stage.
Finally, we analyze the similarity of the gener-ated stories with the stories in the training set. Wequantify this by measuring the maximum and av-erage longest common subsequence of tokens ofa generated story with all human-written storiesfrom the training set. High LCS values would indi-cate models are copying large subparts from exist-ing stories rather than creatively writing new sto-ries. Results shown in Figure ?? indicate that ourproposed decomposition copies slightly less longsequences from the training set compared to thebaselines — by separating verb and entity gener-ation into distinct parts, we generate fewer longsequences already present in the training set.
Human Evaluation To compare overall storyquality using various decomposition strategies, weconduct human evaluation using a crowdworkingplatform. Judges are shown two different storiesthat were generated based on the same human-
9To make likelihoods are comparable across models.
Figure 7: Example generated action plan for the SRL+ NER Entity Anonymization model. It shows a plau-sible sequence of actions for a character.
written prompt (but do not see the prompt). Eval-uators are asked to mark which story they prefer.100 stories are evaluated for each model by 3 dif-ferent judges. To reduce variance, stories from allmodels are trimmed to 200 words.
Figure 6 shows that human evaluators prefer ournovel decompositions over a carefully tuned Fu-sion model from Fan et al. (2018) by about 60%in a blind comparison. We see additive gains frommodelling actions and entities. In a second study,evaluators compared various baselines against sto-ries generated by our strongest model, which usesSRL-based action plans and coreference-based en-tity anonymization. In all cases, our full decompo-sition is preferred.
6.2 Effect of SRL Decomposition
Human-written stories feature a wide varietyof events, while neural models are plagued bygeneric generations and repetition. Table 2 quan-tifies model performance on two metrics to assessaction diversity: (1) the number of unique verbsgenerated, averaged across all stories (2) the per-centage of diverse verbs, measured by the percentof all verbs generated in the test set that are notone of the top 5 most frequent verbs. A higherpercentage indicates more diverse events.10
Our decomposition using the SRL predicate-argument structure improves the model’s abilityto generate diverse verbs. Adding verb attentionleads to further improvement. Qualitatively, themodel can often outline clear action sequences, asshown in Figure 7. However, all models remainfar from matching the diversity of human stories.
10We identify verbs using Spacy: https://spacy.io/
Table 2: Action Generation. Generating the SRL struc-ture improves verb diversity and reduces repetition.
6.3 Comparing Entity Reference Models
We explored a variety of different ways to generatethe full text of abstracted entities—using differentamounts of context and different granularities ofsubword generation. To compare these models,we calculated their accuracy at predicting the cor-rect reference in Table 3. Each model evaluatesn = 10, 50, 100 different entities in the test set, 1real and n−1 randomly sampled distractors. Mod-els must give the true mention the highest likeli-hood. We analyze accuracy on the first mentionof an entity, an assessment of novelty, and subse-quent references, an assessment of consistency.
Effect of Sub-word Modelling Table 3 showsthat modelling a character-level vocabulary for en-tity generation outperforms BPE and word-basedmodels, because of the diversity of entity names.This result highlights a key advantage of multi-stage modelling: the usage of specialized mod-elling techniques for each sub-task.
Effect of Additional Context Entity referencesshould be contextual. Firstly, names must beappropriate for the story setting—Bilbo Bagginsmight be more appropriate for a fantasy novel.Subsequent references to the character may bebriefer, depending on context—for example, he ismore likely to be referred to as he or Bilbo than hisfull name in the next sentence.
We compare three models ability to fillentities based on context (using coreference-anonymization): a model that does not receive thestory, a model that uses only leftward context (asin Clark et al. (2018)), and a model with accessto the full story. We show in Table 3 that havingaccess to the full story provides the best perfor-mance. Having no access to any of the story de-creases ranking accuracy, even though the modelstill receives the local context window of the entityas input. The left story context model performs
Figure 8: Generating entity references for differentgenres, using entity-anonymized human written stories.Models use the story context to fill in relevant entities.Color indicates coreferent clusters.
better, but looking at the complete story providesadditional gains. We note that full-story contextcan only be provided in a multi-stage generationapproach.
Qualitative Examples Figure 8 shows exam-ples of entity naming in three stories of differentgenres. We evaluate different genres to examineif generated entities adapt to the style of the story.We show that models can adapt to the context—forexample generating The princess and The Queenwhen the context includes monarchy.
6.4 Effect of Entity AnonymizationTo understand the effectiveness of the entity gen-eration models, we examine their performance byanalyzing generation diversity.
Diversity of Entity Names Human-written sto-ries often contain many diverse, novel names forpeople and places. However, these tokens arerare and subsequently difficult for standard neuralmodels to generate. Table 4 shows that the fusionmodel and baseline decomposition strategies gen-erate very few unique entities in each story. Gener-ated entities are often generic names such as John.
Our proposed decompositions generate substan-tially more unique entities than strong baselines.
Table 3: Accuracy at choosing the correct reference string for a mention, discriminating against 10, 50 and 100random distractors. We break out results for the first mention of an entity (requiring novelty to produce an appro-priate name in the context) and subsequent references (typically pronouns, nominal references, or shorter forms ofnames). We compare the effect of sub-word modelling and providing longer contexts.
Model # Unique EntitiesHuman Stories 2.99Fusion 0.47Summary 0.67Keyword 0.81Compression 0.21SRL + NER Entity Anonymization 2.16SRL + Coreference Anonymization 1.59
Table 4: Diversity of entity names. Baseline modelsgenerate few unique entities per story. Our decomposi-tions generate more, but still fewer than human stories.Using coreference resolution to build entity clusters re-duces diversity here—partly due to re-using existingnames more, and partly due to greater use of pronouns.
Table 5: Analysis of non-singleton coreference clus-ters. Baseline models generate very few different coref-erence chains, and repetitive mentions within clusters.Our models generate larger and more diverse clusters.
Interestingly, we found that using coreferenceresolution for entity anonymization led to fewerunique entity names than generating the names in-dependently. This result can be explained by thecoreference-based model re-using previous namesmore frequently, as well as using more pronouns.
Coherence of Entity Clusters Well structuredstories will refer back to previously mentionedcharacters and events in a consistent manner. Toevaluate if the generated stories have these charac-teristics, we examine the coreference properties inTable 5. We quantify the average number of coref-
erence clusters and the diversity of entities withineach cluster (e.g. the cluster Bilbo, he, the hobbitis more diverse than the cluster he, he, he).
Our full model produces more non-singletoncoreference chains, suggesting greater coherence,and also gives different mentions of the same en-tity more diverse names. However, both numbersare still lower than for human generated stories,indicating potential for future work.
Qualitative Example Figure 9 displays a sen-tence constructed to require the generation of anentity as the final word. The fusion model doesnot perform any implicit coreference to associatethe allergy with his dog. In contrast, coreferenceentity fill produces a high quality completion.
7 Related Work
Decomposing natural language generation intoseveral steps has been extensively explored (Reiterand Dale, 2000; Gatt and Krahmer, 2018). In clas-sical approaches to text generation, various stageswere used to produce final written text. For ex-ample, algorithms were developed to determinecontent and discourse at an abstract level, thensentence aggregation and lexicalization, and fi-nally steps to resolve referring expressions (Hovy,1990; Dalianis and Hovy, 1993; Wahlster et al.,1993; Ratnaparkhi, 2000; Malouf, 2000). Ourwork builds upon these approaches.
Story Generation with Planning Story genera-tion using a plan has been explored using manydifferent techniques. Traditional approaches or-ganized sequences of character actions with handcrafted models (Riedl and Young, 2010; Porte-ous and Cavazza, 2009). Recent work extendedthis to modelling story events (Martin et al., 2017;Mostafazadeh et al., 2016), plot graphs (Li et al.,2013), plot summaries (Appling and Riedl, 2009),
Figure 9: Constructed sentence where the last wordrefers to an entity. The coreference model is able totrack the entities, whereas the fusion model relies heav-ily on local context to generate the next words.
story fragments or vignettes (Riedl, 2010), or usedsequences of images (Huang et al., 2016) or de-scriptions (Jain et al., 2017).
We build on previous work that decomposesgeneration. Xu et al. (2018) learn a skeleton ex-traction model and a generative model conditionedupon the skeleton, using reinforcement learning totrain jointly. Zhou et al. (2018) train a storyline ex-traction model for news articles, but require super-vision from manually annotated storylines. Yaoet al. (2019) use RAKE (Rose et al., 2010) to ex-tract storylines, and condition upon the storyline towrite the story using dynamic and static schemasthat govern if the storyline can change.
Entity Language Models An outstanding chal-lenge in text generation is modelling and trackingentities. Centering (Grosz et al., 1995) gives a the-oretical account of how referring expressions forentities are chosen in discourse context. Namedentity recognition has been incorporated into lan-guage models since at least Gotoh et al. (1999),and can improve domain adaptation (Liu and Liu,2007). Language models have been extended tomodel entities based on information such as entitytype (Parvez et al., 2018). Recent work has in-corporated learning representations of entities andother unknown words (Kobayashi et al., 2017), aswell as explicitly model entities by dynamicallyupdating these representations to track changesover time and context (Ji et al., 2017). Dynamicupdates to entity representations are used in otherstory generation models (Clark et al., 2018).
Non-Autoregressive Generation Our methodproposes decomposing left-to-right generationinto multiple steps. Recent work has explorednon-autoregressive generation for more efficientlanguage modeling and machine translation. Fordet al. (2018) developed two-pass language models,generating templates then filling in words. Thepartially filled templates could be seen as an in-
termediary representation similar to generating acompressed story. Other models allow arbitraryorder generation using insertion operations (Guet al., 2019; Stern et al., 2019) and Gu et al. (2017)explored parallel decoding for machine transla-tion. In contrast, we focus on decomposing gen-eration to focus on planning, rather than efficientdecoding at inference time.
8 Conclusion
We proposed an effective method for writing shortstories by separating the generation of actions andentities. We show through human evaluation andautomated metrics that our novel decompositionimproves story quality.
ReferencesD Scott Appling and Mark O Riedl. 2009. Representa-
tions for learning to summarize plots.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proc. of ICLR.
Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018.Neural text generation in stories using entity repre-sentations as context. In Proceedings of the 2018Conference of the North American Chapter of theAssociation for Computational Linguistics: HumanLanguage Technologies, Volume 1 (Long Papers),volume 1, pages 2250–2260.
Zihang Dai, Zhilin Yang, Yiming Yang, William WCohen, Jaime Carbonell, Quoc V Le, and RuslanSalakhutdinov. 2018. Transformer-xl: Languagemodeling with longer-term dependency.
Hercules Dalianis and Eduard Hovy. 1993. Aggrega-tion in natural language generation. In EuropeanWorkshop on Trends in Natural Language Genera-tion, pages 88–105. Springer.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi-erarchical neural story generation. In Proceedingsof the 56th Annual Meeting of the Association forComputational Linguistics.
Katja Filippova and Yasemin Altun. 2013. Overcom-ing the lack of parallel data in sentence compression.In Proceedings of the 2013 Conference on EmpiricalMethods in Natural Language Processing.
Nicolas Ford, Daniel Duckworth, MohammadNorouzi, and George E Dahl. 2018. The importanceof generation order in language modeling. arXivpreprint arXiv:1808.07910.
Albert Gatt and Emiel Krahmer. 2018. Survey of thestate of the art in natural language generation: Coretasks, applications and evaluation. Journal of Artifi-cial Intelligence Research, 61:65–170.
Jonas Gehring, Michael Auli, David Grangier, DenisYarats, and Yann N Dauphin. 2017. ConvolutionalSequence to Sequence Learning. In Proc. of ICML.
Yoshihiko Gotoh, Steve Renals, and Gethin Williams.1999. Named entity tagged language models. InAcoustics, Speech, and Signal Processing, 1999.Proceedings., 1999 IEEE International Conferenceon, volume 1, pages 513–516. IEEE.
Barbara J Grosz, Scott Weinstein, and Aravind K Joshi.1995. Centering: A framework for modeling the lo-cal coherence of discourse. Computational linguis-tics, 21(2):203–225.
Jiatao Gu, James Bradbury, Caiming Xiong, Vic-tor OK Li, and Richard Socher. 2017. Non-autoregressive neural machine translation. arXivpreprint arXiv:1711.02281.
Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019.Insertion-based decoding with automaticallyinferred generation order. arXiv preprintarXiv:1902.01370.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle-moyer. 2017. Deep semantic role labeling: Whatworks and whats next. In Proceedings of the An-nual Meeting of the Association for ComputationalLinguistics.
Karl Moritz Hermann, Tomas Kocisky, EdwardGrefenstette, Lasse Espeholt, Will Kay, Mustafa Su-leyman, and Phil Blunsom. 2015. Teaching ma-chines to read and comprehend. In Advances inneural information processing systems, pages 1693–1701.
Eduard H Hovy. 1990. Pragmatics and natural lan-guage generation. Artificial Intelligence, 43(2):153–197.
Ting-Hao Kenneth Huang, Francis Ferraro, NasrinMostafazadeh, Ishan Misra, Aishwarya Agrawal, Ja-cob Devlin, Ross Girshick, Xiaodong He, PushmeetKohli, Dhruv Batra, et al. 2016. Visual storytelling.In Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 1233–1239.
Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mo-hak Sukhwani, Anirban Laha, and Karthik Sankara-narayanan. 2017. Story generation from sequenceof independent short descriptions. arXiv preprintarXiv:1707.05501.
Yangfeng Ji, Chenhao Tan, Sebastian Martschat, YejinChoi, and Noah A Smith. 2017. Dynamic entityrepresentations in neural language models. arXivpreprint arXiv:1708.00781.
Sosuke Kobayashi, Naoaki Okazaki, and KentaroInui. 2017. A neural language model for dy-namically representing the meanings of unknownwords and entities in a discourse. arXiv preprintarXiv:1709.01679.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, Volume 2 (Short Papers), vol-ume 2, pages 687–692.
Boyang Li, Stephen Lee-Urban, George Johnston, andMark Riedl. 2013. Story generation with crowd-sourced plot graphs.
Feifan Liu and Yang Liu. 2007. Unsupervised lan-guage model adaptation incorporating named entityinformation. In Proceedings of the 45th AnnualMeeting of the Association of Computational Lin-guistics, pages 672–679.
Peter J Liu, Mohammad Saleh, Etienne Pot, BenGoodrich, Ryan Sepassi, Lukasz Kaiser, andNoam Shazeer. 2018. Generating wikipedia bysummarizing long sequences. arXiv preprintarXiv:1801.10198.
Robert Malouf. 2000. The order of prenominal adjec-tives in natural language generation. In Proceedingsof the 38th Annual Meeting of the Association forComputational Linguistics.
Lara J Martin, Prithviraj Ammanabrolu, William Han-cock, Shruti Singh, Brent Harrison, and Mark ORiedl. 2017. Event representations for automatedstory generation with deep neural nets. arXivpreprint arXiv:1706.01331.
Nasrin Mostafazadeh, Alyson Grealish, NathanaelChambers, James Allen, and Lucy Vanderwende.2016. Caters: Causal and temporal relation schemefor semantic annotation of event structures. In Pro-ceedings of the Fourth Workshop on Events, pages51–61.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre,Bing Xiang, et al. 2016. Abstractive text summa-rization using sequence-to-sequence rnns and be-yond. arXiv preprint arXiv:1602.06023.
Myle Ott, Sergey Edunov, Alexei Baevski, AngelaFan, Sam Gross, Nathan Ng, David Grangier, andMichael Auli. 2019. fairseq: A fast, extensibletoolkit for sequence modeling. In Proceedings ofNAACL-HLT 2019: Demonstrations.
Md Rizwan Parvez, Saikat Chakraborty, BaishakhiRay, and Kai-Wei Chang. 2018. Building languagemodels for text with named entities. arXiv preprintarXiv:1805.04836.
Julie Porteous and Marc Cavazza. 2009. Controllingnarrative generation with planning trajectories: therole of constraints. In Joint International Confer-ence on Interactive Digital Storytelling, pages 234–245. Springer.
Adwait Ratnaparkhi. 2000. Trainable methods forsurface natural language generation. In Proceed-ings of the 1st North American chapter of the As-sociation for Computational Linguistics conference,pages 194–201. Association for Computational Lin-guistics.
Ehud Reiter and Robert Dale. 2000. Building naturallanguage generation systems. Cambridge universitypress.
Mark O Riedl. 2010. Story planning: Creativitythrough exploration, retrieval, and analogical trans-formation. Minds and Machines, 20(4):589–614.
Mark O Riedl and Robert Michael Young. 2010. Nar-rative planning: Balancing plot and character. Jour-nal of Artificial Intelligence Research, 39:217–268.
Stuart Rose, Dave Engel, Nick Cramer, and WendyCowley. 2010. Automatic keyword extraction fromindividual documents. Text Mining: Applicationsand Theory.
Abigail See, Peter J Liu, and Christopher D Man-ning. 2017. Get to the point: Summarizationwith pointer-generator networks. arXiv preprintarXiv:1704.04368.
Rico Sennrich, Barry Haddow, and Alexandra Birch.2015. Neural machine translation of rare words withsubword units. arXiv preprint arXiv:1508.07909.
Mitchell Stern, William Chan, Jamie Kiros, and JakobUszkoreit. 2019. Insertion transformer: Flexible se-quence generation via insertion operations. arXivpreprint arXiv:1902.03249.
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen,and Xiaodong Shi. 2018. Deep semantic role label-ing with self-attention. In AAAI Conference on Arti-ficial Intelligence.
Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention Is AllYou Need. In Proc. of NIPS.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.2015. Pointer networks. In Advances in Neural In-formation Processing Systems, pages 2692–2700.
Wolfgang Wahlster, Elisabeth Andre, Wolfgang Fin-kler, Hans-Jurgen Profitlich, and Thomas Rist.1993. Plan-based integration of natural languageand graphics generation. Artificial intelligence,63(1-2):387–427.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin,and Michael Auli. 2019. Pay less attention withlightweight and dynamic convolutions. In Interna-tional Conference on Learning Representations.
Jingjing Xu, Yi Zhang, Qi Zeng, Xuancheng Ren,Xiaoyan Cai, and Xu Sun. 2018. A skeleton-based model for promoting coherence among sen-tences in narrative story generation. arXiv preprintarXiv:1808.06945.
Lili Yao, Nanyun Peng, Ralph Weischedel, KevinKnight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. InAssociation for the Advancement of Artificial Intel-ligence.
Deyu Zhou, Linsen Guo, and Yulan He. 2018. Neuralstoryline extraction model for storyline generationfrom news articles. In Proceedings of the 2018 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), vol-ume 1, pages 1727–1736.
