Neural Text Summarization - Piji Lilipiji.com/slides/neuralsum.pdf · 2021. 2. 26. · Vietnamese...

transcript

Neural Text Summarization

Piji Li

NLP Center, Tencent AI Labpijili@tencent.com

Paper Reading, Sep.6, 2018

Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 1 / 63

Table of Contents

1 Introduction

2 Methods

3 Conclusion

Table of Contents

1 Introduction

2 Methods

3 Conclusion

IntroductionText Summarization

The goal of automatic text summarization is to automatically producea succinct summary, preserving the most important information for asingle document or a set of documents about the same topic (event).

7/11/2017 mogren.one/graphics/illustrations/mogren_summarization.svg

http://mogren.one/graphics/illustrations/mogren_summarization.svg 1/1

IntroductionText Summarization - Categories

Input:

Single-Document Summarization (SDS)Multi-Document Summarization (MDS)

IntroductionSingle-Document Summarization

Cambodian leader Hun Sen on Friday rejected opposition parties 'demands for talks outside the country , accusing them of trying to ``internationalize '' the political crisis .Government and opposition parties have asked King NorodomSihanouk to host a summit meeting after a series of post-electionnegotiations between the two opposition groups and Hun Sen 's party toform a new government failed .Opposition leaders Prince Norodom Ranariddh and Sam Rainsy , citingHun Sen 's threats to arrest opposition figures after two alleged attemptson his life , said they could not negotiate freely in Cambodia and calledfor talks at Sihanouk 's residence in Beijing .Hun Sen , however ,rejected that .``I would like to make it clear that all meetings related to Cambodianaffairs must be conducted in the Kingdom of Cambodia , '' Hun Sentold reporters after a Cabinet meeting on Friday .`` No-one shouldinternationalize Cambodian affairs .It is detrimental to the sovereignty of Cambodia , '' he said .Hun Sen 'sCambodian People 's Party won 64 of the 122 parliamentary seats inJuly 's elections , short of the two-thirds majority needed to form agovernment on its own .Ranariddh and Sam Rainsy have charged thatHun Sen 's victory in the elections was achieved through widespreadfraud .They have demanded a thorough investigation into their electioncomplaints as a precondition for their cooperation in getting thenational assembly moving and a new government formed …….

Cambodian government rejects opposition's call for talks abroad

Document

Summary

Figure 1: Single-document summarization.

IntroductionMulti-Document Summarization

Fingerprints and photos of two men who boarded the doomed Malaysia Airlines passenger jet are

being sent to U.S. authorities so they can be compared against records of known terrorists and

criminals. The cause of the plane's disappearance has baffled investigators and they have not said

that they believed that terrorism was involved, but they are also not ruling anything out. The

investigation into the disappearance of the jetliner with 239 passengers and crew has centered so

far around the fact that two passengers used passports stolen in Thailand from an Austrian and an

Italian. The plane which left Kuala Lumpur, Malaysia, was headed for Beijing. Three of the

passengers, one adult and two children, were American. ……

(CNN) -- A delegation of painters and calligraphers, a group of Buddhists returning from a

religious gathering in Kuala Lumpur, a three-generation family, nine senior travelers and five

toddlers. Most of the 227 passengers on board missing Malaysia Airlines Flight 370 were Chinese,

according to the airline's flight manifest. The 12 missing crew members on the flight that

disappeared early Saturday were Malaysian. The airline's list showed the passengers hailed from 14

countries, but later it was learned that two people named on the manifest -- an Austrian and an

Italian -- whose passports had been stolen were not aboard the plane. The plane was carrying five

children under 5 years old, the airline said. ……

Vietnamese aircraft spotted what they suspected was one of the doors belonging to the ill-fated

Malaysia Airlines Flight MH370 on Sunday, as troubling questions emerged about how two

passengers managed to board the Boeing 777 using stolen passports. The discovery comes as

officials consider the possibility that the plane disintegrated mid-flight, a senior source told Reuters.

The state-run Thanh Nien newspaper cited Lt. Gen. Vo Van Tuan, deputy chief of staff of Vietnam's

army, as saying searchers in a low-flying plane had spotted an object suspected of being a door

from the missing jet. It was found in waters about 56 miles south of Tho Chu island, in the same

area where oil slicks were spotted Saturday. ……

Flight MH370, carrying 239

people vanished over the

South China Sea in less than

an hour after taking off from

Kuala Lumpur, with two

passengers boarded the

Boeing 777 using stolen

passports. Possible reasons

could be an abrupt breakup of

the plane or an act of

terrorism. The government

was determining the "true

identities" of the passengers

who used the stolen passports.

Investigators were trying to

determine the path of the

plane by analysing civilian

and military radar data while

ships and aircraft from seven

countries scouring the seas

around Malaysia and south of

Vietnam.

Documents Summary

Figure 2: Multi-document summarization for the topic “Malaysia AirlinesDisappearance”.

IntroductionText Summarization - Categories

Input:

Single-Document Summarization (SDS)Multi-Document Summarization (MDS)

Output:

ExtractiveCompressiveAbstractive

Machine learning methods:

SupervisedUnsupervised

IntroductionText Summarization - History

Since 1950s:

Concept Weight (Luhn, 1958), Centroid (Radev et al., 2004), LexRank(Erkan and Radev, 2004), TextRank (Mihalcea and Tarau, 2004),Sparse Coding (He et al., 2012; Li et al., 2015)Feature+Regression (Min et al., 2012; Wang et al., 2013)

Most of the summarization methods are extractive.

Abstractive summarization is full of challenges. Some indirectmethods employ sentence fusing (Barzilay and McKeown, 2005) orphrase merging (Bing et al., 2015).

The indirect strategies will do harm to the linguistic quality of theconstructed sentences.

IntroductionText Summarization - History

Before the neural summarization era...silent

2015 (Rush et al., 2015)

Table of Contents

1 Introduction

2 Methods

3 Conclusion

MethodsEssential Idea

Salience Detection (Words, Sentences)

MethodsInspiration from DBN, DNN, CNN

Liu, Yan, Sheng-hua Zhong, andWenjie Li. “Query-OrientedMulti-Document Summarizationvia Unsupervised Deep Learn-ing.” In AAAI. 2012.

Denil, Misha, Alban Demiraj, NalKalchbrenner, Phil Blunsom, andNando de Freitas. “Modelling,visualising and summarising doc-uments with a single convolu-tional neural network.” arXivpreprint arXiv:1406.3830 (2014).

Figure 3: Visualization of Parameters.Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 13 / 63

MethodsBetter Semantic Representations

Since 1950s:

Concept Weight (Luhn, 1958), Centroid (Radev et al., 2004), LexRank(Erkan and Radev, 2004), TextRank (Mihalcea and Tarau, 2004),Sparse Coding (He et al., 2012; Li et al., 2015)

Bag-of-Words (BoWs)

MethodsBetter Semantic Representations

Word2vec (Mikolov et al., 2013), Paragraph Vector (Le and Mikolov,2014), RNN-Sent (Tang et al., 2015), CNN-Sent (Kim, 2014)

Improve the performance of PageRank and Data Reconstructionbased models.

Works:

Kageback, Mikael, Olof Mogren, Nina Tahmasebi, and Devdatt Dub-hashi. “Extractive summarization using continuous vector spacemodels.” In CVSC 2014.Yin, Wenpeng, and Yulong Pei. ”Optimizing Sentence Modeling andSelection for Document Summarization.” In IJCAI 2015.Li, Piji, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. ”Cascadedattention based unsupervised information distillation for compres-sive summarization.” In EMNLP 2017.

MethodsInspiration from NMT

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. ”Neuralmachine translation by jointly learning to align and translate.”arXiv preprint arXiv:1409.0473 (2014). (citation:4300+)

Figure 4: Attention-based seq2seq framework. Figure from OpenNMT (Kleinet al., 2017)

Methods

MethodsA Neural Attention Model for Abstractive Sentence Summarization

Rush, Alexander M., Sumit Chopra, and Jason Weston. ”A neuralattention model for abstractive sentence summarization.”EMNLP (2015). (citation:570+)

Figure 5: (a) NNLM decoder with additional encoder element. (b) Attentionbased encoder.

MethodsA Neural Attention Model for Abstractive Sentence Summarization

Rush, Alexander M., Sumit Chopra, and Jason Weston. ”A neuralattention model for abstractive sentence summarization.”EMNLP (2015). (citation:570+)

MethodsLCSTS: A Large Scale Chinese Short Text Summarization Dataset

Hu, Baotian, Qingcai Chen, and Fangze Zhu. ”LCSTS: A LargeScale Chinese Short Text Summarization Dataset.” EMNLP(2015). (citation:49)

（a）（b）

Figure 6: (a) Encoder-Decoder. (b) Attention based Decoder.

MethodsLCSTS: A Large Scale Chinese Short Text Summarization Dataset

Hu, Baotian, Qingcai Chen, and Fangze Zhu. ”LCSTS: A LargeScale Chinese Short Text Summarization Dataset.” EMNLP(2015). (citation:49)

MethodsGenerating News Headlines with Recurrent Neural Networks

Lopyrev, Konstantin. ”Generating news headlines with recurrentneural networks.” arXiv preprint arXiv:1512.01712 (2015).(citation:28)

Investigations of several NMT models.

Methods

MethodsAbstractive sentence summarization with attentive recurrent neural networks

Chopra, Sumit, Michael Auli, and Alexander M. Rush. ”Abstractivesentence summarization with attentive recurrent neuralnetworks.” NAACL, pp. 93-98. 2016. (citation:138)

MethodsAbstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Nallapati, Ramesh, Bowen Zhou, Cicero dos Santos, Ca glar Gulcehre,and Bing Xiang. ”Abstractive Text Summarization usingSequence-to-sequence RNNs and Beyond.” CoNLL 2016 (2016):280. (citation:183)

3 pages version in Feb. 2016.

Many tricks (nmt, copy, coverage, hierarchical, external knowledge).

MethodsWhy Copy?

Extraction

MethodsCopy Mechanism

Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. ”Pointer networks.”In NIPS, pp. 2692-2700. 2015. (citation:352)

Gulcehre, Caglar, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, andYoshua Bengio. ”Pointing the Unknown Words.” In ACL, vol. 1,pp. 140-149. 2016. (citation:126)

Gu, Jiatao, Zhengdong Lu, Hang Li, and Victor OK Li.”Incorporating Copying Mechanism in Sequence-to-SequenceLearning.” In ACL, vol. 1, pp. 1631-1640. 2016. (citation:192)

MethodsCopy Mechanism

Figure 7: Pointer-generator model. (See et al., 2017)

MethodsCopy Mechanism – Performance

MethodsWhy Coverage?

Diversity

MethodsCoverage Mechanism

Tu, Zhaopeng, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. ”ModelingCoverage for Neural Machine Translation.” In ACL 2016. (citation:187)

Chen, Qian, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang.”Distraction-based neural networks for modeling documents.” InIJCAI 2016. (citation:28)

MethodsCoverage Mechanism

Figure 8: Operation of coverage mechanism.

MethodsCoverage Mechanism – Performance

MethodsMore Works in 20161

Cheng, Jianpeng, and Mirella Lapata. ”Neural Summarization byExtracting Sentences and Words.” In ACL, 2016. (citation:108)

Cao, Ziqiang, Wenjie Li, Sujian Li, Furu Wei, and Yanran Li.”AttSum: Joint Learning of Focusing and Summarization with NeuralAttention.” In COLING, 2016.

Zeng, Wenyuan, Wenjie Luo, Sanja Fidler, and Raquel Urtasun.”Efficient summarization with read-again and copy mechanism.”arXiv preprint arXiv:1611.03382 (2016).

Miao, Yishu, and Phil Blunsom. ”Language as a Latent Variable:Discrete Generative Models for Sentence Compression.” In EMNLP.2016.

1https://github.com/lipiji/App-DL#text-summarizationPiji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 38 / 63

Methods

Inspirations from the traditional summarization methods.

Methods

Nallapati, Ramesh, Feifei Zhai, and Bowen Zhou. ”SummaRuNNer:A Recurrent Neural Network Based Sequence Model forExtractive Summarization of Documents.” In AAAI, pp.3075-3081. 2017. (citation:58)

MethodsAbstractive document summarization with a graph-based attentional neural model

Tan, Jiwei, Xiaojun Wan, and Jianguo Xiao. ”Abstractivedocument summarization with a graph-based attentional neuralmodel.” In ACL 2017. (citation:24)

ACL Outstanding Paper.

MethodsAbstractive document summarization with a graph-based attentional neural model

Tan, Jiwei, Xiaojun Wan, and Jianguo Xiao. ”Abstractivedocument summarization with a graph-based attentional neuralmodel.” In ACL 2017. (citation:24)

MethodsSelective Encoding for Abstractive Sentence Summarization

Zhou, Qingyu, Nan Yang, Furu Wei, and Ming Zhou. ”SelectiveEncoding for Abstractive Sentence Summarization.” In ACL2017. (citation:24)

Methods

Recall the Copy and Coverage Mechanism in 2016.

MethodsSelective Encoding for Abstractive Sentence Summarization

See, Abigail, Peter J. Liu, and Christopher D. Manning. ”Get ToThe Point: Summarization with Pointer-Generator Networks.”In ACL 2017. (citation:114)Writing? Figures?

Figure 9: Pointer-Generator Networks.Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 46 / 63

Methods

Reinforcement Learning.

MethodsA deep reinforced model for abstractive summarization

Paulus, Romain, Caiming Xiong, and Richard Socher. ”A deepreinforced model for abstractive summarization.” arXiv preprintarXiv:1705.04304 (2017). (citation:107)

Intra-attention modeling.

Reinforced learning trick.

Paulus, Romain, Caiming Xiong, and Richard Socher. ”A deepreinforced model for abstractive summarization.” arXiv preprintarXiv:1705.04304 (2017). (citation:107)

Methods

MethodsReinforcement Learning based Methods

Celikyilmaz, Asli, Antoine Bosselut, Xiaodong He, and Yejin Choi.”Deep Communicating Agents for Abstractive Summarization.”In NAACL 2018.

Wu, Yuxiang, and Baotian Hu. ”Learning to Extract CoherentSummary via Deep Reinforcement Learning.” In AAAI 2018.

Wang, Li, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du.”A reinforced topic-aware convolutional sequence-to-sequencemodel for abstractive text summarization.” In IJCAI 2018.

Chen, Yen-Chun, and Mohit Bansal. ”Fast AbstractiveSummarization with Reinforce-Selected Sentence Rewriting.”arXiv preprint arXiv:1805.11080 (2018).

Keneshloo, Yaser, Tian Shi, Chandan K. Reddy, and NarenRamakrishnan. ”Deep Reinforcement Learning For Sequence toSequence Models.” arXiv preprint arXiv:1805.09461 (2018).

MethodsCNN-seq2seq, Transformer

Wang, Li, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du.”A reinforced topic-aware convolutional sequence-to-sequencemodel for abstractive text summarization.” In IJCAI 2018.

MethodsRecent Works

Wojciech Kryscinski, Romain Paulus, Caiming Xiong, Richard Socher.”Improving Abstraction in Text Summarization .” arXiv preprintarXiv:1808.07913 (2018).

Zhang, Xingxing, Mirella Lapata, Furu Wei, and Ming Zhou. ”NeuralLatent Extractive Document Summarization.” arXiv preprintarXiv:1808.07187 (2018).

Sebastian Gehrmann, Yuntian Deng, Alexander M. Rush. ”Bottom-UpAbstractive Summarization.” arXiv preprint arXiv:1808.10792 (2018).

Methods

https://github.com/lipiji/App-DL#text-summarization

Table of Contents

1 Introduction

2 Methods

3 Conclusion

Conclusion

Challenges:

Long text abstractive summarization.Abstractive multi-document summarization.

Thanks a lot!Q & A

References I

Regina Barzilay and Kathleen R McKeown. Sentence fusion for multi-document news summarization. Computational Linguistics, 31(3):297–328, 2005.

Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passon-neau. Abstractive multi-document summarization via phrase selection andmerging. In Proceedings of the 53rd Annual Meeting of the Associationfor Computational Linguistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 1: Long Papers), volume 1,pages 1587–1597, 2015.

Gunes Erkan and Dragomir R Radev. Lexrank: Graph-based lexical cen-trality as salience in text summarization. Journal of Artificial IntelligenceResearch, 22:457–479, 2004.

References II

Zhanying He, Chun Chen, Jiajun Bu, Can Wang, Lijun Zhang, Deng Cai,and Xiaofei He. Document summarization based on data reconstruction.In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intel-ligence, pages 620–626. AAAI Press, 2012.

Yoon Kim. Convolutional neural networks for sentence classification. InProceedings of the 2014 Conference on Empirical Methods in NaturalLanguage Processing, pages 1746–1751, 2014.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander MRush. Opennmt: Open-source toolkit for neural machine translation.arXiv preprint arXiv:1701.02810, 2017.

Quoc Le and Tomas Mikolov. Distributed representations of sentences anddocuments. In International Conference on Machine Learning, pages1188–1196, 2014.

References III

Piji Li, Lidong Bing, Wai Lam, Hang Li, and Yi Liao. Reader-aware multi-document summarization via sparse coding. In The 24th InternationalJoint Conference on Artificial Intelligence, pages 1270–1276, 2015.

Hans Peter Luhn. The automatic creation of literature abstracts. IBMJournal of research and development, 2(2):159–165, 1958.

Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Pro-ceedings of the 2004 conference on empirical methods in natural languageprocessing, 2004.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficientestimation of word representations in vector space. arXiv preprintarXiv:1301.3781, 2013.

Ziheng Lin Min, Yen Kan Chew, and Lim Tan. Exploiting category-specificinformation for multi-document summarization. The 21th InternationalConference on Computational Linguistics (COLING), pages 2903–2108,2012.

References IV

Dragomir R Radev, Hongyan Jing, Ma lgorzata Stys, and Daniel Tam.Centroid-based summarization of multiple documents. Information Pro-cessing & Management, 40(6):919–938, 2004.

Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attentionmodel for abstractive sentence summarization. In Proceedings of the 2015Conference on Empirical Methods in Natural Language Processing, pages379–389, 2015.

Abigail See, Peter J Liu, and Christopher D Manning. Get to the point:Summarization with pointer-generator networks. In Proceedings of the55th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), volume 1, pages 1073–1083, 2017.

Duyu Tang, Bing Qin, and Ting Liu. Document modeling with gated re-current neural network for sentiment classification. In Proceedings of the2015 conference on empirical methods in natural language processing,pages 1422–1432, 2015.

References V

Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and ClaireCardie. A sentence compression based framework to query-focused multi-document summarization. In Proceedings of the 51st Annual Meeting ofthe Association for Computational Linguistics (Volume 1: Long Papers),volume 1, pages 1384–1394, 2013.