A Hierarchical Encoder-Decoder for ParagraphSummarization

Farzaneh Mahdisoltani

Department of Computer ScienceUniversity of Toronto

CSC2541, Fall 2016

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 1 / 8

Paragraph Summarization

Summarization Phenomena:¯

Deletion

Paraphrase

Generalization

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 2 / 8

Related Work

Extractive Summarization: Stitching words and phrases from thesource

Dorr et al. (2003), Cohn et al. (2008), Woodsend et al. (2010)

Abstractive Summarization: Generating richer summaries based onunderstanding the input

Attention based: Rush et al. (2015)Statistical machine translation for headline generation: Banko et.al.(2000)RNN with attention: Chopra et al.(2016)Encoder-Decoder: Hu et al.(2015)

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 3 / 8

Hierarchichal Encoder-Decoder Model for Summarization

S1

S2

S3

S4

S5S6

Sentence to Vector enc(P)

Paragraph

enc(S1)

enc(S2)

enc(S3)

enc(S4)

enc(S5)enc(S6)

Paragraph to Vector

Sentence Level Decoder

enc(O1)

enc(O2)

enc(O3)

Word Level Decoder

O1

O2

O3

Summary Abstract Concepts

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 4 / 8

Hierarchichal Encoder-Decoder Model for Summarization

Objective Function:∑

t P(otw |o<tw , os , hp)

Encoder Decoder

WordLevel

SentenceLevel

h1i h2

i h3i h4

i

si

wi1 wi

2 wi3 wi

4

h1i h1

j h2j h3

j h4j h5

j

vj1 vj

2 vj3 vj

4 vj5

yj

h1s h2

s start

d1s

o1s

d1w d2w d3w

o1w o2w o3d

o1w o2w

h1i h1

j h2j h3

j h4j h5

j

wj1 wj

2 wj3 wj

4 wj5

sj

p

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 5 / 8

Selecting the Correct Summary from Candidates

S1S2S3S4S5S6

Sentence to Vector enc(P)

Paragraph

enc(S1)enc(S2)enc(S3)enc(S4)enc(S5)enc(S6)

Paragraph to Vector

enc(O1)enc(O2)enc(O3)

O1O2O3

Sentence to Vector enc(S1)

Paragraph to Vector

enc(O1)enc(O2)enc(O3)

O1O2O3

Sentence to Vector enc(S2)

Paragraph to Vector

enc(O1)enc(O2)enc(O3)

O1O2O3

Sentence to Vector enc(S3)

Paragraph to Vector

enc(O1)enc(O2)enc(O3)

O1O2O3

Sentence to Vector enc(S4)

Paragraph to Vector

Candidates

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 6 / 8

Preliminary Results

Paragraph Representation Method Accuracy

Random 25%Word2Vec 65%Skip-Thought Vectors 62%Bag of Words 67%RNN Encoder 57%Hierarchichal Encoder 54%

Table: Training the encoder for multiple choice question

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 7 / 8

Thank You

Farzaneh Mahdisoltani (University of Toronto)A Hierarchical Encoder-Decoder for Paragraph SummarizationCSC2541, Fall 2016 8 / 8