A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

A Hierarchical Model of Reviews forAspect-based Sentiment Analysis

Sebastian Ruder

PhD Candidate, Social Semantics Unit, Insight Centre, NUIGResearch Scientist, Aylien Ltd., Dublin

24.08.16

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Agenda

1 Introduction

2 A brief history of Aspect-based Sentiment Analysis

3 Task description

4 Data

5 State-of-the-art approaches and motivation

6 Deep Learning background

7 Model

8 Experiments

9 Results and takeaways

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Introduction

Figure: Aspect-based Sentiment Analysis (ABSA)

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

A Brief History of Aspect-based Sentiment Analysis

Main driver of research: shared tasks at SemEvalworkshops

2014. First SemEval task on ABSA [Pontiki et al., 2014]:English reviews for laptops and restaurants

2015. Second SemEval task [Pontiki et al., 2015]:Extension and consolidation of previous subtasks

2016. Third SemEval task on ABSA [Pontiki et al., 2016]:Extension to new languages and domains

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Task Description

Subtask 1. Sentence-level ABSA:Slot 1. Aspect category: FOOD#QUALITY, FOOD#PRICE,etc.Slot 2. Opinion Target Expression: food, service, etc.Slot 3. Sentiment Polarity: positive, negative,neutral

Subtask 2. Text-level ABSA: FOOD#QUALITY:positive, FOOD#PRICE: negative, etc.

Subtask 3. Out-of-domain ABSA.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Task Description

Subtask 1. Sentence-level ABSA:Slot 1. Aspect category: FOOD#QUALITY, FOOD#PRICE,etc.Slot 2. Opinion Target Expression: food, service, etc.Slot 3. Sentiment Polarity: positive, negative,neutral

Subtask 2. Text-level ABSA: FOOD#QUALITY:positive, FOOD#PRICE: negative, etc.

Subtask 3. Out-of-domain ABSA.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Data

Language Domain # of # ofReviews Sentences

English Restaurants 440 2676English Laptops 530 3303Arabic Hotels 2291 6029Chinese Phones 200 9521Chinese Cameras 200 8040Dutch Restaurants 400 2286Dutch Phones 270 1697French Restaurants 455 2429Russian Restaurants 405 4299Spanish Restaurants 913 2951Turkish Restaurants 339 1248

Table: Number of reviews and sentences for every language-domainpair in the SemEval 2016 ABSA task [Pontiki et al., 2016].

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

An example sentence.

1 <s e n t e n c e i d=” 347 : 0 ”>2 <t e x t> I bought i t f o r r e a l l y cheap a l s o

and i t s AMAZING .</ t e x t>3 <O p i n i o n s>4 <Opin ion c a t e g o r y=”LAPTOP#PRICE”

p o l a r i t y=” p o s i t i v e ”/>5 <Opin ion c a t e g o r y=”LAPTOP#GENERAL”

p o l a r i t y=” p o s i t i v e ”/>6 </ O p i n i o n s>7 </ s e n t e n c e>

Figure: Example XML entry in a SemEval 2016 ABSA dataset.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

State-of-the-art Approaches and Motivation

State-of-the-art approaches use a lot of additionalinformation, e.g. domain-specific parsers and lexicons[Brun et al., 2014, Brun et al., 2016] as well as largesentiment lexicons [Kumar et al., 2016]

Can we achieve performance that is on-par or betterjust using the information contained in the review?

What information can we leverage?

The sentence.The aspect.The context of the surrounding sentences / the structureof the review.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Review structure

Elaboration

Background

that they cookwith only sim-ple ingredients.

I am amazedat the qualityof the food

I love thisrestaurant.

Figure: RST [Mann and Thompson, 1988] structure of an examplereview.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

RNNs

Recurrent Neural Networks (RNNs) and LSTMs arestate-of-the-art for many text classification and sequencetagging tasks.

Figure: An RNN takes an input xt at every time step t and producesan output ht .

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bidirectional RNNs

Bidirectional RNNs allow RNNs to ”look ahead”, workeven better in practice.

Figure: A bidirectional RNN: One RNN processes the inputleft-to-right; the other one right-to-left. The output yt at every timestep t is the concatenation of the outputs of the RNNs at thecorresponding time step.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

LSTM

An LSTM adds input, output, and forget gates to anRNN, is able to model long-range dependencies essentialfor capturing sentiment.

Figure: An LSTM cell.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Putting things together...

Sentence. Use a sentence-level bidirectional LSTM tocapture the sentence context.

Review. Use a review-level bidirectional LSTM to capturethe review context.

Aspect. Feed the aspect representation together with thesentence representation into the review-level LSTM.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Our model

Food is great. Service is top notch.FOOD#QUALITY

SERVICE#GENERAL

LSTM LSTM LSTM

LSTM LSTM LSTM 0

0 LSTM LSTM LSTM

LSTM LSTM LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

OUT OUT

0

0

Output

Outputlayer

Review-levelbackward LSTM

Review-levelforward LSTM

Sentence-levelbackward LSTMSentence-levelforward LSTM

Aspect/wordembeddings

Figure: The bidirectional hierarchical LSTM (H-LSTM) for ABSA.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Experiments

Hyperparameter tuning on development set

Dropout of 0.5 before and after LSTM cell

Pre-trained 300-dimensional GloVe word embeddings forEnglish, random embeddings for other languages1

Comparison models:

Best: best model of shared task [Pontiki et al., 2016] foreach domain-language pairIIT-TUDA: best single model of the competition[Kumar et al., 2016]CNN: sentence-level convolutional neural network[Ruder et al., 2016]LSTM: sentence-level Bi-LSTM

1Polyglot embeddings [Al-Rfou et al., 2013] (64 dimensions) did notimprove performance.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Results

Language Domain Best IIT CNN LSTM H-LSTM

English Restaurants 88.1 86.7 82.1 81.4 85.3Spanish Restaurants 83.6 83.6 79.6 75.7 79.5French Restaurants 78.8 72.2 73.2 69.8 73.6Russian Restaurants 77.9 73.6 75.1 73.9 78.1Dutch Restaurants 77.8 77.0 75.0 73.6 82.2Turkish Restaurants 84.3 84.3 74.2 73.6 76.7Arabic Hotels 82.7 81.7 82.7 80.5 82.8English Laptops 82.8 82.8 78.4 76.0 80.1Dutch Phones 83.3 82.6 83.3 81.8 81.3Chinese Cameras 80.5 - 78.2 77.6 78.6Chinese Phones 73.3 - 72.4 70.3 74.1

Table: Results of our system (H-LSTM) in comparison to the bestsystem for each pair (Best), the best single system (IIT-TUDA), asentence-level CNN (CNN), and our sentence-level LSTM (LSTM).

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Takeaways

Knowledge of surrounding sentences / review context ishelpful.

Id Sentence LSTM H-LSTM

1.1 No Comparison negative positive

1.2It has great sushi and

positive positiveeven better service.

2.1Green Tea creme

positive positivebrulee is a must!

2.2Don’t leave the

negative positiverestaurant without it.

Table: Example sentences where knowledge of other sentences in thereview (not necessarily neighbors) helps to disambiguate thesentiment of the sentence in question.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Takeaways

Pre-trained embeddings increase performance across alllanguages significantly (more results in final version).Gathering multilingual corpora is worth it.

H-LSTM is better than state-of-the-art particularly forlow-resource languages where reliable parsers are notavailable.

Generally, too little training data to completelycompensate for lack of domain information; lack of datadoes not allow using more sophisticated models, e.g.attention.

Gap to best model in English, Spanish and French is stilllarge. LSTMs can also use sentiment lexicon, but bestintegration is not obvious (use scalar scores,embed/bucket scores, filter based on occurrence, etc.).

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Presentation is based on:Sebastian Ruder, Parsa Ghaffari, John G. Breslin (2016). AHierarchical Model of Reviews for Aspect-based SentimentAnalysis. EMNLP, Austin, Texas, US.

Credit for RNN and LSTM images: Christopher Olah.

Thank you for your attention!

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bibliography I

[Al-Rfou et al., 2013] Al-Rfou, R., Perozzi, B., and Skiena, S.(2013).Polyglot: Distributed Word Representations for MultilingualNLP.Proceedings of the Seventeenth Conference onComputational Natural Language Learning, pages 183–192.

[Brun et al., 2016] Brun, C., Perez, J., and Roux, C. (2016).XRCE at SemEval-2016 Task 5: Feedbacked EnsembleModelling on Syntactico-Semantic Knowledge for AspectBased Sentiment Analysis.pages 282–286.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bibliography II

[Brun et al., 2014] Brun, C., Popa, D., and Roux, C. (2014).XRCE: Hybrid Classification for Aspect-based SentimentAnalysis.SemEval 2014, (SemEval):838–842.

[Kumar et al., 2016] Kumar, A., Kohail, S., Kumar, A., Ekbal,A., and Biemann, C. (2016).IIT-TUDA at SemEval-2016 Task 5: Beyond SentimentLexicon: Combining Domain Dependency and DistributionalSemantics Features for Aspect Based Sentiment Analysis.Proceedings of the 10th International Workshop on SemanticEvaluation, (SemEval).

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bibliography III

[Mann and Thompson, 1988] Mann, W. C. and Thompson,S. A. (1988).Rhetorical Structure Theory: Toward a functional theory oftext organization.

[Pontiki et al., 2016] Pontiki, M., Galanis, D., Papageorgiou,H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M.,Al-Ayyoub, M., Zhao, Y., Qin, B., Clercq, O. D., Hoste, V.,Apidianaki, M., Tannier, X., Loukachevitch, N., Kotelnikov,E., Bel, N., Jimenez-Zafra, S. M., and Eryigit, G. (2016).SemEval-2016 Task 5: Aspect-Based Sentiment Analysis.In Proceedings of the 10th International Workshop onSemantic Evaluation, San Diego, California. Association forComputational Linguistics.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bibliography IV

[Pontiki et al., 2015] Pontiki, M., Galanis, D., Papageorgiou,H., Manandhar, S., and Androutsopoulos, I. (2015).SemEval-2015 Task 12: Aspect Based Sentiment Analysis.Proceedings of the 9th International Workshop on SemanticEvaluation (SemEval 2015), pages 486–495.

[Pontiki et al., 2014] Pontiki, M., Galanis, D., Pavlopoulos, J.,Papageorgiou, H., Androutsopoulos, I., and Manandhar, S.(2014).SemEval-2014 Task 4: Aspect Based Sentiment Analysis.Proceedings of the 8th International Workshop on SemanticEvaluation (SemEval 2014), pages 27–35.

A HierarchicalModel of

Reviews forABSA

SebastianRuder

Introduction

A BriefHistory ofABSA

Task

Data

SotA &Motivation

DLBackground

Model

Experiments

Results &Takeaways

Bibliography

Bibliography V

[Ruder et al., 2016] Ruder, S., Ghaffari, P., and Breslin, J. G.(2016).INSIGHT-1 at SemEval-2016 Task 5: Deep Learning forMultilingual Aspect-based Sentiment Analysis.Proceedings of the 10th International Workshop on SemanticEvaluation.