Analysis, design and implementation of a Multi-

Criteria Recommender System based on Aspect

Extraction and Sentiment Analysis techniques

Instructors: Student:

Prof. Giovanni Semeraro Davide GIANNICO

Dott. Marco de Gemmis

Department of Computer ScienceMaster of Computer Science (Msc)

Outline

2

Intro

Multi-Criteria RecSys

Proposed approach

Experimentation

Conclusion

Future work

3

Need: necessity of the user to be supported during

complex decisional processes

Tendency: development of on-line platforms which provide

recommendations to the user and where the community

expresses the own opinion for a item class

Some stats 315M unique views per month & 200M reviews(TripAdvisor, 2014);

168M unique views per month & 35M reviews (Amazon, 2013);

139M unique views per month & 67M reviews (Yelp, 2014).

Scenery

Opportunity

4

Take advantage of the reviews informative power

incorporating such information in the recommendation

process.

Concept: make «value» from raw data.

Recommender Systems

5

Recommender Systems (RecSys) are decision support and

information filtering tools

The main goal is helping users which access to a data

source for discovering information or items that could be

interesting for them

Recently, RecSys area has focused on Multi-Criteria

RecSys[AMK11]

[AMK11] G. Adomavicius, N. Manouselis, Y. Kwon. ‘Multi-Criteria Recommender Systems’. In:

Recommender Systems Handbook, pp 769-803, Springer US, 2011

Multi-Criteria Recommender Systems

6

Techniques which provide recommendations to the user,

modeling the utility concept espressed by the user for an

item as a ratings vectore on several criteria

Focus: multi-criteria item-based collaborative filtering

Item-to-item similarity matrix

1.Item-to-item matrix calculus

2. Rating prediction

multi-static-criteria ratings

Input:

multi-criteria ratings

Output: recommendations

𝑠𝑖𝑚 Itemi, Itemj= …

…

IB Collaborative Filtering RecSys

Discussed Problem

7

State-of-the-art Approach: default criteria taxonomies on which users can rate

Limits and challenges:

Proposed criteria are not exhaustive respect to userpreferences need to «customize» the criteria

Vagueness of the criteria need of fine-grained criteria

Proposed Approach

8

Automatic identification of criteria from the reviews

using Aspect Extraction techniques

Sentiment Analysis for associating a preference level to

the extracted criteria (implicit rating)

Extension of multi-criteria item-based recommendation

algorithms exploiting the information extracted

Aspect Extraction e Sentiment Analysis

Step[V11]

9

Aspect

Extraction

Sentiment

Analysis

review<Main Aspect,Sub-Aspect,Rilevance,

Sentiment Score,holder>

<location,station, 0.8, 0.9, >

<staff, courtesy, 0.9, -0.6, >

<…, …, …, …, >

Opinion Retrieval

Engine (ORE) system

[V11] V. Giuliani. ‘Analisi, progettazione e sviluppo di un framework per l'opinion retrieval basato su tecniche

automatiche di aspect extraction’. In Tesi di Laurea in Informatica Magistrale, 2011

(multi-ORE-criteria ratings)

Algorithm A: #multi-ORE-criteria (1/2)

10

review dataset

multi-ORE-criteria ratings

Aspect

Extraction

Sentiment

Analysis

ORE System

Multi-Criteria Item-Based Collaborative Filtering RecSys which

uses the extracted ratings from the reviews for calculating the

item-to-item similarity matrix

Algorithm A: #multi-ORE-criteria (2/2)

11

multi-ORE-ratings Item-item similarity

matrix

(Multi-dimensional Pearson

coefficient)

𝑠𝑖𝑚 𝐼𝑡𝑒𝑚𝑖, Item𝑗 =

𝑢∈U,c=1..k 𝑅𝑢,𝑐,𝑖 − 𝑅 𝑖)(𝑅𝑢,𝑐,𝑖 − 𝑅 𝑗

𝑢∈U,c=1…k 𝑅𝑢,𝑐,𝑖 − 𝑅 𝑖

2𝑅𝑢,𝑐,𝑖 − 𝑅 𝑗

2

(Rating Prediction)

𝑃𝑢,𝑖 = 𝑎𝑙𝑙 𝑠𝑖𝑚𝑖𝑙𝑎𝑟 𝑖𝑡𝑒𝑚𝑠 𝑗(𝑠𝑗 ∗ 𝑅𝑢)

𝑛( , , 0.8 )

( , , 0.6 )

predicted ratings

Algorithm B: Multi-criteria Aspect-based Recommender

system based on sentimenT Analysis (#MARTA) (1/2)

12

Item #1Aspect

Extraction

Sentiment

Analysis

Position

Food

Staff-courtesy

Breakfast-buffet

<position, *,0.9, ..>(review#1)

<staff,courtesy, -0.6, ..>(review#2)<position, *,0.7, ..>(review#2)

(Item#1 summary)

If <aspect> 𝜖 summary

add <aspect> to the running average

else

add <aspect> to summary

Item#1 review set

(Item#1 multi-ORE-ratings)

…

…

Multi-criteria CF RecSys in which the item-to-item similarity matrix

is calculated considering each item description (summary)

13

Position

Food

Staff-courtesy

Breakfast-buffet

…

Food

Room-clean

Pool

Item-item similarity matrix

Item #1 summary

Item #2 summary

(Multi-dimensional

similarity measure)

(Rating Prediction)

( , , 0.9 )

( , , 0.7 )

predicted ratings

Algorithm B: Multi-criteria Aspect-based Recommender

system based on sentimenT Analysis (#MARTA) (2/2)

Room-view

…

𝑠𝑖𝑚 Itemi, Itemj= …

Developed system: Btravel (1/2)

14

Developed system: Btravel (2/2)

15

16

EXPERIMENTATION

Design

17

Research questions

Q1) #multi-ORE-criteria algorithm overtakes the single-criteria(#single-criteria) and multi-criteria techniques (#multi-static-criteria), which use default criteria, results ?

Q2) Multi-criteria Aspect-based Recommender system based on sentimenT Analysis(#MARTA) algorithm overtakes the state-of-the-art approaches results?

10-Fold Cross Validation. For each run the training set isused for calculating the similarity matrix

Evaluation of the approaches on the respective test set

Evaluation metrics: MAE, RMSE, Precision, Recall and F-Measure

Statistical validation using the Wilcoxon test

Dataset

18

Amazon

TripAdvisor

Yelp

#ratings #users #items #ratings/

user

#ratings/

item

sparsity

355.949 2.850 2.820 124,73 126,06 0,96

#ratings #users #items #ratings/

user

#ratings/

item

sparsity

229.905 45.981 11.537 5 19,9 0,99

#ratings #users #items #ratings/

user

#ratings/

item

sparsity

208.135 1.386 1.580 150,16 131,73 0,90

Q1 Results (#multi-ORE-criteria vs #single-

criteria and #multi-static-criteria)

19

Dataset TripAdvisor

MAE

SVD 0,87

#single-criteria 1,32

#multi-static-criteria 1,10

#Multi-ORE-criteria 0,96

0,87

1,32

1,10

0,96

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

MA

E

F-Measure

SVD 0,59

#single-criteria 0,52

#multi-static-criteria 0,53

#multi-ORE-criteria 0,71

0,59

0,52 0,53

0,71

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

F-M

easu

re

Q2 Results (#MARTA)

20

DatasetYelp (sparsity: 0,99)

MAE

SVD 0,91

#single-criteria 1,18

#multi-ORE-criteria 1,13

#MARTA 0,85

0,91

1,181,13

0,85

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

MA

E

F-Measure

SVD 0,58

#single-criteria 0,49

#multi-ORE-criteria 0,55

#MARTA 0,63

0,58

0,49

0,55

0,63

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

F-M

easu

re

Q2 Results (#MARTA)

21

Dataset TripAdvisor (sparsity: 0,96)

MAE

SVD 0,87

#single-criteria 1,32

#multi-ORE-criteria 0,96

#MARTA 0,87

0,87

1,32

0,960,87

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

MA

E

F-Measure

SVD 0,59

#single-criteria 0,51

#multi-ORE-criteria 0,71

#MARTA 0,58

0,59

0,51

0,71

0,58

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

F-M

easu

re

Q2 Results (#MARTA)

22

Dataset Amazon (sparsity: 0,90)

MAE

SVD 0,71

#single-criteria 0,65

#multi-ORE-criteria 0,55

#MARTA 0,71

0,710,65

0,55

0,71

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

MA

E

F-Measure

#SVD 0,62

#single-criteria 0,65

#multi-ORE-criteria 0,74

#MARTA 0,63

0,62

0,65

0,74

0,63

0,54

0,56

0,58

0,60

0,62

0,64

0,66

0,68

0,70

0,72

0,74

0,76

F-M

easu

re

Analysis of the results

23

Q1) #multi-ORE-criteria algorithm overtakes the single-

criteria(#single-criteria) and multi-criteria techniques (#multi-

static-criteria), which use default criteria, results ?

#multi-ORE-criteria showed better results than #multi-

static-criteria and #single-criteria (MAE and F1)

Q2) Multi-criteria Aspect-based Recommender system based on

sentimenT Analysis(#MARTA) algorithm overtakes the state-of-

the-art approaches results?

#MARTA showed good results especially on sparse

dataset

Conclusions

24

Indipendence from the domain

Influenced by the review quality

Good performance especially on the item classification

(relevant or not)

Future work

25

Considering the extracted opinions relevance score

Testing more advanced Aspect Extraction and Sentiment

Analysis techniques

Learning weighing schemes for each user and considering

the context for better performance