Date post: | 17-Jan-2017 |
Category: |
Science |
Upload: | mark-graus |
View: | 221 times |
Download: | 0 times |
Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference
Elicitation?
Mark P. Graus
Martijn C. Willemsen
Human-Technology Interaction Group
Eindhoven University of Technology
Summary
• We wanted to see if we could make people chose less popular items in a choice-based preference elicitation recommender system by showing them trailers.
• We tested this in a between subjects user study.
• We found that after watching trailers people chose less popular items, while user experience was not negatively affected.
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '162
Motivation9/16/2016
Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16
3
Latent Feature Diversification
• Can we reduce choice overload through diversification based on the latent features of a matrix factorization model?
Willemsen, M. C., Graus, M. P., & Knijnenburg, B. P. (2016). Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Modeling and User-Adapted Interaction, 1–43. http://doi.org/10.1007/s11257-016-9178-6
Latent Feature 1
Late
nt
Feat
ure
2
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '164
Latent Feature Diversification Findings
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '165
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
low mid high
stan
dar
diz
ed
sco
re
diversification
Perceived diversity
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
low mid highst
and
ard
ize
d s
core
diversification
Expected choice difficulty
Late
nt
Feat
ure
2
Latent Feature 1
4) Iteration 2
Choice-Based Preference Elicitation
• Can we improve the user experience during cold start by having people choose between items instead of rating items?
Graus, M. P., & Willemsen, M. C. (2015). Improving the User Experience during Cold Start through Choice-Based Preference Elicitation. In Proceedings of the 9th ACM Conference on Recommender Systems - RecSys’15 (pp. 273–276). New York, New York, USA: ACM Press. http://doi.org/10.1145/2792838.2799681
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '166
How does this work? Step 1
Latent Feature 1Late
nt
Feat
ure
2
Iteration 1a: Diversified choice set is calculated from a matrix factorization model (red items)
Iteration 1b: User vector (blue arrow) is moved towards chosen item (green item), items with lowest predicted rating are discarded (greyedout items)
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '167
How does this work? Step 2
Iteration 2: New diversified choice set (blue items)
End of Iteration 2: with updated vector and more items discarded based on second choice (green item)
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '168
Choice-Based Preference Elicitation Findings
• People are more satisfied with choice-based than rating-based interfaces
• This comes mainly because of increased popularity (items with many ratings)
But we do not want to
recommend popular
items!
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '169
Satisfaction with Chosen
ItemPopularity
Difficulty
Intra List Similarity
-2.407(.381)
p<.001
-.240 (.145)
p<.1
-.479 (.111)
p<.001
-.257 (.045)
p<.001
14.00 (4.51)
p<.01
Choice-Based List
+
+
- -
+
Why do people end up with popular items?
• Our hypothesis• Users don’t know all movies, hard to judge based on
metadata alone
• People choose movies they know
• People know movies that are popular
• Choosing popular movies results in popular recommendations
• Our Solution• Provide trailers as additional information for making choices
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1610
Rationale
• In the music domain• Implicit Feedback
• Movie domain• Implicit feedback is sparse
• I (can) listen to 100s of tracks in a week, but I can’t watch 100s of movies a week (and sustain my job).
• We can approximate experiencing movies by providing trailers
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1611
Study9/16/2016
Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16
12
Choice-Based Interface with or without Trailers
• N = 71
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1613
Expected Effects
Trailers
Perceived Diversity
Informativeness
Perceived Novelty
Choice Satisfaction
System Satisfaction
Popularity of Chosen Items
- +
+
+?
-?-
-
+
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1614
Set Up
• Random assignment
• Choice Based Preference Elicitation – 9 choices of 10 items
[with/without trailers]
• Recommendation List – Top-10 Items [with/without trailers]
• Survey to measure User Experience
• Informativeness
• Perceived Diversity
• Perceived Novelty
• System Satisfaction
• Choice Satisfaction
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1615
Results9/16/2016
Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16
16
Do trailers affect the popularity of chosen items?
• Checked through repeated measures (10 choices)
• Popularity is expressed as the rank ordering by number of ratings in MovieLens dataset
• Trailers do not decrease popularity of choices• The popularity rank of the item chosen in each choice
set• The average popularity rank of all items in each choice
set
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1617
However: Relative Popularity of Choice
• average popularity rank of choice set – popularity rank of chosen item
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1618
If we look at people that actually watched trailers
• People that watch trailers are
more likely to pick less popular
movies from the lists
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1619
User Experience
Choice Satisfaction
Perceived Diversity
System Satisfaction
Informativeness
.570 (.295)p < 0.1
-.604 (.091)p < 0.01
.244 (.162)n.s.
.785 (.115)p < 0.01
-.266 (.122)p < 0.05
Trailers.611 (.256)p < 0.05
-.570 (.259)p < 0.05
-
+
-
-
+
9/16/2016
+
Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16
20
Conclusions9/16/2016
Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @ RecSys '16
21
What we found
• Providing people with trailers does make them choose less popular items.
• No indication that the overall satisfaction is affected negatively or positively
• As opposed to initial study where popularity resulted in increased satisfaction
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1622
Limitations
• When were trailers watched?• In the preference elicitation task?
• In the decision task?
Future Work
• How do trailers affect a more standard rating interface?
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1623
Thank You
• Questions/remarks?
Mark Graus – PhD Student
Human-Technology Interaction Group
Eindhoven University of Technology
https://twitter.com/newmarrk
https://linkedin.com/in/markgraus
http://www.marrk.nl
9/16/2016Graus, Willemsen: Can Trailers Help to Alleviate ... IntRS @
RecSys '1624