Toward Better Interactions in Recommender Systems:

Cycling and Serpentining Approaches for Top-N Item Lists

Qian Zhao, F. Maxwell Harper, Joseph A. Konstan (GroupLens)Gediminas Adomavicius (Dept. of Information and Decision

Sciences, University of Minnesota)Martijn Willemsen (Human-Technology Interaction group,

Eindhoven University of Technology)

1

Recommender Systems

» Recommender systems typically display the top-N recommended items in order.

2

Example: MovieLens.org Top Picks

3

What’s wrong with this recommender?

4

1st Visit

5

2nd Visit

6

3rd ... Visit

7

8

What’s wrong with this recommender?

9

1st Page

10

5th Page

11

10th … Page

12

13

Excitement

Show Time/Further Exploration

Re-thinking Top-N Recommendation Lists

» Static Top-N in order à the best design for user interaction and temporal experience

» Two missing factors• Fresh vs. stale• Further exploration à worse quality/experience

14

Prior Work: Dynamic/Interactive Recommenders

» Classics• Temporal dynamics (Koren et al. 2010)• CARS (Adomavicius et al. 2011)• Incremental matrix factorization (Luo et al. 2012)

» Interactive (reinforcement) machine learning• Markov decision processes (Shani et al. 2002)• Contextual bandits (Lu et al. 2010)

15

This Work: Cycling and Serpentining

» Cycling demotes items that have been viewed several (3+) times, exposing fresher recommendations.

» Serpentining spreads top recommended items across several pages, offering high-quality items on each page as a user continues to explore.

16

Cycling

Movie Score

M1 5.0

M2 4.9

M3 4.6

M4 4.0

17

Movie Score #display

M1 5.0 3

M2 4.9 3

Cycling

Movie Score

M1 5.0

M2 4.9

M3 4.6

M4 4.0

18

Movie Score #display

M3 4.6 0

M4 4.0 0

M1 5.0 3

M2 4.9 3

SerpentiningMovie Score

M1 5.0

M2 4.9

M3 4.8

M4 4.7

M5 4.6

M6 4.5

M7 4.4

M8 4.3

19

Movie Score

M1 5.0

M3 4.8

M5 4.6

M7 4.4

p. 1

p. 2

p. 1

p. 2

SerpentiningMovie Score

M1 5.0

M2 4.9

M3 4.8

M4 4.7

M5 4.6

M6 4.5

M7 4.4

M8 4.3

20

Movie Score

M1 5.0

M3 4.8

M5 4.6

M7 4.4

M2 4.9

M4 4.7

M6 4.5

M8 4.3

p. 1

p. 2

p. 1

p. 2

When to Cycle

» Within-session: each time when users go back to home page• More perceived change but more confusion?• Reflective of current recommenders in terms of the

change, e.g. Youtube.» Between-session: when users sign in next time• Less disorienting?• But, can users perceive the change?

21

The novel contribution of this work

» Cycling and serpentining approaches are not studied before.• e.g. Youtube is not cycling (Davidson et al. 2010).

» Better understand the effects of dynamic top-N lists on user experience.

22

A between-subjects field experiment on MovieLens.org» Measurements• Objective activity level• Subjective perception (through surveys)

23

Objective Activity Level

» opt out rate (bad experience)» number of page views (negative efficiency or

positive engagement)» number of interested actions. i.e. clicks,

wishlist (positive engagement)» interested rate: number of interested actions

per page view (positive efficiency)

24

Subjective Perception

» Specific aspects• accuracy, familiarity, diversity, novelty• change, freshness, confusion, boredom

» Overall experience• usefulness and satisfaction

» Surveys embedded in user browsing activities, i.e. sampling user experience

25

Survey Prompt

26

Survey Display

27

28

Data

» March 22 – May 14, 2016 (1.6 months)» 5158 invited users (having more than 15

ratings and two sessions)» 987 users joined the experiment» Analyzing activities of the first half month

(802 users) and all survey responses (~900)

29

Activity Statistics in the Experiment

Activity #Activities #Usersfront page 43,371 987

top picks page 43,231(mean page depth: 9.14) 821

rating 163,242 943click 107,955 924

wishlisting 23,032 473

30

ResultsNo Cycling Within-session

CyclingBetween-session Cycling

No Serpentining control condition ? ?

Serpentining ? ? ?

31

ResultsNo Cycling Within-session

CyclingBetween-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

Serpentining

32

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

Results

33

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

34

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

35

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

too complicated manipulationno interesting sig. results

see the paper for details

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

36

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

see the paper

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

Some users hate it! L

37

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

see the paper

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

Staying users engage more!

38

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

see the paper

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

Staying users engage more.

More browsing to get recommendations! L

39

No Cycling Within-sessionCycling

Between-session Cycling

No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +

accuracy: -familiarity: -usefulness: -change: +freshness: +

#page views: +#interested: +

accuracy: -confusion: +change: +

Serpentining #page views: +#interested: +

accuracy: -familiarity: -usefulness: -

see the paper

Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics

Staying users engage more.

Delighted to explore more interesting recommendations! J

Summary

» Within-session cycling• Higher churning risk• Negative subjective perception, but positive

freshness• Higher level of user activities and interested rate

» Between-session cycling or serpentining• Higher level of user activities• Negative effects on subjective perception

40

Efficiency or engagement?

41

Efficiency or engagement?

42

An experience design perspective on recommenders: There is a tradeoff between serving come-and-go users vs. encouraging deeper interaction/engagement !

Messages

» Better understanding of the trade-off between efficiency vs. engagement can help design a better recommender user experience!

» Cycling and serpentining top-N recommendation lists have benefits (higher engagement) but also costs (negative perception)!

» More work combining algorithms and user experience is needed!

43

Thanks! Questions?

» Title: Toward Better Interactions in Recommender Systems: Cycling and Serpentining Approaches for Top-N Item Lists

» Authors: Qian Zhao, Gediminas Adomavicius, F. Maxwell Harper, Martijn Willemsen, Joseph A. Konstan

» Contact• zhaox331@umn.edu• http://www-users.cs.umn.edu/~qian/

44

45