1

Music Recommendation by Unified Hypergraph: Combining Social Media

Information and Music Content

Jiajun Bu, Shulong Tan, Chun Chen, Can Wang, Hao Wu, Lijun Zhang and Xiaofei He

Zhejiang University

2

Multi-type Media Fusion• Content analysis– text– Image– Audio– Video– ……

• Social analysis– Friendship– Interest group– Resource collection– Tag– ……

Hypergraph

3

Outlines

• Music Recommendation

• Social media information

• Unified Hypergraph Model

• Music Recommendation on Hypergraph (MRH)

• Experimental results

4

Music Recommendation We have huge amount of music available in music

social communities It is difficult to find music we would potentially like Music Recommendation is needed!

Recommended music by the Last.fm.

Traditional Music Recommendation Traditional music recommendation methods

only utilize limited kinds of social informationCollaborative Filtering (CF) only uses

rating informationAcoustic-based method only utilizes

acoustic featuresHybrid method just combines these two

5

6

Music Social Community[www.pandora.com]

Actions which users can do on resources

Social activities between users

7

Users can also make

friends.

Users can bookmark resources by tags.

Users can listen to music. These are music tracks

this user likes best.

Introduction to Last.fm

Users can

join in groups

8

Social Media Information in Last.fmFriendshipsMemberships

Tagging relations

Listening relations

Inclusion relations

9

Social Media Information The rich social media information is valuable

for music recommendation.

To build the users’ preference profiles.

To predict users’ interests from their friends.

To recommend music tracks by albums or artists.

…

10

u1

u2 t2

t1

r2

r1

How About Graph Model? Use traditional graph to model social media

information but fail to keep high-order relations in social media information

(u1, t1, r1)

(u1, t2, r2)

(u2, t2, r1)

It is unclear whether u2

bookmarks r1, r2, or both.

?

11

Unified Hypergraph Model Using a unified hypergraph to model multi-type

objects and the high-order relationsEach edge in a hypergraph, called a hyperedge, is an arbitrary non-empty subset of the vertex setModeling each high-order relation by a hyperedge, so hypergraphs can capture high-order relations naturally

(u1, t1, r1)

(u1, t2, r2)

(u2, t2, r1)

The high-order relations among the

three types of objects can be naturally represented as

triples.

12

Unified Hypergraph Construction

The six types of objects form the vertex set of the unified hypergraph.

Each type of relations corresponds to a certain type of hyperedges in the unified

hypergraph.

tag

tag

tag

album

album

13

Hyperedges Construction Details :a hyperedge

corresponding to each pairwise

friendship

:a hyperedge corresponding to

each group : a hyperedge for each user-track listening relation

:a hyperedge corresponding to each tagging relation

:a hyperedge for each album or artist

: a hyperedge for each track-track similarity relation

14

Ranking on Unified HypergraphSetting a user as the query

…Track List

Tracks have more strong

“hyperpaths” to the query user will get higher ranking scores

15

Notation• A unified hypergraph • : Vertex-hyperedge incidence matrix

16

Notation-2• : the degree of a hyperedge is the number

of vertices in the hyperedge : • : the degree of a vertex is the weight sum

of all hyperedges the vertex belongs to:

• Dv , De and W : diagonal matrices consisting of hyperedge degrees, vertex degrees and hyperedge weights

17

Problem Definition

• Given some query vertices from , rank the other vertices on the unified hypergraph according to their relevance to the queries.

• : the ranking score of the i-th object• : the vector of ranking scores• : the query vector

18

Cost Function

Vertices contained in many common hyperedges should have similar ranking scores

Obtained ranking scores should be similar to pre-given labels

The optimal ranking result is achieved when Q(f) is minimized

19

Matrix-vector Form

20

Optimal Solution

Requiring that the gradient of Q(f) vanish gives the following this

equation

We define

Note: all the matrices are highly sparse!

21

Music recommendation on Hypergraph (MRH)

• The offline training phase: Constructing matrix H and WComputing matrix Dv and De

Calculating , where

• The online recommendation phase:Building the query vector yComputing the ranking results f*Recommending top tracks which not listened

General Ranking Framework

22

Setting a user as the query

…User List

…Group List

…Tag List

…Track List

…Album List

…Artist List

For friend recommendation

For artist recommendation

For group recommendation

For album recommendation

For topic recommendation

For music recommendation

Personalized Tag Recommendation

23

Setting a user and an resource as the queries

…Tag ListPersonalized Tag

recommendation for the target user and

resource

24

Objects and Relations in Our DatasetObjects

Relations

25

Compared AlgorithmsAlgorithms Information Used

User-based Collaborative Filtering (CF) R3

Acoustic-based music recommendation (AB) R3, R9

Ranking on Unified Graph (RUG) R1, R2, R3, R4, R5, R6, R7, R8, R9

Our proposed music recommendation on Hypergraph method (MRH)

MRH-hybrid R3, R9

MRH-social R1, R2, R3, R4, R5, R6, R7, R8

MRH R1, R2, R3, R4, R5, R6, R7, R8, R9

• R1: friendship relations• R2: membership relations• R3: listening relations• R4: tagging relations on tracks• R5: tagging relations on albums

• R6: tagging relations on artists• R7: track-album inclusion relations• R8: album-artist inclusion relations• R9: similarities between tracks

26

Performance Comparison

It is clear that our proposed algorithm significantly outperforms the other recommendation algorithms

Comparison of recommendation algorithms in terms of MAP and F1.

Comparison of recommendation algorithms in terms of NDCG.

27

CF algorithm does not work well too. This is probably because the user-track matrix in our data set is highly sparse

Acoustic-based (AB) method works worst. That is because acoustic-based method incurs the semantic gap and similarities based on

acoustic content are not always consistent with human knowledge

The superiority of MRH over RUG indicates that the hypergraph is indeed a better choice for modeling complex relations in social media information

Our proposed method alleviates these problems. MRH-hybrid only uses similarity relations among music tracks and listening relations,

but it works much better than AB and CF

Precision-Recall Curves

Comparing to MRH-social, MRH uses similarity relations among tracks additionally. We find that using this acoustic information can improve the recommendation result, especially when we only care top ranking music

tracks.

28

The baseline is MRH only using listening relations

Social Information Contribution

Comparison of MRH on different subsets of social media information in terms of MAP and F1.

MRH using listening relations and inclusion relations

MRH using listening relations and tagging relations

MRH using listening relations, and social relations

There is a little improvement at lower ranks obtained by social relations. Intuitively, the users’ tastes may be inferred from friendship

and membership relations.Using inclusion relations among resources, we can recommend music

tracks in the same or similar albums, as well as the tracks performed by the same or similar artists. So the performance is improved greatly.

Tagging relations do not improve the performance. That is because there is a strong correlation between listening relations and tagging relations, and

thus the usage of tagging relations is limited

29

System of a Down

War?

An Example

No. 793

Top 5 Recommended Tracks for No.793

Dirty Window

Deer Dance

Spit It Out

Know

System of a Down

Slipknot

Metallica

System of a Down

From:

From:

From:

From:

From:

The reason these three tracks are recommended is that user No. 793 likes music come from System of a

Down best.

User No. 793 joins in the groups about metal and named Slipknot. Users in these groups are fans of Metallica (one of the four

most popular heavy metal band. )and Slipknot

Conclusion

• We use the unified hypergraph model to fuse multi-type media, includes multi-type social media information and music content.Social media information is very useful for music recommendation.Hypergraphs can accurately capture the high-order relations among various types of objects.

30

31