1
Personalized Music Emotion Recognition via Model Adaptation
Ju-Chiang Wang, Yi-Hsuan Yang, Hsin-Min Wang, and Skyh-Kang Jeng
Academia Sinica, National Taiwan University,Taipei, Taiwan
2
Outline
• Introduction
• The Acoustic Emotion Gaussians (AEG) Model
• Personalization via MAP Adaptation
• Music Emotion Recognition using AEG
• Evaluation and Result
• Conclusion
3
Introduction
• Developing a computational model that comprehends the affective content from musical audio signal, for automatic music emotion recognition and content-based music retrieval
• Emotion perception in music is in nature subjective (fairly user-dependent)– A general music emotion recognition (MER) system
could be insufficient– One’s personal device is desirable to understand
his/her perception of music emotion– Adaptive MER method, efficient and effective
4
Basic Idea
• The UBM-GMM system for speaker adaptation– State-of-the-art systems for speaker recognition– A large background GMM (UBM), representing the
speaker-independent distribution of acoustic features– Obtain the speaker-dependent GMM via model
adaptation with the speech data of a specific speaker• Adaptive MER method for personalization
– A probabilistic background emotion model, learn the broad emotion perception of music from general users
– Personalize the background emotion model via model adaptation in an online and dynamic fashion
5
Multi-Dimensional Emotion• Emotions are considered as numerical values
(instead of discrete labels) over two emotion dimensions, i.e., Valence and Arousal (Activation)
• Good visualization, a unified model
Mr. Emo developed by Yang and Chen
6
The Valence-Arousal Annotations
• Different emotions may be elicited from a song• Assumption: the VA annotation of a song can be
drawn from a Gaussian distribution, as observed• Learn from the multiple annotations and the acoustic
features of the corresponding song• Predict the emotion as a single Gaussian
7
The Acoustic Emotion Gaussians Model
• Represent the acoustic features of a song by a probabilistic histogram vector
• Develop a model to comprehend the relationship between acoustic features and VA annotations– Wang et al. (2012), “The acoustic emotion Gaussians model for
emotion-based music annotation and retrieval,” Proc. ACM Multimedia (full paper)
Acoustic GMM Posterior Distributions
8
Construct Feature Reference Model
A1 A2AK-1
AK A3A4
Each component representsa specific pattern
EM Training
A Universal Music Database
Acoustic GMM
Music Tracks& Audio Signal
Frame-based Features
… …
… …
Global Set of frame vectors randomlyselected from each track
…
Music Tracks& Audio Signal
A Universal Music Database
Music Tracks& Audio Signal
9
Represent a Song into Probabilistic Space
1
2
K-1
K…
Posterior Probabilities over the Acoustic GMM
…
A1
A2
AK-1
Acoustic GMM
AK
…
Feature Vectors Histogram:Acoustic GMM Posterior
prob
1 2 K-1 K…
10
Generative Process of VA GMM
• Key idea: Each component in acoustic GMM can generate a component VA Gaussian
Audio Signal of Each Clip
A Mixture of Gaussians in the VA Space
…
A1
A2
AK-1
Acoustic GMM
AK
1
2
K-1
K…
Viewed as a set of acoustic codewords
11
The Likelihood Function of VA GMM
• Each training clip is annotated by multiple users {uj}, indexed by j
• An annotated corpus: assume each annotation eij of clip si can be generated by a weighted VA GMM with {qik}!
• Generating the Corpus-level likelihood and maximize it using the EM algorithm
1 1 1 1
( | ) ( | ) ( | , )jU KN N
i ik ij k ki i j k
p p s q= = = =
= = å E E e m S
1
( | ) ( | , )K
ij i ik ij k kk
p s q=
=åe e Sm
Acoustic GMM posterior
Clip-level likelihood:Each user contributes equally
parameters of each latent VA Gaussian to learn
Annotation-levelLikelihood
12
Personalizing VA GMM via MAP
• Apply the Maximum A Posteriori (MAP) adaptation• Suppose we have a set of personally annotated songs
{ei, qi}, i =1,…,M• The posterior probability over each component zk for ei is
• The expected sufficient statistics with posterior and ei1
( | , )( | , )
( | , )ik i k k
k i i K
iq i q qq
p zq
q=
=å
ee
e
m
m
Sq
S
1
1
( | , )( )
( | , )
M
k i i iik M
k i ii
p zE
p z=
=
¬åå
e e
em
q
q
1
1
( | , )( )
( | , )
W Tk ij i i ii
k W
k i ii
p zE
p z=
=
=åå
e e e
e
qS
q
13
MAP for GMM: Parameter Interpolation
• The updated parameters for the personalized VA GMM can be derived by interpolation
• The effective number of component zk for the target user
• The interpolation factors (data-dependent) can be set by
( ) (1 ) ,k k k k kEa a¢ ¬ + -m m m
( ) (1 )( ) .T Tk k k k k k k k kEa a¢ ¢ ¢¬ + - + -S S S m m m m
1
( | , )M
k k i ii
M p z=
= å e q
k
kk M
M
Parameter interpolation between the expectation and background
14
Graphical Interpretation – MAP Adaptation
1
2
3 6
5
4
Interpolation Factor
Acoustic GMM Posterior
The personal annotation can be applied to clips exclusive to the background training set
15
Music Emotion Recognition
• Given the acoustic GMM posterior of a test song, predict the emotion as a single VA Gaussian
1
2
K-1
K
…
Acoustic GMM Posterior Learned VA GMM Predicted Single Gaussian
1
ˆˆ( | ) ( | , )K
k ij k kk
p s q=
=åe e m S
^
^
^
^
…
{ , }*m *S
16
Find the Representative Gaussian
• Minimize the cumulative weighted relative entropy– The representative Gaussian has the minimal cumulative
distance from all the component VA Gaussians
• The optimal parameters of the Gaussian are
( )KL{ , }
1
ˆ( | , ) argmin ( | , ) || ( | , )K
k k kk
p D p pq* *
=
= åe e eS
S S Sm
m m m
*
1
ˆ K
k kk
q=
= åm m
( )* * *
1
ˆ ( )( )K
Tk k k k
k
q=
= + - -åS S m m m m
17
Evaluation – Dataset and Acoustic Features
• MER60– 60 music clips, each is 30-second– 99 users in total, each clip annotated by 40 subjects– 6 users have annotated all the clips– Evaluate the personalization based on these 6 users
• Bag-of-frames representation, perform the analysis of emotion at the clip-level, instead of frame-level– 70Dim: dynamic, spectral, timbre (13 MFCCs, 13 delta MFCCs,
and 13 delta-delta MFCCs), and tonal
18
Evaluation – Incremental Setting
• Incremental adaptation experiment per target user– Randomly split all the clips (w/ annotations) into 6 folds – Perform 6-fold CV
• Hold out one fold for testing• The rest 5 folds: All the annotations except the
target user’s to train a background VA GMM• Add one fold of annotation of target user into the
adaptation pool (P=5 iterations loop) – Adaptation pool to adapt the background VA
GMM– Evaluate the prediction performance of the test
fold
19
Evaluation – Result
• Metric (ALLi): compute the log-likelihood of the predicted Gaussian with the ground truth annotation of the target user
20
Conclusion and Future Work
• The AEG model provides a principled probabilistic framework that is technically sound, and flexible for adaptation
• We have presented a novel MAP-based adaptation technique which is very efficient for personalizing the AEG model
• Demonstrated the effectiveness of the proposed method for personalizing MER in an incremental learning manner
• We will investigate the maximum likelihood linear regression (MLLR) that learns a linear transformation over the parameters of the AEG model
