Music Emotion Tracking with Continuous Conditional Neural Fields and Relative Representation

Abstract

This working notes paper introduces the system proposed by the Rainbow group for the MediaEval Emotion in Music 2014 task. The task is concerned with predicting dynamic emotion labels for an excerpt of a song and for our approach we use Continuous Conditional Neural Fields and relative feature representation both of which have been developed or adapted by our group.

1. INTRODUCTION

The Emotion in Music task is concerned with providing dynamic arousal and valence labels and is described in the paper by Aljanaki et al.[ 1 ].

The use of relative feature representation has already been introduced to the eld of dynamic music annotation and tested on MoodSwings dataset [ 4 ] by Imbrasaite_ et al.[ 2 ]. They have shown substantial improvement over using standard feature representation with the standard Support Vector Regression (SVR) approach as well as comparable performance to more complicated machine learning techniques such as Continuous Conditional Random Fields.

Continuous Conditional Neural Fields (CCNF) have also been used for dynamic music annotation by Imbrasaite_ et al.[ 3 ]. In our experiments we have achieved results that clearly outperformed SVR when using standard feature representation and produced similar results to using relative feature representation. It was suspected that the short extracts (only 15s) and little variation in emotion were the main reasons why the model was not able to achieve better results. In this paper we are applying the same techniques to a dataset that improves on both accounts with a hope of clearer results. 2.1

METHOD

Feature extraction and representation

In our system we used two feature sets. Both feature sets were extracted by OpenSMILE using a standard set of features. As CCNF can su er when dealing with a large feature vector and fail to converge, we used a limited set of statistical descriptors extracted from the features limiting the total number of features to 150.

The rst feature set was used as is, in the standard fea2.2

CCNF

Our CCNF model consists of a undirected graphical model that can model the conditional probability of a continuous valued vector y (for example emotion in valence space) depending on continuous x (in this case, audio features).

In our discussion we will use the following notation: x = fx1; x2; : : : ; xng is a set of observed input variables, X is a matrix where the ith column represents xi, y = fy1; y2; : : : ; yng is a set of output variables that we wish to predict, xi 2 Rm and yi 2 R (patch expert response), n is the length of the sequence of interest.

Our model for a particular set of observations is a conditional probability distribution with the probability density function: exp( ) P (yjx) = R 1 exp( )dy (1)

1

We de ne two types of features in our model: vertex features fk and edge features gk. Our potential function is de ned as:

K1 = X X i k=1

K2 kfk(yi; xi; k) + X X kgk(yi; yj ) i;j k=1 We constrain k > 0 and k > 0, while is unconstrained. The model parameters = f 1; 2; : : : K1g,

= f 1; 2; : : : K1g, and = f 1; 2; : : : K2g are learned and used for inference during testing

The vertex features fk represent the mapping from the xi to yi through a one layer neural network, where k is the weight vector for a particular neuron k. h( ; xi) = 1 + e

T xi (2) (3)

The number of vertex features K1 is determined experimentally during cross-validation, and in our experiments we tried K1 = f5; 10; 20; 30g.

The edge features gk represent the similarities between observations yi and yj . This is also a ected by the neighborhood measure S(k), which allows us to control the existence of such connections.

gk(yi; yj ) = 21 Si(;kj)(yi (5)

In our linear chain CCNF model, gk enforces smoothness between neighboring nodes. We de ne a single edge feature, i.e. K2 = 1. We de ne S(1) to be 1 only when the two nodes i and j are neighbors in a chain, otherwise it is 0. 2.2.1

Learning and Inference We are given training data fx(q); y(q) M gq=1 of M song samples, together with their corresponding dimensional continuous emotion labels. The dimensions are trained separately, but all the parameters ( , and ) for each dimension are optimised jointly.

We convert the Eq.(1) into multivariate Gaussian form. It helps with the derivation of the partial derivatives of loglikelihood, and with the inference.

For learning we can use the constrained limited memory Broyden-Fletcher-Goldfarb-Shanno algorithm for nding locally optimal model parameters. We use the standard Matlab implementation of the algorithm. In order to make the optimisation both more accurate and faster we used the partial derivatives of the log P (yjx), which are straightforward to derive and are similar to those of CCRF [ 2 ].

A more thorough description of the model as well as the code to reproduce the results can be found at http://www.cl.cam.ac.uk/research/rainbow/projects/ccnf/

RESULTS

In order to get a better understanding of where CCNF stands in terms of performance, we decided to compare it to another standard approach used in the eld. We used Support Vector Regression (SVR) model with the Radial Basis Function kernel in the same way we used CCNF|we trained a model for each axis, using 2-fold cross-validation to pick the best parameters for training. The experimental design was identical to the one used in our previous paper [ 3 ], which makes the results comparable not only to the baseline method in this challenge, but also between several datasets.

There are several interesting trends visible from the results (see Table 1). First of all, CCNF combined with the relative feature representation clearly outperforms all the other methods for the arousal axis, as well as the baseline method. Secondly, the spread for correlation for CCNF model is twice as big as the one for SVR, while there is little di erence between the spread for RMSE for the di erent methods. In fact, there is little di erence in performance between the di erent methods and the di erent representations used for the valence axis. 4.

FURTHER INSIGHTS

We found it interesting to compare the results achieved with this dataset to those achieved with the MoodSwings dataset. This shows how much of an impact the dataset has on the performance and even the ranking of di erent methods. In our previous work CCNF clearly outperformed SVR with the standard feature representation, while the results with the relative feature representation were comparable between the two models. With this dataset, we would have to draw very di erent conclusions|with the standard representation the results were comparable, if not better for SVR, while there was a clear di erence between the two when using the relative feature representation for the arousal axis, with CCNF clearly outperforming SVR. This maybe due to the fact that there are more training (and testing) samples in this dataset, the extracts are longer and, possibly, better suited to the task.

The valence axis is still proving problematic. The fact that quite heavyweight techniques are not able to outperform simple models with small feature vectors seems to be indicating that we are approaching the problem from a wrong angle. Improving results for the valence axis should be the top priority for our future work.

References

[1] A. Aljanaki , Y.-H. Yang , and M. Soleymani . Emotion in music task at mediaeval 2014 . In MediaEval Workshop, 2014 .

[2] V. Imbrasaite , T. Baltrusaitis , and P. Robinson . Emotion tracking in music using continuous conditional random elds and relative feature representation . In Proc. of ICME. IEEE , 2013 .

[3] V. Imbrasaite _, T. Baltrusaitis , and P. Robinson . Ccnf for continuous emotion tracking in music: Comparison with ccrf and relative feature representation . In Proc. of ICME. IEEE , 2014 .

[4] J. A. Speck , E. M. Schmidt , B. G. Morton , and Y. E. Kim . A comparative study of collaborative vs. traditional music mood annotation . Proc. of ISMIR , 2011 .