In this paper we describe TUM's approach for the MediaEval's \Emotion in Music" task. The goal of this task is to automatically estimate the emotions expressed by music (in terms of Arousal and Valence) in a time-continuous fashion. Our system consists of Long-Short Term Memory Recurrent Neural Networks (LSTM-RNN) for dynamic Arousal and Valence regression. We used two di erent sets of acoustic and psychoacoustic features that have been previously proven as e ective for emotion prediction in music and speech. The best model yielded an average Pearson's correlation coe cient of 0.354 (Arousal) and 0.198 (Valence), and an average Root Mean Squared Error of 0.102 (Arousal) and 0.079 (Valence).
The MediaEval 2014 \Emotion in Music" task comprises
two subtasks. The rst subtask (Subtask 1), pertains to
the development of new features to automatically estimate
Arousal and Valence for music excerpts. The second subtask
(Subtask 2), consists of the estimation of Arousal and
Valence scores continuously in time for a second long segments
of the same music excerpts. In both tasks, the development
set consists of 744 instances and the evaluation set
comprises 1000 instances. For more details, please refer to [
We used two features sets in our experiments. The rst
feature set (FS1) consists of the o cial set of low-level audio
descriptors (LLDs) used in the 2013 INTERSPEECH
Computational Paralinguistics Challenge (ComPareE; see [
As regressors, and given the importance of the temporal
context in emotional responses to music (e.g., [
We used a multi-task learning framework for the joint learning of Arousal and Valence time-continuous values. A cross-validation procedure was used in the development phase, where we created an extra fold to estimate the performance of our approaches during the development phase. The fold subdivision followed a modulus based scheme (instance ID modulus 11). The instances yielding a remainder of 10 were left out to create a small test set for performance estimation. On the remaining instances, a 10-fold cross-validation was performed. We computed 5 trials of the same model each with randomized initial weights in the range [-0.1,0.1]. Our basic architecture consisted of deep LSTM-RNN with 2 hidden layers. We optimised the number of LSTM blocks in each hidden layer, as well as the learning rate (a momentum of 0.9 was used for all tests), and the standard deviation of the Gaussian noise applied to the input activations (used to alleviate the e ects of over- tting). An early stopping strategy was also used to avoid over tting the training data { training was stopped after 20 iterations without improvement of the validation set performance (sum of squared errors). The instances in the 10 training sets were presented in random order to the model during training. The input (acoustic features) and output (emotion features) data were standardised to zero mean and unit variance on the correspondent training sets used in each cross-validation fold.
In four of our ve runs (see next subsection) we pretrained the rst hidden layer. Our unsupervised pre-training strategy consisted of de-noising LSTM-RNN auto-encoders. We rst created a LSTM-RNN with a single hidden layer trained to predict the input features (y(t) = x(t)). Both the development and test set instances were used to train the DAE. In order to avoid over- tting, in each training epoch and timestep t, we added a noise vector n to x(t), sampled from a Gaussian distribution with zero mean and variance n. After determining the auto-encoder weights a second hidden layer was added. In two of the runs, all of the weights were trained using the regression targets and keeping the rst layer weights constant. In the other two, the rst layer weights were retrained. 2.2
We submitted ve runs for Subtask 2. All runs consisted of LSTM-RNNs using two hidden layers in order to attempt modeling high-level abstractions in the data (Deep Learning). The speci cs of each run are as follows: Run 1) The basic architecture was directly trained using the regression targets and FS1; Run 2) We pre-trained the rst layer, added a second one, and all weights (with the exception of the rst layer weights that were kept constant) were trained using the regression targets and FS1; Run 3) Same as Run 2, but all weights (including the rst layer weights) were trained using the regression targets and FS1; Run 4) Same as Run 2, but using FS2; Run 5) Same as Run 3, but using FS2; The submitted results for each test run consisted of the average outputs of the ve best models (across all folds and trials) as estimated using the method described in Section 2.1.
In Table 1, we report the o cial challenge metrics (r Pearson's linear correlation coe cient; and RM SE - Root Mean Squared Error) calculated individually for each music piece and averaged across all pieces (standard deviations also shown) of the test set. In short, we observe that Run 4 lead to the best results. Individual two-tailed t-tests revealed that: a) r(Arousal) was signi cantly higher for Run 4 compared to Run 1, Run 2, Run 5 (p < 0:0001), and Run 3 (p < 0:01); b) r(V alence) was higher for Run 4 compared to all other runs, but only signi cantly higher than Run 3 (p < 0:05); c) RM SE(Arousal) was signi cantly lower for Run 4 compared to all other runs (p < 0:0001); d) RM SE(V alence) was signi cantly lower for Run 4 compared to all other runs (p < 0:0001) except Run 5.
Run 4 consisted of a LSTM-RNN with two layers, including a pre-trained rst layer (with weights kept constant while training using the regression targets) and FS2 as input. The optimised architecture consisted of 200 and 5 LSTM blocks ( rst and second layers, respectively), trained with a learning rate of 10 6 and Gaussian noise with a variance of 0.5 applied to the inputs during development (no noise added when processing the test set). RM SE
Run 1 Run 2 Run 3 Run 4 Run 5 Run 1 Run 2 Run 3 Run 4 Run 5 4.
The LSTM-RNN approaches to the 2014 MediaEval \Emotion in Music" task all delivered consistent improvements over the baselines. The results reveal the importance of ne-tuning the feature set and the deep learning strategy, which could be attributed to the relatively small training set. 5.
This work was partially supported by the ERC in the European Community's 7th Framework Program under grant agreements No. 338164 (Starting Grant iHEARu to Bjorn Schuller) and 230331 (Advanced Grant PROPEREMO to Klaus Scherer). 6.