In this year's MediaEval task, Emotion and Theme Recognition in Music Using Jamendo, the goal is to assign emotion and theme tags to songs. In this paper, we describe our-Team TaiInn (Innsbruck)approach for this task. We use a neural network model consisting of both convolutional and recurrent layers and utilize spectral, highlevel as well as rhythm features. Our approach achieves a ROC-AUC score of 0.723 on the provided test set.
At this year’s MediaEval workshop, the task Emotion and Theme Recognition in Music Using Jamendo deals with detecting mood and theme tags for songs that are described by audio descriptors. Those songs are drawn from Jamendo1, and collected in a data set created by Bogdanov et al. [3]. The tags that have to be predicted cover a wide range of emotions and themes, and include tags like sad, heavy, relaxing, and children. More details can be found in [1].
We use a neural network model to approach this task and show that our model achieves a performance that is comparable to the provided baseline (ROC-AUC of 0.723 for our best run), but requires much fewer training epochs to reach this (16 vs 1,000). We also show that generating more training samples by drawing random windows from the provided mel-spectrograms improves results, and that incorporating high-level features into the model architecture reduces the number of needed epochs in training while giving almost the same results. Our code is available on GitHub2.
Our approach to the task is an extension to the CRNN (convolutional recurrent neural network) model proposed by Choi et al. [4]. Hence, we utilize a neural network consisting of a combination of convolutional, recurrent as well as dense blocks. The network is structured in a way that allows to eficiently combine diferent types of features, similar to the approach used by Zangerle et al. [6] for combining low- and high-level features in a neural network model for the task of hit song prediction. †authors contributed equally to this work 1https://www.jamendo.com 2https://github.com/dbis-uibk/MediaEval2019
In the remainder of this section, we will explain our approach to the task in detail. Section 2.1 describes how we use the provided data to train our models. Section 2.2 explains the structure of our model. Finally, Section 2.3 details the runs we submitted for the task. 2.1
For training our models, we used both the precomputed mel-spectrograms and audio features computed with Essentia [2] that were provided by the task organizers. From those, we utilized the melspectrograms as well as high-level features (like genre and danceability), and rhythm features (beats per minute). As training data, the task organizers provided a set of 9,949 songs. We used two diferent training sets. The Base training set is the set provided by the task organizers. For the mel-spectrograms, we use a window of width 1,366 around the temporal center of the song. For the RandomSampling training set, on the other hand, we generated more training samples by taking five random windows of width 1,366 from each song. This training set therefore provides us with 49,745 training samples. 2.2
Our approach uses a neural network model consisting of convolutional, recurrent, and dense layers. The mel-spectrograms are fed into a series of convolutional layers, followed by two recurrent layers using GRU (gated recurrent unit) cells. High-level and rhythm features bypass this part of the network and directly feed into the dense part that comes after the recurrent layers. A schematic overview of the base network structure is given in Figure 1. Some of our submissions feature modifications to this structure. Those modifications are described in more detail in Section 2.3.
Every convolutional block consists of a 2D convolutional layer
followed by batch normalization, ELU activation and a max pooling
layer. After every block, a dropout of 10% is applied. The
convolutional layers have kernel sizes of (
The model is trained using the Adam optimizer [5]. As loss function, we used binary cross-entropy. The output of this model is a vector of probabilities, where every entry in the vector holds the probability for one of the labels. To then decide which labels to assign to a given song, we determine the best probability thresholds for every label individually, using the provided validation set. For this, we use a variation of the elbow method on the ROC curve: We compute the per label ROC curve on the given validation set. Based on this ROC curve we compute the straight line between the left most point and the right most point. This line is used as a reference to find the point on the ROC curve that has the largest orthogonal distance to that line. We then use the corresponding threshold as our decision boundary for that label. 2.3
Using the data and model described above, we build multiple submission runs: • Run #1: The base model, but without the high-level and rhythm features, trained on the Base training set. • Run #2: The base model, but without the high-level and rhythm features, trained on the RandomSampling training set. • Run #3: The base model, but without the high-level and rhythm features, and with an attention mechanism after the second recurrent layer, trained on the RandomSampling training set. • Run #4: The base model, as described in Section 2.2, trained on the RandomSampling training set. • Run #5: The base model, with an attention mechanism after the second recurrent layer, trained on the RandomSampling training set. 3
The results for the submissions described in Section 2.3, as well as how many epochs were used for training the respective model, are summarized in Table 1. As can be seen, all of our runs outperform the popularity baseline, but do not manage to beat the VGG-ish baseline. The best performing runs are run #2 and #3, which both perform very close to the VGG-ish baseline in terms of ROC-AUC and PR-AUC, but significantly worse in terms of F 1 score. On the other hand, our runs require much fewer training epochs to achieve those results. The fact that ROC-AUC and PR-AUC are similar but F1 is worse suggests that our choice of probability thresholds for label assignments is suboptimal.
Comparing the results for our runs which use both spectrograms as well as high-level features as input (#4, #5) with the runs that use only mel-spectrograms (#1, #2, and #3), we observe that the models without high-level features seem to perform better overall. Despite this, the models using high-level features learn faster, with the diference between 8 and 16 epochs of training being much smaller for run #4 than it is for run #1.
We can also see that random sampling for increasing the number of training samples helps to improve results. The PR-AUC increases from 8% in run #1 to 11% in run #2, which only difer in that run #2 uses random sampling while run #1 does not. 4
In this paper, we described our approach to the Emotion and Theme Recognition in Music Using Jamendo task at MediaEval 2019. Our best approach achieved a ROC-AUC score of 0.723, which is slightly worse than the best baseline provided by the task organizers (0.726), but needs much fewer training epochs than the baseline. Potential future work on this approach includes trying diferent ways to determine probability thresholds for label assignments and applying this approach to other data sets for the same task.
Emotion and Theme Recognition in Music Using Jamendo