We present a deep convolutional neural network approach for emotions and themes recognition in music using the MTG-Jamendo dataset. The model takes mel spectrograms as input and tries to leverage translation invariance in the time dimension while allowing convolution filters to depend on the frequency. It has led the lileonardo team to achieve the highest score of the 2021 MediaEval Multimedia Evaluation benchmark for this task1.
Emotions and Themes Recognition in Music using Jamendo is a
multi-label classification task of the 2021 MediaEval Multimedia
Evaluation benchmark. Its goal is to automatically recognize the
emotions and themes conveyed in a music recording by means of
audio analysis. We refer to [
In the wake of its successes in computer vision, the use of
convolution neural networks has become a very common approach for
audio and music tagging and often leads to state-of-the-art results
(see [
A number of previous works highlight the importance of the
choice of filters with respect to the frequency dimension. In [
As it is often the case with multi-label tagging tasks, there is
a pronounced imbalance between positive and negative classes.
The proportion of positive examples for each label in the training
set ranges from 0.6% to 9.3%. To address this issue, the authors of
[
According to the results of [
The task of recognizing emotions and themes seems particularly well adapted to training on small excerpts of music. Indeed, while an instrument tag can be attributed to a song when it is only present on a small part of it, emotions and themes can often be recognized on most parts of the song. Moreover, by reducing the input time to lengths as small as 1 second, we observed that the ability of the network to perform its task did not radically diminish.
We have computed 128 frequency bands mel spectrograms from the original audio files by keeping the sample rate of 44100 Hz, using a ft window length of 2048 with 50% hop length and a lowpass filter with maximum frequency set to 16 kHz (see Figure 1 for a comparison between these mel spectrograms and the original 96 bands mel spectrograms provided by the MTG-Jamendo dataset).
We found that a simple CNN model overfitted rather quickly when trained on small random chunks of the tracks, unless a carefully engineered learning rate scheduling scheme was used. When we tried to train it on the 128 bands spectrograms, it performed significantly worse and overfitted even more quickly.
In order to try and overcome the class imbalance issue, we used a weighted loss, where for each label, the weight of the positive class is inversely proportionate to its frequency in the training set: (, ) = − =1 1 Õ
2 2 1 + log( ) + 1 + (1 − ) log(1 − ) , where is the frequency of the positive class in the train set for label and = 56 is the number of labels.
We also adopted an input stem constituted of three convolutional
layers followed by a max pooling layer, inspired by [
In order to give the model the possibility to detect diferent features at diferent frequencies, we tried and let the convolution iflters of the higher level layers vary with the frequency. Despite the six fold increase in parameters with this approach, it gave better results on the 128 bands mel spectrograms.
The choice of this architecture comes from the hypothesis that the problem is invariant by translation in the time dimension, that the first layers compute low level visual features, and that at a higher level, the characteristic features depend on the frequency and are not similar on the low, middle or high frequency parts of the spectrogram.
We trained the network on randomly chosen small excerpts of each track. Final predictions are obtained by averaging the outputs for every small size segment in the track. 4
We evaluated the model described in Table 1, first with normal 3 by 3 convolutions, then with frequency dependent convolutions in layers filter1–4.
We trained these two models both on the original 96 bands mel spectrograms provided by the MTG-Jamendo dataset and on the 128 bands ones described in the previous section. We also tried input time lengths of 128 and 224, corresponding to excerpts of approximately 3s and 5.2s. Finally, we trained the models with a classic binary cross-entropy loss, with the weighted loss described in section 3 and with the focal loss with parameters = 0.25 and = 2. We then averaged the predictions of the trained models for any given choice of these hyperparameters to obtain the results in Table 2.
All models were trained for 200 epochs using SGD with Nesterov
momentum of 0.9, a learning rate of 0.5 and weight decay of 2e-5.
We used a cosine learning rate decay, mixup with = 0.2 [
A PyTorch implementation is available online2. 2https://github.com/vibour/emotion-theme-recognition The same training process was used for all models both for the sake of simplicity and in order to provide better comparison possibilities. However, the models with frequency dependent convolutions have a much higher number of parameters and may benefit from more regularization. It seems that with appropriate regularization, no overfitting occurs and the network could be trained for a longer number of epochs.
The results obtained on the test set, when broken down by label, are very diferent from those obtained on the validation set, in a way that is very consistent across all the models we have trained. On average, the score is significantly higher on the test set than on the validation set. Some labels (deep, summer, powerful...) are always better predicted on the test set whereas other labels (movie, action, groovy...) are always better predicted on the validation set. The fact that this is consistent across the models seem to show an inherent diference in the data rather than to be indicative of a high variance.
An increase in the input time length results in a small improvement. It is not clear whether this improvement is due to the additional input data seen by the network resulting in a virtual increase in the number of epochs, if it comes from a better management of side efects introduced by padding, or from better averages seen by the fully connected layer. Since the receptive field at the average pooling layer is independent of the input length, the model cannot actually use features present at a longer time scale.
The actual influence of the number of bands in the spectrogram and of the frequency of the low pass filter applied before the ft is not clear and would need further study.
The introduction of residual blocks in the place of the four filter layers seemed to provide a small but limited improvement. A resnet version of the model would be a good candidate to further improve the results.
The author is grateful to François Malabre for very helpful discussions.