We present USC SAIL's submission to the 2020 Emotions and Themes in Music challenge: an ensemble-based convolutional neural network (CNN) model trained using various loss functions. In this work, we investigate the efect of diferent loss functions and resampling strategies on prediction performance, finding that using focal loss improves overall performance on the provided imbalanced, multi-label dataset. Additionally, we report results from varying the receptive field on our base classifier-a CNN-based architecture trained using Mel spectrograms-which also results in better model performance. We conclude that the choice of the loss function is paramount for improving on existing methods in music tagging, particularly in the presence of class imbalance.
Content-based automatic music tagging is a challenging task: a
robust system must accurately predict multi-label tags (such as
mood, theme, or genre) associated with a piece of music, regardless
of the frequency of such labels. Great strides in the field have been
enabled recently by the release of large high-quality music datasets,
like the MTG-Jamendo dataset [
Models based on convolutional neural networks (CNNs) are an
efective choice for a wide variety of audio-based tasks, including
speech recognition [
In sparse multi-label tasks, loss functions are susceptible to
being overwhelmed by the large number of negative labels [
We test focal loss, originally developed for computer vision
models, as a way to train the model to emphasize improving predictions
on samples that have lower confidence [
We use the provided subset of data from MTG-Jamendo and expand
the training set by using instances from the Music4All dataset [
For feature extraction, we first resample all audio to 16 kHz, and then extract 128-bin Mel spectrograms using a 512-sample FFT, with a window size of 32 ms and 50% window overlap. 3 3.1
We use a modified short-chunk CNN with residual connections for
our model, based on the architecture presented by Won et al. [
We explore three loss functions aimed at increasing average classwise performance on an imbalanced dataset. Where applicable, we modify the loss functions for multi-label classification.
3.2.1 Focal Loss. We implement a multi-label version of focal
loss [
(1 − ) ( ) (1) Õ
BCE Loss 0.150 0.766 Focal Loss 0.156 0.778 CB Focal Loss 0.153 0.773 DB Focal Loss 0.153 0.768 Ensemble 0.161 0.781 VGG-ish-Baseline 0.107 0.725
Popular-Baseline 0.031 0.500
Table 1: Test-set performance of our model trained
using various loss functions. BCE stands for "binary
crossentropy"; CB for "class-balanced"; DB for
"distributionbalanced."
where and are equal to and , respectively, if is a label for
the sample . Conversely, and are equal to 1 − and 1 − if
is not a label for . In our experiments, we use = 0.25 and = 2,
as recommended by [
Here, suppresses the contribution to loss from the relatively well-classified examples, focusing instead on harder-to-classify examples. In the case of our dataset, where no single label is present in a majority of instances, the negative classes may be easy for the model to learn. Thus, we instead want to focus on the harder cases where a given label is present. provides an additional weight term for positive and negative classes.
3.2.2 Class-Balanced Loss. We also implement a class-balanced
version of focal loss [
1 − (1 − ) ( ) =1 where is the number of samples in the training set in which label appears and is a hyperparameter. We set to 0.995 for our experiments.
3.2.3 Distribution-Balanced Loss. Lastly, we use
distributionbalanced loss, which was first presented by Wu et al. [
(1 + −( − ) ) =1 1 (1 − )(1 + ( − ) ) Here, is a scale factor for the negative logits, controlling for the preponderance of negative labels, while is a class-specific rebalancing weight that tries to close the gap between the expected number of samples and actual number of samples for a given class after resampling. Wu et al. used a BCE variant of this equation, as shown above; we implement a focal-loss-based function for our study. 4 4.1
We submitted three models to the challenge: the short-chunk CNN model described above using focal loss, an ensemble model which combines multiple CNN-based models trained using the above four (2) (3)
BCE Loss 0.179 0.163 0.101 Focal Loss 0.174 0.171 0.113 CB Focal Loss 0.173 0.170 0.104 DB Focal Loss 0.173 0.170 0.105
Ensemble 0.182 0.179 0.109 Table 2: Class-wise subset performance of various loss functions in terms of PR-AUC. diferent loss functions, and an identical ensemble model that is trained without any outside data (Million Song Dataset or Music4All).1 4.2
We display the PR-AUC and ROC-AUC test-set performance of each loss function approach against provided baselines in Table 1.
We find that the model trained using focal loss produces the best performance both in terms of PR-AUC and ROC-AUC. All variants of focal loss outperform binary cross-entropy, and our final ensemble of averaging the predictions from models trained using the four diferent loss functions achieves the highest performance.
Additionally, to further investigate the efects of using diferent loss functions on class-wise performance, we split the label set into head, middle, and tail classes, based on frequency in the training set. Head classes contain over 550 samples, middle classes contain between 200 and 550 samples, and tail classes contain at most 200 samples. The results in terms of PR-AUC are displayed in Table 2.
Indeed, we find that the focal loss-based methods perform better than BCE loss on the less-frequent classes (both middle and tail subsets). In our experiments, this comes with a slight penalty in performance on the head classes, but leads to overall better performance, as well as a better-performing ensemble model.
Lastly, we try class-aware sampling [
We present an ensemble-based CNN model trained using focal loss for this year’s submission to the 2020 Emotions and Themes in Music challenge. We find that focal loss helps predict labels that do not occur frequently in the dataset, and that emphasizing correct predictions of these labels results in better model performance. We posit that the choice of a loss function is an essential consideration when developing a prediction model for multi-label classification, particularly in music processing. Future work will determine if our approach generalizes to other methods for automatic music tagging.
This study is done at the Center for Computational Media Intelligence at University of Southern California’s Signal Analysis and Interpretation Laboratory and supported by research awards from Google and the U.S. Chamber of Commerce Foundation. 1Our code can be found at https://github.com/usc-sail/media-eval-2020