We present Mirable's submission to the 2021 Emotions and Themes in Music challenge. In this work, we intend to address the question: can we leverage semi-supervised learning techniques on music emotion recognition? With that, we experiment with noisy student training, which has improved model performance in the image classification domain. As the noisy student method requires a strong teacher model, we further delve into the factors including (i) input training length and (ii) complementary music representations to further boost the performance of the teacher model. For (i), we ifnd that models trained with short input length perform better in PR-AUC, whereas those trained with long input length perform better in ROC-AUC. For (ii), we find that using harmonic pitch class profiles (HPCP) consistently improve tagging performance, which suggests that harmonic representation is useful for music emotion tagging. Finally, we find that noisy student method only improves tagging results for the case of long training length. Additionally, we ifnd that ensembling representations trained with diferent training lengths can improve tagging results significantly, which suggest a possible direction to explore incorporating multiple temporal resolutions in the network architecture for future work.
Emotions and themes are high-level musical attributes that are
abstract and highly subjective. Obtaining emotion labels typically
require human annotation, which can be time consuming and
potentially costly. Is it possible to use semi-supervised learning
techniques, such that we can leverage on unlabelled music tracks to
learn emotion tags, while only using a small amount of labelled
data? Following this question, we intend to explore the usage of
noisy student training [
We extract Mel-spectrograms with 128 bins from raw audio using a sampling rate of 44.1kHz, and the Mel-spectrograms are downsampled with an averaging factor of 10 along the temporal dimension. The number of time steps for each Mel-spectrogram vary baseline long-normal long-hpcp long-hpcp-noisy short-normal short-hpcp short-hpcp-noisy ensemble
ROC-AUC
PR-AUC
F-Score
according to the training strategy, which will be discussed in
Section 2.3. For data augmentation, we perform time masking and
frequency masking, similar to the idea in SpecAugment [
As shown in Figure 1, our base model architecture is similar to
CRNN [
For the long training length mode, we use the first ≈ 185 seconds of the track, which corresponds to 1600 time steps in the Melspectrogram after average pooling. For the short training length mode, we chunk each track into samples of length ≈ 9.25 seconds, which corresponds to 80 time steps in the Mel-spectrogram after average pooling. During evaluation, we average the logits of all chunks to obtain the final output for each track. 2.4
HPCP [
Noisy student training [
In this work, we use the corresponding HPCP models for each long and short training length mode as the teacher model. We only use the predictions which are > 0.1 as positive pseudo-labels, and those < 1−6 as negative pseudo-labels. Both decision thresholds are determined by conducting an empirical evaluation on the predicted value distribution using the teacher model, carried out on the training and validation set. We take the leftmost 5% percentile for the negative label distribution, and the rightmost 5% percentile for the positive label distribution to ensure better confidence. 2.6
Finally, we investigate the results of combining the output of both long and short training length models, by simply taking the weighted sum of their best models: = ·ℎ + (1 − ) ·. We use the validation set to find the ratio which gives the best results. 3
For the training length factor, we find that models trained with long input length perform better in ROC-AUC, but models trained with short input length perform significantly better in PR-AUC. According to Table 2, this is because the former has a higher TNR, while the latter has a higher TPR. Since PR-AUC focuses more on the minority class (in this case the positive class) and ROC-AUC focuses on both, the latter model scores better in PR-AUC. We also find that adding HPCP improves tagging results consistently for both cases, which suggests that harmonic representation is important for music emotion recognition.
For noisy student training, the results are rather inconclusive. We find slight improvements in the long training length case, but the result degrades for the short training length case. Also, we only run noisy student training for 1 iteration, as we find the results consistently degrade for subsequent iterations. Additionally, we try to add more unlabelled tracks from the Lakh MP3 dataset (≈ 45, 000 30 seconds track) to increase the training dataset size, but we do not observe any performance improvement. We infer that noisy student method might not necessarily work well for music emotion recognition tasks, due to the abstract nature and subjectivity of emotion and theme labels. Hence, a small subset of emotion labels might not be suficient to represent the full dataset.
For model ensembling, we choose to ensemble the ‘long-noisy’ model and the ‘short-normal’ model. We find that = 0.7 is optimal through our validation set, hence suggesting that the final output gives more weightage to the short training length model. From the test set results, we can also see that this ensemble method improves the tagging performance significantly, which suggest that combining diferent views of audio in terms of temporal resolution can produce better learnt representations. 4
While investigating the related work, we find that this work still uses a relatively long training length (even for short length we use ≈ 9 seconds, as compared to previous works with ≈ 2 to 5 seconds), and low temporal resolution, which we intend to change in our future work. For future work, we are interested in tweaking the network architecture to capture views of diferent temporal resolutions in the audio sample. We would also like to explore using noisy student training with diferent model architectures and datasets of a much larger scale.