MediaEval 2020 provided a subset of the MTG-Jamendo dataset, aimed to recognize mood and theme in music. Team HCMUS proposes several solutions to build eficient classifiers to solve this problem. In addition to the mel-spectrogram features, new features extracted from the wavenet model is extracted and utilized to train the EficientNet model. As evaluated by the jury, our best result achieved of 0.142 in PR-AUC and 0.76 in the ROC-AUC measurement. With fast training and lightweight features, our proposed methods are potential to work well with deeper neural networks.
Emotions and Themes in Music task in MediaEval [
To be able to solve this task, the authors have tried many methods,
using many kinds of models, input features or loss functions. Our
best result is an ensemble of two kinds of diferent methods, one
using provided mel-spectrogram features with EficientNet model
and the other using waveNet features with MobileNetV2 model
[
Data augmentation is important when training neural network
model. Traditional audio augmentation methods try to modify the
speed of the waveforms or alter the original signal samples with
noises, this method need much computational cost. With
SpecAugment approach[
WaveNet model is applicable in many problem of signal
processing, time series forecasting and music generation[
We follow many approaches which include two main inputs: melspectrogram features and wavenet features. 3.1
As in the figure, the green part shows the audio with only one label mood/theme, the yellow part shows the audio with 2 to 3 moods, the red part shows the audio with more than 3 moods. Number of sample audio for training is 9949 with a total of 17885 moods. On average, each class will have 319 audio with a standard deviation of 202.75. The maximum number of moods of an audio is 8. Mood / theme that appears most is happy with 927 audios.
We can see that the data is extremely unbalanced, and some classes have no audio representing it entirely. Therefore, it is necessary to have a way to reduce the complexity of the data. 3.2
Data preprocessing 3.2.1 Data balance: To reduce the ambiguity of the data, the authors try to change each audio’s label from multi-label to single label, keeping the most significant tag of each audio, reduce standard deviation, give preference to moods with little data.
3.2.2 Features preprocessing: Wavenet feature: Based on the
idea of using wavenet as classifier for raw waveform music audio
[
Based on the dataset’s statistic, the minimum length of audio is 30 seconds and based on the limitation of the authors’ training machine, sound samples greater than 400 seconds in length will be
trimmed to take the middle part. Each sample is again randomly cut for 30 seconds and then extract features from them. This approach is quite subjective and causes loss of input data, we planned to experiment with random cutting from 400 seconds of audios after each epoch. The output of a 30 seconds audio is 16 frames multiply with 937-time steps.
Mel-spectrogram: Each sample feature has 96 channels and time frames are randomly cropped to 6950 after each epoch. 3.3
SpecAugment: To train models more eficiently, the authors include segmentation method SpecAugment which was introduced by Google. This method masks blocks of consecutive time steps and channels in each mel-spectrogram. The result when using this method is increased significantly, PR-AUC-macro is improved from 0.134 to 0.139.
Each input have 70% chance to be augmented by using SpecAugment, each mel-spectrogram will have two blocks of time masking and two blocks of channel masking. 3.4
Since both mel-spectrogram features and wavenet feature can be expressed as images, the authors use convolutional models such as MobileNet and EficientNet. The mel-spectrogram features are passed to EficientNet-B0, on the other hand, the waveNet features are passed to MobileNetV2 and EficientNet-B7. Because waveNet features are not large enough to fit EficientNet-B7, the authors duplicate the number of channels so that these kinds of features can be used.
In addition, we also tested the SVM model, InceptionNet, Resnet,
and to capture the long-term temporal characteristics, self-attention
was added as in the method of AMLAG 2019[
For the loss function, binary cross entropy loss (BCE) is applied
for both MobileNet V2 and EficientNet. The authors also try to
apply Focal Loss[
Our experiments are done on a computer server with Nvidia Quadro k6000 graphic card. Method A,B and D are not submitted to the challenge. We realize that data balancing method leads to a better result comparing to the original dataset with default labels. Based on the experiments on the validation set, our ensemble models are calculated with factors of 0.7 and 0.3 for mel-spectrogram features and wavenet features to gain the best results.
D E (run3) F (run1) G (run4)
Mel-spectrogram EficientNet-B0 Mel-spectrogram EficientNet-B0 with data processing Mel-spectrogram EficientNet-B0 using augmentation WaveNet MobileNetV2 WaveNet EficientNet-B7
Ensemble C and D Ensemble C and E PR-AUC-macro 0.127 0.134 0.139 0.102 0.105 0.1413 0.1414
The EficientNet model was shown to be more eficient than previous models in the mood and theme classification problem. The results can be improved by training mel-spectrogram features on other more complex EficientNet models.
Although the result when training on wavenet features is not higher than mel-spectrogram features, but when assembling two models using these features, the results are improved, it shows that wavenet can extract other aspects of the dataset. Because the wavenet features were extracted by using a pretrained model, the augmentation methods have not been fully applied, for the future work, there are still more improvements to come when training WaveNet-style autoencoder models on the Jamendo dataset.
Research is supported with computing infrastructure by SELAB and AILAB, University of Science, Vietnam National University Ho Chi Minh City.