Our team, "YL-UTokyo", participated in the task: Emotion and Theme Recognition in Music Using Jamendo. The goal of this task is to recognize moods and themes conveyed by the audio tracks. We tried several Convolutional Neural Networks with diferent architectures or mechanisms. As a result, we nfid that a relatively shallow network achieved better performance on this task.
We participated in one of the tasks in MediaEval 2019: Emotion and Theme Recognition in Music Using Jamendo [2]. This task involves the prediction of moods and themes conveyed by a music track. Moods are often defined as feelings conveyed by the music (e.g. happy, sad, dark, melancholy) and themes are associated with events or contexts where the music is suited to be played (e.g. epic, melodic, christmas, love, film, space).
In the task, there are three types of audio representations, including traditional handcrafted audio features, mel-spectrograms, and raw audio inputs. We only used the mel-spectrograms as input to train our model (Figure 1). We tried several Convolutional Neural Networks (CNNs) to find a suitable model for this task. The simplest but efective model we tried is provided by the organizers. It only consists of five convolutional layers and one dense layer at last. We also tried other models with more layers, but they didn’t always achieve better results. As a result, the model that achieved the best performance in our experiments is a shallow neural network with only six convolutional layers and one dense layer.
Image classification performance has improved greatly with the advent of large datasets such as ImageNet [5] using CNN architectures such as VGG [9], Inception [10], and ResNet [6]. There are also many research of music emotion recognition or music classification using CNN architectures [ 4, 7]. Even though statistical machine learning (e.g. Support Vector Machine [8] and Random Forest [1]) can still achieve good performance in some tasks, deep learning, especially CNN based method, is more popular and can achieve better performance in most tasks. For large-scale datasets, deep learning is much more practicable than statistical machine learning.
Mel-spectrogram
Conv 3x3x32
MP (
MP (
MP (
MP (
MP (
MP (
We concentrated on finding the most suitable CNN architecture for the task. The baseline is a simple but efective model consisting of five convolutional layers and a final dense layer. We also tried other models with deeper architecture. We tried models with 6, 16, 18 or 25 convolutional layers. In particular, the most shallow model we considered is a fully convolution neural network with ELU activations, six 3x3 convolutional layers, and 32, 64, 128, 256, 512, 256 units for each layer respectively (Table 1).
We also tried some models with the residual architecture [6]. The convolutional block consists of 1x1, 3x3 and 1x1 convolutional layer sequentially. This is the architecture for inputs and outputs with the same size and unit number. For the block that maps inputs Shengzhou Yi, Xueting Wang and Toshihiko Yamasaki x to outputs with smaller size and more units, the stride of 3x3 convolutional layer is two and the shortcut is a 1x1 convolutional layer for downsampling (Figure 2). 3.2
The dataset includes 17,982 music tracks with mood and theme annotations. The split for training, validation and test is about 2 : 1 : 1. In total, there are 56 tags, and tracks can possibly have more than one tag. There are three types of audio representations, including traditional handcrafted audio features, mel-spectrograms, and raw audio inputs. The traditional handcrafted audio features are from Essentia [3] using the feature extractor for AcousticBrainz. These features were used in the MediaEval genre recognition tasks. The number of mel-bands of the mel-spectrograms is 96. The raw audio inputs are in MP3 format with 44.1 kHz sampling rate. 3.3
We only used the pre-computed mel-spectrograms (Figure 1) as inputs, and we used diferent data augumentation methods in training, validation and test dataset. Let T be the length of input section [frame]. For training dataset, we randomly cropped a T -frame section from each audio track in every epoch. For validation and test dataset, we respectively cropped 10 and 20 T -frame sections from each audio track at regular intervals. We averaged the predictions over all sections of each audio track. The length of input section T is 1,280 frames. We trained our networking using Adam with the batch size of 64 and the learning rate of 0.001. 4
We compared the performance of the models that have diferent architectures or mechanisms in Table 2. Suprisingly, the model that achieved the best performance in our experiments was a relatively shallow model that only consists of six convolutional layers, the architecture of which is detailed introduced in Section 3.1. Moreover, the top-5 and bottom-5 tag-wise AUCs of the 6-layer model are showned in Table 3. The performance achieved by the best 6-layer model is in the fifth place among all 29 submissions.
The network with 25 convolutional layers consists of one 7x7, 24 3x3 convolutional layers and five max pooling layers for downsampling. It’s commonly believed that deep models can achieve a better performance in image classification task. However, the model with deep architecture didn’t always achieve a better performance in this task. We also tried residual architecture that commonly used for improving the performance of neural networks. However, the models with residual architecture didn’t have an advantage in performance. 5
The number of samples (18K) in the dataset is relatively smaller than some image datasets (e.g. CIFAR-10: 60K, MS-COCO: 200K, ImageNet: 517K) and the length of audio data (>30s) is relatively longer than some sound datasets (e.g. UrbanSound8K: <4s, ESC-50: 5s, AudioSet: 10s). According to our experience, the generalization ability of models is especially important in this task. Therefore, it is reasonable that relatively shallow VGG-based network with strong generalization ability can achive better performance.
In the future, we plan to use all of the audio representations because we think it is interesting that we treat audio recognition as a multimodal task. Traditional handcrafted audio features and the raw audio inputs may bring great improvement in the performance of our model. 6
In our experiments, we applied several convolutional neural networks to recogonize the emotion and theme of music. A shallow VGG-based network that consists of six convolutional layers achieved the best performance with PR-AUC-macro of 0.1256 and ROC-AUC-macro of 0.7532. We think that the generalization ability of the models is very important in this task. The link to our source code: https://github.com/YiShengzhou12330379/Emotionand-Theme-Recognition-in-Music-Using-Jamendo.
Emotion and Theme Recognition in Music Using Jamendo