We present the UAI-CNRL submission to MediaEval 2020 task on Emotion and Theme Recognition in Music. We make use of the ResNet34 architecture, coupled with a self-attention module to detect moods/themes in music tracks. The autotagging-moodtheme subset of the MTG-Jamendo dataset was used to train the model. We show that the proposed model outperforms the provided VGG-ish and popularity baselines.
Convolutional neural networks (CNNs) have been successful in extracting meaningful features for tasks such as image recognition [10, 14] and object detection [10]. In the field of audio processing, CNNs have been used for a variety of tasks, such as automatic † Authors Contributed Equally § https://github.com/alishdipani/Multimediaeval2020-emotions-and-themes-inmusic
Music has been shown to induce a variety of emotions such as happiness, sadness, and anger [7,8,27]. This induction of emotions can be attributed to different intrinsic properties such as tempo, rhythm variations, intensity, mode and extrinsic properties such as the association of music with personal events and previous experiences [12,23]. These emotional responses could also be one of the important motivators for humans to listen to music [20][21][22].
Automatic tagging and detection of emotions of music is a difficult task considering the subjectivity of human emotions. The MTG-Jamendo dataset [4] aims at tackling several such autotagging tasks by providing royalty-free audios of consistent quality with several tags for genre, instruments and mood/theme. The Emotion and Theme Recognition Task of MediaEval 2020 uses the mood/theme subset of the MTG-Jamendo dataset. The task is as follows -given audio, automatically detect one or multiple moods/themes out of 56 given tags, for example, fun, sad, romantic, happy [3].
In this paper, we describe our approach (team name: UAI-CNRL) for this task by using convolutional neural networks to extract features from the mel-spectrograms of the audios and multi-head self-attention to predict the mood/theme by processing the extracted features. Our approach achieves better performance than the baselines. tagging [6], source separation [30], music emotion classification [16] and speaker identification [18].
Transformer networks which use self-attention layers [28] have been successful in tackling language tasks involving long-range dependencies. They have also been used in the field of audio processing for many tasks, such as automatic tagging [29], source separation [5], and speech recognition [2].
A combination of these methods have been demonstrated to achieve state-of-the-art performance [2,9,32]. Inspired by these, we use convolution layers to extract features from mel-spectrograms and self-attention layers to process those features to predict the moods/themes.
We make use of a popular convolutional neural network architecture, the ResNet [10] as a feature extractor to extract compact representations of our data. We pair this with self-attention [28] in order to capture long-term temporal attributes of the given data. We also make use of batch normalization [11] and dropout [24] in order to further regularize the model. We describe the model architecture in this section. Our code and trained model are available at this URL § .
Residual connections make training deep neural networks easier, since they address the problem of vanishing gradients. We make use of a standard ResNet34 architecture to take advantage of this property. This is preceded by two convolutional layers in order to reshape the data into a form that can be fed into the ResNet. Another convolutional layer is used after the ResNet feature extractor to reduce the number of channels.
The MTG-Jamendo dataset consists of tracks of varying lengths, a majority of which are over 200 seconds. Using self-attention, we attempt to capture long-range temporal attributes and summarize the sequence of music representation. Our model architecture is inspired by the works done in [25], which uses multi-head attention along with positional encoding. 2 layers, each consisting of 4 attention heads were used. The input sequence length and embedding size used were unchanged. 3.3.2 SpecAugment. SpecAugment [19] is an augmentation technique used for speech recognition, which involves augmenting the spectrogram itself, instead of the waveform data. SpecAugment modifies the spectrogram by warping it in the time axis, masking blocks of frequency channels, and masking blocks of time steps. This makes the model more robust to missing information in terms of the input speech data as well as frequency information.
Other transformation techniques, such as random cropping and random scaling were used to further augment the given data.
This section describes the details of data pre-processing, architecture and other training details.
We use the mel-spectrograms provided in the MTG-Jamendo dataset for the purpose of training. Random cropping and scaling are used to augment and transform the data into a tensor of length 4096 (approximately 87.4 seconds). Additionally, SpecAugment is used to augment the dataset.
• The input tensor of shape (1,96,4096) is divided into 16 segments length-wise, each new segment being of length 256. • Each segment is then processed through 2 convolutional layers, in order to obtain a representation with 3 channels. • The obtained representation is then passed into the ResNet34 feature extractor, followed by a convolutional layer to obtain an intermediate representation.
• The feature maps are then passed through the self-attention module, followed by a series of linear layers to obtain the final class scores. Dropout is used to regularise the training process. • The model returns the outputs of the self-attention module and the feature maps (after passing them through the linear layers). Both outputs are used to compute the loss and perform backpropagation, but only the outputs of the selfattention module are used to make predictions.
The model was trained with the Adam [13] optimizer, at a learning rate of 1e-4, for 35 epochs. The values of 𝛽 1 and 𝛽 2 were set to 0.9 and 0.999 respectively. Binary cross entropy loss was used as the loss function.
The proposed model produces results that improve on those of the given VGG-ish and popularity baselines. We obtain an ROC-AUCmacro metric of 0.7360 and a PR-AUC-macro metric of 0.1275. For comparison, the baseline VGG-ish model produces an ROC-AUC macro of 0.7258 and a PR-AUC macro of 0.1077. Detailed results can be found in Table 1.
In this section, we discuss other approaches that we considered towards the problem statement. These may be used as pointers towards future work on tasks involving this dataset. Our approach can be broken down into two parts -first, the extraction of features from the audio data and second, processing the extracted features to predict the moods/themes. Both these parts could be potentially improved upon, and we mention a few ways to do so below.
With respect to feature extraction:
• Using a wider range of features to aid the classification task instead of using mel-spectrograms. For example, the LEAF frontend proposed by [1] can be used for this approach. • Using self-supervised approach to extract features, such as wav2vec 2.0 [2]. This would also reduce reliance on labelled data. • Using temporal convolutional networks [15] to extract features directly from audio instead of using mel-spectrograms.
With respect to the processing of extracted features:
• Using dual path processing inspired by [17] in order to capture long-term dependencies while also reducing computational load. • Exploring ways of processing the raw audio data with more powerful models, such as WaveNet [26] in order to obtain better insights into the dataset, and theme recognition in general.
We thank Shell Xu Hu for helpful discussions.
MediaEval'20, December 14-15 2020, Online