Music emotion recognition (MER) is a key issue in user contextaware recommendation. Many existing methods require hand-crafted features on audio and lyrics. Here we propose a new end-to-end method for recognizing emotions of tracks from their acoustic signals and lyrics via multimodal deep neural networks. We evaluate our method on about 7,000 K-pop tracks labeled as positive or negative emotion. The proposed method is compared to end-to-end unimodal models using audio signals or lyrics only. The experimental results show that our multimodal model achieves the best accuracy as 80%, and we discuss the reasons of these results.
Music emotion recognition (MER) is a core technology of contextaware music recommendation. Users usually want music to amplify their emotions while partying or driving, for examples. Music recommendation using content-based MER allows music’s emotion to be aligned with that of users in these scenarios. However, this is challenging because it is still unclear how music is causing emotions. It is known that numerous factors such as tone, pace, and lyrics are related to determine music emotion.
Existing studies tackle MER in various ways. They mainly
formulate MER as either a classification or a regression problem. Laurier
et al. use four emotion categories in [
Here we simplify MER as a polarity emotion (positive / negative) classification of tracks to reduce the uncertainty from many emotion categories, considering an application to simple music recommendation scenario. We propose an end-to-end multimodal neural network models without an additional feature engineering process. We also create new dataset including tracks served on a Korean music streaming service to guarantee the high-quality data. 2
We describe our dataset from a famous Korean music streaming service, NAVER Music 0. This consists of 3,742 positive and 3,742 negative tracks with their lyrics, and they are represented into mel-spectrograms. How could we seperate a positive and negative track? There are tracks tagged by editors in NAVER Music. We use a predefined emotion word dictionary to seperate positive and negative tags. For instance, positive emotion words are ‘happy’, and ‘cheerful’, while negative emotion words are ‘sad’, and ‘lonely’. Then, we filter out the tracks whose tags include both positive and negative words. We reject the tracks whose length is less than a minute or lyrics include less than 30 words. We use the first one minute of each mel-spectrograms, and only use noun, verb, adjective, and adverb in words of lyrics. Finally, each mel-spectrogram is represented as a 128 by 1024 matrix including 128 mels and 1024 time slots corresponding to one minute length of acoustic signals. Also, we have (27,496, 400) word vectors where vocabulary size |V | = 27, 496 and the maximum length of word sequences is 400. 3
ua = [Maxpoolinд(Conv(xa))]5
Five 1D convolution layers whose filter sizes are all 3 have 128, 128, 128, 64, and 64 output of filters, respectively. Filter sizes of five max pooling layers are 3, 3, 3, 2, and 2. We use the exponential linear unit (elu) function as a non-linear function of convolution layers.
On top of that, we put two RNN layers (GRU) whose output dimensionality is 64 and one fully connected layer whose weight matrix is F CaW is to build an audio embedding vector va with length 64 before merging with lyrics branch.
va = F CaW {GRU 2(ua)}
At the bottom of the lyrics branch, there is an embedding layer (weight matrix: F CeW ) whose output dimensionality is 200 followed by an 1D convolution layer whose filter size and number of output are 3 and 250 where an input word vector is xl. On top of that, we put global 1D max pooling layer because it is more robust to noise words than non-global one. As in the audio branch, there is one fully connected layer whose weight matrix is F Cl W on it to build an lyrics embedding vector with length 64.
vl = F Cl W {GlobalMaxpoolinд(Conv(F CeW xl))}
We concatenate two branches because it shows the best performance. Lastly, we produce final output by taking this concatenated vector as an input of one fully connected layer (weight matrix: F CmW ) whose output dimensionality is 64 followed by a softmax layer to compute the binary cross-entorpy loss.
o = So f tmax (ReLU (F CmW {Concatenate(va, vl)}))
As a result, output vector o includes two values, the probabilities of a positive and negative emotional track. 4
We compare the classification accuracies of models using audio, lyrics, and both. We test five unimodal models and one multimodal model which consists of the best unimodal models for audio and lyrics as in Figure 1. To test these models, we implemented them (1) (2) (3) (4)
Data Model Audio CNN
RNN
CNN+RNN Lyrics CNN
RNN Both CNN+RNN, CNN with Keras on Tensorflow while using Tesla M40 GPU. Table 1 shows the classification accuracies for each model. We randomly split the dataset into 90% for training and the rest for validation, and obtain the results shown in Table 1 after 5 runs of the test for each model. 1D CNN + RNN model is the best among models using audio. 1D CNN model is the best among models using lyrics. The reason why 1D CNN is better than RNN to predict from lyrics may be that the word sequences are too long. It is also notable that the model for lyrics works better than that for audio. Overall, the multimodal model using audio and lyrics shows the best accuracy, 0.8046. Figure 2 presents validation accuracy and loss of the best model for each modality (audio, lyrics, both). The model for lyrics and both shows little more stable convergence than that for audio. 5
We define MER as a polarity emotion classification and propose a multimodal neural network model trained in an end-to-end manner without additional feature engineering. We present lyrics are better features than audio on our problem, and our multimodal models proves the best accurcacy, 80% compared to unimodal models. We will further investigate end-to-end deep learning strategies with more tracks and emotional categories. Furthermore, we will apply our method to the context-aware music recommendation service of Clova, Cloud-based AI-assistant platform developed as a collaboration project by NAVER-LINE1.