Continuous-time prediction of self reported musical emotions is a challenging problem with many applications. However, there are relatively few studies on design of Deep learning models for the above problem. Existing methods for the same problem has used LSTMs, with modest success. In this work, we describe an attentive LSTM based approach for emotion prediction from music clips. We postulate that attending to speci c regions in the past gives the model, a better chance of predicting the emotions evoked by present notes. We validate our model through extensive experimentation on the standard 1000 Songs for Emotional Analysis of Music dataset, which is annotated with arousal and valence values in continuous time. We nd that the attentive models signi cantly improve the prediction performance of arousal and valence over vanilla LSTM, both in terms of R2 and Kendall- metrics.
Music is well known as an e ective means of eliciting emotions in listeners [
In this study, we propose to use the attention mechanism with a deep RNN
structure composed of LSTMs, to predict the perceived emotion in each de ned
time frame of music continuously. We use the well known ComPare 2013 [
Current state-of-the-art methods for audio sentiment analysis are mostly based
on deep neural network. RNNs are a class of neural networks that are suited
for time series data. They use the outputs of network units at time t as input
to other units at time t + 1. This allows RNNs to store temporal information
present within the input data. Though in theory, RNNs can keep track of
arbitrary long-term dependencies in the input sequences, practically they su er from
the problem of vanishing gradients [
Though RNN-LSTMs are useful, it must be acknowledged that the di culty
of successfully capturing the context increases with length of input sequence
[
Most of the MIR tasks utilizing deep RNN-LSTM structures need
considerable amount of training data to produce good results. In the domain of music
emotion recognition, one such widely used dataset is the 1000 Songs for
Emotional Analysis of Music [
In the present work, we use the 1000 Songs for Emotional Analysis of
Music dataset [
The key component of an LSTM is the cell state C, through which the relevant context/dependency information between the elements in the input sequence ows, with careful regulations by the forget gate (f ), input gate (i) and the output gate (o). Intuitively, it can be understood that the forget gate (equation1) decides which information is irrelevant and can be thrown away from the cell state.
ft = (Wf [ht 1; xt] + bf ) Next, the input gate (equation2) regulates what new information needs to be stored in the cell state, with the help of a vector of new candidate values (equation3).
Thus, an update to the cell state is performed (equation4).
it = (Wi [ht 1; xt] + bi) C~t = tanh(WC [ht 1; xt] + bC )
Ct = ft
Ct 1 + it ~ Ct Finally, the output gate (equation5) decides the output of the network (equation6), based on a ltered version of the cell state.
ot = (Wo [ht 1; xt] + bo)
ht = ot tanh(Ct) In the above equations, xt and ht denote the input and output at time t. Each W and b denote the associated weights and biases of each of the gates. The LSTM gates use the sigmoid function as the activation function. Though RNNLSTMs (equation 7) provide a great way to carry relevant information from one step to the next through the cell state, the di culty increases with length of input sequence. Basically, the neural network has to compress all the necessary information of an input sequence into a single xed length vector, the last hidden state (equation 8). This may become problematic for the neural network in case of longer input sequences.
ht = f (xt; ht 1) c = q(h1; h2; : : : ; hT ) = hT (1) (2) (3) (4) (5) (6) (7) (8) 3.3
To address this issue, Bahdanau et. al. [
p(yijy1; y2; : : : ; yi 1; x) = g(yi 1; si; ci) (9) si = f (si 1; yi; ci) ci = X
Tx j=1 ijhj Each time, the context vector ci is calculated as a weighted sum of all the hidden states (equation 12). The idea being, for each output, the context vector will attend on those parts of the input sequence, which are more relevant for that particular output, by assigning higher weights to the associated encoderside hidden states, using an alignment model. In equation 13, eij is the score of how well inputs around position j and the output at position i align or match. ij =
exp(eij)
PkT=x1 exp(eik) eij = a(si 1; hj) (12) (13)
We use a modi ed form of attention, as the problem we are tackling is music mood regression and not machine translation, thus there is no need for a decoder side architecture. The encoder encodes the input into a set of hidden states and the attention is applied on them to produce target arousal and valence values. Generally, the encoder in neural machine translation reads the input sequence x = (x1; x2; : : : ; xT ) { which is a sequence of vectors { and produces the hidden states (h1; h2; : : : ; hT ), using some RNN approach like in equation 7. In case LSTM is used, equation 7 takes the speci c form of equation 6. In most cases, the whole set of hidden states (h1; h2; : : : ; hT ) are available to compute the context vector for the translation. So, all the hidden states are used for the context vector , either with attention (equation 11) or without (equation 8). This also makes sense for natural language processing, as the translation of an input xt might depend on any input xi, where both i < t or i > t are possible.
But, when we listen to music, the emotion associated with the music at tth second is seldom in uenced by the music following it. Rather, it might be argued, that the associated emotions at the tth second will be more dependent on any music preceding it. Let the output be y = (y1; y2; : : : ; yT ). For the tth output, yt, it will be a function of a) the present hidden state ht, b) the previous output yt 1, c) the unique context vector ct.
p(ytjy1; y2; : : : ; yt 1; x) = g1(ht; yt 1; ct) (10) (11) (14) (15) The unique context vector ct depends on the sequence of annotations (h1; h2; : : : ; ht 1), and is computed as a weighted sum of these annotations hj. So, the model is attending to each hj, corresponding to each of the inputs.
t 1
ct = X
j=1
tjhj
As in Bahdanau et. al.'s [
exp(etj ) Ptk=11 exp(etk) (16) (17) 4 4.1
10-fold cross validation was used on the training and test sets. Evaluation
measures are computed and reported on the entire test set and not by averaging
across folds. We compare the proposed attention approach to the more
traditional LSTM-RNN approach, which has provided good results in the past [
The networks used for the current work are assigned names depending on whether
they apply attention (AT) or not (NAT), followed by the layer sizes. For
example, for an LSTM, no attention network with 1 layer of 128 hidden units, the
name is LSTM NAT 128. For an LSTM, attention network with 2 layers of 700
and 128 hidden units each, the name is LSTM AT 700 128. Thus, all networks
belonging to each proposed model class are assigned the su xes a) LSTM, no
attention is LSTM NAT, b) LSTM with attention is LSTM AT. We replicate one
of the best models proposed in Weninger et.al's work [
P Q p(P + Q + T ) (P + Q + U ) (19) where, P is the number of concordant pairs, Q the number of discordant pairs, T the number of ties only in target set (y1 : : : yn), and U the number of ties only in predicted set (f1 : : : fn). The mean absolute error (MAE) is given for reference. In the next section, we report the results of applying the proposed model for dynamic music emotion regression. 5.1
In the current work, we use four main types of LSTM-RNN models, which are { No Attention (LSTM NAT)
Single Layer: Eg. LSTM NAT 128
Double Layer: Eg. LSTM NAT 700 128 { With Attention (LSTM AT)
Single Layer: Eg. LSTM AT 128
Double Layer: Eg. LSTM AT 400 128
In the rst set of experiments, the performances of di erent models using di erent network topologies are compared. The best results obtained from each of these models are summarized in table 1.
In the case of LSTM NAT networks, separate models for arousal and valence are trained using the LSTM-RNN architecture. Performances of networks having one hidden layer with 128, 300, 400, 512, 700, 1024 units and two hidden layers with (700, 128) and (700, 400) units are calculated. Table 2 reports the results for regression without attention, using di erent network topologies. For the single-layer topologies, a clear trend can be seen for arousal. With the increase in layer size (L1 Size), RA2 increases. A increases till L1 Size = 700, but decreases for L1 Size = 1024. The metrics for valence does not follow a clear trend. It can be seen that LSTM NAT 700 performs best in terms of all the evaluation metrics considered, for both arousal and valence, giving RA2 = 0.70, A = 0.21, RV2 = 0.39, and V = 0.10. Though, LSTM NAT 1024 performs better for arousal (RA2 = 0.73), its performance dips for valence (RV2 = 0.39). is also reduced for both arousal and valence. The two-layer topologies of this model, LSTM NAT 700 128 and LSTM NAT 700 400 perform comparable to the best single layer network LSTM NAT 700 and LSTM NAT 1024 for arousal, both in terms of RA2 and A. The performance for valence decreases in the 2-layer topologies. Thus, increasing layer size might help improve performance for arousal, but not for valence. Also, increasing the number of hidden layers might be unable to produce any signi cant improvement in performance for both arousal and valence.
The performances of of LSTM AT networks, using di erent network topologies are presented in table 3. Performances of networks having one hidden layer with 32, 64, 128, 300, 400 units and two hidden layers with (300, 128) and (400, 128) units are calculated. A clear trend for performances of arousal and valence predictions are observed in this case. For arousal, among the singlelayer topologies, best performance is recorded for the networks LSTM AT 300 and LSTM AT 400, for R2. It can be seen that addition of the attention mechanism improves the performance according to both metrics. For both arousal and valence, the best performances among all the models used is recorded for LSTM AT 400, with RA2 = 0.75 and RV2 = 0.53. Henceforth, for all comparison purposes, we use this model as the best proposed model of this study. Increase in number of layers produce comparable performance for both arousal and valence and no signi cant change is observed. (a) Arousal LSTM AT 400 -(b) Valence
LSTM AT 400 Best
Prediction
Best
Prediction (c) Arousal LSTM NAT 400
Baseline
Prediction -(d) Valence LSTM NAT 400
Baseline
Prediction
ne-grained emotion prediction
In the second set of experiments, the best models for arousal and valence
predictions, as obtained in the previous section, are used for ne-grained (per 500
ms) emotion prediction of some music clips. For arousal and valence predictions,
we use the LSTM AT 400 model (table 1). We choose two clips from the 1000
Songs for Emotional Analysis of Music [
In the third set of experiments, we use the best proposed model LSTM AT 400 and the baseline model LSTM NAT 400 on the validation set, to group the clips into error bins for arousal and valence prediction. These are shown as histograms in gure 3. Comparing gures 3(a) and 3(c), it can be seen that, for the proposed model, the number of clips with higher values of errors are less, in case of arousal. In case of valence, for the proposed model, almost all the clips are grouped into the error bins 0.05 ( gure 3(b). Whereas for the baseline model, a signi cant number of clips across bins are present. 6
We demonstrate that the state of the art models for continuous-time emotion prediction perform modestly, thus emphasizing the need for further research in this area. We have proposed an attentive LSTM based model which improves the state of the art performance signi cantly, on standard benchmark dataset with standard metrics.