Audio Sentiment Analysis is an increasingly popular research area which extends the conventional text-based sentiment analysis to depend on the effectiveness of acoustic features extracted from speech. However, current progress on audio sentiment analysis mainly focuses on extracting homogeneous acoustic features or doesn't fuse heterogeneous features effectively. In this paper, we propose an utterance-based deep neural network model, which has a parallel combination of CNN and LSTM based network, to obtain representative features termed Audio Sentiment Vector (ASV), that can maximally reflect sentiment information in an audio. Specifically, our model is trained by utterance-level labels and ASV can be extracted and fused creatively from two branches. In the CNN model branch, spectrum graphs produced by signals are fed as inputs while in the LSTM model branch, inputs include spectral centroid, MFCC and other recognized traditional acoustic features extracted from dependent utterances in an audio. Besides, BiLSTM with attention mechanism is used for feature fusion. Extensive experiments have been conducted to show our model can recognize audio sentiment precisely and quickly, and demonstrate our ASV are better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms the state-of-the-art approach by 9.33% on MOSI dataset.
Sentiment Analysis is a well-studied research area in Natural
Language Processing (NLP) (Pang B et at al. 2008), which
is the computational study of peoples’ opinions, sentiments,
appraisals, and attitudes towards entities such as products,
services, organizations and so on
In recent years, there are three main methods for audio
sentiment analysis. Firstly, utilizes automatic speech
recognition (ASR) technology to convert speech into texts,
following by conventional text-based sentiment detection
systems (S. Ezzat et at al. 2012). Secondly, adopts a
generative model operating directly on the raw audio waveform
However, for converting speech into texts, by recognizing
each word said by the person in an audio, change them into
the word embedding and use some techniques in NLP, like
TF-IDF and bag of words model. The result is not always
accurate, because sentiment detection accuracy depends on
being able to reliably detect a very focused vocabulary in
the spoken comments
We believe the information extracted from a single utterance must have dependency on its context. For example, a flash of loud expression may not indicate a person has a strong emotion since it maybe just caused by a cough while continuous loud one is far more likely to indicate the speaker has a strong emotion.
In this paper, based on a large number of experiments, we extract the features of each utterance in an audio through the Librosa toolkit, and obtain four most effective features representing sentiment information, merge them by adopting a BiLSTM with attention mechanism. Moreover, we design a novel model called Audio Feature Fusion-Attention based CNN and RNN (AFF-ACRNN) for audio sentiment analysis. Spectrum graphs and selected traditional acoustic features are fed as input in two separate branches, we can obtain a new fusion of audio feature vector before the softmax layer, which we call the Audio Sentiment Vector (ASV). Finally, the output of the softmax layer is the class of sentiment.
Major contributions of the paper are that: We propose an effective AFF-ACRNN model for audio sentiment analysis, through combining multiple traditional acoustic features and spectrum graphs to learn more comprehensive sentiment information in audio.
Our model is language insensitive and pay more attention to acoustic features of the original audio rather than words recognized from the audio.
Experimental results indicate that the proposed method
outperforms the state-of-the-art methods
The rest of the paper is organized as follows. In the following section, we will review related work. In Section 3, we will exhibit more details of our methodology. In Section 4, experiments and results are presented, and conclusion follows in Section 5.
Current state-of-the-art methods for audio sentiment analysis are mostly based on deep neural network. In this section, we briefly present the advances on audio sentiment analysis task by utilizing deep learning, and then we give a summary on the progress of extracting the audio feature representation.
It has been demonstrated that LSTM (Hochreiter S et at al. 1997) are well-suited to make predictions based on time series data, by utilizing a cell to remember values over arbitrary time intervals and the three gates(input gate i, output gate o, forget gate f ) to regulate the flow of information into and out of the cell, which can be described as follows: ft = (Wf [ht 1; xt] + bf ) it = (Wi [ht 1; xt] + bi) ot = (Wo [ht 1; xt] + bo) where ht = ot tanh(Ct) is the output of the last cell and xt is the input of current cell. Besides, the current cell state Ct can be updated by the following formula:
Ct = tanh(Wc [ht 1; xt] + bc)
Ct = ft
Ct 1 + it
Ct where Ct 1 stands for the previous cell state.
One of the most effective variant of LTSM is the bidirectional LSTM. Each input sequence will be fed into both the forward and backward LSTM layers and thus a hidden layer receives an input by joining forward and backward LSTM layers.
CNN
Researchers have found pitch and energy related features
playing a key role in affect recognition
In this section, we describe the proposed AFF-ACRNN model for audio sentiment analysis in details. We firstly introduce an overview of the whole neural network architecture. After that, two separate branch of AFF-ACRNN will be explained in details. Finally, we talk about the fusion mechanism used in our model.
We concentrate on a model that has two parallel branches, the utterance based BiLSTM branch (UB-BiLSTM) and the spectrum based CNN branch (SBCNN), whose core mechanisms are based on LSTM and CNN. One branch of proposed model uses the BiLSTM to extract temporal information between adjacent utterances, another branch uses the renowned CNN based network to extract features from spectrum graph that sequence model cannot achieve. Furthermore, audio feature vector of each piece of utterance is the input of the proposed neural network that based on Audio Feature Fusion (AFF), we can obtain a new fusion audio feature vector before the softmax layer, which we call the Audio Sentiment Vector (ASV). Finally, the output of the softmax layer produces our final sentiment classification results, as shown in Figure 1.
AFF2 AFF1 Librosa
Softmax
LASV Attention
Dropout
V2 BiLSTM Dropout
V3 BiLSTM
Dropout BiLSTM
BiLSTM
BiLSTM utterance1 utterance2 utterance3
V1 BiLSTM
Dropout F1_1 .. F1_n
F2_1 ... F2_n
F3_1 ... F3_n V1_1 .. V1_n
V2_1 ... V2_n
V3_1 ... V3_n
LSTM Layers The hidden layers of LSTM have
selfrecurrent weights. These enable the cell in the memory
block to retain previous information
Algorithm 1 Related Procedure 1: procedure LSTM BRANCH 2: for i:[0,n] do 3: fi = getAudioF eature(ui) 4: ASVi = getASV (fi) 5: end for 6: for i:[0,M] do //M is the number of videos 7: inputi = GetT opU tter(vi) 8: ufi = getU tterF eature(inputi) 9: end for 10: shuf f le(v) 11: end procedure 12: procedure CNN BRANCH 13: for i:[0,n] do 14: xi get SpectrogramImage(ui) 15: ci CNNModel(xi) 16: li BiLSTM(ci) 17: end for 18: end procedure 19: procedure FIND CORRESPONDING LABEL 20: for i:[0:2199] do 21: rename(ui) == for better order in sorting 22: N ameAndLabel = createIndex(ui) 23: == A dictionary [utterance Name: Label] 24: end for 25: Labelx = N ameAndLabel(ux) 26: end procedure CNN Layers Similar to the UB-BiLSTM model proposed above, we extracted the spectrum graph of each utterance through the Librosa toolkit and use it as the input of our CNN branch. After a lot of experiments, we found that the audio feature vector learned by the ResNet152 network structure has the best effect on the final sentiment classification, so we choose the ResNet model in this branch. The convolutional layer performs 2-dimensional convolution between the spectrum graph and the predefined linear filters. To enable the network to extract complementary features and learn the characteristics of input spectrum graph, a number of filters with different functions are used. A more refined audio feature vector is obtained through deep convolutional neural network, and then put into the BiLSTM layer to learn related sentiment information between adjacent utterances. Finally, before the softmax layer, we get another effective vector CASV extracted by our CNN framework, as shown in Figure 3. The process is described in CNN branch procedure in Algorithm 1.
Fusion Layers Through the LSTM and CNN branches
proposed above, we can extract two refined audio sentiment
vectors, LASV and CASV for each utterance. We use these
two kinds of vectors in parallel as the input of BiLSTM in
AFF-ACRNN model. While effectively learning the relevant
sentiment information of adjacent utterance, we extract the
Audio Sentiment Vector (ASV) that has the greatest
influence on the sentiment classification in the three utterances
through the action of the attention mechanism. Finally, the
final sentiment classification result is obtained by softmax
layer. In
Inspired by human visual attention, the attention mechanism
is proposed by
Rx = H Px = tanh(Wh[x] H) Ax = sof tmax(w[x]T
T x
Px)
hx = tanh(Wm[x] Rx + Wn[x] hx) Where Wm[x] and Wn[x] are weights to be learned while training.
In this section, we exhibit our experimental results and
the analysis of our proposed model. More specifically, our
model is trained and evaluated on utterance-level audio from
CMU-MOSI dataset
Evaluation Metrics We evaluate our performance by weighted accuracy on both 2-class, 5-class and 7-class classification.
weighted accuracy = correct utterances utterances
Additionally, F-Score is used to evaluate 2-class classification.
F =(1 + 2)
precision recall ( 2 precision) + recall Where represents the weight between precision and recall. During our evaluation process, we set = 1 since we regard precision and recall has the same weight thus F1-score is adopted.
However, in 5-class and 7-class classification, we use Macro F1-Score to evaluate the result.
M acro F1= 1 n P F1n n where n represents the number of classification and F1n is the F1 score on nth category.
Dataset details CMU-MOSI dataset is rich in sentiment expressions, consisting 2199 opinionated utterances, 93 videos by 89 speakers. The videos address a large array of topics, such as movies, books, and products. Videos were crawled from YouTube and segmented into utterances where each utterance is annotated with scores between 3 (strongly negative) and +3 (strongly positive) by five annotators. We took the average of these five annotations as the sentiment polarity and considered three conditions where consists of two classes (positive and negative), five classes (strong positive, positive, neutral, negative and strong negative) and seven classes (strong positive, positive, weak positive, neutral, strong negative,negative and weak negative). Datasets MOSI
Train utterance 1616 2199 video 65 93
Test
utterance
583
437
video
28
79
Our train/test splits of the dataset are completely disjoint
with respect to speakers. In order to better compare with the
previous work, similar to
ture is implemented based on the open-source deep learning framework Keras. More specifically, for proposed UBBiLSTM framework, after a lot of experiments, we extracted the most four representative audio features of each utterance in a video through Librosa toolkit, which are MFCC, spectral centroid, chroma stft and spectral contrast respectively. In data processing, we make each utterance one-to-one correspondence with the label and rename the utterance. Accordingly, we extend each utterance to a feature matrix of 256 33 dimensions. The output dimension of the first layer of BiLSTM is 128, and the second layer is 32. The output dimension of the first layer of Dense is 200, and the second is 2.
For proposed CNN framework, the input images are warped into a fixed size of 512 512. If the bounding box of the training samples provided, we firstly crop the images and then warp them to the fixed size. To train the feature encoder, we follow the fine-tuning training strategy.
In all experiments, our networks are trained by Adam or SGD optimizer. In the LSTM branch, we initiate the learning rate to be 0.0001, and there are 200 epochs in the training part with batch size equals to 30 in each epoch. In the CNN branch, we initiate the learning rate to be 0.001, and there are 200 epochs in training Resnet-152 with batch size equals to 20 in each epoch.
Comparison of different feature combinations. Firstly, we have considered seven types of acoustic features that can best represent an audio,which mainly includes MFCC, rootmean-square energy, spectral and tonal features. A lot of experiments have be done in order to get the best feature combinations with different model on three types of classification. The results are listed in Table 2. What’s more, we have also compared the different performance of our LASV extracted from LSTM-based and BiLSTM-based fusion model. As the Table 2 shows, the performance of LASV that extracted from BiLSTM-based model behaves better, since the acoustic information behind may also have impact Single Type Two Types Thress Types
Five Types Six Types Seven Types LSTM BiLSTM LSTM BiLSTM LSTM BiLSTM
LSTM BiLSTM LSTM BiLSTM LSTM BiLSTM on the acoustic information previous. It can be seen that the best number of feature combination is four and those four features are MFCC, spectral centroid, spectral contrast and chroma stft. That means the other three features, which are root-mean-square energy, spectral contrast and tonal centroid may introduce some noise or misleading in our sentiment analysis since all seven types of features do not have the best result.
We have compared our CASV performance extracted from
the spectral map with several genres of popular models of
CNN and its variants: LeNet
and UB-Bilstm At last, we have performed fusion experiments between several best SBCNN and UB-BiLSTM and UB-LSTM. More accurately, we choose the three best SBCNN, which are ReSNet18, ResNet50 and ResNet152 to combine with the two kinds of utterance dependent LSTM. The best combination is UB-BiLSTM with Res152. The final result is shown in Table 4.
Comparison with traditional method. Apart from training deep neural network, a bunch of traditional binary classifiers has been used for sentiment analysis. In order to demonstrate the effectiveness of our model, we firstly compare our model with those traditional methods.
[I2C2, 2017]
[BAJECE, 2018]
As shown in Table 5, we use weighted accuracy (ACC) and F1-Score to evaluate our results. Especially, for the ACC on MOUD, our proposed model outperforms the best model, SVM classifier, by 11.51%.
paper introduced a LSTM-based model to utilize the contextual information extracted form each utterance in an video. However, the input of the neural network model only has one type of feature, which is MFCC. This means all the utterance information is merely represented by one single feature. The acoustic information contained by the feature is somewhat duplicated and is bound to omit much sentiment information that might be hidden in many other useful features. What’s worse, one type of feature means the input vector should be large enough to make sure that it carries enough information before it is fed into the neural network. This will undoubtedly increase the parameters to be trained in the network and meanwhile, it is time consuming and computation costly.
Our proposed model not only extracts the feature or sentiment vector from four types of traditional recognized acoustic features, have considered utterance dependency, but also extracts the feature from the spectrum graph, which may reveal some sentiment information that acoustic features cannot reflect. The final AFF-ACRNN consists of the best combination of SBCNN and UB-BiLSTM and outperforms the state-of-the-art approach by 9.33% in binary classification on MOSI dataset and by 8.75% on MOUD. The results are shown in the Table 6.
We have also run our model on one audio whose length is 10s for 1000 times and the average time to get the sentiment calssification result from input is only 655.94ms which thanks to our concentrated ASV extracted from AFFACRNN.
The above experimental results have already shown us that the proposed method has a great improvement in the performance of audio sentiment analysis. In order to get the best structure of our AFF-ACRNN model, we have tested two separate branch respectively, and compare the final AFFACRNN with traditional or state-of-art method. Weighted accuracy and F1-Score, Macro F1-Score are used as metrics to evaluate the model’s performance. In the UB-Bilstm branch, a lot of experiments have shown that four types of heterogeneous traditional features trained by BiLSTM will have the best result, whose weighted accuracy is 68.72% on MOSI. In the SBCNN branch, we have carried out seven experiments to prove the ResNet152 used in SBCNN will have Methods LeNet AlexNet VGG16
ZFNet ResNet18 ResNet50
the best result, for instance, with the weighted accuracy of 65.42% on MOSI, due to its extreme depth and the helpful residual units used to prevent degradation. We selected six best combinations of SBCNN and UB-BiLSTM and find that the best is ResNet152 used in SBCNN with UB-Bilstm, whose weighted accuracy is 69.42% on MOSI and outperforms not only the traditional classifier like SVM, but also the state-of-the-art approach by 9.33% on MOSI dataset. Attention mechanism is used in both branch to subtly combine the heterogeneous acoustic features and choose the feature vectors that have the greatest impact on the sentiment classification. Furthermore, in the experiment of using MOSI as training set and verification set and MOUD as test set, it also shows that our proposed model has strong generalization ability.
In this paper, we propose a novel utterance-based deep neural network model termed AFF-ACRNN, which has a parallel combination of CNN and LSTM based network, to obtain representative features termed ASV, that can maximally reflect sentiment information in an utterance from an audio. We extract several traditional heterogeneous acoustic features by Librosa toolkit and choose the four most representative features through a large number of experiments, and regard them as the input of the neural network. We can get CASV and LASV from the CNN branch and the LSTM branch respectively, and finally merge the two branches to obtain the final ASV for sentiment classification of each utterance. Besides, BiLSTM with attention mechanism is used for feature fusion. The experimental results show our model can recognize audio sentiment precisely and quickly, and demonstrate our heterogeneous ASV are better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms the state-of-the-art approach by 9.33% on MOSI dataset. We have also tested our model on MOUD to prove the model won’t heavily depend on language types. In the future, we will combine the feature engineering technologies to further discuss the fusion dimension of audio features and consider the fusion of different dimensions of different categories of features, and even apply them to multimodal sentiment analysis.