Emotion recognition from speech is a research area that focuses on grasping genuine feelings from audio data. It makes it possible to extract various useful data points from sound that are further used to improve decision-making. This research was conducted to test an emotion recognition toolkit on real-world recording of phone calls in Latvian language. This scenario presents at least two significant challenges: mismatch between real-world data and "artificially" created data, and lack of the training data for the Latvian language. The study mainly focuses on investigating training data requirements for successful emotion recognition.
Nowadays, emotion recognition from speech is a highly The main dataset used in this paper consists of technical
relevant topic. It is used in a wide variety of applications support phone calls. Recordings are done with an 8 kHz
from businesses to governmental bodies. For example, sampling rate and a single channel. The dataset included
in call centers it helps to monitor client support quality audio recordings of 39 conversations, that held in
Latand to study clients’ reaction to certain emotional trig- vian which were further separated into 6,171 segments.
gers. Multiple studies have been conducted on emotion Qualitative analysis of telephone conversations was
perrecognition from speech signal. However, most of the formed, annotating in several layers’ potential afective
papers investigate machine learning model performance features - afect dimensions, linguistic units,
paralinguison public artificially created datasets such as EMODB[
When it comes to emotion recognition from speech in tailed description of dataset creation process and
qualitaLatvian language, the literature is even shallower. Nei- tive analysis of the dataset is presented in [
Additionaly, several public emotional speech datasets were included in the research. EMODB, IEMOCAP, TESS, and RAVDESS were used to increase the dataset size, as well as to see how our model performs compared to the state-of-the-art models.
tion recognition. Each input audio was converted into the 39-dimensional mel-frequency cepstral coeficients (MFCC) feature vector and passed through the model. Due to the relatively small dataset size, a shallow neural network was used to prevent overfitting. The final model was comprised of two LSTM layers, two fully connected layers and a softmax output layer. For additional regularization, a 30% dropout after each layer was added. Figure 1 displays the model architecture.
Categorical cross-entropy was used as a loss function and an Adam optimizer was used for backward propagation. The model training was performed using batches of 64 observations. Each model was trained for 200 epochs with validation after each epoch. Next model weights that yielded the highest accuracy were retrieved. Finally, all trained models are compared based upon their accuracy on the test set.
First, model performance on public datasets was evaluated. TESS and RAVDESS were combined into one dataset, separated into train (80%) and test (20%) sets and used to train the model for 200 epochs. The final test accuracy (Figure 2) was 86.02%. In addition, a similar experiment was conducted with an IEMOCAP dataset which was comprised of conversations between actors.
The final test accuracy (Figure 3) was 60.65% which is
slightly lower than state-of-the-art[
derstand if adding additional data from public datasets can improve the performance of models. Because phone call data is recorded with 8 kHz sampling rate, but public datasets are 16 kHz, following 4 experiments were performed.
In the first experiment, the model was trained on the phone call data in its original format. In the second experiment, IEMOCAP, TESS, RAVDESS and EMODB were downsampled to 8 kHz and added to the train set. In the third experiment, phone call data (both train and test) were upsampled to 16 kHz. In the fourth experiment, IEMOCAP, TESS, RAVDESS and EMODB were added to the upsampled train set. The results of those experiments are summarized in Table 2.
Finally, an untrained human person was asked to guess the emotions in the same test set to estimate the human error. Audio segments were presented in random order, so that the person can not analyse the overall semantics and context of the conversations, and have to rely solely on the acoustics, similarly to the deep learning model. The test accuracy ended up being 22.72% which indicates that predicting emotions in the random segments of phone calls is not a trivial exercise even for a human.
This research investigated emotion recognition from real world phone call data. The output of the research can be summarized according to the following points: • Emotion recognition on real-world data is a more dificult exercise than emotion recognition on artificially created datasets • The model architecture proposed in this paper is capable of surpassing a untrained human-level error for the given exercise • Augmenting training phone call data with artificially created datasets does not seem to help to improve model performance • At this stage increasing data volume twofold marginally improves model performance • Upsampling and downsampling the audio data neither improves nor worsens the performance of the models
For further research it might be worth trying to increase the training dataset further (at least 500-1000% increase). Given the untrained human-level error, it seems that in order to predict emotions accurately, even human needs more context than a single utterance, preferably whole conversations. Therefore, increasing input context is interesting avenue for the follow-up work. Furthermore, defining an emotion as a set of dimensions and predicting each dimension separately might improve forecasting accuracy.
ing from the research project "Competence Centre of Information and Communication Technologies" of EU Structural funds, contract No. 1.2.1.1/18/A/003 signed between IT Competence Centre and Central Finance and Contracting Agency, Research No. 2.9. “Automated multilingual subtitling”.