Recognition of emotional state through the sound of the voice is a crucial element in human interactions. This process, known as emotional prosody, allows an individual's emotions to be interpreted without the need to see his or her face or body language. The ability to identify emotions through voice is critical in areas such as psychology, health care, marketing and human-computer interaction technology. Moreover, in artificial intelligence systems and voice interfaces, the analysis of emotional prosody can improve the user experience and make interactions more personalized and intuitive. This study explores the robustness of a neural network in recognizing emotions within speech signals, considering the presence of environmental noise at diferent SNR levels. The study involves using public datasets containing diferent classes of emotions, training a CNN network to classify these emotional states, and then testing to evaluate the obtained performance. The results show that emotion recognition in speech is more accurate when the recording is clean and free of other noises. However, misclassification depends on the type of noise present and the type of emotion: indeed, it turns out that some emotions are more frequently misclassified than others. In fact, it is observed that accuracy is higher when using RAW data rather than MFCCs, with an accuracy of 75% compared to 60%. In addition, adding noise to the recordings results in a decrease in model performance.
Speech signal is the fastest and most natural method of
communication among humans, prompting researchers to focus
on making this mode of communication equally eficient
in human-machine communication [
Emotion Recognition (ER) is the process of identifying
human emotions, and the use of technology for this process
represents a relatively young area of research. So far, much
of the research in this field has involved recognition of facial
expressions through video and images, vocal expressions
through audio, expressions written in text, and physiology
measured by wearable devices. Emotion recognition can
be used in various ways depending on the context, such as
in speech or face recognition, referred to as “Speech
Emotion Recognition" (SER) [
Despite significant attention to this field, the task of SER systems remains challenging to implement for several reasons. First, it is not always clear which linguistic features are most relevant in distinguishing an emotion. In addition, there is considerable acoustic variability introduced by factors such as speed, accent, and voice volume, which can afect the waveform of audio files but still express the same emotion, making it complex for machines to understand the emotion and locate useful information.
In [
The potential applications of this technology are
numerous: in [
In this paper, the authors propose a study that aims to
explore the robustness of a neural network in recognizing
emotions within speech signals, considering the presence
of environmental noise at diferent levels of signal-to-noise
ratio (SNR). Using public datasets containing a wide range
of emotion classes, such as the RAVDESS [
The comparison of using raw audio data (RAW) and Mel’s frequency cepstral coeficients (MFCC) as inputs to the model aims to determine which type of input is more robust than noise and produces better results in emotion recognition. In addition, the study includes analysis of emotion misclassification trends to identify any patterns in the confounds between emotion classes. By adding noise of diferent types to voice recordings and evaluating the performance of the model under these conditions, the study seeks to understand how noise afects the emotion recognition process and what challenges may arise in realistic contexts. The ultimate goal is to contribute to the creation of more robust and reliable models for voice emotion analysis, with potential applications in areas such as automotive and telephone services.
The paper is organized as follows: In the 2 section, the methodology used in this paper for emotional recognition using speech signals is reviewed. In fact, this section discusses the dataset used for training and testing the network, the pre-processing phase of audio signals, and the addition of noise to validate the robustness of the model. In the 3 section, the results obtained in the case of audio sequences without and with added ambient noise are discussed.
The proposed method is to use neural networks to recognize emotions from audio signals. A public dataset is employed to train the models and evaluate them realistically. Next, audio data are preprocessed and two input approaches are explored: raw audio data and MFCC parameters. The robustness of the model to the addition of environmental noise is also evaluated. Finally, the neural network is trained on both inputs to classify the emotional state through the speech signal. The above is described in detail in the subsections to follow.
The use of public datasets is a crucial element in the
development of neural network-based emotion recognition models.
The availability of well-annotated and diverse datasets
allows researchers to train and evaluate their models under
realistic conditions, ensuring greater generalization and
reliability of the results obtained [
Prominent among the many datasets in the literature are RAVDESS (Ryerson Audio Database for Emotional Speech and Song) and USC-IEMOCAP (University of Southern California-Emotional Motion Capture). These datasets ofer a wide range of voice recordings of individuals expressing a variety of emotions, allowing scholars to explore and analyze in detail the acoustic features associated with each emotion.
There are also other datasets available in diferent languages, such as German, Chinese, Mandarin, and others, which are useful for the development of SER systems in those languages.
The RAVDESS dataset, which is a large set of audiovisual ifles commonly used in Speech Emotion Recognition (SER) systems, was selected for this study. The dataset includes recordings of 24 actors from North America, including 12 female and 12 male actors. The recordings include both speech and singing and are categorized by loudness (normal or loud) and by emotion type, with 8 emotions labeled: calm, neutral, happiness, surprise, sadness, fear, anger, and disgust.
The choice of this dataset is also due to the fact that it avoids false positives regarding gender and volume classiifcation. In addition, for this study, only simple audio data related to recordings were analyzed, excluding video and sung recordings.
Before providing the input data to the neural network, a pre-processing and feature extraction phase was performed on the audio data.
The first operation was to equalize the length of the recordings to 4 seconds.
Then, two parallel studies were conducted: the first study is related to using the RAW data of the audios as input to the neural network; the second study is related to extracting MFCC parameters from the audio signal. 46–50
In both cases, a moving window with an ofset of 200 milliseconds was applied in the input generation process, which starting from the four-second audio file generated about ten files of two seconds each.
Finally, the dataset was divided into learning (70%) and testing (30%) sets.
On the data belonging exclusively to the test set, further
manipulations were carried out: first, noise was added to
the recordings. The types of noise used in the study include:
• Babble-type noise [
Then, the powers were normalized in order to obtain a determined signal-to-noise ratio (SNR) with respect to the useful signal. The SNR values used in the study were 10 dB, 15 dB and 20 dB.
The neural network used in this study is of the 1D type
and has also been used in [
In addition, two diferent neural network trainings were conducted: one using the raw 2-second data of the audio signal as input, and the other using a vector of 50 MFCC parameters extracted from voice recordings of the same duration. Both trainings aimed to evaluate the model’s ability in classifying emotional state through the speech signal.
Analysis of the experimental data provides insight into the performance of the emotion recognition model under varying input conditions. Through the examination of both clean and noise-added data, relevant observations emerge regarding the overall performance of the system. This study focuses on three main scenarios: clean datasets, datasets with babble-type noise, and datasets with ofice-type noise. In each of these scenarios, a performance comparison is conducted between the use of MFCC parameters and RAW raw data.
Analyzing the results obtained from the 2-second-length noise-free data, it can be seen that the accuracy is higher when RAW data (63%) is used compared to MFCC data (55%).
By applying a recurrence filter with observation window equal to 2 seconds and ofset of 200 milliseconds to these data, the accuracy increases to 75% in the case of RAW data and 60% in the case of MFCC data. These results are displayed in Figure 1.
Next, looking at the confusion matrix in Figure 2, we see that using MFCC data there is a misclassification rate of 16% between Neutral and Happy, 19% between Fearful and Happy, and 16% between Fearful and Sad.
Testing data to which babble-type noise has been added, a decrease in model performance is observed. In addition, for the same SNR, the test detects better results using RAW files rather than MFCCs, as shown in Figure 4. Also, as expected, as SNR increases, accuracy decreases.
In tests with RAW recordings to which babble-type noise was added with a signal-to-noise ratio of 10 dB, it can be seen that the “Neutral" and “Sad" emotions are the most misclassified. In particular, it is observed that Neutral is often interchanged with Calm, Happy and Surprise, while Sad is strongly misclassified with Fearful, as shown in Figure 5.
By raising the SNR value to 20 dB, a general increase in network performance is evident in Figure 6, with the Sad emotion no longer being strongly misclassified. However, Neutral is still mistaken for Calm and Sad, while Surprise (which goes from an accuracy of 68% to 59%) is mistaken for Fearful.
Similarly to the noisy babble-type recordings, the ofice-type recordings also showed lower accuracy results than the test with clean data, as could be expected. Also, as in the case of babble noise, for the same SNR, a decrease in accuracy is observed if MFCC instead of RAW data is tested. It is also noted that as the signal-to-noise ratio (SNR) increases, the accuracy decreases, as evidenced in Figure 7.
By analyzing the confusion matrices related to RAW files with ofice-type noise and signal-to-noise ratio of 10 dB, it is possible to observe the results of the model in emotion class recognition. Under these test conditions, it is observed that the network strongly misclassifies the Sad emotion as Fearful 46% of the time, while Disgust is frequently misclassified as Angry and Fearful. Such a confusion matrix is shown in Figure 8.
By increasing the SNR value to 20 dB, a general increase in network performance is observed, with Sad and Disgust emotions no longer being strongly misclassified. Fearful goes from an accuracy of 77% to 69%, while Neutral goes from 49% to 44% accuracy. The confusion matrix of RAW ifles with ofice-type noise and SNR 20 dB is shown in Figure 9.
In this study, the robustness of a neural network in the face of noise variation in audio data in the context of Speech Emotion Recognition was examined.
From the tests conducted and related results, it appears that emotion recognition in speech is more accurate using RAW data (75%) than using MFCCs as input (60%).
In addition, worse performance was observed in the presence of noise in recordings. In particular, in crowded contexts where the acoustic cocktail party phenomenon occurs, SER-type systems experience dificulties in performing their task (accuracy decreases up to 30% in the worst case, with an SNR of 10 dB), while in contexts where noise difers significantly from human speech, better results are obtained (accuracy decreases up to 10% in the worst case, with an SNR of 10 dB).
In general, the most frequently misclassified classes turn out to be “neutral" and “sad."
These results outline interesting prospects for future developments, especially in real-world and potentially realtime applications. For example, in automotive or telephone services, where the recorded and analyzed voice usually belongs only to the involved interlocutors.