Humans communicate their desires through spoken language, which expresses various emotions. This process has led to the development of speech recognition systems, where machine learning enables computers to recognize and analyze vocal cues to interpret emotions, resulting in application creation focused on human-machine interaction. Advancements in technology, the evolution of artificial intelligence, and the influence of deep learning via CNN architectures have propelled research in emotion recognition systems forward. In this paper, we evaluated our method for detecting and classifying emotions in two architectural models (Model-A and Model-B) that utilized Mel-frequency cepstral coeficients to extract features from audio files. The experiments were conducted using the TESS and Crema-d audio file databases. The outcomes are promising, showing an accuracy of 54,07% for Model-B with the Crema-d dataset and 98,92% for Model-A with the TESS dataset.
Speech is a means of communication with the outside world. Each human being has his speech, and it is thanks to natural language that individuals can discuss and understand each other. Speech is unique and expressed through a well-defined language addressed to an interlocutor (oneself). It allows us to express needs such as feelings, sufering, aspirations, observations, and the formulation of requests. It also allows us to constitute diferent natural languages and dialects.
Speech is the most popular tool of expression because it is easier to speak than to write or make a diagram. Nonetheless, The process of producing speech, from the brain to the articulation of the mouth using the vocal cords, is intricate. This dificulty makes the automation of speech by machine complicated [1].
The system of the phonatory apparatus of speech is an acoustic mechanism that difers from other sensory devices. It comprises elements such as the vocal cords, the oral and nasal cavities, the air, the nervous system, and the tongue and lips [3].
• The throat (pharynx), which is the largest surface of the neck and head, is made up of three primary parts: the hypopharynx, the nasopharynx, and the oropharynx.
• The hypopharynx, found behind the pharynx, is a section of the throat. It has a flap-like structure that acts as a lid for the larynx and shuts when swallowing to prevent food and liquid from entering the trachea. • The nasopharynx is located behind the nose, and its purpose is to transfer the air to breathe down through the voice box of the throat and into the lungs. • The oropharynx is situated posterior to the mouth and is responsible for conveying food and liquid to the digestive tract and stomach. • The larynx is the voice box, and it is eficient because it protects the lungs from food, drink, and foreign bodies and acts as a corridor for air from the nasopharynx. • The epiglotis protects the lungs and only lets air pass through. • The vocal tract extends from the vocal cords to the lips and measures approximately 17.5 cm in length. It consists of two types of ducts: a pharyngeal cavity and an oral cavity. • The vocal cords open when breathing and close when swallowing or producing voice sounds.
Understanding the vocal process is straightforward. It starts with the movement of air passing through the two vocal cords (see Fig.2), which are soft and vibrate as air flows through them, producing 100 to 1 000 vibrations per second. • Articulators are the tongue, lips, jaws, mouth, etc. These articulators allow for the modification of the shape of the vocal tract.
• The nasal cavity forms a part of the vocal tract and is situated beneath the velum. Speech production system:
The human speech production system involves multiple stages before the voice is produced. It operates rapidly and is complex, relying on the respiratory system, the phonatory system, and the articulatory system, all working together. These systems collectively create speech and sound. The respiratory system manages the intake and release of air in the lungs. Inhalation involves the diaphragm lowering and the intercostal muscles facilitating vacuum creation in the lungs for air intake. Exhalation entails the diaphragm relaxing, allowing air to escape and produce sounds. Air then passes through the larynx, containing muscles and cartilages called the vocal cords, with the space between them known as the glottis. The vocal cords, capable of rapid opening and closing, can reach up to 400 movements per second in children. The articulators, situated between the arytenoids, enable the vocal cords to move. The articulation process starts with air leaving the larynx, passing through the pharynx before being modulated by resonators like the lips, tongue, mandible, and velum, which impart characteristics to the sound (air flowing freely results in a vowel, while encountering an obstacle yields a consonant).
Analysis of a person’s emotions is an important area of research in Natural Language Processing (NLP). For example, it allows us to recognize the individual’s anxiety. In certain cases, we use the analysis of emotions to diagnose diseases. NLP is a branch of deep learning research. Because deep learning has revolutionized the world of artificial intelligence, such as in medical research for early diagnosis of diseases such as COVID-19 [5] or in the field of biometric recognition [6].
The strategy of our approach consists of: 1. Making a consistent bibliographic study in the speech emotion recognition field. 2. Extracting adequate scientific knowledge to apply them in our approach. 3. Using quality and available datasets that are made available to the scientific community. 4. Applying our approach to deep neural networks of the LSTM type using the Mel-frequency cepstral coeficients (MFCC). 5. Testing our experiments on two datasets, which are TESS and CREMA-D. 6. Evaluating the results obtained with the evaluation parameters: number of trained and untrained parameters, max accuracy val, accuracy test, score, loss, Precision, Recall, F1-score, and Support. 7. Establishing a Comparative Table between our Results and those of the state of the art.
The rest of our paper is structured as follows: Section 2 is devoted to a literature review of speech recognition systems with the individual analysis of emotions and the datasets used. Section 3 is dedicated to the implementation and methodology of the application of our approach with the use of the architecture of the deep neural network of the LSTM type by focusing on the use of Mel-frequency cepstral coeficients (MFCC). Section 4 concerns the experimental results obtained on the datasets with the evaluation metrics used to validate our approach. In conclusion, we will wrap up with a section discussing the primary obstacles and future research directions about speech recognition systems (SER).
In May 2020, De Pinto G. et al. [7] developed a deep neural network for speech recognition with eight emotions. The developed model is based on convolutional neural networks (CNN). The tests were carried out on the RAVDESS database, and the results obtained are of the order of the F1 score of 91,00%, the emotion class Angry with a score of 95,00%, and the class Sad with 87,00%. In February 2022, Puri, T. et al. [8] built a hybrid architecture (LSTM + CNN) using the Hidden Markov Model and Deep Neural Networks (DNN) for a speech recognition system based on eight emotions. They applied their approach to the RAVDESS dataset with a three-branch division strategy (for males and females); the first branch concerns emotions in two positive classes (male and female); the second branch is divided into three emotion classes (positive, negative, and neutral); and finally, the last branch is divided into eight diferent emotion classes. They obtained an accuracy of 98,00%. In September 2022, Gupta, M. V. et al. [9] developed a computer system to detect stress, which has behavioral, emotional, and physical efects. The authors proposed a cascaded RNN-LSTM architectural system and applied their approach to the RAVDESS dataset and obtained an accuracy of 91,00%. In October 2022, Ullah, S. et al. [10] developed an architecture model based on speech recognition of emotions with human-machine interaction. The researchers proposed a one-dimensional CNN (convolutional neural network). They tested it on a combined emotional dataset (Crema-D, Ravdess, Savee, and Tess) with a feature set and classifier ZCR+energy+entropy of energy+RMS+MFCC. The proposed model obtained an accuracy of 92,62%. In November 2022, Vijayan, D. M. et al. [11] proposed two architecture models for speech recognition based on deep learning. The aim is to analyze the emotions of speech and classify them while extracting the spatial and temporal technical characteristics. The first model is a combined CNN-LSTM architecture, and the second is a CNN-Transform encoder architecture. The RAVDESS database was used for the experimentation.The first model obtained an accuracy of 74,00% while the second achieved a better high precision with 82,00%. In January 2023, Ahmed, R. et al. [12] contributed to the implementation of a speech recognition system based on emotion analysis and feature extraction motivation employed on Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). The authors deployed three diferent architectures: the first architecture uses 1D CNN followed by FCN networks (Model-A), the second architecture uses 1D CNN followed by LSTM-FCN networks (Model-B), and the third architecture uses 1D CNN followed by GRU-FCN networks (Model-C). They also used data augmentation by adding Gaussian noise. The experiments were carried out on five databases, and they obtained very accuracy with 95,62% for RAVDESS, 99,46% for TESS, 90,47% for CREMA-D, 95,42% for EMO-DB, and 93,22% for SAVEE. In March 2023, Shah, N. et al. [13] contributed to creating a powerful computational model based on the Mel frequency cepstral coeficients by combining three datasets: RAVDESS, TESS, and SAVEE. The model uses two classifiers, Random Forest and Boosting Ensemble, and the prediction accuracy results are 86,30% for the first classifier and 85,80% for the second classifier. Another learning model was experimented with on the dataset and obtained an accuracy of 75,00%. In July 2023, Bhawesh, K.et al. [14] developed four neural networks for emotion speech recognition using features such as MFCC, chrominance, spectral attenuation, etc. The architecture models used are LSTM, CNN, MLP, and Random Forest models. The experiments were carried out on a combined data set (SAVEE, RAVDESS, CREMA-D, and TESS), and the results obtained with accuracies are of the order of 57,50% for the 1st model, 75,80% for the 2nd model, 59,60% for the 3rd model, 67,80% for the last model. In December 2023, Tyagi, S. et al. [15] developed a computer application based on vocal emotions that allows to understand and identify human emotions from speech. The proposed prototype uses the LSTM architecture with a GWO optimization on a combined dataset with the following databases SAVEE, TESS, EMO-DB, and RAVDESS, and obtained a good accuracy of 65,47% (SAVES), 99,93% (TESS), 78,00% (EMO-DB), and 87,00% (RAVDESS) respectively. In February 2024, Lata, S. et al. [16] experimented with two neural network architectures. The first model is a hybrid architecture of a convolutional neural network (CNN) and long short-term memory (LSTM). The second model consists of an architecture composed of MFCC+LSTM. The aim is to exploit a stack of depth layers in linear form to improve the accuracy of the speech sentiment recognition system. The tests were carried out on the TESS database, and the results obtained are encouraging, with an accuracy of 98,00% for the first model and 96,00% for the second model. In March 2024, Yuan, Z. et al. [17] developed a computer module that relies on a speech recognition system, and more particularly on identity information (this method harms the model generalization). The authors proposed a DTNet-type neural network to dissociate acoustic features from emotional features. The experiments were tested on two databases. They obtained an accuracy of 74,80% for IEMOCAP and 95,00% for Emo-DB. In April 2024, Islam A. et al. [18] built a consistent computing system for speech emotion detection and enhancement. In this context, the authors experimented with their approaches by merging three databases: RAVDESS, TESS, and CREMA-D. They also proposed a hybrid architectural model using CNN and BiLSTM for the eight emotions. The proposed model is based on root mean square energy (RMSE), zero crossing rate (ZCR), and Mel frequency cepstral coeficient (MFCC). The proposed model achieved an accuracy of 97,80%. In June 2024, Akinpelu, S. et al. [19] proposed a computer application of speech recognition based on machine learning to detect emotions. This system is based on the principle of the Vision Transformer (ViT) model. It allows the capture of the characteristics in the images that are adequate indicators of emotional states from the input data of the mel spectrogram introduced into the model. The TESS (Toronto English Speech Set) and EMODB (Berlin Emotional Database) were used for the experiments. The results are satisfactory, as they obtained an accuracy of 98,00% (TESS), 91,00% (EMO-DB), and 93,00% (TESS+EMO-DB). In August 2024, Hossain, I. et al. [20] implemented a hybrid model to extract information and improve prediction accuracy with probability calculated. The model uses the convolutional neural network (CNN) architecture to extract features from the speech spectrogram. After that, the long short-term memory processes the features (LSTM). The authors used a KNN classifier to classify emotions and make predictions. They conducted experiments using the TESS database and achieved an accuracy of 98,21%.
The proposed approach is based on two architectural models, and one of them has given excellent results and is even better than some methods in the literature. The contributions of our model are defined as follows: • The development of a new neural network architecture based on the long short-term memory (LSTM) network, which is dedicated to the classification and analysis of human emotions. • The proposed LSTM structure features nine layers, beginning with an input layer for data and ending with an output layer that uses the softmax activation function to classify seven distinct emotions. It is composed of seven hidden layers that implement optimization methods, such as dropout and dense layers. • The suggested LSTM model achieved an accuracy of 98,92% for classifying emotions using the
TESS database, including pleasant (ps), anger, happiness, disgust, fear, sadness, and neutral. • The LSTM model is lightweight and contains about 307 655 parameters. It allows for accelerated learning and reduces the computation time for emotion class prediction.
The basis of a speech-emotion recognition system is the quality of the audio file database because a good-quality dataset implies an eficient and robust system. If the audio file database contains noise, it must go through refining preprocessing to clean it, and this is done through the process of filtering, encapsulation, and integration [21]. A SER goes through the data collection phase, preprocessing, feature extraction, feature selection, classification, and recognition [ 22]. The data collection stage is the most sensitive phase of the system (Collect the data thoroughly and ensure that it meets high-quality standards). The University of Marburg in Slovenia was the birthplace of the first database for an SER system, as per the literature [23]. It consists of six types of emotions, and the audio files are in MPEG-4 format [24]. The number of utterances in the dataset is 186 for each emotional category.
Before entering the data into the neural network for learning, it is necessary to go through a preprocessing for feature extraction of the audio files and transform them into mathematical coeficients for the classification and recognition of emotions. For this, we need the following parameters: • Mel scale: It is a mathematical scale that allows the height to be measured (acoustic scale), so the unit of measurement is the Mel, with a sound characteristic of high or low. • Frequency: represents the number of oscillations (vibrations) of the sound per second, with the unit of measurement Hertz. • Chromagram: represents the intensity of the audio signal at a given moment, it is made up of a chrome vector (size of 12 dimensions with 12 semitones of the chromatic scale). • Pitch: represents the vibration frequency corresponding to the sounds. • Fourier transform: allows the decomposition of the periodic signals and is the link between the temporal signal and the frequency representation.
Subsequently, the Mel spectrogram will be used for individual identification with speech recognition and emotional states. It is represented by an image whose x-axis represents time, and the y-axis represents the frequency with the application of a logarithmic scale in the dot diagram [25].
The Mel-frequency cepstral coeficients (MFCC) are used for speech recognition, and this is done by following the steps above: 1. Calculate the Fourier transform of the signal. 2. Establish a power mapping of the spectrum obtained previously on the Mel scale using interleaved triangular windows. 3. Collect the power recordings for each Mel frequency. 4. Observe the discrete cosine transform (DCT) of the list of mel log powers as if it were a signal. 5. The amplitudes of the resulting spectrum are called MFCC.
6. For each audio recording, 40 MFCC values [26] are used as input data into the neural network.
The proposed neural network Model-A is of the LSTM (language short-term memory) type [27], with as input the first audio recording vector and these 40 values of the MFCC coeficients. The network comprises an LSTM input layer of 256 neurons and an output layer with the softmax activation function, and these seven neurons correspond to emotions (selection probabilities). The network has seven hidden layers with four dropout layers of 0,2 %, three dense layers with the relu activation function, a dense layer of 128 neurons, a dense layer of 64, and another thick layer of 32 neurons (see Fig.3).
Model-B is a hybrid CNN-LSTM architecture [28], and it is a combined neural network architecture with a total of 18 layers. The input layer consists of 32 neurons with a relu activation function and, as input data, a vector that contains 40 MFCC coeficient values. An output layer with the softmax activation function and the six neurons (correspond to the emotions of speech recognition). The network is composed of 16 hidden layers with five dropout layers (2 layers of 30% and three layers of 50%), two convolution layers of 32 and 64 neurons with a relu activation function, two max-pooling layers of two dimensions, a flattened layer, two LSTM layers with 128 neurons, a dense layer of 128 neurons with the relu activation function, and finally a batch normalization layer (see Fig.4).
During the experimentation phase in both neural network architecture models we used the following hyperparameters (see Table 1).
Dataset 1: The TESS dataset is a database that contains audio files for seven emotions (pleasant (ps), anger, happiness, disgust, fear, sadness, and neutral) [29]. These recordings were made by two actresses aged 26 and 64 recruited in the Toronto area as English-speaking actresses with university studies and musical training. The dataset contains 2 800 audio files (These sound recordings were based on the Northwestern University Hearing Test Number 6) with a set of 200 words relating to a sentence that says "Say the word ........".
Dataset 2: The CREMA-D data set consists of 7 442 original recordings featuring 91 actors, with 48 male and 43 female participants aged between 20 and 74 from diverse racial and ethnic backgrounds such as Hispanic, African, Caucasian, Asian, American, and Unspecified [ 30]. The actors delivered 12 specific sentences while expressing six diferent emotions (Happy, Neutral, Sad, Fear, Anger, and Disgust) at four varying levels (Low, Medium, High, and Unspecified).
To evaluate the results obtained from the experiments of our approach on the two datasets, we used the following criteria: number of trained parameters (train param), number of untrained parameters (untrain param), total number of parameters (total param), top loss (validation), top accuracy (validation), score (test), and accuracy (test).
On the other hand, we have taken specific evaluation criteria to measure the performance of our approach on the image corpus. F1-score, Recall, Precision, and Support parameters are calculated based on the TP (true positive), TN (true negative), FP (false positive), and FN (false negative) [31]. These parameters are defined as follows: =
+ =
+
1 − =
2 × ×
+
(
(
Figure 6 illustrates the outcomes of the confusion matrix from experiments conducted on the TESS database for identifying the emotions: anger, happiness, pleasant, disgust, fear, sadness, and neutral. From this matrix, we observe that the accuracy for the emotion categories fear, sad, angry, disgust, and happy is notably higher than that for the other categories; conversely, the neutral and pleasant emotions exhibit a lower accuracy. This indicates that the model maintains an almost uniform distribution across all emotion categories without clear distinctions. Furthermore, the overall accuracy of the emotion recognition system stands at 98,92 %.
The application of our approach and the experiments and tests carried out on the two emotion databases Crema-d and TESS with the two architectural models LSTM and CNN + LSTM demonstrate the results obtained with the CNN + LSTM model on the dataset Crema-d with an accuracy of 54,07% and the LSTM model on the TESS dataset with an accuracy of 98,92%. This also implies that the proposed model is so efective with the LSTM architecture with all the classes of emotions that there is an almost uniform distribution in the accuracy, which is very adaptable for an emotion recognition system.
The introduction of deep learning has significantly enhanced research in areas like medicine [ 32], biometrics [33], and even more speech recognition systems, especially through the use of convolutional neural network (CNN) architectures. We aim to enhance our model by utilizing a more extensive dataset of emotions, and additionally, we will analyze speech to determine whether the patient is experiencing a psychological disorder.
7356
Either: The author(s) have not employed any Generative AI tools. [5] D. R. Hammou, Classification and detection of covid-19 in human respiratory lungs using convolutional neural network architectures, in: 2021 International Conference on Artificial Intelligence for Cyber Security Systems and Privacy (AI-CSP), IEEE, 2021. doi:10.1109/ai-csp52968.2021.9671158. [6] D. R. Hammou, S. A. Mahmoudi, R. Adjoudj, Multi-Biometric Iris Recognition System Using Consensus Between Convolutional Neural Network Architectures, Int. J. Organ. Collect. Intell. 12.1 (2022) 1–30. doi:10.4018/ijoci.305210. [7] M. G. de Pinto, M. Polignano, P. Lops, G. Semeraro, Emotions Understanding Model from Spoken Language using Deep Neural Networks and Mel-Frequency Cepstral Coeficients, in: 2020 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), IEEE, 2020. doi:10.1109/eais48028. 2020.9122698. [8] T. Puri, M. Soni, G. Dhiman, O. Ibrahim Khalaf, M. alazzam, I. Raza Khan, Detection of Emotion of Speech for RAVDESS Audio Using Hybrid Convolution Neural Network, J. Healthc. Eng. (2022) 1–9. doi:10.1155/2022/8472947. [9] M. V. Gupta, S. Vaikole, A. D. Oza, A. Patel, D. P. Burduhos-Nergis, D. D. Burduhos-Nergis, AudioVisual Stress Classification Using Cascaded RNN-LSTM Networks, Bioengineering 9.10 (2022) 510. doi:10.3390/bioengineering9100510. [10] S. Ullah, Q. A. Sahib, Faizullah, S. Ullahh, I. U. Haq, I. Ullah, Speech Emotion Recognition Using Deep Neural Networks, in: 2022 International Conference on IT and Industrial Technologies (ICIT), IEEE, 2022. doi:10.1109/icit56493.2022.9989197. [11] D. M. Vijayan, A. A. V, G. R, A. N. S. A, R. C. Roy, Development and Analysis of Convolutional Neural Network based Accurate Speech Emotion Recognition Models, in: 2022 IEEE 19th India Council International Conference (INDICON), IEEE, 2022. doi:10.1109/indicon56171.2022.10040174. [12] M. Rayhan Ahmed, S. Islam, A. K. M. Muzahidul Islam, S. Shatabda, An Ensemble 1D-CNN-LSTMGRU Model with Data Augmentation for Speech Emotion Recognition, Expert Syst. With Appl. (2023) 119633. doi:10.1016/j.eswa.2023.119633. [13] N. Shah, K. Sood, J. Arora, Speech emotion recognition for psychotherapy: an analysis of traditional machine learning and deep learning techniques, in: 2023 IEEE 13th Annual Computing and Communication Workshop and Conference (CCWC), IEEE, 2023. doi:10.1109/ccwc57344.2023. 10099344. [14] K. Bhawesh, D. Mustafi, A Comparison of Deep Learning and Machine Learning Models for Speech Emotion Recognition Using Multiple Features, in: 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), IEEE, 2023. doi:10.1109/ icccnt56998.2023.10307514. [15] S. Tyagi, S. Szénási, Optimizing Speech Emotion Recognition with Deep Learning and Grey Wolf
Optimization: A Multi-Dataset Approach, Algorithms 17.3 (2024) 90. doi:10.3390/a17030090. [16] S. Lata, N. Kishore, P. Sangwan, A Comparative Analysis of CNNLSTM and MFCCLSTM for Sentiment Recognition from Speech Signals,” Int J Intell Syst Appl Eng, vol. 12, no. 21s, pp. 4392–4402, Mar. (2024), Accessed: Nov. 22, 2024. [Online]. Available: https://ijisae.org/index.php/ IJISAE/article/view/6295. [17] Z. Yuan, C. L. Philip Chen, S. Li, T. Zhang, Disentanglement Network: Disentangle the Emotional Features from Acoustic Features for Speech Emotion Recognition, in: ICASSP 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2024. doi:10.1109/icassp48485.2024.10448044. [18] A. Islam, M. Foysal, M. I. Ahmed, Emotion Recognition from Speech Audio Signals using CNNBiLSTM Hybrid Model, in: 2024 3rd International Conference on Advancement in Electrical and Electronic Engineering (ICAEEE), IEEE, 2024. doi:10.1109/icaeee62219.2024.10561755. [19] S. Akinpelu, S. Viriri, A. Adegun, An enhanced speech emotion recognition using vision transformer, Sci. Rep. 14.1 (2024). doi:10.1038/s41598-024-63776-4. [20] I. Hossain, M. Islam, T. Nahrin, M. Rashed, M. Rahman, Improving Speech Emotion Recognition and Classification Accuracy Using Hybrid CNN-LSTM-KNN Model. (2024), International Journal of Research Publication and Reviews Journal Homepage: Www.ijrpr.com, 5. Retrieved from URL:https://ijrpr.com/uploads/V5ISSUE8/IJRPR32597.pdf.