A promising area of research and development that can significantly increase the eficacy and accuracy of mental health assessments is using artificial intelligence (AI) and machine learning algorithms to analyse voice and facial expressions simultaneously in a video stream. More studies are required to completely comprehend the capabilities and limitations of these technologies and guarantee their ethical and efective usage in clinical settings. Collaborative robots (cobots) can potentially change how mental evaluations of autistic children are approached completely. ChatGPT is an efective language model that can understand and produce human-like text. When used in conjunction with the Cobot, this technology enables children with autism to interact and communicate in a way that is natural to them. In this article, we introduce a novel method for analysing emotional detection using voice analysis and facial recognition that has been tested on the IEMOCAP database. The outcomes session, which illustrates the tool's potential use in healthcare, concludes the paper. emotion recognition, spoken language, deep learning, convolutional neural network, web-shaped model In the field of digital health, there has been extensive research on emotion recognition utilising spoken language and facial expressions [1, 2]. Deep learning techniques are also included among the strategies involved to solve this problem[3]: it is possible to train convolutional neural networks (CNN) on large collections of photos with the proper emotions labelled on them and subsequently recognize picture patterns associated with emotions [4]. Furthermore, recurrent neural networks (RNN) can be used to analyze audio data for spoken words [5]. An assortment of audio recordings that have each been given a diferent emotion can be used to train the RNN. The network can then develop the ability to identify audio data patterns that represent various emotional states. The CNN and RNN can both be trained independently ∗Corresponding author. †These authors contributed equally.
CEUR ceur-ws.org
(D. Greco)
CEUR
Workshop
Proceedings
before being integrated into a single model that can identify emotions through both spoken and
facial expressions. To distinguish emotions from spoken language and facial expressions, other
deep learning model types, such as long short-term memory (LSTM) networks or transformers,
can also be utilised[
Audio-visual emotion recognition has been recently reviewed in several survey publications, e.g.
[
Mainly using Bayesian models, where each emotion category is modelled by a GMM/HMM. For
instance, in order to fuse numerous signals for an Error-Weighted Classifier (EWC), [
Supervised learning is definitely the most used for several years for emotion recognition in
conversational speech, either from audio or video. For instance, [
Unsupervised learning can be useful either for emotion clustering or feature extraction. Emotion
clustering consists in discovering emotion groups which do not necessarily match with unique
labels. For e.g. a cluster grouping ‘excitement’ and ‘anger’ signals may indicate ‘high arousal’,
whereas ‘calm’ and ‘sadness’, when clustered together, indicate ‘low arousal’. Then, such a type
of clustering would be useful in proposing a novel classification of emotions. This track has
been investigated in [
They can be classified into three main categories: a) speech emotion datasets, b) facial expression
datasets and c) video and/or multimodal emotion recognition. Most of these datasets use either
spontaneous or acted speech or scenes. The corresponding labels are generally provided either
in a categorical or a dimensional scheme. Categorical labels usually follow either the basic
emotion model proposed by Eckman or Plutchik’s compound model, known also as the wheel
of emotions. On the other hand, the dimensional model represents emotions in a 3-D coordinate
system where each emotion is characterized by a score of i) valence, i.e. positive or negative, ii)
activation, i.e. high or low, and iii) dominance [
Therefore, most emotion recognition datasets are built following either the basic, the
compound or the dimensional model. For instance, speech emotion recognition datasets like
EMO-DB and EMOVO contain every 6 basic emotions plus neutrality [
In recent years, a substantial body of research has been conducted on the use of convolutional neural networks (CNNs) for voice emotion recognition. The complexity and dynamic nature of speech signals, which are influenced by several elements like speaking style, pronunciation, and accent, make it dificult to recognize emotions in speech. However, it is also a vital area of research since emotions are fundamental to communication and their understanding can have a big impact on a variety of applications, including speech-based human-computer interaction, afective computing, and mental health diagnostics.
They represent a set of features intentionally designed for speech emotion recognition. Such features have been selected to form the well-known GeMAPS feature set[18]. This feature set includes several types of prosodic, acoustic and spectral descriptors, computed at the raw signal level. Several variants of GeMAPS features are available in OpenSmile toolbox [19], each for a particular task in afective speech computing. These features were proven to give high afective speech recognition rates in several challenges, such as Interspeech’09 emotion challenge [20], Computational Paralinguistics challenge (ComParE) [21]. However, they belong to the paradigm of explicit feature extraction, whereas novel feature extraction methods are moving towards the concept of an end-to-end learning process.
A more recent technique of feature extraction in speech emotion recognition consists in the
usage of spectrogram images as an implicit representation of the Mel-Frequency Cepstral
Coeficients (MFCCs)[ 22]. A spectrogram is a time-frequency representation of speech signals
that can be used to identify the spectral and temporal aspects of speech, which are significant
markers of emotional state. The collection of MFCCs, on the other hand, is resilient to changes
in speaking style, pronunciation, and accent. They summarize the spectral envelope of speech
signals. The spectrogram can be utilized as input for speech emotion classification in several
ways by a) extracting feature vectors composed of MFCC coeficients and their Δ and Δ-Δ, i.e.
their first and second derivatives; b) feature embedding, usually through learning latent features
collected at the code layer of an autoencoder [
When it comes to using spectrogram images as input, convolutional neural networks (CNN) are preferred as classifier methods, since they excel in classifying images and they can accurately identify patterns in the input data. When recognizing emotions in speech, a CNN uses the spectrogram or MFCCs as input and trains on a sizable dataset of speech signals to learn how to categorize the emotions. The combination of feature extraction, CNNs, and numerous other techniques can considerably enhance the precision, robustness, and generalization of emotion recognition in speech. The absence of significant and diverse datasets, the dificulty of defining a set of emotions that is consistent, and the impact of numerous confounding variables on speech signals are just a few of the dificulties the area is now experiencing.
From learning platforms to human-computer and human-robot interaction, facial emotion recognition (FER) is widely used in a variety of activities. It can also be used as an intrinsic technique for face recognition issues to produce an expression-free face classifier. Most approaches focus on building ever-deeper neural networks that regard an expression as a still image or, in certain cases, as a sequence of succeeding frames that reflect the expression’s temporal component [23].
FER usually starts with the extraction of facial features, which is strictly succeeded by an emotion classification method. This section presents the most recent works that led to the experimentation of diferent techniques. Methods for feature extraction are mainly divided into geometric methods and appearance-based methods. In particular, geometric methods focus on the shape, scale, orientation, and location of the various parts of the face, such as the nose, mouth, eyes and eyebrows [24]. In this context, Active Shape Model is a feature-matching solution that focuses on point features and measures several shape variations and a range of adaptive models on the face [25]. The authors in [26] propose a technique that consists of an estimate of parameters extracted from the wavelet transform, and then an SVM classifier performs the emotion classification. In [ 27], oriented gradient histograms (HOG), Gabor, and local binary pattern (LBP) have been used as feature extractors; subsequently, a simple k-nearest neighbour (kNN) was used for the classification. The experiments carried out aimed to demonstrate the efectiveness of these three feature extractors in the FER. The authors in [ 28] propose a system of FER jointly with the gender and age of the subject to understand how much these afect facial expressions. The authors used the classic KNN and SVM classifiers and deep learning models such as CNN and VGG16.
The computational cost of the training phase applies to these, which could take hours or days
to complete. This work proposes the Web Shaped Model (WSM), a geometric method to extract
features that can distinguish between various facial emotions. It uses a virtual geometric pattern
resembling a spider’s web drawn on a face. The Web Shaped model was introduced in [
An experimental setup has been established to carry out both tasks simultaneously and jointly in order to evaluate the eficacy of merging spoken language and facial expressions for emotion recognition. As a result, the following tests were carried out using conversational speech videos that were either improvised or scripted and were included in the IEMOCAP dataset [17]:
The combined process is illustrated in Fig. 3, both for facial images and speech emotion classification.
To implement the audio signal classifier, we opted to use spectrogram images as input and CNNs as classifiers. Thus, no explicit features, such as OpenSmile LLD’s [ 19] were used. The rationale behind such a choice is simply empirical, based on the performance of each type of input. Feature extraction and classification are achieved at the frame level, using the following process: • Each audio signal, generally corresponding to one uttered sentence, is segmented into overlapping short frames, of 100 ms each. Actually, such a duration is the minimum that can carry emotion. • For each frame, a series of spectrogram images is extracted, applying a short-time Fourier transform at overlapping windows. • The spectrogram images are fed into a CNN network, composed of several 2-D convolutional layers, each followed by a max-pooling layer. The last layer is a 1-D Dense layer with a softmax activation that returns a classification probability. • Once all frames belonging to a given sentence are classified, a majority voting is applied to decide about the sentence’s label.
The results of this workflow obtained on a separate test set randomly extracted from the same session of IEMOCAP database used for training are mentioned in Table 1.
For each time interval of the images, the WSM coding from each face was extracted and labelled with the corresponding emotion. The classifier was then trained and tested with the same fragments used in audio testing. In previous articles, the spider web has been used only for a single face, in this work the emotions have been classified by video intervals. Each image in the video range is classified and the classification is carried out for a single frame. Of all the predicted classes in the range, the majority voting value is considered to determine the predicted class in the whole fragment. The classification was done with various known state-of-the-art classifiers to compare with which we obtained the best performance. The best-performing classifier is the Random Forest Classifier with the parameters: • The whole dataset is used to build each tree. • tree splitting criterion = entropy. This criterion is equivalent to minimizing the crossentropy and multinomial deviance) between the true labels and the probabilistic prediction.
• number of estimators = 1000.
Results show diferences for each type of input, i.e voice signals and facial images, and for each category of emotion. Whereas some emotions, like happiness and sadness are pretty well recognized, either by audio or image classification, other ones are much less recognized. For instance, audio-based classification presents low recall rates for most emotions, which indicates high false positive rates. This may be due to the inability of the spectrogram to decorrelate emotions and voices. Actually, once the CNN is trained on a certain pattern of the spectrogram, it learns the characteristic spectral coeficients of the voice, and hence, becomes more prone to recognize the voice than the emotion. In other words, if a voice is frequently labelled as happy, it would also be labelled so even if the target emotion is diferent. FER results are highly dependent on the type of images acquired. The emotions of the IEMOCAP dataset are realistic and include many microexpressions that are unlikely to be captured by cameras with a framerate lower than 30FPS [30]. The problem in question is of a technical nature and also justifies the low result of the other class, which is a collection of unidentified emotions. Otherwise, the general classification trend follows speech recognition in which classic emotions such as sadness and happiness are quite well classified.
Multimodal emotion recognition from audio and video channels can open new avenues in mental health monitoring. Information gathered from the conversation with a Cobot and ChatGPT can be reviewed in real-time, providing the evaluator with immediate feedback. Since children with autism often have dificulty taking typical exams that rely on lengthy questionnaires or standardized exams, this can be extremely useful for them. To explore this avenue, we presented preliminary results on emotion recognition on the IEMOCAP database, which provides voice data and video. Experiments demonstrate how separately it is possible to detect this information. Future experiments may consist in: a) improving the results of emotion recognition by each type of classifier, b) integrating both classifiers into a single multimodal framework obtained from the conjunction of both extracted feature sets using data integration with fuzzy methods [31], and applying visual transformers to introduce recurrent learning. unsupervised learning approach, Int. J. Comput. Intell. Syst. 14 (1) (2021) 23–35. [16] S. Rovetta, Z. Mnasri, F. Masulli, Emotional content comparison in speech signal using feature embedding, Progresses in Artificial Intelligence and Neural Systems (2021) 45–55. [17] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, S. S.
Narayanan, Iemocap: Interactive emotional dyadic motion capture database, Language Resources and Evaluation 42 (4) (2008) 335–359. [18] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, K. P. Truong, The geneva minimalistic acoustic parameter set (gemaps) for voice research and afective computing, IEEE Transactions on Afective Computing 7 (2) (2015) 190–202. [19] F. Eyben, B. Schuller, opensmile:) the munich open-source large-scale multimedia feature extractor, ACM SIGMultimedia Records 6 (4) (2015) 4–13. [20] B. Schuller, S. Steidl, A. Batliner, The interspeech 2009 emotion challenge, in: Interspeech,
Vol. 2009, 2009, p. 312. [21] B. Schuller, S. Steidl, A. Batliner, S. Hantke, F. Hönig, J. R. Orozco-Arroyave, E. Nöth, Y. Zhang, F. Weninger, The interspeech 2015 computational paralinguistics challenge: nativeness, parkinson’s & eating condition, in: Sixteenth annual conference of the international speech communication association, 2015. [22] R. M. Stern, A. Schafer, Robust modeling of speech signals, IEEE Transactions on Speech and Audio Processing 7 (5) (1999) 525–532. doi:10.1109/89.784104. [23] S. Lee, A brief review of deep learning for facial expression recognition, Available at SSRN 4318896 (2023).
URL http://dx.doi.org/10.2139/ssrn.4318896 [24] N. Rathour, R. Singh, A. Gehlot, S. V. Akram, A. K. Thakur, A. Kumar, The decadal perspective of facial emotion processing and recognition: A survey, Displays 75 (2022) 102330. doi:10.1016/j.displa.2022.102330. [25] T. Cootes, C. Taylor, D. Cooper, J. Graham, Active shape models-their training and application, Computer Vision and Image Understanding 61 (1) (1995) 38–59. doi: 10.1006/cviu.1995.1004. [26] J. R. Kumar, R., M. Sundaram, N. Arumugam, et al., Face feature extraction fohr emotion recognition using statistical parameters from subband selective multilevel stationary biorthogonal wavelet transform, Soft Computing 25 (2021) 5483–5501. doi:10.1007/ s00500-020-05550-y. [27] S. Subudhiray, H. K. Palo, N. Das, K-nearest neighbor based facial emotion recognition using efective features, IAES International Journal of Artificial Intelligence 12 (1) (2023) 57. [28] S. T. Chavali, C. T. Kandavalli, T. M. Sugash, R. Subramani, Smart facial emotion recognition with gender and age factor estimation, Procedia Computer Science 218 (2023) 113–123. doi:10.1016/j.procs.2022.12.407. [29] V. Kazemi, J. Sullivan, One millisecond face alignment with an ensemble of regression trees, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1867–1874. [30] D. Borza, R. Danescu, R. Itu, A. Darabant, High-speed video system for micro-expression detection and recognition, Sensors 17 (12) (2017) 2913. doi:10.3390/s17122913. [31] A. Ciaramella, D. Nardone, A. Staiano, Data integration by fuzzy similarity-based hierarchical clustering, BMC Bioinformatics 21 (Suppl 10) (2020) 350. doi:10.1186/ s12859- 020- 03567- 6.