Recognizing and understanding emotions is essential for improving human-computer interaction and has diverse applications, including supporting mental health and increasing customer satisfaction in various industries. As interactive systems are increasingly expected to perceive, understand, and express emotions like humans, automatic emotion recognition becomes crucial in practical contexts. Social networks, which are the main channels for rapid information dissemination, often display users' emotions through their shared messages, which has a significant impact on the spread of misinformation. By integrating emotion recognition tools into disinformation detection, we can improve our ability to counter information manipulation and foster a more transparent, evidence-based information environment. This research aims to develop, evaluate, and apply multimodal emotion recognition methods, particularly in political contexts, to combat misinformation and analyze the use of fallacies in political discourse.
CEUR ceur-ws.org
Emotions are complex responses, both psychological and physiological, that people experience
to various stimuli. These reactions are often associated with specific thoughts, feelings, and
behaviors [
Understanding and recognizing emotions is critical not only for improving human-computer
interaction, but also for a wide range of applications, from supporting mental health to improving
customer satisfaction in a variety of industries [
Therefore, automatic Emotion Recognition (ER) plays an important role in real-world
applications, as interactive machines are increasingly expected to be able to perceive, understand,
and express emotions on the same level as humans [
Social networks have become the primary platform for the rapid dissemination of information
to the general public. Despite its benefits, the lack of efective regulatory measures has led to a
lfood of fake news and rumors on the Internet, making it dificult to distinguish between truthful
information and misinformation, thus afecting the eficiency of information exchange on social
networks [
The objective of this PhD thesis is to design and evaluate of diferent multimodal ER approaches and apply them in diferent domains such as political to identify and address misinformation, as well as to understand the use of fallacies in political discourse. We focused our proposal on Spanish because it is the third most used language on the Internet, only after English and Chinese. Despite its importance, there remains a noticeable lack of linguistic resources to perform multimodal ER in some domains, such as the political domain.
The research hypotheses in this paper are related to the multimodal emotion detection approach for solving misinformation and fallacy identification tasks in the political domain. Our first research hypothesis is that the multimodal ER approach improves the performance of text-based emotion analysis. Our second research hypothesis is that the multimodal ER approach improves misinformation and fallacy identification in the political domain.
To validate these research hypotheses, we established the following objectives: • OB01. Creation of a multimodal corpus in Spanish for emotion recognition and for disinformation detection. This corpus will be built from a crawler that extracts videos, texts, and audios from diferent YouTube channels related to politics, sports and entertainment, as well as from recordings of sessions of the Congress of Deputies. • OB02. Evaluation of diferent multimodal approaches that combine features of diferent modalities for ER. In this case, it can be the combination of audio with text, text with images or the combination of all three. Moreover, we are also evaluating our proposal by participating in several shared task regarding multimodal classification from diferent workshops. • OB03. Identification of misinformation and fallacies using ER. To validate our hypothesis, we are compiling diferent multimodal political disinformation corpus and making them available to the scientific community.
In this section, we describe the methodology and experiments we have created to help us validate our hypotheses.
Our research is primarily focused on addressing the limitations of current approaches to detecting misinformation and deception in the political domain through the analysis of emotions.
In the preliminary stages of our research, we recognized the need for emotion analysis in a wide range of applications, from supporting mental health to improving customer satisfaction in a variety of industries. In addition, we identified the current lack of multimodal resources in Spanish for ER in speech, despite the fact that Spanish is one of the most widely spoken languages in the world, with millions of speakers in diferent continents. The few available datasets sufer in some cases from poor annotation, making it dificult to teach models to accurately identify emotions. In other cases, voice recordings are made by professionals under conditions that do not reflect real-life situations, resulting in a lack of authenticity in emotional expression [7] and poor results in real-life scenarios [8]. Therefore, one of the current challenges in Spanish speech ER is the creation of more robust and representative datasets.
To address this problem, we have created and published a new corpus for multimodal ER in Spanish (Spanish MEACorpus 2023) [9], which contains 13.16 hours of speech divided into 5,129 labeled segments, taking into account Ekman’s six basic emotions (disgust, anger, happiness, sadness, fear, neutral) and annotated by three members of our research group (two men and one woman, all middle-aged). In this paper, we have also evaluated diferent multimodal approaches that combine speech representation techniques and linguistic models to perform emotion classification. We have evaluated approaches ranging from simple text-based emotion detection to approaches based on the fusion of automatic speech recognition models such as Wav2Vec2BERT [10] with pre-trained models such as BETO. Among the model fusion approaches, we have evaluated the late fusion approach, which consists of concatenating or averaging the outputs obtained in Wav2Vec-BERT and BETO; fusion with multi-head cross-attention mechanism, which incorporates a cross-attention mechanism in the hidden state vector of the last layer to better capture complex audio-text relationships, generate richer contextual representations, and increase the generalization capability of the model; and ensemble learning, which is based on combining feature sets using ensemble learning, where the output of each neural network model (Wav2Vec2-BERT and MarIA) is combined by averaging the predictions of each emotion (mean) or selecting the prediction with the highest probability (maximum).
As a contribution to my Ph.D. thesis, this paper shows that multimodal ER models combining audio and text perform better than unimodal models based on text alone, demonstrating that audio and text features complement each other. In addition, the late fusion approach with concatenation strategy is identified to perform better.
Moreover, this dataset has been used as a basis for the organization of the EmoSPeech task [11] in IberLEF 2024, which consists of two subtasks: text-based automatic ER and multimodal automatic ER. The novelty of this task lies in its multimodal approach to ER, analyzing the performance of language models on Spanish MEACorpus 2023.
Furthermore, we have participated in several shared task related to multimodal classification in several relevant evaluation forums such as CLEF and SemEval. First, in SemEval-2024 Task 4[12], which objective is identify persuasive techniques in memes. For this task, we evaluated LlaVa to extract image descriptions and combine them with the meme text. Our system performed well in all subtasks, achieving the tenth best result with a Hierarchical F1 of 64.774%, the fourth best in Subtask 2a with a Hierarchical F1 of 69.003%, and the eighth best in Subtask 2b with a Macro F1 of 78.660%. Second, in SemEval-2024 Task 10 [13], focused on recognizing and reasoning about emotional changes in conversations. This task included diferent languages such as English and Hindi. Our best result was the 6th position in Subtask 2 with an F1 score of 26%. Third, on EXIST 2024 [14], focused on the identification and categorization of sexism in memes. We used the CLIP model to extract the embedded text and image, and then combined them by diagonal multiplication to obtain the classification models. We ranked 33rd in sexism identification and 18th in both source intent and sexism categorization.
Participating in these evaluation forums has allowed me to validate the proposed approaches for multimodal classification that combine features from images and text. In the future, the plan is to integrate representations from diferent modalities such as text, images, and audio to improve the task of emotion identification.
Recent advances in Artificial Intelligence (AI) and the emergence of Large Language Models
(LLMs) like Instruct-GPT [15], ChatGPT, and GPT-4 [16], have made it easier to generate false
information that appears very convincing [17]. Therefore, there is an urgent need to detect
misinformation eficiently and efectively. Most studies provide high-level summaries of the
methods, techniques, and features used to detect rumors and fake news, while others discuss
possible applications of these methods, including posture detection and source detection [
Currently, disinformation detection approaches consist of three main components: (i) the
dataset used to develop them, (ii) the methods used to perform the detection, and (iii) the features
considered in these methods [
Detection methods can be divided into those based on conventional Machine Learning (ML) and Deep Learning (DL) techniques. We tested new approaches to misinformation detection, such as using the “5W1H” technique commonly used by journalists to clearly and explicitly present the most important information in a news item and to assess the reliability of the language presented in the news. We participated in FLARES at IberLEF 2024 [18], which uses the 5W1H technique to assess the reliability of language in news items. This task is divided into two subtasks: (i) 5W1H item identification and (ii) 5W1H-based reliability assessment. We participated in both tasks. For Task 1, we developed a Named Entity Recognition (NER) model using tight transformation models such as BERT and MarIA, and integrating Part-Of-Speech (POS) features and Syntactic Dependencies (Dep). Our BETO + POS + Dep model achieved the 1https://www.snopes.com/ 2https://www.factcheck.org/ 3https://www.politifact.com/ second best result with a score of 56.778%. In Task 2, which focused on assessing the reliability of 5W1H entities, our approach based on contextual adaptation of the MarIA model achieved the best result with a score of 65.820%.
Within the afective feature for disinformation detection, dual emotion features seek to
account for the importance of considering diferent emotional perspective in disinformation
identification, i.e., both the emotion of the publisher, which refers to emotions conveyed in an
original post that starts a thread on social networks, and the social emotion, which refers to
emotions expressed in follow-up posts that respond and/or comment on the original post [
Emotion-based misinformation detection combines emotion and sentiment information with
other features to maximize performance. According to [
According to the study conducted by [
Fallacious arguments were originally defined as defective inferences, i.e., types of arguments that are logically invalid. More recently, from a pragmatic perspective, they have been defined as violations of the typical performance rules of a particular ideal type of argumentative engagement [22], and as inappropriate shifts between diferent types of dialog, noting that the intended move is inappropriate within the applied pragmatic context [23]. Although it occurs in a variety of settings, political debate serves as a natural testing ground for this form of fallacious reasoning. For example, the ad hominem fallacy, which attacks the person or entity making the argument rather than the argument itself, is one of the most commonly cited fallacies in political debate. Such arguments can sound convincing and are intended to mislead the audience into believing the validity of the argument [24].
Currently, there are several types of fallacies, such as hasty generalization, ad hominem, ad populum, false causality, circular reasoning, appeal to emotions, red herring, false deduction, credibility fallacy, false dilemma, straw man, and intention [25]. After analyzing these fallacies, we found that some of them are closely related to the emotion of contempt. For example, contempt may lead one to disqualify the opponent by attacking his character, intelligence, or morality instead of addressing his arguments, which leads to ad hominem fallacies. In addition, contempt can motivate the presentation of arguments in an exaggerated or oversimplified manner in order to distort the opponent’s argument, which is a straw man fallacy. Another fallacy related to the emotion of contempt is the appeal to emotion, which consists of appealing to emotions rather than presenting a logical argument, i.e., using disgust or contempt to evoke a strong emotional response that distracts from the logic of the argument. For example, the question “Do you really want to support someone who advocates such disgusting practices?” for the opponent.
In conclusion, the use of emotions for disinformation detection is an area that has already been explored and experimented with, and emotions have been shown to be complementary to disinformation detection. Integrating emotion recognition tools and techniques into disinformation detection will strengthen the ability to combat information manipulation and promote a more transparent and evidence-based information environment. However, challenges remain, such as the lack of public datasets in Spanish and problems related to multimodality. Regarding fallacies, a strong relationship with emotions is observed, but studies exploring this possibility to improve detection are still lacking.
Therefore, my thesis will focus on the creation of a multimodal public dataset in Spanish
for emotion identification and misinformation detection, aiming to address the lack of public
datasets in these two research areas. In addition, diferent multimodal approaches for
emotionbased misinformation detection, which is one of the main challenges raised in [
We are currently in the phase of identifying emotions related to misinformation and fallacies, as well as collecting more multimodal data in the Spanish MEACorpus 2023. In addition, we are adding examples of the emotion of contempt to the corpus in order to improve the performance of the models in identifying misinformation and fallacies.
This Ph.D. thesis aims to investigate multimodal emotion recognition and apply it in the political domain to detect misinformation or fallacies. To achieve this goal, first, the current state of the art in the field was reviewed. Second, a multimodal dataset of audio and text, called Spanish MEACorpus 2023 [9], has been compiled and published for emotion recognition in Spanish and diferent multimodal approaches have been evaluated, from text-based ER to approaches based on the fusion of automatic speech recognition models such as Wav2Vec2-BERT with pretrained models such as BETO. Thirdly, we have participated in diferent shared tasks of diferent evaluation forums such as CLEF, IberLEF, and SemEval to validate our proposed approaches for the task of multimodal classification, emotion detection and disinformation detection.
We found that emotion improves the identification of mental illness [ 26]. Besides, we participated in several tasks using emotion recognition to improve mental illness identification and hopeful discourse. We are currently in the process of identifying which emotions are most correlated with misinformation and fallacies. We have found that the emotion of contempt is strictly related to certain fallacies, such as ad hominem, straw man, and appeal to emotion. To this end, we are extending the Spanish MEACorpus 2023 dataset by adding examples of the emotion contempt in order to create a consistent model for detecting misinformation and fallacies.
This work is part of the research projects LaTe4PoliticES (PID2022-138099OB-I00) funded by MICIU/AEI/10.13039/501100011033 and the European Regional Development Fund (ERDF)-a way of making Europe and LT-SWM (TED2021-131167B-I00) funded by MICIU/AEI/10.13039/ 501100011033 and by the European Union NextGenerationEU/PRTR, and ”Services based on language technologies for political microtargeting“ (22252/PDC/23) funded by the Autonomous Community of the Region of Murcia through the Regional Support Program for the Transfer and Valorization of Knowledge and Scientific Entrepreneurship of the Seneca Foundation, Science and Technology Agency of the Region of Murcia. Mr. Ronghao Pan is supported by the Programa Investigo grant, funded by the Region of Murcia, the Spanish Ministry of Labour and Social Economy and the European Union - NextGenerationEU under the “Plan de Recuperación, Transformación y Resiliencia (PRTR)”. //www.sciencedirect.com/science/article/pii/S1566253524000782. doi:https://doi.org/ 10.1016/j.inffus.2024.102300. [7] M. B. Akçay, K. Oğuz, Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers, Speech Communication 116 (2020) 56–76. URL: https://www.sciencedirect.com/science/article/pii/S0167639319302262. doi:https://doi.org/10.1016/j.specom.2019.12.001. [8] Y. Singh, S. Goel, An eficient algorithm for recognition of emotions from speaker and language independent speech using deep learning, Multimedia Tools and Applications 80 (2021). doi:10.1007/s11042-020-10399-2. [9] R. Pan, J. A. García-Díaz, M. Ángel Rodríguez-García, R. Valencia-García, Spanish meacorpus 2023: A multimodal speech–text corpus for emotion analysis in spanish from natural environments, Computer Standards & Interfaces 90 (2024) 103856. URL: https://www.sciencedirect.com/science/article/pii/S0920548924000254. doi:https://doi. org/10.1016/j.csi.2024.103856. [10] L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-A. Duquenne, B. Ellis, H. Elsahar, J. Haaheim, et al., Seamless: Multilingual expressive and streaming speech translation, arXiv preprint arXiv:2312.05187 (2023). [11] R. Pan, J. A. García-Díaz, M. Á. Rodríguez-García, F. García-Sanchez, R. Valencia-García, Overview of EmoSPeech 2024@IberLEF: Multimodal Speech-text Emotion Recognition in Spanish, Procesamiento del Lenguaje Natural 73 (2024). [12] D. Dimitrov, F. Alam, M. Hasanain, A. Hasnat, F. Silvestri, P. Nakov, G. Da San Martino, Semeval-2024 task 4: Multilingual detection of persuasion techniques in memes, in: Proceedings of the 18th International Workshop on Semantic Evaluation, SemEval 2024, Mexico City, Mexico, 2024. [13] S. Kumar, M. S. Akhtar, E. Cambria, T. Chakraborty, Semeval 2024 – task 10: Emotion discovery and reasoning its flip in conversation (ediref), in: Proceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, 2024. URL: https://arxiv.org/abs/ 2402.18944. [14] L. Plaza, J. Carrillo-de Albornoz, V. Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amigó, J. Gonzalo, R. Morante, D. Spina, Overview of exist 2024 – learning with disagreement for sexism identification and characterization in social networks and memes (extended overview, in: Working Notes of CLEF 2024- Conference and Labs of the Evaluation Forum, Springer, 2024. [15] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in neural information processing systems 35 (2022) 27730–27744. [16] R. OpenAI, Gpt-4 technical report. arxiv 2303.08774, View in Article 2 (2023). [17] Y. Yan, B. Li, J. Feng, Y. Du, Z. Lu, M. Huang, Y. Li, Research on the impact of trends related to chatgpt, Procedia Computer Science 221 (2023) 1284–1291. URL: https://www.sciencedirect.com/science/article/pii/S187705092300875X. doi:https://doi. org/10.1016/j.procs.2023.08.117, tenth International Conference on Information Technology and Quantitative Management (ITQM 2023). [18] R. Sepúlveda-Torres, A. Bonet-Jover, I. Diab, I. Guillén-Pacho, I. Cabrera-de Castro, C. Badenes-Olmedo, E. Saquete, M. T. Martín-Valdivia, P. Martínez-Barco, L. A. UreñaLópez, Overview of flares at iberlef 2024: Fine-grained language-based reliability detection in spanish news, Procesamiento del Lenguaje Natural 73 (2024). [19] E. Mancini, F. Ruggeri, P. Torroni, Multimodal fallacy classification in political debates, in: Y. Graham, M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, St. Julian’s, Malta, 2024, pp. 170–178. URL: https://aclanthology. org/2024.eacl-short.16. [20] R. Kumari, N. Ashok, P. K. Agrawal, T. Ghosal, A. Ekbal, Identifying multimodal misinformation leveraging novelty detection and emotion recognition, J. Intell. Inf. Syst. 61 (2023) 673–694. URL: https://doi.org/10.1007/s10844-023-00789-x. doi:10.1007/ s10844- 023- 00789- x. [21] Z. Jin, J. Cao, H. Guo, Y. Zhang, J. Luo, Multimodal fusion with recurrent neural networks for rumor detection on microblogs, Proceedings of the 25th ACM international conference on Multimedia (2017). URL: https://api.semanticscholar.org/CorpusID:22504814. [22] F. H. van Eemeren, Crucial concepts in argumentation theory, Amsterdam University
Press, 2001. [23] F. H. V. Eemeren, R. Grootendorst, Fallacies in pragma-dialectical perspective, Argumentation 1 (1987) 283–301. doi:10.1007/bf00136779. [24] P. Gofredo, M. Chaves, S. Villata, E. Cabrio, Argument-based detection and classification of fallacies in political debates, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 11101–11112. URL: https://aclanthology. org/2023.emnlp-main.684. doi:10.18653/v1/2023.emnlp- main.684. [25] F. L. C. y José A. Troyano y Fernando Enríquez y F. Javier Ortega, Detección y clasificación de falacias prototípicas y espontáneas en español, Procesamiento del Lenguaje Natural 71 (2023) 53–62. URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6542. [26] A. Salmerón-Ríos, J. A. García-Díaz, R. Pan, R. Valencia-García, Fine grain emotion analysis in spanish using linguistic features and transformers, PeerJ Computer Science 10 (2024) e1992. doi:10.7717/peerj- cs.1992.