This paper presents a comprehensive analysis of ensemble models for the shared task "Conspiracy Theories Against Critical Thinking Narratives" for PAN at CLEF 2024. Through a data collection involving Telegram conversations on COVID-19, two distinct corpora in English and Spanish were assembled and manually labeled to diferentiate between "critical" and "conspiracy" texts. The study employed ensemble models, comprising seven trained transformer-based models per language-task pair, to address two key tasks: distinguishing between critical and conspiracy texts (binary classification) and detecting spans for six diferent categories that can be found on the texts (multi-label span classification). The results unveiled the competitive performance of ensemble models, particularly in securing notable rankings surpassing the mean of all participants' results in both tasks.
Conspiracy theories (CT) are narratives that seek to explain the causes of significant situations or
events for society, suggesting the existence of secret plans secretly carried out by actors who abuse their
power to achieve their own objectives without caring about depriving people of their rights, freedoms,
prosperity, health or knowledge [
In addition to the behavioral change in believers of these theories, another significant harm is the mistrust they generate towards various medical treatments and the decrease in trust in public health institutions and health professionals. This hinders the implementation of public health measures and the response to health emergencies. For these reasons, it is urgent to identify and address conspiracy theories to mitigate their harmful efects.
Although conspiracy theories are not limited to the field of health, they have been a persistent issue over the years, causing significant harm to the population. A clear example is the case of the smallpox vaccine, discovered by Edward Jenner in 1796, which represented a monumental advance with the potential to improve public health significantly. However, it also led to the creation of a CT [ 4]. It is likely that people did not properly understand how it worked, which led to the spread of rumors warning of horn growth resulting from its use.
And this is not the only case of conspiracy theories related to vaccines. In fact, they have been a recurring theme. For instance, in 1981, Dr. John Wilson claimed that the DPT vaccine caused convulsions and brain damage [5]. In 1998, Andrew Wakefield published an article suggesting a link between the MMR vaccine and autism [6], although it should be noted that this article was retracted by the journal in which it was published. More recently, the COVID-19 pandemic has fueled the spread of numerous conspiracy theories regarding vaccination against this virus [7].
In general, the propagation of conspiracy theories could have several negative efects, among which we can highlight some of them: • Social Division and Polarization: They exacerbate social divisions by promoting extreme and exclusionary beliefs, hindering rational dialogue and societal cohesion. • Dissemination of Misinformation: They contribute to spreading false and unverified information, leading to confusion and potentially harmful decisions. • Loss of trust in authorities and experts: They foster distrust towards governmental, scientific, and public health institutions, as well as towards experts in diferent areas. • Psychological Impact: They induce anxiety, fear, and paranoia among believers, negatively afecting their emotional and mental well-being. • Impaired Decision-Making: Believers may base decisions on misinformation or biased information, impeding informed and rational decision-making processes.
In the health field, conspiracy theories have had significant adverse efects. In Pakistan, for example, there is a belief that the polio vaccine was developed by the CIA to sterilize Muslim men [8], which has led many people to reject it. Another example is the theory that the U.S. government created HIV/AIDS to reduce the African-American population, a widespread belief among this community that has resulted in less frequent condom use [9].
Furthermore, certain sectors of society maintain mistrust towards specific drugs, alleging they inflict greater harm than the diseases they aim to cure. For instance, there exists a theory attributing the majority of deaths among AIDS patients to retroviral drugs. This conspiracy theory holds particular influence in sub-Saharan Africa, where it receives support from influential figures [10].
There are several reasons why conspiracy theories can be widely spread. Among the most prominent ones is their propagation by celebrities through digital media [11], which causes many of their followers to start believing in them. In addition, it is dificult to absolutely determine their falsity, together with the degree of plausibility attributed to them by each person [12], significantly contributes to their dissemination. Critical thinking can help people to better evaluate the information they receive in daily life and thus avoid fraud and harmful habits. For example, critical thinking can be useful in diferentiating reliable medical information from unfounded claims, helping in decision-making about appropriate treatments and lifestyle. When a person with high levels of intelligence, but low levels of critical thinking, believes in a conspiracy theory, they can generate very well-supported arguments to support the false information [13]. These arguments can be quickly propagated through digital media and are dificult to detect.
This year’s goal at PAN 2024 is to analyze texts reflecting oppositional thinking, specifically distinguishing between conspiracy theories and critical thinking narratives. This task addresses two significant challenges for the NLP community: (subtask 1, a binary classification task) diferentiating between conspiracy and critical narratives, and (subtask 2, a multi-label span classification task) identifying key elements of narratives that fuel intergroup conflict. Making this distinction is crucial because mislabeling a text as conspiratorial when it is merely oppositional to mainstream views could push individuals who are simply questioning mainstream perspectives closer to conspiracy communities [14, 15].
Conspiracy theories represent a significant danger, as they can negatively influence people’s behavior, afecting trust in institutions and fostering disinformation. Intelligence is often thought of as synonymous with critical thinking, however, these terms are not the same. In reality, intelligence alone does not always translate into critical thinking. Over the course of history, there have been people with high levels of intelligence who nevertheless have demonstrated a lack of critical thinking in some areas, for instance, Sir. Arthur Conan Doyle a brilliant writer who believed in spiritualism and fairies, despite clear evidence to the contrary [16].
A recent study [13] explores the connections between critical thinking, intelligence and the predisposition to believe in conspiracy theories. The authors note that while intelligence can help people formulate more sophisticated arguments, it does not always protect them from false beliefs. On the other hand, critical thinkers use logical rules, standards of evidence and other criteria that must be met for the product of a thought to be considered good, making them less likely to believe in unsubstantiated claims.
Intelligence is generally associated with good cognitive processing or intellectual abilities and the potential to learn and reason well. Intelligent people tend to perform well in basic real-world domains, such as academic performance and job success but sometimes find it dificult to adapt in other realworld situations [17]. Intelligence without critical thinking can sometimes result in more convincing arguments that support false beliefs. These persuasive arguments can mislead many people into accepting these false ideas.
In a companion study [18], the impact of cognitive styles, such as analytical thinking, critical thinking, and scientific reasoning, on the propensity to believe in conspiracy theories was examined. The findings suggest that individuals who exhibit a stronger inclination towards analytical thinking and scientific reasoning are less susceptible to conspiracy theories due to their more rigorous and evidence-based approach to evaluating information.
As a matter of fact, in recent years, there has been a notable surge in the recognition and analysis of conspiracy theories. This trend mirrors the growing acknowledgment of the significant impact that misinformation and disinformation can have on societies, particularly in the age of digital interconnectedness. Research endeavors[19], have increasingly focused on understanding the dynamics behind the propagation of conspiracy theories.
However, it’s crucial to recognize that the identification and mitigation of conspiracy theories are part of a broader spectrum of tasks aimed at combating misinformation and preserving the integrity of information ecosystems. Alongside the detection of conspiracy theories, researchers and practitioners are also confronted with related challenges, including the identification and containment of rumors[ 20], the mitigation of the spread of fake news[21], the recognition of clickbait content [22] designed to manipulate user engagement, and the indispensable task of fact-checking[23].
In this contemporary landscape, where information dissemination is facilitated by sophisticated technologies and platforms, the importance of discerning false information from genuine content cannot be overstated. The rise of AI-driven text generation capabilities, for instance, presents both opportunities and challenges. On one hand, these advancements ofer innovative approaches to understanding and combating misinformation. On the other hand, they underscore the urgency of developing robust mechanisms to diferentiate between authentic and fabricated texts.
The data collection involved gathering textual data from Telegram conversations concerning COVID-19. These texts were then manually labeled to distinguish between "critical" and "conspiracy" categories. Two corpora were employed for this study, one in English and the other in Spanish. Each set contained 4000 entries for training purposes and an additional 1000 entries for testing [24]. In this work, the use of k-fold cross validation was not implemented due to limited computational resources. Instead, the 10% of the training set was split for validation experiments, preserving the initial class balance provided, as shown in Table 1. The main hypothesis was that a single train-validation split could lead to a scenario where the model stability were more consistent than averaging results over multiple folds specially for subtask 2.
The dataset entries had two diferent representations: the original sentence and the sentence split by tokens designed for subtask 2. The approach implemented was to use the original sentence representation for subtask 1, leveraging the tokenization step to each model’s tokenizer and to use the list of tokens for subtask 2, trying to preserve the majority of the tokens labeled after the preprocess and cleaning step. The following procedures for text cleaning were implemented for both tasks: • Small combinations of numbers and letters (with lengths ranging from 2 to 4) were removed. • Combinations of alternating letters and numbers were removed (e.g. tokens such as 1df324D identified in URLs). • Special words for URLs were removed. • English contractions such as ’re or n’t were normalized by using the complete word (are and not). • Numbers in date format and hour were tagged using the labels date, hour for English and fecha, hora for Spanish. • The rest of the numbers were tagged using the label number.
• Repeated strings of three or more characters were normalized (e.g. aaa to a).
Significantly, both corpora manifest an inherent class distribution imbalance, characterized by a larger proportion of inputs labeled as "critical" in contrast to those categorized as "conspiracy", which is illustrated in Figure 1.
The baseline model provided was a transformer-based model designed for multitask learning to address both tasks. While this generally leads to better results, it can also make the model more complex and dificult to train, particularly in balancing the loss of both tasks to prevent one task from negatively impacting the performance of the other.
The proposed solution in this work involved using an ensemble of transformer-based models in the form of several Small Language Models (SLMs) to address each task-language pair independently, thus training them as single-task learning models using low computational resources. This methodology facilitated the aggregation of multiple logits and aimed to improve overall performance. The training process consisted of developing seven distinct SLMs for each language and task. Subsequently, the top five models for each language-task pair were selected based on specific evaluation metrics. For subtask 1, the oficial evaluation metrics were the Matthews correlation coeficient (MCC) [ 25] and macro F1-score, while subtask 2 was evaluated using the span-F1 metric [26].
Figure 2 illustrates the ensemble strategy utilized in this work for subtask 1. All logits obtained by each SLM were multiplied by a weight based on the scores of the evaluation metrics. Subsequently, the logits were aggregated and rounded, to get the final outcome of the ensemble model. The same strategy was applied for subtask 2, where instead of getting a single outcome per SLM, a matrix ∈ R× was obtained and aggregated aftwewards, as depicted in Figure 3. In summary, two ensemble models were evaluated per task-language pair, one using all seven trained SLMs and another using the top 5 best trained SLMs. Each ensemble model employed a mean voting classifier.
This work’s rigorous selection process led to the identification of several transformer-based models for both subtasks within the English corpus. The transformer library by huggingface provides wrappers for sequence classification and token classification tasks. The following enumeration provides a concise description of the models assessed.
• BERT [27]: Demonstrates a significant performance in understanding context and semantics, making it a natural choice. The baseline provided was constructed utilizing it. • RoBERTa [28]: Employs an optimized pretraining and can achieve better results than BERT. • BigBird [29]: Handles long sequences through sparse attention mechanisms. The English corpus comprises several long sequences of tokens that exceed the typical maximum length (512) accepted by SLMs. • Electra [30]: Utilizes a generator-discriminator architecture for enhanced eficiency, ofering robustness against adversarial attacks and enhancing generalization capabilities. • T5 [31]: Adopts a text-to-text framework that can handle diverse tasks. Although it is a textgenerating model, it can be used as a binary classification by adding a classification module (a linear layer on top of the pooled output). For classification tasks, the output of the first token is processed and classified. Huggingface has an implementation of this model’s variant. • XLM-RoBERTa [32]: Extends RoBERTa to multiple languages, producing distinct representations of the inputs, potentially ofering a complementary perspective on the tasks. • MDeBERTa [33]: Designs eficient multilingual representations like XLM-RoBERTa, thereby providing another perspective of the tasks.
Table 2 displays the metric outcomes for each SLM to subtask 1 on the English corpus. Notably, MDeBERTa and T5 models achieved the most favorable results, outperforming the rest. Conversely, Table 3 showcases the results for the macro span-f1 metric associated with subtask 2 on the English corpus. Here, the multilingual model MDeBERTa and Electra emerged as the best models, while T5 exhibited comparatively insignificant results.
In alignment with the specific demands of the Spanish corpus, a tailored selection of seven models was employed, all implemented using the Hugging Face Transformers library. The following list provides a description of these models: • BETO [34]: Encompasses proficient linguistic understanding and contextual comprehension of the Spanish language, and it served as the baseline model for the subtasks. • Bertin [35]: Contributes to the linguistic analysis of Spanish language, providing an alternative model for addressing linguistic nuances. • MarIA [36]: Demonstrates proficiency and eficacy in addressing the complexities of the Spanish language, being trained by large amounts of Spanish texts. • TwHIN-BERT [37]: Enhances capabilities in processing linguistic structures, being tailored for hate speech detection in Spanish, particularly on social media. • mT5 [38]: Ofers a multilingual variant of the T5 model, and enriches the analytical repertoire available for the Spanish language. It is also a generative text model. • XLM-RoBERTa [32]: Proposes another variant of the inputs, ofering an additional multilingual perspective on the tasks. • MDeBERTa [33]: As a third multilingual representation, it ofers valuable insights, augmenting the analytical approach of the solution approach.
Table 4 presents the metric outcomes for each trained model concerning subtask 1 for Spanish language. Remarkably, MarIA and MDeBERTa demonstrated the most promising results, surpassing its counterparts. On the other hand, Table 5 delineates the results for subtask 2 on the Spanish corpus. For this language, the multilingual model MDeBERTa emerged as the leading performer, while mT5 displayed relatively negligible results, mirroring the outcomes obtained by its counterpart in the English experiments.
The shared task allowed a maximum of two submissions per subtask. For our submissions, we opted to present two ensemble models per subtask: an ensemble version comprising all seven models trained per language-task pair, alongside another submission featuring the top five models. Table 6 provides a comprehensive overview of the oficial results attained per submission for subtask 1, incorporating the attained placement, while Table 7 delineates the results for subtask 2. The best models were determined on their competitiveness across the Matthews Correlation Coeficient (MCC) metric and span-F1 metric, for both subtasks respectively. Due to complications encountered during the experimentation phase, the evaluation of the ensemble model comprising the top 5 models for Spanish was precluded. For subtask 1, the optimal ensemble model surpassed the baseline performance for the English language. However, the submitted ensemble model for the Spanish language did not exhibit a similar performance. Conversely, for subtask 2, the optimal ensemble model successfully outperformed the baselines for both languages. In this subtask, the ensemble model with five learners was the best approach for the English language, while the ensemble model with seven learners was the best for the Spanish language. The results obtained for the Spanish language were significantly higher than its baseline, which implies the learners successfully contributed diferent information to the final solutions.
The ensemble model’s combination of diverse SLM architectures contributed to robustness and generalization, thereby enhancing performance across both tasks. However, certain limitations and areas for improvement were identified. A small fixed validation set was used, but a cross-validation strategy might lead to better performance, specially for obtaining more accurate weights for the base models. The ensemble used a weighted mean voting classifier that can be replaced for a more sophisticated meta model like a logistic regression classifier. The single-task learning approach did not outperform all the baseline results obtained using a multitask learning approach. The shared knowledge from both subtasks might enhance the results and the generalization of the final predictions.
The disparities in performance between tasks could be attributed to the inherent complexity and ambiguity associated with detecting diferent classes among texts, necessitating further exploration and refinement of the approach’s methodologies and feature representations. By leveraging insights gleaned from the model performance analysis, future iterations of the ensemble model can be refined to enhance robustness and eficacy within the domain of conspiracy theories and critical thinking narratives.
This work was done with partial support from the Mexican Government through Consejo Nacional de Humanidades Ciencias y Tecnologías (CONAHCYT) and Instituto Politécnico Nacional (IPN). [4] M. V. Eve Dubé, N. E. MacDonald, Vaccine hesitancy, vaccine refusal and the anti-vaccine movement: influence, impact and implications, Expert Review of Vaccines 14 (2015) 99– 117. URL: https://doi.org/10.1586/14760584.2015.964212. doi:10.1586/14760584.2015.964212. arXiv:https://doi.org/10.1586/14760584.2015.964212, pMID: 25373435. [5] J. T. Wilson, Dpt vaccine and serious neurological illness: current status of the controversy,
Pediatrics 68 (1981) 650–651. [6] A. Wakefield, S. Murch, A. Anthony, J. Linnell, D. Casson, M. Malik, M. Berelowitz, A. Dhillon, M. Thomson, P. Harvey, A. Valentine, S. Davies, J. Walker-Smith, Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children, The Lancet 351 (1998) 637–641. [7] N. Corbu, R. Buturoiu, V. Frunzaru, G. Guiu, Vaccine-related conspiracy and counter-conspiracy narratives. silencing efects, Communications 49 (2024) 339–360. URL: https://doi.org/10.1515/ commun-2022-0022. doi:doi:10.1515/commun-2022-0022. [8] G. E. Andrade, A. Hussain, Polio in pakistan: Political, sociological, and epidemiological factors,
Cureus 10 (2018) e3502. doi:10.7759/cureus.3502. [9] L. M. Bogart, S. T. Bird, Exploring the relationship of conspiracy beliefs about hiv/aids to sexual behaviors and attitudes among african-american adults, Journal of the National Medical Association 95 (2003) 1057. [10] P. Fourie, M. Meyer, The Politics of AIDS Denialism, Routledge, New York, 2010. [11] G. Andrade, Medical conspiracy theories: cognitive science and implications for ethics, Medicine, Health Care and Philosophy 23 (2020) 505–518. URL: https://doi.org/10.1007/s11019-020-09951-6. doi:10.1007/s11019-020-09951-6. [12] M. Frenken, A. Reusch, R. Imhof, “just because it’s a conspiracy theory doesn’t mean they’re not out to get you”: Diferentiating the correlates of judgments of plausible versus implausible conspiracy theories, Social Psychological and Personality Science (2024) 19485506241240506. URL: https://doi.org/10.1177/19485506241240506. doi:10.1177/19485506241240506. [13] D. A. Bensley, Critical thinking, intelligence, and unsubstantiated beliefs: An integrative review, Journal of Intelligence 11 (2023). URL: https://www.mdpi.com/2079-3200/11/11/207. doi:10.3390/ jintelligence11110207. [14] A. A. Ayele, N. Babakov, J. Bevendorf, X. Bonet Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fröbe, D. Korenčić, M. Mayerl, D. Moskovskiy, A. Mukherjee, A. Panchenko, M. Potthast, F. Rangel, N. Rizwan, P. Rosso, F. Schneider, A. Smirnova, E. Stamatatos, B. Stein, M. Taulé, D. Ustalov, X. Wang, M. Wiegmann, S. M. Yimam, E. Zangerle, Overview of pan 2024: Multiauthor writing style analysis, multilingual text detoxification, oppositional thinking analysis, and generative ai authorship verification - condensed lab overview, in: Proceedings of the Fifteenth International Conference of the CLEF Association CLEF-2024, Springer, 2024, pp. 3–10. [15] D. Korenčić, B. Chulvi, X. Bonet Casals, M. Taulé, P. Rosso, F. Rangel, Overview of the oppositional thinking analysis pan task at clef 2024, in: G. Faggioli, N. Ferro, P. Galuvakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [16] T. Waters, Magic and the british middle classes, 1750–1900, Journal of British Studies 54 (2015) 632–653. URL: http://www.jstor.org/stable/24702123. [17] D. F. Halpern, D. S. Dunn, Critical thinking: A model of intelligence for solving real-world problems, Journal of Intelligence 9 (2021). URL: https://www.mdpi.com/2079-3200/9/2/22. doi:10. 3390/jintelligence9020022. [18] B. Gjoneska, Conspiratorial beliefs and cognitive styles: An integrated look on analytic thinking, critical thinking, and scientific reasoning in relation to (dis)trust in conspiracy theories, Frontiers in Psychology 12 (2021). URL: https://www.frontiersin.org/journals/psychology/articles/10.3389/ fpsyg.2021.736838. doi:10.3389/fpsyg.2021.736838. [19] A. Giachanou, B. Ghanem, P. Rosso, Detection of conspiracy propagators using psycho-linguistic characteristics, Journal of Information Science 49 (2023) 3–17. URL: https://doi.org/10.1177/0165551520985486. doi:10.1177/0165551520985486. arXiv:https://doi.org/10.1177/0165551520985486. [20] G. Gorrell, E. Kochkina, M. Liakata, A. Aker, A. Zubiaga, K. Bontcheva, L. Derczynski, Semeval-2019 task 7: Rumoureval 2019: Determining rumour veracity and support for rumours, in: Proceedings of the 13th International Workshop on Semantic Evaluation: NAACL HLT 2019, Association for Computational Linguistics, 2019, pp. 845–854. [21] N. Capuano, G. Fenza, V. Loia, F. D. Nota, Content-based fake news detection with machine and deep learning: A systematic review, Neurocomputing 530 (2023) 91–103. [22] A. Anand, T. Chakraborty, N. Park, We used neural networks to detect clickbaits: You won’t believe what happened next!, in: Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017, Proceedings 39, Springer, 2017, pp. 541–547. [23] N. Walter, J. Cohen, R. L. Holbert, Y. Morag, Fact-checking: A meta-analysis of what works and for whom, Political communication 37 (2020) 350–375. [24] D. Korenčić, B. Chulvi, X. B. Casals, M. Taulé, P. Rosso, Pan24 oppositional thinking analysis [data set] (2024). URL: https://doi.org/10.5281/zenodo.11199642. doi:10.5281/zenodo.11199642. [25] D. Chicco, N. Tötsch, G. Jurman, The matthews correlation coeficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation, BioData mining 14 (2021) 1–22. [26] G. Da San Martino, S. Yu, A. Barrón-Cedeño, R. Petrov, P. Nakov, et al., Fine-grained analysis of propaganda in news articles, in: Proceedings of EMNLP-IJCNLP 2019-2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, 2019, pp. 5636–5646. [27] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [28] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,
Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [29] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al., Big bird: Transformers for longer sequences, Advances in neural information processing systems 33 (2020) 17283–17297. [30] K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning, Electra: Pre-training text encoders as discriminators rather than generators, arXiv preprint arXiv:2003.10555 (2020). [31] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of machine learning research 21 (2020) 1–67. [32] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019). [33] P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, arXiv preprint arXiv:2111.09543 (2021). [34] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained bert model and evaluation data, arXiv preprint arXiv:2308.02976 (2023). [35] J. D. la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury, Bertin: Eficient pre-training of a spanish language model using perplexity sampling, Procesamiento del Lenguaje Natural 68 (2022) 13–23. URL: http://journal.sepln.org/ sepln/ojs/ojs/index.php/pln/article/view/6403. [36] A. G. Fandiño, J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, M. Villegas, Maria: Spanish language models, Procesamiento del Lenguaje Natural 68 (2022). URL: https://upcommons.upc.edu/handle/2117/367156#.YyMTB4X9A-0. mendeley. doi:10.26342/2022-68-3. [37] X. Zhang, Y. Malkov, O. Florez, S. Park, B. McWilliams, J. Han, A. El-Kishky, Twhin-bert: A sociallyenriched pre-trained language model for multilingual tweet representations, arXiv preprint arXiv:2209.07562 (2022). [38] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mt5: A massively multilingual pre-trained text-to-text transformer, arXiv preprint arXiv:2010.11934 (2020).