The dissemination of inappropriate speeches, such as sexist language, on social media has a negative impact on internet users. To promote the technical development of automatic sexism detection, the EXIST lab has been engaged in this field for the past three years. This paper presents a technical report from the FraunhoferSIT team participating in the EXIST shared task for 2024. To address the issue of detecting sexism in tweets, we have experimented with ensemble learning algorithms. Additionally, we implemented a data augmentation method through synonym replacement using rule-based techniques and language models to increase the size of the training data. We participated in tasks 1 to 3. In general, the proposed system did not demonstrate a competitive performance among other systems in the challenge. However, it was observed that it exhibited a better performance in regression tasks compared to its performance in classification tasks.
The advent of social media platforms and their prevalence have provided individuals with the opportunity
to express personal opinions freely and in a timely manner, or to engage in online communication with
others. While technology has undoubtedly facilitated interaction between people, it has also contributed
to the dissemination of inappropriate content online, such as content with a sexist intention. Since the
increased prevalence of discriminatory, harassing, and other forms of sexist content on social media can
cause emotional distress to individuals concerned [
Given the variety of linguistic techniques employed in the formation of sexist sentences as stated
in [
Previous EXIST shared tasks have involved participants experimenting with diferent methods and proposing various systems for automatic sexism detection in tweets. Among the methodologies explored, ensemble methods have been widely used. Nevertheless, their application has been mostly in conjunction with language models. Based on the considerable success of machine learning algorithms in performing classification and regression tasks, there is potential for their investigation in the task of sexism detection by leveraging ensemble techniques for machine learning models. Consequently, this paper investigates the utilization of an ensemble of machine learning models with the help of stacking to enhance the accuracy and robustness of machine learning system for sexism detection. Our paper contributes to the exploration and development of techniques for the research field of the automatic sexism detection on social media.
The structure of the paper is organized as follows: firstly, we present the application of ensemble learning for automatic sexism detection in the past EXIST challenge. Then, we provide a description of the tasks performed and the dataset we used for these tasks. After that, we depict the methodology employed to address the given tasks and outline our experimental setup. Subsequently, we report on the results we achieved in the competition and discuss the limitations of the proposed system. Finally, we conclude this paper with an overview of future work.
A review of the EXIST competition over the past three years found that the potential of various learning
methods suitable for NLP tasks has been widely discussed by participants. The techniques for the
detection of sexism that have been explored include contrastive learning, transfer learning, ensemble
learning and others. While both Transformer models [
The Transformer-based language models have been specifically designed to address NLP tasks
and achieve state-of-the-art performance in numerous NLP benchmarks. In previous challenges, both
monolingual [
Despite the significant advances in NLP tasks achieved by language models, the performance of
individual models in classification tasks remains limited. Consequently, ensemble methods are frequently
employed to enhance the capabilities of individual models. For instance, [
Although ensemble approaches have been demonstrated to be efective in performing the given shared task in comparison to the performance of baseline models, [18] proposed a novel approach that combines transfer learning and ensemble learning. This approach was employed to enhance the feature learning efect of pre-trained and fine-tuned Transformer models. Furthermore, [ 19] proposed a biensemble method, which merges two ensemble approaches: an ensemble consisting of two architectures and an ensemble composed of several trained models of the same architecture.
In addition to the study of ensembles of language models, ensembles of machine learning models have also been examined. For instance, the approach taken by [20] involved the use of an ensemble of machine learning models, whereby the probabilities predicted by these models were aggregated. Nevertheless, despite their potential, the ensembles of machine learning models have not been subjected to suficient investigation within the context of the EXIST challenge. Hence, it is necessary to conduct further research into their capabilities.
The EXIST 2024 shared task comprises two parts: the classification of tweets (Tasks 1 to 3) and the
classification of memes (Tasks 4 to 6) [
The process of identifying instances of sexism in texts is inherently subjective. To enhance the learning
efect of the system and avoid author bias, EXIST 2024 employs a Learning With Disagreement (LeWiDi)
paradigm for the development of the dataset and the evaluation of the systems [
Since we were only engaged in the tweet classification tasks, we utilized only the tweets dataset provided by the EXIST shared task. As shown in Table 1, the tweets dataset is divided into three distinct sets: the train set, the development set, and the test set. Both the train set and the development set comprise labeled tweets in English and Spanish, with a total of 6920 and 1038 tweets, respectively. The test set, on the other hand, includes 2076 unlabeled English and Spanish tweets.
As previously stated (see Section 3), our team participated in the first three tasks related to tweet classification. These tasks can be identified as three types of classification problems: binary classification (Task 1), multi-class classification (Task 2), and multi-label classification (Task 3). In addition to providing hard labels for tweet classification, we also submitted soft labels for each task, which are considered to target the issue of regression. In this section, we will present the methodology employed in the given shared task to address classification and regression problems.
Tweets often contain hashtags, mentions, and URLs. The removal of mentions and URLs from tweets has been demonstrated to have a minimal efect on the interpretation of the original tweets. Hashtags typically represent the topics of tweets. Since the topics of tweets have been predefined for the purpose of sexism detection, the hashtags are considered to have limited importance. Therefore, in the preprocessing stage, hashtags, mentions, and URLs were removed from the tweets properly.
Emojis are frequently utilized in tweets as a means of expressing emotions. They are typically employed to convey the attitude of the author, to extend and strengthen the emotion expressed, or to reverse the textual meaning in instances of sarcasm or irony. Given the potential for emojis to influence the interpretation of tweets, it is recommended that they are treated as a constituent of tweet texts. Consequently, all emojis were retained and converted into text format using the emoji python library [21]. Emojis from Spanish tweets were converted into Spanish text, while emojis from English tweets were converted into English text.
Data augmentation encompasses a range of techniques designed to increase the quantity and diversity of the training data associated with a given dataset, thereby enhancing the accuracy and robustness of machine learning systems [22]. It is used to address the issue of overfitting on the training set, which can arise due to limited labeled data [23]. Furthermore, it serves to mitigate bias and fix class imbalance [24]. Data augmentation, which originated from research fields of computer vision [ 25], has attracted considerable attention in the NLP community in recent years. The data augmentation methods specialized for NLP tasks have been actively investigated to address the issues of limited data for specific NLP research areas.
Despite the advantages that data augmentation techniques bring to text classification settings, data augmentation remains a challenging task due to the dificulty of defining textual transformations that preserve the labels. To ensure the compatibility of the labels on the original texts with those on the augmented texts, we employed two augmentation methods: synonym replacement [26] and contextual augmentation [27]. Both of these methods were applied at the word level. These approaches have been demonstrated to maintain the labels of the original texts, as the semantic meanings of the augmented versions remain unaltered.
WordNet [28, 29] is an English lexical database that groups words into sets of synonyms. The implementation of the synonym replacement method using WordNet for English tweets has been demonstrated to provide high-quality data. Due to the limitations of the available Spanish synonym lexical database, we selected three diferent Transformer language models from HuggingFace [ 30] for the implementation of the contextual augmentation method. The models employed include BERTIN [31], ALBERT Base Spanish [32], and RoBERTuito [33]. In total, the original train and development datasets were subjected to a tenfold augmentation using the aforementioned methods.
Ensemble methods are learning algorithms that are constructed based on the combinations of a set of learning algorithms with the objective of enhancing the overall prediction performance of multiple single learning algorithms. The rationale behind the development of these ensembles is that the error rate induced by the ensemble of diferent learning algorithms has the potential to be compensated in comparison to the prediction performance of single learning algorithms [34, 35]. In addition, the accuracy and diversity of the individual learning algorithms employed in the construction of the ensemble still play a critical role in enhancing the robustness of the ensemble [36].
A variety of techniques has been developed for constructing ensembles, including bagging (bootstrap aggregating) [37], boosting [38, 39], AdaBoost (adaptive boosting) [40], voting [41], and stacking [42]. As stated in [35], ensemble methods are considered the state-of-the-art solution for many machine learning challenges. They have also been widely used in previous years for the EXIST shared task, as outlined in Section 2. However, the ensemble learning methods reported in the EXIST working notes are predominantly in conjunction with Transformer models. The ensemble of machine learning models remain under-researched for the detection of sexism in tweets. Thus, for the EXIST 2024 shared task, we utilize ensemble of machine learning models.
In the presented tasks, both soft labels and hard labels can be provided for tweets. The task of providing soft labels for tweets can be considered a problem of regression, while the task of providing hard labels for tweets is referred to as a classification problem. As the ensemble method of stacking machine learning models can be applied to both classification and regression problems, we employed the stacking ensemble to address the tasks outlined in Section 3. In the context of resolving these two problems, we diferentiate in our approach to model selection for the stacking ensemble. All models employed were implemented from the Scikit-learn library [43].
The base models selected for the given classification tasks are commonly employed in supervised learning and can be applied to both classification and regression problems. Nevertheless, it has been demonstrated that MNB, DT, kNN, and LR are more suitable for classification tasks than for regression tasks. In contrast, SGD presents an optimisation technique that can be used to optimise models for both classification and regression tasks. Additionally, all of these models demonstrate scalability to large datasets and ofers various methods to manage overfitting.
The ET, also known as Extremely Randomised Trees, is a tree-based ensemble method for supervised classification and regression problems. Its distinctive nature involves randomizing both attribute and cut-point choices while splitting a tree node and building completely randomized trees whose structures are independent of the output values of the learning sample [44]. This characteristic suggests that it has the potential to be used as a meta-estimator for stacking ensemble learning.
In the case of the regression tasks, we applied Random Forest (RF), Gradient Boosting (GB), and Stochastic Gradient Descent (SGD) as base models, and AdaBoost as the meta-model. In essence, despite the diferences in their methodologies and types of base learners, these algorithms share the common objective of enhancing prediction accuracy. This can be attained by employing either a tree-based ensemble learning approach, comprising RF, GB, and AdaBoost, or an optimisation technique, such as SGD. Furthermore, each of these algorithms has been developed to handle issues such as overfitting, scalability, and the inherent bias-variance trade-of in the learning process.
In this section, we present the oficial evaluation results of our submissions to the competition (see
Table 2, 3, and 4). The evaluations were performed in two modes: Hard-Hard evaluation and Soft-Soft
evaluation, using the oficial metric Information Contrast Measure (ICM) and selectively the 1 score [
In the first task of identifying instances of sexism in tweets, we achieved a general ICM score of 0.4895 for the Soft-Soft evaluation and 0.6166 for the Hard-Hard evaluation. In the second task of categorising the source intention in tweets, we achieved a general ICM score of 0.1708 for the Soft-Soft evaluation and 0.3665 for the Hard-Hard evaluation. In the third task of sexism categorisation, the general ICM score was 0.2259 for the Soft-Soft evaluation and 0.3273 for the Hard-Hard evaluation. The results achieved in the competition indicate that both the classification and regression systems proposed performed the best in Task 3 in comparison to their performance in Tasks 1 and 2. This was particularly evident in the Soft-Soft evaluation.
Although the stacked ensemble of machine learning models employed were tailored to address the classification and regression tasks for the EXIST challenge, they did not demonstrate a competitive performance among other systems in the challenge. Given the absence of a universal methodology for the selection of the best model combinations for stacking ensemble, our approach involved the identification of models based on their individual performance on the tasks of sexism detection. Furthermore, the time-consuming nature of the stacking ensemble method precluded the development of the optimal combination of machine learning models to target the characteristic of each task.
In this paper, we presented our participation in the EXIST 2024 shared task [
The oficial evaluation results indicate that the system proposed in this report did not perform competitively as other systems within the challenge. Nevertheless, the system exhibited comparatively better performance in regression tasks than in classification tasks.
For future work, it would be beneficial to experiment with various ensemble combinations and to gain a more in-depth insight into model choice in order to identify the most suitable ensemble of models for sexist context. In addition to machine learning models, Transformer models can also be stacked to outperform their single performance on both classification and regression tasks. 2022. URL: https://ceur-ws.org/Vol-3202/exist-paper2.pdf. [16] J. A. García-Díaz, S. M. Jiménez-Zafra, R. Colomo-Palacios, R. Valencia-García, Umuteam at exist 2022: Knowledge integration and ensemble learning for multilingual sexism identification and categorization using linguistic features and transformers, in: IberLEF 2022, CEUR Workshop Proceedings, A Coruña, Spain, 2022. URL: https://ceur-ws.org/Vol-3202/exist-paper14.pdf. [17] A. Younus, M. A. Qureshi, A framework for sexism detection on social media via byt5 and tabnet, in: IberLEF 2022, CEUR Workshop Proceedings, A Coruña, Spain, 2022. URL: https: //ceur-ws.org/Vol-3202/exist-paper15.pdf. [18] J. Böck, M. Schütz, D. Liakhovets, Q. Satriani, A. Babic, D. Slijepčević, M. Zeppelzauer, A. Schindler, Ait_fhstp at exist 2023 benchmark: Sexism detection by transfer learning, sentiment and toxicity embeddings and hand-crafted features, in: CLEF 2023: Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Thessaloniki, Greece, 2023. URL: https://ceur-ws.org/ Vol-3497/paper-074.pdf. [19] E. Villa-Cueva, F. Sanches-Vega, A. P. López-Monroy, Bi-ensemble of transformer for online bilingual sexism detection, in: IberLEF 2022, CEUR Workshop Proceedings, A Coruña, Spain, 2022.
URL: https://ceur-ws.org/Vol-3202/exist-paper4.pdf. [20] S. Ravi, S. Kelkar, A. K. Madasamy, Lstm-attention architecture for online bilingual sexism detection, in: CLEF 2023: Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Thessaloniki, Greece, 2023. URL: https://ceur-ws.org/Vol-3497/paper-089.pdf. [21] T. Kim, K. Wurster, Emoji terminal output for python, 2024. URL: https://github.com/carpedm20/ emoji/. [22] E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, Q. V. Le, Autoaugment: Learning augmentation strategies from data, in: IEEE Conference on Computer Vision and Pattern Recognistion, 2019, pp. 113–123. URL: https://doi.org/10.1109/CVPR.2019.00020. [23] J. Chen, D. Tam, C. Rafel, M. Bansal, D. Yang, An empirical survey of data augmentation for limited data learning in nlp, in: Transactions of the Association for Computational Linguistics, volume 11, 2023, pp. 191–211. URL: https://doi.org/10.1162/tacl_a_00542. [24] S. Y. Feng, V. Gangal, J. Wei, S. Chandar, S. Vosoughi, T. Mitamura, E. Hovy, A survey of data augmentation approaches for nlp, 2021. URL: https://doi.org/10.48550/arXiv.2105.03075. [25] M. Bayer, M.-A. Kaufhold, C. Reuter, A survey on data augmentation for text classification, in:
ACM Computing Surveys, volume 55, 2022, pp. 1–39. URL: https://doi.org/10.1145/3544558. [26] J. Wei, K. Zou, Eda: Easy data augmentation techniques for boosting performance on text classification tasks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langauge Processing, 2019, pp. 6382–6388. URL: https://aclanthology.org/D19-1670.pdf. [27] S. Kobayashi, Contextual augmentation: Data augmentation by words with paradigmatic relations, in: Proceedings of NAACL-HLT, Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 452–457. [28] G. A. Miller, Wordnet: A lexical database for english, Communications of the ACM 38 (1995) 39–41. [29] C. Fellbaum, WordNet: An Electronic Lexical Database, MA: MIT Press, Cambridge, 1998. [30] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771. [31] J. D. la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury, Bertin: Eficient pre-training of a spanish language model using perplexity sampling, Procesamiento del Lenguaje Natural 68 (2022) 13–23. URL: http://journal.sepln.org/ sepln/ojs/ojs/index.php/pln/article/view/6403. [32] J. Cañete, Albert base spanish, HuggingFace, 2022. URL: https://huggingface.co/dccuchile/ albert-base-spanish. [33] J. M. Pérez, D. A. Furman, L. Alonso Alemany, F. M. Luque, RoBERTuito: a pre-trained language