In this paper, we propose o novel method that leverages a Mixture of Transformers (MoT) based on the language performance of each model. We employ simple, yet efective, preprocessing modules that are connected to the state-of-the-art Transformer and we compare the performance of general-purpose, task-specific, and data source-specific flavors of English and multi-language models. This novel approach manages to obtain good results for all tasks, with the best performance in soft-label evaluations rather than hard-label evaluations. We propose 3 types of mixtures that performed best on training data and we notice that they behave well against unseen data. The proposed architecture is easily up-gradable, has low resource costs, and provides good overall results in the EXIST 2024 competition.
• general-purpose transformers • task-specific: harmful speech detection • data-source specific: trained on Tweets
This article is structured as follows. In Section 2, we present the current state-of-the-art methods for harmful contentment detection. In Section 3, we analyze the dataset and present the experimental setup. In Section 4, we present and discuss our results. Finally, Section 5, we draw the main conclusions of this work.
The tasks proposed for this lab aim at mitigating harmful speech, more specifically sexist and ofensive
language, from social networks. With our work, we plan to improve the proposed approach in the
previous edition of our team [
The idea of leveraging multiple simple models is not new and has been previously used for this
task successfully [
Other works focus on language-independent models by training word embeddings [
Finally, the current literature also focuses on how harmful content is spread online [15, 16, 17] and how its efects can be mitigated on social platforms [18, 19, 20].
To better understand the task at hand we propose a simple Exploratory Data Analysis (EDA), as we want to use a mixture of models, based on the language of the tweets. In Table 1, we observe that the proposed split of train-test 79% − 21% has the same distribution across the languages. With the balanced distribution of tweets by language 53% − 47%, a mixture involving multi-lingual and English-only models makes sense and the comparisons of models will provide interesting results.
For our experiments, we propose a mixture of English-only and multi-lingual transformer-based models (Table 2) as we want to showcase our mixture of models architecture based on the language of the tweets (Figure 1).
The output module leverages 3 types of mixtures, in terms of output weight, for the best English and multi-language models. We consider the dominant model the English one, in case the language of the input is English, otherwise the multi-lingual one. When we present the results we will highlight the best English model like this and the best multi-lingual model like this . The leveraged mixtures are: 1. Half-Half 2. Dominant 75% 3. Dominant
Since for this competition, Task 2 and 3 are defined to take advantage of Task 1’s output, our system does the same and propagates the mixtures. This means that in Tasks 2 and Task 3, for each mixture type, the corresponding mixture from Task 1 is used.
ModelName twitter-roberta [21] [22] twitter-xlm-roberta-base-sentiment-multilingual [21] twitter-xlm-roberta-base-sentiment [23] bert-toxic-comment-classification [24] distilbert-uncased-english [25] distilbert-base-multilingual-cased-sentiments [26] MiniLM-L12-H384 [27] xlm-roberta [28] roberta-hate-speech-dynabench-r4 [29]
For all the tasks, we use early stopping with 3 epochs of tolerance and the following hyper-parameters, obtained while training the best model strategy: • _ = 2− 5 • ___ℎ_ = 32 • ___ℎ_ = 32 • ℎ_ = 0.01 • _ℎ = 50
The metrics used in the competition, for which the engine will be optimized, are ICM-Hard, ICM-Hard Norm, F1, Cross Entropy, Majority class, Minority class, and Oracle most voted. To provide models that perform well we are using F1 for Tasks 1 and 2 and for Task 3 we are using a custom Mean Squared Error.
As for the hyper-parameter tuning, each model is optimized as it would be handling the task alone, for the current implementation. 3.2.1. Task 1 The first task is a binary classification one. The system has to decide whether or not a given tweet contains sexist expressions or behaviors. The dataset is annotated by multiple evaluators, each providing their own label. To obtain only one label for each tweet, i.e., ‘YES’ or ‘NO’, we take the majority label.
In Table 3, we observe that the best-performing models for this task are the ones fine-tuned on Twitter data. As mentioned in the previous section, we are going to use a mixture of the best English-based model and the best multi-lingual model.
We notice that the models are close when it comes to performance, all are in the range of 71% − 79%, but when it comes to the resources used there is a meaningful diference. The least resources are used by MiniLM [27], which is a super-pruned version of the regular LMs. The most resource-intensive models are the ones that are trained in multiple extra iterations over the regular LMs, namely the multi-lingual tranformers. The base for the multi-lingual transformers is XLM Roberta. Each is further trained on platform-specific data, i.e., tweets from Twitter, and task-specific data, i.e., harmful speech. 3.2.2. Task 2 The second task is multi-class classification, namely “Source Intention”. Building on top of the first one, it aims to categorize the message according to the intention of the author. This provides insights into the role played by social networks in the emission and dissemination of sexist messages. To unify the results, we use the same approach as for Task 1, a majority vote with equal weight for the labels. Furthermore, we are augmenting the output by leveraging the output from Task 1, working as a ‘YES’ or ‘NO’ filter that tells us if the model needs to be run on the input or not.
For this problem, we observe that the range of results is wider (Table 4), i.e., 46% − 61%. The models perform significantly worse than they previously did, but this is expected as the output of Task 1 is also leveraged.
As expected, the specialized models are outperforming the others. Moreover, the English model, which is solely focused on the task at hand rather than the data, has the overall best performance by a small margin. Resource-wise, the behavior is not reflected on the macro level as it was for Task 1. As such, MiniLM, despite training for the most epochs among the smaller models, is not the least demanding. However, it remains the most eficient model per iteration. 3.2.3. Task 3 Task 3 is a multi-label classification focusing on identifying diferent sexism categories for each tweet that was labeled as sexist by Task 1. Unlike Task 1, tweets have multiple sexist labels. Thus, our proposed approach computes a probability for each label considering the number of annotations, using an equal weight for each annotation. As the metrics are custom, the loss is 1/ , and we did not need to represent it in Table 5. The custom Mean Square Error (CustomMSE) is adapted to the way we build the probabilities of each class.
We observe an almost perfect mirror of what happened before (Table 5), with the best performing models being the data and task-specific ones. For the resources side, we notice that this time MiniLM trained more than twice the number of epochs that the others.
Table 6 presents the oficial results from the leaderboard. For a more comprehensive analysis, please refer to the Results chapter available on the oficial site. We are showcasing only the best ranking, as most of the submissions are one after another in the rankings with a maximum drift of 3 places.
We notice that the best overall mixture is 2, − 75%, and the least is mixture 1, − . The best-performing outputs are on the English tasks for the soft evaluation rather than the hard one. One interesting aspect is that we obtained the lowest performance for Task 1, but the other two that are leveraging its output behave better, with a slight margin. Another interesting aspect is that Task 3, the one that leverages the custom metric, has the best results out of all tasks, with consistent placement.
One thing to notice is the diference between the soft and the hard evaluation, for all language splits, where for Task 1 the diference is quite significant and for the other 2 not that much, considering that in most cases the outputs of each team were one after another and each team had 3 possible outputs that can be submitted.
To conclude the Mixture of Transformers provides promising results with good resource requirements, with the second proposed mixture, − 75%, performing on average the best, with the diference in performance between them being not that significant.
We notice a similar performance as we did in the previous iteration, namely in [
One thing that we did not tackle, but we previously mentioned, is experimenting with the way we weigh each label, based on the meta-data of the annotator, but the literature has mixed views on this.
Another interesting approach is to consider a dynamic number of Transformers, for each language, based on performance, as we observe that sometimes the performance is close for multiple models.
This work is supported in part by • The German Academic Exchange Service (DAAD) through the project “iTracing: Automatic
Misinformation Fact-Checking” (DAAD grant no. 91809005). • The Academy of Romanian Scientists through the funding of project “SCAN-NEWS: Smart system for deteCting And mitigatiNg misinformation and fake news in social media” (AO S,R-TEAMS-III). aware community detection system for real-time disaster reporting, IEEE Transactions on Knowledge and Data Engineering (2024) 1–12. doi:10.1109/tkde.2024.3417232. [14] C.-O. Truică, E.-S. Apostol, P. Karras, DANES: Deep Neural Network Ensemble Architecture for Social and Textual Context-aware Fake News Detection, Knowledge-Based Systems 294 (2024) 1–13(111715). doi:10.1016/j.knosys.2024.111715. [15] A. Petrescu, C.-O. Truică, E.-S. Apostol, Sentiment Analysis of Events in Social Media, in: 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), IEEE, 2019, pp. 143–149. doi:10.1109/iccp48234.2019.8959677. [16] A. Petrescu, C.-O. Truică, E.-S. Apostol, A. Paschke, EDSA-Ensemble: an Event Detection Sentiment
Analysis Ensemble Architecture, 2023. arXiv:2301.12805. [17] C.-O. Truică, E.-S. Apostol, T. S, tefu, P. Karras, A Deep Learning Architecture for Audience Interest Prediction of News Topic on Social Media, in: International Conference on Extending Database Technology (EDBT2021), 2021, pp. 588–599. doi:10.5441/002/EDBT.2021.69. [18] A. Petrescu, C.-O. Truică, E.-S. Apostol, P. Karras, Sparse Shield: Social Network Immunization vs.
Harmful Speech, in: ACM International Conference on Information and Knowledge Management (CIKM2021), ACM, 2021, pp. 1426–1436. doi:10.1145/3459637.3482481. [19] C.-O. Truică, E.-S. Apostol, R.-C. Nicolescu, P. Karras, MCWDST: a Minimum-Cost Weighted Directed Spanning Tree Algorithm for Real-Time Fake News Mitigation in Social Media, IEEE Access 11 (2023) 125861–25873. doi:10.1109/ACCESS.2023.3331220. [20] E.-S. Apostol, Özgur Coban, C.-O. Truică, Contain: A community-based algorithm for network immunization, Engineering Science and Technology, an International Journal 55 (2024) 1–10(101728). doi:10.1016/j.jestch.2024.101728. [21] J. Camacho-Collados, K. Rezaee, T. Riahi, A. Ushio, D. Loureiro, D. Antypas, J. Boisson, L. Espinosa Anke, F. Liu, E. Martínez Cámara, Tweetnlp: Cutting-edge natural language processing for social media, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2022, pp. 38–49. doi:10.18653/v1/2022.emnlp-demos.5. [22] D. Loureiro, F. Barbieri, L. Neves, L. Espinosa Anke, J. Camacho-collados, TimeLMs: Diachronic language models from Twitter, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 251–260. doi:10.18653/v1/2022.acl-demo.25. [23] F. Barbieri, L. Espinosa Anke, J. Camacho-Collados, XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 258–266. [24] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, 2019. doi:10.18653/v1/n19-1423. [25] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. arXiv:1910.01108. [26] M. Laurer, W. van Atteveldt, A. Casas, K. Welbers, Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert-nli, Political Analysis 32 (2024) 84–100. doi:10.1017/pan.2023.20. [27] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou, Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. arXiv:2002.10957. [28] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, 2020, pp. 8440–8451. doi:10.18653/v1/2020.acl-main.747. [29] B. Vidgen, T. Thrush, Z. Waseem, D. Kiela, Learning from the worst: Dynamically generated datasets to improve online hate detection, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), ACL, 2021, pp. 1667–1682. doi:10.18653/v1/ 2021.acl-long.132.