This paper presents the participation of the UMUTeam in the MeO endEs shared task at IberLEF 2021. This task involves the identi cation and categorisation of o ensiveness in Spanish comments from di erent social networks (YouTube, Instagram and Twitter), and Mexican Spanish tweets. Speci cally, four subtasks were proposed: the rst one on multi-class classi cation of o ensiveness types, the second one also concerning multi-class classi cation but with contextual information, the third one on a binary classi cation of texts as o ensives or non-o ensives, and the last one also regarding a binary classi cation but with metadata. Subtasks 1 and 2 focus on generic Spanish, and subtasks 3 and 4 on Mexican Spanish. We have participated in the four subtasks with the aim of promoting the automatic identi cation of o ensiveness in Spanish variants. Our proposal for solving these subtasks is based on the combination of linguistic features (including ne-grained negation features) and embeddings using transformers and ensemble learning. We ranked in second place in subtask 1 with a micro-averaged F1-score of 87.8289%, rst in subtask 2 with a micro-averaged F1-score of 87.8289%, fth in subtask 3 with a macro-averaged F1-score of 67.0588%, and rst in subtask 4 with a macro-averaged F1-score of 66.9449%.
This work describes the participation of the UMUTeam in the shared task MeO endEs 2021 [22], organised in the IberLEF 2021 workshop [21] and aimed at the identi cation of o ensive language and its categories. A text is considered o ensive if it contains hurtful, derogatory or obscene comments made by one person to another person [25]. The use of o ensive language in social media has increased in recent years. Some users make use of the freedom of expression o ered by these media to communicate in an o ensive way. This poses a major problem for society as o ensive comments can cause signi cant harm to the people they are directed at, such as depression or suicide.
The aim of MeO endEs 2021 is to promote the development of tools to detect and recognise o ensive language and its categories in Spanish and Mexican Spanish. Speci cally, they are proposed four subtasks: { Subtask 1: Non-contextual multiclass classi cation for generic Spanish. Identify the type of o ensiveness used in each of the given comment: non-o ensive (NO), non-o ensive but with inadequate language (NOM), o ensive and target is a person (OFP) or, o ensive and target is a group of people or collective (OFG). { Subtask 2: Contextual multiclass classi cation for generic Spanish. Use metadata as an additional source of information to identify the type of o ensiveness: NO, NOM, OFP or OFG. { Subtask 3: Non-contextual binary classi cation for Mexican Spanish. Classify each of the given tweet as o ensive or non-o ensive. { Subtask 4: Contextual binary classi cation for Mexican Spanish. Use metadata as an additional source of information to classify each tweet as o ensive or non-o ensive.
The remainder of this manuscript is organised as follow. First, in Section 2, a short overview on workshops regarding o ensiveness and Spanish corpora is presented. Next, in Section 3, we give some insights regarding the datasets that were made available to the participants. Following, in Section 4, the methodology of our proposal is described. In Section 5, we show the results achieved by our team and compare them with those obtained by the rest of participants. Finally, the conclusions and future research directions are shown in Section 6. 2
O ensive language detection is a task of recent interest due to the proliferation
of this type of language in social media. One of the strategies used to stop
messages with o ensive content is to report these messages, but doing this manually
is not feasible due to the large amount of information that is published daily on
the Web. Therefore, research e orts are being invested to automate this process.
Studies on o ensive language have focused on hate speech [
Of the tasks previously mentioned only AMI and HatEval provided datasets on generic Spanish and MEX-A3T on Spanish Mexican, which are the languages under studied in the MeO endEs task. On the one hand, the two AMI datasets consist of documents written in English, Spanish and Italian and are annotated according to three levels: misogyny (misogyny or not misogyny), misogynistic category (discredit, derailing, dominance, sexual harassment and threats of violence, and stereotype and objecti cation) and target (individuals or groups). In the AMI shared task of IberEval and Evalita, two tasks were proposed: a binary classi cation on misogyny identi cation and a categorisation of misogynistic behaviours and targets. On the other hand, the HatEval dataset is composed of tweets written in Spanish and English related to hate-speech towards women and immigrants. Similar to AMI, two tasks were proposed in HatEval: a binary hate speech detection against immigrants and women, and an aggressive behaviour and target classi cation in which the participants were encouraged to discern between aggressive or not aggressive messages, to later identify if the victim of the harassment is a person or a collective. Finally, MEX-A3T proposed a binary aggressiveness detection track focused on identifying aggressive tweets written in Mexican Spanish. The MEX-A3T dataset was compiled from Mexico City and contains documents with o ensive, vulgar, and aggressive language. 3
Subtask 1 and subtask 2 are multi-classi cation, in which documents were tagged as non-o ensive (NO), non-o ensive but with inadequate language (NOM), and o ensive, discerning whether the target is a person (OFP) or a group (OFG). According to the description provided by the organisers, the dataset was compiled from multiple social networks, including YouTube, Instagram and Twitter. The dataset was provided to the participants in two sets, depending on whether the documents were labelled by three or ten annotators. It is worth mentioning that the organisers of the task divided the corpus into three splits, namely, train, development, and test. However, the o cial development set consisted only in 100 examples and, therefore, as the labels were not balanced, some classes get under represented. Therefore, we decided to merge train and dev, and generate two custom splits into a partition of 80-20. The original splits can be downloaded at https://github.com/pendrag/MeO endEs. The dataset distribution for subtasks 1 and 2 are depicted on Table 1, in which we can observe that the label OFG is the one with fewer instances.
In case of generic Spanish, in subtask 2, some contextual information were provided along with the dataset, regarding the author of the document. This metadata includes information about the social media in which the comment was posted, the name of the channel or the main user involved (also known as the in uencer), and its gender.
For subtasks 3 and 4, concerning Spanish Mexican, the dataset consisted into documents labelled as o ensive or non-o ensive. According to the organisers, this dataset was compiled from Twitter and labelled at rst place as: o ensive, aggressive, and vulgar but non o ensive, but nally merged as binary class. The distribution of the labels across the di erent split is shown in Table 2. We can observe that the relation among non-o ensive and o ensive documents is near 2.6 which can be considered a strong imbalance.
For subtask 4, the organisers provided a large variety of contextual data including the date of publication, its number of retweets, the number of times the tweet has been marked as liked by users, and whether the tweet is a reply or an original comment, among other contextual features. 4
To accomplish all subtasks, our proposal is grounded on the combination of
different feature sets by means of ensembles. Particularly, we focus on two types of
features sets. On the one hand, we employ linguistic and interpretable features,
compiled by UMUTextStats (LF) [
Regarding the linguistic features, the UMUTextStats tool is capable to categorise a total of 365 features regarding semantics, pragmatics, lexical process, social media, gurative language, or correction and style, among others. Concerning the negation features we extract the list of negation cues appearing in each text (simple cues (e.g., \no"/ not ), continuous cues (e.g. \en mi vida"/ in my life) and discontinuous cues (e.g. \ni...ni"/ nor...nor ) and compute their total. With regard to the contextual features (CF), as commented during corpus analysis (see Section 3), the datasets were very di erent regarding complexity and number of features. For the generic Spanish dataset we encode the features regarding media and gender with one-hot encoding. In case of Mexican Spanish, we keep the features provided by the organisers of the task but we include extra features from the date of the posts, to know whether a tweet was posted during weekend or working day, and we divide the day into several time slots to distinguish the tweets written in the morning, afternoon, evening and night.
For all feature sets we carry out a process of preprocessing and normalisation
of the features. First, for the linguistic features we scale each feature
independently in a range [
To combine the features we evaluate di erent forms of ensembles of the best model per feature set. Speci cally, we evaluate three types of ensembles: (1) based on majority voting (mode), (2) based on a modi ed version of the majority voting ensemble, weighting each vote with the weighted F1-score achieved on the validation set, and (3) training a logistic regression model from the predictions of each neural network model.
Prior to our participation we evaluated di erent neural networks models and traditional machine learning models in order to get some insights for the reliability of each feature set used in combination or separately, as well as which hyper-parameters of the networks worked well for each subtask. We, therefore, performed an hyperparameter optimisation stage evaluating 110 neural network per feature set in isolation and in combination, both for generic Spanish (subtasks 1 and 2) and for Mexican Spanish (subtasks 3 and 4). We ranked each model based on the micro-averaged F1-score (we used this measure because in CodaLab the organisers indicated that submission would be evaluated with it, but nally for subtasks 3 and 4 they used macro-averaged F1-score). The feature sets evaluated during this stage were LF, NE, SE, BE, BF, and CF. During this stage we also evaluated convolutional and recurrent neural networks with WE from Spanish pre-trained word embeddings from fastText, gloVe, and word2vec.
For each neural network tested during the hyperparameter optimisation we evaluated di erent depths and di erent number of neurons (8, 16, 48, 64, 128, 256). The layers and the neurons were organised in shapes, including funnel, rhombus, long funnel, brick, diamond, and triangle. It is worth noting that the best results were achieved by simple models in the majority of cases, with one or two layers and few neurons per layer. The majority of architectures evaluated were Multilayer perceptrons because we rely on sentence xed embeddings and linguistic features, thus, spatial and temporal data cannot be exploded. However, in case of WE, we also evaluated Convolutional Neural Networks (CNN), Bidirectional Long Short Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU). In addition, we tried di erent dropout rates to avoid over tting (0, 0.1, 0.2, and 0.3) and several activation functions including relu, sigmoid, tanh, selu, and elu. We also included an early stopping mechanism and a learning rate scheduler.
Table 3 depicts the best parameters for each feature set for the generic Spanish and Mexican Spanish datasets. We can observe that the best results were achieved with multi-layer perceptrons, even for the WE, and with shallow neural networks with one or two hidden layers. Deep neural networks achieved the best result for NE in both datasets, with 5 and 7 hidden layers respectively, and WE, both with 6 hidden layers. CF required a very simpler network for generic Spanish but a complex one for Mexican Spanish. This fact can be explained due to the simplicity of the CF for generic Spanish, including only the gender and media type of the documents. It draws attention, when comparing generic and Mexican Spanish, that the models for generic Spanish are generally simpler than those for Mexican Spanish, despite having more instances and being multiclass. 5
This section is divided into the two main subtasks regarding o ensive content detection in generic Spanish (see Subsection 5.1) and Mexican Spanish (see Subsection 5.2). 5.1
Subtasks 1 and 2. Non-contextual and contextual multiclass classi cation for generic Spanish Subtask 1 and subtask 2 were evaluated by the following metrics: micro-averaged, macro-averaged and weighted-averaged versions of precision (P), recall (R) and F1-score (F1), and Mean Squared Error (MSE). The measure selected by the organisers for ranking the systems was the micro-averaged F1-score.
Each team could participate with up to three submissions and select the best of them as o cial result. Table 4 contains the results of each of our runs for subtask 1, non-contextual multiclass classi cation for generic Spanish. The rst run consists of an ensemble of neural networks models trained with BE, LF, NE, SE, and BF. The ensemble was built using a logistic regression from the individual probabilities of each model. The second run, which is our o cial result, consists of the same type of ensemble but only with BF, SE, and BE; that is, by removing the linguistic features. We can observe than in this case, we achieved slightly better micro-averaged F1-score but worse macro-averaged precision and F1-score, which suggest that the usage of linguistic features are bene cial for the classes NOM, OFG, and OFP. The third run consists of another type of ensemble, based on the weighted mode of the individual predictions of each model. The weights for this model were calculated based on the weighted F1-score achieved with our custom validation set. We can observe than this run achieved worst results than the ensembles based on logistic regression except for MSE.
As it has been previously mentioned, we selected our second run to participate in subtask 1.The o cial leader board is depicted in Table 5. We reached the second best result with a micro-averaged F1-score of 87.8289%. with a di erence of 0.3307% with the best result, achieved by saroyehun (88.1596% of microaveraged F1-score), and followed by xjywing (87.3291% of micro-averaged F1score) with a di erence of 0.4998%. Due to the high number of non-o ensive instances, the results of the micro-averaged F1-score among all participants is similar. By looking the macro-averaged F1-score, we can observe that Marta NG BD and Timen achieved lower results, which can indicate that some minority labels were not classi ed. This subtask also includes a regression metric based on the probabilities assigned for each class, that is measured using the Mean Squared Error (MSE). As it can be observed, we got an MSE of 0.41134 and the results achieved by the rest of the participants (lower is better) matches with the o cial rank.
Regarding substask 2, contextual multiclass classi cation for generic Spanish, we employed the same techniques than for subtask 1. Table 6 contains the results of our three runs. The results for the rst run, which consisted of the ensemble based on logistic regression over BE, LF, NE, SE, and BF but including contextual features (CF) in the ensemble, improves the micro-averaged F1-score achieved in subtask 1 in 0.0294%. This behaviour was expected as the contextual features of Spanish were few. However, the results improve the macro-averaged metrics, which indicates that the contextual features contribute to the texts with o ensive or vulgar content. For the second run, however, we evaluate a di erent approach from subtask 1. We retrain the model including the validation set but the results were not good. Specially, we can observe a signi cant drop in the macro-averaged recall. Finally, for the third run we adopted a similar approach for the second run of the rst subtask (see Table 4) removing the linguistic features but keeping the ensemble model based on logistic regression from the training dataset. As it can be observed, this run achieves worst micro-averaged F1-score and macro-averaged precision, recall and F1-score. In this case, the rst run is the one we selected for the o cial leader board of subtask 2. As it can be observed from Table 7, only two participants sent runs and we achieved the rst position in this subtask with a micro-average F1-score of 87.8289%.
Figure 1 contains the ten top-ranked linguistic features according to mutual information. We can observe that general and negative psycho-linguistic processes are strong features to discern among non-o ensive documents with the rest of the classes. However, they are not indicators to discern among o ensive (OFP, OFG) or vulgar language (NOM). We also found informal speech and swear language are the strongest correlation with the label NOM. O ensive speech is also a strong marker related to when the o ensive message is towards a person (OFP) and when the usage of the language is vulgar or not adequate (NOM). As it can be observed, the number of words have greater in uence on NO and OFG classes. We can nd a correlation of this feature with the number of syllables, the overall length of the documents and readability scores. 5.2
Subtasks 3 and 4. Non-contextual and contextual binary classi cation for Mexican Spanish The subtasks of contextual and non-contextual binary classi cation for MexicanSpanish were ranked by macro-averaged F1-score of the o ensive class. Similar to subtasks 1 and 2, the number of participants was lower in the contextual subtask, with only three participants in subtask 4.
Table 8 depicts the results achieved individually for each of our runs in subtask 3, non-contextual binary classi cation for Mexican Spanish. Similar to subtask 1 and subtask 2, the rst run consisted of an ensemble based on logistic regression from the probabilities of the neural networks models trained with
OFG psycholinguistic processes
negative psycholinguistic processes
negative general stylometry corpus words count
register offensive speech stylometry corpus syllabes count
stylometry corpus length register informal speech swear stylometry corpus words per
sentence stylometry corpus inflesz stylometry corpus readability 0% 25% 50% 75% 100% only one feature set (BE, LF, NE, SE, and BF). We achieved a macro-averaged F1-score of 66.7791%. Our second run is the one that is on the leader board and it consisted of removing the linguistic features (LF and NE). We can observe than in this case, better results are achieved for macro-averaged metrics. One explanation for this fact is that the linguistic and negation features could provide contradictory results for Mexican Spanish as they were designed for generic Spanish. Third run also consisted of an ensemble based on regression but using BE, LF, NE, and BF. We achieved better results than in the rst run by excluding the SE features.
For subtask 3 we selected our second run as o cial result. Table 9 depicts the results of the leader board for this subtask. We reached position 5 in the rank with a macro-averaged F1-score of the o ensive class of 67.0588%. The best result was achieved by vic gomez with a macro-averaged F1-score of 70.2619%. Compared with the best results, we achieved a similar macro-averaged recall score, even greater than the two best overall submits, but their macro-averaged precision was higher (76% and 75% vs. our 66.50%). Other teams achieved even greater macro-averaged precision (xjywing with 88.8333%, aomar with 87.5000%, and 91.8333%) with smaller macro-averaged recall.
Regarding subtask 4, contextual binary classi cation for Mexican Spanish, we sent three runs whose results are shown in Table 10. The rst one, consisted of an ensemble model based on logistic regression of neural networks models for the following feature set: LF, NE, BE, BF, SE, and CF. In the second run, we sent the same model but we adjusted the CF features by using Robust Scaler, as we found heavy outliers in the data. However, as we can observe, the results were the same that we achieved with our rst run. Finally, our third run consisted of an ensemble with the following feature set: LF, NE, BF, and CF. Compared with the rst run, we removed SE. This run was submitted because we achieved good results with our custom validation set.
In subtask 4, as previously stated, we submit the third run to the o cial leader board, that achieve slightly better macro-averaged precision and slightly worse macro-averaged recall than the other two runs. Table 11 depicts the ofcial ranking for subtask 4. Only three teams sent runs for this subtask. We achieved the rst position, with a macro-averaged F1-score for the o ensive class of 66.9449%, improving slightly our results from the previous subtask. Compared with the second best result, we achieved slightly worse macro-averaged recall but higher macro-averaged precision.
non-offensive offensive register offensive speech strong
social media mentions register offensive speech morphosyntax morphology verbs subjunctive compound pluperfect morphosyntax morphology verbs
function main morphosyntax morphology verbs
nonfinite infinitive lexical social analytic psycholinguistic processes
negative general social media jargon errors orthographics misspelled words 0% 25% 50% 75% 100% Fig. 2. Mutual Information of the ten top-ranked linguistic features averaged by label for the Mexican-Spanish dataset
Figure 2 shows the ten top-ranked linguistic features for the Spanish Mexican dataset averaged by label. As it was expected, linguistic features related to o ensive speech are highly related to the o ensive class but it also appears signi cantly on tweets labelled as non-o ensive, which suggests that there are texts considered as non-o ensive that contain o ensive words. Regarding social media, the usage of mentions is most common on o ensive tweets, which suggests that o ensive speech is more common to individuals than groups. However, social media jargon that includes words such as retweets, posts, or direct messages are more common on non-o ensive tweets. The rest of the features only have slightly variations regarding the label, as happens on the usage of verbs on subjunctive compound pluperfect or the number of misspellings that are slightly more common on o ensive tweets. On contrasts, verbs in in nitive, or analytic thinking are more common on non-o ensive tweets. 6
In this working notes we describe the participation of the UMUTeam in the task MeO endES regarding o ensive language detection in di erent variants of Spanish. Our proposal for solving the di erent subtasks have been grounded on the combination of linguistic and negation features with state-of-the-art transformers, combined as ensembles of neural networks classi ers or combined in the same neural network. Our results have achieved very good results in the o cial leader board. We reached the rst position on subtasks 2 and 4 regarding o ensive detection with contextual features on generic and Mexican Spanish respectively, and position 2 and 5 on the non-contextual subtasks. It is worth mentioning, however, that the number of participants who sent runs for contextual tasks was lower than for non-contextual features tasks.
The following insights were obtained during the participation in this task. First, the results achieved indicate that linguistic features and transformers mutually bene t from each other, increasing their reliability. Our results indicate that ensembles learned from training a logistic regression machine-learning classi er from the individual probabilities of each model achieve better results than ensembles based on the mode (the label most voted) or the weighted mode. Second, we observe than sentence- xed embeddings from the ne-tuned model of BETO, which we called BF, outperform plain BE vectors in all cases, and it is more easier to combine them with other feature sets. Third, during the hyperparameter evaluation stage we observed that simpler neural network models with only a few layers and few neurons behave better than models with more than four hidden layers. Forth, we found that those linguistic features concerning negative processes, such as anger, sadness, anxiety, are discriminatory features regarding o ensive language detection. However, our results suggest that these features are reliable to distinguish between non-o ensive documents from the ones that include o ensive or vulgar language but not to di erentiate among di erent types of o ensive language.
Lastly, we are pleased with the opportunity we have been give to participate in this task. This work has been a collaboration between Universidad de Murcia and Universidad de Jaen.
This work was supported by the Spanish National Research Agency (AEI) through project LaTe4PSP (PID2019-107652RB-I00/AEI/10.13039/501100011033). In addition, Jose Antonio Garc a-D az has been supported by Banco Santander and University of Murcia through the industrial doctorate programme. Salud Mar a Jimenez-Zafra has been partially supported by a grant from Fondo Social Europeo, Administration of the Junta de Andaluc a (DOC 01073), Fondo Europeo de Desarrollo Regional (FEDER) and LIVING-LANG project (RTI2018094653-B-C21) from the Spanish Government. 10. Garc a-D az, J.A., Canovas-Garc a, M., Colomo-Palacios, R., Valencia-Garc a, R.: Detecting misogyny in spanish tweets. an approach based on linguistics features and word embeddings. Future Generation Computer Systems 114, 506 { 518 (2021). https://doi.org/10.1016/j.future.2020.08.032, http://www.sciencedirect.com/science/article/pii/S0167739X20301928 11. Jimenez-Zafra, S.M.: Negation processing in spanish and its application to sentiment analysis. Procesamiento del Lenguaje Natural 66, 193{196 (2021) 12. Jimenez-Zafra, S.M., Cruz-D az, N.P., Taboada, M., Mart n-Valdivia, M.T.: Negation detection for sentiment analysis: A case study in spanish. Natural Language Engineering 27(2), 225{248 (2021) 13. Jimenez-Zafra, S.M., Morante, R., Blanco, E., Valdivia, M.T.M., Lopez, L.A.U.: Detecting negation cues and scopes in spanish. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 6902{6911 (2020) 14. Jimenez-Zafra, S.M., Taule, M., Mart n-Valdivia, M.T., Urena-Lopez, L.A., Mart , M.A.: Sfu review sp-neg: a spanish corpus annotated with negation for sentiment analysis. a typology of negation patterns. Language Resources and Evaluation 52(2), 533{569 (2018) 15. Kumar, R., Bhanodai, G., Pamula, R., Chennuru, M.R.: Trac-1 shared task on aggression identi cation: Iit (ism)@ coling'18. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). pp. 58{65 (2018) 16. Kumar, R., Ojha, A.K., Malmasi, S., Zampieri, M.: Benchmarking aggression identi cation in social media. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). pp. 1{11 (2018) 17. Kumar, R., Ojha, A.K., Malmasi, S., Zampieri, M.: Evaluating aggression identi cation in social media. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying. pp. 1{5. European Language Resources Association (ELRA), Marseille, France (May 2020), https://www.aclweb.org/anthology/2020.trac-1.1 18. Malmasi, S., Zampieri, M.: Detecting hate speech in social media. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. pp. 467{472 (2017) 19. Mandl, T., Modha, S., Kumar M, A., Chakravarthi, B.R.: Overview of the hasoc track at re 2020: Hate speech and o ensive language identi cation in tamil, malayalam, hindi, english and german. In: Forum for Information Retrieval Evaluation. pp. 29{32 (2020) 20. Mandl, T., Modha, S., Majumder, P., Patel, D., Dave, M., Mandlia, C., Patel, A.: Overview of the hasoc track at re 2019: Hate speech and o ensive content identi cation in indo-european languages. In: Proceedings of the 11th forum for information retrieval evaluation. pp. 14{17 (2019) 21. Montes, M., Rosso, P., Gonzalo, J., Aragon, E., Agerri, R., Angel Alvarez Carmona, M., Alvarez Mellado, E., de Albornoz, J.C., Chiruzzo, L., Freitas, L., Adorno, H.G., Gutierrez, Y., Jimenez-Zafra, S.M., Lima, S., de Arco, F.M.P., (eds.), M.T.: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021). (CEUR Workshop Proceedings) (2021) 22. Plaza-del-Arco, F.M., Casavantes, M., Escalante, H., Mart n-Valdivia, M.T., Montejo-Raez, A., Montes-y-Gomez, M., Jarqu n-Vasquez, H., Villasen~or-Pineda, L.: Overview of the MeO endEs task on o ensive text detection at IberLEF 2021.
Procesamiento del Lenguaje Natural 67(0) (2021) 23. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bertnetworks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019), http://arxiv.org/abs/1908.10084 24. Stru , J.M., Siegel, M., Ruppenhofer, J., Wiegand, M., Klenner, M., et al.: Overview of germeval task 2, 2019 shared task on the identi cation of o ensive language (2019) 25. Wiegand, M., Siegel, M., Ruppenhofer, J.: Overview of the germeval 2018 shared task on the identi cation of o ensive language. In: 14th Conference on Natural Language Processing KONVENS 2018 (2018) 26. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Semeval-2019 task 6: Identifying and categorizing o ensive language in social media (o enseval). In: Proceedings of the 13th International Workshop on Semantic Evaluation. pp. 75{86 (2019) 27. Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., Coltekin, C.: Semeval-2020 task 12: Multilingual o ensive language identi cation in social media (o enseval 2020). In: Proceedings of the Fourteenth Workshop on Semantic Evaluation. pp. 1425{1447 (2020)