Introduction

In the last years, there have been many initiatives in the NLP and Machine Learning community, to guide research efforts towards solutions in the automatic detection of threats and risks to the users of social networks. Those threats include aggressiveness, hate speech, harassment, racism, misogyny, among many others. For spanish language, those efforts have been promoted by academic competitions in specific tasks, such as the events organized by TASS [6,10], PAN [5] and particularly, MEX-A3T@IberLEF [1,2,7] and MeOffendEs@IberLEF [12,13], which includes a track for aggresiveness and offensiveness identification task respectively for tweets in Mexican spanish.

Detection of offensive comments or posts in social media is not easy, because it is not depending on the presence or absence of specific words. As an example, consider the next tweets taken from MEX-A3T 2020 training corpus:

In the first case, the tweet is non-offensive, even when it contains vulgar and rude language. The second tweet is offensive, although the language is less vulgar than the first one. Based on that, we argue that it is necessary to take into account the context in which words are used.

There are many proposals to tackle offensive and aggressive content detection in social media for spanish language, with document representations based on n-grams (word and character level), and word embeddings, with classifiers based on standard machine learning and deep learning approaches [1,3,7]. However, in the last year, there is a visible trend in the use of contextualized representations of words, such as Bi-LSTM, Bi-GRU, and Transformers-based models, such as BETO [8] with and without fine tunning [2,4,16,17]. State of the art results for Mexican spanish on this task has been reached with a bagging-like scheme, by combining different BERT models trained on different augmented datasets [11].

Similar to [11], we propose an ensemble of BERT models, but also, we use a pre-processing step in order to obtain valuable text descriptions of specialized language used in tweets, followed by an extension of the training corpus. The empirical evaluation shows that the proposed approach obtained the best results in the challenge by ranking first place. In the following sections, our proposal is explained in detail.

This document is organized as follows: Section 2 describes the dataset and the experimental settings. Section 3 describes the proposed pipeline, and Section 3.4 the experimental results. Finally, Section 4 outlines the conclusions.

Dataset and model settings

OffendMex corpus consists of a training set of 5060 tweets and a validation set of 76 tweets, from the total, 80% was used for training and 20% for evaluation purposes. The dataset has a length mean of 24.11 and a maximum of 60 tokens, with an unbalanced ratio of 2.66 and the offensive class as the minority.

For this task, a pre-trained BERT model on Spanish was used [8], and for the fine-tuning step for small datasets (less than 100,000), we used the exhaustive search over the recommended hyperparameters [9] and we choose the best one on the evaluation set. As a result, we used a BERT model with a training batch size of 16 for 4 epochs, and an Adam optimizer with a learning rate of 3e-5.

Pipeline

In this section, the proposed pipeline is described: the corpus pre-processing step, second, the extended corpus step, and finally, the BERT ensemble step.

Step 1: Pre-processing

From other classification tasks such as irony or sentiment classification has been proved that adding the tweet jargon like hashtags, emojis, and emoticons as text descriptions improves tweet classification tasks through deep learning models like BERT [14,15].

Our procedure is the following:

-Hashtags are split into words (see Figure 1) using the python word ninja library (https://github.com/keredson/wordninja) with a Spanish dictionary made with the Spanish fasttext vocabulary (https://fasttext.cc/docs/en/crawlvectors.html) . -Emojis are replaced with their text meaning in Spanish (see Figure 1) given by the python library emoji (https://github.com/carpedm20/emoji/). -Emoticons are replaced with a text representation in Spanish similar to the emojis meanings (see Table 1). -Words out of vocabulary are replaced with the corresponding words (see Table 1). After pre-processing step, the maximum corpus length increases twice (see Figure 2); this is the reason why the maximum sequence length of BERT model is 128. Table 1. Expressions and their Spanish text representations for emoticons and words out of vocabulary. Note that the main idea is to describe with words the meaning of the emoticon. For example, for the emoticon :) we replace by "smiling face".

EXPRESSIONS SPANISH

Step 2: Extend training corpus

It was demonstrated [16] that increasing the training corpus examples with other corpus labeled with a related task such as hate speech could improve model performance. In this pipeline, we choose to use hate speech and negative sentiment from the HatEval 2019 in Spanish [4] and TASS 2019 for Mexican Spanish [10] corpus, respectively. The methodology proposed to add examples from other corpus as a way to improve model performance and reduce the unbalanced ratio is shown in Figure 3, and consists on the following three steps. In the first step the corpus must be preprocessed by the method described in section 3.1, the second step consists on training with the OffendMex 2021 corpus by the method described in section 3.3, and make inference on the HatEval and TASS corpus, and then, we select only those examples whose weights in the classifier are greater or equal to 0.95, which are added to the OffendMex corpus as offensive examples. Finally, the step three consists of training from scratch the model again. The intuitive idea of this step is to augment the training data only with those instances that could improve the classification score.

Step 3: Bert ensemble

In order to alleviate BERT instability of fine-tuning on small samples and unbalanced datasets, it was shown [11] that using single BERTs as weak models and through an ensemble of 20 BERTs and a weighted voting scheme, which means that accumulating the softmax layer outputs and selecting the class with the maximum weight makes a more robust model (see Figure 4).

Results

After following the proposed pipeline described in Section 3, we obtained the results shown in Table 2, where it can be seen that each step on the proposed pipeline helps to improve the model performance, as we expected. The best result achieved a F1 score on the evaluation set of 71.07 with a 20 BERT ensemble, pre-processing, and finally adding more examples to the training corpus.

Conclusions

This work presented a pipeline of three steps for offensiveness detection on Mexican Spanish social media that effectively achieved first place on the MeOf-fendEs@IberLEF 2021 subtask 3 competition with a F1 score of 0.7026 on the test set. Our experimental results on the evaluation set shown that each step on the pipeline improves the model performance. We thought this pipeline could be implemented quickly and successfully in other related tasks such as aggressiveness detection.