This paper describes a system created for the DETOXIS 2021 shared task, framed within the IberLEF 2021 workshop. We present an approach mainly based in ne-tuned BERT models using a GridSearch and Data Augmentation with MLM substitution. This approach only takes into account the textual data from the dataset to prove the power of language models. Our models far outperform the baselines and achieve results close to the state-of-the-art.
Polarization can be a very problematic issue in society, especially on social media.
There are manual mechanisms to report these behaviors, however they can be
slow and ine cient. To address this, we can use NLP to detect automatically
these undesirable toxic behaviors. The DETOXIS (DEtection of TOxicity in
comments In Spanish) [
This article summarizes our participation in all the DETOXIS tasks. Given
the success of Transformer-inspired language models [
In the next section, we will brie y see some previous work related to this topic. In Section 3 we will go through a brief description of the tasks and the corpus. Then, in Section 4, we will explain the main ideas behind the proposed models. In Section 5 we will present a summary of the experiments we carried out and the results we got. Finally, in Section 6 we will expose the main conclusions of our work and results and we will also propose some ideas for future work.
There is an extensive bibliography on Sentiment Analysis and text classi cation in social networks, however not that much work has been done about identifying and classifying toxic behaviors until 2019.
Most of the toxicity detection datasets are focused on classifying what kind
of toxicity is present in the text, instead of the level of toxicity. Basile et al.
[
However, all have a common denominator. The best results have been
obtained with some kind of Transformers or BERT-based model. For instance, the
best models in Stru et al., [
This is a clear indicator of the trends in the state-of-the-art for this topic. 3
The main corpus consists of 3463 comments posted in Spanish online newspapers and forums for the train split and 890 for the test one. They were collected from August 2017 to July 2020. Furthermore, the articles were selected taking into account their potential toxicity and the number of comments in them (more than 50 comments).
For the rst task, the comments are annotated into two categories; toxic and not toxic. The second task consists of further classifying that toxicity into four levels of toxicity; toxicity level 0=not toxic, toxicity level 1=mildly toxic, toxicity level 2=toxic and toxicity level 3=very toxic.
In addition to the classi cation labels, for every sample there are also annotation about the argumentation, constructiveness, stance, target, stereotype, sarcasm, mockery, insult, improper language, aggressiveness and intolerance. However none of these are public in the test set, so they will be ignored in this study. Finally, for every comment, there is also a label indicating if the comment is a response to another comment or not. This information will not be used either because we will only focus on the textual data.
The metrics used to evaluate the results are the F-measure for the task1 and
the Closeness Evaluation Metric (CEM) [
In the table above we can see the distribution of the samples in the train split. The most notable fact is that 67% of the samples are not toxic, so the dataset is unbalanced. For the second task, we can also see a very notable unbalance.
In the table below, we can see some illustrative examples of the data and their labels: We performed a simple preprocessing where we substituted some expressions with a more normalized form: { Every URL was replaced with the token \[URL]" so we don't get strange tokens when the tokenizer tries to process and URL. Furthermore, no semantic information about toxicity can be inferred from a URL, the only information relevant for the model is that there is a URL in that token. { Finally we normalized every laugh (\jasjajajajj" ! \haha") so we minimize the noise of the misspellings, common in social networks. 4.2
We created some baselines so we can compare our models properly. We selected a HashingVectorizer + RandomForest. This way, we can compare our models to a classic feature extraction model. 4.3
We used BETO [
We tried di erent training strategies given that the classes are related to each
other:
{ The simplest approach we tried is treating both tasks as the same one, with
a multiclass classi cation model. For the rst task, everything di erent from
not toxic would be considered toxic.
{ For the second approach, we rst trained a binary classi cation model to
distinguish between not toxic and toxic for the rst task. Then we trained
a multiclass model to classify between the three levels of toxicity.
{ Similarly to the last approach, we then tried to transform the second task
into three di erent binary classi cation problems; classifying between not
toxic and the rest of the classes, mildly toxic and toxic or very toxic,
and between toxic and very toxic. With this, we tried to have very speci c
models that can di erentiate slight changes in toxicity.
{ As there are not too many samples for the last models in the previous
approach to learn correctly, we also tried with a transfer learning approach,
similar to the one presented by Sun et al. [
In addition, for the ne-tuning process, we carried out a Grid-search optimization over the main parameters of the neural network: learning rate, batch size and dropout rate. The search was performed with a 5-fold strati ed crossvalidation with the following grid: Learning rate, (1e 6; 1e 5; 3e 5; 5e 5; 1e 4); batch size, (8; 16; 32) and dropout rate, (0:08; 0:1; 0:12). The best parameters for both models were: learning rate, 1e 5; batch size, 16 and dropout rate, 0:1. 4.4
As the dataset is relatively small, we decided to run Data Augmentation techniques. The selected strategy was the Data Augmentation through the masking of words with a Masked Language Model, BETO.
For every sample in the dataset, we randomly masked 15% of the tokens and used BETO to predict them, creating a modi ed sample. With this method, we obtained double the amount of the original samples.
We trained all the models with a NVIDIA Tesla P100-PCIE-16GB GPU and a Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz CPU with 500GB of RAM memory.
The software we used was Python3.8, transformers 4.5.1 [
In the Table 3 we can see the results for our models in the test set of the
rst task. Note that the ChainBOW baseline, Word2VecSpacy baseline and the
SINAI team (winner of the task) results are taken from the task Overview [
We can see that there is almost no di erence between the binary and multiclass models. This might happen because there is a great di erence in the amount of samples and toxicity between the not toxic comments and the rest, which makes them easy to indentify in every situation. Finally we can see the the Data Augmentation strategy obtained around 0:02 points more than the models without augmentation. Our result was the second best in the competition, which proves that the simplicity of using BETO with some Grid-Search can yield really good results.
For the second task, the results were similar to the ones obtained in the rst
task. In the Table 4 we can look at them in more detail. Again, ChainBOW
baseline, Word2VecSpacy baseline and the SINAI team results are taken from
the task Overview [
This results are placed fth among all the participating teams (24), which proves that our approach, given it's simplicity and the lack of any linguistic analysis, is very good.
Model CEM Word2VecSpacy 0.6116 HV+RF 0.6214 ChainBOW 0.6535 BETO-2 models-aug 0.6891 BETO-multiclass-aug 0.6913 BETO-3 models-aug 0.704 BETO-transfer 0.7172 BETO-transfer-aug 0.7189 SINAI team 0.7495
Table 4. Results for task2 6
Through this shared task, we have seen that NLP can be of great help in detecting and classifying unwanted toxic behavior in social networks and there is still a long way to go.
The results obtained by our systems are very promising given their great performance and their simplicity. This compilation of methods is very signi cant because it could lead to much better results when combined with other improvements from the state-of-the-art.
We believe that our results could improve a lot using speci c language models
trained with corpora from social networks like TWilBert [
This work has been partially funded by the Instituto de Ingenier a del Conocimiento (IIC) and the hardware used was also provided by the IIC.