This shared task system description depicts two neural network architectures submitted to the EXIST task at IberLEF 2021, among them the twelfth classi ed in the second sub-task. We present in detail the approach and topologies used to obtain the two systems which we submitted. Both systems are based on pretrained language models and solve the two subtasks simultaneously, with the rst system using di erent networks for English and Spanish and the second using a multilingual approach.
EXIST (sEXism Identi cation in Social neTworks)[16] is a shared task in Automatic Misogyny Identi cation in social networks at IberLEF 2021[14], a comparative evaluation campaign for Natural Language Processing Systems in Spanish and other Iberian languages. It aims to detect online proof of sexism in Spanish written language, which may help to determine the evolution of new equality policies in online environments, as well as to encourage better behaviours in society. AI and NLP researchers are working on Automatic Misogyny Identi cation (AMI) shared tasks like this one to distinguish misogynist contents from non-misogynous ones and to categorize their type [4, 7{9].
EXIST is divided into two subtasks: { Task 2: Sexism Categorization. It is a multiclass classi cation task. The same texts analyzed in task 1 have to be classi ed into one of the ve categories decided by the organization, which are idelogical and inequality, stereotyping and dominance, objecti cation, sexual violence and misogyny and non-sexual violence [16].
In this working notes we are going to explain our approach on this shared task and how we designed and trained the submitted models. 2
We have submitted two systems capable of making predictions for the two
subtasks. Both systems are based on pretrained Transformer models [17], and both
were designed and trained using biome.text [
These two systems were trained directly over the categories of the second task but were used to predict both tasks: if any of the categories of the second task surpassed a given threshold (independently calculated for each neural network), it is predicted as 'sexist' for the rst task; otherwise, it is predicted as 'nonsexist'. The category of the second task is chosen as the output category from the neural network with the highest probability. 2.1
Our rst system, denoted as Run 1 in the submitted results, has been designed
using two Deep Neural Networks, one for English and one for Spanish Language.
The Spanish Transformer-based language model was BETO, a BERT model
trained on a big Spanish corpus [
Both neural networks were ne-tuned for the task, each one trained with the dataset corresponding to its language, and evaluated using the macro-averaged F-measure. The system was created combining those two networks in a basic decision tree: if the record of the test set to predict was in English, the English network is invoked to make the prediction; otherwise, the Spanish network was used. 2.2
Our second system, denoted as Run 2 in the submitted results, has been
designed using one Deep Neural Network, following a multilingual approach. The
Transformer-based language model used was twitter-XLM-roBERTa-base for
Sentiment Analysis, a XLM-roBERTa-base model trained on 198M tweets and
netuned for sentiment analysis [
This neural network was ne-tuned for the given task, using all records in the training dataset, and also evaluated using macro-averaged F-measure. At the end of the pipeline, this system was capable of predicting both English and Spanish input text. 3
Both systems were trained using the same procedure, even though the hyperparameters obtained after optimizing each neural network and the thresholds used for predicting for each system were di erent.
For the parameter updates, we used the AdamW algorithm [13]. The parameters optimized can be seen in Table 1, along with their search spaces at the start of the hyperparameter process. These parameters were optimized by means of the Ray Tune Library [12], which is tightly integrated in biome.text.
Several Hyperparameter Optimization Processes (HPOs) were performed
for each of the three neural networks, and each subsequent HPO xed some
parameters and reduced the search space for others, until we got the
bestperforming neural networks at the last HPO process. Spanish and Multilingual
neural networks needed four HPO processes, and English neural network needed
ve HPO processes. The reference metric for all these processes was
macroaveraged F-measure of Task 2. The training was done on a computer with 2
Tesla V100. These HPO processes included ASHA trial schedulers to terminate
low-performing trials [11] and a tree-structured Parzen Estimator as search
algorithm [
Once the best-performing models were obtained, a quick sweep across several random initialization seeds was performed, and then another sweep was made across di erent threshold values from 0.15 to 0.85, adding 0.05 in each step. The Parameters Learning Rate Weight Decay Batch Size Warmup Steps (LR Scheduler) Steps per epoch (LR Scheduler) Pooler Type Hidden Size (Pooler) Number of layers (Pooler) Bidirectional (Pooler) Threshold 8 12 354 gry 128 1 True 0.5 8 91 343 gru 128 1 True 0.55 16 14 348 gru 64 1 True 0.5 result of these last processes was the nal model for the Spanish and English languages (which, together, compose System 1) and for the Multilingual approach (System 2).
In Table 2 we included the details of each of the three nal models: the Spanish model, the English model and the Multilingual Model . In Table 3 we present the evaluation metrics of both tasks for each of the submitted runs on the validation and the tests data sets, as well as the model size. Tables 4 and 5 show a comparison between the submitted runs, the best models of the shared task and the baselines models (provided by the organization), divided by tasks. System 1 obtained our highest score in both tasks. Our better model was System 1 (which made run 1), which was the twelfth classi ed for task 2 and the forty sixth for task 1. System 2 underpeformed System 1, being the thirty rst classi ed on task 2 and the fty sixth classi ed on task 1.
Both results obtained on Task 2 are close to the best ones of the competition, being 0.03 and and 0.09 F-measure points away from the winner, respectively. However, our results for Task 1 are signi cantly worse, which means that our initial premise (training a system to predict label and, if any label is predicted, to also predict 'sexist') was not e ective.
We also found that the multilingual approach simpli ed the training (we only had to train one pipeline instead of two) while obtaining good inference results. It did not reach the top performing models of the competition for the second task, and it performed even worse on task 1, but we nd it a valid alternative to classic monolingual training. 5
To face this shared task, we designed two di erent systems with which we made the predictions that composed our two submitted runs. System 1 was designed with two Deep Neural Networks, one for English predictions (using Twitter-roBERTa-base O ensive Language Identi cation as the pretrained language model) and one for the Spanish predictions (using BETO as the pretrained language model). In System 2 we followed a multilingual approach, using only one Deep Neural Network to make predictions in both English and Spanish (with twitter-XLM-roBERTa-base for Sentiment Analysis as the pretrained language model). Both systems followed a multilabel approach described in previous section, with which we were able to make prediction for Tasks 1 and 2 without making di erent pipelines.
We conclude that the exploitation of the transfer capabilities of a pretrained language model and its optimized ne tuning to the target domain provides a conceptually easy system architecture and seems to be the most straight forward method to achieve competitive performance, especially for tasks where training data is scarce. We also found that, for these types of competitions, creating a model for each subtask is the best-performing approach. Better results on task 1 could have been obtained if we had trained Deep Neural Networks on the binary classi cation task.