This paper details the participation of DiTana-PV team in the sEXism Identification in Social neTworks (EXIST) task at CLEF 2024. Specifically, we focused on Tasks 4 and 6, which involved identifying and categorizing sexism in memes. Our primary objective was to evaluate the efect of machine translation on model performance, as well as to explore data augmentation techniques and task combination strategies. By translating Spanish data to English and leveraging a pretrained BERTweet model fine-tuned for sexism detection, we aimed to improve classification accuracy. This work highlights the potential of translation and data handling techniques to enhance multilingual NLP tasks, contributing to more inclusive and efective AI applications in social media analysis.
The proposed lab of CLEF 2024 was sEXism Identification in Social neTworks (EXIST) [
The information provided includes the memes images and their transcriptions into texts, along with some information about the annotators. We divided the dataset into three diferent partitions: train, validation and test. Tab. 1 shows the sample distribution of each language.
In Tab. 2 we can see how each class for Task 4 is distributed along both languages for the train and validation partitions. As we trained the models with the hard labels there were some samples that half of the annotators labeled as sexist and the other half as not sexist (tie). After some experimentation, the best solution found to this situation was to discard the ambiguous samples, considering them noisy samples.
To measure the performance of our proposed systems, the Intermediate Concept Measure (ICM) [
Here we described our main objectives and experimental findings for our participation in this competition. All this objectives focused on text processing, omitting the part of image processing for the memes.
The main objective of participating in this competition was to develop models that could detect sexism in memes and categorize it. More specifically, we were interested in how Machine Translation could afect the model’s performance.
As Machine Translation has advanced enormously and there are far more resources in English than in any other language, some minority languages that may not have enough resources to train this kind of models could benefit from this: the idea of previously translating the content to English and using any model that was trained with all the available data in English.
This is an open line of research that we think has lots of future because it could mean a democratization of the benefits of Machine Learning for all languages, regardless of their available resources.
We also aimed to evaluate how the use of Data Augmentation afects the performance of the models. Since it is a very widespread technique in the world of Machine Learning, especially in Computer Vision, but it has been proven that NLP tasks can also take advantage from it.
As the dataset was unbalanced, as we could see on Tab. 2, another of our objectives was to work with this type of datasets in which not all classes have the same presence and therefore a series of measures must be taken to prevent the model from being biased.
The last of our objectives, specially useful for Task 6, was to see what efect caused the combination of inferences for both tasks. We wanted to try to help a model used for Task 6 with the predictions of a model used for Task 4. This will be further explained in the following sections.
In this section we will explain in detail the diferent approaches that best worked for us.
After some preliminary experiments we discovered that the results in the English dataset were always some points higher than in the Spanish dataset, so we decided to automatically translate the Spanish samples into English samples, assuming the loss of quality that such automatic translation could entail. This approach was interesting because there are lots of resources in English that could improve our results.
For both tasks and languages (as we translated the Spanish samples to English), we used as pretrained
model the BERTweet-large-sexism-detector [
As seen in Tab. 2 there were two main problems with hard labels: there were some ambiguous ones and the amount of sexist and not sexist samples was unbalanced. We decided to discard the ambiguous samples, because taking them into account in the training process just added noise and led to worse performance. To try and solve the class-imbalance, for Task 4, we applied a weighted-loss function in order to give more weight to the not sexist class.
In some of the models, we also performed Data Augmentation to increase the amount of training
data. This technique has proven quite good results in the vast majority of Machine Learning tasks and
NLP is not an exception [
For Task 6, we trained the models for recognising six diferent labels for each meme: the ones for the type of sexism detected plus an extra one for not sexism detected at all. As one of our purposes was to try to combine the information from both tasks, we also focused on training a model for predicting just the main five labels for the type of sexism inside the meme.
We would use our inferences from Task 4 to detect if the meme was sexist or not, previously, and then in case the meme was classified as sexist, the second model would predict what type(s) of sexism were inside the given meme. This approach, as will be seen further in this paper, has been proven to work finely.
In this section there is a description of each one of the six models that were presented to the competition. All these were fine-tuned for 10 epochs with an NVIDIA RTX 3090 with 24GB. The relevant hyperparameters are: 8 samples per device, and a learning rate of 5 · 10− 5 with a linear scheduler.
For Task 4 we developed 3 diferent models following the diferent approaches explained. For all the scenarios described, one separate model was trained for each language.
The first pair of models ( M14) were trained with all the samples from the train partition in English and in translated Spanish, for the English model we also added the validation partition in translated Spanish and vice versa. The second set of models (M24) were trained exactly as M14 but we added a weighted-loss function to correct the explained class-imbalanced. The last pair of models (M34) were trained as M24 but increasing the training dataset applying Data Augmentation.
For this task, we developed 5 diferent models, according to the approaches in the previous sections: a pair for predicting 6 labels, a pair for predicting 5 labels and a single model for predicting six labels for both languages together. For the first two pairs, there is a model for each language (English and translated Spanish).
For the first pair M16, the Spanish model was trained with translated Spanish and the English one was trained with the English samples. The second pair, M26, follows the same logic but for making ifve label predictions. The last one M36 is not a pair but a single model which was trained with both translated Spanish and English samples.
In this section we will discuss the results obtained for each of the models presented.
For the classification models, we can see that just by translating the Spanish samples to English, combining all of the samples in the training process and using as checkpoint the BERTweet model already fine-tuned in a sexism detection task, was enough to achieve a good result. Although, as we can see in the Annex A, it is lower than what was obtained during validation.
Adding weights to the loss function to gives more importance to the not sexist samples as they appear fewer times, allowing us to enhance a little bit the model’s performance, as was already proved during validation.
When we applied Data Augmentation techniques we got worse results. This was surprising because during validation for Spanish it improved the results, and for English it got lower results but it was far more what we won in Spanish than what we lost in English. We hypothesize the reason why Data Augmentation this is because we already inserted noise in the Spanish samples through Machine Translation, thus the addition of more noise through paraphrasing with BERT was too much.
For the categorization models in Task 6, the results indicate a varied performance across diferent approaches. As shown in Table 4, the models trained to predict six labels for each language separately (M16) achieved the highest performance in terms of ICM-Hard and ICM-Hard Norm metrics, although its Macro F1 score was slightly lower than that of the M26 models.
The M16 models, which predicts six labels, performed the best in terms of ICM-Hard and ICM-Hard Norm. This suggests that having a separate model for each language and focusing on the six distinct labels allowed the model to better capture the nuances of sexism categorization.
Interestingly, the M26 models, which was trained to predict five labels, did not outperform the first models, indicating that removing the not sexist label from the categorization task, together with the possible error the previous model (as it was a joint prediction) may have introduced, might have led to a loss of valuable context needed for accurate categorization.
The single model trained on both English and translated Spanish samples (M36) performed the worst. This may be due to the increased complexity and noise introduced by combining data from two languages, even after translation. The challenges of handling translated text, which may not perfectly capture the original sentiment or nuances, likely contributed to the poorer performance.
The results obtained from our participation in the sEXism Identification in Social neTworks (EXIST) competition demonstrate the potential impact of machine translation and data augmentation on improving model performance for sexism detection and categorization tasks.
Our experiments highlighted the benefits of translating minority language datasets into English, utilizing the wealth of available resources and pretrained models in English to enhance performance. This approach showed significant improvements in classification accuracy, suggesting a promising direction for future research aimed at democratizing the benefits of machine learning across languages with fewer resources.
The implementation of a weighted-loss function efectively addressed class imbalance, further improving model performance. However, the addition of data augmentation techniques, while beneficial during validation, did not consistently enhance results in the final evaluation, indicating the need for careful consideration of noise introduced by such methods, especially when combined with machine-translated data.
In Task 6, combining inferences from Task 4 to aid categorization proved to be a viable strategy, though the overall performance varied across diferent model configurations. This underscores the complexity of multi-label classification tasks and the importance of tailored model training approaches.
Overall, this highlights the value of leveraging translation and sophisticated data handling techniques to improve model accuracy in NLP tasks involving multiple languages. Future work could focus on further refining these methods, exploring visual-textual integration for meme analysis, and investigating more robust data augmentation strategies to mitigate noise.
These contributions provide a foundation for advancing sexism detection in multilingual contexts, paving the way for more inclusive and efective AI applications in social network analysis.
This work is partially supported by the Valencian Graduate School and Research Network of Artificial Intelligence (ValgrAI).
In this appendix section illustrate the results that each of the models obtained for our validation partition of the dataset for each task, distributed by language.
As we can see in Tab. 5 the starting pint was not brilliant, although we could improve our results for the Spanish split by four points just by translating the samples to English and using the BERTweet ifne-tuned model for sexism detection. The weighted-loss function has also allowed to increase some points and the Data Augmentation has also enhanced the model performance.
In Tab. 6 we can see that, just comparing the baselines using BERT and BETO, the English partition gets a better performance. As in the Spanish validation split, just with the combination of the English and translated Spanish datasets and using the BERTweet-large-sexism-detector model as checkpoint, we already obtained good results. Adding the weighted-loss improved them, as happened with the Spanish dataset. In this case, diferently from what happened in the Spanish dataset, the Data Augmentation did not improve model’s performance. Model M16, which trained to predict six labels separately for each language, achieved a moderate performance. However, its F-Measure indicates room for improvement, suggesting that the model may struggle with certain categories or nuances in sexism categorization. The model trained to predict five labels M16 exhibited lower performance across all metrics compared to M16. This could be attributed to the removal of the not sexist label from the categorization task, as well as mixing its predictions with the ones from the previous model, potentially leading to loss of valuable context for accurate classification. The results can be seen in Tab. 7
For the Engish partition, as shown in Tab. 8 the first model achieved moderate performance, with comparable metrics to its counterpart on the Spanish partition. While the ICM-Hard and ICM-Hard Norm scores indicate reasonable categorization accuracy.
The single model trained on both English and translated Spanish samples demonstrated the best performance among the three models on both languages. Despite the diferences in validation data composition, this model achieved significantly higher ICM-Hard and ICM-Hard Norm scores, indicating better categorization accuracy and similarity to ground truth labels. However, it’s important to note that the validation data composition for M36 difered from that of M1 6 and M26, incorporating both English and translated Spanish samples. As such, the higher performance of M36 cannot be directly compared to the other models due to this diference in validation data composition. -0.511 -1.450 1.762 0.220 0.143 0.370