Sexism, far from being merely a conceptual issue, is a concerning and pervasive social health problem that negatively impacts individuals' well-being and perception. In today's digital era, as sexism permeates online platforms, the creation of systems that detect this type of content is a challenging yet essential task. This paper presents the approach of the CIMAT-GTO team to Task 1 of EXIST 2024, which involves identifying tweets with sexism-related content. Our proposal takes advantage of the reasoning capabilities of Llama 3 in a two-step process. Initially, we generate rationales to analyze the nature of the tweets. Then, in a second step, we let the model reflect on the previously produced reasoning. The intuitive idea is to create text that supports opposite categories, and expect the model to contrast valid and invalid reasons by itself. We then use these generated rationales as extra information to complement the tweets and fine tune a Twitter-specialized XLM-RoBERTa model. Our experiments showed that incorporating Llama 3's rationales improves performance compared to only using tweets and yields competitive results in the task, demonstrating the potential of these methods.
In today’s world, social media is an essential platform for the communication and difusion of information
and opinions among individuals. However, social media interactions are often related to misleading
or harmful content. For instance, users might directly express bias in their own generated content.
Alternatively, they could engage by sharing and commenting on biased content created by other users.
Within these interactions, one major social concern is sexism, defined as prejudice or discrimination
based on sex or gender [
Sexism negatively afects the psychological well-being of women and men not only in everyday
face-to-face interactions [
Instances of harmful sexist expressions in social media include hostile behavior and negative
evaluations of female job candidates [
In the modern world, the eficient identification of sensitive content is crucial due to the vast volume
of data, human biases, and the profound impact of this task. Despite advancements in computer science
and the deployment of more accurate and sophisticated models, the challenge remains unsolved. In
this context, several eforts include shared tasks to address this issue, such as Automatic Misogyny
Identification at IberEval [
This paper describes CIMAT-GTO’s participation in Task 1 of EXIST 2024, which tackles the binary detection of sexism in tweets. We propose a technique that takes advantage of the reasoning capabilities of Large Language Models (LLMs). In a first step, the LLM generates reasoning supporting the target categories for Task 1, which are sexism-related and not-sexism-related. In a second step, we exploit and take advantage of this valuable information and feed it to the LLM to reflect on this reasoning. The reasoning outputs are then used to extend the information provided by the tweets in fine tuning an XLM-RoBERTa model pre-trained on multilingual tweets.
EXIST shared tasks aim to detect and capture sexism-related content in social networks while identifying
intention and fine-grained topics [
For tweets, three primary tasks were established.
• Task 1: Binary classification to identify tweets with sexism-related content. • Task 2: Multi-class classification to identify the intention of the tweet.
• Task 3: Multi-label classification to categorize the types of sexism expressed.
Analogous tasks were introduced regarding meme images. Systems that address such shared tasks can be set in two contexts: a hard setting, in which systems aim to predict a hard conventional output category or set of categories, and a soft setting, in which systems are intended to provide probabilities instead. In the case of Task 1, the categories consist of sexist and non-sexist tweets. In this paper, we address Task 1 in the hard context.
The tweet dataset contains 10, 034 tweets in Spanish and English; 6920 for training, 1038 for development, and 2076 for testing. Each tweet was labeled by six annotators selected such that they had diferent demographic characteristics to minimize bias in the labeling. The age ranges and gender of the annotators belonged to the sets {18 − 22, 23 − 45, 46+} and { , }. All tweets were annotated such that there was one annotator from each of the six possible gender-age groups.
For Task 1, annotators were required to indicate whether the tweet was related to sexism or not. In the following, we will refer to these two categories as sexist and not sexist for simplicity purposes, since that’s the labeling convention used in EXIST. It is worth noting that the sexist category not only encompasses tweets with direct harmful messages but also tweets where sexism-related situations are being discussed or exposed. Therefore, this task is not only to detect direct hateful behavior against women.
Table 1 summarizes the train dataset partition according to language and label assigned in Task 1. The dataset is mostly balanced among languages and sexism-related and not-sexism-related categories. A minority, less than 13% of the tweets, do not have a majority class attached to them.
During EXIST 2023, the most commonly used approaches to address binary sexism classification included
the use of variations and combinations of the following three [
While classical methods and architectures remain popular and competitive, models like GPT, Llama,
or Gemini have not yet been deeply explored. In EXIST 2023, GPT-based large language model cascades
were shown to be competitive and ranked among the top systems. Notwithstanding the fact of being
efective, their strategy appears to be used in a classification setting rather than generating text [
Recent research on Large Language Models is focused on exploiting their “reasoning” capabilities.
An overview of the current state-of-the-art knowledge on reasoning in LLMs provides a review of
diferent techniques [
Given that fine-tuning massive generative LLMs is not eficient regarding computational resources,
prompt-based methods have gained popularity because of their knowledge capacities. Adding prompt
pieces in zero-shot scenarios such as “Think step by step” [
In few-shot scenarios, techniques like chains of thought have been shown to improve the answers by
demonstrating a thought process and encouraging the model to provide its own behind the answer [
All these strategies require a prompt-refining process, to some extent, to provide better context and enhance the use of these generative LLMs. Still, this refinement process is usually not automatic and is done through a highly qualitative assessment.
Another important consideration is the source and target languages used to prompt an LLM. Studies
have shown a disparity between the performance of LLMs in English and non-English languages, with
LLMs generally performing better in English [
We will focus on the data’s hard label predictions. Although we will not dismiss the individual annotators’
labels, we will filter the tweets so that only those classified as sexist and not sexist by majority vote will
be included. We preprocess the tweets using the library “pysentimiento” [
4. Emojis are replaced with their text descriptions.
As a baseline, we worked with a Twitter-specific multilingual language model that consists of an
XLM-RoBERTa architecture trained on multilingual tweets [
We have an experimental setup in two stages: In the first stage, we generate “reasoning” texts using an LLM that aims to understand the tweets’ nature. In the second stage, we use the generated texts to process them further with a pre-trained XLM-RoBERTa model. We explain these details in the following section.
We relied on an autoregressive LLM to generate text analysis of a tweet. Rather than asking the model to assign a label and provide an explanation in one straightforward step, we created rationales that support the tweet’s target categories, sexist and not sexist. Then, we let the model compare both arguments and choose the most accurate. We hypothesize that this is better than the direct closed question “Is this tweet sexist?” because, despite the validity of the reasonings for each category, the model internally evaluates the correctness of the statements produced. The setting is the same for both tweets in English and Spanish, and all the generated analyses are in English.
The first two steps occur independently: 1. Positive Reasoning. Generating analysis that supported the idea that the tweet was related to sexism. 2. Negative Reasoning. Generating analysis that supported the idea that the tweet was not related to sexism.
In a further step, the LLM is asked to “reflect” on the opposite texts it produced: 3. Stacked Comparison Reasoning. The model is fed with the information generated in the Positive and Negative Reasonings and has the chance to compare them.
This process is illustrated in Figure 1. The LLM generates three rationales that provide insight into the nature of the tweet. We call all the rationales to be Tweet Reasonings (marked as purple boxes in Figure 1), and they will be further used in the next section.
As an experimental setting, we ask the model to answer as a gender equality specialist, to respond
as unbiased as possible or to be concise. The exact prompts and an output example can be found in
Appendix A. The LLM selected for this reasoning stage was Llama 3, an open-source auto-regressive
language model developed by Meta AI [
In a conclusive step, we ask the model to provide a label synthesizing the reflection made in the
our proposed methodology, but we use it to assess the method’s gain in performance. The answers provided by this method are reported as B-StackLlama. Even though this workflow encourages the model to produce better-structured answers, the final classification is still not accurate enough. The second stage, which will be explained in the next section, addresses this issue. 5.1.1. Fine tune of XLM-RoBERTa In this stage, we test the reasoning provided by the LLM in the previous step as a supplement to the tweets. To enhance the XLM-RoBERTa baseline described before, we experimented with feeding the tweet concatenated with the reasonings generated. Due to the maximum input token restriction, we are limited to incorporating one reasoning at a time.
We fine tune three diferent XLM-RoBERTa models using diferent inputs. The input of each model consists of the concatenation using the special separator token [SEP] of a tweet with one of its corresponding Tweet Reasonings generated in the previous stage. The diferent reasonings Positive, Negative or Stacked Comparison Reasoning yield a diferent fine-tuned model, which we will call P-LLM-R-Stack-Ra,
Figure 2 shows the process realized. In addition to the individual {P, N, C}-X-LLM-R-Stack-Ra models, we create ensembles of individual models aiming to capture a more accurate label. Ensembles are generated by aggregating the scores that individual systems assign to the labels. The final label is designated as the one with the highest score.
A more general task can be addressed as an aggregate of fine-grained tasks. In particular, there
are questionnaire-based retrieval models used to provide a final response diagnosis, like the case of
depression [
Using this idea, in addition to the previous approach, we created a list of binary and close-option questions meant to identify the nature of the tweet. The questionnaire focused on identifying ofensiveness, intention, and whether fine-grained sexism-related topics were expressed directly or passively. The complete questionnaire is included in Appendix B. The answers to these questions were fed through a multi-layer feed-forward neural network that aimed to predict all three tweet-related tasks. Because this method did not outperform the scores obtained by the previous method on the validation set by itself, we used it only to enhance the previous techniques in an ensemble setting in the fashion described before. Reference for this method is Q-Llama-MLP.
Although this method seems promising, the quality of the answers is sensible to question formulation. Appendix B shows an example of the refinement through a qualitative assessment of a question that provides more accurate results in its final versions. We believe that all the questions in the original questionnaire can be refined into versions that lead to more accurate and representative answers and can be further used both to address the binary task and to identify fine-grained topics.
In this section, we discuss the performance of our proposals over the validation set and the oficial evaluation metrics at EXIST 2024.
This section is dedicated to evaluating the efectiveness and the impact of the key components of the methods proposed in the previous sections. We include the system Ensemble , which is the ensemble merging the two best performing individual systems N-LLM-R-Stack-Ra and C-LLM-R-Stack-Ra. In extension to this ensemble, we also include Ensemble, which in addition to individual systems
described before, the result of equally aggregating the scores of the individual scores of each category to obtain the ensemble scores.
We present in Table 2 the F1 of the positive class for the diferent systems proposed considering both tweets in English and Spanish of the validation set. As observed, models {N, C, P}-LLM-R-Stack-Ra outperform the baseline XLM-RoBERTa-Baseline. This demonstrates the benefit of incorporating the reasoning against using only tweets. The Negative and Stacked Comparison Reasonings incorporation perform slightly better than the Positive Reasonings. We also observe that B-StackLlama underperforms XLM-RoBERTa-Baseline, indicating the deficiency of relying only on the reasonings. Regarding Q-LlamaMLP, it does not beat XLM-RoBERTa-Baseline, partly due to the redundancy and inability to capture the questions’ nature and meaning. Despite that, Q-Llama-MLP enhances the ensemble’s performance slightly. The best performance overall belongs to Ensemble.
During the prompting process of B-StackLkama, the binary final answer underestimates sexismrelated tweets. The performance by itself is poor because the two-step process is biased to produce a final negative label. We additionally observed the length of the Tweet Reasonings also influences the quality of the response: too short, and the analysis might not have enough details; too long, and the answer might be repetitive, afecting the performance during the nfie-tuning process of {N, C, P}-LLM-R-Stack-Ra. We set up the reasoning generation to contain at most 200 tokens of the Positive and
showing the variation of the answer based on the token length is also included in Appendix A
The oficial metrics for EXIST 2024 are ICM-hard and F1 of the positive class, and scores are divided by
language [
The single system N-LLMRStack-Ra performs less efectively than Ensemble . We hypothesize that the Negative and Stacked Comparison Reasonings complement the tweets with diferent information during the fine-tuning process. In the case of the Negative Reasoning, as provided for all tweets, we expect to learn an internal diferentiation between accurate and inaccurate supporting facts of the not-sexist category. In the Comparison Stacked Reasoning, we expect to capture the contrast between reasonings and correct the preference of one over the other if necessary. We think that the distinct aspects and relationships these reasonings capture contribute to the improved performance in Ensemble .
The ensemble Ensemble is the best performer of our systems. In particular, the diference between the top score with this system considering the set of evaluation tweets in English and Spanish is less than 0.01 in the ICM-Hard Norm and the F1 of the positive class, which shows the competitivity of our method. Performance of this model is just slightly better than Ensemble , and we believe that even though Q-Llama-MLP did not outperform the individual systems in validation, adding the output in Ensemble provides a slight correction of the underestimation of sexist tweets.
Performance over the Spanish tweets is slightly better than performance over English tweets, and we conjecture that the multilingual setting benefits performance for both English and Spanish languages.
In this paper, we propose a methodology to detect sexism that takes advantage of the capabilities to generate rationales of LLMs, which was not used in EXIST’s previous editions. These rationales show the potential of the LLM to support the binary classes and the capabilities to compare two diferent reasoning processes. As shown in this paper, the LLM reflection alone is not enough to achieve competitive results, and it is biased to underestimate sexist tweets, which is not desirable. That is why using the rationales enhances the baseline results provided by an XLM-RoBERTa that is fine-tuned only using the tweets. It allows for information that explores the nature of the tweet and a more accurate classification. Results show that the proposed models are competitive and open the panorama of how internal knowledge and reasoning capabilities of autoregressive LLMs can address this task.
In future work, we plan to extend this approach to address other tasks, refine the prompts to obtain better rationales, and study the capabilities and limitations of other autoregressive LLMs. Questionnaire results can be explored to dive into the fine-grained classifications of sexism and analyze source intention and topic classification. A more in-depth exploration regarding the length of the rationales generated, the efects of prompting variation, alternative LLMs, and bias remains to be explored. This method also shows insight into the biases encoded in the large language models, as the model we chose (Llama 3) is misled to provide inaccurate explanations of the wrong category classification and, as observed in the scores, fails to choose the correct classification label by itself.
It is important to note that the systems developed in this work predict binary labels based on annotators’ majority vote and might overlook people’s perceptions at an individual level. Another important distinction is label names, where instead of “sexist” and “not-sexist” they could be more accurately described as “sexism-related” and “non-sexism-related” to recognize two diferent perspectives: negative intention (difusion of biased content) and positive intention that treats sensitive content (bias awareness and discussion).
The reasoning generated by the LLMs put into evidence the biased internal views they provide, where sexism-related content is underestimated. Using these labels by themselves or trusting the rationales generated should be considered carefully, as they can be highly misleading. Given the implications of deploying detection systems for sensitive content, the proposed solution requires a more in-depth analysis by social scientists, and ethics and fairness experts. Misusing these systems might have significant implications, including the potential non-detection of toxic and dangerous content or the unintended censorship of discussions about social problems.
Villarreal-Haro acknowledges CONAHCYT for its support provided by the program Becas Nacionales Para Estudios de Posgrados (CVU 1309535). We thank CONAHCYT for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies and CIMAT Bajio Super-computing Laboratory (#300832). Sanchez-Vega acknowledges CONAHCYT for its support through the program “Investigadoras e Investigadores por México” (Project ID.11989, No.1311). Rosales-Perez acknowledges CONAHCYT for its support through the project grant Búsqueda de arquitecturas neuronales eficientes y efectivas (CBF2023-2024-2797).
In the experimental templates, we tried to set an objective environment for the answer given by the system role. The exact prompts used for the task are the following:
The first two prompts take as input {tweet}, and generate the Positive and Negative Reasonings are the following:
Positive Prompt.
Negative Prompt.
Positive and Negative Reasonings are limited to a maximum of 200 generated tokens. In a further step, the tweet and both the Positive and Negative Reasonings are provided so that the model self-evaluates both.
Comparison Prompt.
The Comparison Stacked Reasoning is limited to a maximum of 250 tokens.
We show an example of the LLM output to illustrate the type of text reasoning we generated.
The original tweet is the following:
The positive and negative reasoning generated are:
Positive Reasoning: Let’s break down the tweet and identify the elements that suggest sexism: 1. The tweet starts by mentioning the author is writing a university essay in a pub, which is a common scenario. This sets the context for the rest of the tweet. 2. The author mentions that a "random old man" is asking them "drunk questions" while they’re trying to concentrate. This already has a slightly confrontational tone, as the author is implying that the man is interrupting and being a distraction.
specifically suggests sexism. The man’s comment is making an assumption about the author’s future plans and goals, which are likely to be academic in nature. The comment implies that the author’s ultimate goal is to get married and abandon their academic pursuits, which is a stereotype often perpetuated about women.
Negative Reasoning: The tweet does not contain elements of sexism because the content is focused on the author’s experience of being interrupted by an old man while trying to write an essay, rather than any specific gender-based discrimination or harassment. The tweet is more about the author’s frustration with the distraction and the old man’s opinion about the author’s future plans, which is not related to the author’s gender. While the author uses the hashtag #everydaysexism, which is often used to describe the subtle and pervasive forms of sexism that people face in their daily lives, the tweet itself does not contain any explicit sexist language, attitudes, or behaviors. The tweet is more about the author’s annoyance with the old man’s behavior and the disruption it caused, rather than any gender-based issue. In fact, the tweet could be interpreted as an example of ageism, as the old man’s comment about the author’s future plans is not specific to the author’s gender, but rather a stereotypical comment.
The comparison reasoning where the model self-evaluates the outputs produced is the following: Stacked Comparison Reasoning: Analysis 1 is the most accurate. The tweet contains elements of sexism because: 1. The old man’s comment "good luck, but you’ll just end up getting married and not use it anyway" is a stereotypical and gendered assumption about the author’s future plans and goals, which are likely to be academic in nature. The comment implies that the author’s ultimate goal is to get married and abandon their academic pursuits, which is a stereotype often perpetuated about women.
incorrect because it does not recognize the gendered nature of the old man’s comment and the impact it has on the author. Analysis 2 is also incorrect because it does not identify the specific elements of sexism in the tweet, instead, it focuses on broader issues of sexism and gender inequality. Analysis 1 accurately identifies the elements of sexism in the tweet, including the stereotypical and gendered assumption made by the old man.
It is essential to remark on some things. First, the model’s answer quality varies with length. If we ask the model to generate positive reasoning with a smaller amount of tokens, the answer does not provide as many insights that support the target category. The following is an example of a positive reasoning limited to less than 50 tokens.
Positive Reasoning:
As we can observe, limiting the reasoning to fewer tokens in the rationale prevents the model from fully breaking down the elements that it identifies as relevant. Allowing the model to produce a much longer output will result in repetition of the rationale generating.
Secondly, due to the safety settings of the model, in certain topics, the generated answer is as follows: Positive Reasoning:
In the second step of our methodology, we expect that even though the model does not provide a rationale supporting the targeted position, the model is learning that this type of analysis also provides information about the nature of the tweet and helps to learn about the topics contained.
Finally, these are all automatic reasonings generated, and as shown in the example, are internally capable of supporting both sexist and not-sexist categories. We noticed that this two-step scheme is biased to predict as a final answer negative results. So, these models and the extracted knowledge shall be used carefully.
As we discussed before, refining the questions impact the accuracy of the answers. Table 4 shows an example of the qualitative analysis of the evolution of a question if asked to explain the answer provided.
The complete list of fine-grained sexist questions is the following. The model was asked to answer with “Yes/No” for binary questions and only with the answer identificator for closed-option questions.