In this article we describe the LabTL-INAOE participation in the MiSonGyny 2025 shared task. We present a method for detecting misogyny in song lyrics by a two-step classification framework based on a confidence-based partitioning strategy. In the first step, a base classifier is used to assign labels to all instances. These labels are then evaluated in order to assign them a confidence score. Based on this score, the instances are partitioned into two groups: trustworthy and untrustworthy. In the second step, the untrustworthy instances are relabeled using a more sophisticated model, in this case, a large language model with prompting. The proposed approach was evaluated in the two subtasks comprised in MiSonGyny 2025, yielding competitive results. Beyond quantitative performance, the method enhances explainability by providing confidence information at the instance level, making it especially useful for content moderation and other contexts where explainability and human oversight are essential.
Misogyny is defined as hateful behavior toward women, manifested in violent and cruel acts against
them for the fact of being women [
This situation is fueled by the belief in the supposed inferiority of women and the overvaluation of
male dominance, the latter being reinforced by factors such as traditionalism, family environment, and
social media [
There are studies that focus on analyzing specific genres to study the phenomenon of misogyny, for
example, in genres such as hip-hop, rap, reggaeton, among other musical genres. [
This paper describes our participation in the MiSonGyny 2025 shared task; we developed a two-step classification method for detecting misogyny in song lyrics. In the first step, a base classifier is used to assign labels to all instances. These labels are then evaluated by a confidence-based partitioning strategy that computes a confidence score. Based on this score, the instances are divided into two groups: trustworthy and untrustworthy. In the second step, the untrustworthy instances are relabeled using a more sophisticated model, in this case, a large language model (LLM) with prompting.
This strategy is inspired by the saying “diferent strokes for diferent folks” . Some songs express misogyny explicitly, such as "Run for Your Life" by The Beatles: “Well, I’d rather see you dead, little girl Than to be with another man”. Other lyrics require deeper interpretation, such as "Buenos días, amor" by José José: “Me perdí en tu vientre cuando aún dormías... Sé que estabas enfadada, pero no dijiste nada, El que calla otorga y sé que estás enamorada” (I was lost in your womb while you were still sleeping...I know you were angry, but you didn’t say anything. Silence gives consent, and I know you’re in love.). This excerpt implies a non-consensual sexual situation, though in a subtle manner. Finally, some lyrics require understanding of cultural and temporal context, such as "There Goes My Everything" by Elvis Presley: “There goes my only possession, Oh, there goes my everything” Although presented as a romantic sentiment, the idea of "possession" may be interpreted as inherently misogynistic.
As these examples show, each instance presents unique challenges in accurately detecting misogynistic content. Some can be handled by standard classifiers, while others demand more advanced models or even human judgment. Our proposed system adapts the analysis strategy according to the perceived dificulty of each instance.
This paper is organized as follows. In Section 2, we briefly introduce an overview of shared tasks related to identifying misogyny and hate speech in songs. In Section 3, we describe the experimental settings and the obtained results during the developing phase. In Section 4, we present the oficial results obtained in MiSonGyny 2025 shared task. Finally, in Section 5, we conclude the paper.
The identification of misogyny is part of the phenomena covered by hate speech, which is defined
as a conscious and deliberate public statement intended to denigrate a group of people based on
characteristics such as race, color, ethnicity, gender, sexual orientation, nationality, religion or political
afiliation [
Detecting hate speech is very challenging since it takes many forms in social media: it can be
manifested verbally, non-verbally, and symbolically [
Within the field of NLP, several shared tasks have been proposed that provide labeled data and promote
the development of proposals to solve diferent problems. To the present day, some tasks focused on
the detection of misogyny or similar problems have been presented. Such as "Automatic Misogyny
Identification" (AMI) in Evalita 2018 and Evalita 2020, focused on the identification of misogyny, the
categorization of misogynistic behavior, and the classification of targets in tweets in Italian and English
[
Regarding tasks related to the detection of hate speech in songs, a subtask of HOMO-MEX 2024 was presented: Hate Speech Detection Towards the Mexican Spanish-speaking LGBT+ Population, where the HOMOLYRICS corpus was presented, composed of Spanish song lyrics that may or may not contain LGBT+phobic text. For this task, the main proposals were based on the use of traditional machine learning methods such as the Naive Bayes classifier and SVM classifier, others were focused on the use of prompting with LLMs such as Falcon or Llama 2, and the majority focused on the use of Transformers models and Transformers ensembles such as BERT, BETO, XLM RoBERTa, RoBerta, BERTweet, DistilBER, and mDeBERTa [18].
In this work, we propose a two-step classification framework based on a confidence-based partitioning strategy. The main idea is to initially classify each test instance using a base model, selected for its simplicity or eficiency. Then, instead of accepting the predicted label as definitive, the model evaluates the confidence of that classification.
The confidence score is designed to reflect the attraction of a given instance to the most representative examples of the classes, both, the predicted class and the opposing class. If an instance is strongly attracted to the core examples of its predicted class and only weakly attracted to those of the opposite class, then the confidence in the prediction should be considered high.
To compute this score, we adopt a strategy inspired by the K-Strongest Strengths (KSS) algorithm [19]. This approach models the idea of attraction forces between texts, where each instance exerts a force on another based on their semantic similarity and relevance (masses). The mass of an example is derived from its importance within the training data; in this case, we use the cumulative frequency of class-relevant words. By combining these elements, we compute the average attraction force exerted on the instance by both the predicted class and the opposing class.
The resulting confidence score is obtained by calculating the ratio between the average attraction force from the predicted class and that from the opposing class. This ratio determines whether an instance is trustworthy (i.e., classified with confidence) or untrustworthy (i.e., uncertain or subtle). Dificult instances are passed to a more sophisticated and costly model (e.g., a GPT-based classifier via prompting) for relabeling.
Our method consists of four steps, which are illustrated in Figure 1. This framework is particularly well-suited for subjective classification tasks, where the boundaries between classes are not always clearly defined, such as the detection of misogyny in song lyrics.
1. Songs Data: The data includes song phrases of varying lengths, from a variety of genres, and covering a wide range of topics. They are divided into training and test sets for both tasks. 2. Preprocessing: Instances are standardized by converting all text to lowercase, removing line breaks, punctuation marks, and structural indicators such as choir, verse, etc. 3. First classifier: The first model used to assign labels is RoBERTuito, which was fine-tuned for this task. From this model, we obtain both the predicted labels to be later evaluated, and the
embeddings, which serve as the base representation of the song lyrics and are used to compute the confidence score. [20]. 4. Confidence score: This process is inspired by the kSS algorithm (k-strongest strengths classification algorithm), which draws an analogy with Newton’s Law of Universal Gravitation. Unlike traditional approaches that rely solely on the labels of neighboring instances, kSS uses the gravitational forces exerted by training instances to assess their influence [19].
Following this principle, we construct the attraction forces for each test instance. Using the word embeddings from the Transformer model, we calculate the cosine distances between the training and test objects.
To assign a mass to each training instance, we rely on the relevance of specific n-grams for each class. These n-grams are extracted using the SS3 algorithm [21], which identifies the features that contribute the most to distinguishing between classes. For each class, we construct a list of the most informative n-grams. Then, for every training instance, we count the frequency of the n-grams from the list corresponding to its predicted class. The resulting count determines the instance’s mass: the higher the frequency of class-relevant n-grams it contains, the greater its mass. Finally, the attraction force it exerts on a test instance is computed using the relationship between the previously calculated mass and the cosine distance, as defined in Formula 1. (1) (2) strength(, ) =
() dist2(, ) For each class, the 10 strongest forces acting on the test instance are selected to calculate the confidence score. we consider the predicted label assigned by the base model and the distribution of forces. As shown in Formula 2, we calculate the average force from the predicted class and divide it by the average force from the opposing class(es). The result is a single value representing the confidence score of the instance. Instances with scores above a given threshold (typically between 1.0 and 1.5) are considered trustworthy and retain their assigned label. Those with lower scores are marked as untrustworthy, as they exhibit traits associated with multiple classes, making them challenging to classify automatically.
Confidence Score = pred opp 5. Second classifier: Instances deemed untrustworthy (i.e., with low confidence scores) are passed to a large language model (LLM), specifically GPT-4o-mini [ 22], for re-labeling via prompting (the following section shows the prompts used). The final labels for this subset are those produced by the LLM.
For training purposes, task organizers provided a dataset for each subtask: • Task 1: It has a total of 2104 training data, divided into 642 instances for the misogyny class, and 1462 instances for the non-misogyny class. The test data has a total of 527 instances. • Task 2: It has a total of 1168 training instances, divided into 435 instances marked as Sexualization(s), 129 instances marked as Violence (V), 78 instances marked as Hate (H), and 526 instances marked as Not Related (NR). The test data has a total of 293 instances.
Based on the results obtained during system validation, the proposed method was applied to the test partition, separating the instances according to their confidence level. The results of this separation for each task are described below: • Task 1:The method identified 102 instances (19.35%) as untrustworthy, out of a total of 527. These were labeled using prompting with the model GPT-4o-mini, using a specific prompt designed for this task.
"Eres un modelo de clasificación de canciones misóginas que asigna una categoría a cada oración, esto tomando en cuenta que la misoginia se define como el odio o prejuicio contra las mujeres, puede manifestarse lingüísticamente de numerosas maneras, incluida la exclusión social, la discriminación, la hostilidad, las amenazas de violencia y la cosificación sexual. Las clases posibles para clasificar son: ’Misogino’, ’No Misogino’. Responde solo con la etiqueta exacta, 0 para No Misogino, 1 para Misogino." "You are a misogynistic song classification model that assigns a category to each sentence. This is based on the definition of misogyny as hatred or prejudice against women, which can be linguistically expressed in numerous ways, including social exclusion, discrimination, hostility, threats of violence, and sexual objectification. The possible classes for classification are: ’Misogynistic’, ’Not Misogynistic’. Respond only with the exact label: 0 for Not Misogynistic, 1 for Misogynistic." • Task 2: 58 instances (19.80%) were detected as untrustworthy, out of a total of 239, also labeled with GPT-4o-mini using the same approach. The prompt used is shown below: "Eres un modelo de clasificación de canciones misóginas que asigna una categoría a cada oración, esto tomando en cuenta que la misoginia se define como el odio o prejuicio contra las mujeres, puede manifestarse lingüísticamente de numerosas maneras, incluida la exclusión social, la discriminación, la hostilidad, las amenazas de violencia y la cosificación sexual. Sexualización (S) : Frases que describen o sugieren actos sexuales, lenguaje sexual o insinuaciones. Violencia (V) : Frases que se refieren a agresiones físicas o verbales, amenazas o acciones violentas. Odio (H) : Frases que contienen lenguaje ofensivo o discriminatorio, expresiones de desprecio u hostilidad hacia un grupo o individuo. No relacionado (NR) : Frases que no entran en las categorías anteriores y carecen de contenido sexual, violento o de odio.. Responde solo con la etiqueta exacta, S, V, H o NR." "You are a misogynistic song classification model that assigns a category to each sentence, taking into account that misogyny is defined as hatred or prejudice against women, it can manifest itself linguistically in numerous ways, including social exclusion, discrimination, hostility, threats of violence and sexual objectification. Sexualization (S): Phrases that describe or suggest sexual acts, sexual language or innuendos. Violence (V): Phrases that refer to physical or verbal aggression, threats or violent actions. Hate (H): Phrases that contain ofensive or discriminatory language, expressions of contempt or hostility towards a group or individual. Unrelated (NR): Phrases that do not fall into the previous categories and lack sexual, violent or hateful content. Answer only with the exact label, S, V, H or NR." For both tasks, the final labels were generated by joining the preditions on the two subsets, that is: • The labels obtained with the fine-tuned RoBERTuito model for the easy instances. • The labels generated by prompting with GPT-4o-mini for the hard instances.
The results obtained are summarized in Table 2. As can be seen, the performance values remain practically unchanged compared to the use of the base model, indicating that the separation strategy does not significantly improve the classification. While our method achieves a slight improvement in Task 1, there is no diference in Task 2. This is mainly due to the fact that Task 2 poses additional challenges, as it is a multi-class problem with some classes having very few instances, which limits their representativeness and reduces the efectiveness of the proposed approach.
Although the proposed approach does not lead to a substantial increase in quantitative performance metrics, it provides significant qualitative advantages. In particular, it enhances the explainability of the classification process and ofers valuable support for human decision-making. This is especially relevant in sensitive applications such as content moderation, where the ability to understand and justify predictions is as important as their accuracy. Rather than aiming solely for metric improvement, the method focuses on enriching the interpretability of results, enabling more informed and transparent decisions. 4.3.1. Highlights of the method • Lists of relevant n-grams: For each class, the system generates a set of representative n-grams that are used to calculate the masses within the attraction model. Table 3 shows the most significant n-grams that characterize each category.
Classes Uni-grams Bi-grams Misogynist bmaubeyv,em,saemxoi,,fploawrt,yv,ammuoje,reensc,ipmearr,ac,apmearr,ecou,loto,cnaorvtieo., spiaqlucaierraej,od.addt yankee, j balvin, tbbwaoebyryfkr,iimenngod,m,tommuoyc,vhefil,nosgweyx,,oleputar’ssretgylfo,.,woonmteonp,, bbietdch,,ass, Iffucykouof.want, daddy yankee, j balvin, Not alma, vivir, dolor, jamás, luz, fin, adiós, triste, amar, tantas cosas, pido perdón, ser feliz, Misogynist soledad, llorar, silencio, sufrir, cariño, morir. si pudiera, solo queda, tal vez.
lsoovuel,, lloivnee,lpinaeinss,,ncervye,rs,illiegnhcte,,esnudfe,r,gaofeocdtbioyne,, dsaied., taSololbmtehaahntayrpetpmhyia,niignfssI, icIso,aumplodal,yobgeiz.e, • Trust ranking per instance: Each song is given a confidence score that allows it to be ranked from most to least reliable. This ranking is useful for prioritizing subtle cases or candidates for human review. The system automatically identifies a small subset of sentences (around 20 %) that contain subtle forms of sexist language, which can be reviewed with more sophisticated tools or by human moderators.
To demonstrate the usefulness of the method, below we show concrete examples that were marked as trustworthy or untrustworthy. Table 4 presents examples of sentences classified as trustworthy. The misogynistic examples include clear expressions of sexualization and objectification. In contrast, sentences classified as not misogynistic, although they may contain afective or animated language, do not have evidence of hateful or prejudiced content.
On the other hand, Table 5 shows examples of sentences labeled as misogynistic by GPT-4o-mini, initially considered implicit or untrustworthy by the model. In the first case, metaphors of physical violence are detected. In the second, there is a subtle allusion to female sexual freedom, which can be interpreted in diferent ways, some may attribute and even explain the use of the adjective to the person’s infidelity, whereas others may notice a clear misogynistic expression, which shows that many of these examples end up being so subtle that automatic methods could not perfectly identify them. These situations reflect the complexity of language and the need for more refined analysis mechanisms or human intervention.
The proposed system introduces a strategy for performing a confidence assessment that allows instances to be divided into trustworthy and untrustworthy subsets. This division allows eficient models like RoBERTuito to be applied to easy instances and more powerful models like GPT-4o-mini to be reserved for the hard instances that require a more in-depth analysis. Although the quantitative results do not show a significant improvement in classification metrics, the added value of the system lies in its ability to explain the decision-making process, identify challenges and provide useful tools for content moderation.
The system’s main contributions include: (i) the identification of relevant n-grams for each class, (ii) the calculation of an interpretable confidence score for each instance, and (iii) the ability to generate a reduced subset of implicit examples as candidates for human review. These features make the system not only function as a classifier, but also as a tool for analyzing and interpreting textual content, particularly valuable in sensitive contexts such as detecting misogyny in song lyrics.
In summary, this work contributes a hybrid approach that prioritizes explainability and eficiency in the use of computational resources, opening new possibilities for the design of moderation systems that are fairer, more understandable, and adaptable to complex social contexts.
Generative AI tools were used solely within the proposed classification approach to assist in the processing of certain instances. Grammarly was employed to suggest minor grammatical and stylistic corrections. No generative AI systems were used to write or edit the content of this manuscript.