Gender bias and discrimination against women are still manifested in popular culture through various types of discourse present in social media, TV shows, and music. In particular, song lyrics often express misogyny in explicit ways (e.g., objectification of women's bodies, violent threats), but also in covert forms masked as romantic expression. This paper reports our participation in the MiSonGyny 2025 shared task, which involves both the detection and fine-grained classification of misogynistic content in lyrics. Our experiments explore the efectiveness of BERT-like models and large language models (LLMs) for these tasks.
The spread of misogynistic discourse remains a persistent issue contributing to gender inequality.
Harmful content towards women in social media platforms and other ways of communication implies a
critical scenario that promotes gender inequality. It also contributes “to an environment where women
feel unsafe, targeted or silenced”[
We can consider misogyny within the concept of hate speech (HS) understood as “any communication
denigrating a person or group, judging by specific features such as color, ethnicity, gender, sexual
orientation, nationality, and religion” [
This scenario presents a challenge for research studies trying to understand the semantic patterns
hidden beneath a wide range of written and visual features in discourses such as social media platforms,
news articles or TV shows. For instance, even song lyrics are contaminated with ideologically-biased
ideas based on misogynistic attitudes, and we believe that it is crucial to address this concern, given the
influential role music plays in shaping societal perspectives. The powerful influence of music is raised
in Davies [
In this regard, an interesting work by Betti et al. [
This scenario confirms that the exploration of misogyny in song lyrics has opened new avenues for multidisciplinary research, and it is a call to action highlighting the need for models attuned to the intricacies of lyrical language. Continued eforts in dataset development, bias mitigation, and model refinement are essential to further progress in this critical area of study.
The following work aims to investigate this gap by participating in the MySonGyny shared task
[
In recent years, the proliferation of HS on digital platforms has sparked significant interest within the
NLP community. Numerous resources, shared tasks, and benchmark datasets [
Misogyny, as a specific form of HS, has received particular attention. Existing studies have tackled
this issue primarily through the textual analysis of comments extracted from social media platforms,
especially X [
Despite these advances, most approaches still rely heavily on social media data, which limits the
generalization of models to other forms of communication. Creative and artistic texts, such as song
lyrics, present unique challenges for the automatic detection of HS. Although some recent work has
addressed this issue [
From a methodological standpoint, recent advances in Deep Learning (DL) have significantly shaped
the development of HS detection systems, with Transformer-based architectures [
The MiSonGyny 2025 shared task [
Although they might address themes related to women, they can do so without perpetuating stereotypes or negative attitudes. 2. Fine-grained Misogyny Speech Detection. A hierarchical multi-class classification task aimed at detecting the type of speech present in a song. The categories can be described as follows: • Sexualization (S): Phrases which describe or suggest sexual acts, sexual language, or insinuations. • Violence (V): Refers to lyrics containing physical or verbal aggressions, threats or violent actions. • Hate (H): Lyrics which contain ofensive or discriminatory language, expressions of contempt, or hostility towards a women or a group of women. • Not Related (NR): Phrases that do not fall into the above categories and lack sexual, violent, or hateful content.
For the evaluation of the proposed systems across both sub-tasks, standard classification metrics
such as Precision (Pr), Recall (Rc) and F1-score were proposed by the organizers, with F1-score being
the selected metric for determining the ranking on the oficial test set [
Table 1 summarizes the metrics used for the evaluation of each sub-task, indicating the oficial metrics used for ranking.
This section describes the dataset used in the MiSonGyny 2025 shared task [
On the one hand, Figure 2a presents the data distribution for the first subtask. Although we are provided with approximately 2300 samples, it is important to note on the class imbalance present for this sub-task, since the non-misogynistic songs make up almost 70% of the dataset.
On the other hand, Figure 2b presents the data distribution for the second subtask. Since this subtask is a hierarchical one, all the samples on this subtask are directly extracted from the previous. Likewise, the most common class is the one related to the absence of misogyny (i.e., NR for this subtask). By examining the categories related to the presence of hate speech, it can be observed that the sexualization class is the most common among them with 435 samples, whereas the violence and hate classes only make up less than 20% of the overall dataset, presenting a very skewed distribution. (a) Misogyny Speech Detection sub-task. (b) Fine-grained Misogyny Speech Detection subtask.
Song lyrics possess a unique structure, driven by their domain, that requires specific NLP considerations. For instance, they frequently feature pervasive repetition, like recurring choruses and verses, which, while integral to musical composition, ofers little to no information for the given tasks.
Beyond repetition, lyrics often contain elements irrelevant to their core message. This includes ad-libs (often sounds rather than coherent sentences) and song delimiters such as [Intro] or [Chorus]. As shown in Table 2, these elements, coupled with the initial song length, frequently push them beyond common token limits like BERT’s 512-token maximum.
Believing these structural and superfluous elements do not significantly influence hate speech presence, we preprocessed our data by removing ad-libs, song section delimiters, blank lines, and repeated verses and sections. As Table 2 reflects, this step allows approximately 75% of the songs to ift within the BERT model’s token limit, enabling more eficient and relevant analysis focused on the unique and semantically relevant lyrical content.
The proposed system architecture is based on fine-tuning a pre-trained RoBERTa model [
As we pointed out in Section 3.2, one of the problems we can anticipate for both sub-tasks is the lack of
data for some categories. In order to address this problem, we have decided to employ data augmentation
techniques using LLMs to generate new data for the misogynistic lyrics. More specifically, we have
explored data augmentation by paraphrasing [
Rather than direct paraphrasing, we explicitly prompt the LLM with a definition of the target finegrained misogyny category and instruct it to first identify misogynistic verses related to that category. Subsequently, the model generates three distinct paraphrases, with each version adhering to specific rules: the first paraphrases all content while maintaining the core misogynistic theme, the second replaces non-misogynistic verses with new content while only paraphrasing the misogynistic elements, and the third reorganizes and paraphrases the entire song while preserving narrative coherence and the defined misogynistic essence. Prior to generation, the LLM is also tasked with explaining and justifying its intended changes and how the new versions retain the original misogynistic nature, encouraging the model to take intermediate steps before generating any data. Our paraphrase prompt can be found in Appendix 8.1.
On the other hand, for synthetic data creation, our intention was to generate entirely new song lyrics from an existing one, critically ensuring the preservation of the original’s misogynistic elements in the new composition. Prior to generating these new lyrics, we instruct the model to extract crucial information: the song’s main themes, specific misogynistic verses, its genre, and its overall mood. Subsequently, the model is tasked with creating a new song that retains the same mood and themes but belongs to a diferent music genre. Our synthetic data generation prompt can be found in Appendix 8.2.
A primary consideration when designing our classification prompts is that it is not necessary to create a separate prompt for each task. As highlighted in Section 3.2, the second sub-task is dependent on the first. Consequently, successfully addressing the second sub-task enables us to infer the categories for the initial sub-task, which minimizes both employed time and economic costs if we were to use a provider API.
Furthermore, our classification prompts adhere to the conventional zero-shot and one/three-shot
schemes for few-shot learning. To enhance the LLM performance, we incorporated CoT reasoning by
instructing the model to meticulously analyse song lyrics and formulate its responses through a
stepby-step process [
Instead of randomly selecting the samples for our in-context learning prompts, we have utilized
a model which combines both automatic extraction as well as manual revision. To do so, we first
calculated embeddings for all song lyrics with a multilingual embedding model 3 [
Once embeddings were computed, our goal was to select both representative and ambiguous examples for each class. We achieved this by calculating each song lyric’s distance to the centroid of every class. Representative samples were chosen based on their proximity to their own class centroid, providing the LLM with clear examples of what to look for. Conversely, ambiguous samples were picked if they were closer to another class’s centroid, exposing the LLM to "dificult" examples.
Given its reliance on semantic similarity, a linguist manually evaluated the top-K representative and ambiguous samples extracted by the algorithm. Thus, the selected samples are guaranteed to represent the phenomena of each of the categories from the second sub-task.
Our 0-shot classification prompt can be found in Appendix ??.
For each task, we created our private training and validation datasets by splitting the original training dataset provided by the organizers, using an 80/20 ratio while stratifying the split to ensure proportionate representation of classes. Subsequently, when applying the data augmentation techniques mentioned on Section 4.3.1, we experimented with various training configurations; for instance, some setups involved replacing all samples of the minority class with synthetic data, whereas others combined both the original misogynistic samples with the synthetic ones. Critically, data augmentation was exclusively applied to samples already present in our private training dataset, ensuring no data leakage into the private validation sets.
Our RoBERTa models were fine-tuned following several key recommendations presented in [
Models were trained with a batch size of 16 for a maximum of 10 epochs. To optimize training
duration and prevent overfitting, early stopping was implemented based on the validation loss, and the
model checkpoint achieving the lowest validation loss was selected. Input sequences were dynamically
padded to the maximum length within each batch to ensure maximum computation eficiency.
5.3. LLM
All experimentation involving LLMs was conducted using the OpenAI model GPT-4.1-mini [
Regarding more specific settings, we only tweaked the temperature of the model depending on the given task. On the one hand, classification tasks were run with a temperature of 0.5, promoting more consistent and deterministic outputs, yet still allowing the model suficient flexibility to provide nuanced reasoning, especially for misogyny detection. On the other hand, data augmentation was performed with a temperature of 0.7, thus enabling the model to explore more creative and diverse generations for the resulting data.
Prior to presenting the performance of the system, we evaluated the quality of data generated through augmentation techniques. The results, reflected in Table 3, demonstrate that all the paraphrased data, across all variations, ofer a high degree of semantic similarity to the original content, as shown by consistently high BERT-scores. However, the similarity across the three paraphrasing methods is also very close, suggesting that despite specifying rules for diversity, the model did not introduce as much variability as expected.
In contrast, the synthetically generated data stands out for its lower cosine similarity, indicating greater lexical and structural diversity. Despite this, its BERT-score remains relatively high (0.76), implying that the generated content still retains much of the original meaning.
Table 4 displays that our RoBERTa model trained on the original dataset consistently achieves the best overall performance across most metrics. When the original data is replaced with paraphrased versions (Par. {1,2,3}), a performance drop is observed in all cases, with the third paraphrasing variant showing the most pronounced decline. A similar trend is evident when training solely on synthetic data (Synthetic), which also underperforms compared to the original configuration. Furthermore, combining the original dataset with either paraphrased (Original + Par. {1,2,3}) or synthetic data (Original + Synthethic) does not yield improvements over the baseline. This suggests that the augmented samples may introduce noise or redundancy rather than helpful variability, hindering model performance.
Table 5 presents the results for the GPT-4.1-mini model across diferent prompt configurations. As anticipated, increasing the number of examples in the prompt leads the LLM model to yield results that more closely resemble those obtained by our RoBERTa-based approach, indicating the benefit of in-context learning. Note that the LLM results on classification do not incorporate the AUROC metric, since it relies on class probabilities in order to calculate its value.
Table 7 presents the results of the diferent dataset techniques utilized compared to the original dataset for the RoBERTa model. As can be seen, the results are very similar to the ones we observed earlier. (d) 3-shot with GPT-4.1-mini. In other words, replacing the original data by the generated from the LLM (Par. {1,2,3} and Synthetic) ofers a worse performance in comparison with the original baseline. However, a tiny performance improvement can be observed once we combine the generated data from the LLM alongside the original data.
Table 8 presents the results for the Fine-grained Misogyny Speech Detection using GPT-4.1-mini with diferent prompting strategies. As shown, the LLM consistently outperforms our RoBERTa system-based approach for all prompts, including 0-shot.
To further analyse the results, we examined the system outputs using Figure 5. For the RoBERTa model with the original dataset (Figure 5a), high recall is observed for NR and S, which suggests that the model is overfitting on these categories since they are the most predominant on this task. However, the model significantly struggles with hate and violence, showing a very low recall of these crucial categories, frequently misclassifying them as NR or S.
The LLM model’s confusion matrices (Figures 5b, 5c, and 5d) show a progressive improvement with more examples. While initially biased towards predicting NR (especially in 0-shot), recall for S consistently improves across all scenarios. H also sees a significant recall improvement from 0-shot to 3-shot. However, the V category remains challenging for the LLM, showing only marginal improvements in recall across settings. We will analyse specific outputs for the V category in the subsequent section to better understand these limitations.
For the oficial test set, we sent the predictions from the RoBERTa model trained with the original training dataset configuration as well as the LLM predictions using the diferent prompts. Table 9 reflects our results in the oficial test set as well as a comparison with the winner of the subtask. In addition, our test results prove that the LLM proposal ofers better results in the given task. Moreover, our best submission is the 3-shot prompt, which has achieved a third position in the overall ranking.
The concise results explained in the last Section 5.1, provide a rigorous context for a qualitative analysis that allows us to explore the GPT-4.1-mini data-augmentation behaviour. In this subsection, we explore the rationale of the LLM behind the generation of paraphrased samples. Table 10 shows an example of a paraphrased song.
In general, taking into account the conditions of the paraphrasing prompt, we observe three main issues. Firstly, there is a lack of creativity when the LLM tries linguistic paraphrasing strategies. The three versions of the paraphrase show poor writing innovation reflected in verses that are almost equal between versions. For instance, "No matter how far you are", "even if you are far", "even if you’re far away". The LLM only adds some synonyms or changes the verb tense keeping some of the verses almost identical to the original song. Secondly, we noticed that the length of the paraphrases is significantly shorter than the original (especially in the third paraphrase) as it can be observed in Table 10. Lastly, after exploring the generated samples, we noticed that the LLM does not apply the condition of the (d) 3-shot with GPT-4.1-mini. third paraphrase in the prompt: "[...] paraphrase all misogynistic and non-misogynistic verses of the song, but modify the structure of the song as well as the order of the verses [...]", as it can be seen from the third example in the table, the order and structure of verses are kept the same as in the original.
In conclusion, after our exploration of the data augmentation results, we can infer that creative
writing is an outstanding challenge in generative AI. This is an ongoing discussed subject ([
In this subsection, we further present examples of the responses produced by the GPT-4.1-mini when detecting the categories from the second sub-task (NR, S, H, V) to the diferent prompts: 0-shot, 1-shot and 3-shot. With this aim, we establish a comparison among the three diferent prompting strategies and explore the performance results in detail. The following three tables present three significant examples of the response of the model in diferent categories. Some of them include interesting information about the detailed reasoning of GPT-4.1 when making a classification decision. For instance, we can observe the presence of semantic arguments and linguistic expressions to justify the LLM choice and how its interpretation of the semantic context of the song afects its final decision.
Table 11 presents the responses to a song categorized as "Sexualization". As we mentioned in Section 5.1, this is one of the predominant categories within the dataset, but even though the LLM performance Original: <Estoy tan lejos de ti Y a pesar de la enorme distancia Te siento juntito a mí; Corazón, corazón; alma con alma Y siento en mi ser tus besos No importa que estés tan lejos Estoy pensando en tu amor Y a lo loco platico contigo Te cuento de mi dolor Y aunque me hagas feliz No te lo digo Y vuelvo a sentir tus besos No importa que estés tan lejos El cielo empieza a clarear Y mis ojos se llenan de sueño Contigo voy a soñar Porque, quieran o no Yo soy tu dueño Y siempre tendré tus besos No importa que estés tan lejos Estoy pensando en tu amor.> Paraphrase 1: <Me siento lejos de tu piel Aunque estemos distantes, Te imagino bien cerca de mí; Alma con alma, latidos juntos, Y en mi cuerpo guardo tus caricias, No importa que estés tan lejos. Mi mente se llena de tu amor, Hablo contigo en mi delirio, Te comparto mi tristeza, Aunque no lo diga, tú me completas, Y vuelvo a sentir tus labios, No importa que estés tan lejos La madrugada comienza a salir, Los párpados se ciegan de cansancio, Contigo quiero soñar, Porque, guste o no, soy quien te posee, Y siempre tendré tus besos, No importa que estés tan lejos.> Paraphrase 2: <Estoy lejos de tu figura Pienso en ti sin medida, Siento tu calor en la distancia, Corazón junto al mío, Tus caricias me persiguen, Aunque estés lejos de aquí. Me absorto en las llamas de tu querer, Hablo contigo en mis sueños, Te revelo mi tristeza, Aunque no lo confiese, Y tus labios son míos, No importa que estés lejos. Cuando el alba empieza a brillar, Mis ojos se cierran de fatiga, Quiero perderme en tus sueños, Porque, quieras o no, Soy el dueño de tu cuerpo, Y siempre poseeré tus besos, Aunque estés lejos.> Paraphrase 3: <Soy quien controla tu cuerpo Y siempre tendré tus besos, No importa donde estés. Aunque la distancia crezca, Te siento muy cerca de mí, En mi alma estás presente, Y tus caricias me envuelven. Hablo contigo en silencio, Revelo mi pena oculta, Y aunque no te lo diga, Mi deseo por ti no muere. Al amanecer me rindo al sueño, Cierro los ojos imaginando, Que tus labios son míos, Aunque estés muy lejos.> shows high results, there are some cases where it struggles to detect "Sexualization". As we observe, this song is classified as "Non-Related" in 0-shot and 1-shot responses. The LLM gives diferent arguments to support its classification decision, but some of them are inconsistent: "using afectionate and flattering language (for example: ’you’re perfect’, ’doll, princess’, ’I love all of you’), without dehumanizing or objectifying" (0-shot) and "barbie, doll, princess physical descriptions are expressed [...] without any degrading intent or objectification"(1-shot). In both responses, linguistic cues carry explicit meanings of women body objectification that are interpreted as mutual sexual desire. This is reformulated in the 3-shot response in which the model improves its performance considering the association of women with figures such as Barbie or a doll as clearly objectifying and subordinating women.
The second table (Table 12) shows a comparison of the LLM responses to a song categorized as "Violence" by the true label. This particular example demonstrates how the model struggles to detect instances of violent actions against women even when they are explicitly expressed through linguistic cues. As we observe, the model misclassifies this category as "Sexualization" and tries to justify the presence of some linguistic expressions that carry significant semantic weight such as "I could be your rapist", "I’ll take you to the limit" or "I’ll make you move to extremes". These expressions are detected by the LLM as sexual objectification or sexual subordination but are understood as sexual and dominance play within the context of "Sexualization". In this sense, the responses of 0-shot and 1-shot tend to misunderstand the violence expressed by the song masked under sexualization and domination play. In addition, in 3-shot the model mitigates the violent connotation of expressions such as "I could be your rapist" and supports its decision mentioning that these linguistic instances carry an implicit meaning of 0-shot: <The lyrics of the song "Es Un Secreto" do not meet the general definition of misogyny [...] Nor does it perpetuate gender stereotypes that promote subordination [...]. The content is primarily romantic [...] using afectionate and flattering language (for example: "you’re perfect," " doll, princess," "I love all of you"), without dehumanizing or objectifying women in a degrading way. [...] The portrayal of sexuality is not used as a tool for domination or degradation, but rather as an expression of desire and connection between two people. Consequently, the correct classification is ’Not Related’ (NR).> 1-shot: <The lyrics of "Es Un Secreto" do not meet the established definition of misogyny, [...] Nor do they perpetuate stereotypes of female subordination or objectify the body or its parts in a degrading manner [...]. The use of metaphors such as “Barbie, doll, princess” and physical descriptions (“your face, your body, your eyes, also your skin”) is expressed in a tone of admiration and appreciation, without degrading intent or objectification [...] the song conveys an ideal of connection between two parties where desire is shared and consensual. Therefore, the song should be classified as ’Not Related’ (NR)[...].> 3-shot: <The song "Es Un Secreto" is classified as ’Sexualization’(S) because, [...] it does perpetuate the objectification and subordination of women through sexualization. The lyrics repeatedly highlight female physical attributes [...] and use terms like “Barbie, doll, princess,” which objectify women by comparing them to idealized and superficial figures. Additionally, there is an underlying intention of domination [...], and the invitation to “dance sensually” and “please,” which suggest a dynamic where the woman is primarily seen as an object of sexual pleasure. [...] it fits the definition of misogyny in terms of promoting the subordination and objectification of the female body with an implicit intent of sexual domination, clearly placing it within the category of Sexualization.> sexual subordination of women. In conclusion, this is one of the examples that illustrates how "Violence" against women goes undetected by the model.
Finally, the third table presents a comparison of the model response to a song in the category "Hate". "Hate" is one of the categories from sub-task two that improves with 3-shot while it presents some performance limitations in 0-shot and 1-shot according to the confusion matrix presented in Figure 5 (5b, 5c, 5d). The following example illustrates this scenario in which the song is classified as "Non-Related" in 0-shot and 1-shot. According to the model, in both responses, the song "contains neither explicit disdain nor violent intentions against women". In addition, in 0-shot, the model supports the idea that “although the narrator expresses a certain degree of contempt or resentment due to having been abandoned, the language used is neither ofensive nor discriminatory toward women”. Also, in 1-shot, it states a similar reason: “Pejorative language is used to describe the man with whom the woman was involved, such as the expression (’that bastard’)[...]the underlying intention seems to align more with a narrative centered on betrayal and personal empowerment.” There is a significant improvement in 3-shot since the model recognizes linguistic cues that carry explicit expressions of misogyny and hate speech against women such as "Consider yourself dead; your time is up." Also, in 3-shot, the model interprets the expression “that bastard” as one of the expressions conveying “disdain and resentment toward the woman.”
From a qualitative perspective, we consider two key aspects from the examples analyzed in this
section. On the one hand, there is a clear improvement in the classification performance of the model.
As previously discussed, the category of “Sexualization” is the most accurately classified, for the reasons
outlined earlier. However, in some instances, the model overlooks certain linguistic cues that may be
crucial for detecting the appropriate category. The categories of “Hate” and “Violence” exhibit notable
0-shot:
<The song contains language that suggests sexual acts and domination games, specifically in
verses like “I’ll be your butler and you’ll play the role of Lady Fine” and “Or I could be your
rapist[...], which imply a sexual relationship based on domination[...]. There are also explicit references
[...]which objectify the female body with a clearly sexual intent. Although the lyrics use figurative
language and a framework of a “seduction game,” the depiction of scenes in which the woman is
objectified and placed in a subordinate role during sexual acts fits the definition of misogyny
through sexualization. No direct expressions of physical or psychological aggression (Violence
category) or explicit hate speech (Hate category) are identified. Therefore, the appropriate classification
is Sexualization(S).>
1-shot:
<The song "Juego de Seducción" is classified as Sexualization (S) [...] the woman is portrayed in a passive
or subordinate role within an implicitly dominant context. [...]especially the line “Or I could be
your rapist, imagination” suggest a power dynamic that implies an unequal relationship in which
the woman is positioned as an object within a sexual game of domination and submission.
[...] “rapist” [...] directly invokes a violent sexual act [...]. Furthermore, the chorus [...] reinforce the
dominant and controlling intent from a sexual angle that instrumentalizes female sexuality. There is
no direct threat of physical violence or clear expression of hate, so it should not be classified
under Violence (V) or Hate (H). [...]. For these reasons, it clearly fits under the category Sexualization
(S).>
3-shot:
<The song "Juego de Seducción" meets the definition of misogyny [...] the lyrics suggest roleplay [...]
and explicitly mention the possibility of being “your rapist”, which implies violent intent and
submission in a sexual context. Although the violence is not directly expressed as a real threat
or harm, the reference to rape within a game scenario sexualizes the woman as a passive,
subordinate object, reinforcing subordination [...] “I’ll take you to the edge” and “your very skin”
are expressions that objectify the female body in a context of sexual domination. Therefore, the
misogyny present is mainly expressed through sexualization, with an implicit component of domination
and submission[...]>
classification limitations and reveal that the arguments developed by the model are significantly more
biased in these cases compared to the “Sexualization” category. Nevertheless, while the classification
of “Hate” improves considerably in responses using 3-shot prompting, the model still struggles to
accurately classify songs containing semantic content related to violence against women. On the other
hand, after a detailed examination of the generated responses, the misclassification of “Violence” as
“Sexualization” or “Non-Related” may be due to the presence of more linguistically complex structures
that convey implicit meanings, which are far more subtle than the explicit language found in songs
classified as "Sexualization". In addition, we could suggest that the model may exhibit greater bias
when interpreting this type of content as a consequence of the perpetuation of gender biases in AI. The
presence of misogyny is a one of the most complex issues to be detected since it requires a nuanced
understanding of language and context [
This work has addressed the detection of misogyny—an already well-explored topic in NLP—within the under-researched yet culturally significant domain of song lyrics. By participating in the MiSonGyny 2025 shared task, we contribute to expanding misogyny detection into new discursive contexts.
Our experiments with a RoBERTa-based system confirm that fine-tuned transformer architectures remain efective for this task, achieving fifth place in the general misogyny detection subtask. However, their performance sufers under data imbalance, particularly in the fine-grained classification scenario, where overfitting is more pronounced, achieving only 31st position on the test ranking.
In contrast, our exploration of LLMs investigated the impact of in-context learning and chain-ofthought prompting. Despite limited data, LLMs demonstrated strong performance in the fine-grained subtask, securing third place and outperforming BERT-like models. Beyond raw performance, we analysed the LLM’s reasoning capabilities, noting that while it efectively identified explicit categories like Sexualization (S), it struggled with more implicit ones like Violence (V) and Hostility (H). These dificulties may stem from the subtlety of these expressions and the model’s inability to justify its predictions with coherent rationale—issues clearly reflected in the confusion matrix.
Furthermore, we examined the use of LLMs for data augmentation through paraphrasing and synthetic
sample generation. However, the results suggest limited variability in the augmented data, potentially
due to constraints imposed by safety filters or a lack of creativity in LLM-generated text [
Building on our LLM experimentation, we plan to extend our work beyond closed models, utilizing
open LLMs like the Gemma [
On the other hand, although the following work addresses the detection of misogyny in song lyrics,
we believe there is future research to be conducted regarding the detection of stereotypes [
In conclusion, our study presents a multifaceted evaluation of misogyny detection in lyrics, comparing traditional fine-tuned models and LLMs, and highlighting their respective strengths and limitations. We provide both quantitative metrics and qualitative insights, ofering new perspectives on LLM reasoning and their potential for future work in data augmentation and interpretability.
This work has been developed under the framework of the following projects respectively: MARTINI (Grant PCI2022-135008-2) and FairTransNLP-Stereotypes (Grant PID2021-124361OB-C31) both funded by MCIN/AEI/10.13039/501100011033 and by European Union NextGenerationEU/PRTR.
During the preparation of this work, the authors used used ChatGPT and DeepL in order to: Text Translation, Paraphrase and reword and Improve Writing style. After using these services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
OBJECTIVE:Generate three paraphrases of the lyrics of the following song, which contains misogynistic elements and/or hate speech, specifically of the type {TYPE_MISOGYNY}. The output must be structured in XML format.
DEFINITION OF MISOGYNY: We speak of a song’s lyrics as containing misogyny when they contain hate speech or disdain directed at women. Lyrics that promote the subordination or objectification of women also fall into this category.
DEFINITION OF {MISOGYNY_TYPE}: {MISOGYNY_TYPE_DEFINITION}.
ADDITIONAL RULES: • Do not remove misogynistic and/or problematic elements from any of the new versions. • You do not need to respect the rhyme of the verses, but you do need to keep the central theme as well as the misogyny of {TYPE_MISOGYNY} in the song. • For the first paraphrase, paraphrase all misogynistic and non-misogynistic content, but keep the hate speech nature of {TYPE_MISOGYNY} as per the definition provided. • For the second paraphrase, it paraphrases all the misogynistic verses of the song, while the non-misogynistic verses are replaced by entirely new ones in line with the original subject matter of the song as well as its message. The paraphrases of the misogynistic elements should be in accordance with the definition of {TYPE_MISOGYNY} provided. • For the third paraphrase, paraphrase all misogynistic and non-misogynistic verses of the song, but modify the structure of the song as well as the order of the verses, always maintaining a narrative coherence that resembles the original lyrics but conforms to the definition of {TYPE_MISOGYNY} provided. • Before paraphrasing, think about the changes you are going to make and explain them briefly. Briefly justify why your new version still retains the original essence as well as why it is misogynistic according to the definition of {TYPE_MISOGYNY} we have provided.
OBJECTIVE: Generate a new song lyric from the lyrics of the following song, which contains misogynistic elements and/or hate speech, specifically of the type {MISOGYNY_TYPE}. DEFINITION OF MISOGYNY: We speak of a song’s lyrics as containing misogyny when they contain hate speech or disdain directed at women. Lyrics that promote the subordination or objectification of women also fall into this category.
DEFINITION OF {MISOGYNY_TYPE}: {MISOGYNY_TYPE_DEFINITION}. RULES: • Preserving misogyny: It is imperative that the song contains similar misogynistic elements in its new versions. The specific type of misogyny {MISOGYNY_TYPE} and the essence of the message must remain clearly recognisable. • Maintain Original Themes: The newly generated song should respect much of the original themes proposed in the song. For example, if the song is about heartbreak, the new song should focus on this theme, while if it is about sexual acts, the new version should talk about this as well. • Rhymes: The new version does not need to respect the rhymes between verses, but it does need to maintain misogynistic elements in its final version. • Mood: Use the feeling/mood associated with the original song to generate the new song. If the song emanates a feeling of sadness, the new song should maintain this essence, while if it emanates a feeling of revenge or hatred, the new song should reflect these feelings. • Rhetorical figures of speech: Introduce at least 2 or 3 metaphors or similes in the song.
However, these figures of speech should NOT afect the explicitness of the misogyny, if any. • Length: The resulting song should be approximately the length of the original song in terms of the number of verses. • DON’T JUST PARAPHRASE OR COPY THE ORIGINAL SONG: Generate a new song from the one provided, BUT don’t mimic the original song in its entirety. Be creative. • Identify all elements of the original song that relate to the definition of {MISOGYNY_TYPE}.
These new elements must appear, in one form or another, in the newly generated version. • Extract and argue the main themes, the genre as well as the mood of the next song. • Generate a new song: Consider all of the above elements to generate a new song of acceptable quality that is suficiently diferent from the original.
Once you have generated the new song, justify why it maintains the themes of the original song as well as the misogynistic elements. Please respond using the following outline: <response> <elements_m>Here you should write down all the misogynistic elements you have identified in the song along with their relationship to {MISOGYNY_TYPE}. Argue your answer by citing specific excerpts from the song that have allowed you to reach this conclusion. </elements_m> <main_themes>Include here all the main themes surrounding the song with an argument for the presence of these"</main_themes>. <mood>Mention here the main moods conveyed by this song</mood>. <genre> Mention here the genre to which the song belongs</genre>. <new_lyrics> The lyrics of the new song you have generated </new_lyrics>. <justification>Justification of why the new version you have generated maintains the misogyny. </justification> </response> Here are the lyrics to the song: {lyrics}