Gender-neutral rewriting (GNR) aims to reformulate text to eliminate unnecessary gender specifications while preserving meaning, a particularly challenging task in grammatical-gender languages like Italian. In this work, we conduct the first systematic evaluation of state-of-the-art large language models (LLMs) for Italian GNR, introducing a two-dimensional framework that measures both neutrality and semantic fidelity to the input. We compare few-shot prompting across multiple LLMs, fine-tune selected models, and apply targeted cleaning to boost task relevance. Our findings show that open-weight LLMs outperform the only existing model dedicated to GNR in Italian, whereas our fine-tuned models match or exceed the best open-weight LLM's performance at a fraction of its size. Finally, we discuss the trade-of between optimizing the training data for neutrality and meaning preservation.
of the Senate has received the information). A further
challenge in automatic GNR is preserving the meaning
Language technologies reinforce existing gender stereo- of the original sentence beyond gender expression, to
types and binary assumptions by disproportionately avoid generating output sentences that are neutral but
favoring masculine references or representations [
Gender-neutral rewriting (GNR) has emerged as a nat- specific gendered phenomena, such as member nouns ural language generation task aimed at producing texts [13], or use neologistic [14] inclusive devices such as free from unnecessary gender specifications [ 10, 11]. This neomorphemes and graphemic solutions [15, 16, 17] that task is particularly challenging in grammatical-gender convey neutrality, but are not necessarily acceptable in languages, such as Italian, due to the pervasive encod- all contexts. Currently, the sole model dedicated to Italian ing of gender in the morphology. Consider the sen- GNR was developed by Greco et al. [18], which, however, tence ‘Tutti i senatori sono stati informati’ (equivalent was developed and tested on proprietary, not publicly to AllM theM senatorsM have beenM informedM): almost available data, hindering reproducibility and progress. every word is morphologically inflected for (masculine) Towards addressing this gap, this paper explores the gender. Rephrasing this sentence in a gender-neutral potential of state-of-the-art (SOTA) large language modway may require significant changes, e.g. ‘ Ogni mem- els (LLMs) to perform GNR in Italian. Specifically, we bro del Senato ha ricevuto l’informazione’ (Every member explore both prompting and fine-tuning approaches and assess both neutrality and meaning preservation in the tCicLsi,CS-eitpt2e0m25b:eErl2e4ve—nt2h6I,t2a0li2a5n, CCaognlfiearrein,cIteaolyn Computational Linguis- reformulated texts. * Corresponding author. Our contributions are threefold: i) The first systematic $ apiergentili@fbk.eu (A. Piergentili); bsavoldi@fbk.eu evaluation of SOTA LLMs for Italian GNR under a two(B. Savoldi); negri@fbk.eu (M. Negri); bentivo@fbk.eu dimensional framework measuring both neutrality and (L. Bentivogli) meaning preservation; ii) A set of experiments in fine(B. 0S0a0v0o-l0d0i0);30-020101-70-010323-88(8A1.1-P4i3e3rg0e(nMti.liN);e0g0r0i)0;-00000002--03000611--78438107-2231 tuning LLMs for GNR, enabling compact models to rival (L. Bentivogli) significantly larger-sized models; iii) An investigation of © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License the GNR performance trade-of between meaning preserAttribution 4.0 International (CC BY 4.0). vation and neutrality in the outputs of LLMs fine-tuned on sentence similarity-optimized data.1 REF-G
EN REF-N
EN
Spero di essere stato chiaro su questo punto.
I hope that I am clear in this.
Spero di avere espresso con chiarezza questo punto.
I hope that I have expressed this point clearly.
to prevent expressions that reinforce gender hierarchies
or render non-binary identities invisible, promoting fair- In REF-N is italicized.
ness and inclusion in alignment with UN Sustainable
Development Goals of gender equality.2 In
grammaticalgender languages like Italian, inclusive language is both work has explored gender-neutral translation [
to remove explicit gender markings referring to human
Gender-Inclusive Rewriting In recent years, sexism entities, without altering the sentence beyond what is
and gender-exclusionary practices have been increas- necessary for neutralization, ensuring semantic
equivaingly addressed in NLP, focusing initially on binary gen- lence to the input. We run a set of experiments evaluating
der bias and more recently expanding to non-binary in- diferent systems and approaches to GNR. Here, we first
clusive language technologies [
Intralingual inclusive rewriting has primarily been ex- 3.1. Evaluation plored in English [10, 11, 12], where gender marking is scarce. Similar eforts in languages with grammatical gender include research on German [15], Portuguese [16], and French [17, 13], either by using innovative forms or targeting specific instances of gendered languages—such as masculine generics in member nouns. In Italian, prior 1We release models and data at https://huggingface.co/FBK-MT 2See https://sdgs.un.org/goals/goal5 3English presents fewer challenges as gender marking is primarily limited to pronouns, allowing focused solutions like the singular they [23]. 4See for instance the EU Parliament guidelines for gender-neutral language: https://www.europarl.europa.eu/cmsdata/151780/GNL_ Guidelines_EN.pdf
our GNR experiments on mGeNTE [
LLM-as-a-Judge [43] approach proposed by Piergentili et al. [44], which provides sentence-level binary gendered/neutral assessments, and was shown to be highly accurate in both human- and model-generated texts. We use their optimal configuration for monolingual evaluation.5 We compute the percentage of neutralized sentences over the whole test set (750 entries).
To evaluate meaning preservation in GNR, we use
BERTScore [45], an attested BERT-based [46] metric
measuring the semantic similarity of two texts (the higher
the better, indicating close similarity). We use BERTScore
rather than common string-matching metrics like BLEU
[47] and TER [
To identify reference values to guide the interpretation of BERTScore in GNR, we compute the distribution of BERTScore of mGeNTE REF-N sentences against the respective REF-G.6 As these neutral reformulations were produced by human experts, the BERTScore distribution provides an empirical estimate of human-level performance in meaning preservation in GNR. We take the mean BERTScore minus one standard deviation 5Prompt: ‘Mono+P+L’; GPT model: gpt-4o-2024-08-06 6We only use Set-N entries in this computation. Rewrite, Italian Riformula la seguente frase utilizzando un linguaggio neutro rispetto al genere dei referenti umani, evitando l’uso di forme maschili e femminili.
Rewrite the following Italian sentence using a gender-neutral language in reference to human beings, avoiding masculine or feminine forms.
GFG, English Rewrite, English Sei un riscrittore di frasi italiane con l’obiettivo di rendere i testi You are a rewriter of Italian sentences with the goal of making neutrali rispetto al genere dei referenti umani. Ti viene fornita una texts gender-neutral with respect to human referents. You are frase che contiene riferimenti a persone in forme marcate per given a sentence that contains references to people using genere, come il maschile sovraesteso o coppie binarie. gender-marked forms (such as masculine generics or binary pairs). Il tuo compito è riformulare la frase in modo da: Your task is to rewrite the sentence to: Per farlo, usa strategie come:
To do this, use strategies such as: • rimuovere riferimenti espliciti al genere quando non
necessari; • mantenere inalterato il significato originale; • preservare lo stile e la leggibilità del testo. • sostantivi collettivi (“la cittadinanza”, “il personale”,
“l’utenza”); • perifrasi impersonali (“si dovrebbe”, “si consiglia”); • forme passive (“l’accesso è consentito”); • forme imperative (“allega il documento”); • pronomi relativi e costruzioni subordinate (“chi ha svolto
attività di pesca”); • termini epiceni (“ogni giudice”, “gentile collega”); • termini neutri (“l’individuo”, “la persona interessata”, “il
membro”). • evita l’uso del maschile come forma generica e non usare
forme grafiche non standard come asterischi o schwa; • evita doppie formulazioni come “il/a cittadino/a” oppure
“il professore o la professoressa”; • non rimuovere parti della frase che non richiedono
modifiche (ad esempio, i nomi propri); • fornisci solo la frase riformulata. • remove explicit gender references when they are not
necessary; • preserve the original meaning; • maintain the style and readability of the text. • collective nouns (“la cittadinanza”, “il personale”,
“l’utenza”); • impersonal phrases (“si dovrebbe”, “si consiglia”); • passive constructions (“l’accesso è consentito”); • imperative constructions (“allega il documento”); • relative pronouns and subordinate clauses (“chi ha svolto
attività di pesca”); • epicene terms (“ogni giudice”, “gentile collega”); • neutral terms (“l’individuo”, “la persona interessata”, “il
membro”). • avoid using the masculine form as a generic and do not
use non-standard spellings such as asterisks or schwa; • avoid binary formulations such as “il/a cittadino/a” or “il
professore o la professoressa”; • do not remove any part of the sentence that does not
need to be rewritten (e.g. proper names); • only return the reformulated sentence.
IMPORTANTE: IMPORTANT: 3.3. Few-Shot Prompting
in the selection described above,9 to investigate the
performance of LLMs without any task-specific fine-tuning.
We use two prompt formats:
• GFG: a concise rewriting instruction, originally
used by Frenda et al. [
We instead test its of-the-shelf generation capabilities.
complex instruction on models’ performance. Moreover,
we experiment with these two prompt formats by
formulating them in both Italian an English, to investigate
whether the language used is a relevant factor as well.
The content of the prompts is reported in Table 3. We
include the same 8 task exemplars—or shots—with all
prompts, to elicit the in-context learning ability of LLMs
[
of larger models or closed systems. Namely, we fine-tune
LLaMAntino, Velvet, LLama 3.1, Phi 4, and the 8B and
14B Qwen3 models.
neutrality classifier. 10 This data consists in gendered
Italian sentences and their gender-neutral counterparts,
all generated starting from a dictionary of masculine,
feminine, and neutral expressions, through a multi-step
prompting pipeline. We repurpose this data to fine-tune we compare the gendered and neutral sentences in the
autoregressive LLMs for GNR. We prepare the data as dataset using BERTScore to identify dataset entries with
chat-formatted input, where each instance consists of a semantically divergent gendered-neutral sentences.
Figuser role message containing a gendered sentence, and ure 1 reports the BERTScore values for the entire dataset.
an assistant role message containing the correspond- We observe that while the score distribution is skewed
ing neutral sentence. Consistent with the models’ prior towards almost-perfect values, there is a notable tail of
instruction-following fine-tuning, this method adopts a gendered-neutral sentence pairs with a rather divergent
conversational prompt–response format while strictly semantic content. To investigate the impact of such data
adhering to a causal token-prediction objective [
As the sentences were partly LLM-generated, we note training alongside the full dataset: a clean subset
obthat the content of the gendered-neutral pairs may not tained by filtering out the bottom 50% of sentence pairs
always be aligned due to the unpredictability of LLMs based on the BERTScore values. Statistics about the
finein open-ended generation.11 To investigate this aspect, tuning data are reported in Table 4.
10More specifically, we use the cleaned version of the dataset
later released by Savoldi et al. [
3.4.2. Method
12We run our experiments on nodes with 4 NVIDIA A100 GPUs with
64 GB VRAM each. balance between hardware constraints12 and consistency English), we report each model’s average performance, across model sizes and requirements: learning rate: along with the range of neutrality and BERTScore values 2 × 10− 4, batch size: 8 for the 8B models, 4 for the observed across prompting conditions. In Appendix A 14B models. We use early stopping with a patience of 20 we provide the complete and detailed results obtained steps for the 8B models and 40 steps for the 14B models. with the two prompt formats, separately for Italian and English instructions.
Generally, and with rare exceptions, all models’
4. Results BERTScore values are well above the quality threshold
we identified in §3.1. This means that the models do not
4.1. Few-Shot Prompting Results generate unrelated or additional text, confirming that
Figure 2 summarizes the results of the few-shot prompt- their outputs remain adherent to the input and free of
ing experiments showing all models’ performance in “hallucinations” [
LLama 3.3 perform better than the Italian models. The rest of the Qwen3 family generally underperforms, with the high BERTScore suggesting that they make little to no change to the gendered sentences. The only model performing well on both axes is GPT 4.1, which tops at 89.07% neutralization accuracy and 93.21 BERTScore, indicating that it correctly alters the parts of the sentences expressing the gender of human beings while leaving the rest untouched.
Overall, we find that the LLMs we tested perform very diferently in GNR in Italian, and that failure in this setting consists in overlooking the relevant (gendered) parts of the input to act upon, and/or unsuccessfully rendering them gender-neutral. 4.2. Fine-Tuning Results
Figure 3. We first notice that on the neutrality axis all ifne-tuned models outperform the baseline, except for LLamantino/clean configuration. LLamantino shows the narrowest gains overall, and in one case even a drop in neutrality, echoing its weaker few-shot prompting results and suggesting it may be ill-suited to GNR. In four out of six instances, and always with the full dataset, the finetuned models also outperform the best performer among the open-weight models in the prompting experiments, i.e. LLama 3.3 70B with the GFG English prompt, though with a significant drop in BERTScore.
Such a drop indicates that these models fail by
hallucinating unrelated content in their attempt to neutralize,
rather than by leaving the input sentences untouched
as observed in the prompting experiments (§4.1). This
is possibly due to two factors: the significantly smaller
size of the fine-tuned models with respect to LLama 3.3
70B (1/9 or 1/5, for the 8B and 14B models respectively),
as larger LLMs have been shown to exhibit greater
robustness and lower variance in downstream performance
after fine-tuning compared to smaller counterparts [
While full yields the highest improvements in neutrality, only clean improves performance on both axes while keeping BERTScore within the human-level range.
However, it yields significantly smaller gains in neutrality and even causes drops for two models (LLamantino, Phi 4). We hypothesize that clean may be excessively conditioned by the data filtering method, i.e. a BERTScore based selection. In other words, by selecting only dataset entries with almost perfect BERTScore values we are optimizing the models to perform well on the sentence similarity dimension—as measured by BERTScore—rather than GNR.
The impact of metric-based data selection To investigate the hypothesis above, we evaluate the same outputs against the gendered inputs with another semantic similarity metric: BARTScore [57].13 BERTScore and 13While similar in name and scope, BERTScore and BARTScore function diferently. The first computes a sum of token-level cosine similarities between two sentences’ embeddings encoded by a BERT (encoder-only) model; the latter is computed as the weighted sum of the log-probabilities that a pretrained BART (encoder-decoder) model assigns to each token in the generated text. In our experiBARTScore evaluations are visualized in Figure 4. To Through fine-tuning experiments we showed that comunderstand whether outputs of the models fine-tuned pact models can match or exceed the best open-weight on clean are actually very semantically similar to the LLM at a fraction of its size. Moreover, our BERTScorecorresponding input, and whether those models simply based data cleaning highlighted a trade-of: models learned to game BERTScore, we compute14 the Pear- trained on cleaned data achieve human-level BERTScore son r and Spearman correlation coeficients between but show smaller neutrality gains and exhibit ranking BERTScore and BARTScore assessments. The first cap- diferences against another similarity metric, signaling tures linear correlations between the two metrics’ raw over-fitting on BERTScore. Future work should take this scores, while the latter measures how well the relation- trade-of into account and create dedicated, high-quality ship between the two variables can be described by a parallel data to aim at reaching the performance of the monotonic function, by comparing the rankings of the commercial system with open-weight models. scores rather than their raw values. This combination allows us to assess both the alignment of the scores and the consistency in how the two metrics rank the outputs. Acknowledgments
We find that in full, r equals 0.814 and equals 0.907, whereas in clean they are 0.914 and 0.679 respectively.15 We acknowledge the support of the project InnovAction: r is high in both cases, indicating a strong linear corre- Network Italiano dei Centri per l’Innovazione Tecnologlation between the two metrics—stronger in clean, as ica (CUP B47H2200437000), funded by MIMIT with NPRR in that case the data points are more tightly clustered, - NextGenerationEU funds, in collaboration with Piazza skewed towards higher values. This confirms that the Copernico S.r.l. We also received funding from the PNRR metrics generally agree on the quality of the outputs. project FAIR - Future AI Research (PE00000013), under The substantial drop in , instead, indicates that there the NRRP MUR program funded by NextGenerationEU. are many instances in clean where the monotonic trend Finally, we acknowledge the CINECA award under the is broken, i.e., higher BERTScore does not necessarily ISCRA initiative (AGeNTE), for the availability of highcorrespond to higher BARTScore. This suggests that performance computing resources and support. the clean models also learned to game BERTScore by reproducing features rewarded by that metric. References
With respect to our hypothesis: by selecting highsimilarity pairs for the clean dataset, we efectively steered models toward preserving semantic alignment with the input; however, this emphasis on similarity appears to have hampered their improvement in neutralization. Indeed, the models learned to preserve the input to an excessive degree, as confirmed by the high r coeficient and high BARTScore values shown in Figure 4. We interpret our results as evidence of a broader trade-of between optimizing for neutrality and for sentence similarity. Our findings underscore the need for data curation strategies that strike a balance between neutrality and similarity, achieving the flexibility required for efective GNR.
We presented the first systematic investigation of state-ofthe-art large language models for Italian gender-neutral rewriting under a two-dimensional evaluation of neutrality and meaning preservation. In our few-shot prompting experiments, open-weight models outperformed the only existing Italian-specific system but remained behind a closed commercial system.
ments, we use the BART model facebook/bart-large [58]. 14We use the Python library SciPy [59]. 15All p-values < 0.05.
ing of the ACL, ACL, Online, 2020, pp. 5454–5476. 2023, pp. 8747–8759. URL: https://aclanthology.org/ URL: https://aclanthology.org/2020.acl-main.485/. 2023.findings-emnlp.585/. [6] S. Dev, M. Monajatipoor, A. Ovalle, A. Subramo- [17] P. Lerner, C. Grouin, INCLURE: a dataset and toolkit nian, J. Phillips, K.-W. Chang, Harms of gender for inclusive French translation, in: Proc. of the exclusivity and challenges in non-binary represen- 17th Workshop on Building and Using Comparable tation in language technologies, in: Proc. of the Corpora (BUCC) @ LREC-COLING 2024, ELRA and 2021 Conference on Empirical Methods in Natural ICCL, Torino, Italia, 2024, pp. 59–68. URL: https: Language Processing, ACL, Online and Punta Cana, //aclanthology.org/2024.bucc-1.7/. Dominican Republic, 2021, pp. 1968–1994. URL: [18] S. Greco, M. La Quatra, L. Cagliero, T. Cerquitelli, https://aclanthology.org/2021.emnlp-main.150/. Towards ai-assisted inclusive language writing in [7] U. Gabriel, P. M. Gygax, E. A. Kuhn, Neutralis- italian formal communications, ACM Trans. Intell. ing linguistic sexism: Promising but cumbersome?, Syst. Technol. (2025). URL: https://doi.org/10.1145/ Group Processes & Intergroup Relations 21 (2018) 3729237.
844–858. [19] B. Papadopoulos, Morphological Gender Innova[8] APA, Publication Manual of the APA, 7th ed., 2020. tions in Spanish of Gender queer Speakers, Depart[9] A. Piergentili, D. Fucci, B. Savoldi, L. Bentivogli, ment of Spanish and Portuguese, University of CaliM. Negri, Gender neutralization for an inclusive fornia, UC Berkeley, 2019. URL: https://escholarship. machine translation: from theoretical foundations org/uc/item/6j73t666. to open challenges, in: Proc. of the First Work- [20] G. S. di Carlo, Is italy ready for gender-inclusive shop on Gender-Inclusive Translation Technolo- language? an attitude and usage study among gies, EAMT, Tampere, Finland, 2023, pp. 71–83. italian speakers, in: Inclusiveness Beyond the URL: https://aclanthology.org/2023.gitt-1.7/. (Non)binary in Romance Languages, 1st edition ed., [10] T. Sun, K. Webster, A. Shah, W. Y. Wang, M. Johnson, Routledge, 2024, p. 21. URL: https://doi.org/10.4324/ They, them, theirs: Rewriting with gender-neutral 9781003432906.
english, 2021. arXiv:2102.06788. [21] G. V. Silva, C. Soares, Inclusiveness Beyond the [11] E. Vanmassenhove, C. Emmery, D. Shterionov, Neu- (Non)binary in Romance Languages: Research and Tral Rewriter: A rule-based and neural approach Classroom Implementation, 1st ed., Routledge, Lonto automatic rewriting into gender neutral alterna- don, 2024. doi:10.4324/9781003432906. tives, in: Proc. of the 2021 Conference on Empirical [22] P. Gygax, S. Sato, A. Öttl, U. Gabriel, The masMethods in Natural Language Processing, ACL, On- culine form in grammatically gendered languages line and Punta Cana, Dominican Republic, 2021, and its multiple interpretations: a challenge for pp. 8940–8948. URL: https://aclanthology.org/2021. our cognitive system, Language Sciences 83 emnlp-main.704/. (2021) 101328. URL: https://www.sciencedirect.com/ [12] M. Bartl, S. Leavy, From ‘showgirls’ to ‘performers’: science/article/pii/S0388000120300619.
Fine-tuning with gender-inclusive language for bias [23] L. Ackerman, Syntactic and cognitive issues in inreduction in LLMs, in: Proc. of the 5th Workshop vestigating gendered coreference, Glossa: a journal on Gender Bias in Natural Language Processing of general linguistics 4 (2019). (GeBNLP), ACL, Bangkok, Thailand, 2024, pp. 280– [24] M. Rosola, S. Frenda, A. T. Cignarella, M. Pelle294. URL: https://aclanthology.org/2024.gebnlp-1. grini, A. Marra, M. Floris, Beyond obscuration and 18/. visibility: Thoughts on the diferent strategies of [13] E. Doyen, A. Todirascu, Genre: A french gender- gender-fair language in Italian, in: Proc. of the 9th neutral rewriting system using collective nouns, Italian Conference on Computational Linguistics 2025. arXiv:2505.23630. (CLiC-it 2023), CEUR Workshop Proc., Venice, Italy, [14] E. Rose, M. Winig, J. Nash, K. Roepke, K. Con- 2023, pp. 369–378. URL: https://aclanthology.org/ rod, Variation in acceptability of neologistic 2023.clicit-1.44/.
English pronouns, Proc. of the Linguis- [25] G. Comandini, Salve a tutt@, tutt*, tuttu, tuttx e tic Society of America 8 (2023) 5526. URL: tutt@: l’uso delle strategie di neutralizzazione di https://journals.linguisticsociety.org/proceedings/ genere nella comunità queer online. indagine su un index.php/PLSA/article/view/5526. corpus di italiano scritto informale sul web., Testo [15] D. Pomerenke, Inclusify: A benchmark and e Senso 23 (2021) 43–64.
a model for gender-inclusive german, 2022. [26] J. Silveira, Generic Masculine Words and ThinkarXiv:2212.02564. ing, Women’s Studies International Quarterly 3 [16] L. Veloso, L. Coheur, R. Ribeiro, A rewriting ap- (1980) 165–178. URL: https://www.sciencedirect. proach for gender inclusivity in Portuguese, in: com/science/article/pii/S0148068580921132. Findings of the ACL: EMNLP 2023, ACL, Singapore, [27] A. H. Bailey, A. Williams, A. Cimpian, Based on across the categories are highlighted , and the best overall performer is in bold.
AVG Inclusively
across the categories are highlighted , and the best overall performer is in bold.
Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) in order to: Paraphrase and reword, Improve writing style, and Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.