Natural Language Understanding and Generation models sufer from a limited capability of understanding the nuances of inclusive communication as they are trained on massive data, often including significant portions of non-inclusive content. Even when the models are specifically designed to address non-inclusive language detection or reformulation, they disregard, to a large extent, inclusivenessrelated features that are likely correlated with the inclusive language nuances, such as the discourse type, level of inclusiveness, and intended context of use. To assess the importance of additional inclusiveness-related features, we collect a new corpus of Italian administrative documents humanly annotated by linguistic experts. Linguistic experts not only highlight non-inclusive text snippets and propose possible reformulations, but also annotate multi-aspect labels related to diferent inclusive language nuances. We empirically show that a multi-task learning approach that leverages the multi-aspect annotations can improve the non-inclusive text reformulation performance, thereby confirming the potential of expert-annotated data in inclusive language processing.
Non-inclusive expressions are widespread in humanly
written documents [
To mitigate the presence of bias in data, applications based
on AI rely on human supervision for model training and
post-processing evaluation. This is quite common in the
areas of Natural Language Understanding and Generative AI,
in which applications like Large Language Models (LLMs)
provide end-users with conversational and language editing
services [
The computational linguistic community has agreed
on the need to leverage human expert annotations in
experience-based learning for bias detection and
mitigation [
In the context of inclusive language understanding and
generation, most of the previous work exploits rule-based
or round-trip translations to annotate texts for inclusivity
issues [
A limited body of work has been devoted to generating
and exploiting multi-faceted expert human annotations to
drive AI models for inclusive language, e.g., [
This paper proposes an expert-annotated dataset covering these new aspects and investigates their usefulness in enhancing the performance of the task of non-inclusive text reformulation in the absence of rule-based editing models.
To this end, we enrich a corpus of Italian administrative documents with multi-aspect annotations, providing more insights into the inclusive language nuances. The purpose is to enable the study of new features describing inclusiveness aspects neglected by existing approaches, such as the level of inclusiveness, register, and genre. By enriching the language descriptions with new inclusiveness-related features, we provide the research community with new resources to enhance the understanding and writing capabilities of AIbased solutions.
We also collect preliminary results on the use of multiaspect annotations in a multi-task learning approach to enhance non-inclusive language reformulations. The results confirm the potential of the inclusiveness-related expert annotations.
The term annotation is often used to indicate the process
by which textual data are subjected to a tightly interrelated
two-phase activity [
Unlike human annotators, AI models often lack cognitive abilities such as common sense reasoning and generalization capabilities due to the relatively limited numbers of linguistic examples used for model training compared to the impressive variety of natural language forms.
Human annotators need suficient expertise to interpret nuanced linguistic phenomena and assign appropriate labels adequately. Their annotations are at the base of a supervised learning process. The trained models can progressively learn from annotated data as automatized humans do, but at a scale not possible through manual work alone. Annotation of Italian administrative documents. We have designed and utilized a novel benchmark dataset for inclusive language writing in Italian. This dataset comprises administrative communications sourced from the Italian public administration, spanning across both national and regional levels. We annotate the corpus at the sentence level. To this end, we set up a heterogeneous team of 13 linguistic experts with diverse experiences and expertise in inclusive language. The team consists of predominantly female individuals, all native Italian speakers. All the annotators are educated: 57% have at least 10 years of experience in linguistics, and 50% have at least 3 years of experience in inclusive language. In addition, the annotators received, on average, about 30 hours of training specific to inclusive language annotations.
Each human annotator independently assigns inclusiveness-related metadata to the document sentences. Each sentence can be enriched with multiple annotations. The annotations consist of (a) The reformulation of any non-inclusive piece of text, i.e., an alternative inclusive form; (b) The level of inclusiveness of the input sentence indicating whether a sentence is non-inclusive, inclusive, or not pertinent; (c) The register or intended context of use, i.e., Standard, Specialized, or Informative/Educational; (d) the discourse type or genre, i.e., Legal, Administrative, Technical, or Informative/Educational.
Additional contextual aspects could be included in future annotations to enhance models’ understanding of inclusive language usage further. By jointly providing those annotations, the experts aimed to capture inclusive language’s nuanced, multi-faceted nature.
By learning language inclusiveness patterns from a diversified, context-dependent set of expert annotations, AI models gain exposure to subtle interpretive diferences. The consistency across annotations is ensured through detailed guidelines and instructions provided to experts. Before full annotation, a collaborative analysis of a sample set identifies any divergent interpretations to refine guidance.
ber of annotated sentences for each aspect, separately for the training, validation, and test sets.
Task ID NILR ILC RC GC
Train
Example of annotations. Table 2 shows an example of an Italian annotated sentence (as well as the corresponding English translation for non-Italian readers). Linguistics experts assign diferent annotations to each sentence. In this example, they have assigned three labels to the sentence. Regarding inclusiveness, the sentence has been categorized as non-inclusive because it contains “Il Presidente” (i.e., Chair/President) and “Rettore” (i.e., Rector), which are masculine declensions of professional roles. In addition, the sentence also contains “suo decreto”, which refers to a decree that comes from a male person, so the sentence is not inclusive. The discourse sequence is of the administrative type, as the content refers to an administrative topic, and the used language is specialized, as the content describes specific and technical aspects.
for Italian Inclusive Language
We conduct an empirical analysis to examine the impact of utilizing expert annotations in inclusive language generation. Specifically, we investigate the advantages of simultaneously addressing two key objectives: reformulating non-inclusive language and predicting various aspects of inclusiveness.
Tasks. Given a non-inclusive piece of text , the NonInclusive Language Reformulation (NILR) task aims at generating an equivalent inclusive natural language form. The NILR task is a sequence-to-sequence problem, where the input is a non-inclusive sentence and the output is the corresponding inclusive sentence.
Given and an aspect , the goal is to predict the ’s value for . can be the level of inclusiveness, register or intended context of use, and discourse type or genre. According to the aspect under analysis, the corresponding sub-tasks are denoted by Inclusiveness Level Classification (ILC), Register Classification (RC), and Genre Classification Non-inclusivo
Amministrativo
Specialistico Non-inclusive
Administrative
Specialized (GC). The ILC, RC, and GC tasks are treated as separate classification problems, where the input is a sentence and the output is the corresponding aspect value.
formance of models trained using diferent learning approaches, we conducted experiments in both single-task and multi-task learning settings.
In Single-Task Learning, we exclusively focus on the task
of Non-Inclusive Language Reformulation (NILR),
disregarding all aspect-related annotations. We leverage an
encoderdecoder architecture, specifically BART-IT [
Conversely, in Multi-Task Learning, we integrate the NILR task with Aspect Classification tasks during training (i.e., ILC, RC, and GC). For the additional tasks, we specifically leverage the encoder component of the model, which extracts representations of the input text. The encoder component is additionally trained with a classification objective. Each task is associated with a separate classification head, trained to predict the corresponding aspect value for the input sentence. By interleaving these tasks during training, the model learns to simultaneously address NILR and create encoder representations that capture various aspects related to inclusiveness.
Evaluation Metrics. We evaluate the quality of the text
reformulation using a standard train-validation-test split
on our expert-annotated data. To compare the
automatically generated and expected reformulations, we use the
established ROUGE F1-scores [
To complement the quantitative evaluation, we also perform a qualitative evaluation of the achieved results. We involved six human evaluators who were asked to label each model-generated sentence as: correct if it accurately maintained the original meaning while using inclusive language appropriately for the context; partially correct if some aspects were reformed correctly, but others were missed or inaccurate; or not correct if the rewriting fundamentally failed to capture the original meaning or usage intention. This multi-level feedback aims at capturing the models’ ability to perform the rewriting task sensitively across diferent scenarios beyond just string-matching metrics.
To each reformulation, we assign a score to each annotation as follows: 1 for correct, 0.5 for partially correct, and 0 for incorrect. The final score for each reformulation is computed as the average over all the expert annotations ( = 6). Finally, we average the scores for all the reformulations ( = 30) to obtain a single score for each model. Results’ overview. Columns 2, 3, and 4 in Table 3 show the ROUGE scores for both models. The multi-task learning achieves the best performance on all the quantitative metrics. Regarding the human evaluation, we obtained 6 annotations for 30 reformulations for each model. For the model trained with the single task configuration, 93 were correct, 55 were partially correct, and 32 were incorrect. Instead, for the multi-task model, 101 were correct, 49 were partially correct, and 30 were incorrect. Column 5 reports the average human evaluation scores for both models. The human scores are coherent with the quantitative ones, showing that the model trained under multi-task settings beneifts from the additional labels. Based on these preliminary results, we can conclude that the nuanced and multidimensional annotations of inclusive language have the potential to develop a more comprehensive approach to modeling inclusive language.
This paper discussed and experimentally demonstrated that the role and contribution of human annotators are of paramount importance in improving the quality of NLP results and the writing capability of generative approaches in inclusive communication. Starting from a new Italian administrative corpus, we enriched it with a variety of annotations with the help of a team of language experts. This included (i) reformulating gendered language and acronyms, (ii) rewriting to enhance readability for the visually impaired, and (iii) defining the intended context of use (register) and text genre. The preliminary experimental results on the annotated corpus are promising and highlight the potential of the newly proposed annotations to develop a more comprehensive and richer approach that improves the ability of the generative algorithm to propose comprehensive and integrative reformulations.
Limitations. i) The annotation is language-specific , limited to the Italian language, thereby constraining its utility in multilingual scenarios; and ii) It is formal communicationspecific . Tailored to tackle the challenge of inclusive language in administrative and academic settings, the natural language tasks are exclusively trained on administrative documents, potentially lacking suitability for diverse contexts like legal and web communications.
Future work. As part of the E-MIMIC1 (Empowering Multilingual Inclusive Communication) project, we are currently working on a multilingual annotation process to overcome these issues and foster inclusive communication across different domains and languages. A team of experts is annotating a large corpus of documents according to linguistic criteria to label linguistic resources in a multilingual setting.
Finally, we want to exploit text-based explainability
techniques [
Ethical Considerations. All the gathered documents are public and therefore freely accessible on the internet. All references to proper names of people and institutions have been anonymized and replaced with random names for privacy reasons.
This study was carried out within the project ”E-MIMIC: Empowering Multilingual Inclusive Communication”, funded by the Ministero dell’Universitá e della Ricerca - with the PRIN 2022 (D.D. 104 - 02/02/2022) program. 1https://dbdmg.polito.it/e-mimic/