Large Language Models (LLMs) have exhibited impressive natural language processing capabilities but often perpetuate social biases inherent in their training data. To evaluate this, we introduce MultiLingual Augmented Bias Testing (MLA-BiTe), a framework that improves prior bias evaluation methods by enabling systematic multilingual bias testing. MLA-BiTe leverages automated translation and paraphrasing techniques to support extended assessments across diverse linguistic settings. In this study, we evaluate the efectiveness of MLA-BiTe by testing four state-of-the-art LLMs in six languages-including two low-resource languages-focusing on seven sensitive categories of discrimination.
Large Language Models (LLMs) have become central to modern Natural Language Processing (NLP)
applications, excelling in tasks such as machine translation [
Enabling non-technical stakeholders, such as HR departments, Ethics Committees, and Diversity & Inclusion oficers, to evaluate AI behavior in multiple languages is essential for trustworthy and inclusive AI development. However, extending bias evaluation to languages that are oficial yet underused in professional settings, such as Luxembourgish or Catalan, poses challenges. In many cases, the people responsible for defining fairness criteria are not fluent in these languages, making it dificult to craft or validate suitable prompts. This problem is exacerbated in countries with many co-oficial or regional languages, such as South Africa and India.
Manual prompt translation and paraphrasing are slow and not scalable. To address this, we explore
whether LLMs can automate these tasks in a reliable way, enabling consistent and culturally appropriate
prompt augmentation for bias evaluation. If successful, this would reduce reliance on native speakers,
lower the barrier to entry for non-technical users, and enable broader participation in AI fairness
assessments. We introduce MultiLingual Augmented Bias Testing (MLA-BiTe), a framework designed
to extend existing methods, particularly LangBiTe [
To address RQ1, we employ in-context learning to generate semantically consistent paraphrases and translations of existing LangBiTe prompts, comparing diferent ordering strategies. For RQ2, we assess the outcomes of these augmented prompts across six languages, including both high-resource (English, Spanish, French, German) and low-resource (Catalan, Luxembourgish) cases. This allows us to analyze whether linguistic resource availability correlates with the extent of detected bias. MLA-BiTe thus provides a scalable, inclusive solution for evaluating LLM fairness in multilingual settings, especially where technical and linguistic resources are limited.
LLMs have achieved widespread popularity and are becoming pervasive for text classification, content
generation, language translation, and text summarization, among many other tasks. However, because
their training typically relies on large datasets derived from web crawls, they often fail to address ethical
concerns and tend to mirror biases prevalent on the Internet [
Language Bias Testing (LangBiTe) [
MLA-BiTe operates exclusively on inputs provided to the underlying framework, such as the PromptTemplates employed by LangBiTe. Because its core logic is decoupled from the specific framework implementation, MLA-BiTe can readily accommodate inputs from other prompt-based bias evaluation frameworks without necessitating alterations to their internal code structures.
Algorithm 1 MLA-BiTe pipeline Input: : PromptTemplates, : a LLM, : set of languages to translate into, : number of paraphrases Output: generated PromptTemplates 1: translator ← initialize_translator() 2: paraphraser ← initialize_paraphraser() 3: ← () 4: for in do 5: transl_pt ← translate(translator, , ) 6: paraph_pt ← paraphrase(paraphraser, transl_pt, ) 7: .append(paraph_pt) 8: end for
Specifically, within LangBiTe, translation and paraphrasing procedures are implemented at the template level, not at the individual prompt level, that is, prior to the instantiation of template placeholders with targeted communities. This choice is justified because a single template with placeholders ! intended for filling from a set of target communities can yield up to !(− )! distinct test prompts. Performing translation and paraphrasing at the template level rather than at the prompt level significantly enhances the eficiency and scalability of the approach.
Moreover, translating and paraphrasing at the individual prompt level would result in prompts derived from the same template being syntactically divergent. This divergence would complicate the interpretation of results, making it challenging to discern whether a failed test prompt is due to variations in the ordering of community placeholders or subtle syntactic diferences. By applying operations at the template level, the approach ensures that generated test prompts are syntactically uniform, thereby enhancing the comparability and interpretability of the evaluation outcomes.
Algorithm 1 outlines the overall workflow of MLA-BiTe. The tool takes as input a list of PromptTemplates ( ), an that acts as both translator and paraphraser, the set of target languages for translating the original , and the desired number of paraphrases for each translation. It is worth noting that separate LLMs could be used for translation and paraphrasing. However, for simplicity, this work assumes the use of a single for both tasks.
Initially, the translator is set up using the , and the paraphraser is configured with the same , along with the specified number of desired paraphrases (lines 1–2). The list of generated PromptTemplates, , is initialized as empty (line 3). Next, each in is translated by the translator into each language in (line 5). The translated output, transl_pt, is then paraphrased times using the paraphraser (line 6). Please refer to Section 4.3 for additional information regarding the choice of this pipeline. Finally, the newly generated PromptTemplates are appended to (line 7).
It is important to note that if is empty, meaning no translation is needed, transl_pt will be identical to . Similarly, if no augmentation is required (i.e., = 0), paraph_pt will be the same as transl_pt.
This section outlines the evaluation setup addressing RQ1, which investigates whether LLM-based
translation and paraphrasing can efectively augment test templates across multiple languages, and
which ordering of these steps yields the most reliable expansions. A preliminary evaluation phase was
conducted to identify the most suitable LLM and configuration.
4.1. Setup
The implementation of MLA-BiTe was carried out in Python 3.11. Four non-reasoning state-of-the-art
LLMs were evaluated: Claude 3.5 Sonnet, Gemini Pro 1.5, Llama3 405b, and GPT-4o. All models were
queried using a temperature of 1 to balance creativity and consistency, with all other parameters
set to default. The evaluations were run between 5 and 7 February 2025 using the latest available
model versions. Test cases were sourced from the LangBiTe GitHub repository [
The initial step was to assess model performance in translation and paraphrasing. For translation, a random 20% of Spanish test cases (61 templates) were selected and translated bidirectionally between English, Spanish, and Catalan, yielding six translations per case. GPT-4o and Gemini 1.5 successfully handled all cases, while Claude 3.5 had a 10% failure rate and Llama3 405b failed only marginally.
Translation quality was evaluated using cosine similarity (semantic alignment) and BLEU (syntactic divergence). GPT-4o consistently achieved the highest average semantic alignment across language pairs, especially for Spanish-Catalan, which also showed the best cross-language performance overall. In the paraphrasing task, models were tested with =2, 5, and 10 paraphrases per template, compared using cosine similarity and BLEU. All models preserved the semantic meaning well, with cosine similarity scores averaging between 0.85 and 0.95. Increasing had negligible impact on semantic preservation, but higher values induced greater syntactic variation. Based on these evaluations, GPT-4o was selected for the main experiments due to its superior mean performance, especially in semantic similarity, and its consistent instruction-following across both tasks.
Following the model selection phase, we evaluated the optimal ordering of paraphrasing and translation, as this sequencing can influence the quality of the final output. We considered two pipelines: paraphrasing-to-translation (P2T ) and translation-to-paraphrasing (T2P). In both cases, we used a fixed number of paraphrases, = 5, and focused on bidirectional translation pairs: English–Spanish and Spanish–Catalan. In the P2T pipeline, the paraphrases were first generated in the source language and then translated into the target language. Conversely, in the T2P pipeline, the original template was ifrst translated and then paraphrased in the target language. Each output sentence was compared to its corresponding human-written input using cosine similarity to assess semantic preservation.
As shown in Figure 1, P2T exhibits slightly higher semantic similarity in the EN→ES and ES→CA directions, while T2P performs marginally better in the reverse directions. Overall, the impact of pipeline ordering is limited. For the main evaluation, we adopt the T2P configuration due to its comparable performance and slightly more consistent behavior across languages. Further research is needed to fully explore language-specific efects and confirm these findings more broadly.
In Section 4, we addressed RQ1, demonstrating that LLM-based translation and paraphrasing efectively
augment bias-testing templates. We also observed that applying paraphrasing before translation yields
slightly better results than the reverse. In this section, we address RQ2, namely whether low-resource
languages exhibit more bias than high-resource languages when tested with augmented multilingual
templates. All raw results are available in our public GitHub repository [
We focus on six languages from two major Indo-European families, Romance and West Germanic, to ensure both linguistic diversity and coverage of high- and low-resource settings. Selection is based on typological variation, geographic spread, and the availability of ground truth data.
Romance languages: Spanish (ES), Catalan (CA), and French (FR) represent varying resource levels and regional contexts within the Romance family. Spanish and Catalan are included from the original LangBiTe study, while French serves for cross-validation.
West Germanic languages: English (EN), German (DE), and Luxembourgish (LB) cover a comparable resource range. English and German are high-resource, while Luxembourgish ofers a low-resource counterpart analogous to Catalan, supporting comparative analysis across families.
The set of templates described in Section 4.1 was used for the main experiment. English served as
the source language, from which the test cases were translated into the target languages. For the
paraphrasing component, we set the number of variations to = 1. The communities analyzed in this
study are the same as those considered in [
Figure 2 presents a series of spider (radar) plots illustrating each LLM’s performance across the
sensitive categories for each language included in this study. Hereafter, we define each unique
concernlanguage combination as a test batch. Within each plot, the radial axes represent the percentage of tests
passed by a given model for a particular test batch, thus enabling a direct comparison of how efectively
diferent LLMs handle sensitive content. Note that these results reflect only tests for which valid and
interpretable answers were obtained. Although the framework allows up to three retries per test, some
responses remained unprocessable. As described in [
Several noteworthy observations emerge from Figure 2. First, English and Spanish consistently yield the highest scores across the bias categories, irrespective of the model. This finding aligns with earlier results indicating that widely used languages with substantial training corpora tend to produce more accurate automated bias-detection outcomes. By contrast, Catalan and Luxembourgish exhibit greater variability in categories such as Politics and Racism, likely because lower-resource languages contain sparser training data that may limit the models’ ability to handle culturally specific terms and nuances.
The models themselves also vary in their performance. GPT-4o generally achieves high scores across most categories, particularly Ageism, Sexism, and Xenophobia, indicating strong coverage of related keywords and contexts. Gemini 1.5 Flash often excels in Religion and Lgbtiqphobia, suggesting it can efectively capture nuanced expressions of bias across languages in these domains. Meanwhile, Claude 3.5 Sonnet typically maintains moderate to high consistency in Sexism and Racism across multiple languages but sometimes fluctuates in Politics, reflecting challenges associated with localized political terminology. Llama3 405B demonstrates comparatively mixed results: it excels in certain instances of Racism and Ageism, yet may underperform in categories such as Politics or Xenophobia for lower-resource languages. For categories like Lgbtiqphobia and Xenophobia, all four LLMs exhibit relatively high detection rates in most languages. This consistency may stem from the more universal nature of terms referring to LGBTIQ+ identities or xenophobic attitudes. By contrast, Politics emerges as the most variable concern, with each model showing inconsistencies across diferent languages.
Similarly, Sexism and Ageism produce mid-range consistency across models, suggesting that while many overtly disparaging or discriminatory terms are well covered, subtler connotations may elude LB LB
CA FR CA FR
ES DE ES DE
FR CA FR
DE ES DE
CA FR 80 ts
s 70 Teed
s 60 fsPa
o 50 taeg
n 40 rcee
P 30
ES DE Religion
Sexism straightforward keyword matching or demand deeper contextual understanding. Lastly, Religion tends to be comparatively stable across both languages and models, presumably due to shared or borrowed religious terminology and the availability of well-established keywords that more readily transfer from English prompts to other languages.
Figure 3 aggregates the results shown in Figure 2 by language and model, alongside the mean outcomes. As depicted, Llama3 405B is the most biased LLM overall, while GPT-4o and Claude 3.5 Sonnet exhibit the strongest overall performance, with scores around 75%. Regarding performance by language, models generally perform best on high-resource languages, achieving their highest average scores in English, and appear to exhibit more social biases when tested on lower-resource languages. Notably, Luxembourgish stands out as the language with the highest discrimination rates overall. GPT-4o on Catalan, however, is an outlier, achieving the second-best score among all language-model pairs.
A limitation of our setup is that GPT-4o was used both as the translation/paraphrasing model and as
one of the evaluated LLMs. This dual role may confer a slight advantage to it, by aligning evaluation
inputs more closely with its own generation style. While this choice ensured consistent and high-quality
multilingual augmentation, it may act as a confounding factor, an issue made more salient by recent
evidence that LLMs can propagate hidden patterns across tasks, even when such patterns are not
semantically related to the target outputs [
Given the variance observed in Figure 2 across diferent bias categories, it is also evident that choosing an LLM may require a case-by-case approach. Individual models can exhibit strong performance in some categories while underperforming in others, especially when targeting localized cultural or linguistic nuances. Hence, a nuanced selection process that accounts for both language and bias category may be necessary to optimize bias detection and mitigation.
In conclusion, and in direct response to RQ2, these findings suggest that LLMs exhibit higher social biases when data augmentation is performed for low-resource languages. Nonetheless, the model best suited for each task may vary depending on the specific bias category and language under consideration.
We complement the results presented in Section 5.2 with a Pearson correlation analysis on the performance of the same model/concern pairs across diferent languages. This analysis highlights both common patterns and divergences in behavior across languages. The outcomes (Figure 4) reveal that, contrary to initial expectations, LLMs do not consistently exhibit comparable biases in linguistically related languages. For instance, while German and English (both West-Germanic languages) display the highest performance similarity across all language comparisons, the biases observed in Luxembourgish are more closely aligned with those detected in Spanish and Catalan than with German or English. A more granular examination of individual bias dimensions (see Figure 2) further underscores these unexpected findings. Notably, LLMs display marked performance variations across several categories of bias, including Ageism, Lgbtiqphobia, Racism, and Sexism. For example, GPT-4o performs comparatively poorly in the Racism category for Catalan and French, whereas Gemini 1.5 exhibits pronounced diferences in Sexism performance between these two languages. Collectively, these observations indicate that linguistic proximity does not necessarily translate into similar bias patterns across diferent LLMs.
From Figure 2 it also emerges that political bias is a notable outlier to our observations. In evaluating
the political bias of language models, it is essential to highlight the limitations of LangBiTe’s default
template library, and their obtained paraphrases, particularly when the queries are predominantly
centered around U.S politics and require a neutral stance. What we see in Figure 2 is that most models
take an ideological side when prompted about U.S. political issues, whereas the oracles expect no
positioning at all. Nevertheless, while LangBiTe’s templates provide valuable insights into U.S.-related
political leanings from generative AI models, they may not fully capture the diferences and complex
nuances of political discourse in other countries and languages. Political ideologies and the framing
of society matters can vary significantly across diverse national or regional contexts. In addition,
political ideologies and stances tend to evolve over time and are generally too complex to be placed
on a one-dimensional spectrum [
As mentioned in Section 5, not all LLMs duly follow LangBiTe’s output formatting instructions. This leads to computing errors, since the response may not be correctly interpreted—not even by the LLM-as-judge. Most of the templates requiring a structured output are targeting Sexism (42 out of 65 templates) and Racism (46 out of 98), resulting in a higher number of errors for these ethical concerns.
Across the four LLMs, clear patterns emerged when using prompts in Catalan. Claude 3.5 Sonnet often focused on grammar instead of addressing the intent of prompts for Lgbtiqphobia, which clearly afected the results. A similar phenomenon occurred with GPT-4o in Xenophobia tests. Probabilistic tests were a challenge for Claude 3.5 Sonnet, whereas Gemini and GPT-4o demonstrated their neutrality across ethical concerns for such prompts. On the other hand, Llama3 405B struggled to follow JSON formatting instructions. Positive discrimination toward women is suggested by responses from GPT-4o and Llama3 405B. Common weaknesses across models included political leanings, discriminatory responses towards 20-year-olds, and inconsistencies in religion-based conflict scenarios.
We find similar patterns when analysing responses for the Luxembourgish language. Concerning Lgbtiqphobia, LLMs typically state whether a given sentence is written in correct Luxembourgish rather than giving an opinion on the presented claim. Other instances of not understanding the intent of the given prompt include instances of Llama 405B misinterpreting the term ’be worth less than’ as meaning ’having fewer of’ when presented with the question whether one ethnic group is worth less than another. Such misunderstandings often can lead to false positives when evaluating the outputs of the LLMs. Similarly to Catalan, Claude and Gemini tend to struggle with producing probabilities while GPT-4o and Llama 405B output neutral responses. However, when presented with questions regarding the probability of given ethnic groups being able to do given professions, Claude and Gemini usually give neutral answers while GPT-4o and Llama 405B fail to produce probabilities or give biased answers. With regard to political biases, Claude was the only LLM explicitly stating to be neutral while other LLMs either show consistent left/liberal biases or produce conflicting answers.
This study introduced MLA-BiTe, a framework that enhances bias evaluation by enabling systematic multilingual testing. Leveraging automated translation and paraphrasing, we generated input templates compatible with LangBiTe and tested the approach on four LLMs across six languages, including both high-resource (English, Spanish, French, German) and low-resource (Catalan, Luxembourgish) cases. Our first research question examined whether LLM-based translation and paraphrasing can efectively augment bias-testing templates. Results show that both methods improve comprehensiveness, with paraphrasing before translation yielding slightly more reliable outcomes. The second question explored whether low-resource languages display higher bias levels. Our findings confirm that these languages tend to show more variability in bias detection, especially in sensitive categories such as Politics and Racism, suggesting that training data richness significantly impacts model consistency.
The evaluation also revealed model-specific diferences, indicating that architecture and data composition shape how biases manifest. Interestingly, linguistic similarity did not strongly correlate with bias similarity, suggesting that cross-linguistic bias patterns are not solely dictated by language family.
Future work will expand the evaluation to more LLMs and a broader set of languages, particularly
outside the Indo-European family, where typological features such as noun class and verb morphology
may require specialized evaluation strategies. We also plan to extend the framework to the image domain
through ImageBiTe [
Ultimately, MLA-BiTe provides a scalable path toward inclusive and context-sensitive bias evaluation. Continued expansion and adaptation will be essential for addressing the nuanced challenges posed by multilingual and multimodal AI systems.
This work has been partially funded by the Luxembourg National Research Fund (FNR) PEARL program (grant agreement 16544475); by Citcom.ai, co-funded by EU/Digital Europe and, in Luxembourg, by the Feder; the research network RED2022-134647-T and the project PID2023-147592OB-I00 “SE4GenAI”, both funded by MCIN/AEI/10.13039/501100011033 The authors have not employed any Generative AI tools.