Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These ifndings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.
As Large Language Models (LLMs) become increasingly embedded in educational tools, they are
reshaping how students access academic support [
In multilingual educational settings, this imbalance can have serious implications. Students interacting
with the same model in diferent languages may receive responses that vary not only in fluency, but in
clarity, accuracy, and pedagogical value [
In this paper, we present a scalable, automated pipeline for evaluating how language impacts the
quality of LLM-generated math solutions. Covering generation, translation, solution, and evaluation,
the framework supports consistent multilingual comparison across hundreds of exercises. Using three
commercial models (GPT-4o-mini (OpenAI) [
LLMs are increasingly used in education to support both teachers and students. For educators, they
assist with lesson planning, content creation, and assessment, helping reduce workload and improve
eficiency [
Nonetheless, AI fairness in education is an increasing concern [
These tendencies raise concerns about the inclusivity and cultural relevance of AI-generated
educational content. In mathematical reasoning tasks, LLMs have shown tendency to perform well in
high-resource languages such as English and Chinese but struggle with languages like Korean due
to dificulties processing non-English input [
Such biases have real consequences. In grading and assessment contexts, models may favor Western
argument structures, writing styles, and measurement systems placing non-Western students at a
disadvantage. More broadly, the dominance of English-centric models reflects global educational
inequality, as advanced AI tools remain most accessible to wealthier, English-speaking regions [
This study uses a dataset of 628 math exercises spanning five topic areas and grade levels B through H
(equivalent to Grades 2-10), based on the oficial German K-10 mathematics curriculum [
All exercises were initially written in English, then translated into German and Arabic using the GPT-4o-mini API. Given their concise and context-independent structure, the exercises were well-suited for machine translation. Nonetheless, all translations were manually audited to ensure semantic and linguistic appropriateness. Minor edits were applied as needed to maintain consistency across languages.
Three LLMs were used in this study: GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus. Each was prompted to generate step-by-step solutions in English, German, and Arabic. The prompts, written in the target language, asked the model to explain how to solve the given problem rather than to provide a final answer. This setup was intended to mirror how K-10 learners might seek guided support or clarification. The resulting solutions were saved in separate CSV files for each language-model combination. 1Code and data are available at https://github.com/iug-htw/solution_evaluator.
To evaluate solution quality, we adopted a hybrid evaluation framework in which LLMs acted as judges.
LLM-based evaluation is gaining traction as a scalable alternative to human or code-based assessments
[
For each math exercise, three step-by-step solutions, one in English, German, and Arabic, were presented as alternative responses to the same prompt. These were then assessed and compared by a panel of LLM judges based on their clarity, accuracy, and pedagogical efectiveness.
To ensure impartiality and prevent indirect bias from models evaluating their own outputs, we implemented a held-out judging strategy. In each round, the model under evaluation was excluded from the judging panel and replaced with a neutral model, Claude 3.5 Haiku [17]. The remaining three judges were selected from GPT-4o-mini, Gemini-2.5-flash, Qwen-plus, and Claude.
Prior to evaluation, we identified technical terms relevant to each exercise, as the accurate use and explanation of such terms are crucial for clarity and learner comprehension. To do so, we employed a script that extracted these terms from each exercise using the GPT-4o-mini model.
During evaluation, solutions were presented in randomized order to each judge to mitigate position
bias. Position bias occurs when the order of presentation influences rankings, a phenomenon observed
both in LLMs and human decision-making [
We followed a comparative assessment methodology, which has been shown to outperform direct scoring in terms of alignment with human preferences [19, 20]. Judges were asked to rank the three solutions from 1 (best) to 3 (worst) and provide a short justification to their judgment. Rankings were based on multiple educational criteria: conceptual understanding, clarity of explanation, step-by-step reasoning, correct terminology, accuracy, recognition of common mistakes, pedagogical suitability, generalizability, and grade-level appropriateness.
To identify the best-performing language for each exercise, we applied a majority voting scheme. If two or more judges ranked the same solution highest, the language of that solution was recorded as the top performer. If the judges couldn’t agree on a single best-performing solution, the result was labeled a tie (TIE), ensuring that ambiguous cases were recorded without forcing a winner.
Table 2 summarizes the performance of solutions per language, categorizing them for each LLM based on how frequently they were ranked as the best, mid-tier, or worst. Across all three models, English solutions were most frequently favored. GPT-4o-mini showed the strongest language preference, heavily favoring English and consistently rating Arabic lowest. Qwen-plus followed a similar trend, though with less extreme separation. Gemini-2.5-flash produced a more balanced distribution overall; while English still led in top rankings, the diference between German and Arabic was narrower, and German received the most “Worst” labels in that case.
In addition to collecting quantitative rankings, we asked the LLM judges to provide brief justifications for each evaluation (see examples in Table 3). Analysis of over 4,000 justifications revealed consistent emphasis on clarity, structured reasoning, and appropriate use of mathematical terminology. English solutions were most frequently praised as “comprehensive,” “well-structured,” or “clear,” reflecting strong positive sentiment overall. German responses received more mixed feedback, often described as adequate but occasionally critiqued for being “less detailed” or assuming prior knowledge. Arabic solutions had the highest concentration of negative sentiment, with justifications noting a lack of clarity or depth in explanation. Notably, comparative framing (e.g., “slightly better,” “strongest”) was common, confirming that judges made relative, not absolute, assessments.
The results indicate a clear disparity in performance across languages. English solutions were rated most frequently the best, while Arabic was generally rated the worst, although not uniformly across models. German solutions typically occupied the middle range. GPT-4o-mini showed the most extreme language separation, assigning overwhelmingly high scores to English and low scores to Arabic. Gemini-2.5-flash diverged slightly, assigning the highest number of “Worst” rankings to German rather than Arabic. However, the margin between German and Arabic was relatively small (246 vs. 223), and Gemini’s rankings were more evenly distributed across languages overall.
It is worth noting that, while we did not repeat the full experiment under fixed conditions, the full pipeline was executed multiple times during development with variations in datasets, prompts, and code refinements. Despite fluctuations in exact scores (as expected with stochastic outputs), the relative pattern remained consistent: English was favored, Arabic was disadvantaged, and German remained in between. This recurring trend across settings suggests that the observed disparities are robust rather than incidental.
These gaps were not solely driven by solution correctness. Even when all three solutions were mathematically accurate, models showed clear preferences for responses that included explicit reasoning, pedagogical structure, and generalizability. Analysis of the provided justifications show that the LLMs consistently reward explanations that go beyond getting the answer right to showing why the method works. Another key insight is that each language showed strengths in diferent areas. Arabic was often rewarded for its warmth and accessibility, especially at lower grade levels. German performed well when structure and terminology were key but sometimes assumed prior knowledge. These patterns suggest that language-specific conventions shape both how explanations are presented and how models evaluate their educational value.
Nonetheless, variations in linguistic complexity likely contributed to the observed performance patterns. German has a complex sentence structure with long compound words, flexible word order, [en]: "...well-structured and clearly explains the concept [en]: "...while correct, lacks some of the engaging eleusing appropriate terminology and visual analogies." ments...may not connect with younger students" [de]: "...excels in clarity, structure, and appropriate use [de]: "... although correct, assumes prior knowledge of of technical terms...avoids assumptions..." rearranging the density formula..." [ar]: "...provides a student-friendly approach to solving [ar]: "... presents the information in a slightly less structhe problem...making it easy for a 4th grader to follow." tured manner..." and grammatical rules. These characteristics can make mathematical expression and phrasing more dificult for LLMs to manage, which may explain its intermediate performance. Arabic with its rich morphology and right-to-left script posed even greater challenges. This morphological richness can compress tokens, producing shorter outputs that feel abrupt next to more elaborated English or German responses. Moreover, LLMs may also lack exposure to Arabic instructional texts with formal, school-style structure. As a result, LLMs may misinterpret Arabic’s concise or direct solutions as less educationally complete, even when correct. These surface-level diferences can shape how LLMs perceive clarity and instructional value, even when conceptual accuracy is maintained.
The decision to fully automate the pipeline was driven by the need for scalable, reproducible analysis. Manual evaluation methods, while valuable, limits consistency and scope, making it dificult to compare results at scale. Our automated approach ensures uniform treatment across hundreds of exercises and enables easy replication or extension to new models, languages, or curricula. It also reduces human bias in assessment and supports more systematic multilingual benchmarking.
Finally, the findings carry clear pedagogical implications. Teachers cannot assume equal support across languages, as LLMs may provide stronger guidance in English than other languages. LLMs should therefore be treated as supplementary tools, ideally with teacher oversight or cross-lingual checks. At the same time, the pipeline itself serves as an analytical resource, helping educators pinpoint curriculum areas where linguistic gaps pose the greatest risk, such as topics requiring precise terminology or multi-step reasoning. This allows teachers to adapt instruction accordingly to promote more equitable learning in multilingual classrooms.
This study presents a scalable, fully automated pipeline for evaluating LLM-generated math solutions across languages. Applying it to English, German, and Arabic revealed consistent disparities in explanation quality, with English generally favored and Arabic more frequently disadvantaged. These diferences were not solely about correctness but reflected variation in reasoning depth and pedagogical clarity.
As LLMs become increasingly integrated into classrooms and tutoring platforms, such disparities risk reinforcing existing educational inequalities. Our findings underscore the importance of more linguistically inclusive development and evaluation practices. To ensure equitable access, future work should prioritize improving performance in underrepresented languages through targeted fine-tuning, diverse training data, and culturally informed prompt design. Expanding this evaluation framework to additional languages and domains can support the development of more accessible and fair educational AI systems worldwide.
This paper portrays the work carried out in the context of the KIWI project (16DHBKI071) that is generously funded by the Federal Ministry of Research, Technology and Space (BMFTR).
During the preparation of this work, the authors used OpenAI’s GPT-4.1 to check grammar and spelling and to enhance the writing style. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [16] B. Yuan, J. Hu, An Exploration of Higher Education Course Evaluation by Large Language Models, 2024. arXiv:2411.02455. [17] Anthropic, Claude 3.5 haiku, 2024. URL: https://www.anthropic.com/news/ 3-5-models-and-computer-use. [18] S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. A. Popa, I. Stoica,
JudgeBench: A Benchmark for Evaluating LLM-based Judges, 2024. arXiv:2410.12784. [19] Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulić, A. Korhonen, N. Collier, Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators, in: D. Das, D. Chen, Y. Artizi, A. Fan (Eds.), First Conference on Language Modeling COLM 2024, OpenReview, 2024. URL: https://openreview.net/group?id=colmweb.org/COLM/2024/Conference#tab-accept, ifrst Conference on Language Modeling 2024, COLM 2024. [20] A. Liusie, P. Manakul, M. J. F. Gales, LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models, in: Y. Graham, M. Purver (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, St. Julian’s, Malta, 2024, pp. 139–151. doi:10.18653/v1/2024.eacl-long.8.