Recent Large Language Models (LLMs) have shown impressive performance in addressing complex aspects of human language. These models have also demonstrated significant capabilities in processing and generating Italian text, achieving state-ofthe-art results on current benchmarks for the Italian language. However, the number and quality of such benchmarks is still insuficient. A case in point is the “Open Ita LLM Leaderboard” which only supports three benchmarks, despite being one of the most popular evaluation suite for the evaluation of Italian-language LLMs. In this paper, we analyze the current limitations of existing evaluation suites and propose two ways of addressing this gap: i) a new suite of automatically-translated benchmarks, drawn from the most popular English benchmarks; and ii) the adaptation of existing manual datasets so that they can be used to complement the evaluation of Italian LLMs. We discuss the pros and cons of both approaches, releasing our data to foster further research on the evaluation of Italian-language LLMs.
CEUR
ceur-ws.org
1. Introduction
showing impressive results on an increasing range of
standard benchmarks, thanks in particular to their
reasoning and in-context-learning capabilities [
As of today, most of the existing Italian benchmarks
are translations of English datasets, which may not fully
capture the nuances and unique characteristics of the
Italian language. Nevertheless, the ability to automatically
translate English benchmarks into Italian is valuable and
enticing for two main reasons. First, it provides a way
tions of existing datasets from English and the creation
of brand-new datasets in Italian, there is the option of
adapting existing Italian datasets that were originally
created for a diferent purpose, to measure the capabilities
way to evaluate and compare models, and this is now es- ities of modern LLMs to be fully analyzed, even though
tion. This direction has gained traction over the past few LLM Leaderboard”. This is a collection of three datasets –
months, with eforts that focus on repurposing Italian HellaSwag [
In this paper, we follow both directions and intro- of high-quality (thanks to the fact that the translations duce ITA-Bench, a more comprehensive benchmark suite were produced using GPT-3.5), there are still several isfor the evaluation of Italian-language LLMs. First, ITA- sues that limit the quality of this evaluation suite: Bench proposes a new extended suite of benchmarks Coverage: Open Ita LLM Leaderboard only covers created by automatically translating the most popular three benchmarks. There are plenty of other datasets English benchmarks into Italian. Second, ITA-Bench in- that are generally used to test the capabilities of LLMs cludes existing manually curated datasets, adapted to en- in English, so limiting the assessment of Italian LLMs to hance the evaluation framework for Italian LLMs. These just three datasets may result in the evaluation of some two complementary approaches aim to bridge the eval- important aspects of their capabilities in Italian being uation gap and provide a more thorough understand- overlooked. ing of the capabilities of Italian-language LLMs. With Reproducibility: The code and models used to transITA-Bench, we hope to foster further development and late these three benchmarks are not directly available, refinement of evaluation techniques for Italian LLMs, making it hard – if not impossible – to reproduce the ultimately contributing to the broader field of multilin- translations.3 gual NLP. ITA-Bench is available at https://github.com/ Transparency: The fact that the translations are not sapienzanlp/ita-bench. reproducible makes it dificult to analyze whether there are errors or there is margin for improvement in the 2. ITA-Bench: a New Evaluation translation process originally used to translate the three benchmarks.
Suite for Italian LLMs English specificity: Despite the translation process, these benchmarks actually remain tied to the English lanIn this section, we introduce our methodology for the guage. Indeed, the prompts used as input to the language creation of ITA-Bench, a more comprehensive evalua- model contain parts that are in English (for example, in tion suite for Italian LLMs. Our objective is to focus on the creation of the examples used for few-shot evaluathe Italian language and, more specifically, to create a tion). This is undesirable because it inherently favours benchmark suite that is able to test a wide variety of LLMs that are bilingual, more specifically, LLMs that can aspects of LLMs that “generate” Italian text. To accom- “speak” fluent English in addition to Italian. plish this objective we focus on two distinct directions: Uniformity: The translation of benchmarks from Eni) translating existing English benchmarks that are cur- glish to a target language is usually done on a benchmarkrently used to evaluate the capabilities of state-of-the-art by-benchmark basis. On one hand, this allows developers LLMs in English, and ii) adapting existing Italian bench- to specialize the translation code to each dataset; on the marks, drawing from popular repositories, conferences, other hand, this approach prevents the translation proshared tasks, and community initiatives, such as the sev- cess from being comparable across datasets, which makes eral EVALITA editions1 and SemEval tasks.2 In the case performing a root-cause-analysis on the origin of an error of adaptation of existing datasets, most of the work con- in the translated dataset more complex. sists in adapting the scope of the tasks, i.e., since many of these tasks were not designed to evaluate LLMs, the core of the work lies in reframing the problem in a way 2.1.2. Re-translating English benchmarks that a prompt can be used to test the capability of a particular LLM to solve a specific task. Table 1 reports the overall statistics of the datasets that we consider for our ITA-Bench suite.
dressing the issues that are present in existing benchmark translations, including the ones used in Open Ita LLM Leaderboard. More specifically, we introduce a new library called OBenTO (Open Benchmark Translation for the Others) that is designed to translate existing benchmarks in a uniform, reproducible and fullytransparent way. Moreover, it is also designed to be easily extensible, in such a way that the research community can add new benchmark translations and even
marks is not known. Also note that OpenAI has already deprecated many GPT 3.5 versions.
Italian produced via translation is perhaps the “Open Ita
2https://semeval.github.io/
new languages besides Italian. We release OBenTO at campaigns and the SemEval shared tasks. These sources
https://github.com/sapienzanlp/obento. provide Italian data and annotations for a variety of tasks,
covering a broad spectrum of linguistic capabilities and
Translation model. The OBenTO library is designed phenomena in the Italian language.
to be easily adaptable to new backbones, but at the time of The key step in adapting these Italian benchmarks –
writing this article, the library relies on TowerLLM [
MMLU [
PIQA [
Winogrande [24, WG]: a commonsense reasoning dataset that requires choosing between two options based on coreference resolution.
Multi-domain Documents was first presented at EVALITA-2023. The task uses standard NER classes, namely, Person, Organization, and Location, to tag entities in a text. In ITA-Bench, we adapt NERMuD and create task instances comprised of three elements: i) the sentence that contains the entity mention, ii) the mention of the entity in the sentence, and iii) the correct class associated with the mention in the given
Dataset Train set Valid set Test set PRELEARN [28]: Prerequisite RElation LEARNARC-C 1068 286 1132 ing is a task from EVALITA 2020 on concept prerequisite ARC-E 2157 549 2258 learning. This task consists in identifying whether a BGoSoMlQ8K 97349793 32-59 13-19 concept A is a prerequisite of another concept B, i.e., HS 39722 9998 - if learning concept B requires having already learnt MMLU 269 1402 13127 concept A. The original dataset comes with four domains, PIQA 15038 1713 - namely, Geometry, Precalculus, Physics, and Data Mining, STcruiQthfulQA -- 799823 9-85 and we maintain these same domains in ITA-Bench. Winogrande 4717 1176 - WiC [29]: Word-in-Context for Italian. We focus AMI 7014 - 2908 on the binary-classification sub-task of the original WiC 2805 500 500 formulation. In ITA-Bench, an LLM is tasked with NERMuD 14529 4079 3943 determining if a word occurring in two diferent PPrReETLEENASRN 52833278 -- 14566909 sentences 1 and 2 has the same meaning in 1 and 2. DISCOTEX 16000 - 1600 QUANDHO [30]: The QUestion ANswering Data GhigliotinAI 62 - 553 for Italian HistOry dataset is an Italian questionQUANDHO 384 - 1416 answering dataset focused on Italy’s history during the Table 1 ifrst half of the 20th century. It provides Wikipedia Statistics of the ITA-Bench datasets, for each dataset the car- passages that may contain the answer to specific dinalities of the training, validation and test set are reported. questions. Each question in the dataset appears in multiple (, ) pairs, where the answer can be either correct or incorrect. In ITA-Bench, we select context. We distinguish between two subdomains: ADG, the pair with an answer marked as correct and three writings and speeches from the Italian politician Alcide distractors from the occurrences of incorrect answers De Gasperi, and WN, news texts from the past decades. paired with the same question.
DISCOTEX [27]: Assessing DIScourse COherence in GhigliottinAI: Starting from two diferent EVALITA Italian TEXts is a task focused on modelling discourse tasks, nlp4fun [31] and ghigliottin-AI [32], we collect coherence in real-word Italian texts. In ITA-Bench, we about 600 diferent games extracted from the TV focus only on the first sub-task of DISCOTEX: Last Sen- show and the boardgame of “L’Eredità”, a popular tence Classification , where, given a short input paragraph quiz game in Italy. “La Ghigliottina” is a challenging and a sentence, the goal is to tell whether the sentence game that requires extensive knowledge of the Italian is a valid continuation of the paragraph. To assess the culture. The goal is to find a single word that links five capability of an LLM to solve this task, we reframe seemingly unrelated words. However, since multiple DISCOTEX as a multi-choice question answering task. solutions are often possible and computing all potential More specifically, given an input paragraph, the LLM is answers is impractical, in ITA-Bench, we reframe the tasked with selecting the most appropriate continuation problem as a multi-choice question answering task, from among five options that we provide (the original i.e., a simplified version in which four possible words dataset does not provide distractors). Therefore, for the are given and, among these, only one can be linked to subset of instances with valid continuations, we create all the five input words. In ITA-Bench, we also select a set of distractors by sampling continuations from three distractor words in such a way that the distractors other instances at random. Instead, for the instances are linked to three of the five input words. We ensure with invalid continuations, we create a new correct that the distractors are not too similar one to the other option “nessuna delle precedenti” (none of the above), and by maximixing the cosine distance of their FastText add a set of four random distractors from other instances. embeddings. The distractors are also designed to be at PreTENS4: Presupposed Taxonomies was first most one character shorter or longer than the correct proposed for SemEval-2022. This task focuses on word, resulting in a task that is easy for humans but semantic competence, and evaluates the ability of challenging for LLMs. an LLM to recognize valid taxonomic relationships between two nominal arguments. For example, this 3. Evaluation Results can require recognizing whether or not a concept is a subclass of another concept. In ITA-Bench, an LLM is tasked with identifying whether the relationship between two concepts in the same sentence is acceptable.
on ITA-Bench: we first present the results on the automatically-translated benchmarks and then on the adapted benchmarks. ITA-Bench implements all the task formulations using the lm-evaluation-harness li
Size Name 0.4B Minerva-350M-base-v1.0 1B Minerva-1B-base-v1.0 3B OpenELM-3B 3B XGLM-2.9B 3B Minerva-3B-base-v1.0 7B OLMo-7B-0724-hf 7B LLaMAntino-2-7b 7B Minerva-7B-base-v1.0 7B Mistral-7B-v0.1 8B Llama-3.1-8B 7B Mistral-7B-Instruct-v0.1 7B Maestrale-chat-v0.4-beta 8B LLaMa-3.1-8B-Instruct 8B LLaMAntino-3-ANITA 9B Italia-9B-Instruct-v0.1 brary [33], which allows us to calculate the likelihoods the results on GhigliottinAI, Italian LLMs seem to perfor each possible continuation in a simple and compa- form well and surpass the results obtained by English rable way, as lm-evaluation-harness is also used by models. This may indicate that this task needs a diferHugging Face for the Open LLM Leaderboard. ent type of competence and/or knowledge in order to be solved. Indeed, we hypothesize that the task requires 3.1. Automatic Translation a deeper understanding of some elements of the Italian culture, e.g., entities and concepts that are more comThe results of various LLMs on our translated bench- monly known in Italy than in other countries. Therefore, marks are reported in Table 2, which provides an pretraining and fine-tuning on Italian documents might overview of the zero-shot scores on cloze style task for- be the key to obtaining better results in GhigliottinAI. mulations, i.e., the input prompt to an LLM includes only the question without the possible answers. More specifically, we compare the results of several open-weight 4. Manual Error Analysis LLMs having diferent sizes, ranging from less than 1B parameters up to 9B parameters and focusing on LLMs that have been pretrained, fine-tuned and/or adapted on/to the Italian language. As we can see, the scores of the LLMs are roughly correlated to their size in terms of number of parameters. Notably, the smaller versions of the Minerva LLMs are able to compete with larger models, thanks to the fact that a significant portion of their pretraining dataset is composed of Italian text (rather than English).
automatically-translated data, we conduct a manual
error analysis. To this end, we examine the translations
into Italian produced by four language models: two
open-source ones, namely, TowerInstruct-7B-v0.25 and
TowerInstruct-Mistral-7B-v0.26 [
reports the scores of diferent state-of-the-art models, As the source of the data for our linguistic analysis, we ranging from 350M parameters models to 9B parameters. rely on the ARC dataset, which includes multiple-choice Here, we focus on the results of the LLMs in cloze style question answering in a wide range of domains. Specifitasks, except for QUANDHO and DISCOTEX, as ITA- cally, we randomly select a sample of 100 instances from Bench supports only the multi-choice formulation for these two tasks. Unsurprisingly, the size of the LLMs and their pretraining data are discriminators for reaching better results. Most importantly, even the strongest Italian LLMs, such as ANITA, still struggle to compete against their English counterparts. However, as we can see from 5https://huggingface.co/Unbabel/TowerInstruct-7B-v0.2 6https://huggingface.co/Unbabel/TowerInstruct-Mistral-7B-v0.2 7We employ the OBenTO pipeline to process the translations generated by the open-source models. As for GPT-3.5-turbo, we use the translations available at: https://huggingface.co/datasets/ alexandrainst/m_arc. We also translate the datasets using GPT-4omini with a pipeline similar to the one used for GPT-3.5-turbo. the ARC Challenge dataset and we manually analyze the employ a new annotator to assess their quality based on quality of the translations produced by all language mod- our guidelines. We obtain a Cohen’s kappa of 0.85, which els considered. For each instance, we assess the degree of indicates a strong agreement. comprehensibility and fidelity of the translation of both questions and answers, assigning a binary label which 4.2. Results indicates whether a translation is acceptable or not. Crucially, we distinguish between minor and major errors Our analysis shows that GPT-4o-mini outperforms all its depending on the impact on the comprehensibility and competitors. With an error rate8 of 4%, it is markedly ifdelity of the target translation. We then identify error more accurate than TowerInstruct-7B-v0.2, which expatterns, some of which we describe below, highlighting hibits an error rate of 23%. TowerInstruct-Mistral-7Bthe cases in which the translation impedes understanding v0.2 and GPT-3.5-turbo show a similar performance, that of either the questions or the answers, or fails to faith- is, 8% and 9% error rate, respectively. Finally, the most fully reproduce the source text, thus altering the original frequent error patterns are omissions, especially when meaning. Finally, we discuss the results of our analysis. considering open-source models, and incorrect terminolAnnotation guidelines are reported in Appendix A. ogy.
5. Conclusion As part of our manual annotation process, we identify error patterns, of which we report four key ones, namely: In this paper, we introduce a novel evaluation suite i) omissions, which consist in omitting one or multiple aimed at advancing the Italian community’s ability to source words in the translation; ii) incorrect terminol- assess the competencies of LLMs on Italian data. Our ogy, that is, the incorrect translation of one or multiple approach follows two main directions. First, we define terms into the target language; iii) untranslated source a novel pipeline called OBenTO, which involves transtext, where one or multiple source words are reported lating state-of-the-art English benchmarks into Italian. as-is in the translation, despite these words not being Second, we rephrase existing Italian benchmarks to be commonly used in the target language; and iv) grammat- used for prompting and testing large language models. ical errors, which include orthographic, morphological Additionally, we conduct a comprehensive evaluation and syntactical errors. Instances of the aforementioned of the quality of automatically translated benchmarks, error patterns can be found in Appendix B. highlighting the inherent challenges of such an approach and analyzing the errors made by four LLMs. We hope 4.1.2. Inter-annotator agreement that our work can provide a solid evaluation framework for evaluating the capabilities of current and future LLMs In order to assess the reliability of our manual annota- in Italian. tions, we compute the inter-annotator agreement. With this aim in view, we select the already-annotated trans- 8We emphasize that this error rate does not provide a nuanced evalulations produced by one randomly-chosen model and sauticohnaosf flutheencayfoarenmdeidnitoiomnaetdicaintyd. other crucial aspects of translation, Acknowledgments
project PE0000013-FAIR, which fully funds his fellowship. Federico Martelli and Roberto Navigli acknowledge the support of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content, Progetti di Interesse Nazionale - PRIN 2020). Finally, we acknowledge the work of the M.Sc. students of Prof. Navigli’s multilingual NLP course for the Academic Year 2024, whose contributions provided valuable insights and ideas for the adaptation of existing Italian benchmarks. We acknowledge the CINECA award IsB28_medit under the ISCRA initiative for the availability of high-performance computing resources and support.
A. Annotation Guidelines
adopted to ensure consistency throughout our analysis. Annotators receive a document containing the source text and the translations produced by four language models,
translation does not align with the context of the source text. 7. Addition of one or more words: Additional words or phrases (not present in the source text) are included in the translation. [...] Un uomo con una combinazione di alleli RR per il tratto produce uno zigote con una donna con una combinazione di alleli rr per il tratto. Quale combinazione di alleli potrebbe verificarsi nello zigote?
B. Examples of key error patterns
reported in Table 10. We present all prompts in the same
format as the LM-Evaluation-Harness implementation. To
In this section, we report examples of the key error pat- ensure clarity and conciseness, we use Jinja templating9
terns described in Section 4.1.1. Specifically, we report in- for all prompts.
stances of omissions (Table 4), incorrect terminology (Table
5), untranslated source words (Table 6) and grammatical
errors (Table 7). Errors are highlighted by square brackets D. In-domain Results
in red. Importantly, all examples in the aforementioned
Tables are considered major errors, with the sole excep- PRELEARN and NERMuD have been reported as average
tion of the first instance of omission reported in Table 4. accuracies on the main part of this paper. Results are
Specifically, the omission of the word pests has a limited reported in Table 11 and Table 12, looking at each domain
impact on the comprehensibility and fidelity of the trans- separately for the two diferent tasks. While results for
lation and, therefore, for the purposes of the task at hand the zero-shot setting are reported in Table 13 and in
and our analysis, the translation is considered accept- Table 14. We reported the results twice, dividing the
able. As for untranslated source words, we note several multi-choice and cloze style prompt setting.
issues in the data. As reported in Table 4, we note that
GPT-4o-mini translates the term weathering as the Italian E. Other results for adapted tasks
equivalent of erosion. However, weathering and erosion
are two diferent geological processes. In fact, weathering In this section we report other results about adapted
(which could be translated into Italian as degradazione tasks. More precisely, in Table 15 are collected the metrics
meteorica) refers to the breaking down of rocks and min- for the adapted tasks in zero shot setting, where all the
erals at their original location through physical, chemical, tasks are proposed in cloze style prompting, except for
or biological means, without the material being moved DISCOTEX and QUANDHO which are reported in
multielsewhere. In contrast, erosion involves the removal and choice prompting.
transportation of weathered material by agents such as Since we employed a Multi-Choice (MC) style
promptwater, wind, or ice. Hence, in translating weathering as ing for all adapted tasks. Table 16 presents the results
the Italian equivalent of the word erosion, the model fails for these tasks in the zero-shot setting, while Table 17
to capture the precise meaning of the source term, signif- shows the results in the five-shot setting.
icantly altering the content of the source text. Our error
analysis also shows that MT systems still struggle with
disambiguation of concepts [
C. Adapted Tasks Prompts
adapted tasks. The cloze style prompts are reported in Table 8, while multi-choice-style prompts can be seen in Table 9. For each task we also defined a system prompt, which consists of a text prepended to the model before the sample prompts, the proposed system prompts are
TowerInstruct-7B-v0.2 Una cellula [eugleno] possiede una struttura chiamata [macula occhiolare] che rileva la luce. Un [parameno] non possiede una [macula occhiolare] e quindi non riesce a rilevare la luce. Perché un [parameno] non ha bisogno di una [macula occhiolare]? A plateau is most likely formed by [a] runof from a river. [b] weathering by waves. [c] erosion of rock debris. [d] a buildup of cooled lava.
Un plateau è più probabilmente formato da [a] deflusso da un fiume. [b] [erosione] da onde. [c] erosione di detriti rocciosi. [d] un accumulo di lava rafreddata. In un bicchiere è [stato versato] dell’acqua fino a metà. Sono stati messi cinque cubetti di ghiaccio nel bicchiere, facendo sì che il livello dell’acqua raggiungesse il bordo del bicchiere. Quale delle seguenti afermazioni spiega al meglio l’aumento del livello dell’acqua? Which of the following is most likely an adaptation that resulted from habitat destruction?
Qual è più probabile un[’]adattamento che è risultato dalla distruzione dell’habitat? PRELEARN NERMuD PreTENS PRELEARN QuandHO DISCOTEX GhigliottinAI WiC PRELEARN QuandHO DISCOTEX GhigliottinAI WiC
Indica se i seguenti tweet presentano caratteristiche misogine.
Data una frase e un’entità, indica se tale entità rappresenta un luogo, un’organizzazione o una persona. Indica se le seguenti frasi hanno senso.
Dati due concetti A e B, indica se il primo concetto è un prerequisito per il secondo.
Il concetto A è prerequisito per il concetto B, se per comprendere B devi prima aver compreso A. I seguenti concetti appartengono al dominio: {{domain}}.
Ti saranno poste domande di storia italiana.
Identifica quali paragrafi contengono la risposta alle domande date.
Ti verranno poste delle domande nelle quali è presente un paragrafo, e come possibili risposte varie frasi che possono essere o meno la continuazione del paragrafo.
Indica la frase che rappresenta la continuazione più probabile del paragrafo, oppure “nessuna delle precedenti” se nessuna delle continuazioni è corretta.
Ti viene chiesto di risolvere il gioco della ghigliottina.
Il gioco della ghigliottina consiste nel trovare un concetto che lega cinque parole date. Tale concetto è esprimibile tramite una singola parola.
Date due frasi, che contengono un lemma in comune, indica se tale lemma ha lo stesso significato in entrambe le frasi. WN WN Minerva-350M-base-v1.0 Minerva-1B-base-v1.0 XGLM-2.9B OpenELM-3B Minerva-3B-base-v1.0