Introduction

Large Language Models (LLMs) are models based on the Transformer architecture capable of solving a wide variety of Natural Language Generation (NLG) tasks, even those not encountered during training, due to their extensive training and large number of parameters. Thanks to their remarkable skills, interest in LLMs is now at its climax, resulting in a proliferation of open-weight models (e.g. LLaMA, Mistral, and many others). Among the several challenges related to the development of LLMs, one of the most critical is their evaluation [1]. One approach to tackle this issue has been to build benchmarks that collect different datasets, with the aim of obtaining a more comprehensive evaluation of the model's overall capabilities. Currently, there is a leaderboard 1 [2] which 1 https://huggingface.co/spaces/open-llm-leaderboard/open_llm_ leaderboard keeps track of the capabilities of openly available LLMs. Specifically, the models are tested on six tasks that span different abilities a language model should have, e.g. reasoning or text completion. Regarding their reasoning abilities, the models are tested by solving closed-ended tasks. Specifically, multiple-choice question answering tasks are provided, where a question is given with a list of possible alternatives associated with an identifier (a letter, a number, and so on). Intuitively, since the model has also been pre-trained on closed-ended question-answering data, it should be able to generalize and understand the correct choice out of the available ones. Furthermore, rather than generating the output directly, the probabilities learned by the model are studied, using log-likelihood to assess which option is more likely to be correct. For the English language, this evaluation methodology has been a standard approach to assess the capabilities of LLMs. However, when adapting a model to a new language, due to the low amount of non-English data that has been used to pre-train such models, this methodology may not be as sound. The model only has to generate the correct option identifier, therefore this is not really testing the ability of the model of generating high-quality text in another language. The goal of this work is to understand whether a new evaluation setting applied to languageadapted LLMs may give more insight than the traditional approach. Therefore, our contributions are the following:

• We test two evaluation settings for languageadapted LLMs changing the structure of closed-ended question answering tasks; • We evaluate the performance of state-of-the-art models on these settings; • We study the sensitivity that the models have for the input prompt.

Related Works

Language Model evaluation has been a research focus ever since the first Decoder-only models, which were designed for natural language generation. One of the most remarkable skills regarding LLMs reasoning has been in-context learning. In particular, fewshot learning has been increasingly used. The idea is that providing examples of input-output in the model prompt should affect positively the generation process [3].

There are multiple leaderboards which evaluate open LLMs on non-English languages, e.g. Open PL LLM Leaderboard [4] for Polish or Open KO LLM Leaderboard [5] for Korean. These leaderboards are often based on the lm evaluation harness framework [6], which has been a milestone in the evaluation of LLMs. LLM evaluation can also depend on the topic at hand. There are some works which focus on mathematical reasoning [7] as well as factuality [8].

These evaluation settings often rely on closed-ended tasks, specifically multiple-choice question answering. The idea is to calculate the log-likelihood of the next token to generate for the option identifiers. However, this may not be the best setting to evaluate LLMs. Wang et al. [9] studied this on Instruction-tuned LLMs by training a classifier to predict which possible option to associate with the generated answer. This was done to glance over additional text generated by the model (e.g. the generated text could be "The answer is B. " as opposed to the simple "B." token). They found that the log-likelihood and the generated text decisions were often not matching.

Regarding Italian evaluation, some works have approached this challenge. Bacciu et al. [10] released another version of the Open Italian LLM Leaderboard, considering a different variety of tasks. Mercorio et al. [11] released a benchmark based on questions that can be found in the INVALSI test, an Italian educational test, to further test the knowledge and reasoning abilities of these models on a dataset that is natively in Italian rather than obtained through machine translation. The latter is one of the main problems when evaluating these models, due to the lack of resources w.r.t. English language, datasets that are used at the state-of-the-art are translated using machine translation models. Still, all this effort made to evaluate Italian-adapted LLMs mainly relies on closed-ended tasks.

Experiments

We study pre-trained and language-adapted models to test their capabilities in the resolution of Italian language tasks. Specifically, we want to modify the typical formatting that is used in multiple choice question answering to study if the models are capable of correctly following and generating Italian text. Usually, the format shown in Listing 1 is used, where <QUESTION> is the question the model has to answer, <IDENTIFIER_i> and <OPTION_i> are the option identifier, which is usually a letter or a number, and the text of the possible answer to the previously provided question respectively. <COR-RECT_IDENTIFIER> is the identifier of the option that is the correct answer to the question.

<QUESTION>: <IDENTIFIER_1> <OPTION_1> <IDENTIFIER_2> <OPTION_2> ... <IDENTIFIER_N> <OPTION_N> <CORRECT_IDENTIFIER> Listing 1: closed-ended format

We aim to modify the task so that the model has to generate the text of the correct option instead of the identifier. To do so, we consider two main evaluation settings:

• Open-ended (OE): we remove the available options and only supply the question in the prompt; • Closed-ended no identifiers (CE-NI): we format the options without an identifier, the model has to write the corresponding text of the correct option.

In particular, for the CE-NI setting, we apply the format shown in Listing 2, where <CORRECT_OPTION> is the text of the option that represents the correct answer to the question. Generally models are also evaluated by calculating the log-likelihood rather than generating text directly. The chosen option is then selected based on the highest value. We choose to perform a generative task instead, to check whether the models are capable of generating the answer string only without additional text and to also check if they generate something outside of the provided options. To evaluate this case, we use the BLEU, ROUGE-L and BertScore F1 metrics, which are reference metrics used to evaluate the correspondence of a generated sentence with a base one. BLEU and ROUGE-L focus on matching n-grams, while BertScore leverages pre-trained Bert models to assess the semantic similarity between words of the two texts. Furthermore, we consider four different possible prompt formats:

• Plain (P): there is no formatting, the text of the task is provided as it is in the prompt, only a "Risposta:" string is added at the end; Furthermore, for the few-shot formats, we consider two distinct numbers of examples to provide in the prompt: one-shot and five-shots. The intuition is that a language-adapted LLM should significantly improve performance even when provided with a single example.

We consider these prompt formats because most of the evaluation settings for Italian LLMs are done without applying the chat template. We argue that this choice may not be the best one when considering Instruct models that have been trained using a specific prompt format to continue a conversation. They should be evaluated using the same prompt format since it is also the one that will be used in case of deployment.

To set up the experimental protocol, we use the lmevaluation-harness library [6], which provides an immediate and intuitive command line to automatically evaluate LLMs on previously defined as well as custom tasks. Specifically, we define custom tasks within the library following the previously defined evaluation settings. To do so, we consider the following datasets:

• ARC-Challenge [12]: consists of multiplechoice science exam questions, the Challenge set consists of complex questions that were not correctly answered by both a retrieval and cooccurrence method;

• MMLU [13]: consists of multiple-choice questions from 57 different topics (e.g. mathematics, computer science, and so on), requiring problemsolving abilities and knowledge to answer correctly; • EXAMS [14]: consists of multiple-choice questions from high school exams. The dataset contains different subsets curated for different languages and optionally contains additional paragraphs regarding the question (extracted from Wikipedia); • WWBM [15]: consists of multiple-choice questions spanning a wide range of topics. The questions come from the Italian version of the "Who Wants to Be a Millionaire?" board game where contestants answer progressively difficult questions.

The question-answer instances are split into different categories depending on the difficulty of the question itself.

For the Italian version of these datasets, both EXAMS and WWBM are provided with splits in the Italian language natively. For ARC and MMLU, instead, we use the Italian version provided in the library for the okapi task released by Lai et al. [16], who performed automatic translation of the original datasets using GPT-3.5 Turbo for several languages. For all of these datasets, we define two custom tasks which apply the OE and CE-NI evaluation settings automatically. The examples used in the few-shot settings are taken from the validation splits of the datasets. For EXAMS, we use the train split as a test split (since it is not provided), while for WWBM, we remove the first five instances from the original dataset and use them as a validation split.

Regarding the models, we experiment using the following:

• Italia-9B-Instruct-v0.1 2 : trained from scratch with a focus on the Italian language (90% of data in Italian and the rest in English) with instructiontuning for conversational purposes; • LLaMAntino-2-chat-13b-hf-UltraChat-ITA [17]: instruction-tuning of LLaMAntino-2-chat-13b-hf-ITA (an Italian-adapted LLM) using a translated version of the UltraChat dataset; • LLaMAntino-3-ANITA-8B-Inst-DPO-ITA [18]: fine-tuning, DPO and adaptation using a mixture of Italian and English datasets starting from the LLaMA-3-8B-Instruct model; • maestrale-chat-v0.4-alpha-sft 3

Table 1

Results for the OE setting. For the few-shots formats, the number of given shots is also provided next to the format name. The best result for each dataset and for each metric is in bold

• Meta-Llama-3-8B 4 and Meta-Llama-3-8B-Instruct 5 : latest version of the LLaMA family of models released by META (base and instruct version respectively); • Minerva-3B-base-v1.0 6 : trained from scratch to be a proficient bilingual base model (English and Italian); • zefiro-7b-dpo-ITA 7 : based on zephyr by Tunstall et al. [19], DPO training done on top of zefiro-7b-sft-ITA.

Furthermore, to test whether bilingual training helps the model solve these tasks, we instruction-tuned two 4 https://huggingface.co/meta-llama/Meta-Llama-3-8B 5 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 6 https://huggingface.co/sapienzanlp/Minerva-3B-base-v1.0 7 https://huggingface.co/mii-community/zefiro-7b-dpo-ITA new models. We start from the Meta-LLaMA-3-8B-Instruct checkpoint and fine-tune the model on 40, 000 instances from 3 different datasets: databricks-dolly-15k, OpenOrca and UltraChat. The datasets are automatically translated to Italian using ChatGPT 3.5. We consider two different settings, one where 20, 000 instances are kept for each language (Italian and English), and one where 40, 000 instances are kept for the Italian language only. For instruction tuning, we used LoRA with 𝑟 equal to 16 and alpha equal to 16, targeting all linear layers of the model. Other hyperparameters are effective batch size equal to 128, learning rate equal to 2𝑒 − 5, weight decay equal to 0.01 and warmup steps equal to 5. In both cases, the instances to be used during the training are chosen at random.

For all experiments, we use the greedy-decoding generation strategy with a maximum number of tokens to

Table 2

Results for the CE-NI setting. For the few-shots formats, the number of given shots is also provided next to the format name. The best result for each dataset and for each metric is in bold generate equal to 64. This limit was set for computational requirements and the value was chosen after studying the datasets to assess the number of tokens required for each answer. There was no combination of tokenizer and dataset which had a 95% percentile greater than 50 for token count, therefore we can safely set the previously defined boundary. We also set torch.bfloat16 and use flash-attention-2 [20] to speed up the generation process.

Inference was always done with batch size set to 1 to maximize the quality of the generated text. Furthermore, we consider changing the number of few-shots that are given in the prompt. Our assumption is that the models may learn to follow the patterns given in the examples, and therefore the Italian language generation may become more likely thanks to the additional information conveyed in the prompt. We aim to mitigate this potential bias by decreasing the number of shots. Thus, the number of shots for all settings using a few-shot strategy was set to either 1 or 5.

We report the results of the OE setting in Table 1 and of the CE-NI setting in Table 2 and comment them in the following section.

Hardware and Software Configuration

Our experimental setup consisted of a multi-node cluster provided by Fastweb SpA and equipped with Nvidia H100 GPUs for distributed training and evaluation. We used a suite of open-source libraries, including Transformers from Hugging Face [21], which provides seamless integration with PyTorch [22] and DeepSpeed [23], as well as Unsloth 8 and TRL [24]. This software stack has been instrumental in efficiently handling large data sets and complex models. This configuration allowed for parallelization of computations, significantly reducing training and evaluation time. DeepSpeed optimized memory usage and communication between nodes, allowing us to effortlessly scale evaluation processes across multiple model architectures.

The hardware-software combination ensured efficient, cost-effective, and reproducible experiments, which are critical for comparing multiple models and training new ones efficiently.

Findings and Additional Tests

Analyzing the results, it is clear that the OE strategy did not yield very satisfactory results for BLEU and ROUGE-L. We associate this with the difficulty of generating a response matching exactly the ground truth when the text that can be generated is not constrained in any way. To further support this point, we can see that the BertScore of some experiments yields good results, hinting that the semantics of the content that has been generated is similar to that of the ground truth.

Regarding the CE-NI strategy, the obtained results are much better for all metrics. Therefore providing the options in the input prompt greatly helped the model in limiting its generation to follow the provided options. Surprisingly, with respect to the Italian leaderboard where fine-tuned versions of the LLaMA 3 family were shown to have much better results, here the results are in line with the base models (or even worse in some cases). Furthermore, one of the best-performing models is maestralechat-v0.4-alpha-sft, which consistently outperforms the LLaMA 3 models in most cases.

For both settings the obtained results show that providing input-output examples in the prompt greatly enhances the results for all settings.

For both settings, primarily Instruct models were used. Upon analyzing the generated results, we observed instances where the model provided the correct result but appended an additional substring (e.g., the model began explaining the reasoning behind its response). To assess if this might have affected the result, we performed an additional test where we checked if the ground truth string was a substring of the generated output (after removing punctuation and trailing whitespaces as well as lowercasing the two strings). We report the complete results in Appendix C. Overall, some models show an improvement in performance, but the results still do not beat maestrale-chat-v0.4-alpha-sft.

We provide some generation examples in Appendix B.

Conclusions and Future Works

We have carried out a study on the effectiveness of evaluation of Italian-adapted LLMs on closed-ended tasks, multiple-choice question answering tasks specifically.

We have experimented with two settings: an open-ended one and a closed-ended one without option identifiers. The results show better performance for the latter. Furthermore, they also show that, with respect to the Open Italian LLM Leaderboard, there are significant differences regarding model performance. We can conclude that the evaluation of Italian-adapted models should follow a more rigorous procedure which does not mainly rely on closed-ended tasks. We release the code that was used on GitHub 9 . In the future, we plan to further work on the topic and attempt to define best practices for the evaluation of these models.

Model Format ARC_IT MMLU_IT EXAMS WBMM

Italia-9B-Instruct-v0.