Challenge: Introduction and Motivation

Assessing the quality of Large Language Models is a challenging task because these models can virtually perform any task that can be presented through natural language.

To address this difficulty, each model needs to be tested on several tasks at once. To help provide new benchmarks to evaluate LLMs in Italian, We propose two benchmarks, Invalsi MATE and Invalsi ITA the first meant to evaluate LLMs' mathematical understanding and the second to evaluate their language understanding, both in Italian.

These benchmarks originate from the Invalsi tests, which have been used in the past for demographic studies [1,2,3], but, to the best of our knowledge we are the first to use them to test LLMs performance in Italian [4], followed only later by others [5].

There are several benchmarks to evaluate mathematical understanding of LLMs based on English tests [6,7,8] and there are also several multi-domain benchmarks involving Italian [9], however there aren't any specifically focused on mathematical understanding in Italian. We focus on high-school questions, an English benchmark similar to Invalsi MATE is the GSM8k one, [8], which contains 8,500 high-school questions.

Language Models understanding of language in English is also well studied, there are several benchmarks meant to measure the ability of language models to understand language constructs in English, such as [10,11], also arranged into extensive suites [12]. On the contrary there are fewer examples of these tests for the Italian language.

Therefore we propose Invalsi ITA which contains questions that are usually split among several different benchmarks, e.g. MNLI [13], SQuAD [14] and others from the GLUE suite [15]. The questions in the dataset cover several aspects of language understanding, ranging from the ability to extract specific information, such as the date when something happened to more complex information such as whether two events implicate each other or not.

These two datasets allow us to measure two key abilities of language models in Italian, to make the comparison among different models more fair we cast all questions as multiple choice and measure models' performance by selecting the answer with the highest likelihood according Testo Elisa è uscita da casa questa mattina alle ore 8:15. Elisa è rientrata nel pomeriggio alle ore 1:15 Domanda Quanto tempo è stata fuori casa Elisa? A. 5 ore B. 7 ore C. 9 ore D. 11 ore (a) scelta multipla question from Invalsi MATE.

Testo

Se moltiplichi per 2 un numero naturale e dal risultato sottrai 1, ottieni sempre un numero pari. Domanda Vero o Falso? (b) vero/falso question from Invalsi MATE.

Testo

Filippo dice: per trovare il numero della mia maglietta aggiungi una decina e sei unità al numero 4. Domanda Qual è il numero della maglietta di Filippo? (c) numero question from Invalsi MATE.

Testo

Luca lancia due dadi a sei facce non truccati. Domanda Completa la frase inserendo una delle espressioni: La probabilità che la somma dei punti sia 12 è maggiore della, minore della, uguale alla probabilità che la somma sia 2. to the model.

We measure the performance of 4 strong large models, mixtral instruct [16], mistral instruct [17], llama 3 8b instruct [18], anita 8b dpo [19], the first three are Englishfirst and the fourth is fine-tuned in Italian, and the current only Italian-first model minerva 3b. We show that on Invalsi ITA the best model among those we tested is mixtral instruct, which reaches an accuracy of 0.8, while on Invalsi MATE the highest accuracy is 0.55, also achieved by mixtral instruct.

Both language understanding and mathematical understanding are key abilities for students as well as language models, particularly since these models are often used in learning environments. By adding these benchmarks to the CALAMITA suite we hope they will help the development of LLMs in Italian by providing a more comprehensive evaluation of their abilities and thus fostering the research and development of models in this language.

The CALAMITA special event [20], which has aims to establishing a shared benchmark for LLMs in Italian, is a first step towars a systematic evaluation of LLMs in this language. We hope that the Invalsi challenge will enrich the Linguistic and Mathematical understanding branches of this shared benchmark.

Challenge: Description

The challenge is composed of two tasks: Invalsi MATE and Invalsi ITA. For each task, we provide a detailed description of the data, the metrics used for evaluation, and the limitations of the data.

Task 1: Mathematical Understanding in Italian (Invalsi MATE)

The first task consists in answering mathematical questions in Italian. These questions are meant for students from 6 to 18 years of age, therefore the kind of question can vary significantly, from simpler, example-based, ones that don't require any knowledge besides counting, to more complex ones requiring basic geometry and calculus training and knowledge, never beyond what is demanded in basic high-school tests. The questions are of 4 kinds, scelta multipla, completa frase, vero/falso and numero:

• scelta multipla (multiple choice): the question requires to pick the right answer among four possible ones; • vero/falso (true/false): the question requires to pick the right answer between true and false; • numero (number): the question requires to pick a number that is the correct answer to the question; • completa frase (fill the gap): the question requires to fill one or more missing words to make the text coherent.

Of the four question types, scelta multipla and vero/falso are naturally multiple choice, with scelta multipla always having 4 possible answers (A, B, C, D) and vero/falso always 2, (true, false). Questions of the numero type, are not naturally multiple choice, since the answer is a number among all possible ones (some of the answers will be a year, e.g. 1948, while others can be a decimal number of liters of milk, e.g. 0.2), to address this we add 3 extra answers that are realistic but wrong to make the questions multiple choice. Finally, completa frase questions, which are only few (20), are too difficult to turn into multiple choice without changing their meaning, and therefore we exclude them.

Task 2: Language Understanding in Italian (Invalsi ITA)

The second task consists in answering Italian language understanding questions, similarly to task 1, these questions are also appropriate for students between 6 and 18 years old, and they are overall not too difficult to answer. Most of the questions concern a text passage that has to be included in the model context, making this evaluation more costly because the context becomes considerably larger. The text passage is where the difficulty difference between ages is more evident, since it can be a simple and short story for primary school students, while they are generally longer and more involved texts for older students.

The questions are of 4 different types, scelta multipla, binaria, domanda aperta and altro:

• scelta multipla (multiple choice): the question requires to pick the right answer among four possible ones; • binaria (binary): the question requires to pick the right answer about a binary property of a statement, e.g. True -False, Before -After, etc.

• domanda aperta (open question): the question requires to pick the passage in the text that answers the question. • altro (other): A small share of questions belong to open-ended questions with varying scope that are hard to put under a single label.

Similar to Invalsi MATE, this task involves only multiple-choice questions, evaluated through a likelihood approach. Both scelta multipla and binaria are naturally of this kind, the first with 4 options (A, B, C D) and the second with 2 options that change for each question. Both domanda aperta and altro questions are hard to turn into multiple-choice ones and therefore we discard them. Also for Invalsi ITA, this involves only discarding about 180 questions out of 1297, therefore the task only involves 1117 samples.

Data description

Origin of data

The dataset is built upon the questions from the Invalsi tests of the last 15 years. These tests are administered to students yearly. There are three different Invalsi tests, Language, Mathematics and English. For the scope of this datasets we don't look into the English test, but we limit ourselves to the Italian language and Mathematics ones. The original questions can be accessed here 1Some of the questions from the original tests contain visual content as part of the question, we omit these questions since we focus on the language understanding abilities of the models. The data is first collected as is from the webpage and afterwards it is manually checked for errors and inconsistencies from two annotators. The annotators have MSc in Mathematics and Computer Science, which gives them sufficient knowledge to identify issues in the Invalsi MATE questions. For Invalsi ITA, the annotators don't have an appropriate background, however, the questions are simple enough that they can be easily understood and checked for errors by anybody who has completed the mandatory education.

Data format

The Invalsi MATE dataset has 8 different columns:

• testo: this field contains the context needed to answer the questions, it is often empty for Invalsi MATE since most of the context is part of the domanda field itself; • domanda: this field contains the question itself, including possible answer options, e.g. for scelta multipla questions; • risposta: this field contains the correct answer; • test_id: this field is just an id to identify each sample; • tipo: this field indicates the question type, among scelta multipla, vero/falso, numero and completa frase; • alt1, alt2 and alt3: this three fields indicate the alternative values for the numero questions since this we chose ourselves and are not indicated in the domanda field.

The Invalsi ITA dataset has the same fields as the Invalsi MATE one with the exception that the testo field is often present and generally the longest.

We evaluate in a zero-shot fashion just providing the model with question and using a likelihood based method, we pick as the model's answer the one with the highest likelihood among the options available. This is always possible since we have recast all the questions as multiple choice ones. We also don't use chain of thought prompts or similar methods. Since this is the first attempt to build a dataset on mathematical understanding in Italian, currently we evaluate with the simplest approach.

Detailed data statistics

The data does not have a train and a test split because we have a limited number of samples. The Invalsi MATE split is composed of 420 samples, of which 400 are used in the benchmark, since we exclude the 20 questions marked as completa frase since they can´t be made into multiple choice.

Figure 2a shows the percentage of questions of each kind in the dataset, scelta multipla has the largest share, 58%, the second most present is numero, 24.7% and then vero/falso and completa frase are fewer. Table 1 has a random row that shows the performance if one were to pick random questions, moreover the correct answers for each question type are approximately evenly distributed among labels. Specifically, scelta multipla questions have answers distributed as follows, 46 are labelled A, 87 B, 71 C and 40 D, showing a moderate balance. Similarly for vero/falso questions there 24 questions with positive answer and 30 with negative answer.

Similarly, Figure 2b shows the percentage of questions of each of the kinds present in Invalsi ITA, scelta multipla is by a large margin the most present, composing 76.4% of all the questions, binaria is second with 10.9% while domanda aperta and altro are fewer.

Table 2 shows the performance one can achieve picking answers at random in each split, and moreover the correct answers are evenly distributed for each label also in this dataset. In particular, for Invalsi ITA 254 of the scelta multipla questions have answer A, 255 B, 263 C and 205 D which is comparable to the distribution in the Invalsi MATE dataset and similarly does binaria.

Metrics

Since all the datasets and splits we study have balanced labels, we choose to measure accuracy. In particular, since all the questions in the datasets we propose are multiple choice, it is straightforward to measure accuracy even if there are questions of different kinds, by simply counting |𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑎𝑛𝑠𝑤𝑒𝑟𝑠|/|𝑎𝑙𝑙 𝑎𝑛𝑠𝑤𝑒𝑟𝑠|.

To see how challenging our benchmark is , we measure four powerful language models based on mistral, mixtral and llama 3, in particular, we measure the performance of mistral instruct, mixtral instruct, anita 8b dpo and llama 3 8b instruct.

These models have between 7 and 54 billion parameters, three of them, mistral instruct, anita 8b dpo and llama 3 8b instruct are purely autoregressive transformers, while mixtral instruct is a MoE architecture that has 54 billion parameters but only uses 14 billion at inference. We test them all in the same way, using a likelihood based approach.

Table 1 shows the performance of these models on Invalsi MATE on the whole dataset, in the ALL column and on each split scelta multipla, vero/falso and numero in the respective columns. mixtral instruct is the clear winner among the models we tested, it beats the second best, llama 3 8b instruct by 7% accuracy on the entire dataset. On the separate splits, mixtral instruct is best overall, however the second best model changes, with llama 3 8b instruct being second in scelta multipla with a 7% gap, anita 8b dpo second on vero/falso with a smaller 2% performance gap and mistral instruct being second best in numero with a 3% gap.

The total accuracy is bound by 55% showing that the Invalsi MATE task is challenging for models of the sizes we tested, up to 54B parameters, and that the performance a model achieves provides valuable insights about how well it can perform mathematical reasoning in Italian.

Table 2 shows the performance of the same models on Invalsi ITA, the model ranking stays the same, with mixtral instruct the strongest and anita 8b dpo second best. Performance on Invalsi ITA is higher across all fields with mixtral instruct achieving 80% accuracy. Unlike what happens for Invalsi MATE, in Invalsi ITA the second best model is the same across the board, anita 8b dpo is the second in scelta multipla as well as in binaria with a performance gap around 10% in all question types.

The accuracy of the best model on all the questions at once is 80% showing that the models we tested perform well in the language understanding in Italian.

Conclusions

We propose two Tasks, Invalsi MATE and Invalsi ITA the first for the evaluation of mathematical understanding and the second for the evaluation of language understanding in Italian.

For Invalsi MATE we have collected 420 questions divided into 4 types, scelta multipla, vero/falso, numero and completa frase and we evaluate 4 strong language models that are near SOTA in their weight range, mixtral instruct, mistral instruct, llama 3 8b instruct and anita 8b dpo. We find that this models are still far from perfect mathematical understanding in Italian with the highest accuracy, achieved by mixtral instruct being 55%.

For Invalsi ITA we have collected 1297 questions divided into 4 types, scelta multipla, binaria, domanda aperta and altro, we tested the same models also on this benchmark and found that models are stronger at language understanding, with the highest accuracy in this task at 80%, also in this case, achieved by mixtral instruct.

Both mathematical and Language understanding are key abilities for LLMs, we believe that our two benchmarks will foster the development of LLMs in Italian and pave the way for new more challenging benchmarks on mathematical and language understanding in Italian.

Limitations

The main limitations of the benchmark we propose lies in Task 2, Invalsi ITA we show that the models we test achieve very high accuracy, up to 80% on this benchmark, making it possibly too simple for newer and larger models, nevertheless, current Italian first LLMs are not comparable to larger English-first ones and therefore we believe it can still be valuable in this transitory phase. On the contrary Invalsi MATE is very challenging and it seems that models won't saturate it soon.

We believe that there is a limited risk from contamination from both existing English and Italian tests.

Concerning direct contamination, we were unable to find any web page that would expose the answers openly without needing any sort of authentication, making it difficult to crawl these data automatically, therefore, while the questions might be present in the training set of some of the models, we deem it unlikely that the answers were there too.

Concerning contamination through translation from English, the Invalsi questions are carefully crafted to match the grade of the students that will undertake them, therefore we believe it is unlikely that they are taken from English questions available in other online sources, but rather created specifically for each new annual test.