In recent years, multi-choice question answering (MCQA) has established itself as a powerful method to test the factual knowledge and reasoning abilities embedded in large language models (LLMs) as a byproduct of the language modelling objective [1,2,3,4]. The evaluation of MCQAs can be easily automated, offering a significant advantage over other benchmarking formats such as open-end text responses. In addition, with appropriately targeted prompting, the limited num-coherence of discourse, and encourage the presence of linguistic constructions that reflect the source language rather than the target one [9]. The second issue relates to culture and societal norms: text translated from English to Italian will lack topical biases, preferences, conventions, and ways of expressing ideas that are unique to Italian culture. Thus, while the text may be expressed in the Italian language, its content and underlying norms will continue to represent an Anglo-centric, predominantly American perspective.
More generally, training data for LLMs is biased towards English content, and as a result, there is often a gap between English and non-English performance [10]. For example, the Common Crawl dataset 2 , often used as a base for more refined datasets to be employed in the pre-training of LLMs, is composed of 45% English content, while the data for languages such as Spanish, French, Italian, and Chinese are all below 5% each, with the only exceptions being Russian (6.2%) and German (5.1%).
Creating a large-scale multi-choice question answering benchmark using original Italian data will make it possible to investigate the Italian abilities of LLMs in a more natural and transparent way, possibly also leading to a better understanding of how to make multilingual models better at Italian. It will also serve as a core benchmark for assessing the performance of monolingual Italian LLMs. If similar datasets are collected natively for other languages, Mult-IT can be part of a larger MCQA benchmark which is multilingual in the truest sense.
Mult-IT, presented at CALAMITA [11], is the first massive Multi-Choice-Question-Answering dataset specifically designed for the Italian language which draws on Italian culture and Italian-focused knowledge. By providing a comprehensive, culturally relevant benchmark for the Italian language, we aim to set a precedent for the development of similar resources in other languages and cultures, ultimately contributing to a more diverse and inclusive AI landscape.
This challenge involves a multiple-choice questioning answering task. The model is prompted with a simple instruction (see Box 1), followed by a question and a set of three and five possible answers, depending on the source and topic of the question. Among these answers, only one is correct, and the others are distractors. The model is expected to identify the correct answer and return the letter corresponding to the option deemed correct.
All questions in the benchmark have been manually crafted for the purpose of training or testing students, job applicants, or learners across a range of topics, including general knowledge and more specialised subjects. These questions make up the Mult-IT dataset: Multiple Choice Questions on Various Topics in Italian, which we are introducing in this contribution. The details of the dataset are described in Section 3. The defining feature of the Mult-IT challenge is that all of the MCQs are natively Italian, both in language and in content. While this is an advantage to gain a better understanding of model behaviour on Italian data, we do expect a decline in model performance. Considering that even models which have been trained on multilingual data have a heavy bias towards English and American-centric culture, it is expected that the correctness of the answer may be affected by a cultural (and possibly language) gap. Should the battery of the models tested also include Italian monolingual or bilingual English-Italian models trained on a substantial amount of Italian text, this benchmark will make it possible to underscore differences in performance possibly associated to the language specificity of such models.
Mult-IT contains quizzes designed to assess candidates' knowledge in open competitive exams, whether for admission to national universities or for positions in Italian institutions. This approach offers several advantages. First of all, these public competitions encompass very general topics such as language comprehension, basic history, and common knowledge, but also more specialised ones, focusing on specific laws needed for certain professions or the security measures required for jobs such as policemen or firefighters. Our benchmark, therefore, contains questions that range from a low level of difficulty to a very high and specific level, setting high standards for the performance of the models, and it may also be useful to assess specific knowledge valuable for the adoption of models in Public Administration scenarios. The inclusion of profession-specific questions in Mult-IT tests the ability of LLMs to apply their knowledge in practical, real-world scenarios, a feature that could prove particularly valuable in assessing the potential of AI systems to support specialised fields and decision-making processes in professional and administrative contexts in the Italian landscape. Moreover, the quizzes contained in the dataset also present challenges regarding reasoning, such as logical thinking and mathematical reasoning, as well as quizzes specifically designed to assess knowledge and mastery of the Italian language, for example, text comprehension or detailed understanding of grammatical phenomena.
Mult-IT consists of two core subsets, which are divided by the origin of the data. Both subsets are made of quizzes that test knowledge of general Italian culture and that are used in the public recruitment processes for governmentbased positions. They are described in more details below.
A is a collection of MCQs provided by Alpha test. It contains a total of 1,692 questions in Italian, spanning over 17 categories which corresponds to topics featuring in entry exams for Italian universities (see Table 1 below for details) or question answering tests employed in public competitions. The quizzes in the dataset falling into the categories of law, pedagogy, psychology and criminology originates from public competitions. For each question, four or five possible answers are provided, out of which only one is correct. An example from topic 'sinonimi' (synonyms) is shown in Figure 1.
(A) tenero, (B) doloroso, (C) pigro, (D) insensibile Mult-IT-C Mult-IT-C is a large collection of MCQs, organised in groups of questions ("quizzes") around multiple topics, which we have obtained from publicly accessible online platforms through data-gathering and web-scraping. The quizzes are meant to be used by people who need to prepare to apply for job positions in the public sector. One of the most interesting feature of Mult-IT-C is its size: it contains more than 100,000 questions, making it almost six times larger than MMLU. An example from topic 'geografia' (geography) is shown in Figure 2.
In quale nazione si trova il Lago Balaton?
Mult-IT-A All the materials of Mult-IT-A were obtained thanks to the generosity of Alpha Test3 . Alpha Test S.r.l. is an Italian publishing house and educational training company, founded in Milan in 1987, that specialises in study aid materials and courses for high school, university, professional tests, exams and certifications. Alpha Test is the main reference for high school students preparing for university admission. Each year, Alpha Test gathers new data from the entrance exams of public and private universities and military schools, mainly in the form of multiple choice questions. The publishing house enhances such materials with comments and explanations, and creates variations or completely new versions of the original quizzes. All the materials in Mult-IT-A have been sourced from original, public data, and represent a varied sample of quizzes about general culture, STEM, and juridical disciplines.
All the materials of Mult-IT-C were obtained using a web-scraping process from the website "Concorsi Pubblici"4 via customised Python scripts. While there exist many websites collecting public competitions exams, we found Concorsi Pubblici to be the most complete. Because the same public competition can be listed in several platforms, gathering all the data from a single websites avoided the risks of data duplication. The quizzes on Concorsi Pubblici are organised by topic (see Appendix A), and were extracted in time interval 1997-2024.
Overall, the data format is consistent across the two Mult-IT subsets, which allows for a single evaluation procedure on Mult-IT. The larger size of the Mult-IT-C dataset allowed us to include additional information, including details about the quiz's administration presented in the form of quiz blocks and a multi-level topic taxonomy. This feature is absent in Mult-IT-A as questions are collected by subject without being grouped in quizzes.
Common data fields in all Mult-IT are:
• origin: It can be either 'C' or 'A', to discern if the question belongs to Multi-IT-A or Multi-it-C • question: The question.
• choices: The list of possible answers.
• answer: The array-index corresponding to the correct answer in the choices array.
These common fields are crucial to the evaluation task, but each of the sub-portions has additional information added.
Examples taken from Mult-IT-A are given in Figure 3.
{ "origin": "A", "topic": "informatica", "question": "Le dimensioni del monitor si misurano in:", "choices": [ "megahertz", "pixel", "centimetri", "pollici" ], "answer": 3 }, { "origin": "A", "topic": "psicologia e sociologia del disadattamento", "question": "Come viene definito lo stimolo funzionale a provocare un cambiamento?" , "choices": [ "Stress", "Output", "Input", "Matrice" ], "answer": 0 }, The only additional field is Topic, pointing to the topic of the question. Its distribution and statistics about token count and char count are available in Figure 6.
The dataset consists of two files: quiz.jsonl contains the actual questions, while metadata.jsonl contains additional information about the questions. An example from the quiz.jsonl file is given in Figure 4.
Data fields unique of this subportion are:
• quiz_id: The ID of the quiz to which the question pertains. • question_id: The unique identifier of the question inside the quiz. In combination with the quiz_id it forms the unique identifier of each question and can be used to retrieve the metadata of the question from the metadata.jsonl file.
Data fields of the metadata are:
• id: The unique identifier of the question.
• title: Title of the quiz sourced from the original website. • tags: List of word tags.
{ "quiz_id": 2250, "question_id": 20, "question": "La Costituzione riconosce allo Stato una potesta' legislativa esclusiva in materia di:\n\n", "choices": [ "organizzazione della rete scolastica", "norme generali sull'istruzione", "ricerca scientifica e tecnologica", "istruzione professionale" ], "answer": 1 } { "quiz_id": 1253, "question_id": 63, "question": "In un ingranaggio con piu' ruote dentate, una ruota denominata R1 ha 25 denti e fa muovere una seconda ruota denominata R2 da 50 denti, che a sua volta fa muovere una terza ruota R3 da 150 denti. Se la ruota dentata R3 fa un giro e mezzo, quanti ne fa la ruota dentata R1?\n\n", "choices": [ "3", "5", "6", "9", "12" ], "answer": 4 } An example of an item from the metadata file is given in Figure5.
We evaluate our models in a zero-shot setting, thereby imitating the conditions of a real use-case scenario. The prompt we chose is designed to encourage the model to output only the letter corresponding to the answer. The original prompt, together with its English translation, is presented in Box 1.
{ "id": 2250, "title": "Area 3 Giuridico Amministrativa Finanziaria -25 domande concorso dirigente scolastico Miur", "tags": [ "concorsi dirigenti scolastici", "concorso dirigente scolastico", "dirigente scolastico concorso dirigente scolastico 2017", "miur", "miur concorso", "miur concorso dirigente scolastico miur concorsi scuola", "concorso scuola", "bando concorso scuola", "bandi miur" ], "class_lev1": [ "Miur", "dirigente scolastico" ], "source": [ "Fgl Cgil, Miur" ], "difficulty": [ "medio" ], "class_lev2": [ "Istruzione", "Altre" ], "class_lev3": [ "Societa' e Diritto", "Altro" ] }
Di seguito è riportata una domanda a scelta multipla e varie possibili risposte, ciascuna indicata da una lettera. Scegli la risposta che meglio risponde alla domanda, e riporta in output soltanto la lettera corrispondente a quella risposta, senza spiegazioni.
Choose the best answer for the question, and report as output only the letter corresponding to that answer, without any explanation.
Box 1: Zero-shot prompt and English translation.
We decided to write the prompt in Italian in order to better represent a multilingual scenario. The prompt does not contain any information about the subject of the question or any other informative cues. In this way, our benchmark not only tests the model in question answering, but also indirectly tests the instruction-following abilities of the model in a language different than English.
Previous work on evaluating the performance of LLMs on MCQ datasets has identified two aspects which can interfere with the model's answers and therefore accuracy. One has to do with the order of possible answers: Wang et al. [12] show that the first presented option out of the possible choices tends to be preferred in the model's answer, making it quite important to take the order of possible answers into account. The other has to do with the prompt's (and even the question's) formulation: Singhal et al. [13] experiment with multiple types of prompts and also show that prompt formulation affects the model's output.
Because the position of the correct answer in the original data was already randomly distributed, which we verified with a supplementary analysis on the data (see Appendix B), performing a random permutation of the possible answers was not necessary.
Mult-IT-A The Mult-IT-A dataset is composed of 1,692 questions, spanning over 17 topics, all centered around knowledge required for entry exams at Italian Universities. The topics, and some additional information on the dataset composition, are provided in Table 1.
On average, questions are 83.76 characters long and contain 25 tokens counted with the tiktoken cl100k base5 tokenizer or 16.5 if counted with the Spacy6 library using the it_core_news_lg model 7 .
Further statistics about quiz distribution and answer position are available in Appendix ?? and C.
It's worth noting that permutating the order of the answers would be recommended to avoid any kind of unbalance, as Multi-IT-A shows an uneven correct answer distribution leaning heavily on the first choice, acquired from the source data.
The Mult-IT-C dataset is composed of 108,773 questions divided into 4,129 quizzes.
To avoid confusion, we decided to give unequivocal names to the items of the subjects. A "quiz" is defined as a set of multiple "questions". Quizzes come from real-world examples, so they are provided with a specific name and Mult-IT-A: Topics included in the dataset, number of questions per topic, total tokens per topic, and average length of question per topic in terms of tokens. The topic "pedagogia" in the Table is short for "pedagogia con particolare riferimento agli interventi relativi all'osservazione e al trattamento dei detenuti e degli internati"; the topic "elementi di diritto costituzionale ed amministrativo" is short for "elementi di diritto costituzionale ed amministrativo con particolare riferimento al rapporto di pubblico impiego".
a categorisation originating from the original data source.
A quiz can contain a variable number of questions. The average number of questions per single quiz is 26, and the maximum is 250. There are 1623 quizzes with more than 25 items, 298 with more than 50, and only 22 with more than 100 items. The original categorisation made by the authors of the website "Concorsi Pubblici" was problematic for our purposes: some categories were near-duplicates of each other, containing only slightly different words. Moreover, we believed that 186 categories were too many for a meaningful visualisation and management of the data. For this reason, we created a hierarchy of three levels in which the first (bottom) level corresponds to the original categorisation of the data, then the second level groups the categorisation into 36 areas, and finally, the third level, the more abstract, has only 7 categories. The drawback of this approach, as it can be seen in the tables and graphs contained in Appendix A, is that in both the supplementary categorisations there is a significant amount of quizzes falling into the category "Other".
Nonetheless, we believe that this abstract categorisation can be good for having a general look at the data composition and thus the performance of the models in terms of macro-areas. On the other hand, keeping the original very detailed categorisation in the data allows for more in-depth analysis of model performances in specific aspects. In Appendix A, all the statistics of the 186 categories are listed in the form of a table. To appreciate the level of specificity reached by the first level of categorisation, it's interesting to notice, as examples, categories such as "Verbs", "Diphthongs", or "Word Meanings" referring to specific language abilities. These categories are then grouped in level 2 as "Linguistic Competence" and in level 3 as "Language". As another example, we can see categories that refer to specific aspects of the Italian Public Administrations: we can see in category 1 fields such as "INPS", that is, National Institute for Social Security, or "ASL" that is "Local Health Authority".
We believe that having such a precise categorisation at our disposal is of great help in understanding the abilities and weaknesses of models in very specific aspects, thus being helpful on one hand for assessing the possibility of direct practical employment of models in Italian public administration and, on the other hand, to improve the scientific understanding of models and how they deal with different kinds of challenges. This last aspect can also be helpful for interpretability studies of LLMs.
On average, questions are 104 characters long, they contain 27.5 tokens counted with "tiktoken cl100k base"8 or 19.8 if counted with the "Spacy" library9 using the "it_core_news_lg" model 10 . The longest question is 1363 token long.
We will use accuracy to evaluate the LLMs' performance on Mult-IT. Accuracy is defined as the ratio of correctly answered questions to the total number of questions, and it is a straightforward and easily interpretable measure of performance on MCQ tasks. Accuracy will be reported overall, and also separately for the two subsets Mult-IT-A and Mult-IT-C.
While accuracy is indeed a straightforward evaluation metric for this task, deciding which is the answer identified by the model as correct is not necessarily as straightforward for a couple of reasons.
As mentioned in Section 3.3, the position of the correct answer in the prompt is randomly distributed, reducing the likelihood of bias resulting from its placement, al-though the models might have a tendency to select the first answer more frequently.
A related issue is the fact that the model's output, in spite of the specific request in the prompt, might not always just be the letter corresponding to the chosen answer. In the case of longer outputs, simple regular expressions will be applied to extract the relevant letter.
In practice, as for all the CALAMITA challenges, the evaluation of the LLMs on Mult-IT will be carried out on the LM-evaluation-harness framework developed by EleutherAI 11 .
The vast majority of data comes from sources linked with Italian public Institutions, and can be considered official documents. For this reason, we expect an high quality regarding the formulation of the quizzes and the correctness of the answers. Nonetheless, given the large amount of data, we cannot guarantee the absence of errors in the single questions. Human errors can happen, even in official selection, although it should be considered a rare occasion. This aspect can be improved by analysing the results obtained by the model in the benchmarks: the more the benchmark is going to be used, the more it will be possible to isolate and eventually remove or correct problematic quizzes with data analytics techniques. Moreover, considered that the quizzes encompass almost a thirty years time span, it is possible that some quizzes, particularly the ones regarding laws, may be outdated. Nonetheless, thanks to the availability of meta- data, it is possible to further refine this dataset to also account for specific historical knowledge about laws by providing metadata to the model. However, we believe that for the first run of this evaluation this point will not create particular issues as we expect the potentially outdated questions to be limited.
Given the publicity of the data, it is possible that the original exams are already present in the models training data as they can easily obtained on the Internet. At the same time, it is likely that some sources, for example complete laws of the Italian legislation, are present in the training data, but we consider this eventuality positive given that one of the benchmark's aim is to evaluate the knowledge and capacity of the model to adapt to the Italian landscape.
Information about license and copyright issues is mandatory.
The authors would like to thank Alpha Test -https:// www.alphatest.it, and in particular Martha Fabbri, for their interest in the Mult-IT CALAMITA challenge and for the extremely valuable exchange of ideas and data, that allowed us to shape a task of high potential impact also in the field of educational training and assessment.
(M. Nissim) https://github.com/mrinaldi97 (M. Rinaldi); https://github.com/Jj-source (J. Gili); https://github.com/rosakun (M. Francis); https://github.com/vivpatti (V. Patti); https://github.com/malvinanissim (M. Nissim) 0009-0004-7488-8855 (M. Rinaldi); 0009-0007-1343-3760 (J. Gili); 0009-0007-7638-9963 (M. Francis); 0000-0001-5991-370X (V. Patti); 0000-0001-5289-0971 (M. Nissim)