B E E P - BEst DrivEr's License Performer:

Abstract

We present BEEP (BEst DrivEr's License Performer), a benchmark challenge to evaluate large language models in the context of a simulated Italian driver's license exam. This challenge tests the models' ability to understand and apply trafic laws, road safety regulations, and vehicle-related knowledge through a series of true/false questions. The dataset is derived from oficial ministerial materials used in the Italian licensing process, specifically targeting Category B licenses. We evaluate models such as LLaMA and Mixtral across multiple categories. In addition, we simulate a driving license test to assess the models' real-world applicability, where the pass rate is determined based on the number of errors allowed. While scaling up model size improved performance, even larger models struggled to pass the exam consistently. The challenge demonstrates the capabilities and limitations of LLMs in handling real-world, high-stakes scenarios, providing insights into their practical use and areas for further improvement.

-

CEUR

ceur-ws.org A

CALAMITA

1. Challenge: Introduction and

Motivation

come a significant breakthrough in Natural Language Processing (NLP) and Artificial Intelligence (AI) [ 1 ]. Assessing model performance is crucial yet challenging, involving multiple critical attributes: models must be precise, resilient, fair, and eficient, among other characteristics [ 2 ].

Developing efective models in underrepresented languages such as Italian is a continuing challenge [3]. This disparity arises from limited and lower-quality data [4] and a development process often prioritising Anglocentric perspectives [ 5 ]. Recently, there has been a surge clusive, moving beyond mere multilingualism to address deeper cultural contexts [6]. For instance, a structured benchmark utilising the INVALSI tests—well-established assessments measuring educational competencies across Italy—represents one such efort to embed culturally relevant content in model evaluation [ 7 ].

This work is part of CALAMITA [8] (Challenge the Abilities of LAnguage Models in ITAlian), an initiative tional Linguistics. CALAMITA aims to develop a comprehensive and evolving benchmark for evaluating the shared platform with a suite of tasks and a live leaderboard, allowing for ongoing assessments of Italian and multilingual LLMs. CALAMITA seeks to build this benchmark through community-driven challenges, inviting researchers to propose tasks and datasets that evaluate specific aspects of LLMs’ performance in Italian. This paper contributes to this collaborative efort by presenting a benchmark that assesses LLMs’ ability to comprehend and apply Italian driving regulations, forming one of the initial tasks in this evolving benchmark.

This challenge evaluates LLM’s ability to comprehend While LLMs have shown remarkable capabilities in understanding and generating human language, their effectiveness in real-world decision-making scenarios remains underexplored, especially in languages such as Italian. This challenge tests whether these models can perform efectively in a linguistically demanding and contextually rich domain. Success in this challenge would demonstrate the model’s ability to generalise language understanding to practical tasks, a crucial step towards their broader application in everyday life.

2. Challenge: Description

BEst DrivEr’s License Performer (BEEP) is a challenge benchmark that focuses on assessing LLMs through a The dataset is formatted with the following columns: • Categorisation Structure - Each question in the dataset is organised within a hierarchical categorisation system consisting of Major Categories, Minor Categories, and Subcategories to ensure precise classification. For example, the Major Category ”Road Signage” includes Minor Categories like ”Warning Signs” and ”Prohibition Signs”, which further break down into Subcategories detailing specific signs such as ”Speed Limit Signs”; • Question Text - The actual content of the question; • True Answer - Can be either true or false; • Figure - A reference for the accompanying figure, if present.

3.3. Example of prompts used

The road can be divided into lanes.

simulated driver’s license exam in Italian. This task re- 3.2. Data format quires a deep understanding of trafic laws and reasoning through driving situations.

In Italy, obtaining a driver’s license is a structured process involving theoretical and practical assessments to ensure drivers are well-versed in road safety, trafic regulations, and practical driving skills. The Italian driver’s license process is governed by strict rules set forth by the Ministero delle Infrastrutture e dei Trasporti (Ministry of Infrastructure and Transport), and the license is recognised across the European Union.

Italy ofers several categories of driver’s licenses, depending on the type of vehicle a person wishes to operate.

We focus on Category B, which is required for cars (up to 3.5 tons) and vehicles with up to 8 seats.

The theoretical exam is crucial to obtaining a driver’s license in Italy, and it is required, along with the practical exam. It assesses the applicant’s knowledge of trafic laws, road signs, and driving regulations. It consists of multiple-choice questions and is typically administered electronically. The candidate must understand trafic regulations, road signs, driving behaviour, and vehicle maintenance. A Category B license test typically consists of 30 questions; a candidate can pass up to 3 errors. Question

The licensing process is not just about learning the rules; it requires candidates to internalise and apply them practically. BEEP reflects this focus on real-world application and safety. The Italian driving system also empha- Options sises road etiquette and the ability to navigate complex trafic situations, particularly in high-density urban areas. Consequently, the challenge aims to mirror this complexity in evaluating LLMs.

Options [ A. True, B. False ]

3. Data description

3.1. Origin of data

Instructions:

You must return the letter corresponding to the correct answer in square brackets.

Answer format: [letter] BEEP is derived from the publicly accessible PDF ”Listato A e B”, which includes all quiz questions related to Italian Answer driver’s license examinations provided by the oficial ministerial listing1. The quizzes consist of true or false [ A ] questions for driving license categories A and B, with data updated as of 01/07/2020. Figure 1: An example question, with instructions and a cor

We extracted the data from the oficial PDF file. The rect answer highlighted. text is segmented by identifying distinct patterns indicating the start of new questions and sections. These segments are classified into predefined categories and We exclusively employed the zero-shot setting in our sub-categories. For each text segment, relevant metadata, evaluation process, where no prior examples were proquestion types (e.g., true/false) and related image num- vided. An illustrative example of a prompt used in this bers are extracted and compiled into a structured format. setting is shown in Figure 1, which demonstrates the The final dataset is exported, ofering a well-organised structure and input format supplied to the model. The collection of questions for the evaluation. decision to have the language model answer with ’[letter]’ rather than simply ’letter’ or ’True/False’ is due to 1Visit ListatoAB for more information at https://www.neca.it/assets/ our use of pattern matching for response extraction. By pdf/ListatoAB.pdf. enforcing a consistent answer format with brackets, we

VEHICLE EQUIPMENT

VISUAL SIGNAL DEVICES AND LIGHTING

ACCIDENTS AND INSURANCE

CAUSES OF ACCIDENTS

ROAD

ROAD AND TRAFFIC DEFINITIONS

TRAFFIC REGULATIONS

STOPPING AND SAFE DISTANCE

MANDATORY DOCUMENTS, AGENTS AND LI

CENSE PLATES

STATIONARY VEHICLE SIGNALS AND ROAD OB

STRUCTIONS

CLASSIFICATION OF VEHICLES

VEHICLE COMPONENTS

TIRES, ADHERENCE AND STABILITY

WARNING LIGHTS AND SYMBOLS

CIVIL AND CRIMINAL LIABILITY AND INSURANCE

STOP, STANDING AND PARKING

DRIVING ON HIGHWAYS

SPEED LIMITS

RIGHT-OF-WAY RULES AND PROCESSIONS

POSITION ON ROADWAY, DIRECTION CHANGE

AND LANE

SPEED REGULATION

OVERTAKING

TRANSPORT OF PEOPLE, LOAD ARRANGEMENT,

PANELS AND TOWING

FIRST AID TO INJURED PEOPLE

SUPPLEMENTARY PANELS

TRAFFIC LIGHT SIGNALS AND POLICEMAN

PROHIBITION SIGNS

INFORMATION SIGNS

MANDATORY SIGNS

WARNING SIGNS

PRIORITY SIGNS

ROAD MARKINGS

Major

DOCUMENTS

VEHICLES

MOTOR VEHICLE

FIRST AID

TRAFFIC SIGNS

SAFETY AND POLLUTION

SEAT BELTS, AIRBAG AND PROTECTIVE HELMET

TEMPORARY AND SUPPLEMENTARY SIGNS ENVIRONMENTAL AND NOISE POLLUTION

can reliably parse responses, reducing ambiguity and en- accuracy is commonly used in classification tasks, particsuring that variations in phrasing or formatting do not ularly in true-false or binary decision evaluations [9]. It interfere with accurate evaluation. measures the proportion of all correct predictions (true positives and negatives) out of the total number of pre3.4. Detailed data statistics dictions made. In other words, it quantifies how well a binary classification system performs by indicating the The questions are organised into the categories described fraction of correctly classified instances (both positive in Tab. 1. This table summarises statistics across various and negative classes) relative to the total number of inroad safety and vehicle regulation categories, provid- stances evaluated. ing detailed insight into major and minor classifications.

Each entry in the table is categorised into broad Major Table 3 Categories such as ”DOCUMENTS,” ”Vehicle Equipment,” Overall accuracy of selected models, ranging from LLaMA to and ”Road Signage,” which are further subdivided into Mixtral, demonstrating their performance on the dataset. more specific Minor Categories. For example, the major category ”DOCUMENTS” includes the minor category Model Overall Accuracy ”Mandatory Documents, Agents, and License Plates,” llama-3-8b-instruct 56.27% ahnigdhalidgmhtiinnigstdraifetriveentdaestpaielcst.s of document requirements lmmlaiimxxttarraa-3ll---887xx027b2b-ib-ni-nsintsrtsurturcutctct 787737...122993%%%

We also include figures associated with specific questions, particularly those addressing trafic signals, road signs, and right-of-way scenarios. These visual elements Table 3 shows the Overall Accuracy obtained by provide additional context and enhance the comprehen- LLAMA3 8B - Instruct2 and others State of the Art models. sion of complex trafic situations. However, for the We evaluate the metrics on the portion of our dataset that CALAMITA challenge, we opted not to include ques- does not require image processing operations. The scaltions containing figures, focusing solely on text-based ing laws hold as it is observed that performance increases questions. This decision ensured that the evaluation of with the number of parameters.

LLMs remains centred on their language comprehension, Table 2 shows the Overall Accuracy stratified by Major knowledge and reasoning abilities rather than visual pro- Category for each tested model. Models perform better in cessing capabilities. Including images would limit partic- the ”SAFETY AND POLLUTION”, ”FIRST AID”, and ”ACipation to multimodal models, excluding many language CIDENTS AND INSURANCE” categories. This may be models that cannot process visual information. By us- possible given the generality of these major categories, as ing only text, we maintain a broader, more accessible opposed to more niche categories such as ‘DOCUMENTS’ benchmark. or ‘VEHICLE EQUIPMENT’, where the performance is worse.

4. Metrics

4.1. Simulated Driving License Test

Since the dataset comprises questions that can only be an- We also test the models by simulating a proper driving swered with true and false, we involved the Overall Accu- licence exam, following the appropriate oficial guidelines racy to evaluate the models’ answers in our task. Overall 2https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct and creating a new indicator. We sampled 1000 samples of 30 questions from the dataset, ensuring each sample was unique. We then counted the correct and incorrect answers for each sample and each evaluated model. The guidelines state that the test is passed if the number of wrong answers is less than or equal to 3. Therefore, we built an indicator for each model that considered the percentage of driving licence exams passed, related to the number of examinations attempted. The results are shown in Tab. 4. As expected, smaller models made many mistakes on average (around 13), which was fatal as it never passed the test in any of the attempts. Even larger models like Mixtral-8x22b did not perform well in most cases. However, we believe more advanced models, such as GPT-4, might succeed more reliably. The data are publicly available online and not subject to copyright restrictions.

Acknowledgments

We thank Thomas Passera for providing the initial code for the dataset’s extraction. Evaluation of the opensource models was conducted on Leonardo supercomputer with the support of CINECA-Italian Super Computing Resource Allocation, class C project IsCb7_LLMEVAL (HP10CIO7T9). [8] G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Francis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Rinaldi, D. Scalena, CALAMITA: Challenge the Abilities of LAnguage Models in ITAlian, in: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), Pisa, Italy, December 4 - December 6, 2024, CEUR Workshop Proceedings, CEUR-WS.org, 2024. [9] C. M. Bishop, Pattern recognition and machine learning, Springer google schola 2 (2006) 1122–1128.

References

[1] Y. Chang , X. Wang , J. Wang , Y. Wu , L. Yang , K. Zhu , Model Total Tests Passed (%) Avg Errors (Std.) H. Chen , X. Yi , C. Wang , Y. Wang , W. Ye , Y. Zhang, llllaammaa--33-- 780bb-i-ninstsrtuructct 640//11000000 ((60.%4%)) 163.8.187((±±22.6 .751) ) Y. Chang , P. S. Yu , Q. Yang , X. Xie , A survey on mixtral-8x7b-instruct 61/1000 (6.1%) 6.79 (±2.24) evaluation of large language models , 2023 . URL: http: mixtral -8x22b -instruct 258/1000 (25.8%) 5.01 (±2 .09) //arxiv.org/abs/2307.03109. arXiv: 2307 . 03109 .

[2] P. Liang , R. Bommasani , T. Lee , D. Tsipras , D. Soylu , It is important to note that this simulated test is not M. Yasunaga , Y. Zhang , D. Narayanan , Y. Wu , A. Kuintegral to the CALAMITA benchmark . While it provides mar , et al., Holistic evaluation of language models, additional insights into the models' performance in a arXiv preprint arXiv: 2211 .09110 ( 2022 ). high-stakes, applied setting, the oficial evaluation metric [3 ] S. Ruder , N. Constant , J. Botha , A. Siddhant , O. Fifocuses solely on overall accuracy . rat, J. Fu, P. Liu, J. Hu , D. Garrette , G. Neubig , et al., Xtreme- r: Towards more challenging and nuanced 5. Limitations multilingual evaluation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural LanConsidering state-of-the-art LLMs, it is possible that guage Processing, Association for Computational one's training sets are contaminated with examples from Linguistics, 2021. the U.S. driving licence test and that these may influence [4 ] J. Kreutzer , I. Caswell , L. Wang , A. Wahab , D. van performance on our benchmark . Furthermore, although Esch, N. Ulzii-Orshikh , A. Tapo , N. Subramani, the benchmark allows the real driving licence test to be A . Sokolov , C. Sikasote , et al., Quality at a glance: An reproduced, it can only assess true-or-false binary an- audit of web-crawled multilingual datasets, Transacswers and not dialogue or reasoning ability . tions of the Association for Computational Linguistics 10 ( 2022 ) 50 - 72 .

[5] Z. Talat , A. Névéol , S. Biderman , M. Clinciu , M. Dey, 6 . Ethical issues S. Longpre , S. Luccioni , M. Masoud , M. Mitchell , D. Radev , et al., You reap what you sow: On the chalAlthough the models may demonstrate positive perfor- lenges of bias evaluation under multilingual settings, mance in this benchmark, it is crucial to recognise that in: Proceedings of BigScience Episode# 5-Workshop such results do not equate to an actual ability to drive or on Challenges & Perspectives in Creating Large Lannavigate safely in real-world environments . The bench- guage Models , 2022 , pp. 26 - 41 . mark assesses the models' ability to process and under- [6] S. Pawar , J. Park , J. Jin , A. Arora , J. Myung , S. Yadav, stand driving-related questions, a far cry from the com- F. G. Haznitrama, I. Song , A. Oh , I. Augenstein , Surplex task of driving a vehicle, which requires perception, vey of cultural awareness in language models: Text decision-making and real-time motor control. and beyond ( 2024 ).

[7] F. Mercorio , M. Mezzanzanica , D. Potertì , A. Serino , A. Seveso , Disce aut deficere: Evaluating llms proifciency on the invalsi italian benchmark , arXiv preprint arXiv:2406.17535 ( 2024 ).