This paper presents a zero-shot evaluation of a large language model applied to automatic reading comprehension in Spanish. The experiment was conducted as part of the PROFE 2025 shared task of IberLEF 2025 workshop. It focused on the first subtask, which consists of multiple choice questions based on DELE exam texts. The model received no training examples, and a task-specific prompt was carefully designed to simulate real test conditions. Specifically, the model tested was Gemini 2.0 Flash, which achieved an accuracy of 0.91, demonstrating strong performance on standardized reading comprehension tasks in Spanish. This result contributes to ongoing research on automatic reading comprehension by showing that large language models can successfully interpret real-world assessment items, provided that they are guided by well-structured and pedagogically motivated prompts.
Automatic reading comprehension is a key area of research within Natural Language Processing (NLP),
concerned with evaluating the ability of language models to understand written texts [
The PROFE 2025 shared task [
According to the Diccionario de términos clave de ELE published by the Instituto Cervantes, reading
comprehension is one of the core communicative skills. It entails not only the decoding of linguistic
information, but also the activation of cognitive, perceptual, attitudinal, and sociocultural processes
that contribute to textual interpretation [
Understanding to what extent LLMs can successfully engage with such multifaceted tasks is critical, as reading comprehension involves far more than syntactic or lexical processing alone. The outcomes of this study provide insights into the current capabilities and limitations of LLMs in Spanish, contributing to the broader discussion on multilingual comprehension and model generalization.
In this context, the SINAI research group contributed to the PROFE 2025 shared task addressing the ifrst subtask, which targets multiple choice reading comprehension exercises in Spanish.
The remainder of this paper is organized as follows. Section 2 describes the PROFE 2025 shared task. Section 3 outlines the methodology and prompt design. Section 4 details the experimental setup and the model used. Section 5 presents the results and a discussion of the main findings. Finally, Section 6 provides the conclusions and outlines directions for future work.
The PROFE shared task [
The task comprises three subtasks but, in the present study, we focus exclusively on the first one. A brief description of each subtask is provided below: 1. Subtask 1: Multiple choice. It involves reading a text, whose length varies according to the proficiency level, and answering a set of multiple choice questions. Each question presents several options, from which the model must select the most appropriate one. 2. Subtask 2: Matching. It requires establishing correspondences between two sets of texts. Items from one set must be matched with their corresponding elements in the other, based on semantic and pragmatic coherence. There is only one possible matching per text, but the first set can contain extra unnecessary texts. 3. Subtask 3: Filling the gap. It consists of completing a partially elided text by inserting appropriate fragments provided in a separate list.
The dataset used in the shared task, IC-UNED-RC-ES, is based on real Spanish language proficiency exams developed by the Instituto Cervantes. These exams, created by certified experts in language assessment, span CEFR levels from A1 to C2 and have been digitized for computational use. The complete dataset contains 282 exams and 855 individual exercises, amounting to a total of 6,146 evaluation points distributed across the three task types: multiple choice, matching, and filling the gap. For this edition of the task, the organizers have used approximately 50% of the available material, while the remaining data have been reserved for future evaluations.
Regarding the evaluation, the main evaluation metric used is accuracy, defined as the proportion of correctly answered items. Performance is assessed from two perspectives. First, at the item level, each correct response is counted individually. Second, at the exam level, a system is considered to have passed an exam if it achieves an accuracy above 60%. This exam-level evaluation is applied only to systems participating in the three subtasks. Accuracy was selected as the primary metric because each question has a single correct answer, and it is also the criterion used to evaluate human candidates in the original DELE exams, enabling direct comparison between human and model performance.
As previously stated, this study focuses on the first subtask of the PROFE 2025 challenge: answering multiple choice reading comprehension questions based on texts from the DELE exams developed by the Instituto Cervantes.
To conduct our experiments, we designed a detailed prompt following established principles of
prompt engineering [
The role assigned to the model was that of an expert in the Spanish language and its teaching as a foreign language, with the additional specification that it was acting as a DELE test developer. This role was selected to match the nature of the task: solving DELE-style comprehension exercises, which require not only linguistic understanding but also familiarity with the structure and intent of standardized language assessments.
In addition to role specification, the prompt incorporated constraints and guidance similar to those
provided to human learners preparing for DELE exams [
Tienes que elegir una de las opciones. Debes comprender la idea principal, hay normalmente una pregunta sobre ello y también la información concreta que hay en el texto, NO sobreentiendas respuestas. Las respuestas verdaderas no siempre son las que más información tengan. En ocasiones debes prestar atención a toda la información y reflexionar, puesto que la respuesta no está en una frase, si no en el conjunto del texto, para ello fíjate en tiempos verbales, marcadores del discurso y pronombres. Recuerda que no siempre son las mismas palabras en el texto y en las preguntas. Algunas preguntas son sobre la intención del texto.
Cada pregunta tiene un identificador único (questionId) que debes respetar exactamente. Estos identificadores tienen este formato: <nivel>_<fecha>_<ejercicio>_<pregunta> Donde: - <nivel> es el nivel del examen (por ejemplo, B2) - <fecha> es la fecha del examen con año de cuatro cifras (por ejemplo, 2010-05-22) - <ejercicio> es el ejercicio, por ejemplo E1 - <pregunta> es el número de la pregunta, como Q1, Q2, etc.
Ejemplo correcto de identificador: B2_2010-05-22_E1_Q3 Debes usar exactamente los identificadores que se te muestran a continuación, sin modificarlos: IDS_VÁLIDOS Devuelve las respuestas en formato JSON con estos identificadores como claves y una sola letra como valor (A, B, C...).
No inventes identificadores ni des explicaciones. No escribas ‘ninguna de las anteriores’.
Ejemplo: { B2_2010-05-22_E1_Q3: A, B2_2010-05-22_E1_Q4: B } TEXTO_EXAMEN
PREGUNTAS: PREGUNTAS_EXAMEN emphasizing attention to textual details, and clarifying that the goal was to select the most appropriate option among several possible answers.
Finally, the prompt concluded with instructions for formatting the output clearly and consistently, ensuring that the model’s responses could be evaluated automatically and without ambiguity.
To arrive at the final prompt configuration, a series of tests were conducted using texts extracted from DELE preparation books in order to evaluate prompt efectiveness prior to the oficial task submission. Furthermore, texts were grouped and evaluated by CEFR level, as the model was found to perform more accurately.
The full prompt used is presented in Figure 1.
This study was conducted using a zero-shot [
The objective was to assess the generalization capacity of the model when faced with unseen domainspecific tasks, specifically standardized reading comprehension exercises from the DELE exams, without relying on fine-tuning or in-context learning.
For the experiments, we used Gemini 2.0 Flash [
The experiments were performed by CEFR levels, generating one response for each exam of the corresponding level. For this, the dataset, provided in JSON format, was filtered to select only the exams of a specific CEFR level (A1, A2, B1, B1, B2, C1 or C2), and each exam was processed as a complete unit including the exam text and all its associated multiple-choice questions. Thus, the model Gemini 2.0 Flash independently evaluated each exam with a dynamically generated prompt (Figure 1). In addition, to ensure robustness, a retry mechanism was implemented that handled possible API speed limits or server errors (RESOURCE_EXHAUSTED), introducing controlled delays between attempts. The parameters dynamically included in the prompt were: • NIVEL_DEL_EXAMEN: Level corresponding to the exam to be evaluated (A1, A2, B1, B1, B2, C1 or C2). • IDS_VÁLIDOS: Explicit control over the inclusion of only valid question IDs. • TEXTO_EXAMEN: Text of the exam to be solved.
• PREGUNTAS_EXAMEN: Exam questions for which an answer is to be generated.
The results obtained in this study are highly promising and suggest that large language models, even in zero-shot configurations, possess substantial capabilities in performing reading comprehension tasks in Spanish. Specifically, the model achieved an accuracy of 0.91, significantly outperforming the provided task baseline of 0.64. This represents a 27 point improvement, underscoring the efectiveness of the prompt design.
Figure 2 presents a bar chart that compares the scores obtained by the diferent participating teams. As shown, our team, SINAI, achieved a result very close to that of the top-performing team, highlighting the competitiveness of our approach.
This level of performance, achieved without prior examples or task specific adjustments, suggests that current language models not only internalize general linguistic patterns but also adopt task oriented reasoning strategies when guided by well-structured instructions.
Nevertheless, it is necessary to assess whether this efectiveness remains consistent across diferent item types and proficiency levels. Although the overall accuracy is high, it would be important to determine whether the model struggles with specific cognitive challenges, such as inference, figurative language, or discourse coherence—phenomena that are often dificult even for human learners.
In addition, during the experimental phase, it was observed that the prompt design had a direct impact on the model’s performance. In particular, prompts that provided clearer, more specific instructions and reflected the communicative context of the exam tended to yield more accurate and coherent responses. This observation suggests that, even in zero-shot settings, the way a task is presented can serve as a trigger for activating the model’s relevant capabilities. It is not merely a matter of issuing an instruction, but of doing so in a manner aligned with the internal logic of the test and the type of processing it requires.
These findings contribute to the field of automatic reading comprehension (ARC) by demonstrating that large language models can efectively solve structured comprehension tasks in Spanish, even without prior task-specific training. The use of authentic test materials and a zero-shot setup highlights the potential of these models for evaluating linguistic and interpretive competence in realistic educational scenarios.
This study has shown that large language models, even in a zero-shot setting, can achieve strong performance in Spanish reading comprehension tasks, as demonstrated by the accuracy of 0.91, which significantly exceeds the task baseline. The success of the proposed approach can be attributed, in part, to the design of the prompt, which closely mirrored the instructions typically given to students preparing for DELE exams.
Beyond the strong results obtained, the findings highlight the growing potential of language models as tools in educational contexts. Their ability to interpret structured tasks and generate reliable responses opens the door to applications such as autonomous exam preparation, exercise generation, and even formative assessment in foreign language teaching.
These findings contribute to ongoing research on automatic reading comprehension, ofering evidence that large language models can perform efectively on real world assessment tasks in Spanish, even in zero-shot conditions.
For future work, it would be valuable to conduct a more detailed evaluation of the model’s performance across diferent CEFR levels and item types. Additionally, it would be of interest to apply the proposed approach to other reading comprehension tasks included in the competition, such as subtask 2 (text matching) and subtask 3 (filling the gap), in which we did not participate in this edition. This would help assess the model’s ability to handle a broader range of evaluation formats.
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Desarrollo Modelos ALIA. This work has also been partially supported by Project CONSENSO (PID2021-122263OB-C21), Project MODERATES (TED2021-130145B-I00) and Project SocialTox (PDC2022-133146-C21) funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, and Project FedDAP (PID2020-116118GA-I00) and Project Trust-ReDaS (PID2020-119478GB-I00) supported by MICINN/AEI/10.13039/501100011033. The research work conducted by Salud María Jiménez-Zafra has been supported by the grant RYC2023-044481-I, funded by MICIU/AEI/10.13039/501100011033 and by ESF+.
During the preparation of this work, the authors used ChatGPT-4-turbo in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.