This paper presents a multi-agent system designed to address the Multiple Choice subtask of the PROFE 2025 shared task, which involves selecting the correct answer to questions derived from Spanish language proficiency exams. The system operates in a zero-shot setting using the Gemini 2.5 Flash large language model. It comprises four specialised agent types working collaboratively to make informed and interpretable decisions. Evaluated on the oficial test set, the system achieved 92.90% accuracy, ranking among the top performers in the shared task.
The Multiple Choice subtask within the PROFE 2025 framework is designed to evaluate the capabilities of automated systems in comprehending and interpreting Spanish texts. Each exercise presents a passage accompanied by a series of multiple-choice questions, each ofering several answer options with only one correct choice. The primary objective for participating systems is to accurately identify the correct answer for each question, thereby demonstrating proficiency in language understanding and reasoning.
The dataset employed for this subtask is the IC-UNED-RC-ES corpus, derived from authentic examinations administered by the Instituto Cervantes. These examinations, meticulously crafted by language assessment experts, aim to measure Spanish language proficiency across various levels, ranging from A1 to C2 as per the Common European Framework of Reference for Languages (CEFR).
The dataset developed for the Multiple Choice subtask comprises questions that span a wide range of proficiency levels, from A1 to C2. Each question is accompanied by a set of candidate answers, among which only one is correct. The core objective for participating systems is to select the correct response from the options provided, thereby demonstrating their capability to interpret and reason over linguistic input.
A distinctive characteristic of the dataset is the variable number of answer choices associated with each question. Specifically, the dataset includes questions with two, three, or four options, introducing an additional layer of complexity. The organisers have released a development set consisting of a very limited sample of ten questions, designed primarily for testing and debugging purposes. A detailed breakdown of the number of questions and corresponding answer options across each CEFR level, for both the development and testing sets, is presented in Table 1. This level distribution highlights not only the diversity of linguistic challenges posed by the task, but also the unbalanced nature of the dataset in terms of dificulty levels, which must be accounted for during model evaluation.
Dev Questions
Dev Options
Test Questions
Test Options
The primary evaluation metric employed for this subtask is accuracy, defined as the proportion of questions for which the system selects the correct answer among the provided alternatives. This metric is both intuitive and interpretable, facilitating benchmarking of system performance. Moreover, its use aligns with established practices in human language assessment, thereby enabling a direct and meaningful comparison between the results obtained by automatic systems and those achieved by human examinees under comparable conditions.
Given the limited availability of development data, we adopt a zero-shot approach leveraging a Large Language Model (LLM) without incorporating any additional fine-tuning or supervised adaptation. To Level A1 A1E A2 A2B1E B1 B1E B2 C1 C2 Total mitigate the challenges posed by low-resource evaluation and enhance answer reliability, we propose a multi-agent architecture specifically designed for the multiple-choice subtask.
The proposed system comprises four distinct groups of agents, each designed to fulfil a specialised role in the decision-making pipeline: (i) Responder Agent, which select the most appropriate answer from the available options; (ii) Blind Responder Agent, which generate a free-form answer without access to the predefined answer options. Their response is subsequently matched to one of the candidates using semantic similarity metrics, allowing us to evaluate alignment between unconstrained reasoning and constrained choices; (iii) Evaluator Agents, which receive the options not selected by the Responder Agent and independently assess whether these alternatives could be correct and (iv) a Moderator Agent, which acts as an adjudicator, resolving disagreements across agents and consolidating their opinions into a final, consensus-based answer.
This architecture is illustrated in Figure 1. All interactions within the system are conducted in Spanish, aligning with the linguistic nature of the task.
For all agents, we employ Gemini-2.5-Flash [
All implementation details, including the source code and the complete set of prompts used in the experiments, are publicly available in our GitHub repository.1
The Responder Agent is tasked with selecting the most appropriate answer from the set of provided options, given a multiple-choice question and its associated context. This agent not only chooses an option but is also required to generate a textual justification supporting its decision. The rationale provides interpretability, allowing downstream agents and human evaluators to better understand the decision-making process.
This agent is based on the simulation of typical test-taking behaviour that reflects the same context and set of choices faced by human test takers.
The Blind Responder Agent addresses the same question as the Responder Agent but without access
to the predefined answer choices. It generates an open-form answer based solely on the question and
context. To reconcile this unconstrained response with the format of a multiple-choice task, a semantic
1https://github.com/JesusFraile/Profe2025-JFraTeam
similarity match is performed between the generated answer and each of the available options. This
matching process is carried out using the paraphrase-multilingual-MiniLM-L12-v2 [
The reason for introducing this agent is because of its ability to emulate an unbiased reasoning process, uninfluenced by distracting choices. This setup allows us to assess whether the correct answer can be naturally inferred from the question-context pair, and to compare this inference with the restricted scenario of multiple choice selection.
For each multiple-choice question, the system instantiates a number of Evaluator Agents equal to the number of available answer options minus one. Each Evaluator Agent is assigned to independently assess one of the options not selected by the Responder Agent. The input provided to each Evaluator consists on the contextual passage, the question, and the specific answer option to be evaluated. Based on this information, the Evaluator determines whether the assigned option could plausibly be correct and generates a textual justification for its judgement.
This design promotes impartial evaluation by avoiding cross-contamination of reasoning between agents. By not exposing Evaluators to the choice of Responder Agent, each judgement remains independent. The inclusion of Evaluators helps to reinforce the critical reasoning capacity of the system, allowing it to reconsider discarded options and thus reducing the risk of overconfident or erroneous selections by the lead Response Agent.
The Moderator Agent acts as the final arbitrator, consolidating the outputs from all other agents to determine the definitive system answer. It reviews the Responder Agent’s chosen answer and justification, the Blind Responder Agent’s response, and the judgments and rationales from all Evaluator Agents. Based on this multi-source evidence, the Moderator selects the most appropriate answer, particularly in cases where disagreements or uncertainty arise among the other agents.
This design decision is inspired by the principle of deliberative consensus. The Moderator serves to synthesise divergent viewpoints into a coherent outcome, ensuring that the final response is not the result of a single heuristic or model decision.
The multi-agent system follows a structured pipeline consisting of the following stages: • Initial Answer Generation. The Responder and the Blind Responder Agent operate simultaneously. The Responder selects an answer from the available options and provides a justification. In parallel, the Blind Responder generates an open-form response without seeing the options. This free-form answer is subsequently aligned with the closest candidate using semantic similarity based on the paraphrase-multilingual-MiniLM-L12-v2 model. • Agreement Check. If both agents independently converge on the same answer (if the Blind Responder Agent’s semantically closest option matches the Responder Agent’s selected answer and the semantic similarity score exceeds a predefined threshold of 0.6) the system accepts this answer immediately. This mechanism leverages agreement between two independently reasoned responses to increase eficiency while minimising spurious matches due to chance or low semantic alignment. • Evaluator Stage. If disagreement occurs, the system initiates the evaluation phase. A set of Evaluator Agents, one per unselected answer option, assess the acceptability of their assigned option. Each Evaluator receives only the context, the question, and the specific answer to evaluate, Responder Agent Only Blind Responder + Similarity Matching Full Multi-Agent System Ezotova JFraTeam (Responder Agent) Ralucaginga Mvictoria Annamwinkler Ymlopez and returns a binary judgement along with a justification. If all Evaluators unanimously reject their assigned options, the original selection by the Responder Agent is confirmed. • Mediation. In cases where at least one Evaluator gives a valid alternative as valid, the system delegates the decision to the Moderator Agent. This agent aggregates all justifications and outputs, resolves conflicts, and synthesises a reasoned consensus. • Finalisation. The Moderator Agent produces the final answer accompanied by an integrated explanation derived from the system’s internal deliberation. This ensures that the output is both well-justified and interpretable.
This modular and deliberative pipeline is designed to adapt its depth of reasoning based on the complexity of each input, combining fast agreement-based shortcuts with deeper analysis when required.
To assess the performance of our proposed system, we conducted a evaluation on the oficial test set of the shared task. The analysis covers three configurations: (i) the standalone Responder Agent; (ii) the Blind Responder Agent coupled with the semantic similarity matching mechanism; and (iii) the full Multi-Agent System.
Table 2 summarises the results of our system in all three configurations. For comparative purposes, we also report the results from the shared task leaderboard, selecting only the best-performing submission from each participating team. Additionally, we include the relative accuracy gap between each configuration of our system and the highest-scoring model overall.
In addition to performance metrics, we conducted a analysis of the decision flow within the multiagent system. This aimed to quantify how frequently each component was activated during inference on the test set. Specifically, we measured the proportion of questions resolved directly by agreement between the Responder and Blind Responder Agents, those that required evaluation by the Evaluator Agents, and those that ultimately triggered the Moderator Agent due to evaluator disagreement. The results are summarised in Table 3.
The results shown in Table 2 and Table 3 provide a detailed evaluation of the multi-agent architecture’s performance and internal dynamics. Notably, both the Responder Agent Only and the Full Multi-Agent System achieved accuracies of 92.96% and 92.90%, respectively. These scores position them within 2.58 and 2.64 percentage points of the top-performing system on the leaderboard. The Responder Agent would rank second overall, while the full multi-agent system would take third place. This demonstrates the system’s robustness and its capacity to approximate top-tier performance through collaborative decision-making.
Conversely, the Blind Responder + Similarity Matching configuration attained a markedly lower accuracy of 55.87%. The semantic similarity module employed in this configuration used the lightweight paraphrase-multilingual-MiniLM-L12-v2 model. While computationally eficient, this lightweight approach struggled to deliver reliable selections. It would be worthwhile in future work to assess the impact of integrating a more powerful semantic similarity model.
Further insights into the system’s internal behaviour were obtained by analysing the contribution frequency of each agent in the decision pipeline (Table 3). Direct agreement between the Responder and Blind Responder Agents resolved 34.21% of the questions, enabling swift decision-making using the most eficient configuration. The remaining 65.79% of cases required additional processing: 49.18% were resolved unanimously by the Evaluator Agents, while 16.61% necessitated final arbitration by the Moderator Agent. These results validate the design of the system’s hierarchical resolution mechanism, in which more computationally intensive processes are activated only when necessary.
In terms of computational cost, the experiments involved a total of approximately 4.333 million processed tokens and 6,373 model requests across the three tasks. These operations correspond to a total of 1,704 questions. On average, this corresponds to approximately 2,543 tokens and 3.74 requests per question.
Despite the strong performance of the full multi-agent system, it is important to note that the Responder Agent Only, operating in a traditional zero-shot setting, still achieved the highest accuracy within our framework and matched the best-performing single submission on the leaderboard. This suggests that while the multi-agent pipeline is efective in resolving ambiguous or conflicting cases, it does not necessarily surpass the capabilities of a well-designed standalone agent when provided with suficient context.
To better understand this discrepancy, further analysis is needed. In particular, it would have been valuable to examine how the accuracy of each system varies across diferent levels of question dificulty or linguistic complexity. Identifying patterns in the types of errors made by the full system versus the Responder Agent could ofer insights into the strengths and limitations of multi-agent deliberation, and guide the development of more adaptive and context-aware coordination strategies.
This paper has introduced a multi-agent architecture designed to solve the task of multiple-choice question answering in Spanish, using zero-shot prompting of a large language model without any task-specific fine-tuning. Our approach leverages the collaborative reasoning of four distinct agent types each with clearly defined roles and contributions to collective decision-making.
Through comprehensive evaluation on the oficial test set of the PROFE 2025 shared task, we have demonstrated that the system achieves competitive performance, closely matching the best-performing submissions. In particular, the Responder Agent alone reached an accuracy of 92.96%, outperforming most leaderboard entries, while the full multi-agent system ofered a comparable performance with added interpretability and robustness mechanisms. This work was supported by DeepInfo (PID2021-127777OB-C22) project.
During the preparation of this work, the authors used GPT-4 in order to: Grammar and spelling check. After using this tool, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.