In this paper, we present an overview of the eleventh edition of the BioASQ challenge, which is part of the fourteenth Conference and Labs of the Evaluation Forum (CLEF). BioASQ is a series of challenges focused on promoting methodologies and systems for large-scale biomedical semantic indexing and question answering. This document provides an overview of the tasks b and Synergy in this year's BioASQ edition. Although fewer teams participated in this edition compared to previous ones, more than 80 systems were submitted by 22 teams for these two tasks. Task 11b was the focus of 19 teams while 5 teams participated in task Synergy. Like the previous year, the high percentage of newly registered teams suggests that the interest of the community in large-scale biomedical semantic indexing and question answering remains strong.
In the eleventh edition of the BioASQ challenge, there were ofered three tasks: (1) a biomedical
question answering task (task b), (2) a task on biomedical question answering for open
developing issues (task Synergy), both tasks considering documents in English, and (3) a new task
focused on the detection, normalization, and indexing of clinical procedures (task MedProcNER),
considering medical documents in Spanish [
Task 11b consists of a large-scale question-answering challenge that involves developing systems for all the stages of question-answering in the biomedical domain. As in previous editions, the task examines four types of questions: “yes/no”, “factoid”, “list” and “summary” questions [7]. In this edition, the training dataset available to the competing teams contains 4,719 questions that are annotated with relevant golden elements and answers from previous versions of the task [8]. The teams had to use this dataset to develop their systems. The details of both training and testing sets for task 11b are shown in Table 1.
Unlike previous challenges, task 11b was divided into four independent bi-weekly batches and the two phases for each batch run for two consecutive days. The two phases of task 11b consist of: (phase A) the retrieval of the required information and (phase B) answering the question, which run for two consecutive days for each batch. In each phase, the participants receive the corresponding test set and have 24 hours to submit the answers of their systems. This year, the ifrst two test sets consisted of 75 questions each, and the remaining two test sets consisted of 90 questions each. For each test set, the respective questions, written in English, were released for phase A and the participants were expected to identify and submit relevant elements from designated resources, including PubMed/MedLine articles and snippets extracted from these articles. Then, the manually selected relevant articles and snippets for these questions were also released in phase B and the participating systems were asked to respond with exact answers, that is entity names or short phrases, and ideal answers, that is natural language summaries of the requested information.
The Synergy task was first introduced in the ninth edition of the BioASQ challenge [ 9] with the goal of creating a synergy between the biomedical experts studying the developing issue of COVID-19 and the automated question-answering systems participating in BioASQ. The experts assess the systems’ responses and their assessment is fed back to the systems in order to help improve them, in a continuous iterative process. Figure 1 sketches this procedure. The competing systems provide their initial answers for open questions on developing problems along with relevant documents and snippets. These answers are then evaluated by experts who then provide feedback to the systems and new or pending questions.
This version of the Synergy task (Synergy11) was structured into four rounds, one every two weeks, and was open to any developing problem considering documents from the current version of PubMed that was designated for each round. As in previous versions of the task, the questions were not required to have definite answers and the answers to the questions could be more volatile. A set of 311 questions on COVID-19 was also available from the previous versions of the Synergy task, together with respective expert feedback and answers, and was provided as a development set.
In each round of the Synergy task, the system responses and expert feedback refer to the same questions, unless they have been closed by the experts for having received a full and definite answer that is not expected to change. In Synergy11, in particular, a team of seven biomedical experts contributed 53 open biomedical questions in total and assessed the retrieved material (i.e. documents and snippets) and responses submitted by the participating systems in each of the four rounds. Table 2 shows the details of the datasets used in task Synergy.
Similar to task 11b, four types of questions are examined in Synergy11: yes/no, factoid, list, and summary, and two types of answers, exact and ideal. Moreover, the assessment of the systems’ performance is based on the evaluation measures used in task 11b. After the completion of the Synergy11 task, enough relevant material was identified for providing an answer to about 79% of the questions. In addition, about 42% of the questions had at least one ideal answer, that had been submitted by the systems, which was considered satisfactory (ground truth) by the expert that posed the question.
This year’s challenge saw more than 80 distinct systems participating in tasks 11b and Synergy11 with a total of 22 teams. Specifically, 19 of these teams submitted on task 11b and 5 on task Synergy11. Furthermore, Figure 2 illustrates the international interest in the challenge as the participating teams originate from various countries around the world.
As already observed in previous years of the challenge, the participation in task b is surpassing the participation of Synergy. The overall number of participating teams this year is similar to last year’s with a slight decrease, as illustrated in Figure 3. However, the high percentage of teams that participated for the first time in the BioASQ challenge (red circles in Figure 2), suggests that the interest of the community in large-scale biomedical semantic indexing and question answering remains strong. A total of 7 new teams participated in this year’s editions of tasks b and Synergy of the BioASQ challenge. 3.1. Task 11b In task 11b, 19 teams competed this year with a total of 76 diferent systems for both phases A and B. In particular, 9 teams with 37 systems participated in phase A, while in phase B, the number of participants and systems were 16 and 59 respectively. 6 teams engaged in both phases.
In task Synergy11, 5 teams participated this year with a total of 12 distinct systems. As this task shares some common ideas with task 11b, some teams participated in both tasks. Specifically, 2 teams participated in both task 11b and Synergy11 as shown in Figure 5. However, as already observed in previous versions of the tasks, less teams participate in Synergy11 than in task 11b. This could be due to the particularities of open questions in Synergy, such as the volatility of answers and the evolving nature of the relevant knowledge which make the task more challenging than traditional question answering.
In this paper, we presented the eleventh version of the BioASQ tasks b and Synergy. Both tasks are already established through the previous versions of the challenge. The participation of teams was comparable to last year’s version of these tasks with a slight decrease. On the other hand, we noticed a high number of newly registered teams. Therefore, we believe that the challenge and the datasets developed for its tasks increase the research community’s interest in question answering and encourage the development of better solutions to aid biomedical researchers’ access to the abundance of biomedical knowledge.
Google was a proud sponsor of the BioASQ Challenge in 2022. The eleventh edition of BioASQ is also sponsored by Ovid. Atypon Systems Inc. is also sponsoring this edition of BioASQ. The MEDLINE/PubMed data resources considered in this work were accessed courtesy of the U.S. National Library of Medicine. mantic indexing and question answering competition, BMC Bioinformatics 16 (2015) 138. doi:10.1186/s12859-015-0564-6. [7] G. Balikas, I. Partalas, A. Kosmopoulos, S. Petridis, P. Malakasiotis, I. Pavlopoulos, I. Androutsopoulos, N. Baskiotis, E. Gaussier, T. Artieres, P. Gallinari, Evaluation Framework Specifications, Project deliverable D4.1, UPMC, 2013. [8] A. Krithara, A. Nentidis, K. Bougiatiotis, G. Paliouras, BioASQ-QA: A manually curated corpus for Biomedical Question Answering, Scientific Data 10 (2023) 170. [9] A. Nentidis, G. Katsimpras, E. Vandorou, A. Krithara, L. Gasco, M. Krallinger, G. Paliouras, Overview of BioASQ 2021: The Ninth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2021, pp. 239–263.