In this paper, we present an overview of the tasks a and b of the ninth edition of BioASQ challenge, together with a newly introduced task on question answering for developing problems called Synergy. All these tasks ran as part of the BioASQ challenge lab in the Conference and Labs of the Evaluation Forum (CLEF) 2021. The main focus of BioASQ is to promote methodologies and systems for large-scale biomedical semantic indexing and question answering. This is achieved through the organization of yearly challenges which enable the participation of teams around the world in developing and comparing their methods on the same benchmark datasets. This year, 42 teams with more than 170 systems participated in the four tasks of the challenge, with six of them focusing on task 9a, 24 on task 9b and 15 on task Synergy. Correspondingly to the previous years, the participation has increased, indicating the established presence of BioASQ challenge in the field.
Taken together, in the ninth version of the BioASQ challenge were ofered four tasks: (1) a
large-scale biomedical semantic indexing task (task 9a), (2) a biomedical question answering task
(task 9b), both considering documents in English, (3) a medical semantic indexing in Spanish
(task MESINESP9), and (4) a new task on biomedical question answering (task Synergy). In this
section, apart from providing a brief description of the two established tasks (9a and 9b) with
focus on diferences from previous versions of the challenge [
Task 9a focuses on classifying articles from the PubMed/MedLine1 digital library into concepts
of the MeSH hierarchy. Specifically, the test sets for the evaluation of the competing systems
consist of new PubMed articles that are not yet annotated by the indexers in NLM. A more
detailed view of each test set can be seen in Table 1. Similarly to the previous years, the
task is divided into three independent batches of 5 weekly test sets each. Two scenarios are
provided: i) on-line and ii) large-scale. The test sets is a collection of new articles without any
restriction on the journal published. To evaluate the participating systems we use standard flat
and hierarchical information retrieval measures as in previous versions of the task [
The aim of Task 9b is to enable the competing teams to develop systems for all the stages of
question answering in the biomedical domain by introducing a large-scale question answering
challenge. Akin to the previous versions of the task, four types of questions are considered:
“yes/no”, “factoid”, “list” and “summary” questions [
Task 9b is divided into two phases: (phase A) the retrieval of the required information and (phase B) answering the question. Moreover, it is split into five independent bi-weekly batches and the two phases for each batch run during two consecutive days. In each phase, the participants receive the corresponding test set and have 24 hours to submit the answers of their systems. More precisely, in phase A, a test set of 100 questions written in English is released and the participants are expected to identify and submit relevant elements from designated resources, including PubMed/MedLine articles, snippets extracted from these articles, concepts and RDF triples. In phase B, the manually selected relevant articles and snippets for these 100 questions are also released and the participating systems are asked to respond with exact answers, that is entity names or short phrases, and ideal answers, that is natural language summaries of the requested information.
The current BioASQ task B is structured in a sequence of phases. First comes the annotation phase; then with a partial overlap runs the challenge; and only when this is finished does the assessment phase start. This leads to minimal interaction between the experts and the participating systems, which is acceptable due to the nature of the questions that are generated. Namely, we are looking for interesting research questions that have a clear, undisputed answer.
This model is less suitable to developing biomedical research topics, such as the case of COVID-19, where new issues appear every day and most of them remain open for some time. A more interactive approach is needed for such cases, aiming at a synergy between the biomedical experts and the automated question answering systems. We envision such an approach as a continuous dialog, where experts issue open questions to the systems and the systems respond to the questions. Then, the experts assess the responses, and their assessment is fed back to the systems, in order to help improving them. Then, the process continues iteratively with new feedback and new system predictions.
Fig. 1 sketches this vision, which motivates the new BioASQ Synergy task. This new task allows biomedical experts to pose unanswered questions for developing problems, such as COVID-19. Participating systems attempt to provide answers, together with supporting material (relevant documents and snippets), which in turn are assessed by the experts and fed back to the systems, together with new questions. At the same time, we are adapting the BioASQ infrastructure and expand the community to address new developing public health issues in the future. In this introductory year, Task Synergy took place in two versions. Each version was structured into four rounds, of systems responses and expert feedback for the same questions. However, some new questions or new modified versions of some questions could be added to the test sets. The details of the datasets used in task Synergy are available in Table 3.
Contrary to the task B, this task was not structured into phases, but both relevant material and answers were received together. However, for new questions only relevant material (documents and snippets) is required until the expert considers that enough material has been gathered during the previous round and mark the questions as “ready to answer". When a question receives a satisfactory answer that is not expected to change, the expert can mark the question as “closed", indicating that no more material and answers are needed for it.
In each round of this task, we consider material from the current version of the COVID-19
Open Research Dataset (CORD-19) [
Overall, 37 teams from institutes around the world participated in the tasks 9a, 9b and Synergy of the challenge with more than 120 distinct systems. Particularly, six of these teams submitted on task 9a, 24 on task 9b and 15 on task Synergy. Furthermore, we can see from Fig. 2, that the participating teams in tasks 9a, 9b and Synergy are originating from various countries around the world, indicating the international interest in the challenge. We observe that a shift towards the most complex question answering task b, already observed in previous years of the challenge, is still apparent this year, as the number of participating teams is slightly increased as shown in Fig. 3. Detailed descriptions for some of the systems will be available at the proceedings of the workshop. 3.1. Task 9a In this year’s Task 9a, 6 teams competed with a total of 21 diferent systems. Teams that have already participated in previous versions of the task include the National Library of Medicine (NLM) team that submitted predictions with five diferent systems, the Fudan University & Atypon team that participated with 4 systems, and the team from the University of Vigo and the University of A Coruña that participated with two systems. On the other hand, two new teams from Roche and Atypon competed for the first time, submitting results with five and three systems respectively. 3.2. Task 9b This year, 90 diferent systems have submitted predictions for Task 9b in total, for both phases A and B. These systems were developed by 24 teams. In phase A, 9 teams participated, submitting results from 34 systems. In phase B, the numbers of participants and systems were 20 and 70 respectively. There were only three teams that engaged in both phases.
In the first two versions of the new task Synergy, introduced this year, 15 teams participated submitting the results from 39 distinct systems. Although significantly diferent from task b, this task is still about biomedical information retrieval and question answering, therefore the systems for both tasks are expected to share some common ideas and techniques. Therefore, some teams participated in both tasks. In particular, 8 teams participated both task 9b and Synergy as shown in Fig. 5.
This paper provides an overview of the ninth version of the BioASQ tasks a and b, along with the newly introduced task Synergy. Tasks 9a and 9b, are already established through the previous eight years of the challenge, and together with the MESINESP9 task on semantic indexing of medical content in Spanish, and the Synergy task, which ran for the first time, consisted the ninth edition of the BioASQ challenge.
Overall, the BioASQ challenge has been matured and established its presence through these years. Besides continuing the annual tasks a and b, this year we ofered a new biomedical question answering task, Synergy. Similar to previous years, the participation of teams increased and therefore, we consider that the challenge keeps meeting its goal to push the research frontier in biomedical semantic indexing and question answering.
Google was a proud sponsor of the BioASQ Challenge in 2020. The ninth edition of BioASQ is also sponsored by the Atypon Systems inc. BioASQ is grateful to NLM for providing the baselines for task 9a and to the CMU team for providing the baselines for task 9b. The MESINESP task is sponsored by the Spanish Plan for advancement of Language Technologies (Plan TL) and the Secretaría de Estado para el Avance Digital (SEAD). BioASQ is also grateful to LILACS, SCIELO and Biblioteca virtual en salud and Instituto de salud Carlos III for providing data for the BioASQ MESINESP task.