BioASQ task B focuses on biomedical information retrieval and question answering. This paper describes the participation and proposed solutions of our team. We build a system based on recent advances in the general domain as well as the approaches from previous years of the competition. We adapt a system based on a pretrained BERT for document and snippet retrieval, question answering and summarization. We describe all approaches we experimented with and show that while neural approaches do well, sometimes baseline approaches have high automatic metrics. The proposed system achieves competitive performance while being general so that it can be applied to other domains as well.
BioASQ [27] is a large scale competition for biomedical research. It provides evaluation measures for various setups like semantic indexing, information retrieval and question answering, all regarding the biomedical domain. The competition takes place annually online, and each year gains more attention from research groups all around the world. The BioASQ provides necessary datasets, evaluation metrics and leaderboards for each of its sub-challenges.
More speci cally, the BioASQ challenge consists of two major objectives which are called \tasks". The rst is semantic indexing, which goal is to construct a search index given a set of documents, such that certain semantic relationships are held between the index terms. The second objective is passage ranking and question answering in various forms, which is given a question to return a piece of text. The returned text must either answer the question directly or contain enough information to derive the answer. In terms of the BioASQ, those objectives are called Task A and Task B, respectively.
In this work, we explore applications of the state-of-the-art model in natural language processing and deep learning in biomedical question answering. As a result, we develop a system, that is capable of providing answers in the form of documents, snippets, exact answers or abstractive text, given biomedical questions from various domains. We evaluate our system on the recent BioASQ 2020 challenge, where it achieves competitive performance. 1.1
Our team participated in the Task B, which involves information retrieval, question answering, summarization and more. This task uses benchmark datasets containing development and test questions, in English, along with gold standard (reference) answers constructed by a team of biomedical experts. The task is separated into two phases.
Phase A The rst phase measures the ability of systems to answer biomedical questions with a list of relevant documents and snippets of text from retrieved documents. The main metric for documents and snippets is the mean average precision (MAP). The average precision is de ned as follows:
AP =
PjrL=j1 P (r) rel(r)
jLRj
where jLj is the number of items in a list predicted by the system, jLRj is the
number of relevant items. P (r) is a precision when only rst r returned items are
considered, and rel(r) is equal to 1 if the r-th returned item is relevant. MAP
and GMAP are arithmetic and geometric means of all questions in the evaluation
set. For the snippets retrieval, precision is measured in terms of characters, and
rel(r) is equal to 1 if the returned item has non-zero overlap with at least one
relevant snippet. Additional metrics are precision, recall and F1 score. A more
detailed description is present in the original paper [
Phase B The second phase evaluates the performance of question answering, given a list of relevant documents and snippets from the previous phase. The questions are of several types: questions where the answer is either yes or no (\yes/no"), questions where the answer is a single term (\factoid"), and questions where the answer is a list of terms (\list"). Additionally, each question has an \ideal" answer, where the aim is to measure the systems' ability to generate paragraph sized passage, that answers the question.
The metrics of phase B are F1-macro for yes/no questions, mean reciprocal
rank (MRR) [29] for factoid questions and F1 score for list questions. To
evaluate answers in natural language the ROUGE [16] scores are used. We should
note that human experts will additionally evaluate all systems after the contest.
However, the results are not available at the time of writing this paper, thus we
use only the automatic measurements to draw our conclusions.
1.2
Most of the contemporary large scale QA systems attempt to ll the gap between
massive source of knowledge and a complex neural reasoning model. Many of the
popular knowledge sources are sets of unstructured or semi-structured texts, like
Wikipedia [
During document retrieval a question answering system can bene t from
structured knowledge as well. There is a rich set of di erent biomedical ontologies
like UMLS [
Many systems perform re-ranking after initial document retrieval. Specialized
neural models like DRMM [
Some systems solve snippet extraction by utilizing methods that were originally developed for document re-ranking. [22] uses the earlier mentioned DRMM in the Task 7B and achieves top results. [23] comes up with another neural approach, employing both textual and conceptual information from question and candidate text. In our work, we experiment with di erent methods and show how strong baselines consistently demonstrate high metrics, given a proper document retriever.
Deep learning has shown its superiority in question answering. In the Task
5b, [30] achieve top scores by training an RNN based neural network. However,
most of the modern advancements in question answering can be attributed to
transformer-based models. Last years' challenges were dominated by systems
that used BERT or its task-speci c adapations, like BioBERT [34], [
Deep neural and transformer based models in particular have shown their
ability to tackle summarization in di erent setups [17], including QA
summarization [15]. However, as [19] noticed, BioASQ summaries tend to look very
similar to the input examples. They exploit this observation and introduce several
solutions, based on sentence re-ranking, and achieve top automatic and human
scores in several batches. There is also an attempt to utilize pointer-generator
networks [26] for BioASQ ideal questions [
In this section, we describe the system we implemented for document and snippet retrieval, as well as our question answering system. We provide the results of di erent approaches which we experimented with during the competition. To assess the performance of di erent methods more accurately, we merge all test batches of 8B task into one and use resulting 500 questions as an evaluation set. Here and during the competition, we created a development set using 100 questions from 6B and 200 questions from 7B task. Our system evolved from batch to batch, achieving its nal shape in batch 5. All the ablation experiments and retrospective evaluations are performed on the system, that was used for the 5-th batch submission. 2.1
For document retrieval, we implement a system conceptually similar to [21]. At rst we extract a list of N document candidates using Anserini implementation of bm25 algorithm [32]. Then we use the BERT model to re-rank candidate documents and output at most ten top-scored documents.
BM25 For initial document retrieval, we used Anserini [32]. We created an
index using the PubMed Baseline Repository of the 2019 year [
BERT Re-ranking The initial set of documents obtained with BM25 is passed to the BERT re-ranker, which assigns relevance scores to documents based on a question. We consider all documents with a score higher than a threshold to be relevant and output at most ten papers with the highest scores. To train BERT re-ranker, we created a binary classi cation dataset. We obtained positive examples from the gold documents of BioASQ dataset. We collected negative examples using BM25 by extracting 200 documents using a question as a query and consider all documents starting from position 100 to be non-relevant if they are not in the gold documents set. As BioASQ dataset for question answering contains questions collected from the past year contests, the relevant documents include only the papers published before the year of the competition. Usually, there are several relevant documents for the question that were published after the year of the contest. To exclude such papers from the negative examples, we calculated the maximum publication year for all relevant documents and ltered all documents published after this year from the negative examples. Experiments We evaluated several approaches to document retrieval. First, we evaluated the performance of the BM25 algorithm, and then we applied di erent modi cations of BERT-based re-ranker. We examined the e ects of relevance score threshold as well as the number of documents obtained from the BM25 stage. The results are presented in table 1. We can see how BERT-based re-ranker consistently improves base BM25 performance.
Since our re-ranker is trained to perform logistic regression, we can vary the decision boundary to achieve an appropriate trade-o between precision and recall. However, the MAP metric, which is used as a nal ranking measure, does not penalize the system for additional non-relevant documents, which means the system should always output as much documents as possible to achieve the highest score, while reducing its practical usefulness. We decided to orient our system towards both precision and recall and as a result we achieve the highest F1 scores across all batches, while maintaining competitive MAP scores. Snippet extraction systems extract a continuous span of text from one of the relevant documents for the given question. We observe that snippets from the BioASQ training set are usually one sentence long, thus our system is designed as a sentence retriever and snippet extraction is formulated as a sentence ranking problem. We experiment with both neural and statistical approaches to tackle this challenge.
Baseline We use a simple statistical baseline for sentence ranking, which is based on measuring entity cooccurrence in question and candidate sentence. For each question and sentence we extract sets of entities Q and S respectively and compute relevance score: relevance(q; s) = jQ \ Sj jQj
We use ScispaCy [20] en core web sm model for extracting entitities. Word2Vec Similarity One approach of determining sentence similarity is to map both query and candidate into the same vector space and measure the distance between them. For embedding word sequences, we use Word2Vec model, pretrained on PubMed texts [18], and compute the mean of individual word embeddings. Suppose the Eq and Es are the embeddings of question and snippet correspondingly. The relevance of snippet for a given question is a cosine similarity between embeddings: relevance(q; s) =
Eq Es jjEqjjjjEsjj BERT Similarity As the transformer pretrained on the biomedical domain should contain a lot of transferable knowledge, we check the zero-shot performance of the pretrained model. Similar to embeddings similarity, we use cosine distance between the embeddings of question and snippet. The embedding of a text span is the contextualized embedding corresponding to the special [CLS] token which is inserted before the tokenized text. The relevance is a cosine similarity between embeddings of question and snippet.
BERT Relevance As the task of snippet retrieval is very similar to
document retrieval, we test a similar approach. We use BERTrel model, trained for
document ranking to assign a relevance score to the pair of question and snippet:
relevance(q; s) = BERTrel(q; s)
Document Scores Finally, after assigning each question-sentence pair a
relevance score, we scale the latter by additional score, based on the position of
the document, which the candidate sentences are extracted from, in the list
of relevant documents. Despite the simplicity of this trick, experiments show
considerable improvements of evaluation metrics, which points out a strong
correlation between the rank of the abstracts and the rank of the snippets from
those abstracts. For each document di from the list of ranked relevant
documents D = d1; d2; : : : ; dn there is a list of sentences Si = si;1; si;2; : : : ; si;m and
the similarity score between query q and sentence si;j is:
score(q; si;j ) =
relevance(q; si;j )
i
Experiments We evaluated all described approaches to snippet retrieval. The
results are presented in table 2. We can see that heuristic of adding
document score into the score of a snippet allows to improve MAP scores for all
approaches signi cantly. In line with the document retrieval, BERT relevance
model has higher precision and recall with lower MAP scores. Surprisingly,
retrieval based on BioBERT cosine similarly performed well even without training
on any BioASQ data. We can consider the approach to be a zero-shot
performance of BioBERT on the task of snippet retrieval.
Factoid and List questions. For factoid and list questions we generate
answers with a single extractive question-answering system. Its design follows the
classical transformer-based approach, described in [
Throughout all the 5 batches we experiment mainly at pre- and post-
processing stages, without substantial changes in the architecture of the system
itself. During preprocessing, we convert input questions to the SQuAD format
[25], where contexts are built from the relevant snippets, that come with each
input question. The postprocessing stage is implemented in the same manner as
[34]. However, for list question we additionally split the resulting extracted spans
by \and/or" and \or" conjunctions, which we observed to be frequently used in
chemical/gene enumerations in various biomedical abstracts. Table 3 shows the
importance of this step.
Yes/No questions. For yes/no questions we formulate the task as a logistic
regression over question-snippet pairs and implement a transformer-based
approach, similar to [34]. We use the ALBERT model and ne-tune it using SQuAD
and BioASQ datasets. In the fth batch, we additionaly use PubMedQA dataset
[
Phase B also includes summarization objective, where a participating system has to generate a paragraph sized text, answering the question. We come up with di erent approaches for tackling this challenge.
Weak baseline BioASQ does not impose any limitations on the source of
the summary. We observed that summaries tend to be one or two sentences
long, reminding how snippets are composed. Straightforward approach is to use
snippets, provided with the question for computing the summary. Our weak
baseline selects the rst snippet from the question for this purpose.
Snippet Reranking Naturally, the rst snippet may not answer the question
directly and clearly, despite being marked as the most relevant. A logical
improvement to the baseline is to select the appropriate snippet, potentially in a
question-aware manner. To make answers more granular, we split snippets by
sentences and the resulting candidate pool contains snippets and snippet
sentences. Sometimes, however, snippets are absent for a given question. In that case
we extract the candidate sentences from the relevant abstracts. For re-ranking,
we use BERTrel trained for document re-ranking, as described in 2.2. Overall,
we can describe this system as sentence-level extractive summarization.
Abstractive Summarization Our nal system performs abstractive
summarization over provided snippets. We use traditional encoder-decoder transformer
architecture [28], where the encoder is based on BioMed-RoBERTa [
In this section, we present an o cial automatic evaluation of our system, comparing to the top competitor system. We denote our system as \PA" which stands for the Paper Analyzer team. We additionally perform a retrospective evaluation of phase A, where the gold answers are available. 3.1
In table 4, we present the results of our document retrieval system on all batches compared to the top competitor. The nal design of our system was implemented only in the fth batch. So, to evaluate our proposed system against our own and other participants' systems from previous batches, we computed evaluation metrics over golden answers, provided by BioASQ for the Phase B. We were able to fully reproduce o cial leaderboard scores for the fth batch and show, that our nal system outperforms all our previous submissions. The retrospective evaluation shows that we signi cantly improved our system during the contest and achieved better results with the nal system. In table 4, we present the results of our snippet retrieval system on all batches compared to the top competitor. Similar to the document retrieval, we performed a retrospective evaluation on all batches for the nal implemented system. The evaluation shows that we signi cantly improved our system during the contest. We submitted only baselines for batches 1 and 2, so we present results only for batches starting with 3. Overall, we achieved moderate results on the question answering task, as we mainly focused on Phase A. We believe this was caused by poor selection of the training dataset. We will analyze errors and perform additional experiments in the future. The performance of our system is presented in tables 5 and 6. We evaluated our systems in all the ve batches. However, we were able to experiment with only one system per batch. The results are presented in the table 7. We show how simple snippet re-ranker can achieve top scores in automatic evaluation. Meanwhile the abstractive summarizer, while providing readable and coherent responses, achieves lower scores, however still very competitive ones. We hope that human evaluation will show the opposite results. We included side-by-side comparison of answers provided by both systems in the appendix (table 8). In this work, we demonstrate several approaches for our rst participation in BioASQ competition. We propose a working system for each of the evaluation objectives in Task 8B, achieving top positions in several batches.
We designate a few areas for improvements. First, a more robust experimentation process is required. After the challenge, we found several errors in document and snippeet retrieval pipelines that could a ect the resulting performance. Second, our system does not utilize semantic indexing, which can be implemented using di erent biomedical ontologies.
Appendix: Side-by-Side Comparison of Snippet Ranking and Generative Models. [14] Zhenzhong Lan et al. \ALBERT: A Lite BERT for Self-supervised Learning of Language Representations". In: (Sept. 2019). arXiv: 1909.11942. url: http://arxiv.org/abs/1909.11942. [15] Mike Lewis et al. \BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension". In: ArXiv abs/1910.13461 (2020). [16] Chin-Yew Lin. \Rouge: A package for automatic evaluation of summaries".
In: Text summarization branches out. 2004, pp. 74{81. [17] Yang Liu and Mirella Lapata. \Text Summarization with Pretrained Encoders". In: EMNLP/IJCNLP. 2019. [18] Ryan McDonald, George Brokos, and Ion Androutsopoulos. \Deep Relevance Ranking Using Enhanced Document-Query Interactions". In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018). doi: 10.18653/v1/d18-1211. url: http://dx.doi. org/10.18653/v1/D18-1211. [19] Diego Molla and Christopher Jones. \Classi cation Betters Regression in Query-Based Multi-document Summarisation Techniques for Question Answering". In: Communications in Computer and Information Science (2020), pp. 624{635. issn: 1865-0937. doi: 10.1007/978-3-030-438876_56. url: http://dx.doi.org/10.1007/978-3-030-43887-6_56. [20] Mark Neumann et al. \ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing". In: Proceedings of the 18th BioNLP Workshop and Shared Task. Florence, Italy: Association for Computational Linguistics, Aug. 2019, pp. 319{327. doi: 10.18653/v1/W19- 5034. eprint: arXiv:1902.07669. url: https://www.aclweb.org/anthology/W195034. [21] Rodrigo Nogueira and Kyunghyun Cho. \Passage Re-ranking with BERT".
In: arXiv e-prints, arXiv:1901.04085 (Jan. 2019), arXiv:1901.04085. arXiv: 1901.04085 [cs.IR]. [22] Dimitris Pappas et al. \AUEB at BioASQ 7: Document and Snippet Retrieval". In: Machine Learning and Knowledge Discovery in Databases. Ed. by Peggy Cellier and Kurt Driessens. Cham: Springer International Publishing, 2020, pp. 607{623. isbn: 978-3-030-43887-6. [23] Monica Pineda-Vargas et al. \A Mixed Information Source Approach for Biomedical Question Answering: MindLab at BioASQ 7B". In: Machine Learning and Knowledge Discovery in Databases. Ed. by Peggy Cellier and Kurt Driessens. Cham: Springer International Publishing, 2020, pp. 595{ 606. isbn: 978-3-030-43887-6. [24] Pranav Rajpurkar, Robin Jia, and Percy Liang. \Know What You Don't Know: Unanswerable Questions for SQuAD". In: arXiv e-prints, arXiv:1806.03822 (June 2018), arXiv:1806.03822. arXiv: 1806.03822 [cs.CL]. [25] Pranav Rajpurkar et al. \SQuAD: 100,000+ Questions for Machine Comprehension of Text". In: arXiv:1606.05250 (June 2016). arXiv: 1606.05250 [cs.CL]. url: http://arxiv.org/abs/1606.05250. [26] Abigail See, Peter J. Liu, and Christopher D. Manning. \Get To The Point: Summarization with Pointer-Generator Networks". In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2017). doi: 10.18653/v1/p17-1099. url: http://dx.doi.org/10.18653/v1/P17-1099. [27] George Tsatsaronis et al. \An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition". In: BMC Bioinformatics 16 (Apr. 2015), p. 138. doi: 10.1186/s12859-015-05646. [28] Ashish Vaswani et al. \Attention is All you Need". In: ArXiv abs/1706.03762 (2017). [29] Ellen M Voorhees. \The TREC question answering track". In: Natural
Language Engineering 7.4 (2001), p. 361. [30] Georg Wiese, Dirk Weissenborn, and Mariana Neves. \Neural Question Answering at BioASQ 5B". In: BioNLP 2017 (2017). doi: 10.18653/v1/ w17-2309. url: http://dx.doi.org/10.18653/v1/W17-2309. [31] Peter Willett. \The Porter stemming algorithm: then and now". In: Program (2006). [32] Peilin Yang, Hui Fang, and Jimmy Lin. \Anserini: Reproducible Ranking Baselines Using Lucene". In: J. Data and Information Quality 10.4 (Oct. 2018). issn: 1936-1955. doi: 10.1145/3239571. url: https://doi.org/ 10.1145/3239571. [33] Wei Yang et al. \End-to-end open-domain question answering with bertserini". In: arXiv preprint arXiv:1902.01718 (2019). [34] Wonjin Yoon et al. \Pre-trained Language Model for Biomedical Question Answering". In: arXiv:1909.08229 (Sept. 2019), arXiv:1909.08229. arXiv: 1909.08229 [cs.CL]. url: http://arxiv.org/abs/1909.08229.