Deep Neural Network (DNN) based Machine Learning models achieved remarkable success in many elds of research. Yet, many recent studies show the limitations of these approaches to generalize to unseen examples and to new domains such as the biomedical domain. Besides, supervised-learning based DNN models require a substantial amount of labeled data which is not readily available for many tasks such as the biomedical question answering task. Transfer Learning is shown to mitigate these challenges by transferring information from auxiliary tasks to improve the performance on a source task, and shown to be especially useful for low-resource tasks. These observations and ndings motivated us to investigate the e ect of transfer learning and multi-task learning on the biomedical question answering task. We proposed a novel multi-task learning model to learn biomedical entities and questions simultaneously. In this work, we explain the three di erent neural models we used to participate for the BioASQ 8B challenge. Our initial results showed that transferring information from the biomedical entity recognition task brings improvement for the biomedical question answering task.
Pretrained language models [
Transfer learning is a general term to describe the learning schemes where
the information from a source task is used to improve the performance on a
target task. It is shown to be especially useful to improve the performance on
low-resource tasks [
BioASQ is a challenge on biomedical semantic indexing and question
answering [
In this section we describe each model we used during the BioASQ Task 8b: Biomedical Semantic Question Answering. During the task, we made submissions using ve di erent models, three of which used an identical neural architecture, but the nal model is determined using di erent evaluation methods. We used BioBert-based Question Answering Model [17] as our baseline model, which we refer to as BioBERT baseline. The second model is an extension of the rst model, which jointly learns all question types using a single architecture. We refer to this model as BioBERT allquestions. We used three variations of this model for our submissions. Finally, we used a novel multi-task learning model that learns biomedical entities and all question types simultaneously. We refer to this model as BioBERT multitask. For the BioASQ 8B challenge, we only submitted answers for the `list', `factoid', and `yes-no' type questions. `Summary' type questions require a fundamentally di erent approach, and was beyond the main scope of this work. 2.1
The raw input format of the BioASQ dataset needs to be pre-processed into the suitable format expected by the BioBERT model. Following Yoon et al. [17], we used a similar pre-processing scheme to convert the BioASQ questions into the SQUAD Question Answering format. In the BioASQ dataset, multiple goldanswers are provided for most questions. Gold answers are denoted as spans inside the snippets provided for each question. During pre-processing, we treated each gold-label snippet and question pair as separate examples to increase the size of the training set. During all our experiments, we only made use of the gold-label snippets. We did not analyze the e ect of appending additional information from external sources such as the links to related documents provided by the BioASQ organizers. Previously, Yoon et al. [17] experimented with various pre-processing methods to bring further improvements. They observed that the bene ts of each strategy depend on the question type and the test-batch. For this reason, we xed the pre-processing method throughout our all experiments to make it clear where the improvements for each proposed model come from. Besides, using only the snippets as input to the neural networks signi cantly reduces the input size and reduces the overall training time. For factoid and list type questions, each gold-label span is used to create a new Question-Passage pair. An example factoid type question and gold-label spans from the provided spans are given in Table 1. The nal predictions for the list type questions are handled during the post-processing step, and explained in the relevant section.
Contrary to the previous work that directly adapts the BERT Question
Answering Model [
Pretrained subword contextual embeddings has shown remarkable progress over
previous approaches on many downstream Natural Language Processing (NLP)
tasks [
The performance of models pretrained on general domain corpora (e.g.,
Wikipedia articles) drops signi cantly when tested on niche domains such as
the biomedical domain. Motivated with this observation, Lee et al. [
For our baseline model, we used this BioBERT-based approach which we refer to as BioBERT baseline. BERT model is extended with two separate additional neural layers to learn di erent question types. The overall architectures are given in Figure 1. For the yes/no type questions, the output for the rst token ([CLS]) ) of the nal layer of BERT is given as input to a fully connected layer with 2-dim output representing the scores for yes/no scores. This is followed by a softmax layer to convert these scores into probabilities. Given a sequence of n question tokens Q = qt : 1 t n, and m passage tokens P = pt : 1 t m , BioBERT outputs m + n + 2 xed-size (L) vectors V = vj : 1 j (m + n + 2). Next, v1 is multiplied with an (L; 2) dimensional matrix W to generate scores, S = fsyes; snog, for yes and no answers:
V = BioBERT (Q; P ) S = v1T W
O = Sof tmax(S) where O = oyes; ono represents the probabilities for each answer, which is the nal output for the yes/no type questions. Similarly for the factoid/list type questions, each vj is multiplied with an (L; 2) dimensional matrix W2 to generate scores S2 = fsstart; sendg, which represent the score for the start, end spans for each token pj inside the input passage P :
V = BioBERT (Q; P )
S2 = vjT W
For training, each BioBERT-initialized model in Figure 1 is ne-tuned on the BioASQ-8b for each question type, separately. The main drawback of this previously proposed model is that the common BioBERT layer, which constitutes the majority of the parameters (only a single layer is added for each question type), is ne-tuned separately for each question type. The bottleneck for developing high-performing biomedical question answering systems is the scarcity of the labeled training sets. This approach further limits the training dataset size, and not ideal for low-resource domains like the biomedical domain. (a) Yes/No model.
(b) Factoid/List model.
Fig. 1: Overall architectures for training separate models for yes/no and factoid/list question types for the BioBERT baseline model [17]. The common BioBERT model layers are netuned from scratch for each type. 2.3
The baseline approach does not expose the model to all the examples in the training dataset. This observation motivated us to propose a novel joint-learning model, which uses a single architecture to learn all question types, which we refer to as BioBERT allquestions. Learning of all question types using a single BERTbased model is not employed before in this domain, to the best of our knowledge. The overview of the proposed joint-learning model is shown in Figure 2. This simple extension to the previously proposed BioBERT-based QA Model [17] allows exposing the model to all the available examples in the training dataset. The common BioBERT layer is trained jointly on all question types. This allows the model to transfer information from other question types for better generalization.
An important part of training joint-learning models is the selection of performance metrics. In the conventional single-task machine learning setting, there is usually a single performance metric. The models are evaluated on a development/validation dataset based on this metric, to determine the best performing model during training. In the joint-learning setting, we can evaluate the models based on their performance on each task separately, or we can evaluate them based on their overall performance. For our submissions for the BioASQ 8B challenge we used the following three joint-learning models: { Overall best-performer { Best yes/no model { Best factoid model
To determine the best-performer in each three cases, we used the average
results over ve test-batches of the Bio-ASQ 6B challenge [
S = [s1; :::; sj; :::; sn] crf loss = f orward score(S; T) path score(S; T; G) where S denotes the scoring matrix containing scores for each label and word pair, G is the gold-label sequence, and T is the transition matrix containing transition scores between each label. f orward score(S; T) denotes the total score of all paths and path score(S; T; G) is the score of the gold label sequence. Ideally, we want all probabilities to accumulate on the gold-label path so that these two scores will be identical.
Inference During the inference mode, we used Viterbi decoding [
As explained in the pre-processing section, we divided each question with multiple gold-label snippets into separate inputs. These examples are merged during post-processing to generate a unique answer for each question. For the postprocessing step, we followed [17] to combine the predictions to the same question for factoid/list type questions. Majority voting is used to nd the highest scoring predictions for each factoid/list type question. For each factoid type question, top N highest scoring predictions are returned where N corresponds to the maximum limit allowed for the BioASQ 8B challenge. For the list type questions, we used 0.50 as the probability threshold, and included all answers that have a higher average probability score.
For the yes/no type questions, we averaged the probability scores for each example belonging to the same question instance to determine the nal answer. 3
In this section we explain details regarding the experiments we conducted. All
experiments are done using a single V100-GPU. For the Question Answering task
we used the BioASQ 6B test sets as our validation set, and used the examples
in the BioASQ 8B training set, for training. For the entity recognition task, we
kept the same train/dev/test split already provided in [
The entity recognition component of the nal multi-task learning model we used
for our submissions is trained on the BC2GM dataset [
In order to evaluate our proposed multi-task learner, we trained the entity
recognition component on three di erent datasets. We used the BC2GM [
The BioASQ 8B training set contains 3,243 questions in total. We did not make use of the 777 summary type questions, so our overall training set contained 2466 questions. For training our models we used only the snippets already provided by the challenge organizers as the relevant passage for each question. Each snippet and question is treated as a unique (Q; P ) pair which is given as input to the question answering component, where Q and P represent `question' and `passage', respectively.
For evaluating our proposed models, we also used the factoid questions from
the BioASQ 6B test set [
Joint-learning model training The joint learner (BioBERT allquestions) is trained on all question types at once. At each iteration a (Q; P ) pair is picked randomly from the whole training set. If the picked example is a `yes/no' type question the `Yes/No' component in Figure 2 is used to generate the output of the model. Otherwise, the `Factoid/List' component is used to generate the `start' and `end' scores for each token inside the given passage P . The loss for each input example is backpropagated to update the weights of 1) the questiontype speci c component, and 2) the common BioBERT component. This way, we allow information transfer between di erent question types. Considering the relatively small sizes of the biomedical question answering datasets, this allows a better utilization of what is available. Besides, this approach reduces the total number of parameters of the nal model almost by half, as the majority of the trainable parameters are the common BioBERT weights. As we have multiple target performance metrics (overall performance and performances on each question type), we continued the training until we could not observe any improvement for any question type on the question answering validation sets.
(BioBERT multitask) is simultaneously trained for the question answering and the entity recognition tasks. At each iteration we ip a random coin to determine the task type (QAS or NER), and use the corresponding component from Figure 3. Similar to the joint-learning model this allows information transfer from the NER dataset examples for the question answering task. The common BioBERT model is updated using examples from both tasks, which allows us to expose the model for a signi cantly larger amount of sentences from the biomedical domain. In this work, entity recognition task is used as an auxiliary task to help improve the nal performance on the target question answering task. For this reason, training is done until we could not observe any improvement on the question answering validation set. 4
In this section we start by giving the results we obtained for evaluating our proposed multi-task learner. We compare BioBERT multitask, which learns both entity recognition and question answering tasks simultaneously, with the jointlearning model BioBERT allquestions, which only focuses on the question answering task. The BioASQ 8B data is used to train both models, and the factoid type questions from the BioASQ 6B challenge is used to evaluate them, which contains ve di erent test batches. For training the entity recognition component of the multi-task learning model, we used three di erent biomedical entity datasets. The results for both models are given in Table 6. Our results showed that learning both tasks simultaneously improved the performance for all entity datasets and for all test batches. For all three datasets we observed that the multi-task learning model outperformed the model that only learns the question answering task on all ve test-batches. These results veri ed our initial claim on transfering information from entity recognition task to improve the performance on the target question answering task, and motivated us to apply the proposed multi-task learning model on the BioASQ 8B test sets.
Next, we give the results obtained on the BioASQ 8B challenge for each model we explained above. For the rst test-batch we only made submissions using two models: BioBERT baseline, BioBERT allquestions. For the other four test-batches we made ve submissions using the three models explained above. To be able to make a clean comparison between the proposed models, we kept the post-processings schemes identical for all our submissions. This is necessary to evaluate our claim on using multi-task learning to improve the performance on biomedical question answering task.
The QAS components of the joint-learning model and the model-task learning model are identical. In order to evaluate our claim on using multi-task learning for question answering, we must compare these models, rather than comparing them with the single-task learning model which uses a di erent architecture (separate models for each question type). The results show that for the factoid questions, the multi-task learning based model outperformed all three joint-learning models for all four test-batches. This clearly shows that leveraging information obtained about genes and proteins may help improve the nal performance on the factoid type questions. The results for list and yes/no type questions are mixed, and the bene ts of multi-task learning are unclear for these types.
In this paper we described the models we used to make submissions for the BioASQ 8B challenge. We proposed a novel multi-task learning model for biomedical entity recognition and question answering tasks. Our results showed that transferring information from the entity recognition task consistently improved the performance on the factoid type questions of the question answering tasks. On all test-batches of both BioASQ 6B and BioASQ 8B challenges, transferring information brought improvement for factoid questions. We believe that further improvements can be achieved by implementing a more sophisticating information sharing between the two tasks. Analyzing the characteristics of each dataset used, can help us understand why transfer learning improves/degrades the performance for each question type.
So far we have only considered using domain-adaptive pretrained models
(BioBERT-based). Recent work on pretraining showed that task-adaptive
pretraining brings additional improvement for low-resource tasks [