Question Entailment in Consumer Health Using a Query Formulation Approach

Abstract

The need for online assistance regarding healthcare has grown significantly; a deficiency which has become readily apparent after the advent of the SARS-COV-2/COVID-19 pandemic. A widespread, trusted means of dispersing the latest medical knowledge could have provided tremendous benefit from a public health standpoint and curtailed the spread of a disease which has claimed lives of millions. Question Answering (QA) systems are well-suited to provide this assistance for medical professionals and the public at large, especially considering the increased adoption of virtual digital assistants such as Samsung's Bixby and Google Assistant in recent years. The overall performance of QA systems can be improved by a variety of methods, including entailment-based methods. In this paper, we propose a Query-Based Framework for Recognizing Question Entailment (QBF-RQE), which leverages a query formulation method to identify whether two questions are in an entailment relationship - with a specific emphasis on Consumer Health Questions (CHQs). Our approach also incorporates type and focus features of CHQs to determine the entailment relationship. We evaluate our approach with the MEDIQA 2019 shared task organized at the ACL-BioNLP workshop. Our method gives 83.48%, while the best-performing

Keywords

• Approach

1. Introduction

The purpose of the entailment recognition (or recognition) task is to classify the entailment relationship between a text pair (usually two separate sentences), which are known as the premise and the hypothesis. The entailment relationship are classified as: entailment (the hypothesis having a similar meaning as the premise), neutral (hypothesis having similar lexical items but has a diferent meaning than the premise), and contradiction (hypothesis having contradicting meaning versus the premise) relation [ 1, 2 ]. Within entailment recognition, there exists Recognizing Question Entailment (RQE) where both the premise and the hypothesis are question sentences. Harabagiu and Hickl showed improving performance for RQE also improves the QA system in the general domain [ 3 ]. Furthermore, Demner-Fushman et al. showed applicability to the CHQ domain as well by augmenting their Consumer Health Information Question Answering (CHiQA) system with a specific module for RQE [ 4 ].

According to Abacha and Demner-Fushman, the definition of entailment in QA is as follows: “a question A entails a question B if every answer to B is also a complete or partial answer to A” [ 5 ]. The primary goal of RQE is to ensure the answers of the premise and the answers of the hypothesis align with the entailment relationship, per the definition of entailment. According to Ben Abacha and Demner-Fushman, achieving this goal in QA requires multiple, diferent approaches [ 6 ].

We propose a new framework – QBF-RQE – which recognizes entailment relations based on the query formulation approach. Our framework leverages insights from Abacha and DemnerFushman’s CHiQA model, which uses type and focus information to form a query to retrieve answers using multiple AI models [ 5 ]. Thus, if the hypothesis has the same focus and/or type as the premise, the retrieved answers to a premise can be a partial or full answer to the answers of the hypothesis. We can state the hypothesis is an entailment of the premise.

While AI models would be ideally trained with premise and hypothesis pairs to achieve high performance, there is a relative lack of suitable, generally-available datasets for CHQA. To directly address this aforementioned lack of available datasets, we attempted several diferent approaches to augment the oficial MEDIQA 2019 training set, via several merging-based methodologies: 1) a module trained with a premise and hypothesis pairs 2) a module trained with question (Entailment Recognition Module: ER Module) and type pairs (Type Recognition Module: TR Module) 3) a module trained with question and focus pairs (Focus Recognition Module: FR Module), as shown in Figure 1.

2. Datasets

In this section, we describe datasets used to train and test our pipeline modules. The overall performance of RQE in CHQ is measured with the MEDIQA 2019 RQE Challenge test set.

2.1. Entailment Datasets

This section describes the dataset used for the ER module. We use diferent combinations of MeQSum, the MEDIQA 2019 training set, and the MEDIQA 2019 NLI dataset for training. 1. MEDIQA2019 RQE Datasets1: This dataset consists of sets of text-hypothesis pairs (clinical questionquestion pairs) provided by Abacha et al., Ben Abacha and Demner-Fushman at NLM. The pairs are labeled either Entailment or Not-Entailment. In the 8,890-pair training set, 4,680 pairs were labeled Entailment and 3,963 pairs were Not-Entailment. In the 302-pair validation set, 129 pairs were labeled Entailment and 173 pairs were Not-Entailment. In the 230-pair test set, the pairs are evenly divided with 115 each. 2. MeQSum2: We leveraged the fact that answers from summarized CHQ should result in the same answers as the original CHQ to include MeQSum to our RQE task training set. The dataset [ 8 ], also provided by the NLM group, includes 1,000 pairs of CHQ and summarized CHQ. 3. MEDIQA2019 NLI Datasets3: While not consisting of pairs in question form, a few teams incorporated MEDIQA-NLI (MedNLI) [ 9 ] in the MEDIQA 2019 RQE task [ 10, 11 ]. The dataset includes clinical sentence pairs: Entailment (3,744 pairs), Neutral (3,744 pairs) and Contradiction (3,744 pairs). Each label has 465 pairs in the validation set, and 474 pairs in the test set. 1https://github.com/abachaa/MEDIQA2019/tree/master/MEDIQA_Task2_RQE 2https://github.com/abachaa/MeQSum 3https://physionet.org/content/mednli-bionlp19/1.0.1/

2.2. Type and Focus Datasets

This section describes the dataset used to train and test the TR module and FR module. For the TR module for RQE task, we use LiveQA, MedInfo and MedQuAD to train the model. For the TR task itself, we use the LiveQA training set, MedInfo and MedQuAD to train, and the LiveQA test set to measure the performance of each model to compare the performance with the baseline (Demner-Fushman et al.). The type names and their frequencies are shown in Table 6 in Appendix A.2. For the FR, we consider disease names as a focus of the CHQs to be consistent with the answer retrieval method of CHiQA. LiveQA, MedQuAD, and MedInfo all are in the CHQ domain and have focus entities labeled. However, the Named Entity Recognition (NER) task to identify disease names are already widely available, and for the purposes of this paper, we do not perform re-training for the NER task with CHQ datasets.

1. TREC-2017 LiveQA4: The TREC-2017 LiveQA: Medical Question Answering Task [ 7 ] organizer provides a dataset (LiveQA) that has 446 pairs in the training set and 104 pairs in the test set. 2. MedInfo5: The MedInfo [ 12 ] dataset is about medication CHQs. The dataset has CHQs, answers, focus, type, section title and URL of the information source.

3. MedQuAD6: MedQuAD [ 5 ] has 47,457 pairs of medical questions/answers created from NIH websites.

3. Methodology

We describe our model in this section. Our model has 3 modules for diferent tasks: ER, TR and FR. The detailed architecture of our model is shown in Figure 1. All the models are transformerbased models, and we use pretrained models publicly available in the Hugging Face repository. The parameters we used are listed in Appendix B.

4https://github.com/abachaa/LiveQA_MedicalTask_TREC2017 5https://github.com/abachaa/Medication_QA_MedInfo2019 6https://github.com/abachaa/MedQuAD

3.1. ER Module

For ER, we experiment with 4 diferent dataset combinations with 6 diferent models. Data With MedNLI, MedQuAD and MEDIQA2019, we create 4 combinations of sets: 1) MEDIQA 2019 training set, 2) MEDIQA 2019 training set + MedNLI, 3) MEDIQA 2019 training set + MeQSUM, 4) MEDIQA 2019 training set + MeQSUM + MedNLI.

Model 1. Bio-Clinical-BERT: Bio-Clinical-BERT [ 13 ] is domain-specific contextual word embedding model, which is initialized with BIOBERT model and trained on all MIMIC notes [ 14 ]. 2. BiomedNLP-PubMedBERT-base-uncased-abstract Gu et al. provide a BERT-based neural language model pretrained on the biomedical NLP benchmark. BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext and BiomedNLP-PubMedBERT-base-uncased -abstract are pretrained models which are available in the Hugging Face repository78. 3. BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed [ 16 ] is a pretrained ELECTRA model-based, biomedical domain-specific language model using discriminators, showing great performance in MedNLI [ 9 ] (Language inference task), i2b2-2010 [ 17 ] (NER and relation extraction task), ShARe/CLEFE [ 18 ] (NER task) and ClinicalSTS [ 19 ] (Sentence Similarity task). 4. BioMed-RoBERTa-base BioMed-RoBERTa-base [ 20 ] is a language model based on the RoBERTa-base [ 21 ] model, fine-tuned with 2.68 million scientific papers from the Semantic Scholar corpus. Both full-text of papers and abstracts were used to train.

3.2. TR Module

Data We union labels of the LiveQA (26 types), MedInfo (17 types) and MedQuAD (16 types in Disease-related questions, 20 types in the drug category), resulting in 38 labels. For similar labels, we prioritized matching with LiveQA labels. Specific details regarding the label union methodology/procedure as well as the labels after union are shown in the bottom row of Table 6 in Appendix A.1.

Model

We use the same models in the TR module as those in the ER Module.

Entailment Score We measure the score as either 1 or 0 (Consistent or Not-Consistent), based on overlapping type labels. If there is any overlap between the type of text and the type of hypothesis, then we consider it Consistent. If there are no overlaps, then it is Not-Consistent.

3.3. FR Module

NER tasks to identify disease names are popular research area and publicly-available datasets and models are easily accessible. For this paper, we used 2 of the state-of-the-art models for the task and selected the one that gives higher accuracy based on the validation set. One is biobert-diseases-ner [ 22 ], which is a BERT-based model trained on NCBI-disease. NER-diseasencbi-bionlp-bc5cdr-PubMed [ 23 ] is a RoBERTa-based model [ 21 ], trained on NCBI-disease and BC5CDR datasets.

7https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract 8https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext Entailment Score We test 2 diferent methods to measure the score: 1) Exact-Match, and 2) Similarity-Based Match. Exact-Match occurs when there is overlap in disease names; which we then classify as entailment. If there is no overlap, then it is not entailment. Similarity-Based Match is utilized to address minor diferences/typos in the disease names. We measure the similarity score between each focus in the premise and the hypothesis. If the similarity scores of focus in the hypothesis and premise score is above a threshold, then we consider the pairs to be in an entailment relationship. The similarity score is measured with the S-BioBert-snlimultinli-stsb sentence similarity model [ 24 ] and the spaCy sentence similarity model [ 25 ]. S-BioBert-snli-multinli-stsb model is BioBERT [ 26 ] finetuned with several language inference datasets: SNLI [ 27 ], MultiNLI [ 28 ] and STS-b [ 29 ]. The spaCy model measures similarity by measuring the distance between word vectors trained on a large English general domain. 3.4. Merge

4. Evaluation

To merge the results, we test a majority-voting and a weighted-voting system. In this section, we discuss the overall performance of the QBF-RQE, along with the performance of each individual module.

4.1. QBF-RQE Results

We measure the performance of the QBF-RQE by calculating the number of correctly-predicted labels over the total number of premise and hypothesis pairs. We then compare our results to the results of the MEDIQA 2019 challenge participants. In Table 1, we list the results of each module for the RQE task, along with the pipeline result with the majority-voting method, and the pipeline result with the weighted-voting methods. The table also includes the list of results from the top 3 best-performing teams at MEDIQA 2019. With the test set, we had 8.58% higher accuracy than the best-performing team.

4.2. Module Performance

We evaluate each module of the pipeline separately.

4.2.1. ER Module We evaluated the performance of 5 diferent models and 4 diferent combinations of datasets for the ER module. As shown in Table 2, augmenting training set with MedNLI, or MeQSum gives higher performance for all models.

Both the MEDIQA training set+MedNLI combination and the MEDIQA training set+MeQSum combination demonstrated a greater than 10% increase vs just the MEDIQA 2019 training set. This shows that augmenting the MEDIQA 2019 helps to improve RQE models. However, merging the MedNLI, MeQSUM and MEDIQA training sets together did not necessarily improve the performance. Combining all datasets gave the best score of 80.99% and the average score of QBF-RQE ER module alone QBF-RQE TR module alone QBF-RQE FR module alone QBF-RQE Merged with majority-voting system QBF-RQE Merged with a weighted-voting system (Entailment-Pair model) QBF-RQE Merged with a weighted-voting system (Type model) QBF-RQE Merged with a weighted-voting system (Focus model) QBF-RQE Entailment ∪ Type&Focus MEDIQA 2019 Participants: Method Description PANLP: Ensemble, transfer learning, re-ranking with BERT, MT-DNN (Zhu et al. [ 10 ]) Sieg: MT-DNN with data RQE+QQP+GARD (Bhaskar et al. [ 30 ]) IIT-KGP: The best model result for Test set - Sci-BERT+Hinge Loss (Sharma and Roychowdhury [ 31 ]) IIT-KGP: The best model result for Val Set - QSpider (Sharma and Roychowdhury [ 31 ])

Baseline - SVM (Ben Abacha et al. [ 32 ]) 79.13%, which is higher than the MEDIQA + MedNLI combination, but lower than the MEDIQA + MeQSUM combination. We can therefore conclude that MedNLI may increase performance with a training set which is relatively small/limited, but if there is a training set that has closer characteristics to the test set, merging with MedNLI may not be advantageous.

For the test accuracy on Table 2, we selected the dataset combination which gave the best accuracy to the validation set (MEDIQA 2019 + MeQSum) and added the MEDIQA 2019 validation set to the training set to train and tested on the MEDIQA 2019 test set. 4.2.2. TR Module Demner-Fushman et al. thoroughly investigated the individual TR and FR models using Recall, Precision and F1 score with LiveQA test set. They used combinations of SVM and rule-based methods (regular expressions) and deep learning methods to extract the Type from CHQs. We consider this method as a baseline and compare it with our models. As shown in Table 3, simply merging LiveQA, MedInfo and MedQuAD showed improved performance.

We use the same models to test the ER purpose, to pick the best performing model and plug it into the pipeline, BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext showed the best performance on both the test and validation set. Results are shown in Table 4. 4.2.3. FR Module ner-disease-ncbi-bionlp-bc5cdr-PubMed shows slightly higher performance than biobert-diseasener model for the RQE task. Therefore, QBF-RQE results listed in the Section 1, ner-disease-ncbiModel with Train-set, MedNLI, MeQSum Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-full text BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt Biomed-RoBERTa-base Model with Train-set, MedNLI Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt Biomed-RoBERTa-base Model with Train-set, MeQSum Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt Biomed-RoBERTa-base Model with Train-set Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt Biomed-RoBERTa-base Model with Train-set, MeQSum, Validation set Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt

Biomed-RoBERTa-base bionlp-bc5cdr-PubMed model was used for the FR module, with the entailment score calculated with the Similarity-Based Match method. Exact accuracy is listed on the Table 5.

4.3. Limitations and Future work

While the performance of the QBF-RQE is generally improved by combining multiple modules, it is important to note that if the accuracy of a single module is significantly lower for a particular use case, the net efect can decrease overall performance. This characteristic is prominent on the test set. Individual module accuracy of the ER module, TR module and FR Data Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt Biomed-RoBERTa-base

SVM+Rule-Based+BiLSTM (Demner-Fushman et al. [ 4 ]) Bio-Clinical-BERT BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext BiomedNLP-PubMedBERT-base-uncased-abstract BioELECTRA-base-discriminator-PubMed BioELECTRA-base-discriminator-PubMed-PMC-lt

Biomed-RoBERTa-base modules are 57.39%, 82.17% and 51.3% respectively. The TR Module has the highest recognition and diference of accuracy between the TR module vs FR and ER modules is more than 20%. With the majority-voting system, we can see the accuracy reduced to 60% from 82.17%. With weighted-voting based on type, the accuracy is increased by 1.31%. Therefore, when using this approach, it is advantageous primarily when the individual modules have a balanced performance profile. Otherwise, simply employ the module with the best performance, particularly when the individual module is has an overwhelmingly superior performance profile. Second, when there is an bias in performance in one module (though not to an overwhelming degree), the weighted merge imparts improved performance. In the future, we hope to explore methods to further improve the performance of each module and hopefully investigate the diferent methods to merge the ER, FR and TR modules.

5. Conclusion

Ideally, the best scenario for RQE would be having one AI model and suitably training the dataset with appropriate premise and hypothesis pairs. But, the CHQ domain lacks such a dataset, which therefore limits the performance of the AI models. However, we showed significant improvement in performance in RQE in the CHQ domain by using the query formulation method inspired by the definition of Entailment in QA. In the future, we hope to investigate diferent ways to incorporate queries and study diferent methods of extracting queries (not limited to question focus and type characteristics) to build a more versatile RQE pipeline.

A. Drug Label Merge Details

A.1. Union Labels

To avoid redundant labels, we merged labels originating in diferent datasets which are functionally identical but possess minor variations in spelling. The bottom row in Table 6 contains the labels resulting from the union of LiveQA, MedInfo and MedQuAD. The labels in black originated in LiveQA. Blue labels are labels that originated in MedInfo that do not exist in LiveQA. Red labels only exist in MedQuAD.

We union the labels manually, setting the priority of labels as: #1 LiveQA, #2 MedInfo and #3 MedQuAD. We prioritized LiveQA due to its previous use in CHiQA research, better facilitating comparisons. Thus, we rename the labels to match the spelling as it exists with LiveQA, if possible. If the label does not exist in the LiveQA but only in MedInfo and MedQuAD, we arbitrarily modified the MedQuAD label to match with MedInfo. For example, side efects in MedInfo is modified to side-efect to match with LiveQA, while side efects, severe reaction in MedQuAD are modified to side-efect to match with LiveQA. In Table 6, the example is represented as “side efects( side efects, side efects,severe reaction )”.

Another manual task is to identify entailment relationships. For the purposes of the paper, if the two types are in an obvious entailment relationship, we unified the labels. The label special instructions, important warning, precautions, are renamed to considerations. In Table 6, the example is represented as “considerations(special instructions, important warning, precautions)”.

A.2. Union Datasets

After merging the 3 datasets with the method mentioned in Appendix A.1, the total number of questions and the type pairs are 48,577 and the total number of labels is 38. Due to the resource limitations and to prevent the dataset from overfitting on MedQuAD characteristics, we only select a max of 500 question and type pairs for each type. Among the 48,577, more than 97% of the dataset is from MedQuAD. With this limit, we have a total of 12,620 pairs. The detailed distribution is listed in Table 7.

B. Parameters

We use default parameters of hugging face for 6 models for ER and TR except warmup_steps, save_steps, batch size, epochs, weight_decay and learning_rate. We perform a grid search method to find an optimal parameter for each model: warmup_steps=100, save_steps = 500, batch size = 16, epochs = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, weight_decay={0.01, 0.1}, learning_rate = {5e-6, 1e-5, 5e-5, 1e-4}. weight_decay of 0.01 gave the best result for all tasks and models. learing_rate of 5e-5 gave the best results for the TR module. For the ER module, the best results were given when learning_rates lies between 1e-5 and 5e-5. Bio-Clinical-BERT, BiomedNLPPubMedBERT-base-uncased-abstract-fulltext, Biomed-RoBERTa-base results are model results with learning_rate of 1e-5. BiomedNLP-PubMedBERT-base-uncased-abstract, BioELECTRAbase-discriminator-PubMed and BioELECTRA-base-discriminator-PubMed-PMC-lt results are models trained with learning_rate of 5e-5. LiveQA MedInfo MedQuAD Union treatment, information, cause, diagnosis, susceptibility, interaction, person-organization, side-efect, efect, ingredient, prevention, symptom, tapering, usage, complication, contraindication, dosage, indication, prognosis, storage-disposal, comparison, inheritance, action, alternative, lifestyle-diet, other-question, genetic changes, resources information, dose, usage, side efects, indication, interaction, action, appearance, usage/time, stopping/tapering, ingredient, action/time, storage and disposal, comparison, contraindication, overdose, alternatives, usage/duration, time, brand names, combination, pronunciation, manufacturer, availability, long term consequences Diseases: information, research (or clinical trial), causes, treatment, prevention, diagnosis (exams and tests), prognosis, complications, symptoms, inheritance, susceptibility, genetic changes, frequency, considerations, contact a medical professional, support groups Drugs: information, interaction with medications, interaction with food, interaction with herbs and supplements, important warning, special instructions, brand names, how does it work, how efective is it, indication, contraindication, learn more, side efects, emergency or overdose, severe reaction, forget a dose, dietary, why get vaccinated, storage and disposal, usage, dose, precaution Medical Entities (ME): information treatment, information(other-question,learn more), cause(causes), diagnosis, susceptibility, interaction(interaction with food, interaction with herbs and supplements,interaction with medications), personorganization(contact a medical professional,support groups), sideefect( side efects ,side efects,severe reaction ), efect( how efective is it), ingredient, prevention, symptom(symptoms), tapering(stopping/tapering), usage, complication(complications), contraindication, dosage(dose,overdose,dose,forget a dose,emergency or overdose), indication, prognosis(long term consequences), storage-disposal(storage and disposal,storage and disposal), comparison, inheritance, action(how does it work), alternative, lifestyle-diet(dietary), genetic changes, resources(research), appearance, time(duration), comparison, alternatives, brand names, combination, pronunciation, manufacturer, availability, frequency, considerations(special instructions,important warning,precautions), why get vaccinated # of Pairs / # of Labels Train: (p) 446 / (L) 23 Test: (P) 104 / (L) 26 (P) 674 / (L) 25 (P) 47,457 / (L-disease) 16, (L-drug) 20, (L-ME) 1 Train: (P) 48,577 / (L) 38 Test: (P) 104 / (L) 26 information symptom treatment consideration cause dosage prognosis diagnosis organization brand names inheritance side_efect usage indication prevention storage_disposal complication frequency lifestyle_diet genetic changes susceptibility resources interaction action efect stages time tapering appearance contraindication ingredient why get vaccinated comparison alternative pronounce name combination manufacturer availability

References

[1] T. Neeraj , Semantic entailment, trishalaneeraj.github.io ( 2020 ). URL: https://trishalaneeraj. github.io/2017-12-22/semantic-entailment.

[2] A. Paramasivam , S. J. Nirmala , A survey on textual entailment based question answering , Journal of King Saud University - Computer and Information Sciences ( 2021 ). URL: https: //www.sciencedirect.com/science/article/pii/S1319157821003311. doi:https://doi.org/ 10.1016/j.jksuci. 2021 . 11 .017.

[3] S. Harabagiu , A. Hickl , Methods for using textual entailment in open-domain question answering , in: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Sydney, Australia, 2006 , pp. 905 - 912 . URL: https://aclanthology.org/P06-1114. doi: 10 .3115/1220175.1220289.

[4] D. Demner-Fushman , Y. Mrabet , A. Ben Abacha , Consumer health information and question answering: helping consumers find answers to their health-related information needs , Journal of the American Medical Informatics Association 27 ( 2019 ) 194 - 201 . doi: 10 . 1093/jamia/ocz152.

[5] A. B. Abacha , D. Demner-Fushman , A question-entailment approach to question answering , CoRR abs/ 1901 .08079 ( 2019 ). URL: http://arxiv.org/abs/ 1901 .08079. arXiv: 1901 .08079.

[6] A. Ben Abacha , D. Demner-Fushman, Recognizing question entailment for medical question answering , in: AMIA 2016 , American Medical Informatics Association Annual Symposium, Chicago, IL, USA, November 12 - 16 , 2016 , 2016 . URL: https://www.ncbi.nlm. nih.gov/pmc/articles/PMC5333286/.

[7] A. B. Abacha , E. Agichtein, Y. Pinter , D. Demner-Fushman, Overview of the medical question answering task at trec 2017 liveqa , in: TREC, 2017 .

[8] A. Ben Abacha , D. Demner-Fushman , On the summarization of consumer health questions, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Florence, Italy, 2019 , pp. 2228 - 2234 . URL: https://aclanthology.org/P19-1215. doi: 10 .18653/v1/ P19 - 1215.

[9] A. Romanov , C. Shivade , Lessons from natural language inference in the clinical domain , CoRR abs/ 1808 .06752 ( 2018 ). URL: http://arxiv.org/abs/ 1808 .06752. arXiv: 1808 .06752.

[10] W. Zhu , X. Zhou , K. Wang , X. Luo , X. Li , Y. Ni , G. Xie, PANLP at MEDIQA 2019: Pretrained language models, transfer learning and knowledge distillation , in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019 , pp. 380 - 388 . URL: https://aclanthology.org/W19-5040. doi: 10 .18653/ v1/ W19 - 5040.

[11] H. Pugaliya , K. Saxena , S. Garg , S. Shalini , P. Gupta , E. Nyberg, T. Mitamura, Pentagon at MEDIQA 2019: Multi-task learning for filtering and re-ranking answers using language inference and question entailment , CoRR abs/ 1907 .01643 ( 2019 ). URL: http://arxiv.org/abs/ 1907 .01643. arXiv: 1907 .01643.

[12] A. Ben Abacha , Y. Mrabet , M. Sharp , T. Goodwin , S. E. Shooshan , D. Demner-Fushman, Bridging the gap between consumers' medication questions and trusted answers , in: MEDINFO 2019 , 2019 .

[13] E. Alsentzer , J. R. Murphy , W. Boag , W. Weng , D. Jin , T. Naumann , M. B. A. McDermott , Publicly available clinical BERT embeddings , CoRR abs/ 1904 .03323 ( 2019 ). URL: http: //arxiv.org/abs/ 1904 .03323. arXiv: 1904 .03323.

[14] A. Johnson , T. Pollard, L. Shen , L.-w. Lehman, M. Feng , M. Ghassemi , B. Moody, P. Szolovits, L. Celi , R. Mark , Mimic-iii, a freely accessible critical care database , Scientific Data 3 ( 2016 ) 160035 . doi: 10 .1038/sdata. 2016 . 35 .

[15] Y. Gu , R. Tinn , H. Cheng, M. Lucas, N. Usuyama , X. Liu , T. Naumann , J. Gao , H. Poon , Domain-specific language model pretraining for biomedical natural language processing , 2020 . arXiv:arXiv: 2007 .15779.

[16] K. r. Kanakarajan , B. Kundumani , M. Sankarasubbu, BioELECTRA:pretrained biomedical text encoder using discriminators , in: Proceedings of the 20th Workshop on Biomedical Language Processing , Association for Computational Linguistics, Online, 2021 , pp. 143 - 154 . URL: https://aclanthology.org/ 2021 .bionlp- 1 .16. doi: 10 .18653/v1/ 2021 .bionlp- 1 . 16 .

[17] Uzuner , B. R. South , S. Shen , S. L. DuVall , 2010 i2b2/ VA challenge on concepts, assertions, and relations in clinical text , Journal of the American Medical Informatics Association 18 ( 2011 ) 552 - 556 . URL: https://doi.org/10.1136/amiajnl-2011-000203. doi: 10 . 1136/amiajnl- 2011 - 000203 . arXiv:https://academic.oup.com/jamia/articlepdf/18/5/552/33015279/18-5-552.pdf.

[18] L. Kelly , L. Goeuriot , H. Suominen , T. Schreck , G. Leroy , D. L. Mowery , S. Velupillai , W. W. Chapman , D. Martínez , G. Zuccon, J. Palotti , Overview of the share/clef ehealth evaluation lab 2013 , in: CLEF, 2013 .

[19] Y. Wang , S. Fu , F. Shen , S. Henry , O. Uzuner , H. Liu, The 2019 n2c2/ohnlp track on clinical semantic textual similarity: Overview , JMIR Med Inform 8 ( 2020 ) e23375 . URL: http://medinform.jmir.org/ 2020 /11/e23375/. doi: 10 .2196/23375.

[20] S. Gururangan , A. Marasović , S. Swayamdipta , K. Lo , I. Beltagy , D. Downey , N. A. Smith, Don't stop pretraining: Adapt language models to domains and tasks , in: Proceedings of ACL , 2020 .

[21] Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , O. Levy , M. Lewis , L. Zettlemoyer , V. Stoyanov , Roberta: A robustly optimized BERT pretraining approach , CoRR abs/ 1907 .11692 ( 2019 ). URL: http://arxiv.org/abs/ 1907 .11692. arXiv: 1907 .11692.

[22] Á . A. Casero , Named entity recognition and normalization in biomedical literature: a practical case in sars-cov-2 literature, 2021 . URL: https://oa.upm.es/67933/.

[23] X. Zhang, raynardj/ner-disease-ncbi-bionlp-bc5cdr- pubmed , 2021 . URL: https: //huggingface.co/raynardj/ner -disease-ncbi-bionlp-bc5cdr-pubmed.

[24] P. Deka , A. Jurek-Loughrey , Unsupervised keyword combination query generation from online health related content for evidence-based fact checking , in: The 23rd International Conference on Information Integration and Web Intelligence , 2021 , pp. 267 - 277 .

[25] M. Honnibal , I. Montani , spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing, 2017 . To appear.

[26] J. Lee , W. Yoon , S. Kim , D. Kim , S. Kim , C. H. So , J. Kang , Biobert: a pre-trained biomedical language representation model for biomedical text mining , CoRR abs/ 1901 .08746 ( 2019 ). URL: http://arxiv.org/abs/ 1901 .08746. arXiv: 1901 .08746.

[27] S. R. Bowman , G. Angeli, C. Potts , C. D. Manning , A large annotated corpus for learning natural language inference , in: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Lisbon, Portugal, 2015 , pp. 632 - 642 . URL: https://aclanthology.org/D15-1075. doi: 10 .18653/v1/ D15 - 1075.

[28] A. Williams , N. Nangia , S. R. Bowman , A broad-coverage challenge corpus for sentence understanding through inference , CoRR abs/1704 .05426 ( 2017 ). URL: http://arxiv.org/abs/ 1704.05426. arXiv: 1704 . 05426 .

[29] D. Cer , M. Diab , E. Agirre , I. Lopez-Gazpio , L. Specia, SemEval -2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation , in: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , Association for Computational Linguistics , Vancouver, Canada, 2017 , pp. 1 - 14 . URL: https://aclanthology. org/S17-2001. doi: 10 .18653/v1/ S17 - 2001.

[30] S. A. Bhaskar , R. Rungta , J. Route , E. Nyberg, T. Mitamura, Sieg at MEDIQA 2019: Multitask neural ensemble for biomedical inference and entailment , in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019 , pp. 462 - 470 . URL: https://aclanthology.org/W19-5049. doi: 10 .18653/ v1/ W19 - 5049.

[31] P. Sharma , S. Roychowdhury, IIT-KGP at MEDIQA 2019: Recognizing question entailment using sci-BERT stacked with a gradient boosting classifier , in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019 , pp. 471 - 477 . URL: https://aclanthology.org/W19-5050. doi: 10 .18653/v1/ W19 - 5050.

[32] A. Ben Abacha , C. Shivade , D. Demner-Fushman, Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering , in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019 , pp. 370 - 379 . URL: https://aclanthology.org/W19-5039. doi: 10 .18653/ v1/ W19 - 5039.