The first weeks of the COVID-19 crisis brought together several researchers from a wide range of domains, who combined their efforts in fighting the pandemic. At the same time, a significant issue in biomedical text mining was brought to the surface; while machine learning methods keep improving, boosting the performance of supervised models in the biomedical natural language processing field (biomedical NLP or BioNLP), the domain topics change rapidly and so do the related textual resources (scientific publications, reports, clinical trials). Thus, while gold standard, annotated datasets provide a solid basis for training, improving and evaluating new methods, they cannot account for emerging topics, new entities and terminology.
Indeed, navigating existing and upcoming literature, on a variety of COVID-related topics was identified as a critical task early on. The CORD-19 dataset and the kaggle CORD-19 challenge, reflected this need and indicated the path to addressing it. The CORD-19 dataset is an ongoing initiative (further described in Section 4.2) to collect resources that could be informative and helpful in coronavirus-related research. The kaggle challenge(s) built upon the early versions of the dataset, and invited research teams to submit systems that would address a set of key-questions spanning across domains and ranging from very information specific ones ('What do we know about vaccines and therapeutics?') to rather generic ones ('What has been published about information sharing and inter-sectoral collaboration?'). At the time no existing resources could account for COVID-specific annotations in text, calling for either unsupervised approaches or models trained on other domains. Deep neural architectures such as BERT-based models have shown great potential in information retrieval (IR) and question-answering (QA), rendering them strong vanilla models. Since COVID-19 related concepts were already incorporated in large knowledge bases such as SNOMED-CT, MeSH and UMLS, we wanted to explore the potential of using such knowledge sources in a post-processing manner in order to enhance such pre-trained models.
Since this is a preliminary study, we focused on different ways to enhance BERT-based embeddings with knowledge extracted from SNOMED-CT. BERT (Bidirectional Encoder Representations from Transformers) is a family of high performance pre-trained language models which produce state-of-the-art results in a wide variety of NLP tasks [1]. BERT's key technical innovation is applying the bidirectional training of Transformers [2] to language modelling. By using multiple attention mechanisms (multi-head attention), the model is able to capture a broader range of relationships between words than would be possible with a single attention mechanism. Moreover, BERT stacks multiple layers of attention, each of which operates on the output of the layer that came before. Through this repeated composition of word embeddings, BERT is able to form very rich representations as it gets to the deepest layers of the model. The choice of knowledge source was also dictated by performance and wide coverage of concepts; SNOMED-CT1 is the most comprehensive clinical healthcare terminology, consisting of more than 350,000 concepts and covering clinical findings, symptoms, diagnoses, procedures, body structures, organisms and other etiologies, substances, pharmaceuticals, devices and specimens among others.
For evaluation we use a subset of the OHSUMED dataset (see Section 4.1.1) to assess the improvement out method can achieve for document retrieval compared to different BERT models, used as baselines. Specifically, we are interested in comparing the potential for improvement across BERT-based models trained on different domains (generic, clinical, biomedical and combinations). To this purpose, given a query we retrieve the initial set of relevant documents calculating similarity of the BERT-based representations between query and document. We then re-rank the document set using SNOMED-CT to define a new concept co-occurrence based metric. We show that SNOMED-based filtering and re-ranking can consistently boost performance across different BERT baseline models in the IR-OHSUMED task. We show that the performance improvement is consistent across models but higher for generic domain models. Additionally, we show that we can obtain results that compare against and even outperform other semantic enhancement approaches. We provide a detailed analysis of the results and discuss how this preliminary study can set the basis for the development of further unsupervised methods, incorporating semantic knowledge in pre-trained embeddings using semantic knowledge graphs. Additionally, we demonstrate how this paradigm can be applied to answer the kaggle CORD-19 challenge questions, and present our modification to maintain robustness on large texts via summarisation.
While there is a range of work that inspired and relates to the work presented in this paper, the main line line of research concerns the use of external knowledge sources (ontologies, knowledge graphs or knowledge-bases) in order to semantically enhance a natural language processing (NLP) model, in a pre-processing, feature-extraction, joint learning or a post-processing fashion. We position our work in the post-processing approaches based on that classification, but we present below an overview of the core approaches for each category, with a focus on the biomedical/clinical domain.
Early on, Zhang et al. [3] proposed a method for semantic relatedness (SR) calculation between terms, showing how we can combine information from Wikipedia and WordNet in an enhanced graph that can then be traversed to obtain a relatedness score. They then showed that the SR extracted form these graphs can improved performance in named entity disambiguation tasks. More recent work, has focused in the potential of using an external knowledge source (ontologies, graphs or knowledge-bases) to identify key concepts in text and then link textual information from different documents [4].
In many downstream tasks, and especially textual classification, significant performance boosts can be obtained by using external knowledge sources to complement the textual representations and provide more informative features [5]. In this approach the extracted features are used with an SVM to obtain the classification output. While such feature engineering was the standard method to infuse external knowledge to supervised models in traditional ML, these are static features that are integrated in a uniform manner for all instances. Deep neural networks (DNN) can better address this limitation; there have been recent attempts to directly exploit external knowledge sources during the training of DNN models, either using "knowledge focused" attention mechanisms that use the external knowledge to obtain better representations of concepts in text [6,7], or by retrofitting information from the knowledge graph to pre-trained language models [8] in a post-processing fashion.
Focusing on the IR task, several publications use external ontologies and knowledge sources such as MESH in order to improve IR performance via semantic query expansion [9,10,11,12]. Agosti et al. [13] considers the relation between text and queries and aims to reduce the semantic gap between queries and documents, by incorporating polysemy and synonymy information during the training of neural networks.
Another strand of work related to our paper concerns the unsupervised or semi-supervised IR in the biomedical domain, as well as IR approaches on the main dataset we are experiment- ing with, namely OHSUMED. More specifically, Liu et al. [14], use UMLS to identify word relations and use this information to retrofit pre-trained word embeddings, enforcing the representations of related words to be closer together. Rais et al., compare different strategies of enriching document representations, using concepts from external knowledge sources in combinations with WSD approaches [15]. They specifically employ UMLS for the concept extraction (using a similar MetaMap enrichment to the one we describe in Section 3.4.1) and show that the replacement of terms with their respective concepts can boost IR performance on OHSUMED when used in combination with WSD approaches. We show in this work that when used in combination with pretrained DNN models (which are more robust in terms of contextualisation of terms) the conceptualisation of terms can boost performance even without the use of WSD approaches and outperform the aforementioned approach. Oh et al. [16] proposes CBEEM which exploits external dataset collections in building a feedback model to improve relevance ranking for biomedical IR. Note that instead of using a hierarchical resource in this case, Oh et al. use large external document collections to cluster documents and thus better contextualise query relevance by incorporating an additional term in the traditional feedback model related to the collection relevance estimation.
Our system assumes a large document resource that is checked against a specific user query. We use the process described in Figure 1 to obtain relevant documents that could answer the user's query. Specifically, we apply the following:
1. Document formatting: Summarisation of full-text documents to obtain a representative set of sentences 2. Semantic text enrichment based on SNOMED-CT: Identifying and aligning concepts mentioned in documents and/or queries with their respective descriptions and neighbour nodes in the knowledge-base.
We use vectorized text representations to compare documents and queries. We employ BERT models for this purpose (see Section 3.3).
We distinguish two main categories of papers when searching the scientific literature, for which we apply different pre-processing before obtaining the document vectors.
1. Abstract only: Refers to articles for which only the title and abstract (and perhaps some metadata) is available to be processed, typically due to licensing. For these papers the document to be vectorised, is the concatenation of the title and abstract. 2. Full-text documents: Typically open publications where the title, the abstract and the main body and full metadata of the paper is available to any reader and/or text mining system. If the abstract is invalid (e.g., <= 3 sentences) using the full text representation can negatively influence the performance of similarity ranking against the query, due to the length discrepancy. To account for such cases (frequent in the early version of CORD-19 dataset), we opt for an extractive summarisation method, described below. The method was applied on the CORD-19 dataset only, since OHSUMED consists exclusively of abstracts.
To reduce the gap between the abstract only and full-text documents, and be able to apply the same IR models, we opted to extracting only the most representative sentences for documents without a clearly specified abstract, in other words, we used single-document extractive summarisation for scientific documents. We thus fine-tune BERTSUM [17], which has been demonstrated to achieve high performance in scientific summarisation tasks [18,19,20].
The BERTSUM model provided by the authors was initially trained on summaries for news articles, whose document characteristics differ significantly compared to those of scientific articles. One of the main underlying differences, concerns the language and vocabulary used, as well as the sentence structure (longer, more complicated sentences). Additionally, the summary size in the newswire domain is significantly smaller compared to the typical scientific summary. We thus construct a new summarisation dataset based on CORD-19 dataset and fine-tune BERTSUM to this specific task, to obtain a model that can better distinguish the important sentences in a long scientific article.
The training dataset was constructed exclusively from papers which have valid and clearly distinguished abstract and main body text as they are defined above. Sentence splitting was applied on both the abstract and the body text of the papers using StandfordNlp [21]. Additional denoising was applied to remove highly frequent abbreviations. After the tokenisation and preprocessing of the texts, each paper consists of 2 parts: πππ π‘πππ‘ = [ππ πππ‘ 1 , ππ πππ‘ 2 , ..., ππ πππ‘ π ] and ππππ¦ = [ππ πππ‘ 1 , ππ πππ‘ 2 , ..., ππ πππ‘ π ] where ππ πππ‘ π , ππ πππ‘ π is the π π‘β sentence of the abstract and body text respectively.
The training dataset was constructed considering that the abstract contains the most important information of the paper (inductive bias). We thus score the ππππ¦ sentences against the πππ π‘ππππ‘ sentences using ROUGE-L score to obtain the π sentences. For the specific dataset and based on the distribution of the abstract sentences we selected π = 3 , since in this value it appeared that there was a golden ratio between redundancy and noise. We assigned label 1 to sentences selected in the oracle summary and 0 otherwise, and fine-tuned the initial BERTSUM model on this dataset.
To retrieve documents relevant to a query, we employ bert-as-a-service [22] and calculate the BERT representation of the user query and each candidate document. We then rank the documents' relevance for each query by calculating the distance between two vector representations. We use cosine similarity for the distance estimation, so if we assume that π π is the document vector and π π is the query vector then the relevance score is calculated as:
We opted for BERT embeddings trained on different domains, with demonstrated high performance in downstream classification tasks. Specifically we chose the following: (1) the original BERT model [1], trained on Wikipedia and BookCorpus, hence fine-tuned for the generic domain, (2) BioBERT [23] trained on Pubmed papers, hence fine-tuned on the biomedical domain and (3) BlueBERT [24], trained in a multi-task setting on a combination of biomedical and clinical data, hence still fine-tuned in the biomedical domain, but accounting for a wider scope of text. We expect the latter to have broader coverage and thus better performance on the queries, and generally we expect BioBERT and BlueBERT to outperform BERT due to the expected overlap between the documents they was trained on and the scientific papers in the evaluation datasets. The base model was used across cases (12-layer, 768-hidden, 12-head). [25] is a collection of medical terms, and their synonyms, descriptions, etc., with an underlying description logic formal model. It contains clinical knowledge that can complement textual information, and help us process new documents. Its core components include concept hierarchy, descriptions, relations and reference sets, of which we focus on:
β’ Concept hierarchy: encoded clinical terms, organised in hierarchies. The hierarchical structure is particularly useful in the case of searching in a corpus of texts with multilevel information (high-level, general concepts vs more technical/specific information). One can move down the hierarchy in order to fetch specific results, or move up the hierarchy in the case of queries that do not match exactly any of the documents' terms.
β’ Descriptions: textual descriptions of concepts. These can be either synonyms or periphrastic definitions of the corresponding terms. Given that a term can appear with multiple surface forms, enhancing a concept with associated description (and the terms mentioned in in) we can map additional relevant text spans to a given concept.
For the purposes of our work, we make use of the concept hierarchy, where we collect the parents of medical concepts in the SNOMED CT hierarchy and the descriptions, where we incorporate alternative, equivalent ways of describing the same medical concept. One of the challenges in terms of transferring these rich SNOMED-CT concepts to raw text, is to be able to identify the relevant terms in text. For this purpose, we employ the MetaMap tool [26], which maps biomedical text to the UMLS metathesaurus. Upon identifying the text spans that correspond to UMLS concepts, we use a mapping between UMLS and SNOMED concepts in order to incorporate the SNOMED knowledge.
For this preliminary study we focus on the existence of SNOMED related concepts in the text as a re-ranking and filtering method for the vector based similarity ranking. More specifically, assume a given user query π π and an ordered list of documents π·, ranked by the cosine similarity scoring described in Eq 1. Upon obtaining the list, we identify a set of text spans πΆπ· π in each document π π β π· that correspond to SNOMED concepts, using the process described in Section 3.4.1. Similarly, we identify a set of SNOMED concepts πΆπ π corresponding to the query π π . For each identified SNOMED concept π such that π β πΆπ· π and/or π β πΆπ π , we navigate the SNOMED-CT hierarchy to identify the hyper-concepts (parents) π βπ¦π as well as the terms contained in the concept π description, π πππ π and then expand the πΆπ· π and πΆπ π sets with the respective π βπ¦π and π πππ π concepts. We then calculate the concept intersection between the two sets as specified in Eq 2 and attribute each document with a renewed score, calculated as:
ππππ’ππππ‘_π ππππ(π π , π π ) = πππππ£ππππ(π π , π π ) β πππππππ‘_π πππ‘ππ(π π , π π )
We then investigate the use of different thresholds on the value of the πππππππ‘_π πππ‘ππ to filter the initial set of documents π·. For the experiments presented in the following sections, when the πππππππ‘_π πππ‘ππ falls below the threshold the value is set set to 1 in Eq 3. We show that this simplified filtering technique can consistently boost performance on different BERT models, and we identify the optimal threshold on the OHSUMED dataset, based on the NDCG metric (Eq. 4).
The OHSUMED test collection is a subset of the MEDLINE database. We consider a commonly used subset [27], consisting of the first 20,000 documents from the 50,216 medical abstracts 1 we present the a more detailed breakdown of query-document sets. As we discuss in the results, the low number of related documents for some queries, has a negative impact on performance which we aim to address in the future work.
We base our evaluation on the normalised discounted cumulative gain (NDCG) metric, used to assess the model's ranking of relevant papers pertaining to a set of queries Q. It is defined for position π β {0, 1, ..., π } as described in Eq. 4:
where πΌ π·πΆπΊ denotes the ideal and highest possible π·πΆπΊ and πππ π (π) refers to the relevance of the π π‘ β result ranked according to query q.
While NDCG is our main metric, we also discuss the results for Precision@N and Recall@N, to provide better insights on the performance of the proposed methods (see Section 4.1.3). The Precision and Recall functions are presented below, assuming an ordered set of relevant documents π πππ· = {πππ 1 , ..., πππ π } and an ordered set of retrieved documents π ππ‘π· = {πππ‘ 1 , ..., πππ‘ π }
In the experiments, we first identified the optimal threshold value for filtering, which is consistently π‘βπππ βπππ = 5 for all models. We can then see, that for the application of filtering with this threshold, NDCG performance improves consistently across models. As explained in Section 4.1.1, the OHSUMED-91 dataset, contains only a subset of documents of the original dataset, hence some queries contain only a single relevant document in the ground truth set, rendering the IR problem more demanding. However, as we show in Figure 2, once we constrain the dataset to contain only queries with a higher number of relevant documents (πΊπ π·), performance improves further for all models. Thus we reach NDCG@100 performance approximating to 0.25 which is comparable with other unsupervised IR methods on the same dataset [16]. BioBERT and BlueBERT models outperform BERT in all versions, with BlueBERT reaching the best performance for most dataset subsets, confirming our initial hypothesis. If we cast a closer glance at the increase over the plain model version compared to the SNOMED-CT filtering, we can see that we get a greater performance increase for the less domain specific models (see Figure 3a). Moreover, we can see that this improvement boost is consistent for all filtering thresholds (Figure 3b, and dataset subsets (Figure 4). This observation underlines the potential of semantic enhancement across domains. More importantly, it demonstrates that a variation of semantic filtering based on a knowledge graph, can successfully be applied to adapt out-of-domain models to a new domain, thus motivating our future research into transfer learning via knowledge graphs.
Regarding the impact of the filtering threshold, we can see that the initial performance of all BERT models improves consistently with the application of filtering with increasing thresholds, until we reach the optimal threshold value.The improvement holds across different dataset subsets (GTD) even if we apply the more relaxed filtering option (π‘βπππ βπππ = 1) which demands that there is at least one concept co-occurence between the document and the query, for the document to be considered valid. Moreover, performance increases consistently until π‘βπππ βπππ = 5 for all models, with a sole exception to the trend for BlueBERT π‘βπππ βπππ = 2. Performance drops for larger thresholds but we have to note that the optimal threshold is related to the length of the queries and documents and needs to be studied separately for different dataset setups. Additionally, the low number of related documents per query significantly impacts the Recall@N values shown in Figure 6 which start from low values for all model and threshold variations. However, we can see that still, comparing the baseline BERT models when there is no concept-based filtering and the case where we use the previously identified optimal threshold for filtering (π‘βπππ βπππ = 5), we get a significant improvement for recall, with emphasis on π < 200 (see 6a). Similar observations were seen in precision, presented in table 2, where we should note that for the BlueBERT model we obtain better performance than the one reported in [15] for P@5 and comparable for P@10. We expect that we would see considerably higher values for datasets with a larger number of related documents per query,
To demonstrate the direct applicability of the described methodology on other datasets without further fine-tuning, we also experiment with an early version of the CORD-19 dataset, as used for the respective kaggle challenge2 . The dataset is a snapshot of the 10th of April 2020, and contains 51045 documents accompanied with their full text and metadata information.
Where available, the abstract is provided as a separate metadata element, however approx. 27K articles had invalid abstracts, based on the criterion described in Section 3.2. Documents are selected based on their expected relevance to the COVID-19 pandemic, covering a wide range of biomedical, clinical and socioeconomic aspects, spanning the period from 1985 to 2020. Since our initial motivation was the kaggle CORD-19 challenge and associated queries, we present below representative examples of query outputs showing also the difference between the highlighted queries. We show the process and output for identifying the relevant passages for one of the main kaggle questions in Figure 7. We added the coronavirus concept to the query when it was not explicitly stated to further adapt to the domain. We show the identified concepts and sample SNOMED-CT mappings (Figure 7), as well as the start of the produced BERTSUM summaries. We provide an interactive query UI on kaggle3 .
Motivated by the rapid evolution of covid-related publications and query topics, we explored options for improving unsupervised IR on emerging queries in the biomedical domain. This is preliminary work, exploring the use of SNOMED-CT to further filter the relevant documents, ranked by BERT model variations. We showed that even with a simple co-occurrence filtering method, we can significantly improve the initial results and achieve comparative performance to other unsupervised work on the same dataset. Specifically, we show that for multi-document queries and using the BlueBERT model as a basis, the filtering method reaches 0.23 for the NDCG@100 metric. Additionally, we show that we can get meaningful gains across different metrics even for models trained on generic data. Indeed, BERT-based results filtered using SNOMED-CT surpass the performance of unfiltered BioBERT results.
The aforementioned outcomes provide solid basis for further experimentation into better exploitation of knowledge graphs and concept hierarchies as a means of boosting IR on new topics in an unsupervised manner. We intend to further establish our findings by applying the described approach to the full OHSUMED corpus, as well as other biomedical datasets (CLEF, TREC CDS, TREC CORD-19, etc). Moreover, we inted to experiment with neural network architectures other than BERT, such as XLNET and ELECTRA [28]. Upon completion, our future work will be focused on two main tracks. Firstly, explore in more detail the potential of the SNOMED-CT hierarchy. More specifically, in this work we only incorporated the description and first parent node of each identified concept, without further traversing the concept graph. We hypothesise that the position of a concept in the hierarchy as well as the neighbourhood size and type we consider for each concept would impact the size of intersecting concepts between queries and documents (hence impacting the threshold value), and would potentially allow us to identify further connections between missed documents. Additionally, the incorporation of different types of neighbours for each concept would allow us to come up with more elaborate re-ranking formulas, taking into account multiple variables to produce the final document score. The second line of future work, concerns the use of external knowledge sources such as SNOMED-CT not in order to obtain a re-ranking functionality, but as a means to achieve transfer learning and distant supervision to better adapt deep neural networks and pretrained embeddings to new domains.