The continuously increasing number of biomedical scholarly publications makes it challenging to construct document recommendation algorithms that can efficiently navigate through literature. Such algorithms would help researchers in finding similar, relevant, and related publications that align with their research interests. Natural Language Processing offers various alternatives to compare publications, ranging from entity recognition to document embeddings. In this paper, we present the results of a comparative analysis of vector-based approaches to assess document similarity in the RELISH corpus. We aim to determine the best approach that resembles relevance without the need for further training. Specifically, we employ five different techniques to generate vectors representing the text in the documents. These techniques employ a combination of various Natural Language Processing frameworks such as Word2Vec, Doc2Vec, dictionary-based Named Entity Recognition, and state-of-theart models based on BERT. To evaluate the document similarity obtained by these approaches, we utilize different evaluation metrics that account for relevance judgment, relevance search, and re-ranking of the relevance search. Our results demonstrate that the most promising approach is an in-house version of document embeddings, starting with word embeddings and using centroids to aggregate them by document.
Recommendation systems have shown to be a successful method to cope with information
overload and retrieval, making it easier to navigate the ever-expanding public information
available online. Such systems have become a key application for scientific publications and their
corresponding repositories [
All of the aforementioned systems are document-to-document recommendation systems; they
offer recommendations for documents that are similar to a particular, i.e., reference, document.
Due to the lack of document-to-document relevance or similarity training datasets, it is a common
approach to use document-to-topic relevance or similarity collections (e.g., those used in the Text
Retrieval Conference – TREC). To overcome this situation, the RElevant LIterature SearcH
(RELISH) consortium created a document-to-document relevant dataset [
Determining similarity between one document with respect to a reference one mainly boils
down to analyzing the similarities between their corresponding texts - a task that is often tackled
using Natural Language Processing (NLP). To achieve this, the textual data from these documents
needs to be transformed into a structured form to enable machine-based document-to-document
similarity approaches. One option is using vectors, i.e., representations of words that encode
words' meaning, making it easier to, for instance, finding close words based on their distance in
the vector space, which could also be similar in meaning or context. To generate these vectors,
various state-of-the-art NLP methods, such as Word2Vec [
Document comparison based on text-based vectors require a comparable measure of document similarity. Metrics such as the Dice coefficient, Jaccard coefficient, Overlap coefficient, Cosine similarity, Okapi Best Matching 25 (BM25), Term Frequency-Inverse Document Frequency (TF-IDF), and more sophisticated techniques that account for context, subject, and term-dependencies can be used to assess these similarities.
In this paper, we describe and compare five different techniques to generate vectors for every PubMed article of the RELISH corpus using NLP frameworks, namely: word2doc2vec, doc2vec, whatizit-dictionary, hybrid-doc2vec, and BERT-based approaches. As per the RELISH corpus, the relevance annotation for each article with respect to the others is categorized into three classes: relevant (definitely relevant), partial (partially relevant), and non-relevant. We employ this relevance categorization as our ground truth to evaluate the performance of our five different approaches. We made use of the Cosine similarity metric to calculate the document similarities and further analyze the document-to-document similarity using three evaluation approaches, namely: distribution-based analysis, normalized Discounted Cumulative Gain, and precision, to account for the categorization, ranking, and relevance of the similarity search.
In this study, we used the RElevance LIterature SearcH (RELISH) consortium [
Using the BioC API, we retrieved an XML file containing the PMID, title, and abstract for each unique entry in the RELISH JSON file. We also recorded the missing PMIDs whose retrieval failed or whose title/abstract was unavailable. In total, we retrieved about 163,189 XML files. This dataset was also transformed into a TSV file consisting of three columns, namely: PMID, title, and abstract, where the text in the title and abstract was preprocessed. Two different preprocessing pipelines were followed, depending on the vector-based approach used. The preprocessing pipeline for word2doc2vec, doc2vec, and the hybrid-doc2vec approaches included converting the text into lower case, removal of punctuations, removal of structural words (e.g., “BACKGROUND:,” “CONCLUSION:,” “METHODS:”), followed by tokenization.
For the BERT-based approach, only white spaces and newlines between the text were removed. The titles and abstracts for each document were combined as a single text input for all the approaches. In addition to this, another TSV file was created from the raw original RELISH JSON file consisting of three columns, namely: PMID1 (reference article), PMID2 (assessed article), and relevance (relevance between the two documents). For simplicity, the relevance was assessed by a score of 0, 1, and 2 for articles that were non-relevant, partially relevant, and definitely relevant, respectively. This TSV file was used for the evaluation tasks, and excluded all RELISH reference articles having a multiple relevance assessment due to being evaluated by more than one annotator.
In this study, we evaluate document-to-document similarity by generating embeddings using five different approaches, namely word2doc2vec, doc2vec, whatizit-dictionary, hybrid-doc2vec, and BERT-based approach. We assess the similarity using the cosine similarity metric. The input for the dictionary-based NER approach (whatizit-dictionary) was the RELISH XML files, whereas the input for all the other four approaches was the RELISH TSV file consisting of three columns (PMID | title | abstract).
In this approach, we made use of the Word2Vec framework [
2.3.2. Doc2vec
This approach utilized the Doc2Vec framework [
This approach utilizes a dictionary-based Named Entity Recognition method that employs the Whatizit tool. Whatizit is a text processing system based on MONQjfa, a deterministic and nondeterministic finite automata for Java [15]. For our study, we make use of a minimal dockerized version of Whatizit, focusing mainly on the automata part of the MONQjfa [16]. The input for Whatizit is a dictionary that recognizes entities in a text, and normalizes them against a controlled vocabulary. In order to annotate the RELISH XML files, we used Medical Subject Headings (MeSH) [17] as our controlled vocabulary. The annotated XML files were then used for generating embeddings in the form of Term Frequency-Inverse Document Frequency (TF-IDF) vectors, where we evaluated the relevance of each MeSH term to a particular RELISH article in the entire RELISH corpus.
This is a hybrid approach that explores the combination of a dictionary-based Named Entity Recognition using the Whatizit tool and the Doc2Vec framework. The main idea of this approach is to transform annotated XML files after the Whatizit processing into a plain text dataset, preprocess the text, and then use the Doc2Vec model. Annotated MeSH terms in the article’s title and abstract are replaced by their MeSH ID, and converted into plain text. This step is followed by applying a standard Doc2Vec process (similar to the Doc2Vec approach) to generate embeddings using the same set of hyperparameter combinations as shown in Table 1.
This approach explores and assesses document-to-document similarity using Bidirectional Encoder Representations (BERT) [11] - based embeddings. BERT from Transformers [18] is a transformer-based machine learning technique for NLP pre-training developed by Google. We used the Sentence-Transformers package [19] to run experiments with BERT models. In this approach, we used two state-of-the-art BERT models relevant to the biomedical domain: BioBERT [20] and SciBERT [21]. BioBERT is a pre-trained language representation model based on the BERT model, which was trained on a large corpus of biomedical text (PubMed abstracts and PMC full-text articles). We used two different versions of the BioBERT model by DMIS-Lab [22]: BioBERT-Base-cased-v1.1 [23] and BioBERT-Large-cased-v1.1 [24]. We obtained vectors of size 768 using the BioBERT-Base-cased-v1.1, and vectors of size 1024 using the BioBERT-Largecased-v1.1. SciBERT is another pre-trained language model based on BERT but trained on a large corpus of scientific text. This model was trained on 1.14M full-paper corpus from semanticscholar.org [25]. We used the cased version model for SciBERT: scibert_scivocab_cased [26]. Similar to BioBERT-Base, we obtained vectors of size 768 for this model.
Table 1 summarizes the set of hyperparameter combinations that were used to generate the embeddings using the three approaches: word2doc2vec, doc2vec, and hybrid-doc2vec. The parameters include the training algorithm, epochs, min_count, vector_size, and window_size. The epochs and min_count were kept constant with values of 15 and 5, respectively. The training algorithm parameter was varied based on the approach: skip-gram (sg: 1) or the continuous bag of words (cbow: 0) for word2doc2vec, and distributed memory (dm: 1) or the distributed bag of words (dbow: 0) for doc2vec and the hybrid-doc2vec approach. Three different values were used for the vector_size: 200, 300, and 400, as well as three different values for the window_size: 5, 6, and 7. skip-gram skip-gram 5, 6, 7 5, 6, 7
We calculated the cosine similarity for all the existing pairs of PMIDs per the original RELISH JSON file and created a four-column matrix (PMID1 | PMID2 | relevance | cosine similarity). We then used a distribution-based analysis to obtain the best hyperparameter configuration. This analysis aims to provide a visual aid to understand the cosine similarity tendency for a given model, as well as to understand how a given model behaves with respect to each relevance category (definitely relevant, partially relevant, and non-relevant).
Firstly, we build a counting table in the form of a four-column matrix consisting of a cosine similarity interval ranging from 0 to 1 in steps of 0.01, giving us a total of 101 intervals, count of 0’s, count 1’s, and count of 2’s. The purpose of this counting table is to represent how many comparisons of each relevance category are found in each discrete cosine interval. We believe that an optimal similarity model will tend to have non-relevant values (0’s) in the lowest values of cosine similarity intervals, while expecting the definitely relevant values (2’s) to be in the higher cosine similarity intervals. We visually studied this tendency for each set of hyperparameter combinations by plotting the counting table as a histogram distribution between the relevance counting vs. cosine intervals for each of the four approaches. We also included the possibility to normalize the distributions so that the cumulative sum of each category adds up to 1.
We further used this analysis to select the optimal model from our set of hyperparameter combinations for each approach by proposing a ROC One vs. All approach. The task at hand was viewed as a problem resembling a multi-class classification problem, where we attempted to categorize whether a cosine similarity value corresponds to a definitely relevant, partially relevant, or a non-relevant pair of documents. We proceeded by converting our problem to a simple binary classification problem by combining both the partially relevant (1’s) and definitely relevant (2’s) category into a single “relevant” category, thereby optimizing for a relevant vs. nonrelevant classification. We calculated the True Positive Rate (TPR) and False Positive Rate (FPR) using the formula stated by equations 1 and 2 from the typical classification metrics, consisting of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). These values were calculated from the counting table. For a given cosine similarity, called CIx, we calculated: • TP: sum of the number of relevant documents for cosine intervals greater than or equal to CIx. • FP: sum of the number of non-relevant documents for cosine intervals greater than or equal to CIx. • FN: sum of the number of relevant documents for cosine intervals smaller than CIx. • TN: sum of the number of non-relevant documents for cosine intervals smaller than CIx. (1) (2)
From the calculated TPR and FPR, we plotted the ROC curve as a representation of the TPR against the FPR and calculated the Area Under the Curve (AUC). The calculated AUC values act as a good measurement of the overall model performance; we used this metric to compare and choose which hyperparameter combination performed best in discriminating between relevant vs. non-relevant pairs of documents for our approaches. For each of our approaches, we ultimately only used the model with the optimal hyperparameters for further downstream evaluation approaches.
To evaluate our five approaches, we made use of the above-mentioned cosine similarity fourcolumn matrix for all the existing pairs of PMIDs (PMID1 | PMID2 | relevance | cosine similarity) along with their relevance scores as per the RELISH JSON file. We evaluated the similarity scores with respect to the RELISH relevance assessment for all approaches using normalized Discounted Cumulative Gain (nDCG@N) and precision@N.
This is an evaluation metric used to rank the recommendations in a document recommendation system based on the relevance. For each of our five approaches, we used the existing pairs of PMIDs with its relevance score (as per the original RELISH JSON) and cosine similarity score between those pairs (PMID1 | PMID2 | relevance | cosine similarity) as input to this evaluation algorithm. The nDCG scores were calculated based on two scores: Discounted Cumulative Gain (DCG) and the ideal Discounted Cumulative Gain (iDCG). In order to account for these two scores, this algorithm works by creating two intermediary matrices, DCG and iDCG. The DCG matrix sorts each PMID entry based on the cosine similarity (from highest to lowest), whereas the iDCG matrix sorts each PMID entry based on the relevance scores (2’s, 1’s, 0’s in that order). For every PMID, the DCG@N and the iDCG@N scores are then calculated using the DCG and the iDCG matrices based on the formula as stated by equations 3 and 4, respectively, where ‘N’ and ‘n’ stand for the number of documents for which we intend to calculate the nDCG score, and ‘i’ stands for the ith document in our document set. The ‘rel’ accounts for the relevance score between the pair of PMIDs (PMID1 and PMID2). (3) (4) (5)
In our case, the values of N we used were 5, 10, 15, 20, 25, and 50. Lastly, we calculated the nDCG@N scores for each PMID by dividing the DCG scores by the iDCG scores as stated by equation 5, and took the average nDCG scores for all the values of N for all five vector approaches.
This performance metric accounts for the fraction of relevant instances among the retrieved instances, in our case, amongst the N retrieved documents. We take as input the cosine similarity four-column matrix (PMID1 | PMID2 | relevance | cosine similarity) which includes the relevance scores that come from the original RELISH JSON as our ground truth. We start by sorting the cosine similarity scores for each reference PMID (highest to lowest) and take the top N similar documents, where N is the precision parameter that takes values of 5, 10, 15, 20, 25, and 50, and count the number of true positives. Amongst the top N similar documents, we consider a document (assessed PMID) to be a true positive if it has a relevance score of 2 (indicating definite relevance) with respect to the reference PMID in the relevance scores file. Equation 6 states the formula for calculating the precision@N. (6)
We repeat this step for all the documents-to-document pairs in the corpus and calculate the average precision scores for each N value for all five vector approaches.
In this study, we evaluated various approaches for measuring similarity between scientific articles using different vector representations. Table 2 displays the AUC scores obtained from the distribution-based analysis for the BERT-based models that we used. We observed that the scibert-scivocab-cased model outperformed the other two models, with an AUC score of 0.65. Based on this finding, we selected scibert as our optimal BERT-based model for the other two evaluation approaches.
We also obtained the optimal hyperparameters for the word2vec, doc2vec, and hybriddoc2vec approaches through the distribution-based analysis, as shown in Table 3. Tables 4 and 5 present the results of the two other evaluation approaches with different vector representations.
Our findings revealed that the word2doc2vec approach outperformed all the other approaches in terms of nDCG and precision scores, followed by the doc2vec approach and the hybrid-doc2vec approach. On the other hand, the traditional TF-IDF method using the whatizitdictionary approach was the least effective out of all.
We found that the use of only one controlled vocabulary, MeSH, may have contributed to the low scores observed in our evaluation. This could be due to the limited number of MeSH terms compared to the vocabulary that emerges from our dataset, or the inefficiency of replacing terms recognized by MeSH into normalized text. We will investigate this further. Moreover, the precision scores for all of our approaches were not particularly high, indicating that none of them naturally captures the relevance. The most promising approach in this regard was word2doc2vec.
Overall, our results suggest that there is no definite winning approach in this study based on the very close AUC scores using the distribution-based analysis and extremely low precision scores. All the evaluation metrics that were used are tailored to a different task at hand. AUC using the distribution-based analysis gives us an idea about the coverage of relevance judgment. In our study, it was tailored to two relevant categories, and converted into a binary classification problem; in a future run, we will turn it into a multi-class classification so we can take into account the three relevance classes (definitely relevant, partially relevant, non-relevant) rather than merging definitely with partially relevant (which is a common practice when working with topicto-document relevance).
Although the hybrid-doc2vec approach was promising, it did not perform well. Our expectation was that using NER before the embeddings would improve results; however, this was not the case. A possible reason is the even poorer performance from the dictionary-based approach that was also used in the hybrid-doc2vec approach. As for the direct use of similarity to assess relevance, our results suggest that this is not possible, at least not for the RELISH corpus. None of the approaches showed high values for precision (they are close to 50%, except for word2doc2vec on P@5 with 0.6574), and none were high enough for a search engine or recommendation system with respect to the nDCG except for the word2doc2vec approach which exhibited the highest nDCG score of 0.78 for nDCG@5.
Our initial experiments (our project is still a work in progress) aimed to generate vector representations for the particular corpus at hand, with the exception of the BERT-based approach which utilized pre-trained models. Different hyperparameter configurations were assessed using the AUC to obtain the best hyperparameter combination for each approach. Our evaluation based on nDCG@n and precision@n revealed that the cosine similarity can serve as an effective reranking mechanism for a known resultset. Although the hybrid-doc2vec and BERT-based approaches were initially expected to perform better, the low nDCG for the hybrid-doc2vec approach could be attributed to the NER process before embeddings, which exhibited the lowest nDCG when used alone.
Our comparison allows us to find a similarity-based approach that exhibits a natural resemblance to relevance without requiring additional training. However, it is important to note that relevance and similarity are related concepts but not interchangeable, thus further finetuning is necessary to optimize for relevance. Our future work will explore this subject as well as further variations of the word2doc2vec and combinations with NER approaches, using multiclassification for the distribution-based analysis used to optimize the hyperparameters (rather than reducing three assessments to two). We also want to try other options for the hybrid approach, such as revising the dictionary, using a different dictionary-based annotator, or doing the replacement after getting the vector space. Additionally, we will use a classifier approach to find a sound approach for relevance. Our ultimate goal is to provide a semantic-based approach that can assist researchers to find relevant literature not only in the well-covered biomedical domain, particularly with respect to Medline abstracts), but also in the agricultural domain, where there is a need for better coverage of non-traditional and non-peer-reviewed publications.
This work was partially supported by the STELLA project funded by the Deutsche Forschungsgemeinschaft DFG (project no. 407518790), the NFDI4DataScience project also funded by DFG (project no. 460234259), and the BMBF-funded de.NBI Cloud within the German Network for Bioinformatics Infrastructure (de.NBI) (031A532B, 031A533A, 031A533B, 031A534A, 031A535A, 031A537A, 031A537B, 031A537C, 031A537D, 031A538A).