Biomedical research has not only academic impact, but also clinical impact. The evaluation of the clinical impact of COVID-19 papers is very interesting. It is still unclear whether the interdisciplinary and entity characteristics of the paper affect the clinical impact. We selected COVID-19 papers published in 2021 for preliminary exploration and got some interesting findings. We found that only 22.43% were cited in clinical trials or clinical guidelines; 47.42% of the papers are biased towards human research in MeSH terminology; On average, 46.1% of papers have the potential to be cited in clinical studies after publication. The interdisciplinary features of the paper are not significantly related to clinical translation intensity, but the biomedical entity features mentioned in the paper are significantly related to clinical translation intensity. The number of Chemical entities, Gene entities and Species entities had significant negative effects on clinical translation intensity. However, the number of Disease entities mentioned in the paper has positive impact on the clinical translation intensity.
1 Introduction
The main purpose of biomedical research is to
serve the public health and improve the well-being
of the people. So, Biomedical research should have
not only academic impact but also clinical impact.
The evaluation of scientific impact has been very
rich, but the identification and measurement of
clinical impact is still relatively weak.
Interdisciplinary research is the main mode of
modern biomedical research. Previous studies have
shown that interdisciplinarity is significantly
related to the citation of papers, and highly cited
papers have a higher interdisciplinary level. In our
previous research on papers published by Lasker
Prize winners in Basic Medicine, we found that the
average number of disciplinary involved in papers
published by winners was positively correlated
with APT, while the number of subjects is not
significantly correlated with clinical citations [
Therefore, more diverse, and complex
interdisciplinary indicators are needed to verify its
correlation with clinical translation intensity in
other samples. Recently, biological entity features
have also been used to predict the clinical
translational potential of a paper or to measure the
translational progress of a paper [
This study includes two objectives: 1) to measure the clinical translation intensity of COVID-19 articles published in 2021; 2) to test the impact of interdisciplinary level and the characteristics of biological entity on the intensity of clinical translation of COVID-19 papers. 2
Related works 2.1 The translation papers measurement intensity of of clinical biomedical
As for the clinical translation intensity of papers,
some methods have been developed, such as
“Research Level” [
It is reported that in 2020, the research on
COVID-19 involved an average of 6-7 disciplines
[
Knowledge entity refers to the unit of
knowledge in scientific articles [
The entity characteristics of a paper are related to the interdisciplinary characteristics, for example, the entity of a biomedical paper is a terminology or a unit of knowledge in the biomedical field. But the two are measured from different perspectives. The interdisciplinary features are measured from the subject of the journal in which the paper's references are published, while biomedical entity features are measured from the biomedical proprietary concepts mentioned in the title and abstract. Entity features are more fine-grained than subject features.
The research process mainly includes three parts: First, data collection and preprocessing; The second is to measure the clinical translation intensity of papers; Third, verify the effect of interdisciplinarity and entity characteristics on the clinical translation intensity of papers.
On March 13, 2023, we exported a total of
120,573 papers published in 2021 from the iSearch
COVID-19 portfolio [
We select three indicators (APT, Human and
Cited_by_Clin) to represent the clinical translation
intensity of the paper from different perspectives.
Data for all three dependent variables are
downloaded from the iCite platform [
APT (Approximate Potential to Translate) is a
clinical translation indicator developed by
Hutchins et al. in 2019 to predict the probability
that a paper will be cited in a clinical paper shortly
after publication [
The indicator “Human” is another clinical translation indicator developed by Hutchins et al. which indirectly reflects whether the research topic of the paper focuses on human health. The range of “Human” value is between 0 and 1. The larger the “Human” value is, the closer the topic of the paper is to human health, and the higher the clinical translation potential of the paper is.
The indicator Cited_by_Clin refers to the absolute number of times each paper is cited by clinical papers such as clinical trials and clinical guidelines. Cited_by_Clin is a continuous variable that is greater than or equal to zero.
We select two kinds of independent variable indicators, including interdisciplinary characteristics and entity characteristics.
We measure interdisciplinary level by the diversity of subjects a paper is classified into. In this paper, we used Variety, 1-GINI, Disparity and Rao-Stirling indicator to measure the interdisciplinary level of the papers. Table 2 lists the interdisciplinary indicators and their connotations.
In this study, PubTator Central (PTC) was used
to extract the biological entities mentioned in the
title and abstract of each article
(https://www.ncbi.nlm.nih.gov/research/pubtator/).
It divides biological entities into six categories:
Gene, Chemical, Disease, CellLine, Mutations, and
Species. The efficiency and accuracy of PubTator
for biomedical entity extraction are superior to
manual text mining [
In this study, Pubtator API was invoked to extract biomedical entities mentioned in the title and abstract of the paper in batches by PMID number. We assign a unique identifier to each individual entity of each entity class. The six indicators that separately count the number of entities of each type mentioned in each paper including CellLine, Chemical, Disease, Gene, Mutation and Species. We performed deprocessing when calculating the number of entities mentioned in each paper. For example, if a paper mentions the same independent entity multiple times, we only count it once. Table 3 summarizes entity characteristic indicators and their definitions.
We select 8 variables as control variables, which are 1) Whether it is a clinical study, clinical guidelines or clinical trial, extracted from iCite; 2) Paper length, extracted from the "PG" field of the core collection of WOS database; 3) Title length, counting the number of words in the title; 4) The number of references, extracted from the "NREF" field of the core collection of WOS database; 5) The number of authors, extracted from the "AU" field of the core collection of WOS database; 6) The number of funds, extracted from the "FU" field of the core collection of WOS database; 7) Paper language, extracted from the "LA" field of the core collection of WOS database. 8) The first corresponding author's institution type, extracted from the "RP" field of the core collection of WOS database.
First, we calculate APT, Human, and Cited_by_Clin for COVID-19 papers published in 2021. In addition, we use the negative binomial regression method to examine the influence of interdisciplinary features and entity features on APT, Human and Cited_by_Clin of the papers.
The study sample was 36,797 papers. APT and Human have values between 0 and 1, while Cited_by_Clin has a maximum value of 168. Therefore, we use two vertical axes in box plot (Figure 2). APT and Human correspond to the scale on the left and Cited_by_Clin to the scale on the right. Table 1 lists the descriptive analysis result of dependent and independent variables.
As is shown in Figure 2, the median of Human is smaller than its average, suggesting that most papers with a larger Human value in the sample. 47.42% of the papers had a Human value greater than 0.5, and 77.57% were not cited in clinical papers. The distribution of APT is close to symmetric, while Human is positively skewed. Cited_by_Clin is a variable with most zeros, and its mean, median, upper, and lower quartiles and minimum values are all zero. We do not see IQR and whiskers, but only many positive outliers.
.006 .003 .006
.02 .008 0 0 0
Table 2 reveals the negative binomial regression results for APT. We found that the interdisciplinarity had no statistical significance on APT (p﹥0.1). The number of chemical entities, gene entities and species entities were significantly negatively correlated with APT, while the number of disease entities was significantly positively correlated with APT. -.019
0 .169 .168 .174 .021 .102 -.493 .07
. .102 .102 .064 .067 .063 .191 .098 .099 .006 .749 .103 .01 .713
Table 3 shows that the interdisciplinarity has no significant impact on the Human of the papers. However, the number of CellLine entities, chemical entities, gene entities, mutation entities and species entities have a negative impact on Human, while the number of disease entities has a positive impact on Human.
.04 .014
As shown in Table 4, interdisciplinarity had no significant effect on the number of clinical citations. The number of chemical entities, gene entities, mutation entities and species entities have a significant negative impact on the number of clinical citations, while the number of CellLine entities and disease entities has a significant positive impact on the number of clinical citations.
Firstly, after measuring the clinical translation intensity of 36,797 COVID-19 papers published in .028 .074 2021, We found that APT, Human and Cited_by_Clin, three clinical translation intensity indicators with different connotations, can provide three dimensions of mutually complementary information for the same paper sample. APT is measured from the perspective of the potential of a paper to be cited by clinical guidelines or clinical trials after publication, Human is measured from the perspective of the proportion of MeSH terms in a paper that are biased toward human beings, and Cited_by_Clin is measured from the perspective of the actual number of citations of a paper by clinical guidelines or clinical trials.
Our team previously investigated the clinical
conversion intensity of papers published by Lasker
Prize winners in Basic Medicine, and found that the
average value of APT and Cited_by_Clin was 0.24
and 0.59, respectively, and 80% of the papers were
not cited in clinical papers [
Secondly, although most previous studies have
confirmed that interdisciplinarity has a significant
impact on the citations of papers, in this study, we
did not detect a significant impact of
interdisciplinarity on the clinical impact of
COVID-19 papers. Some studies have found that
although interdisciplinary research dominates the
academic cooperation network, this competitive
advantage does not translate into immediate returns,
and the impact of these studies is low in the short
term. Interdisciplinary research requires more time
and perseverance to overcome challenges [
We believe that interdisciplinary research is more time consuming and energy consuming, which will not lead to higher impact performance in the short term. In addition, clinical impact puts more emphasis on the flow of knowledge in a paper to clinical application, while the traditional method of measuring interdisciplinary level is very crude, representing the discipline of the paper from the discipline of the journal in which the reference was published. This may lead to the traditional interdisciplinary level is not correlated with the clinical translation strength of the paper.
In contrast, a significant relationship was examined between the entity characteristics of the paper and its clinical impact. The number of chemical entities, gene entities and species entities had significant negative effects on APT. The number of CellLine entities, chemical entities, gene entities, species entities and mutation entities had significant negative effects on Human. The number of chemical entities, gene entities, mutation entities and species entities had significant negative effects on clinical citation. The CellLine entities has a positive relationship with clinical citation. Interestingly, the number of disease entities had a positive impact on APT, Human, and clinical citations.
Some chemical entities are often mentioned in
COVID-19 papers. For example, the effectiveness
of methylprednisolone in treating high-risk
COVID-19 patients [
Similarly, when genome researchers explore the relationship between COVID-19 and host genes, genes related to COVID-19 critical illness, and genomics studies on the natural origin of COVID19, the fewer the number of genetic entities mentioned in the paper may indicate the stronger the breakthrough of the paper content and the higher the clinical translation intensity.
For species entities, the lower the number of species entities mentioned in the paper, the higher the clinical translation intensity. For example, specifically studying the clinical presentation, pathogenesis, or treatment of COVID-19 in only men or women, or in only children or the elderly, will likely be more valuable for clinical diagnosis, treatment, and prevention.
Studies of the effect of drugs or vaccines targeting a single protein, DNA, and SNP mutation on the treatment or prevention of COVID-19 will have higher clinical translational strength than studies that combine multiple mutations.
Cytology studies on COVID-19 also follow a rule, that is, the CellLine entities involved in the research are more specific, the research is more indepth, and the clinical application is more valuable for reference, so the APT and Human of the paper is higher.
However, if there are more disease entities mentioned in COVID-19 papers, it means that this is a study on severe COVID-19 patients with multiple infections or comorbidities, which is a breakthrough study on severe COVID-19 patients, so this kind of research has higher clinical translation intensity.
The paper has some limitations. First, the intensity of clinical translation of a paper varies with the time of publication. COVID-19 papers published in 2021 have only a window of nearly two years to accumulate citations. Second, only some simple entity characteristic variables were used in this study, and future work is to create more diversified and systematic entity characteristic variables to further verify their relationship with clinical translation intensity.
We measured the clinical translation strength of COVID-19 papers published in 2021 and found that only 22.43% were cited in clinical trials or clinical guidelines; 47.42% of the papers are biased towards human research in MeSH terminology; On average, 46.1% of papers have the potential to be cited in clinical studies after publication. The interdisciplinary features of the paper are not significantly related to clinical translation, but the biomedical entity features mentioned in the paper are significantly related to clinical translation. The more disease entities mentioned in the paper, the stronger the clinical translation; However, the more chemical, genetic and species entities are mentioned, the weaker the clinical translation intensity is.