Novel research drives scienti0ic breakthroughs but also has higher uncertainty of being recognized by citation count based metrics. This study proposed two indicators to measure the content novelty of a paper based on the knowledge entities it contains, and explored the relationship between content novelty and scienti0ic impact of papers.It is found that content novelty is negatively correlated with citation impact in our dataset. Our 0indings suggest that science policy in favor of citation count based impact may be biased against novel research.
•Applied computing~Document processing
2 2.1
2.2
The knowledge content of a paper is represented by the Pubtator Central 3 entities and the pairwise combination of entities in the paper. The PubTator Central system provided biomedical concepts such as genes, chemicals that were automatically extracted from each PubMed abstract. The F1 score of this system is higher than 80% [4]. We obtained entities of each article in our dataset by searching PMID in the system via API.
We determined the novelty degree of a paper as the proportion of its new knowledge entities and new knowledge entities pairs that were not appeared in its references. We compared the entities in two sources by exactly matching, which means the same entities in the two sources should be exactly identical.
Formally, the two indicators were computed as follows: (1) The proportion of new knowledge entities in a paper : ! = ""!!" (1) 3 https://www.ncbi.nlm.nih.gov/research/pubtator/ Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) The ! is the number of knowledge entities in paper , while !# is the number of new knowledge entities in paper that were not occurred in its references. (2) The proportion of new knowledge entities in paper :
pairwise combination of The ! is the number of distinct pairwise combination of knowledge entities in paper , while !# is the number of new pairwise combination of knowledge entities in paper that were not appeared in its references. 2.3
Citation count has often been applied to evaluate the scienti0ic impact of publications. In this paper, we also explored the relationship between content novelty and citation counts of papers. We ranked the novelty values of all papers in ascending order. For each decile of novelty, we calculated the average number of citations each paper received over a 3-year period after the publication year, and the proportion of top10% highly cited papers. These processes also applied for each domain of our dataset separately to observe the differences among domains. In addition, we used Pearson correlation coef0icient to measure the strength of association between content novelty and scienti0ic impact. We considered short-term impact (1-year citations), medium-term impact (3-year citations and 5-year citations) and long-term impact (10-year citations) of papers. 3
Figure 1 presents the mean of citation counts for each decile of novelty, where novelty was measured by ent and com indicators respectively. We can observe that citations decrease signi0icantly as the rise of novelty. The paper in the last decile of novelty has more than 10 citations less than the paper in the 0irst decile on average, either in ent or com measured novelty. A slight increase of citation counts is observed in the 10-20 percentile group of novelty that measured by ent. These patterns are robust across the major domains in our dataset.
The proportion of top10% highly cited papers declines with the deciles of novelty, as shown in Figure 2. The result is also robust across domains in our dataset. Only Health Sciences exhibits a slightly different pattern that the proportion of top10% papers has increased in the 10-20 percentile group of ent measured novelty.
Table 1 presents the Pearson correlation coef0icients between content novelty indicators and different impact of papers. It shows that content novelty of papers has signi0icantly negative correlation with scienti0ic impact. However, the correlation coef0icients are not very large, ranging from 0.078 to 0.130.