Citation is a manifestation of the academic impact of scientific papers. Among diverse citation motivations, the most crucial one is that the research elements of a paper, e.g., the proposed problem, methods, models, etc., inspire the research of peers. As the tags of research elements are known as knowledge entities, citations that refer to certain knowledge entities of a cited paper are called entity citations. Both the position and the strength of an entity citation indicate the impact that a certain research element has. In this study, the academic impact of a cited paper is measured by the entity citations. A measurement approach is proposed with the technique of knowledge entity recognition and entity citation detection. The impact of a paper can be more precise and more interpretable with the proposed approach. The findings of this study can enhance the impact evaluation of both papers and knowledge entities, as well as improve the ranking quality in knowledge retrieval applications.
When evaluating the impact of a scientific paper, it is necessary to understand what has
inspired peers' studies besides simply counting the citation number. Among different citation
motivations, the research elements that describe problems, methods, models, and so on are the
most crucial factors that drive citations, as they outline the important parts of solving problems. In
scientific papers, these elements are called knowledge entities [
In previous studies, entity types in the domain of chemistry, biology, and medicine such as
genes [
As research builds upon previous work, these entities can be explicitly or implicitly mentioned in the context of citing papers. Thus, the semantic meaning of citation context may associates with certain knowledge entity. From the perspective of influence, the importance of a knowledge entity is related to the frequency and the position its citation appears in citing papers, and also to the He) influence of the citing papers themselves. Consequently, the more important entities a paper contains, the more impact it has.
This study aims to discover entity citations in the citing papers and evaluate the impact of cited papers. The contribution lies in that it not only helps to explain the exact reasons for the paper's impact but also improves the academic impact measurement at the granularity of knowledge entities. Besides, the study that identifies the important knowledge entities also helps to discover the core knowledge of the research domain and improves the rank of knowledge retrieval.
Academic influence evaluation for publications, journals, or other academic products has been a
hot topic in the field of information science for a long time. The metrics include the citation
number of papers, the impact factor of journals, and the number of likes, comments, and retweets
on social media. However, the accounting of citation is mainly based on coarse-grained objects,
such as papers. While some studies have conducted citation analysis at the lexical level, they often
treat keywords as highlight terms without leveraging specialized knowledge entities [
Nakov et al. consider the sentences surrounding citations as an important tool for the semantic
interpretation of cited papers. They define the text span of citation sentences and illustrate that a
set of citation sentences expresses the same concepts in different ways [
A series of studies contribute to revealing the academic value of cited papers through
microlevel analysis using citation content. Thelwall et al. argue that being cited by a highly valuable
paper indicates a significant influence of the cited paper, integrating the citation frequency of
referenced works into the evaluation system for paper importance [
As shown in Figure 1, the evaluation of the academic impact of scientific papers consists of two phases. Firstly, detecting entity citation by comparing the meaning of citation contexts with the knowledge entities of the cited paper. Secondly, evaluating the impact of cited papers based on the weight of entity citation. The weight combines the importance of the citing paper, the citation strength and the position of the citation context.
Let D, C, and E respectively denote the cited papers, the papers that cite D, and the knowledge
entities of D. The corpus D is composed of titles and abstracts of papers in certain fields. C(di) is
the full text of papers that cites di, di∈ D, m=|C(di)|. e(ki) represents the kth entity identified from
di. s(l j) denotes the lth citation context in c j, c j∈ C ( di ). The knowledge entities are identified by a
public toolkit based on the BiLSTM-CRF framework [
Set E includes the research problems Ep and methods Em. The total number of Ep and Em in di is r and r’ respectively.
The academic impact of di, denoted as I(i), is defined as Formula 1. It considers the importance of the citing paper, the position weight of the entity citation appears, and the strength the entities are mentioned, which are respectively denoted as I j, at and f (ki),t. In this study, I j is the citation number of c j. at is assigned by Entropy Weight Method (EWM) to four different positions, i.e., Introduction & Background, Methods & Dataset, Experiment & Analysis, Conclusion. The weight of position is calculated in section 4.3. And f (ki),t means the number of citation contexts in position t of c j that mention the entity e(i).
k m 4 r+r' I (i)=∑ I j ∙ ∑ at ∑ f (ki),t j=1 t=1 k=1 (1)
In the experiment of detecting entity citation, the parameters for comparing the citation contexts with the knowledge entities are tuned for higher accuracy based on G1. A total of 343 problem 33 entities and 544 method entities are matched with all 10007 citation contexts. G2 is used for evaluation the performance. The accuracy is 79%.Three aspects need to be highlighted. Firstly, the average accuracy is higher when the citation context only includes citation sentence than includes sentences before and after it. It means a larger context does not necessarily introduce desired semantic information about the cited knowledge entities. Secondly, the average accuracy increases from 43.6% to 51.4% when duplicating words from 1 to at most 10 before the citation notes for 5 times, as illustrated in Figure 2. It reminds us that effective information about the cited entities becomes denser in words preceding the citation notes. Thirdly, the accuracy increases further when combining the Word2Vec vectors with SciBERT vectors. The best performance is achieved by setting the weight 0.16 for SciBERT vectors and 0.84 for Word2Vec vectors, see Figure 2. It indicates richer domain knowledge can effectively compensate for the semantic representation since the word2vec model is trained in the domain-specific papers.
Weight of citation position We first identify the position of each citation context in papers of C, then count the frequency of each position. Citation position weights are determined with EWM. The number and the normalized weights are shown in Table 1. What to be noted is although most citations appear in the first two positions, there are still 28% in the last two, which implies that the cited knowledge entities may provide experimental supports or theoretical foundations to the citing papers, and thus have a higher weight.
The weights derived in this study are largely consistent with those obtained through the expert
scoring method in [15] and the questionnaires combined with AHP method as used in [
Academic impact of papers In Figure 3, the total impact value of each paper is illustrated in descending order, also the impact contributed by the problem entities and the method entities of the cited paper D is shown. Generally, the papers with high impact only occupy a small proportion, most papers are not very influential. Notably, for highly impactful papers, contributions from both problem entities and method entities are substantial. The top six impact papers give evidence as shown in Table 2.
In a whole, the impact contributed by method entities is higher than problem entities. Considering the selected scholar is of high impact in the domain whose H-index ranked in the top 25 among the computer scientists of the world till Nov. 2023, according to Research.com, the results indicate the contribution of the scholar to the domain mainly lies in method innovation.
To further confirm the rationale of the academic impact measurement driven by entity citations, we calculate the correlation between the impact score proposed in this study and the citation number of papers, the classical impact metrics. The results in Table 3 indicate a high correlation of the two metrics; and in this research field, the impact contributed by method entities is notably higher than those of problem entities which is probably due to the substantial number of method entities. Scholars cite previously published papers when the research elements of these papers enlighten their studies. They usually state the research elements with concise expression and, in most cases mention original knowledge entities of the cited papers. Such a citation driven by an entity indicates the influence of the inspiring knowledge. We propose an evaluation approach to academic impact with consideration of the entity citation. The contribution lies in that it not only helps to explain the exact reasons for the impact of a cited paper but also improves the academic impact measurement at the granularity of knowledge entities. Similar to traditional metrics like citation counts, our method cannot predict the impact of papers that have not been cited. However, it can reveal the specific reasons for papers' impact. Besides, the study that identifies the important knowledge entities also benefits to discovering the core knowledge of the research domain, and improving the rank of knowledge retrieval.
In the future, publications may be presented at a finer granularity of knowledge units. The method proposed in our study can be directly applied to the evaluation of research outputs, researchers, knowledge discovery, and information services. 35
This study is supported by the National Social Science Foundation of China (grant number 21BTQ065). The paper is presented at the second Workshop on “Innovation Measurement for Scientific Communication (IMSC) in the Era of Big Data” at 2024 ACM/IEEE Joint Conference on Digital Libraries (JCDL).
During the preparation of this work, the authors used ChatGPT, Kimi in order to: translate a small portion of text into English and cheek spelling. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. Organization Conference (lSKO 2024). Advances in Knowledge Organization, Volume 20, 7588.Ergon, Baden-Baden. [15] Siniša Maričić, Spaventi J, Leo Pavičić, et al. Citation context versus the frequency counts of citation histories.[J]. Journal of the American Society for Information Science, 1998, 49(6):530540. [16] Juyoung An, Namhee Kim, Min‐Yen Kan, et al. Exploring characteristics of highly cited authors according to citation location and content[J]. Journal of the Association for Information Science and Technology., 2017, 68(17):1975-1988.