Science of Science (SciSci) is an emerging discipline wherein science is used to study the structure and evolution of science itself using large data sets. The increasing availability of digital data on scholarly outcomes o ers unprecedented opportunities to explore SciSci. In the progress of science, the previously discovered knowledge principally inspires new scienti c ideas, and citation is a reasonably good re ection of this cumulative nature of scienti c research. The researchers that choose potentially in uential references will have a lead over the emerging publications. Although the peer-review process is the mainly reliable way of predicting a paper's future impact, the ability to foresee the lasting impact based on citation records is increasingly essential in the scienti c impact analysis in the era of big data. This paper develops an attention mechanism for the long-term scienti c impact prediction and validates the method based on a real large-scale citation data set. The results break conventional thinking. Instead of accurately simulating the original power-law distribution, emphasizing the limited attention can better stand on the shoulders of giants.
With the advent of the era of big data, people pay more and more attention
to the value of data. The massive volume of publications created every year
has grown into big data that we can't ignore. With the development of SciSci, it
provides a quantitative understanding of scienti c discovery, creativity, and
practice [
(a) The volume of literatures.
(b) The citation Distribution.
Scienti c work is founded on prior research. It is not wise nor possible for
researchers to track all existing related work due to the enormous volume of the
existing publications. In general, researchers follow or cite merely a small
proportion of high-quality publications. SciSci provides several quanti cation methods
for scienti c impact measurement in article-level, author-level, and journal-level.
Much SciSci work has been done on the evaluation metrics for the quality and
in uence of scienti c work [
A lot of SciSci researchers have focused on the characterization of scienti c
impact [
The fact is that the current citation count and the derived metrics can only
capture the past accomplishment. They lack the predictive power to quantify
the future impact [
The other prevalent line utilizes Deep Neural Network (DNN) based
models to solve the scienti c impact prediction problem. Recently, Convolutional
Neural Network (CNN) and Recurrent Neural Network (RNN) have received
considerable attention from academia and the industry. RNN has been proven
to perform particularly well on temporal data series [
All the existing methods try to tune the citation distribution precisely as the original power-law distribution. However, this paper argues that the e ectiveness of quantifying long-term scienti c impact is fundamentally limited in this routine thinking. This paper proposes to put more attention on some speci c items, such as highly cited papers. The authors validate the proposed method on a real large-scale citation data set. Extensive experiment results demonstrate that the proposed method possesses remarkable power at predicting long-term scienti c citation. The most important contribution is that this paper changes the line of thinking in quantifying the long-term scienti c impact. Instead of simulating the original power-law distribution, researchers need to emphasize the limited attention to better stand on giants' shoulders. 2
The primary evaluation metric for scienti c impact is citation count. The received citation count of an individual paper d during time period [0; T ] is characterized by a time-stamped sequence fntdgtT=0, where ntd represents the number of citation counts received by paper d at time t, ntd is an integer greater than or equal to zero. In giving the historical citation records, the goal is to model the future citation count and predict it over an arbitrary time.
Given the literature corpus D, card(D) = M means nd M papers from D. In this paper, we believe that the scienti c impact of a literature article is equal to the number of papers which cite it. The scienti c impact of a literature article d 2 D at time t is de ned as its citation counts ntd: citingdt = fd 2 D; d~ 6= d : d~t cites dg; ntd ~ = card(citingdt): (1)
The underlying assumption of the citation count here is the accumulated citations, which make it possible to quantify citations for di erent items at di erent times. The long-term scienti c impact of the individual item d can be formalized as the following time series fn0d; ; ntd; ; ndT g. Without loss of generality, the number of accumulated citation counts increases over time. And then, we have 0 = n0d ntd ndT = Nd.
In the scienti c impact prediction problem, the input X~ is fn0d; ; ntd; g, for every paper d 2 D, where ntd is the citation counts of paper d at time t. The goal of the scienti c impact prediction problem is to learn a predictive function f to predict the citation counts of an article d after a given period time t. Formally, we have:
f (djX~ ; t) ! n^td; where n^td is the predicted citation count and ntd is the actual one. Based on the learned prediction function, we can predict the citation count of a paper for the next years. For example, the citation count of paper d at time t is given by f (djX~ ; t). (2) 3
As the most e cient scienti c impact prediction method found so far, RNN has already achieved compelling performance in predicting the scienti c impact. This paper embeds the RNN with LSTM units as a baseline and then emphasizes highly cited articles in the proposed attention mechanism. Although many other elds have used the attention mechanism, the proposed method gives new insight into long-term quantifying scienti c impact. Instead of adapting citation distribution to a power-law distribution, this paper's ndings provide a new line of thinking for the SciSci research. (a) The overview.
(b) The LSTM units.
(c) The attention model.
Given a time-stamped sequence fndgt=0, a K-dimensional feature vector X~ =
t T
fx0d; ; xtd; ; xdT g needs to be designed as input. The input space of every
item with popularity records f(x0; n0); ; (xt; nt); ; (xT ; nT )g re ects the
intrinsic quality of the item. Fig. 2(a) gives an overview of the model architecture.
There are two critical components in the architecture: the RNN with LSTM units
and the attention model. As illustrated in Fig. 2(b), it arranges the LSTM units
in the form of RNN with L layers. In the deep neural network, the parameter
L depends on the input scale. RNN is famous for its popularity and well-known
capability for e cient time series learning [
The RNN with LSTM Units. The LSTM units are arranged in the form of
RNN, as illustrated in Fig. 2(b). There are four major components in a standard
LSTM unit, including a memory cell, a forget gate f , an input gate i, and an
output gate o. The gates are responsible for information processing and storage
over arbitrary time intervals. Usually, the outputs of these gates are between 0
and 1. A new study gives suggestions to push the output values of the gates
towards 0 or 1. By doing so, the gates are mostly open or closed instead of in a
middle state [
Intuitively, the input gate controls the extent to which a new value ows into the memory cell. The input gate's function passes through the input gate and is added to the cell state to update it. The following formula for the input gate is used: it =
Wi ht 1; xt + bi ; (3) where matrix Wi collects the weights of the input and recurrent connections. The symbol represents the Sigmoid function. The values of the vector it are between 0 and 1. If one of the values of it is 0 (or close to 0), it means that this input gate is closed, and no new information is allowed into the memory cell at time t. If one of the values is 1, the input gate is open for a new coming value at time t. Otherwise, the gate is in the state of half-open half-clearance.
The forget gate controls the extent to which a value remains in the memory cell. It provides a way to get rid of the previously stored memory value. Here is the formulation of the forget gate: ft =
Wf ht 1; xt + bf ; (4) where Wf is the weight matrix that governs the behavior of the forget gate. Similar to it, ft is also a vector of values between 0 and 1. If one of the values of ft is 0 (or close to 0), it means that the memory cell should remove that piece of information in the corresponding component in the cell. If one of the values is 1, the corresponding information will be kept.
Remembering information for long periods is practically the default behavior of LSTM. The long-term accumulative in uence is formulated as follows: That is, the information in the memory cell consists of two parts: the retained old information ft ct 1 (controlled by the forget gate) and the new coming information it c~t (controlled by the input gate).
The output gate controls the extent to which the value in the cell is used to compute the output activation of the LSTM unit. The following output function is used: ot =
Wo ht 1; xt + bo : The weight matrices and bias vector parameters are needed to be learned during training. This paper updates the current working state as the following formula: ht = t tanh ct : o ct = t f ct 1 + it
c~t; c~t = tanh Wc ht 1; xt + bc : where denotes the Hadamard product (the element-wise multiplication of matrices), c~t is calculated as follows: (5) (6) (7) The items stored in the current working state have an advantage in reading over those stored in long-term memory. In the time series modeling of scienti c impact, the recent items stored in the short-term working state have an advantage over those stored in the long-term memory. The next step introduces the attention mechanism based on ht.
The Attention Model. The arti cial attention mechanism, inspired by the attention behavior in neuroscience, has been applied in deep learning for speech recognition, translation, and visual identi cation of objects.Broadly, attention mechanisms are components of prediction systems that allow the system to focus on di erent subsets of the input sequentially. It aims to capture the critical points and focuses on the relevant parts more than the remote parts as a human does. More speci cally, content-based attention generates attention distribution. Only part of a subset of the input information is focused. The attention function needs to be di erentiable, so that everywhere of the input is focused, just to di erent extents.
The deep learning attention mechanism used in this paper works as follows: given an input X~ = fx0d; ; xtd; ; xdT g, the aforementioned LSTM units generate ~h = fh1; ; ht; ; hT g to represent the hidden patterns of the input. The output is the summary of the ht focusing on information linked to the input. In this formulation, attention produces a xed-length embedding of the input sequence by computing an adaptive weighted average of the state sequence ~h.
The graphical representation of the attention model is shown in Fig. 2(c). The input X~ and the hidden layer ~h of the LSTM network (an RNN composed of LSTM units) are the input of the attention model. Then, it computes the following formula: at = tanh Wa xt; ht ; where Wa is the weight matrix. An important remark here is that each at is computed independently without looking at the other xt0 for t0 6= t. Then, each at is linked to a Softmax layer, which function is given by: (9) (10) (11) t =
eat P eat ; for t = 1; t
; T O = X
txt: t where Pt t = 1, the t is the softmax of the at projected on a learned direction. The output is a weighted arithmetic mean of the input, and the weights re ect the relevance of ~h and the input. It is calculated as the following formula: Finally, the popularity of item d at time t is given by the prediction f (djX~ ; t) = O. 3.2
As widely acknowledged, the citation distribution follows the power-law
distribution. This nding leads the way for research in this domain. Researchers try
to simulate the citation distribution as the power-law distribution. This paper
changes the line of thinking. Although the number of research papers has
exploded, the reading time of scientists has not. At the same time, the attention
shifts toward the top 1% over time [
In the fact of limited attention, the Matthew e ect dominates in quantifying
the long-term scienti c impact. The experiments will con rm it. The citation
count captures the inherent di erences between papers, accounting for the
perceived novelty and the importance of a paper. The "rich-get-richer" phenomenon
summarizes the Matthew e ect of accumulated advantage, i.e., previously
accumulated attention triggers more subsequent attention [
4.1
This section demonstrates the e ectiveness of putting particular emphasis on the vital factor in quantifying the long-term scienti c impact.
The authors extract the data from an academic search and mining platform called AMiner and construct a real large-scale scholarly dataset{Academic Social Network1. The citation network's full graph in this dataset has about 2 million vertices (papers) and 8 million edges (citations). In detail, the dataset is composed of 2; 092; 356 digitalized papers spanning from 1936 to 2016 (for more than 80 years) and 8; 024; 869 citations between them. By convention, the authors eliminate those papers with less than 5 citations during the rst 5 years after publication and only retain the remaining papers as the training data. As a result, 143; 902 papers published from 1956 to 2015 are retained. 4.2
To compare the predictive performance of the proposed attention model against
other models, we introduce several published models that have been used to
predict scienti c impact. Speci cally, the experiments' comparison methods are LR,
CART, SVR (the three basic machine learning methods used in [
This paper uses two basic scienti c impact evaluation metrics: Mean Absolute Percentage Error (MAPE) and Accuracy (ACC). Let ntd be the observed citations of paper d up to time t, and n^td be the predicted one. The MAPE measures the average deviation between the predicted and observed citations over all the papers. For a dataset of M papers, the MAPE is given by: 1 XM n^td M d=1 ntd ntd : (12)
ACC measures the fraction of papers correctly predicted under a given error tolerance . Speci cally, the accuracy of citation prediction over M papers is de ned as:
ACC =
where I[ ] is an indicator function which returns 1 if the statement is true,
otherwise returns 0. We nd that our method always outperforms regardless of
's value. In this paper, we set = 0:3.
The experiment results show that the longer the duration of the training set, the
better the long-term prediction performance. According to our experiment, this
paper sets the training period as 5 years and then predicts the citation counts for
each paper from 1st to 5th after the training period. For example, t = 1 means
that the rst observation year after the training period. In the experiments, the
features with positive contributions are the citation history, the current h-index
of the paper author, and the publication journal level [
attention module of the proposed model to verify the e ectiveness of the attention mechanism. The remainder is RNN with LSTM units (labeled as LT-CCP), proven to be useful in long-term citation count prediction. In the next step, we add the attention mechanism in two di erent ways. Firstly, we add the attention module before the RNN module, labeled as ATT-B-LT (attention before LT-CCP). In a second way, we add the attention module after the RNN module, labeled as ATT-A-LT (attention after LT-CCP). As shown in Fig. 3(b) and Fig. 3(a), the ACC is increased, and the corresponding MAPE is decreased. Both ATT-B-LT and ATT-A-LT perform better than LT-CCP in terms of MAPE and ACC. Introducing the attention module improves the ability of scienti c impact prediction. The e ectiveness of the attention mechanism is veri ed. (a) ACC comparision.
(b) MAPE comparision. (c) LT-CCP (RNN LSTM).
with (d) ATT-B-LT.
(e) ATT-A-LT (DLAM).
In addition, we can see that the ATT-A-LT performs better than ATT-BLT. When the attention model is applied after a deep learning model, it is more e ective than the reverse combination. It indicates that the deep learning model can learn the implicit features underlying the citation records, which boots the performance.
predicted citation distribution of LT-CCP (RNN with LSTM), ATT-B-LT, and ATT-A-LT (DLAM) when t = 5 in Fig. 3(c), Fig. 3(d), and Fig. 3(e), respectively. The LT-CCP (RNN with LSTM) illustrated in Fig. 3(c) shows the best simulation of the power-law distribution. But the ATT-B-LT shown in Fig. 3(d) and the ATT-A-LT (DLAM) shown in Fig. 3(e) present bad simulation of the power-law distribution. The results show that LT-CCP (RNN with LSTM) matches very well with that of real citations, but the ATT-B-LT and the ATTA-LT (DLAM) don't. Usually, it is believed that the more similar the power-law distribution, the whole result is better. At rst glance, it seems that LT-CCP (RNN with LSTM) performs the best.
However, the rst thought is wrong. As veri ed in Fig. 3(a) and Fig. 3(b), the LT-CCP (RNN with LSTM) performs the worst. In fact, the LT-CCP only has a better tting e ect on the papers with little citation counts. On the contrary, the ATT-B-LT and ATT-A-LT (DLAM) have a better tting e ect on the highly cited papers. The methods with attention mechanisms achieve better overall performance than others. It is more accordant with practical prediction requirements that a few papers occupy a vast number of citations. It further proves the e ectiveness of the attention model. The experimental results indicate that we need to change the xed pattern of thinking in quantifying long-term scienti c impact. 6
Scienti c impact evaluation is always a critical point in decision-making concerning recruitment and funding in the scienti c community. The rapid evolution of scienti c research has been creating a massive volume of publications every year. Among the many quanti cation measures of scienti c impact, citation count stands out for its frequent use in the research community. Although the peerreview process is the mainly reliable way of predicting a paper's future impact, the ability to foresee the lasting impact based on citation records is increasingly essential in the scienti c impact analysis in the era of big data.
SciSci provides a quantitative understanding of the scienti c impact based on big data empirical analysis. In this paper, the authors develop an attention mechanism in long-term scienti c impact prediction and verify its e ectiveness. More importantly, this paper provides us great insights into understanding the critical factor in quantifying the long-term scienti c impact. Usually, researchers try to make the predicted citation distribution similar to the original one. However, the experimental results in the paper question this solution. In future research work, we need to change the xed pattern of thinking in quantifying the longterm scienti c impact and emphasize limited attention to better stand on giants' shoulders.
The work is supported by the National Natural Science Foundation of China (NSFC) under Grant No. 61806111 and NSFC for Distinguished Young Scholar under Grant No. 61825602.