We present approaches to estimate content-aware bibliometrics to quantitatively measure the scholarly impact of a publication. Traditional measures to assess quality-related aspects such as citation counts and h-index, do not take into account the content of the publications, which limits their ability to provide rigorous quality-related metrics and can significantly skew the results. Our proposed metric, denoted by Content Informed Index (CII), uses the content of the paper as a source of distant-supervision, to weight the edges of a citation network. These content-aware weights quantify the information in the citation i.e., these weights quantify the extent to which the cited-node informs the citing-node. The weights convert the original unweighted citation network to a weighted one. Consequently, this weighted network can be used to derive impact metrics for the various entities involved, like the publications, authors etc. We evaluate the weights estimated by our approach on three manually annotated datasets, where the annotations quantify the extent of information in the citation. Particularly, we evaluate how well the ranking imposed by our approach associates with the ranking imposed by the manual annotations. The proposed approach achieves up to 103% improvement in performance as compared to second best performing approach.
Scientific, engineering, and technological (SET) innovations have been the drivers behind many of the significant positive advances in our modern economy, society, and life. To measure various impact-related aspects of these innovations various quantitative metrics have been developed and deployed. These metrics play an important role as they are used to influence how resources are allocated, assess the performance of personnel, identify intellectual property (IP)related takeover targets, value a company’s intangible assets (IP is such an asset), and identify strategic and/or emerging competitors.
Citation networks of peered-reviewed scholarly
publications (e.g., journal/conference articles and patents) have
widely been used and studied in order to derive such metrics
for the various entities involved (e.g., articles, researchers,
institutions, companies, journals, conferences, countries,
etc.
In this paper, we propose machine-learning-driven
approaches, that automatically estimate the weights of the
edges in a citation network, such that edges with higher
weights correspond to higher-impact citations. There has
been considerable effort in the past to identify important
citations
While our discussion and evaluation focuses on identifying informing citations, our approach is not restricted to this domain, and can be used to derive impact metrics for the various involved entities. For example, the contentaware weights estimated by the proposed approach convert the original unweighted citation network to a weighted one. Consequently, this weighted network can be used to derive impact metrics for the various involved entities, like the publications, authors etc. For example, to find the impact of a publication, the sum of weights outgoing from its corresponding node can be used to quantify the impact of the publication, instead of using vanilla citation count.
The reminder of the paper is organized as follows. Section 2 presents the related literature review. The paper discusses the proposed method in Section 3 followed by the experiments in Section 4. Section 5 discusses the results. Finally, Section 6 corresponds to the conclusions.
2
The research areas relevant to the work present in this paper belong to citation indexing, citation recommendation, link prediction approaches, distant-supervised credit attribution approaches and citation-intent classification approaches. We briefly discuss these areas below:
A citation index indexes the links between publications that
authors make when they cite other publications. Citation
indexes aim to improve the dissemination and retrieval of
scientific literature. CiteSeer
Citation recommendation describes the task of
recommending citations for a given text. It is an essential task, as
all claims written by the authors need to be backed up
in order to ensure reliability and truthfulness. The
approaches developed for citation recommendation can be
grouped into 4 groups as follows
A link is a connection between two nodes in a network.
As such, link-prediction is the problem of predicting the
existence of a link between two nodes in a network. A
good link-prediction model predicts the likelihood of a link
between two nodes, so it can not only be used to
predict new links, but to also curate the graph by filtering
less-likely links that are already present. Thus, the
linkprediction can be a useful tool to find likely citations in
a citation network. The citation recommendation task
described previously can be thought of as a special case of
linkprediction. Following the taxonomy described in
Various distant-supervised approaches have been developed
for credit-attribution, but the prior have primarily focused on
text documents. A document may be associated with
multiple labels but all the labels do not apply with equal
specificity to the individual parts of the documents. Credit
attribution problem refers to identifying the specificity of
labels to different parts of the document. Various
probabilistic and neural-network based approaches have been
developed to address the credit-attribution problem, such as
Labeled Latent Dirichlet Allocation (LLDA) (Ramage et al.
2009), Partially Labeled Dirichlet Allocation (PLDA)
Another line of work uses distant-supervised
creditattribution for query-understanding in product search.
Examples include, (i) using the reformulation logs as a source
of distant-supervision to estimate a weight for each term in
the query that indicates the importance of the term towards
expressing the query’s product intent
There is a large body of work studying the intent of
citations and devising categorization systems. In general, these
approaches treat citation-intent classification as a text
classification problem, and require the availability of training
data with ground truth annotations. Representative examples
include rule based approaches
Paper 3
Paper 4 Unit Normalization
Paper 1 (Citing paper) ---[Paper 4]---[Paper 3]---[Paper 2]Representation of the historical concepts
Minimize the explanation loss
driven approaches
3
In the absence of labels that define the impact, we assume that the extent to which a cited paper informs the citing paper is an indication of the citation’s impact. Specifically, we assume that each paper Pi can be represented as a set of concepts Ci. Further, we assume that each paper Pi is build on top of a set of historical concepts Hi, and its novelty Ni is the new set of concepts it proposes. The contribution of a cited paper Pj towards the citing paper Pi is the set of concepts Cji = Cj \ Hi. In other terms, the set of concepts Ci is given by:
Ci = Ni [ Hi = Ni [ [[PicitesPj Cji]: The task at hand is to quantify the extent to which Cji contributes towards Hi. To achieve this task, we look into the following directions: • How do we supervise the exercise? We minimize the novelty of paper Pi, by trying to explain the concepts in paper Pi (denoted by Ci) using the historical concepts, i.e., the concepts of the papers it cites (Cj ). We call the loss associated with this minimization as the explanation loss. This gives rise to the following optimization problem: minimize X
Ni = minimize X Ci
Hi:
i
i
To proceed in this direction, we need to answer two
questions, (i) How to represent the the set of concepts
associated with the paper Pi?, and (ii) How do we represent the
set of historical concepts Hi? As we show next, we use
the textual content of the papers to estimate the
representations of Ci and Hi. Thus, we formulate our problem as a
distant-supervised problem, and the content of the papers
acts as a source of distant-supervision.
• How to represent the set of concepts associated with
a paper? For simplicity, we represent the set of
concepts associated with a paper (Ci) as a pretrained
vector representation (embedding) of its abstract, such as
Word2Vec
= subject to
X Pi cites Pj
X Pi cites Pj w~ji w~jir(Cj ) w~j2i = 1 0; 8(i; j):
P w~j2i =
Pi cites Pj 1) to make the representation of r(Hi) agnostic to the number of cited-papers (a paper can cite multiple papers to reference the same borrowed concepts).
The weights w~ji can be thought of as normalized similarity measure between the concepts of the cited paper, and the citation context. Thus, to estimate w~ji, we first estimate unnormalized w~ji, denoted by wji, and then normalize wji so as to have unit norm. The unnormalized weight wji is precisely the extent to which Cj contributes towards Hi (and hence Ci), i.e., the weight that we wish to estimate in this paper. We estimate wji as a multilayer perceptron, that takes as input the representations of the cited paper and the citation context. We use the representation associated with the corresponding concepts as the representations of the cited papers (r(Cj )). Similar to r(Cj ), we use the ScispaCy vector representation for the citation context as the representation of the context, and denote it by r(j ! i).
The above discussion leads to the following formulation: minimize
f subject to
X jjr(Ci)
i w~ji = r
X
Pi cites Pj wji
P Pi cites Pj wj2i w~jir(Cj )jj2 ; 8(i; j); wji = f (r(Cj ); r(Cji)); 8(i; j); wji wji 0; 8(i; j); b; 8(i; j): (1)
The max-bound constraint (wji b) is introduced to limit the projection space of the weights wji. This is because, without this constraint, for a given citing paper Pi, if the set of weights wji minimize Equation (1), then so will any scalar multiplication of the weights wji. This can potentially lead to the estimated weights being incomparable across different citing papers. Having a max bound on the estimated weights helps avoid this scenario. To take care of the constraints, the function f ( ) can be implemented as a L2 regularized multilayer perceptron, with a single output node, and a non-negative mapping at the output node. Not that we do not explicitly set the max-bound b, but it is implicitly set by the L2 regularization of the weights of the function f . The L2 regularization parameter is treated as a hyperparameter. Figure 1 shows an overview of ContentInformed Index (CII).
4
We need to evaluate how well the weights estimated by our
proposed approach quantifies the extent to which a cited
paper informs the citing paper. To this extent, we leverage
various manually annotated datasets (explained later in
Section 4), where the annotations quantify the extent of
information in the citation. The task inherently becomes an
ordinal association, and we need to evaluate how well the
ranking imposed by our proposed method associates with
the ranking imposed by the manual annotations. As a
measure of rank correlation, we use the non-parametric Somers’
Delta
variable has only two distinct classes (binary variable), the
area under the receiver operating characteristic curve (AUC
ROC) statistic is equivalent to
We choose representative baselines from diverse categories
as discussed below:
Link-prediction approaches: The citation weights that
we estimate in this paper can also looked from the
linkprediction perspective, i.e., assigning a score to every
citation (link) in the citation graph, the score portraying the
likelihood of the existence of a link. Thus, the citations that are
noisy, i.e., the edges that do not make sense with the respect
to underlying link-prediction model get smaller weights. We
compare against two link-prediction methods, one based on
classic network embedding approach, and other belonging
to recent Graph Neural Network (GNN) based approaches.
• DeepWalk
To calculate each of the above similarity measures, we use the same pretrained representations as we used as an input to CII, and cosine similarity as the similarity measure, which is a popular similarity measure for text data. The baselines belonging to this category can also be thought of as similaritybased link prediction approaches.
In addition, we also consider another simple baseline,
referred to as Reference Frequency, where we assume that
more frequently the cited paper is referenced in the citing
paper, the higher the chances of the cited paper informing the
citing paper. This assumption has also been used as a feature
in prior supervised approaches
The S2ORC
ACL-
ACL-ARC: The ACL-ARC
citing publications are 145 and the total number of unique cited publications are 413.
SciCite
We treat one of the evaluation datasets (ACL-ARC) as the
validation set, and chose the hyperparameters of our
approaches and baselines with respect to best performance
on this dataset. For DeepWalk, we use the implementation
provided here11, with the default parameters, except the
dimensionality of the estimated representations, which is set
to 200 (for the sake of fairness, as the used 200
dimensional text representations for CII). For the models that
require learning, i.e., the logistic regression part of Deepwalk,
MLP part of Deepwalk, GraphSage, and CII, we used the
ADAM
ACL-ARC respectively. We use the same network architecture for the MLP that we train on top of DeepWalk representations. We train the logistic regression and MLP parts of Deepwalk, GraphSage, and CII for a maximum of 50 epochs, and do early-stopping if the validation performance does not improve for 5 epochs. For GraphSage, we use the implementation provided by DGL12. We used mini-batch size of 1024 for training the models.
5
Table 1 shows the performance of the various approaches on the Somers’ Delta ( ) for each of the datasets ACL-2015, ACL-ARC and SciCite. For ACL-2015 and SciCite, the proposed approach CII outperforms the competing approaches; while for the ACL-ARC dataset, CII performs at par with the best performing approach. The improvement of CII over the second best performing approach is 22% and 103%, on the ACL-2015 and SciCite datasets, respectively.
Interestingly, the simplest baseline, Reference-frequency and its normalized forms are the second best performing approaches. While Reference-frequency performs at par with the CII on the ACL-ARC dataset, it does not perform as good on the other two datasets. This can be attributed to the fact that the number of unique citing papers in ACLARC dataset are relatively small. Thus, many citations in ACL-ARC are shared by the same citing paper, which is not the case with the other two datasets. Thus, as mentioned in Section 4, absolute frequency of referencing a cited-paper may provide a good signal regarding the information borrowed from the cited paper, when comparing with other papers being cited by the same citing paper. Further, even the normalized forms of the Reference-frequency lead to only marginal increase in performance for the ACL-2015 and SciCite datasets. Thus, the simple normalizations (such as mean, max and min normalization used in this paper), are not sufficient to address the difference in citation-behavior that occurs between different papers.
12https://github.com/dmlc/dgl/blob/master/examples/pytorch/ graphsage
Furthermore, we observe that simple similarity based approaches, such as cosine-similarity between pairs of various entities (each combination of citing abstract, citing abstract, and citation-context) performs close to random scoring ( value of close to zero). This validates that the simple similarity measures, like cosine similarity are not sufficient to manifest the the information that a cited-paper lends to the citing-paper; thus, showing the necessity of more expressive approaches, like CII.
In addition, the other learning-based link-predictionbased approaches perform considerably worse than the simple baseline reference-frequency. While on ACL-2015 and SciCite datasets, they perform close to random scoring, the performance on ACL-ARC dataset is better than the random baseline.
In order to understand the patterns that the proposed approach CII learns, we look into the data instances with the highest and lowest predicted weights. As the function f takes as input both the abstract of the cited paper and the citation context, the learnt patterns can be a complex function of the cited paper abstract and the citation context. Thus, for simplicity, we limit the discussion in this section to understand the linguistic patterns in the citation context, and how these patterns associate with the weights predicted for them.
In this direction, we select 10; 000 citation-contexts corresponding to citations with highest predicted weights, and plot the word clouds for these contexts. We repeat the same exercise for the citation-contexts with the lowest predicted weights. Figures 2 and 3 shows the wordclouds for the highest weighted citations and lowest weighted citations, respectively. These figures show some clear discriminatory patters between the highest-weighted and lowest-weighted citations, that relate well with the information carried by a citation. For example, the words such as ‘used’ and ‘using’ are very frequent in the citation contexts of the highest weighted citations. This is expected, as such verbs provide a strong signal that the cited work was indeed employed by the citing paper, and hence the cited paper informed the citing work. Another interesting pattern in the highest weighted citations is the presence of words like ‘fig’, ‘figure’ and ‘table’. Such words are usually present when the authors present or describe important concepts, such as methods and results. As such, citations in these important sections indicates that the cited work is used or extended in the citing paper, which signals importance.
On the other hand, the wordcloud for the least weighted citations (Figure 3) is dominated by weasle words such as ‘may’, ‘many’, ‘however’, etc. The words such as ‘many’ commonly occur in the related work section of the paper, where the paper presents some examples of other related works to emphasize the problem that the citing paper is solving. The words like ‘may’, ‘however’, ‘but’ etc are commonly used to describe some limitation of the cited work. Such citations are expected to be incidental, carrying less information, as compared to other citations. In this paper, we presented approaches to estimate contentaware bibliometrics to accurately quantitatively measure the scholarly impact of a publication. Our distant-supervised approaches use the content of the publications to weight the edges of a citation network, where the weights quantify the extent to which the cited-publication informs the citingpublication. Experiments on the three manually annotated datasets show the advantage of using the proposed method on the competing approaches. Our work makes a step towards developing content-aware bibliometrics, and envision that the proposed method will serve as a motivation to develop other rigorous quality-related metrics.