This study aims at representing the citation based on the citation context extracted from the citation network. Researchers cite papers for various purposes to describe their arguments in a logical structure. Thus, citations have diferent roles depending on what structure they are cited in the paper. In this paper, we first present a definition of the citation context and initialize the embedding vector based on the citation order and location. Then, based on the graph transformer model, we learn contextual citation embeddings. To represent citation context, we consider the following three parts: (i) textual features of paper, (ii) positional features of the citation context, and (iii) structural features of the citation network by applying the self-attention mechanism.
The exponentially increment academic papers cause various services (e.g.,
citation recommendation [
There have been various studies [
To solve this problem, it is necessary to understand the overall context of
the citation in the paper. The content-based approaches [
Thereby, in this paper, we define and extract the citation context in citation
networks. First of all, we assume that the cited papers compose the contents of
the citing paper, and the order and location of the cited papers reflect the role of
each paper in the citing paper. To represent citations, we propose an embedding
method considering (i) textual features of paper, (ii) positional features of the
citation context, and (iii) structural features of the citation network by applying
the self-attention mechanism [
Finally, based on the graph-transformer [
This section introduces the existing methods for analyzing the citation relationship in the citation network. To deal with the large citation network, various studies investigated the co-citation frequency.
Boyack and Klavans [
The above approaches focused on comparing content-based similarities in
consideration of the relationship between cited papers. While these are efective
for application to specific tasks such as citation recommendations and searches,
it is dificult to generate widely used representation by unsupervised learning.
Thereby, a few studies conducted network representation learning [
However, it is dificult to consider the contextual features reflected in the structure of papers. From this perspectives, we extract the context of a citation through citation networks constructed according to the citation section. After that, initial embedding is performed considering the network structure and textual features so that the transformer model can learn various features of citations. In this section, we will introduce the detailed approach about the contextual citation embedding model. As illustrated in Fig. 2, the model composes three components: (1) extracting citation context, (2) initializing the citation embedding, (3) graph-transformer based encoder. Therefore, the graph transformer model learns a representation a target citation by fusing the input initial embedding vectors. To extract the context of the citation in the first component, we define our citation network as follows.
Definition 1 (Citation Network). The citation networks (N ) contains paper node (P). There are citation relationship (C ∈ R|P|×| P|) between paper nodes. When paper pi cites paper pj in the nth section, the citation relationship has weights (w ∈ {0, · · · , n, · · · , N }). This can be formulated as follows: N = ⟨P, C, w, t⟩ , (1) where t refers to a textual feature vector of P.
To consider the diferent compositions of the sections of the paper, we rearrange the paper into four sections from 0 to 3: 0 represents an introduction, 1 represents a related task, 2 represents a methodology, and 3 represents a result. In this case, the maximum section number is 3. Also, for the text features of each paper pi, diferent word embedding models can be used. 3.1
Instead of working on the entire citation network N , we extract the citation context from the citation network. The existing network embedding method uses a node sampling approach that is weighted according to the importance of the node. However, since the importance of cited papers is determined by the purpose of citations, analysis of the purpose and characteristics of each cited paper is necessary.
As stated in Sect. 1, we assume that the citation order and location of the paper relates to the purpose of the citation paper. Thus, we extract various subgraphs for the target paper by sampling the cited paper for each section rather than sampling the entire citation paper. In this section, we define the subgraphs as citation context; Definition 2 (Citation Context). Given an input citation network N , for paper pi in the network, citation context is a set of sampled paper at each section n ∈ [0, N ] This can be formulated as follows: Γ (pi) = ⟨Γ (pi,0), · · · , Γ (pi,N )⟩ , (2) where Γ (pi,n) represents the contextual citation in section n. This can be formulated as follows: Γ (pi,n) = {pj |pj ∈ P∖ {pi} ∧ w(i, j) = n}. (3)
To eficiently extract citation context for a batch of papers during the training of the embedding model, we extend a node sampling algorithm to enable node sampling for each section. The sampling method iteratively samples a list of papers for a target paper pi using adaptive sampling depth Kn by section. Let Spkin− 1 refer to the bag of papers sampled at the (k − 1)th step in nth section. For each paper node pi in Spkin− 1, we randomly sample cited papers in citation network with replacement from pi’s one-hop neighbors at the kth step. Through this process, the papers in pi’s citation context Γ (pi) can cover both local neighbors of pi and papers far away. 3.2
Based on the citation context concept, we obtain the set of sampled subgraph batches for all the nodes as get G = {g1, g2, · · · , g|P|}, where gi represents the subgraph sampled for target paper pi. Diferent from general graph in which the nodes are orderless, in the paper, cited papers are logically constructed, so the order of citations is meaningful. Therefore, the citation context is serialized in the order cited in the paper. Formally, we concatenate the target paper pi and its ordered contextual citation g1, denoted by Ipi = [pi, pi,1, pi,2, · · · , pi,S ], where pi,j is the jth node in gj , and 1 ≤ j ≤ S. In this section, we define paper embeddings along the citation order quoted in a paper. The paper embeddings will be the input to the graph-transformer model.
For textual embedding, we can embed textual feature vector tj into a shared feature space for each paper pj ∈ I(pi) in the citation context gi. Simple fully connected layers can be used for the textual input. This can be formulated as follows: xtext(pj ) = Embedding(tj ) ∈ Rd, (4) xpos(pj ) = Embedding[p(j)] ∈ Rd, where p(j) indicates the position-id of paper pj in Ii.
Our main objective is to obtain the representation of the target paper pi
based on the structural roles. To identify the role of each paper, we use the
embedding method based on Weisfeiler-Lehman (WL) algorithm [
xrole(pj ) = Embedding[r(j)] ∈ Rd, where r(t) refers to the role label.
After computing the three terms of embedding, we aggregate them to be the initial input paper embedding of the graph transformer model. The embedding fusion is formalized as follows: where d indicates dimension of the shared feature space.
The position of a paper in the citation context Ipi reflects the purpose and characteristic of the citation to the target paper pi. Thus, we suggest that the order of papers in Ipi is significant in learning citation representations. The following position-id embedding is used to identify the cited paper order information of an input list, (5) (6) (7) (9) x(pj ) = xtext(pj ) + xpos(pj ) + xrole(pj ) ∈ Rd.
We define the embedding fusion function as the summation of three embedding terms.
Finally, given a target paper pi, we obtain the initial paper embedding of each paper in its substructure cited paper set. The initial paper embedding for the paper in the citation context Ipi can be stacked to a embedding matrix. The embedding matrix is represented by X(pi) = [x(p1), x(p2), · · · , x(pS )] ∈ RS× d. 3.3
The target of the graph-transformer model is to aggregate the initial embedding of each paper and generate a low-dimensional embedding vector for each of paper. A numbers of attention layers are stacked to compose the transformer module. A single layer can be formulated as:
H(l) = attention H(l− 1) = sof tmax
Q(l)K(l)⊤ √d
V(l), (8) where H(l) and H(l− 1) denote the output embedding of the l and (l − 1) layer, Q(l), K(l), and V(l) are the query matrix, key matrix, and value matrix respectively, and d is the dimension of paper embedding. Specicfially, Q(l), K(l), and V(l) are calculated as follows: Q(l) = H(l− 1)WQ(l), K(l) = H(l− 1)WK(l), V(l) = H(l− 1)WV(l), where WQ(l), WK(l), and WV(l) are the weight matrices of the lth attention layer.
The input of the graph-transformer model H(0) is denoted as the embedding matrix of the target paper X(pi). The output of the last attention layer H(L) is defined as the output paper embedding matrix Z of the transformer model. 4
In this paper, we have proposed the learning representation of contextual citation network. We have defined the citation context by sampling a diferent number of papers per section. Using a graph transformer model, paper vectors were output based on salient citations within the citation context. According to our initial assumption, the results of the embedding model can reflect the role of each citations in the paper.
The citation purpose of a paper can change dynamically [
Acknowledgements This work was supported by Korea Foundation for Women In Science, Engineering and Technology (WISET) grant funded by the Ministry of Science and ICT(MSIT) under the team research program for female engineering students. (WISET Contract No. 2021-178)
Jeon et al.