Knowledge graph completion (KGC) aims to discover missing relations of query entities. Current textbased models utilize the entity name and description to infer the tail entity given the head entity and a particular relation. Existing approaches also consider the neighborhood of the head entity. However, these methods tend to model the neighborhood using a flat structure and are only restricted to 1-hop neighbors. In this work, we propose a node neighborhood-enhanced framework for knowledge graph completion. It models the head entity neighborhood from multiple hops using graph neural networks to enrich the head node information. Moreover, we introduce an additional edge link prediction task to improve KGC. Evaluation on two public datasets shows that this framework is simple yet efective. The case study also shows that the model is able to predict explainable predictions.
[
Most existing work typically models the head entity ℎ and the query relation together, then focuses on the modeling for predicting the relation for the tuple (ℎ, , ?) to find the correct tail entity . The modeling for (ℎ, ) tends only to consider the head entity name or descriptions and the query relation name. Besides, existing approaches also consider the neighborhood of the head entity [13, 14]. However, these methods tend to model the neighborhood using a flat structure so that a simple attention-based method can be applied; or they are only restricted to 1-hop neighbors for computational consideration. In this work, we propose considering the neighborhood from multiple hops to improve the richness of the head entity information for further KGC predictions. By doing so, the head entity contains much explicit information that may help the relation prediction, especially when the head entity name/description is short and not informative. We model the neighborhood using a graph neural network, making it easy to propagate information within multiple hops away. Besides, to further enhance the correlation with these neighbors and the tail entity, we add an extra link prediction loss to the KGC task, and we show this could improve the KGC performance.
Our main contribution of this work is to propose a node neighborhood-enhanced model for knowledge graph completion by modeling the neighbors with graph neural networks [15]. Moreover, the experiment shows that this framework is efective in two public datasets. Additional case studies show that this model could provide explainable predictions. We release our code at https://github.com/IreneZihuiLi/NNKGC.
We propose the Node Neighborhood-enhanced model for knowledge Graph Completion (NNKGC), which consists of the node encoder and the relation prediction module, as shown in given an incomplete knowledge graph , we need to find the tail entity by providing the head entity ℎ and the query relation : (ℎ, , ?) .
Node Encoder Typically, we model the head and tail entity nodes separately. In our case, each entity node contains a node name in a short phrase or a keyword (name), and a node description in a sentence (desc). The relation is usually represented by a phrase or a keyword (rel). To incorporate the text features into the knowledge graph, we utilize the pre-trained language models (PLMs), BERT [16], to encode the head entity and relations. Each entity node is represented by: = BERT[, , [SEP], ] To further consider the head node neighborhood to enrich the encoding, we apply a graph neural network [17] to model the neighbors. We first collect all the neighbors within -hop as a list of nodes: 1, 2, ... . Then we encode the head entity and relation as follows: = [ ℎ , 1 , 2
, ... ], ℎ = GNN( , ),
In which, denotes the node embeddings of the entities within the neighborhood, including the head node, as well as the neighbors, and is the adjacency matrix indicating the directed (1) (2) (3) Head
Node Head Node Neighborhood
Tail Node Graph Encoder
Node Encoder
Head Node Neighborhood Embedding Tail Node Embedding
Cosine Similarity
connections among the entities. We define the neighborhood of a head node to be the connected entities, including incoming and outgoing connections. In Fig 1, we illustrate 1-hop (blue) and 2-hop (grey) neighbors and ignore outgoing neighbors for simplicity. There is a number of choices for the graph encoder, we investigate three types: graph convolutional network (GCN) [18], graph attention network (GAT) [19], and GraphSAGE [20].
Relation Prediction To make the prediction on the tail entities, we followed the recent
SimKGC [
Neighborhood Edge Prediction To further strengthen the efect of neighborhood entities for the KGC task, we ask the model to predict neighborhood edges (i.e., the highlighted edge in the figure). In other words, we randomly mask some edges in matrix , and let the model reconstruct these masked relations. There are multiple ways to achieve this, and we apply a variational graph autoencoder (VGAE) [21] to conduct the missing link prediction. It is shown to be efective for several NLP tasks [ 21, 22]. Specifically, we apply vanilla GCNs to model the mean and standard deviation of the node features, then conduct dot-product to predict the existence of a given node pair. Note that this is only done for the neighborhood. It will generate a loss on the reconstructed edges ℒ . So the final loss becomes: ℒ = ℒ + (1 − )ℒ (4)
Here, is an empirical constant to control the efect of the edge prediction, and it ranges between 0 and 1.
Shown in Tab. 1, we conduct knowledge graph completion on two public datasets: FB15k-237
[23], and WN18RR [24]. We follow the previous work [
Hyperparameters The BERT encoder is the pretrained bert-base-uncased model.1 AdamW optimizer [25] was applied for training. For modeling the neighborhood, we typically apply two layers of neural networks, and the dimension for the graph node representation is 768. We set the number of attention heads for the GAT encoder to 3. We set the to 0.2. We conduct experiments on 4 A100 GPUs (with 40GB memory). Training on WN18RR with 30 epochs took about 3 hours; training on FB15k237 with 5 epochs took less than one hour. More experimental hyperparameters can be found in the code URL.
FB15k-237
MRR MRR
MTL-KGC [26]
NNKGC WN18RR
MTL-KGC [26]
NNKGC
Main Results We compare with two embedding-based methods: TransE [
GCN
and SimKGC [
Neighborhood Encoder We then study diferent neighborhood graph encoders in Eq. 3. We compare three widely-used graph models: GCN, GAT, and GraphSAGE. In Fig. 2, we report the average performance on the two datasets. We could observe that GraphSAGE and GAT are slightly better than the vanilla GCN encoder.
WN
FB
We provide a detailed evaluation of various graph encoders for modeling the head node neighborhood, shown in Tab. 4 and 3. In general, the GCN encoder is worse than the other two. In general, GraphSAGE shows better stability in both datasets. 23.8 25.2 25.1 35.4 36.5 36.2 50.6 51.5 51.2
Neighborhood Hops As we focus on the neighborhood, we now study the efect of the neighborhood scale. We conduct experiments on 1-hop, 2-hop and 3-top neighbors to the head entity. Shown in Fig. 3, we compare the evaluation metrics as well as the average performance (Avg. on the WN18RR dataset.
We also show the bar chart of how the number of hops afects the performance for FB15k-237 in Fig. 4. One may observe a similar trend to that of WN18RR. As more hops will bring more neighboring entities, some noises may be added, which may make the performance worse. 65.4 65.8 67.4 57.0 56.4 59.6
Head node description: Star Wars, later retitled Star Wars Episode IV: A New Hope, is a 1977 American epic space opera film written and directed by George Lucas...
Description: The Academy Awards are the oldest awards ceremony for achievements in motion pictures. The Academy Award for Best Production Design recognizes achievement in art direction on a film. The category’s original name was Best Art Direction, but was changed to its current name in 2012 for the 85th Academy Awards.
Neighbor 2, Name: Ben Burtt Description: Benjamin Ben Burtt, Jr. is an American sound designer , film editor, director, screenwriter, and voice actor. He has worked as sound designer on various films including: the Star Wars and Indiana Jones film series...
Neighbor 3, Name: John Williams Description: John Towner Williams is an American composer , conductor and pianist... Node relation: music film
In general, integrating more neighbors is not contributing to the performance. Especially, H@1 drops significantly when the hop number increases. A possible reason might be that more neighbors may introduce more noises which confuses the knowledge graph completion task. Moreover, doing so brings more computational burden to the model as more nodes participate in the graph modeling part. We did not conduct experiments on 4 and more hops, as the total training time is incredibly long.
Case Study We select an example from FB15k-237 in Tab. 5, and study how the neighbor entities and corresponding relations help with the KGC task. Given the head entity to be the
Head node description: any monetary aid Relation: hypernym
Description: a grant to a person or school for some educational project Node relation: hypernym Neighbor 2, Name: grant (VB2) Description: give as judged due or on the basis of merit; the referee awarded a free kick to the team; the jury awarded a million dollars to the plaintif; funds are granted to qualified researchers .
Node relation: derivationally related form movie Star Wars IV, and the relation to be nominated for (an award), our model predicts a ranked possible tail entity list. The entity Academy Award for Best Sound Mixing achieves the highest score, which is precisely the ground truth. We also randomly select three 1-hop neighbors, and their relations with the head entity in the training set. As we can see, the first neighbor is one of the Academy Award categories, and the relation is the same as the query relation. The second neighbor is a sound designer. The description shows that this person worked on the
Head node description: John Grifith Jack London was an American author, journalist, and social activist. He was a pioneer in the then-burgeoning world of commercial magazine fiction and was one of the first fiction writers to obtain worldwide celebrity and a large fortune from his fiction alone...
Neighbor 1, Name: San Francisco Description: San Francisco, oficially the City and County of San Francisco, is the leading financial and cultural center of Northern California and the San Francisco Bay Area...
Neighbor 2, Name: Oakland Description: Oakland, located in the U.S. state of California , is a major West Coast port city and the busiest port for San Francisco Bay and all of Northern California. It is the third largest city in the San Francisco Bay Area, the eighth-largest city in the state, and the 47th-largest city
Description: The University of California, Berkeley, is a public research university located in Berkeley, California, United States . The university occupies 1,232 acres on the eastern side of the San Francisco Bay with the central campus resting on 178 acres. Berkeley is the flagship institution of the 10 campus University of California
Star Wars series. The third neighbor is a composer who participates in the music production for the movie. We highlight some potential triggering tokens in the table. As we can see, some tokens are highly related to the ground truth tail entity, i.e., neighbor 1, the name Academy Award. Besides, the token composer and music are also semantically related to the ground truth tail entity name Best Sound Mixing. From this case study, one can notice that the neighborhood has some positive efects on the KGC task, which is consistent with our motivation. Moreover, we show that our framework has the potential to explain the predictions, which is an essential step for further fact verification.
We present two more case studies. Tab. 6 shows a random example from WN18RR. As we can see, the entity description tends to be shorter, but we can still find text chunks that are highly related to the ground truth tail entity (financial aid and funds). Similarly, Tab. 7 gives an example of finding a person’s nationality. We can observe that the neighbors are some relevant locations about this head entity person (i.e., lives in Oakland, U.S. state of California), which contributes to the final prediction on the tail entity ( United States of America).
In this work, we proposed a node neighborhood-enhanced model for knowledge graph completion, by modeling the neighbors with graph neural networks. We showed that the framework is simple but efective. Case studies also show that it is possible to provide explainable results. [13] J. Zhuo, Q. Zhu, Y. Yue, Y. Zhao, W. Han, A neighborhood-attention fine-grained entity typing for knowledge graph completion, Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (2022). [14] A. Borrego, D. Ayala, I. Hernández, C. R. Rivero, D. Ruiz, Cafe: Knowledge graph completion using neighborhood-aware features, Eng. Appl. Artif. Intell. 103 (2021) 104302. [15] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, P. S. Yu, A comprehensive survey on graph neural networks, IEEE Transactions on Neural Networks and Learning Systems 32 (2019) 4–24. [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/ N19-1423. doi:10.18653/v1/N19-1423. [17] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural network model, in: IEEE International Joint Conference on Neural Networks, IEEE, 2009, pp. 2187–2192. [18] T. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks,
ArXiv abs/1609.02907 (2016). [19] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio’, Y. Bengio, Graph attention networks, ArXiv abs/1710.10903 (2017). [20] W. L. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, in: NIPS, 2017. [21] I. Li, V. Yan, T. Li, R. Qu, D. Radev, Unsupervised cross-domain prerequisite chain learning using variational graph autoencoders, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2021, pp. 1005–1011. [22] Q. Xie, J. Huang, P. Du, M. Peng, J. Nie, Inductive topic variational graph auto-encoder for text classification, in: North American Chapter of the Association for Computational Linguistics, 2021. [23] K. Toutanova, D. Chen, P. Pantel, H. Poon, P. Choudhury, M. Gamon, Representing text for joint embedding of text and knowledge bases, in: Conference on Empirical Methods in Natural Language Processing, 2015. [24] T. Dettmers, P. Minervini, P. Stenetorp, S. Riedel, Convolutional 2d knowledge graph embeddings, in: AAAI Conference on Artificial Intelligence, 2017. [25] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2017. [26] B. Kim, T. Hong, Y. Ko, J. Seo, Multi-task learning for knowledge graph completion with pre-trained language models, in: International Conference on Computational Linguistics, 2020.