Iterative Entity Alignment with Improved Neural Attribute Embedding

Abstract

Entity alignment (EA) aims to detect equivalent entities in di erent knowledge graphs (KGs), which can facilitate the integration of knowledge from multiple sources. Current EA methods usually harness KG embeddings to project entities in various KGs into the same low-dimensional space, where equivalent entities are placed close to each other. Nevertheless, most methods fail to take fully advantage of other sources of information, e.g., attribute information, and overlook the negative impact brought by lack of labelled data. To overcome these de ciencies, in this paper, we propose to generate neural attribute representation by considering both local and global signals. Besides, entity representations are re ned via an iterative training process on the neural network. We evaluate our proposal on real-life datasets against state-of-the-art methods, and the results demonstrate the e ectiveness of our solution.

Keywords

• Entity alignment Attribute information Iterative training

-

Knowledge graphs (KGs) are becoming increasingly important for many downstream applications such as question answering [ 1 ] and sentence generation [ 2 ]. A large number of KGs, e.g., YAGO and DBpedia, have been constructed. However, in reality, these KGs are far from complete. To tackle this problem, various methods have been proposed, among which KG alignment attracts growing attention since it can incorporate complementary knowledge from multiple external KGs. Unfortunately, KGs are usually built in di erent natural languages or with various ontology systems, resulting in the obstacle of integrating knowledge from external KGs to re ne the target KG. As thus, many research works have been devoted to improving the performance of KG alignment.

Current KG alignment approaches lay emphasis on entity alignment (EA), as entities are the pivots connecting di erent KGs. The task of EA aims to identify equivalent entities in di erent KGs. State-of-the-art methods [ 3, 4 ] normally harness translation-based KG embeddings to project entities and relations into a low-dimensional embedding space. The separated embedding spaces are then uni ed by harnessing seed entity pairs. Eventually given a target entity, its counterparts in other KGs can be determined in accordance to the distance in the uni ed embedding space. Nevertheless, Wang et al. [ 5 ] argued that KG embedding might fail to fully mine the structural information and instead they utilize graph convolutional network (GCN) [ 6 ] to generate entity embeddings. Additionally, they proposed to incorporate attribute information to serve as additional signals for EA. Due to the limitation of dataset, attribute names are considered instead of attribute values. Their method has also achieved superior results on existing EA benchmarks.

In [ 5 ], attribute names are represented as one-hot embeddings of the most frequent attributes. However, the most frequent attributes appear with the majority of entities and are not able to help identify a speci c entity. Additionally, the neighbourhood attribute information is completely ignored. For instance, to determine the equivalent entity of entity Michael Jordan, optional attribute hasSpouse would be more useful than obligatory attribute birthDate since every person has a birthday while not necessarily a spouse. Besides, the attributes of Michael Jordan's neighbouring entities, e.g., hasNBAChampionship for Chicago Bulls, can also be harnessed for representing Michael Jordan. Also, the shortage of labelled data (seed entity pairs) is largely overlooked by previous works, which will restrain the quality of entity embeddings, and hence, the performance of EA.

In this paper, to handle these drawbacks, we devise an iterative entity alignment method with improved neural attribute embedding, Inga, which enhances EA performance by harnessing neural network, i.e., GCN, iterative training strategy and re ned attribute information to generate entity representations. In speci c, by incorporating the neighbouring attributes of an entity (local attribute information) and the frequency of an attribute (global attribute information) to form the improved attribute feature vector, more comprehensive signals can be captured in comparison to the one-hot representation [ 5 ]. To deal with the second limitation, an iterative training strategy is utilized to train GCN, which keeps labelling unlabelled instances and select high-quality ones to retrain itself so as to generate better entity embeddings.

The main contributions of this work are: { Attribute representation is improved by considering both local and global information. { We apply an iterative training mechanism on GCN to generate more accurate structure and attribute representations. { We evaluate Inga against state-of-the-art methods on three cross-lingual EA datasets, and the results demonstrate the e ectiveness of our proposal. Related Works. The task of KG alignment can be traced back to traditional ontology matching task [ 7 ]. With the emergence and prevalence of embedding techniques, most KG alignment solutions resort to KG embedding for determining equivalent elements in di erent KGs. Chen et al. [ 3 ] (MTransE) are the rst to utilize TransE to embed entities in each KG into separated embedding spaces, which are then uni ed by di erent alignment models using seed entities pairs. The distance in the uni ed embedding space is used to determine entity pairs. JAPE [ 4 ] introduces attribute type information for re ning structure representation captured by KG embedding. GCN [ 5 ], on the other hand, harnesses GCN, instead of KG embedding, to generate entity representation. Attribute information, represented as one-hot vectors of most frequent attributes, is also utilized to complement structure information.

Methodology

Task De nition. A KG is usually represented as G = (E; R; A; V ), where E, R, A, V denotes entities, relations, attributes and attribute values respectively. Given two KGs, G1 and G2, EA aims to automatically mine new aligned entity pairs based on existing seed entity pairs S = f(ei1; ei2)jei1 2 E1; ei2 2 E2gim=1. e5 e3 e1 e4 e2

GCN

Seed pairs e 11e12 e31 e32 Structure and Attribute Embedding. Equivalent entities in multiple KGs are assumed to have similar neighbours (structure information) and attribute names (attribute information). To capture these information, GCN is utilized to operate on KGs and produce node-level embeddings for all entities. An entity's structure information can be represented by xs, as thus, the matrix encoding structure information of all entities is denoted by Xs, which is randomly initialized and updated during model training in our setting. Similarly, the attribute feature of an entity can be represented by a vector xa, and the corresponding attribute feature matrix for all entities is Xa. The initial attribute matrix is pre-computed, as detailed in the following. Note that following previous works, here we focus on attribute names, instead of attribute values.

In [ 5 ], attribute information is converted into a k-dimension one-hot vector encoding k most frequent attributes. Nonetheless, this setting fails to di erentiate entities or consider the neighboring information. In our model, the most frequent attributes (which we de ne as attributes appearing with more than 80% of entities) are discarded for representing an entity since they appear with many entities and are not discriminative. Among the rest of attributes, we select k most frequent ones as they can better distinguish entities and are not too long-tail (which might result in very sparse attribute matrix). For an entity, its attribute feature vector can be denoted by xa = [x1a; x2a; :::; xka], where xia =

ni Pk j=1 nj : (1) ni is the total times of i-th attribute appearing among the attributes of an entity and its one-hop neighbours, which is harnessed to capture local attribute information. As thus, both local and global attribute information can be encoded.

The inputs of GCN model include Xs, Xa, and adjacency matrix A. By feeding the inputs into GCN model, the output entity embedding matrix is: [Cs; Ca] = GCN (A; [Xs; Xa]); (2) where [; ] denotes the concatenation of two matrices, Cs 2 RN ds is the nal structure embedding matrix and Ca 2 RN da represents the nal attribute embedding matrix. In our model, we harness two 2-layer GCNs to generate embeddings for entities in two KGs respectively. The dimensionalities of structure and attribute feature vectors are set to ds and da for all layers in respective models.

Distance Function. A weighted distance function, which combines structure embedding and attribute embedding, is designed for entity alignment prediction. Concretely, for ei1 2 G1 and ei2 2 G2, the distance can be calculated by: Dis(ei1; ei2) =

Diss(ei1; ei2) + (1 )Disa(ei1; ei2); (3) where is a hyper-parameter balancing the importance of structure embedding distance and attribute embedding distance. The structure (attribute) embedding distance is de ned as the vector norm of cis1 cis2(cia1 cia2) divided by the dimensionality ds(da). The distance between equivalent entities is expected to be as small as possible. As thus, the entity in G2 with the smallest distance from a speci c entity ei1 2 G1 can be regraded as the counterpart of ei1. Loss Function. We use pre-aligned entity pairs S to train GCN models. The training objectives for learning structure embedding and attribute embedding are to minimize the following margin-based ranking loss functions, Js = Ja =

X

X (e1;e2)2S (v1;v2)2S (e1;e2)2S (v1;v2)2S

X X ds [Diss(e1; e2)

Diss(v1; v2) + s]+; da [Disa(e1; e2)

Disa(v1; v2) + a]+; (4) (5) where [x]+ = maxf0; xg, S denotes the set of negative aligned entity pairs; s and a are two positive margins separating positive and negative aligned entity pairs. Loss functions Js and Ja are optimized by stochastic gradient descent (SGD) separately.

Iterative Training. Considering the lack of labelled data, inspired by [ 8 ], we adopt semi-supervised training strategy to enlarge the training set iteratively by including aligned pairs with high con dence during training process.

Once newly-aligned entity pairs are added into S, they are considered as valid training data. However, some false positive pairs may be included, which will hurt the following training process. Consequently, the key challenge is how to choose highly con dent samples from newly-aligned entity pairs to enlarge S. In consequence, we consider candidate entity pairs f(ei1; ej2)jei1 2 G1nS1; ej2 2 G2nS2g, which satisfy ei1 = arg min Dis( ; ej2); ej2 = arg min Dis(ei1; ), as reliable aligned pairs for iterative training, where S1 and S2 are the set of pre-aligned entities in G1 and G2 respectively.

Experiment

Datasets. We adopt the widely used DBP15K datasets in the experiments, which were developed by [ 4 ]. The datasets were constructed from subsets of DBpedia, which has multiple versions in di erent languages. DBP15K consists of three datasets, Chinese-English (Zh-En), Japanese-English (Ja-En), and FrenchEnglish (Fr-En). In each dataset, there are 15 thousand already-known equivalent entity pairs, 30% of which are used for training and 70% of which are for testing. Parameter Settings. In our GCN models, the dimensionality of structure embedding and attribute embedding in all layers were set to ds = 300 and da = 600 respectively. The number of top attributes k is set to 1000. The iterative training processing would not stop until the size of the newly-included set jCj is under a threshold = 100. The margins s and a are set to 3. The hyperparameter in weighted distance function is set to 0.9.

Competing Approaches and Evaluation Metric. Three approaches are utilized for comparison, including MTransE [ 3 ], JAPE [ 4 ], and GCN [ 5 ]. The evaluation metric, Hits@k, measures the proportion of correctly aligned entities in top k ranked candidates. We report the results of Hits@1 (accuracy), Hits@10, and Hits@50 in the experiment. Experiment Results. The experimental results of Inga and three competitors on DBP15K datasets are shown in Table 1. It can be easily observed that MTransE 30.83

JAPE 41.18 GCN 41.25 Inga 50.45 79.12 88.9 86.23 89.79

Zh

En Inga achieves the best performance among most settings on three bi-directional datasets.

Among the four approaches, MTransE achieves relatively worse results. The Hits@1 values of MTransE on all datasets are between 20% and 30%, indicating that translation-based KG embeddings can capture structure information and serve as useful signals for EA. Another KG embedding based method, JAPE, outperforms MTransE signi cantly by over 10% in most cases due to its ability to incorporate attribute information for re ning entity structure embeddings. GCN attains slightly better results than JAPE on J a En and F r En language pairs, indicating the e ectiveness of GCN model for generating structure representation. Inga is built on the architecture of GCN, whereas it improves the results by a large margin. In both alignment directions, Inga outperforms GCN and JAPE by about 3% 12% regarding all Hits@k metrics. This demonstrates the usefulness of the improved attribute feature representation and iterative training strategy.

Noteworthy is that the gap between Inga and the rest approaches is much larger on Hits@1 (accuracy) than other metrics. This reveals that Inga can align more accurate entity pairs, which is critical to EA task.

4 Conclusion

In this paper, we propose a GCN-based model to align entities in di erent KGs by projecting entities into a uni ed embedding space, where equivalent entities are placed close to each other. Attribute representation is improved by capturing more informative attribute features. Furthermore, we devise an iterative training strategy to enlarge training set and generate better entity embeddings via neural network. Our proposal is then evaluated on real-life datasets and the results demonstrate that our model outperforms three state-of-the-art competitors by a large margin. For further work, to take more information especially attribute values as guidance for our model is also necessary.

Acknowledgements. This work was partially supported by NSFC under grants Nos. 61872446, 61876193 and 71690233.

References

References

1. J. Yin , X. Jiang , Z. Lu , L. Shang , H. Li , and X. Li . Neural generative question answering . In Proceedings of IJCAI , pages 2972 { 2978 , 2016 .

2. B. D. Trisedya , J. Qi , R. Zhang , and W. Wang . GTR-LSTM: A triple encoder for sentence generation from RDF data . In Proceedings of ACL , pages 1627 { 1637 , 2018 .

3. M. Chen , Y. Tian , M. Yang , and C. Zaniolo . Multilingual knowledge graph embeddings for cross-lingual knowledge alignment . In Proceedings of IJCAI , pages 1511 { 1517 , 2017 .

4. Z. Sun , W. Hu , and C. Li . Cross-lingual entity alignment via joint attributepreserving embedding . In Proceedings of ISWC , Part I , pages 628 { 644 , 2017 .

5. Z. Wang , Q. Lv , X. Lan , and Y. Zhang. Cross-lingual knowledge graph alignment via graph convolutional networks . In Proceedings of EMNLP , pages 349 { 357 , 2018 .

6. T. N. Kipf and M. Welling . Semi-supervised classi cation with graph convolutional networks . CoRR, abs/1609.02907 , 2016 .

7. F. M. Suchanek , S. Abiteboul , and P. Senellart . PARIS: probabilistic alignment of relations, instances, and schema . PVLDB , 5 ( 3 ): 157 { 168 , 2011 .

8. H. Zhu , R. Xie , Z. Liu , and M. Sun . Iterative entity alignment via joint knowledge embeddings . In Proceedings of IJCAI , pages 4258 { 4264 , 2017 .