Leveraging Multilingual Descriptions for Link Prediction: Initial Experiments

Abstract

In most Knowledge Graphs (KGs), textual descriptions of entities are provided in multiple natural languages. Additional information that is not explicitly represented in the structured part of the KG might be available in these textual descriptions. Link prediction models which make use of entity descriptions usually consider only one language. However, descriptions given in multiple languages may provide complementary information which should be taken into consideration for the tasks such as link prediction. In this poster paper, the bene ts of multilingual embeddings for incorporating multilingual entity descriptions into the task of link prediction in KGs are investigated.

-

Introduction Various Knowledge Graphs (KGs) such as DBpedia and Wikidata have been published to share linked data and have been crucial for many tasks. However, according to the Open World Assumption, KGs are never complete. Due to this fact, di erent KG completion models which map KGs to a low dimensional vector space based on the task of link prediction have been proposed. However, only some of these models such as DKRL [ 10 ], MKBE[ 7 ] , Jointly[ 11 ] , SSP [ 9 ] , and LiteralE [ 6 ] leverage the textual descriptions of entities for the link prediction task [ 4 ]. Furthermore, most popular KGs contain descriptions in two or more languages for a single entity due to the multilingual community working on these KGs (as in Wikidata) or the multilingual nature of its sources (as in DBpedia). The cultural context and bias associated with each of these descriptions induces a di erence with regards to content. However, despite the fact that entity descriptions are available in multiple natural languages, all the existing models including DKRL consider only one language. Figure 1 presents an example scenario showing the di erences in contents of multilingual descriptions of a single entity. In this example, the description in German contains information which does not appear in the English or French descriptions. For instance, the fact that the team is the record winner of the U-19 Asian Cup with twelve titles is only mentioned in the German description.

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

This paper presents part of an ongoing work which is an empirical evaluation of an already published position paper [ 3 ]. Speci cally, in this poster paper, the performance of the existing model DKRL in leveraging multilingual descriptions using multilingual embeddings has been analysed and the results of the initial experiments are discussed. "Die sudkoreanische U-20-Fu ballnationalmannschaft ist ... bei U-20Weltmeisterschaften und U-19-Asienmeisterschaften..."@de /m/05f5sr9 ("South Korea national under-20 football team") "L'equipe de Coree du Sud de football des moins de 20 ... de la Federation de Coree du Sud de football (KFA).@fr). "The Korea Republic national under-20 football team represents South Korea in international youth football competitions."@en As discussed in [ 3 ], an entity alignment model named KDCoE [ 2 ] has demonstrated the advantage of multilingual word embeddings by using a cross-lingual Bilbowa word embedding [ 5 ] to encode multilingual descriptions for the task of cross lingual learning. In this paper, the same approach is adopted to encode multilingual entity descriptions for a link prediction task on a monolingual dataset. In particular, the experiments have been performed with one of the existing models DKRL [ 10 ] using pretrained multilingual word embeddings by MUSE3. DKRL is an extension of TransE [ 1 ], which learns two kinds of vector representations for an entity, i.e., structure-based and description-based representations. DKRL adopts TransE for the structure-based representation and uses CNN to encode entity descriptions for the description-based representations. These two kinds of entity representations are learned simultaneously into the same vector space without forcing them to be uni ed. In our experiments, in order to effectively utilize multilingual descriptions, the embeddings of the words in the descriptions obtained by MUSE are passed as inputs to the encoder.

MUSE has been chosen because this paper deals with multilingual descriptions and MUSE aligns embeddings (speci cally, FastText embeddings) of words

3 https://github.com/facebookresearch/MUSE

in di erent languages into the same vector space. Figure 2 shows the CNN encoder part of DKRL with pretrained word embeddings from multilingual descriptions as inputs. In this section, the experiments conducted to incorporate textual descriptions in English, French, and German into DKRL (for the task of link prediction) are presented. Table 1 shows the dataset created out of FB15K-237 [ 8 ] for the experiments by removing those triples for which either the head or tail entity does not have descriptions in at least one of the three languages mentioned above or have less than 3 words after preprocessing. Since FB15K-237 is a dataset generated from Freebase and the entity descriptions in Freebase are old, the descriptions in all the three languages have been constructed by taking the information from the summary part of their respective Wikipedia pages. During preprocessing, stop words are removed and all phrases are marked using entity names and also by applying Spacy's4 named entity recognizer. The created dataset is available at https://github.com/ISE-FIZKarlsruhe/LinkPrediction-with-Multilingual-Entity-Descriptions.

For the experiments with DKRL and TransE, the code published at https: //github.com/xrb92/DKRL by the authors of the DKRL paper and the code

4 https://spacy.io/

for TransE available at https://github.com/thunlp/OpenKE has been used respectively. The DKRL model has been trained on three varieties of the dataset, given the names DKRLe, DKRLeg, and DKRLegf . For DKRLe, only English descriptions are used whereas for DKRLeg the combination of descriptions in German and English are used. On the other hand, descriptions in all the three languages are used to train DKRLegf . The minimum, maximum, and average number of words are 3, 615, and 107.351 for the descriptions in DKRLe, 9, 970, and 140.591 in DKRLeg, and 18, 1460, and 192.091 in DKRLegf respectively.

In DKRLe, the words are initialized using FastText pretrained embeddings and for the other two models MUSE pretrained embeddings are used. As shown in Table 2, TransE [ 1 ] has also been trained on the new dataset for fair comparison with DKRL. The hyperparameters are chosen from embedding size f50, 100, 150g, margin f1.0, 2.0, 3.0, 4.0, 5.0g, learning rate f0.01, 0.1, 1.0g (following the same procedure as in the paper of TransE). The optimal parameters for TransE on this dataset is embedding size: 100, margin: 4.0, learning rate: 0.1, and epoch: 1000. For all the other models DKRLe, DKRLeg, and DKRLegf , the same procedure as in the original study DKRL has been adopted. The optimal parameters are entity and relation embedding size: 100, learning rate 0.001, margin: 1.0, window-size: 2, dimension of feature map: 100, and word embedding size: 300, for all the three varieties. DKRLe is trained for 1000 epochs where as DKRLeg and DKRLegf are trained for 1200 epochs.

As shown in Table 2, incorporating descriptions into the link prediction task brings improvement over TransE. However, when comparing the di erent varieties of DKRL, it is seen that combining multiple descriptions has only a slight improvement. For instance, hits@10 is the same for DKRLeg and DKRLegf . One potential reason for such results could be the out of vocabulary words in the pretrained word embeddings by MUSE. There are 18.4% and 20% out of vocabulary words for DKRLeg and DKRLegf respectively and they are randomly initialized. In this paper, which is based on an already published position paper, preliminary results from an ongoing work to discuss the bene ts of leveraging multilingual embeddings for the task of link prediction are presented. As a future work, more experiments will be conducted by aligning pretrained FastText embeddings which have bigger vocabulary size, using MUSE, to avoid the problem which rises due to out of vocabulary words. Moreover, another way to improve the results will be investigated which is to learn description-based embeddings of an entity separately for each language and then fusing the vectors using di erent approaches.

References

1. Bordes , A. , Usunier , N. , Garcia-Duran , A. , Weston , J. , Yakhnenko , O. : Translating Embeddings for Modeling Multi-Relational Data . In: NIPS ( 2013 )

2. Chen , M. , Tian , Y. , Chang , K.W. , Skiena , S. , Zaniolo , C. : Co-training embeddings of knowledge graphs and entity descriptions for cross-lingual entity alignment . arXiv preprint arXiv: 1806 . 06478 ( 2018 )

3. Gesese , G.A. , Alam , M. , Sack , H.: Semantic entity enrichment by leveraging multilingual descriptions for link prediction . In: DL4KG@ ESWC ( 2020 )

4. Gesese , G.A. , Biswas , R. , Alam , M. , Sack , H.: A survey on knowledge graph embeddings with literals: Which model links better literal-ly? arXiv preprint arXiv: 1910 . 12507 ( 2019 )

5. Gouws , S. , Bengio , Y. , Corrado , G.: Bilbowa: Fast bilingual distributed representations without word alignments . In: ICML ( 2015 )

6. Kristiadi , A. , Khan , M.A. , Lukovnikov , D. , Lehmann , J. , Fischer , A. : Incorporating literals into knowledge graph embeddings . In: International Semantic Web Conference . pp. 347 { 363 . Springer ( 2019 )

7. Pezeshkpour , P. , Chen , L. , Singh , S. : Embedding multimodal relational data for knowledge base completion . In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . pp. 3208 { 3218 . Association for Computational Linguistics (Oct-Nov 2018 ), https://www.aclweb.org/anthology/D18- 1359

8. Toutanova , K. , Chen , D. : Observed versus latent features for knowledge base and text inference . In: 3rd Workshop on Continuous Vector Space Models and their Compositionality . Association for Computational Linguistics ( 2015 )

9. Xiao , H. , Huang , M. , Meng , L. , Zhu , X. : Ssp: semantic space projection for knowledge graph embedding with text descriptions . In: Thirty-First AAAI Conference on Arti cial Intelligence ( 2017 )

10. Xie , R. , Liu , Z. , Jia , J. , Luan , H. , Sun , M. : Representation learning of knowledge graphs with entity descriptions . In: AAAI ( 2016 )

11. Xu , J. , Qiu , X. , Chen , K. , Huang , X. : Knowledge graph representation with jointly structural and textual encoding . pp. 1318 { 1324 (08 2017 ). https://doi.org/10.24963/ijcai. 2017 /183