RDF2vec is an embedding technique for representing knowledge graph entities in a continuous vector space. In this paper, we investigate the efect of materializing implicit A-box axioms induced by subproperties, as well as symmetric and transitive properties. While it might be a reasonable assumption that such a materialization before computing embeddings might lead to better embeddings, we conduct a set of experiments on DBpedia which demonstrate that the materialization actually has a negative efect on the performance of RDF2vec. In our analysis, we argue that despite the huge body of work devoted on completing missing information in knowledge graphs, such missing implicit information is actually a signal, not a defect, and we show examples illustrating that assumption.
ing knowledge in a knowledge graph before computing
RDFvec [
There is a plethora of work addressing the completion ternatives for completely random walks to generate
seof knowledge graphs [
While those works exploit merely knowledge graph internal signals (e.g., by computing PageRank over the graph), other works include external signals as well. For Proceedings of the CIKM 2020 Workshops, October 19-20, 2020, Galway, Ireland email: andreea@informatik.uni-mannheim.de (A. Iana); heiko@informatik.uni-mannheim.de (H. Paulheim) orcid: 0000-0002-7248-7503 (A. Iana); 0000-0003-4386-8195 (H. Paulheim)
© 2020 Copyright for this paper by its authors. Use permitted under Creative
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
3. Experiments
example, [
A work that explores a similar direction to the one
proposed in this paper is presented in [
additional symmetry, transitivity, and inverse axioms for enriching the ontology. After inspecting the results of DL-Learner, and to avoid false T-box axioms, we used thresholds of 0.53 for symmetric properties, and 0.45 for transitive properties. Since the list of pairs of inverse properties generated by DL-Learner contained quite a few false positives (e.g., dbo:isPartOf being the inverse of dbo:countySeat as the highest scoring result), we manually filtered the top results and kept 14 T-box axioms which we rated as correct.
tion on the quality of RDF2vec embeddings, we repeat In both cases, we identify a number of inverse,
transithe experiments on entity classification and regression, tive, and symmetric properties, as shown in Table 1. The
entity relatedness and similarity and document similar- symmetric properties identified by the two approaches
ity introduced in [
With the enriched ontology, we infer additional A-box
3.1. Experiment Setup axioms on DBpedia. We use two settings, i.e., all
subproperties plus (a) all inverse, transitive, and symmetric
For our experiments, we use the 2016-10 dump of DB- properties found using mappings to Wikidata, and (b)
pedia, which was the latest oficial release during the all plus all inverse, transitive, and symmetric properties
time at which the experiments were conducted. For cre- found with DL-Learner.
ating RDF2Vec embeddings, we use KGvec2go [
Since the original DBpedia ontology provides informa- erties could also be respected. For example, from the tion about subproperties, but does not define any sym- axioms metric, transitive, and inverse properties, we first had to enrich the ontology with such axioms.
links to Wikidata [
from those reported in [
2https://github.com/IBCNServices/pyRDF2Vec 3https://github.com/andreeaiana/rdf2vec-materialization 4https://www.wikidata.org/wiki/Q21510862 5https://www.wikidata.org/wiki/Q18647515 Cerebellar_tonsil
isPartOfAnatomicalStructure Cerebellum . Cerebellum isPartOfAnatomicalStructure
Hindbrain . and the two identified T-box axioms isPartOf a owl:TransitiveProperty . isPartOfAnatomicalStructure
rdfs:subPropertyOf isPartOf . the first iteration adds
Cerebellar_tonsil isPartOf Cerebellum .
Cerebellum isPartOf Hindbrain . whereas the second iteration adds
Cerebellar_tonsil isPartOf Hindbrain .
3. Entity relatedness and entity similarity, based on the KORE50 dataset; and 4. Document similarity, based on the LP50 dataset, where the similarity of two documents is computed from the pairwise similarities of entities identified in the texts. 3.2. Training RDF2vec Embeddings
On all three graphs (Original, Enriched Wikidata, and En- evaluation is defined as follows [
The resulting sequences are then used as input to 10− 3, 10− 2, 0.1, 1, 10, 102, 103, and the best value is
choword2vec. Here, two variants exist, i.e., CBOW and Skip- sen. All other algorithms are run in their respective
stanGram (SG), where SG consistently yielded better results dard configurations. 8,9
in [
3.3. Results on Diferent Tasks
cation, with the results depicted in Tables 2 and 3. For the
Evaluation-Framework/blob/master/doc/Classification.md
9https://github.com/mariaangelapellegrino/ Evaluation-Framework/blob/master/doc/Regression.md
10https://github.com/mariaangelapellegrino/ Evaluation-Framework/blob/master/doc/EntityRelatedness.md
11https://github.com/mariaangelapellegrino/ Evaluation-Framework/blob/master/doc/DocumentSimilarity.md regression task, we can observe that the best result for 3.4. A Closer Look at the Generated each combination of a task and RDF2vec configuration Walks (depth of walks, and dimensionality) is achieved on the unmaterialized graph in 15 out of 20 cases, with linear In order to analyze the findings above, we first tried regression or KNN delivering the best results. If we con- to correlate the findings with the actual change on the sider all combinations of a task, an embedding, and a entities in the respective test sets. However, there is learner, the unmaterialized graph yields better results in no clear trend which can be identified. For example, 39 out of 60 cases. in the classification and regression cases, the dataset
The observations for classification are similar. For 19 which is most negatively impacted by materialization, out of 20 combinations of a task and an RDF2vec con- i.e., the Metacritic Albums dataset, has the lowest change ifguration, the best results are obtained on the original, in its instances’ degree (the avg. degree of the instances unmaterialized graphs, most often with an SVM. If we changes by 0.003% and 0.007% with the Wikidata and consider all combinations of a task, an embedding, and a the DL-Learner enrichment, respectively). On the other learner, the unmaterialized graph yields better results in hand, the increase in the degree of the instances on the 60 out of 80 cases. cities dataset is much stronger (1.03% and 1.04%), while
Moreover, if we look at how much the results degrade the decrease of the predictive models on that dataset is for the materialized graphs, we can observe that the vari- comparatively low. ation is much stronger for the longer walks of depth 8 We also took a closer look at the generated random than the shorter walks of depth 4. walks on the diferent graphs. To that end, we computed
The observations on the other tasks are similar. For distributions of all properties occurring in the random entity similarity, we see that better results are achieved graph walks, for both strategies and for both depths of 4 on the unmaterialized graphs in 16 out of 20 cases, and and 8, which are depicted in Fig. 1. in all of the four overall considerations. As far as entity From those figures, we can observe that the distriburelatedness is concerned, the results on the unmaterial- tion of properties in the walks extracted from the enized graphs are better in 13 out of 20 cases, as well as in riched graphs is drastically diferent from those on the all four overall considerations. It is noteworthy that only original graphs; the Pearson correlation of the distribuin three out of ten cases – enriching the IT companies tion in the enriched and original case is 0.44 in the case test set with DL-Learner and Wikidata, and enriching of walks of depth 4, and only 0.21 in the case of walks the Hollywood celebrities test set with Wikidata – the of depth 8. The property distributions among the two degree of the entities at hand changes. This hints at the enrichment strategies, on the other hand, is very simiefects (both positive and negative) being mainly caused lar, with the respective distributions exposing a Pearson by information being added to the entities connected to correlation of more than 0.99. the entities at hand (e.g., the company producing a video Another observation from the graphs is that the distrigame), which is ultimately reflected in the walks. bution is much more uneven for the walks extracted from
Finally, for document similarity, we see a diferent pic- the enriched graphs, with the most frequent properties ture. Here, the results on the unmaterialized graphs are being present in the walks at a rate of 14-18%, whereas always outperformed by those obtained on the material- the most frequent property has a rate of about 3% in the ized graphs, regardless of whether the embeddings were original walks. The three most prominent properties in computed on the shorter or longer walks. The exact rea- the enriched case – location, country, and locationcounson for this observation is not known. One observation, try – altogether occur in about 20% of the walks in the however, is that the entities in the LP50 dataset have by depth 4 setup, and even 30% of the walks in the depth 8 far the largest average degree (2,088, as opposed to only setup. This means that information related to locations 18 and 19 for the MetacriticMovies and MetacriticAlbums is over-represented in walks extracted from the enriched dataset, respectively). Due to the already pretty large de- graphs. As a consequence, the embeddings tend to focus gree, it is less likely that the materialization skews the on location-related information much more. This obserdistributions in the random walks too much, and, instead, vation might be a possible explanation for the degradaactually adds meaningful information. Another possible tion in results on the music and movies datasets being reason is that the entities in LP50 are very diverse (as more drastic than, e.g., on the cities dataset. opposed to a uniform set of cities, movies, or albums), Finally, we also looked into the correctness of the Aand that in such a diverse dataset, the efect of materi- box axioms added. To that end, we sampled 100 axioms alization is diferent, as it tends to add heterogeneous added with each of the two enrichment approaches, and rather than homogeneous information to the walks. had them manually annotated as true or false by two annotators. For the Wikidata set, the estimated precision is 65.5% (at a Cohen’s Kappa of 0.413), for the DL-Learner dataset, the estimated precision is 61.5% (at a Cohen’s Model / Dataset 500w_4d_200v 500w_4d_200v_Wikidata 500w_4d_200v_dllearner 500w_4d_500v 500w_4d_500v_Wikidata 500w_4d_500v_dllearner 500w_8d_200v 500w_8d_200v_Wikidata 500w_8d_200v_dllearner 500w_8d_500v 500w_8d_500v_Wikidata 500w_8d_500v_dllearner Model / Dataset 500w_4d_200v 500w_4d_200v_Wikidata 500w_4d_200v_dllearner 500w_4d_500v 500w_4d_500v_Wikidata 500w_4d_500v_dllearner 500w_8d_200v 500w_8d_200v_Wikidata 500w_8d_200v_dllearner 500w_8d_500v 500w_8d_500v_Wikidata 500w_8d_500v_dllearner
Results for the Document Similarity Task. w stands for number of walks, d stands for depth of walks, v stands for dimension
added to DBpedia are actually correct. Hence, we conclude that a potential addition of erroneous axioms does not explain the degradation in the downstream tasks.
Since the results show that adding missing knowledge to the knowledge graph actually results in worse RDF2vec embeddings, we want to investigate the characteristics of missing knowledge in DBpedia in general, as well as its impact on RDF2vec and other algorithms.
(b) depth=4, Wikidata 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 locatioloncationCountry country team rdf:type locationCity birthPlace groundregionServedrdfs:seeAlso (e) depth=8, Wikidata 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 locatioloncationCountry country team rdf:type birthPlace locationCity groundregionServedrdfs:seeAlso (f) depth=8, DL-Learner 4.1. Nature of Missing Information in
Knowledge Graphs
One first observation is that information in DBpedia and is contained in DBpedia, while its inverse other knowledge graphs is not missing at random. For a curated knowledge graph, a statement is contained in Nicolaus_Copernicus doctoralStudent the knowledge graph because some person deemed it Georg_Joachim_Rheticus . relevant.12 is not (since Nicolaus Copernicus is mainly known for
Consider, e.g., the relation spouse. It is unarguably other achievements). Adding the inverse statement makes symmetric, nevertheless, in DBpedia, only 9.8k spouse the random walks equally focus on the more important relations are present in both directions, whereas 18.1k statements about Nicolaus Copernicus and the ones cononly exist in one direction. Hence, the relation is noto- sidered less relevant. riously incomplete, and a knowledge graph completion The transitive property adding most axioms to the approach exploiting the symmetry of the spouse relation A-box is the isPartOf relation. For example, chains of could directly add 18.1k missing axioms. geographic containment relations are usually material
One example of a spouse relation that only exists in ized, e.g., two cities in a country being part of a region, a one direction is state, etc. ultimately also being part of that country. For Ayda_Field spouse Robbie_Williams . once, this under-emphasizes diferences between those cities by adding a statement making them more equal.
Ayda Field is mainly known for being the wife of Robbie Moreover, there usually is a direct relation (e.g., counWilliams, while Robbie Williams is mostly known as a try) expressing this in a more concise way, so that the musician. This is encoded by having the relation rep- information added is also redundant. resented in one direction, but not the other. By adding the reverse edge, we cancel out the information that the 4.2. Impact on RDF2vec and Other original statement is more important than its inverse. Algorithms
Adding inverse relations may have a similar efect. One example in our dataset is the completion of doctoral advi- RDF2vec creates random walks on the graph, and uses sors and students by exploiting the inverse relationship those to derive features. Assuming that all statements between the two. For example, the fact in the knowledge graph are there because they were considered relevant, each walk encodes a combination of statements which were considered relevant.
If missing information is added to the graph which was not considered to be relevant, there is a number of efects. First, the set of random walks encodes a mix of 12For the sake of this argument, we can also consider DBpedia a curated knowledge graph, since the source it is created from, i.e., the infoboxes in Wikipedia, is curated. A statement is contained in DBpedia if and only if somebody considers it relevant enough to be added to an infobox in Wikipedia. pieces of information which are relevant and pieces of information which are not relevant. Moreover, since the number of walks in RDF2vec is restricted by an upper bound, adding irrelevant information also lowers the likelihood of relevant information being reflected in a random walk. The later representation learning will then focus on representing relevant and irrelevant information alike, and, ultimately, creates an embedding which works worse.
The efects are not limited to RDF2vec. Translational embedding approaches are likely to expose a similar behavior, since they will include both relevant and irrelevant statements in their optimization target, which is likely to cause a worse embedding.
There are also other fields than embeddings where missing information might actually be a valuable signal.
Consider, for example, a movie recommender system which recommends movies based on actors that played in the movies. DBpedia and other similar knowledge graphs typically contain the most relevant actors for a movie.13 If we were able to complete this relation and add all actors even for minor roles, it would be likely that movie recommendations were created on major and minor roles alike – which are likely to be worse recommendations.
terialization on knowledge graph embeddings created with RDF2vec. The empirical results show that in many cases, such a materialization has a negative efect on downstream applications.
Following up on those observations, we propose a different view on knowledge graph incompleteness. While mostly seen as a defect – i.e., a knowledge graph is incomplete and hence needs to be fixed – we suggest that such an incompleteness can also be a signal. Although certain axioms could be completed by logical inference, they might have been left out intentionally, since the creators of the knowledge graph considered them less relevant.
A natural future step would be to conduct such experiments on other embedding methods as well. While there is a certain rationale that similar efects can be observed on, e.g., translational embeddings as well, empirical evidence is still outstanding.
Overall, this paper has shown and discussed a somewhat unexpected finding, i.e., that materialization an A-box can actually do harm on downstream tasks, and looked at various possible explanations for that observation.
13On average, a movie in DBpedia is connected to 3.7 actors.