We propose a novel approach to enrich Wikidata with the textual content of Wikipedia. Speci cally, we leverage knowledge graph (KG) embedding models to classify the hyperlinks between Wikipedia articles and predict the corresponding facts. For instance, we would like to complete the triple (Berlin, *, Germany) with the relation capital of, given a hyperlink from Berlin to Germany in Wikipedia. While existing KG embedding models can be used for this task of relation prediction, they were not explicitly designed for it and their performance is not satisfactory. In this paper, we propose two methods that greatly improve the performance of these models on this task: rst, a new negative sampling method that balances the roles of entities and relations during training; second, a method to exploit the types of entities in the selection of candidate relations. We obtain accuracy scores as high as 94% on the popular FB15k237 dataset and 75% on WDV5, an extraction of Wikidata. The e ciency of the approach is illustrated on some Wikipedia pages, where new facts unknown to Wikidata are predicted by our method.
In the recent years, Wikipedia has become the largest open-source collection of knowledge. Its textual content is however mostly unstructured, the structured information being mainly limited to the content of infoboxes (e.g., place and date of birth for articles on humans). The hyperlinks make another structure which is not fully integrated in Wikidata yet. The main challenge is that, in order to know the meaning of an hyperlink, an agent needs to read the text in which the hyperlink is embedded. While some hyperlinks do not correspond to relevant facts, we claim that this is a rich source of information to complete Wikidata. To illustrate this, one can look at the level 5 of Wikipedia vital articles1, that is about 40,000 pages serving as a centralized watchlist to track the quality of the most important articles. These Wikipedia pages are linked by slightly more than 3 million hyperlinks, which is far more than the approximately 200,000 facts linking the corresponding entities in Wikidata. For instance, there is a link from the page Henri Poincare to the page Optics in Wikipedia. This suggests the existence of a relation linking the two entities, here eld of work. This fact is not present in Wikidata.
Formally, a KG consists of a set of vertices called entities (e.g., person, place, date, concept) linked by directed edges in the form of triples (h; r; t) where h (resp. t) is the head (resp. the tail) entity and r is a relation carrying the semantic nature of the edge. When a triple is known to be true, it is called a fact.
In this paper, we address the issue of relation prediction: nding the relation linking some given head and tail entities. For instance, we would like to complete the triple (Berlin, *, Germany ) with the relation capital of, assuming the fact is not in the KG. This task is also known as the semanti cation of a link. For this, we leverage the embedding of the entities and relations of the KG to compute scores on possible triples. Though most existing works on embeddings have focused on the task of link prediction, that is, completing either the triple (*, capital of, Germany ) (head prediction) or (Berlin, capital of, *) (tail prediction), we show that embeddings for relation prediction can also perform notably well. We propose two techniques for that: rst, we adapt the training of the models by balancing the role of entities and relations in the negative sampling step and then we use the types of entities to lter candidate relations.
These techniques prove very e cient, allowing a simple embedding model
like TransE [
Contributions. The main contributions of this work are the following: An approach to enrich Wikidata by the semanti cation of the hyperlinks of Wikipedia.
A novel negative sampling technique for improving the ability of KG embedding models to predict relations, without a ecting their performance on link prediction.
A novel ltering technique for relation prediction where candidate relations are selected through the types of the head and tail entities.
A new dataset, WDV5, consisting of the facts between entities of Wikidata corresponding to the level 5 of Wikipedia vital articles2. 2
KG embedding. A KG embedding model is de ned as a function f that computes a score for any triple (h; r; t) using some vector representations of h; t and 2 https://en.wikipedia.org/wiki/Wikipedia:Vital articles/Level/5 r. By extension, the vectors representing entities and relations are called embeddings. KG embeddings have been speci cally designed for link prediction3: given an entity h (resp. t) and a relation r, the model is used to predict an entity t (resp. h) so that the fact (h; r; t) is the most likely to be true. This prediction is done by selecting the entity giving the highest score among all entities.
There are three categories of models depending on the form of the scoring
function f and thus on the way entities and relations interact in the vector space
(see [
Linear models, where h; r; t are linked by a linear relation in the vector space.
Some projections can be added to increase the expressiveness of the model.
Examples of such models include TransE [
Bilinear models, where the relation r is a bilinear form of h and t in the
vector space. Examples include RESCAL [
Given some known fact (h; r; t), the usual way to create a false statement
from it (under the CWA) is to randomly choose either the head entity or the
tail entity and to replace it with another random entity of the KG [
Type Filtering. Most KGs assign one or several type(s) to each entity through
a rdf:type relation (e.g., the P31: \instance of " relation in Wikidata). The
types of entities have mainly been used in link prediction, either to enforce type
constraints in negative sampling or to select the candidate entities [
We make a clear distinction between link prediction (head or tail entity unknown) and relation prediction (relation unknown).
KBs can also enforce type constraints on relations via rdfs:domain and rdfs:range constraints. In relation prediction, selecting candidate relations with these constraints seems natural but they can be missing or too coarse grained making the ltering either too restrictive or with no e ect. In Wikidata, relation constraints are hints for the editors, not rm restrictions4. We propose a method to infer such constraints simply from the rdf:type relation of the KG at hand and use the resulting constraints in relation prediction. We show that it has a major impact on performance.
NLP for relation prediction There are two main tasks tackled by NLP
methods. The rst is relation prediction, also known as relation extraction, consisting
in predicting the semantic relation linking two entities using sentences describing
these entities. The best performing models rely on deep neural networks with
attention mechanisms [
Hyperlink semanti cation. Very few works exist on the semanti cation of
Wikipedia hyperlinks using the graph structure of the KG only. The approach
of Galarraga et al. [
Bernoulli Negative Sampling. The usual negative sampling technique (noted
BerNS) relies on relation-speci c Bernoulli distributions to choose between the
head or the tail which entity of a fact should be replaced to maximize the
probability of the resulting triple to be false [
As an example, consider \author of " which is a one-to-many relation (one author and many potential books). In that case, the head entity (an author) 4 https://www.wikidata.org/wiki/Help:Property constraints portal should be more likely replaced than the tail entity (a book), yielding a false statement with greater probability.
Model training. Training a model comes down to nding its parameters (the
embeddings) so that the scoring function gives high scores to facts and low
scores to false statements. Given a training set of facts denoted (h; r; t), the
corresponding false statements (h0; r; t0) are generated by NS. Then for each pair
of facts (h; r; t) and (h0; r; t0), a loss measuring the gap between the corresponding
scores is computed, `(f (h; r; t); f (h0; r; t0)). This loss ` should be high for close
scores. Examples include the logistic loss and the margin loss [
4.1
Our approach relies on the following techniques.
KG Embedding. KG embedding models can be used for relation prediction the same way they are used in the aforementioned link prediction: given two entities h and t, an embedding model and its scoring function f , the relations of the graph can be ranked by decreasing order of scores: f (h; r1; t) > f (h; r2; t) > > f (h; rk; t). The relation r1 is then predicted, corresponding to the fact (h; r1; t). Note that this method applies to the case of undirected links, by ranking the scores of the predictions for both directed links (h; t) and (t; h). This is especially useful when some relations have no reciprocal.
Balanced Negative Sampling. Simple experiments show that o -the-shelf linear models like TransE perform really badly in relation prediction (3% of Hit@1 on FB15k237, see Table 2a). This suggests that the representation of relations is not as good as that of entities. It turns out that entities and relations play similar roles in the training procedure except for the NS step. Usually, only the entities are randomly replaced to get false statements (see Section 3). We propose a simple modi cation of BerNS to balance the roles of entities and relations during training. Rather than just replacing one of the two entities of a known fact, we replace the relation with some probability p, and apply BerNS otherwise (See Algorithm 1). This new method is called Balanced Negative Sampling (BalNS). The default value for p is set to 12 . Experiments have shown that the value of the parameter has no major impact on the performances of the approach as long as it is greater than 0.1. Algorithm 1: Balanced Negative Sampling (BalNS).
Input: (h; r; t), a fact Input: p, probability to replace the relation Output: (h0; r0; t0), a false statement Data: T , the facts in the KG
Data: pr, Bernoulli parameter for relation r 1 (h0; r0; t0) (h; r; t) 2 while (h0; r0; t0) 2 T do 3 u uniform random variable on [0; 1] 4 if u < p then 5 r0 random relation Type Filtering for relation prediction. Another key technique to improve the quality of relation prediction is through Type Filtering (TF). An entity e is said to have the type t if the fact (e, rdf:type, t ) is known. To predict the relation linking h and t, only relations that are known to link entities of the type(s) of h to entities of the type(s) of t should be considered. Formally, we say that a relation r links type a to type b if there exists some known fact (h; r; t) with the head entity h of type a and the tail entity t of type b. Now for predicting the relation missing in (h; ; t), we propose to consider as candidates only the relations r linking any type of h to any type of t. The corresponding algorithm for relation prediction is described in Algorithm 2. Observe that if either the head entity h and/or the tail entity t is not typed, the candidate relations are then the relations that are involved in a training fact with either h as a head entity or t as a tail entity. In the end, if no relation meet any constraint, there is no ltering, i.e., all relations are selected. Regarding speed, this step has no signi cant impact on the global computation time with proper index: linking entities to their types and types to possible relations. 5 else 6
R 7 if jRj = 0 then 8 R
R 9 r arg max(ff (h; r; t); r 2 Rg) 10 return r
Algorithm 2: Relation prediction with Type Filtering (TF).
Input: h; t, entities Input: f , scoring function Output: r, relation linking h to t Data: T , the facts in the KG
Data: R, the relations in the KG 1 A types of h 2 B types of t 3 if jAj > 0 and jBj > 0 then 4 R fr : 8(a; b) 2 A B; 9h0; t0 : type(h0) = a; type(t0) = b; (h0; r; t0) 2 T g fr 2 R : 9e : (h; r; e) 2 T g [ fr 2 R : 9e : (e; r; t) 2 T g To summarize, our approach relies on the following steps: 1. Training the model (e.g., TransE or ComplEx) with BalNS (Algorithm 1). 2. Predicting relations with TF (Algorithm 2). 4.2
Given an embedding model trained with BalNS and some known fact (h; r; t) of a test set, all relations selected by TF are ranked by decreasing score. The rank of the true relation r is recorded as the recovery rank (if the true relation r is not selected by TF, the rank is set to the maximum). Usual metrics of link prediction like Mean Reciprocal Rank (MRR: average of the inverses of the recovery ranks) and Hit@k (proportion of tests in which the recovery rank is at most k) can then be reported. In the ltered setting, any relation that is ranked better than r and that is known to lead to a fact (i.e., in the training set) is discarded, so that the model is not penalized for predicting known facts that are simply more likely than the target one. 5
The experiments aim at assessing the performance of our approach on existing KGs and at showing its practical interest on a real-world task, i.e., the semanti cation of Wikipedia hyperlinks. All experiments can be reproduced using the publicly available code5 and data 6. 5 https://gitlab.telecom-paris.fr/aboschin/hyperlinks-semanti cation 6 https://netset.telecom-paris.fr/pages/wikivitals+.html Datasets The datasets used in the experiments are shown in Table 1.
Dataset FB15k237 WDV5 Wikivitals+
Entities/nodes Facts/edges Relations Types Typed entities 14,541 310,116 237 73 2,719 39,062 231,744 607 1,206 22,883 39,062 3,008,116
One of the most common datasets used to evaluate the quality of KG
embeddings is a subset of Freebase called FB15k237 [
We also introduce WDV5, a new dataset containing the facts linking entities of Wikidata corresponding to the level 5 of Wikipedia vital articles (see Section 1). To type entities, only typing facts included in the dataset are used (i.e., all types are entities of WDV5). In particular, not all entities are typed (only 56%, see Table 1). It is important to note that WDV5 is a raw extract from Wikidata, without any pre-processing. As such, we expect the corresponding experiments to be more representative of real use cases than those based FB15k237.
For the semanti cation of Wikipedia hyperlinks, we use Wikivitals+, an extraction of the level 5 of Wikipedia vital articles and the hyperlinks between them. We only keep the pages that have a corresponding Wikidata entity. This dataset provides many hyperlinks that are natural candidates for true facts, after relation prediction.
Baseline In order to measure the impact of using a KG embedding model for ranking the candidates selected by TF, we compare our approach to a simple baseline that ranks the candidate relations by popularity in the training set, in number of facts.
Process Two o -the-shelf embedding models were chosen for the experiments:
TransE [
The models were trained using the Adam algorithm for optimization [
In the case of FB15k237, the split between train, validation and test sets is
set by Toutanova et al. [
The results for relation prediction are shown in Table 2 with metrics computed in a ltered setting, for di erent variants of the model so as to assess the respective gains of the proposed techniques:
Original: The base model (either TransE or ComplEx) trained with BerNS. BalNS: The base model trained with BalNS.
TF: The base model evaluated with Type Filtering (TF).
BalNS & TF: The base model trained with BalNS and evaluated with TF.
The original version of TransE is not e cient on FB15k237 (only 3% of
Hit@1). ComplEx performs however notably well on the same dataset (89% of
Hit@1). It seems less sensitive to the unbalanced role of entities and relations
during training. We suspect however that the score of ComplEx on FB15k237
mainly results from over- tting due to lack of new datasets in the KG embedding
literature over the past few years and over-engineering of FB15k237 (it is the
second version of the subset). This has already been argued in [
Balanced Negative Sampling. Training with BalNS has a strong impact on the relation prediction performance of the models: training TransE on FB15k237 with BalNS rather than BerNS increases the Hit@1 from 3% to 91%. This conrms the intuition that the relation embeddings were not well trained. The difference is less impressive for ComplEx on FB15k237 but the original ComplEx model performs already quite well on this dataset. On WDV5, there is a big increase in Hit@1 for both models: 24% for TransE and 28% for ComplEx. Type ltering. TF has a strong impact on the performance of the models. Looking at Hit@1 on WDV5, TransE goes from 45% to 58% and ComplEx goes from 45% to 76%. Note that Type Filtering alone (the baseline) performs almost
Base model Variant TransE ComplEx
Original 0.061 0.033 0.049 BalNS 0.940 0.914 0.972 TF 0.405 0.184 0.744 BalNS & TF 0.957 0.935 0.983 Original 0.928 0.894 0.967 BalNS 0.956 0.934 0.982 TF 0.953 0.927 0.983
BalNS & TF 0.961 0.943 0.983 Baseline TransE ComplEx
Original 0.556 BalNS 0.779 TF 0.711 BalNS & TF 0.821 Original 0.546 BalNS 0.816 TF 0.826
BalNS & TF 0.827 Baseline 0.516 as well as the original embedding models. It is however largely beaten by the combination of TF with scoring by an embedding model. The gain of using and embedding model is very important.
Complete model. The combination of BalNS and TF gives the best results. On FB15k237, the increase in performance of TransE is impressive (Hit@1 from 3% to 94%) and makes this model almost as e cient as ComplEx. This is obtained through additional facts imported from Wikidata for TF but the scores of TransE simply trained with BalNS (and without TF) are already close to those of ComplEx.
On WDV5, all performance metrics are signi cantly improved by our approach. Both models that perform similarly in their original form remain close. On average, ComplEx beats TransE by 1% in Hit@1 but the scores of TransE are much more stable from one split to the other, as shown by the lower standard deviation. The intervals of uctuation of MRR and Hit@1 tend to be reduced if the model is trained with BalNS. This is particularly true for TransE, whose standard deviation for each metric is very small.
It is remarkable to get almost identical performance with TransE and ComplEx, knowing that TransE has half the number of parameters of ComplEx, is more geometrically intuitive and requires 6 times less operations for each gradient descent step during training. 6.2
In order to predict the relation associated to a hyperlink, we use the TransE embedding of WDV5 trained with BalNS and applied using TF. When two pages are linked and the corresponding Wikidata entities are involved in a fact of Wikidata (110,311 out of 3,008,116 hyperlinks), we can compare the predicted relation to the ground-truth. We obtain 84% of accuracy. This good score is expected as the model is trained on WDV5 facts and some hyperlinks indeed correspond to existing facts. However, it is interesting to look at cases where the prediction is di erent from the true fact. We have observed that the model can hardly predict directed relations (e.g., parent-child) or semantically close relations (e.g., employer and educated at for links between scholars and universities). This is not surprising as the only available data is the structure of the KG. Some other mistakes come from the embedding model itself, for example headquarter location always has a lower score than twinned administrative body for some reason, making the headquarter predictions all wrong.
In Table 3, we report for two pages the semanti ed hyperlinks that got the highest scores. It is reassuring to see that most of the resulting facts are true, many of them being however already known in Wikidata. A few mistakes could be avoided using a little bit of context (i.e., text information) but these results suggest that our approach is able to correctly semantify many links.
It seems however di cult to produce automatically a full dataset in this way. First, the scores of embedding models are usually not normalized so comparing them works ne when done locally (e.g., looking at the links of a particular page) but comparing the scores of the three million possible facts is not feasible. Second, many facts that get a high score are very likely but require additional information not present in data. For example the three most likely facts resulting from semanti ed hyperlinks of Wikivitals+ are: (Serbia, member of, World Trade Organization): Serbia's application is still under review. (Taiwan, member of, World Trade Organization): Taiwan is already a member of the WTO through the Chinese Taipei but not in its name. (Kosovo, member of, Interpol ): Kosovo's application was rejected in 2018.
Clearly, some additional textual content is needed in these cases. 7
We have proposed a novel approach to relation prediction by KG embedding. Our approach is based on two key ideas: Balanced Negative Sampling in the training Head Predicted Relation Allergy health specialty Allergy has e ect Allergy has cause Allergy health specialty Allergy health specialty Allergy instance of Allergy drug used for treatment Allergy medical examinations Allergy drug used for treatment Allergy health specialty Allergy possible treatment Allergy symptoms Allergy drug used for treatment Allergy subclass of Allergy health specialty Allergy health specialty Allergy a icts Allergy a icts Allergy possible treatment Allergy symptoms
Tail Score Immunology -0.728593 Rhinorrhea -0.839387
Allergen -0.844231
Pediatrics -0.972245 Internal medicine -1.022068
Disease -1.022585 Adrenaline -1.040919 Blood test -1.165760
Aspirin -1.172504 Hematology -1.219639
Medication -1.221141 Abdominal pain -1.228720
Penicillin -1.235292 Pollution -1.283537
Statistics -1.297987 Epidemiology -1.301585 Immune system -1.374023
Blood -1.376165 Antibiotic -1.406908
Itch -1.416243 (a) Top-20 facts predicted from the page Allergy.
Head Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare Henri Poincare
Predicted Relation employer employer occupation place of death occupation eld of work student of eld of work eld of work eld of work student of eld of work notable work eld of work eld of work place of birth di erent from student of eld of work eld of work
Tail Score University of Paris -0.358521 Ecole Polytechnique -0.412040
Mathematician -0.414613
Paris -0.492610
Engineer -0.537747 Number theory -0.561962 Charles Hermite -0.592644
Epistemology -0.605310
Topology -0.678377 Algebraic geometry -0.712538
Wilhelm Wundt -0.738560
Optics -0.738636 Poincare conjecture -0.739285 Philosophy of science -0.791077
Metaphysics -0.822683
Nancy, France -0.824190 Raymond Poincare -0.830676 Karl Weierstrass -0.843734
Set theory -0.855588 Celestial mechanics -0.877295 of the embedding model, and Type Filtering to select candidate relations. We have shown that this approach performs well using an embedding model as simple as TransE, opening the way to robust and explainable predictions. Our results suggest that the model can be used to enrich Wikidata, by the semanti cation of Wikipedia hyperlinks associated with known entities.
This approach is however not yet fully automatable and performance still needs to be increased for that goal. For future work, we would like to further improve our negative sampling technique by replacing the relation with some probability that depends on the considered fact (h; r; t), instead of some xed probability. It seems also necessary to integrate some context from textual data for example (like the description of the relations and the articles themselves) in order to help the embedding model in its choices. A fully automatized process of enriching Wikidata with semanti ed Wikipedia hyperlinks seems however not out of reach.