Knowledge graph embedding (KGE) has become a prominent topic for many AI-based tasks such as recommendation systems, natural language processing, and link prediction. Inclusion of additional knowledge such as ontology, logical rules and text improves the learning process of KGE models. One of the main characteristics of knowledge graphs (KGs) is the existence of relational patterns (e.g., symmetric and inverse relations) which usually remain unseen by the embedding models. Inclusion of logical rules provides embedding models with additional information about the patterns already present in the KGs. The injection of logical rules has not yet been studied in depth for KGE models. In this paper, we propose an approach for rule-based learning on top of the two embedding models namely RotatE and TransE within this scope of the paper. We rst study the e ect of rule injection in the performance of the selected models. Second, we explore how the removal of leakage from popular KGs such as FB15k and WN18 a ects the results. By leakage we are referring to the patterns exist in the training set from the test set (e.g. if the test set contains (h; r; t) then it also contains t, r, h in the training set which is considered as a symmetric leakage where t, r and h refers to tail, relation and head respectively). Empirical results suggest that incorporation of logical rules in the training process improves the performance of KGE models.
Nowadays, knowledge graphs (KGs) are one of the leading technologies in knowledge representation and representation learning. The main characteristic of this technology is its representation of facts in a triple set KG = f(h; r; t)jh; t 2 E ; r 2 Rg in which h; t; r refer to the head and tail and relation respectively, where E is a set of entities (nodes), and R is the set of relations (links). Relational learning is the act of applying learning models on KGs. Among the already existing learning approaches, knowledge graph embedding (KGE) models have shown in uential results in link prediction. An embedding model provides vector representations of entities and relations in a KG where a triple (h; r; t) is mapped to its corresponding d dimensional vector as h; r, t 2 Rd.
In recent years, there have been a series of embedding models which were
proposed with di erent learning skills. Some of the popular models are TransE [
Empowerment of KGE models can be done internally by adding complementary knowledge to the underlying KGs. In the case of encoding relational patterns, the injection of logical rules is seen as an approach when the model cannot capture the patterns directly from the KG itself.
In order to further study the in uence of logical rules in the learning process of KGE models, rstly, we have developed a pipeline where we are able to inject rules in popular embedding models. Research suggests that rule injection can aid learning of KGE models. After injection of logical rules, we have done experiments to get an intuition about how rule injection is a ecting the performance of KGEs. Secondly, we removed known patterns (leakage) from the test set, which we call re ned test set. In this step, we have additionally evaluated the performance of the underlying models. Results approve our hypothesis such that rule injection can be bene cial to KGE models such as TransE and RotatE. Removal of known patterns as leakage from the test set causes the results of the same KGE models to become worse. Unfortunately, rule injection is unable to help when the leakage has been removed from the test set. Within the scope of this paper we investigate on four types of rules, namely implication, symmetric, inverse and equivalence (Table 1). 2
Since patterns are implicitly hidden all over the KGs, in order to empower KGEs
with complementary knowledge of logical rules, one need to rstly extract them.
Here we represent three of the related rule extraction and injection attempts
in the context of KGEs. There are multiple rule extraction frameworks such
as AMIE [
As mentioned before, KGs include relational patterns. However, they are often
not explicitly modelled via schema axioms or rules. If we want to use the logical
rules as complementary knowledge in KGEs, we need to rst extract patterns
from KGs and make them available for such models. Towards having a set of
rules extracted from KGs, a set of certain steps have been followed in this work
using AMIE [
The KGs that are considered as an input of our rule extractor step are seen
as raw data in the form of RDF (Resource Description Framework) 4. This
representation is also in triple format (s,p,o) (Subject, Predicate and Object).
Subject and object are considered as entities. Predicate is considered as relation
between the two entities. We use two main standard datasets as input KGs
namely FB15k [
The core of rule extraction step is based on AMIE [
Rule generation is a signi cant part for this pipeline. Not all of the extracted rules provide valuable information in the training process. Therefore, rules which have a lower con dence level in the dataset are discarded. Based on the comparisons of the results and evidences in the KGs, a threshold value of 0.8 is being set to abandon unwanted rules.
Grounding generation is required for the rule injection in the selected models. Grounding are the set of triples from the dataset that follows the patters of rules extracted by AMIE. After mining the rules from the underlying KG using AMIE+ and ltering out unwanted rules, we have generated groundings from those ltered rules, which are required for generating rule loss in the training phase. To do so, for each of the rules among the extracted results, the components are decomposed. After matching the corresponding patterns of the rules in the form of premise ! conclusion for each of the class types (e.g., implication, inverse, symmetric and equivalence), the relations from premise and conclusion of the rule pattern is taken with matched class type together into a nal set of rule (named as rule bag). Secondly, after the rule bag is created for each of the items, it is taken and matched with the training triple of the respective KG. An example of how this task of grounding generation from original triple is depicted in Fig. 1. While extracting pattern of rules from KG using AMIE tool we enforce a con dence threshold value of 0.8 to lter out irrelevant pattern of rules. 4
Rule injection is performed by adding groundings in the training phase. Grounding loss has been incorporated alongside of the KGE training by adding it up with the total loss. In this paper, two KGEs have been adopted in such a way that they learn grounding loss and triple score loss simultaneously as a total loss. Based on the total loss of the models (TransE and RotatE), back-propagation is performed and parameters are updated accordingly. In this section, each of the models that we use for rule injection are described in brief. 4.1
TransE [
RotatE [
In our context, we incorporate the groundings to the training phase which we obtained from the extracted rules. In this paper, we de ne this as rule injection. After obtaining the loss of the groundings in training phase, it is added with the mainstream loss. The principal goal of rule injection into KGE models is to Rule De nition Formulation based on score function Symmetric (h; r; t) () (t; r; h) fhr;tfhr=1;tftr;hft+r;2h h;t Implication (h; r1; t) ) (h; r2; t) fhr2;t Inverse (h; r1; t) ) (t; r2; h) fhr1;t Equivalence (h; r1; t) () (h; r2; t) fhr1;t = fhr2;t + h;t
Table 1. Representation of rules
improve the models capability to capture the pattern of rules from knowledge
graphs. De nition and the formulation of used rules are de ned in Table 1 [
We call these scores as output1 and output2. output2 is always bigger than output1 (for TransE and RotatE) based on the formulation. Thus, it can act as a clear baseline for the grounding loss. Our grounding loss can be depicted in the equation GL = (L(output2) L(output1) + ), where refers to the output function of the injected model and is an additional constant value. Here, L can be stated as the abstraction of KGE model's (e.g., RotatE) score function.
Hyperparameter settings. A series of hyperparameters are used to achieve the result of table 2. For FB15K the learning rate is 0.1, the rule loss multiplier mp is 0.05. The embedding dimension D is 200, batch size is 2750, the rule loss multipliers fimplication = 0, inverse = 0.1, symmetric = 0.01, equivalence = 0.05g, number of negative sample N is 10, is 24. For WN18 everything is the same except the rule loss multipliers and the margin . Since for WN18 we have only inverse and symmetric groundings, the multipliers are finverse = 7, symmetric = 0.5g. The is kept at 10.
Results. In this section, we present the results of our evaluations. Selected
comparison criteria are followed by the best practices of evaluation in embedding
models. We consider Mean Rank (MR) and Hits@10 as the main criteria to
evaluate the models [
For both of these criteria, we consider looking into the details of the behaviour of KGE models with or without injection of rules. These are all evaluated on two sets of the data: raw and ltered as discussed in section 3. In table 2, we present the results of our evaluation on FB15k and WN18. The results show that rule injection has a positive in uence in the result. In case of RotatE and TransE rule injection yields better result. It is clear that for FB15k dataset RotatE has a signi cant performance increment in both raw and ltered settings. I.e. the Hits@10 has increased from 0.85 to 0.88.
FB15K Models Standard Leakage Free Without injection With injection Without injection With injection Raw Filtered Raw Filtered Raw Filtered Raw Filtered MR Hits@10 MR Hits@10 MR Hits@10 MR Hits@10 MR Hits@10 MR Hits@10 MR Hits@10 MR Hits@10 RotatE 190 0.5247 38 0.85 150 0.5705 35 0.8791 276 0.4355 113 0.6453 271 0.4354 108 0.6410 TransE 204 0.4605 68 0.6268 156 0.5199 51 0.6998 270 0.4304 123 0.5677 271 0.4251 125 0.5624
WN18
For TransE the Hits@10 has also been increased from 0.63 to 0.70 for FB15k dataset if we inject rules. Though it can be seen from the table that for WN18 dataset, the result does not improve upon injecting rules in ltered settings but upon injection of rules there is signi cant improvement in raw settings. I.e. the Hits@10 has been improved from 0.81 to 0.90 for raw settings. However, if we re ne the test set by removing the leakage, the result in every case is reduced by a signi cant number. Even the injection of the rules does not help in this regard. As seen in the table 2 the results are still poor even after injection. 6
In this work, we showcased the use of logical rules in extraction and injection of knowledge from knowledge graphs into embedding models. One of the main contributions of our work is that we removed leakage from testing set of FB15K, therefore, we have many inverse patterns in training dataset of the underlying KG, but not the leakage in testing. We mainly focused on the two main models namely: a) TransE (baseline model), and b) RotatE (current state-of-the-art model) designed for encoding the patterns. This study is a prototype of approving the role of rules in improving the learning process and performance of KGE models. As future work, we will extend the pipeline as a comprehensive framework to cover most of the other embedding models, and more datasets (FB15k-237, WN18RR) and real KGs. We also consider to automatize the rule extraction and injection which is currently done externally using AMIE with post-processing steps in grounding the rules. We also plan to train RUGE on these KGs that we have created and compare the results.