Knowledge graphs provide significant assistance for many artificial intelligence tasks, but they are usually incomplete. Techniques for knowledge graph completion can improve the coverage of Knowledge Graphs (KGs) by inducing new facts. Traditional methods for completion use structural representations in embedding space, but textual information can also be helpful. Recently, pre-trained language models have shown impressive performance on natural language processing tasks. KG-BERT is a pre-trained language model that is used for knowledge graph completion and achieves appealing performance. However, KG-BERT struggles when the number of facts is inadequate. We consider the inadequacy of data for various relations and the compensation for sparsity via data augmentation. We propose two knowledge graph data augmentation methods to generate facts with novel relations. Specifically, multi-hop relations between two entities are extracted to form multi-hop facts, and implicit relations are generated by horn rules. Moreover, we find multi-hop facts are useful for few-shot learning scenarios. Our system improves the performance of KG-BERT regarding the accuracy of the link prediction task. The experimental results demonstrate that our models make significant enhancements for KG-BERT on several knowledge graph completion benchmarks (e.g., WN18RR and UMLS).
Knowledge graphs (KG) are effective structures for machine learning, composed of
’triples’—pairs of entities and a relation. Given KGs’ inherent incompleteness, efforts are
made to complete them by inferring plausible triples. Transformer-based models like BERT
[
Although KG-BERT demonstrates great achievement on the link prediction task, it can be further improved. Due to the incompleteness of knowledge graphs, some potential relationships could be further explored. For example, there are two triples (Anthony Albanese, born_in_city, Sydney) and (Sydney, located_in, Australia). According to these two triples, Anthony Albanese and Australia can use an alternative relation that may not appear in the knowledge graph to directly represent their relationship, i.e., nationality_of. Besides, the potential relations explored by rules could also give further information. For example, consider the rule co_work(X, Y) ← work_at(X, A) ∧ work_at(Y, A), where X, Y, A can be any entity that satisfies this rule and co_work and work_at are existing relations. Not all entities satisfying the rule’s body will satisfy its head, it usually has a confidence score to show its reliability. If we want to find a relation that could describe (X, Y) while it should be semantically similar with co_work, like one relation expressing work in the same company. This kind of relation is a loose implicit expression of the original one, and it is worthwhile to investigate how to generate relations like it. While the rule learners can mine rules with relatively low confidence, we cannot apply all these rules to KG directly and treat all infrared facts as valid facts to KG.
In this paper, we improve the Knowledge Graph-BERT (KG-BERT) with two innovative data augmentation techniques. First, a multi-hop data augmentation approach boosts the overall performance and offers an effective solution for few-shot learning. Second, the creation of implicit facts using AnyBURL-generated horn rules. Such facts express implicit information by constructing similar relations by dropout. These strategies together significantly improve the robustness and performance of KG-BERT.
KG-BERT: BERT [
Rule Learning: Rule-based models, such as AnyBURL [
The flow chart shown in figure 1 shows how original data and two kinds of augmented data are fed into language models. For an LM-based KGC such as KG-BERT, we directly input the original knowledge graph into the model. For data augmentation via multi-hops, the multi-hop triples are extracted as a knowledge graph and fed into the model. For data augmentation via rules, the implicit triples are extracted as a knowledge graph . Differently to and , it first converts to word embeddings and is implemented dropout and then fed into the model.
Augmentation via Multi-hop Facts: For entities ℎ and , assume a series of triples such as
{(ℎ, 1, 1), (1, 2, 2), ..., (− 1, , )}. We can present their relation in a multi-hop format
like ℎ→−1 1→−2 2→−... − 1→− . In order to make multi-hop triples adaptable for pre-trained
language models, we ignore the entities in the relation and get the triple (ℎ, 1 ⊕ 2 ⊕ ... ⊕ , ).
We convert such triples suitable format and call this method Data Augmentation via Multi-Hop
(DAMH). Assume a multi-hop triple has relations as (ℎ, 1 ⊕ ... ⊕ , ), then the text sequence
W̃︁DAMH can be represented as shown in formula 1. After that, we input W̃︁DAMH into a
pretrained language model and take out the pooled embedding CDAMH as the representation of the
entire multi-hop triple. Besides, constructing 2-hop triples for relations that infrequently appear
in training dataset can effectively improve the performance, which we call this few-shot learning
in knowledge graph completion [
W̃︁DAMH = [w[CLS], Wℎ, w[SEP], W1 , w[SEP], ..., W , w[SEP], W, w[SEP]] (1)
Augmentation via Horn Rules: Rule-based models such as AnyBURL can find potential triples by exploring horn rules. The information implied in these rules can also be used to generate augmented triples. By exploring implicit patterns in a knowledge graph, the rule-based model can obtain rules such as co-work(X,Y) ← work_at(X,A) ∧ work_at(Y,A), where co-work(X,Y) is head and work_at(X,A) ∧ work_at(Y,A) is body. But not all entities X,Y,AX, Y, A satisfying body can get the head. If we need to find an implicit relation which can describe these entity pairs, then it should have some similarity to the original relation.
Inspired by SimCSE [
W̃︁ token = [wt[oCkLenS], Wtℎoken, wt[oSkEePn], Dropout(Wtoken, wt[oSkEePn]), Wtoken, wt[oSkEePn]] (2) (3)
W̃︁ DAI = W̃︁ token + W̃︁ segment + W̃︁ position
We conduct a set of experiments to assess the performance of our data augmentation methods.
We evaluate our Data Augmentation via Implicit triples method on four datasets, while two of
them are open-source WN18RR [
We have released our code and detailed results 1, and generated augmented datasets for training, favoring 2-hop data extraction for optimal performance. DAMH expands the training datasets of WN18RR and UMLS by 158% and 307% respectively. Statistics of datasets are also published in the link. We adopted KG-BERT’s hyper-parameter settings due to high computational costs. Our methods are compared with other baselines on UMLS and WN18RR datasets. Test samples involved creating negative samples by replacing the head or tail entity with other entities not present in the training set.
From the comparison of experimental results in table 1, it can be seen that KG-BERT+DAMH, KG-BERT+DAI, and KG-BERT+DAMH+DAI all outperform the original KG-BERT on both 1Detailed experimental results and code can be found at https://github.com/LSX-Sneakerprogrammer/ KG-Augmentation WN18RR and UMLS datasets. For the Hits@10 metric, the most significant improvement on WN18RR is from KG-BERT+DAI+DAMH (+10.6% compared with KG-BERT), while for UMLS is from KG-BERT+DAMH (+0.5% compared with KG-BERT). For the mean rank metric, KG-BERT+DAMH outperforms the official result of KG-BERT by a significant 12 on WN18RR and a smaller 0.08 on UMLS. These results establish that our augmentation approaches make significant progress on these well-studied tasks.
For few-shot learning, we evaluate models on two datasets, FB15k-237 (few-shot) and NELLONE (few-shot), and compare them to KG-BERT. We can see the results in table 1 for FB15k-237 (few-shot). The improvements of KG-BERT+DAMH are 7.8% and 3, respectively for Hits@10 and Mean Rank. For the NELL-ONE (few-shot) dataset, the KG-BERT+DAMH surpasses KG-BERT on four metrics. These experimental results demonstrate the effectiveness of our method on few-shot learning.
Our study proposes generating augmented triples via multi-hop connections and horn rules, effectively enhancing KG-BERT, particularly with the multi-hop method. Both methods and their combined form remarkably improve performance, with the multi-hop approach notably strengthening few-shot learning tasks. Future research should investigate integrating the augmented data with other knowledge graph completion methods.