We address the challenge of building domain-specific knowledge models for industrial use cases, where labelled data and taxonomic information is initially scarce. Our focus is on inductive link prediction models as a basis for practical tools that support knowledge engineers with exploring text collections and discovering and linking new (so-called open-world) entities to the knowledge graph. We argue that - though neural approaches to text mining have yielded impressive results in the past years current benchmarks do not reflect the typical challenges encountered in the industrial wild properly. Therefore, our first contribution is an open benchmark coined IRT2 (inductive reasoning with text) that (1) covers knowledge graphs of varying sizes (including very small ones), (2) comes with incidental, lowquality text mentions, and (3) includes not only triple completion but also ranking, which is relevant for supporting experts with discovery tasks. We investigate two neural models for inductive link prediction, one based on end-to-end learning and one that learns from the knowledge graph and text data in separate steps. These models compete with a strong bag-of-words baseline. The results show a significant advance in performance for the neural approaches as soon as the available graph data decreases for linking. For ranking, the results are promising, and the neural approaches outperform the sparse retriever by a wide margin.
1 ...Dawn French and Stephen Fry. 2 Tolkien also based elements of his... 3 …together with Walter Moers... 4 ...by Sir Arthur Conan Doyle in 1912...
? profession Author
text collection (contexts) (1) query text for relevant contexts (2) query KG for relevant facts knowledge graph
STEP II: LINKING 1 Stephen Fry profession Author 2 Stephen Fry place of birth Hampstead 3 Stephen Fry religion Atheism 4 Stephen Fry profession Screenwriter !
Stephen Fry ? ?
Explicit knowledge is scarce in these domains. However, what is often commonly available in abundance is text data, such as service tickets, insurance claims, or business reports.
The focus of this paper is to support experts with building knowledge graphs by inspecting text. To do so, we assume that the expert adds triples to the graph in an iterative process, in which he/she locates and identifies new entities in the text collection and links them to the graph. We address two key steps of this process, as illustrated in Figure 1: 1. Discovery/Ranking: The expert explores the text collection based on a partial fact of the KG (i.e. a relation-vertex tuple). He/she specifies an information need (such as “find me actors”, or more specifically “find sentences likely to contain mentions of entities which have the relation profession with the known entity actor”). The system returns a ranked list of potentially interesting sentences, giving rise to a ranking task. 2. Linking: The expert studies the sentences. Once he/she has discovered an interesting entity (such as Frances McDormand), we would like to link to the graph by adding triples, not only via the profession relation but also via other relations (such as the birthplace, languages she speaks, or movies she appeared in). To do so, the system collects textual evidence on the entity in form of other mentions in the text collection. From those, it infers triples that link to the knowledge graph.
We address both tasks using embedding-based semi-inductive [3] open-world [4] link prediction, which is targeted at predicting the likelihood of triples (ℎ, , ). In contrast to standard link prediction, semi-inductive link prediction addresses new entities to be linked to the known graph and the open-world scenario connotes that the new entities are solely described via free text.
Since domain-specific data is often confidential, link prediction research employs open data from graphs such as Freebase [5] or Wikidata [6]. We argue that the insight these benchmarks ofer for industrial knowledge acquisition is limited, and contribute a new benchmark called Inductive Reasoning with Text V2 (IRT2) with the following benefits 2: (1) To assess models at diferent stages of graph construction, we benchmark on graphs of varying size. (2) While text contexts in other datasets consist of concise descriptions of entities, text in practice contains rather incidental mentions of entities. We sample our text data accordingly. (3) To our knowledge, our study is the first to not only address the linking task but also the ranking task.
We evaluate three inductive link models on IRT2: One baseline based on keyword matching, and two neural models that combine a transformer text encoder with a link predictor. We show that the end-to-end approach and separate training both work well, while the latter generally performs a little better (even for small graphs). Also, only through evaluation on smaller graphs, the baseline model is outperformed when linking, which favours the neural models in scenarios where structured data becomes sparse.
Link Prediction Models: Closed-world link prediction (or also knowledge graph completion, KGC) has attracted interest in the research community over the past years. Earlier approaches such as RESCAL [7], TransE [8], and DistMult [9] laid the foundations for many following completion models such as ComplEx [10], ConvE [11], RotatE [12], or KBGAT [13]. We are, however, interested in open-world scenarios, where, given a textual description of an entity, geometric reasoning is applied through the alignment of dense graph- and text-representations. This combines KGC—which aims at the prediction of missing links between existing entities— with language modelling [14, 15, 16] which produces dense vector representations for natural language text. Specifically, in this work, a semi-inductive setting [ 3] is studied, where out-of-kg entities are to be linked to an existing graph using their textual description. One of the first approaches to combining language- and graph-reasoning models is NTN [17], where entity description embeddings are trained to be similar if their entities are connected in the KG. Later approaches build on this idea: DKRL [18], ConMask [4], MIA [19], and KEPLER [20] introduce models that jointly learn the connection between entity and graph representations. A diferent approach decouples the KGC scorer and text encoder such that first the scorer is trained independently and in a separate step a projection from text-based entity descriptions to their associated graph-based embeddings is learned [21, 22, 23]. We study and compare both approaches with the models described in Section 4. Highly related to our work is the recently published approach by Daza et al [24]. Here, a BERT model (BLP) is trained to directly score KGC triples for plausibility using the textual representations instead of training a separate entity embedding. However, we cannot rely on high quality text and thus a separate entity embedding tuned with many diferent observations for input text is more suitable.
KGC Benchmarks: Closed-world KGC models are usually evaluated on subsets of publicly available knowledge graphs such as Freebase [5], DBPedia [25] or Wikidata [6]. Famous benchmarks include FB15K [8], succeeded by FB15k237 [26] due to test-leakage, WN18, succeeded by WN18RR [11], or CoDEx [27], among others. Open-world KGC combines KGs with textual descriptions of its entities. Approaches include the FB20K benchmark as part of DKRL [18], DBPedia50k and DBPedia500k as part of ConMask [4], and FB15k237-OWE as part 2The data is publicly available under https://github.com/lavis-nlp/irt2 of OWE [21]. More recent work introduces Wikidata5M [20] (a large Wikidata subset) and the FB15k237/WN18RR adaptions of [24]. Since data in industrial use cases is not publicly available, we also sample our benchmark data from open knowledge graphs (CoDEx) and text sources (Wikipedia). However, all the above benchmarks tackle the open-world scenario with a single concise description of graph entities. We are, however, interested in linking entities associated with many, noisy text contexts. To our knowledge, the only benchmark to attempt this is IRT [28]. Here, a a set of 30 text contexts with incidental mentions is associated with each entity in the knowledge graph. We extend this benchmark by ofering more text, multiple graph sizes, a greater variety of mentions, and evaluation protocols for both ranking and linking.
We define a knowledge graph (KG) as a directed graph = (, ℛ, , ℳ, ) where is the set of vertices (e.g. = Fargo) and ℛ a set of relation types (e.g. = genre). Triples (ℎ, , ) ∈ ⊆ ×ℛ× connect the vertices, e.g. (Fargo, genre, Thriller Film). Additionally, the set ℳ contains textual mentions, each associated with a vertex via a function : ↦→ (ℳ). For example, the entity v=Thriller Film has the mentions M(v) = { “crime thriller” , . . . , “gangster film ” }. Note that mentions can be ambiguous (e.g. “the film ” as mention of Fargo). Furthermore, each of the mentions is assumed to occur in textual context sentences from a corpus . In general, these contexts do not express triples but solely contain incidental mentions of entities. They are accessible through a function : × ℳ ↦→ (). For example, with = Crime Film and = “gangster film ”: (, ) = { “Corman called it the most accurate, authentic gangster film ” , . . . }. Note that – via M and C – we can access an entity’s mentions and contexts, and vice versa.
Open/Closed-World: While the text corpus contains known (or "closed-world") entities, a second corpus contains undiscovered open-world mentions ℳ. Our goal is to discover these and link them to the graph. We define a closed-world graph (containing subsets of the respective sets of and the text corpus ) and an open-world graph = (, ℛ, , ℳ, ) with ⊆ , ⊆ , ℳ = ℳ ∖ ℳ: Namely a graph with a set of undiscovered mentions not known in the closed-world graph . This gives rise to the two tasks we address:
Ranking Task: The first task is to retrieve text contexts from which contain mentions of interest, i.e. unknown mentions of (possibly unseen) entities which should complete a given entity-relation pair. For example, for relation = genre and tail = crime film , we search for text contexts where any crime films are mentioned.
Linking Task: Now, given an open-world mention was just discovered and marked, all text contexts of are bundled by mention. For each such bundle of text, links ∈ ℛ to the graph vertices must be found. For example: “What genre is associated with all text contexts that contain the mention fargo?” .
We tackle the above tasks with three models: (1) JOINT, where a text encoder and a knowledge graph completion (KGC) scorer are trained jointly, (2) OWE, where an encoder learns to project text representations into a pre-trained graph embedding space, and (3) BOW, a model that employs bag-of-words representations for text similarity. All three models are evaluated on both the ranking and linking tasks. Note that, although our notation is (for brevity) generally focused on finding tails, we do also always include head-prediction for given relation-tail pairs.
Both neural models combine a text encoder and a KGC module (see Figure 2). As a text encoder, we use a pre-trained BERT [16], a popular state of the art transformer [29] encoder. Given a text context , we run it through the encoder and select the [CLS]-token embedding as the context representation ()CLS ∈ R′ (following the observations of [28]). This representation is mapped using a learned afine projection ( , b) into a complex-valued space, obtaining a representation c. The representation is then used to estimate a tripel’s plausibility. We use the ComplEx scorer [10], which, given complex-valued context, relation and tail embeddings c, r, v ∈ C, calculates the plausibility score as (, , ) = Re(c ⊙ r ⊙ v). Note that—since the projection maps from real-valued to complex space—we choose the matrix and bias vector dimensions accordingly ( ∈ R′× 2, b ∈ R2). Overall, our neural models can be written as: (, , ) = ( · ()CLS + b, r, v ) ⏟ c ⏞ Multi-Context Extension As discussed above, instead of a single context , a set Σ of multiple contexts may be available describing the same entity. We would like our entity representations to take the whole set Σ into account. To do so, we use a simple early fusion that averages the text representations before projecting them: c = · ︃( |Σ | ∈Σ 1 ∑︁ ()CLS )︃
+ b ℒJOINT = − ∑︁ ′∈ ︃(
exp((, , ′)) log ∑︀′∈ exp((, , ′)) · 1(,,′)∈ )︃ (1) (2) (3)
We then use the resulting representation c as a drop-in replacement in Equation (1). Concerning training, we use two variants of this setup: In the first variant (referred to as single), training happens on individual contexts as in Equation (1), but we apply Equation (2) in inference. In the second variant (multi), we apply Equation (2) both in training and inference.
As illustrated in Figure 2, we study two diferent training strategies to fit the model parameters (text encoder , projection , b, and knowledge graph embeddings r, v). Our first approach uses a joint (end-to-end) training (JOINT): We iteratively sample a random entity from the knowledge graph. For , we draw a random triple (, , ′) and context . The model’s scores (, , ′) are mapped to a probability distribution using a softmax, and we apply a cross-entropy loss:
ɸ text encoder
BERT parameters projection
W, b
c text-based representation
graph embeddings v, r ψ kgc scorer ℒ cross entropy text contexts 1. joint training
(JOINT) 2. separate training (OWE)
end-to-end gradient flow (2b) align text and graph representations ( c ≈ v ) (2a) train link prediction
Note that – though the above notation samples triples (, , ′) to predict tails ′ – we also sample triples (′, , ) and predict the heads ′ accordingly.
Our second approach follows Shah et al’s OWE model [21] and applies a separate training. Here, training happens in two steps: First, the link prediction model is trained on the closed-world graph, obtaining a graph-based embedding space. Second, the text encoder and projection (, b) are trained to align entities’ text-based representations with the (now fixed) graph-based ones. To do so, we reduce the geometric distance between context representations c = · () + b and their associated graph-based representations, using a mean squared error loss: ℒOWE = 21 · ∑︁ (︀ Re(c) − Re(v)︀) 2 + (︀ Im(c) − Im(v))︀ 2 0<≤ (4)
Inference: This section outlines how the neural models are used in our two tasks. First, in ranking, a closed-world entity and a relation of interest are given, and the task is to retrieve a ranked list of text contexts from the query corpus that potentially contain mentions of relevant open-world entities to be identified by the expert. To do so, we compute the singlecontext scores (, , ) for all contexts ∈ , ∈ ℛ and ∈ . The scores are normalized per relation using the softmax function as to better compare the independently scored contexts (the raw ComplEx scores are unbounded). The contexts in the query corpus are then ranked by this normalized score.
Second, in linking, the expert is interested in a certain mention of interest ∈ ℳ representing an open-world entity. His/her goal is to link said entity to the graph. To do so, he/she selects a relation ∈ ℛ and collects a set of contexts Σ containing to describe the open-world entity. Using the contexts, all entity candidates ∈ are ranked by their scores (Σ , , ). Note that in principle we can repeat this procedure for each relation and also for predicting heads instead of tails.
To evaluate the tasks at hand with a common industrial approach, we compare the above neural models with a bag-of-words approach. Given one or several mentions, this approach concatenates text contexts containing the mentions into documents and then conducts a similarity matching between documents using the well-known BM25[30] keyword matching implemented by the Elasticsearch search engine3.
Ranking: Given an entity relation pair such as = Thriller Film, = genre (“Find text contexts which mention thriller films”), the baseline’s strategy is to find open-world entities that are similar to known closed-world entities ′ for which (′, , ) holds (i.e., known thriller iflms). We sample a set of random representatives ′ and some of their associated text contexts. These contexts are concatenated into a document which is then used as a query against an index containing all open-world contexts .
Linking: Given a mention ∈ ℳ and a relation , we sample contexts containing into a document (), which is used to query against an index containing documents () for all closed-world vertices ∈ . The top-n predicted results allow us to compile a set of reference vertices which we use as blueprints for prediction. For example: When searching for = profession of text contexts containing “s. a. corey” (“Predict the profession of the text contexts containing s. a. corey” ), the result list may contain other texts describing authors. The ifnal prediction of vertices is compiled from the retrieved vertices targets of relation and scored by the inverse of the position of the document in the result list.
Our benchmark picks up the work of [28] and extends it: (1) We study the behaviour of models with varying knowledge graph size, and particularly for small graphs, (2) we test models for both the ranking and linking task. We ofer four variants of the benchmark: tiny, small, medium, and large. The variants ofer diferent ratios of structural information (i.e. graph triples) and associated text. The datasets are based on CoDEx-M [27] which is sampled from Wikidata [6]. The following steps are executed:
1. Identifying Concept Vertices: We assume that in a real-world scenario world-knowledge is more likely to be already modelled in the given KG. Information to be discovered is usually much more volatile. Consider for example a graph for a machine manufacturer. We know that things can catch fire (the world knowledge) but we aim to find the specific parts that actually do. A heuristic to identify such world knowledge is based on the assumption that such entities are characterised by relations which exhibit a strong disproportion between their heads and tails. The ratio between a relation ’s domain size (its number of heads) dom() := |{ℎ | ∃ : (ℎ, , ) ∈ }| and the range size (its number of tails) rg() := |{ | ∃ ℎ : (ℎ, , ) ∈ }| is min(︀ dom(), rg())︀ /max(︀ dom(), rg())︀ . Per size variant of the dataset, we select a subset of the relations ofered by CoDEx, order by ratio and select a subset of those to determine concept entities that will remain in the closed-world split. The appendix in ?? enumerates all relations and corresponding selections in greater detail.
2. Sampling Mentions and Text Contexts: As our graph is based on Wikidata, we can exploit its links to Wikipedia. For each vertex , we identify the associated Wikipedia page and all pages which link back to it (). Using the link text of the hyperlinks, we build the set of mentions which are associated with the vertex (). Lastly, to build the contexts (, ), all occurrences of the mentions in () are identified and the surrounding sentence kept as context. This is a heuristic to obtain incidental text contexts: The entity is mentioned but usually not the subject of the sentence. As example for = Scotland: “Aberdeen is a city in northeast Scotland” .
3. Open/Closed-World Split: The open/closed-world split, in this case, is not done on vertex but mention level. We separate the mention set ℳ into a partition ℳ for closed-worldand ℳ for open-world mentions respectively. First, all mentions associated with concept entities are set aside for the closed-world part. Then, for each remaining vertex , its associated mentions are distributed randomly between open- and closed-world. We set an additional pruning parameter which limits the maximum amount of mentions allowed for a closed-world vertex. With the information about closed-world mentions, we identify all triples whose vertices are associated with them and set them aside as the closed-world triple set ⊆ . The so-called test triples ( ∈ ℳ, ∈ ℛ, ∈ ) (explained below the table) are derived directly from the triple set by identifying in which triples the mention’s vertex occurs. We set aside a subset of the open-world split for validation. The above steps are executed for the four diferent dataset variants. These have diferent parameters set for concept relations, total relations to keep and closed-world mention pruning (see the appendix for more details).
Relations |ℛ| Entities |ℰ| Training Triples | | Training Text || Test Text || Ranking Queries Linking Queries Test Triples
The last four rows of the table describe the scale of the challenges. A ranking query is a tuple ( ∈ , ∈ ℛ) (e.g. (writer, profession) – “Find texts with writers in them” ) and an associated ground truth set of open-world mentions {1, . . . } ∈ to be discovered (e.g. { “Tolkiens” , “the author” , “Corey” , . . . }). A linking query is a tuple ( ∈ , ∈ ℛ) (e.g. (“Tolkiens” , profession) – “What professions does the entity have, given texts in which “Tolkiens” occurs?”) and a set of associated closed-world vertices ∈ (e.g. { author, linguist, . . . }). One can see that both ranking and linking are diferent views of the same data: unique ( ∈ , ∈ ℛ, ∈ ) triples. These triples are presented as test triples in the table above and allow to gauge the ratio of the queries and their associated ground truth. For example, in the most extreme case, ranking on the tiny variant, has 102866/695 ≈ 148 mentions to discover per query on average.
We run a series of experiments to build a solid understanding of the challenge’s dificulty. We employ all three formerly described models (JOINT, OWE, BOW) for both tasks (ranking, linking) on all four splits (tiny, small, medium, and large). We measure hits@k and the mean reciprocal rank (MRR) for each test triple (i.e. micro-averaged). Following common practice, we apply target filtering [ 21] to the scored predictions (mentions for ranking and vertices for ranking). Target filtering is a method where, for a given ranking, only a single true positive of interest is inspected, while all other true positives are removed from the list. This helps to not artificially worsen the result for rankings with many true positives. We ofer the datasets, dataset creation code, and evaluation protocol online4. The model implementations and all presented trained models are also supplied separately5. We evaluate our models on a subset of the test text contexts (400k sentences drawn randomly for ranking; 100 sentences per mention for linking).
We determine the final hyperparameters using a combination of random-, and grid searches over the parameter space. A single closed-world ComplEx model is trained per dataset split and commonly used to train the OWE model using Adagrad [31]. The hyperparameters studied comprise learning rate, L2 regularization weight, and embedding space dimensions. We employ PyKEEN[32] for training and build our own open-world extension. We implement and train the JOINT and OWE models using PyTorch [33] and PyTorch Lightning[34]. Models are optimized using Adam[35]. For each model architecture and dataset split, we run a random parameter sweep to fix a subset of parameters and subject the remaining to a grid search. We study the efect of regularization of the entity and relation embeddings, weight-decay and learning rate of the optimizer, how many layers of the encoder are frozen during training, how many text contexts are sampled per vertex per training, and whether the mention is masked during training. Additionally, we study how many contexts to provide per optimizer step for multi-context models. Generally, all models share a similar set of values for learning-rate, weight-decay and regularization weight. The other parameters are dependent on the split. We found to freeze parts of the encoder works well against overfitting. Contrary to the observations of [ 28], masking did not significantly increase model performance. The final hyperparameters are provided in the appendix (??) and with the models online.
First, we evaluate the linking task as the designated challenge for the presented models. Table 1 details our findings. Generally, a few trends can be observed: (1) The neural models significantly outperform the baseline approach on the three smaller variants but yield inferior results on the large dataset. This supports our claim that a benchmark should ofer insights into varying degrees of structural data scarcity. Even with an abundance of data (usually favouring neural 4https://github.com/lavis-nlp/irt2 5https://github.com/lavis-nlp/irt2m approaches), the BOW baseline can beat the neural model’s performance on large but falls short for the smaller variants. (2) Overall, the OWE approach outperforms JOINT albeit not by a great margin for tiny, small, and large. The biggest performance gap can be observed on medium, where the OWE model profits the most from the decoupled training. We argue that this hints towards the need for better sampling/overfitting strategies during training for JOINT as it seems to struggle with the lower variety of mentions seen during training. (3) Although the multi-context models generally perform better than the single-context counterparts, the diference in performance is not as pronounced as reported by [ 28]. Contrary to their findings we also did not observe much performance improvement through masking the mention. These observations indicate that both models struggle to incorporate much of the textual clues and instead rely heavily on the structural information present in the vertex- and relation embeddings. Overall, the metrics show that the models can link an entity described by text contexts to a given graph with relatively high precision. single multi single multi
Tiny
Secondly, for ranking, we study whether the models which were originally trained to do linkprediction can be employed for discovery, too. We report the hits@100 performance in Table 2. (1) The results show that the scoring mechanism of the neural models outperforms the BM25 search results. At first glance, this is a surprising insight, but can be explained by the nature of text samples. As the mention of interest is usually not the subject of the text context, the query does not directly describe its properties. This leads the ranker astray. For example: searching for other comedians (i.e. ?, profession, comedian) the model uses text samples sampled from the Wikipedia page about New York among others. This leads to other text samples to be retrieved which also talk about things in the context of New York but seldom other comedians. This contextualization of queries (by queried relation) is better considered with the neural models. (2) Overall ranking quality is not high enough for practical application.
We suspect two main factors which hinder efective ranking: First, the scores are calculated independently from each other. This makes an intra-sample comparison of scores dificult. Secondly, the ranking task requires scores first and foremost produced by head prediction. These scores usually are of lower quality because the models are much better at predicting a few biased tails (e.g. professions) than hundreds or thousands of volatile heads (e.g. authors). This is worsened by the formerly described observation that text contexts have a comparatively small influence on the score. To obtain better performance, new approaches must be devised which incorporate the graph context on the one hand and neural semantic retrieval [36, 37] on the other.
We present IRT2, a benchmark to study a knowledge acquisition pipeline to extend a knowledge graph (KG) from noisy text. We ofer knowledge graphs of diferent sizes and text associated with its vertices. The benchmark comprises two tasks, ranking and linking, where first, entities need to be discovered from unlabelled text, and in the second stage, linked to the KG. Alongside, we study three approaches for the tasks at hand: two neural open-world triple scorers and a bagof-words model. We found the models to perform well for the linking task but less so for ranking. For linking, the BOW baseline outperforms the neural models on the large variant. For the remaining, smaller variants, the OWE approach outperforms all other approaches even if graph data is scarce. For ranking, the JOINT model produces the best results. Although the models show a promising direction to be used for ranking, they do not perform at the level for practical application yet. We recommend studying how semantic ranking techniques can profit from the given links between graph- and text data. We invite the community to use the benchmark to (1) devise ways to better incorporate the noisy but abundant text data to make better predictions, (2) find ways to contextualize semantic retrievers with the specific graph information, and (3) derive insights for application in an industrial context with similar constraints and conditions. The dataset and evaluation scripts can be found here: https://github.com/lavis-nlp/irt2.
This work was supported by the BMBF program FH-Kooperativ, project SCENT (13FH003KX0)
This appendix details configuration options used for benchmark construction and model training in detail to allow for the reproduction of the reported results.
Concept Relations Total Relations Closed World Threshold Target Mention Split Target Validation Split Mention Threshold Concept Vertices Training Vertices Training Mentions Training Triples Training Text Contexts Validation Vertices Validation Mentions Validation Vertex Triples Validation Task Triples Validation Text Contexts Test Vertices Test Mentions Test Vertex Triples Test Task Triples Test Text Contexts
Tiny
The final benchmark proportions rely heavily on the selected relations and pruning options for construction. The following table enumerates both the configuration which was used for sampling and the resulting key figures for the training, validation and test splits. Relations are picked manually per split, are ordered by their ratio and a subset is selected as concept relations. The Closed World Threshold determines how many mentions per relation are retained at most in the closed world split. The Mention Threshold says to keep only mentions with at least that many associated text contexts. Validation- and Test Vertex Triples are the triple sets (vertex, relation, vertex) from which the final Task Triples are derived (mention, relation, vertex). They are pruned by filtering out true open-world vertices (the necessary zero-shot entity linking is out of scope for the presented tasks)—which explains the decrease that can be observed for the tiny variant.
The above figures detail the shift in proportions between the dataset variants. The test mention counts decrease proportionally to the increase in training. Concept mention count increases and is equal for M and L. The proportion of concept mentions decreases with increasing dataset size. Correspondingly, validation text context counts stay constant while an increase in training data decreases available test data.
The bar plots above report the absolute counts of triples (called Vertex Triples in the former table) and associated text. The resulting triple split behaves correspondingly to the mention split with a proportional behaviour between training/test data. The overall triple count more than doubles between T and L. Associated text data increases with respect to the increase in closed-world mentions.
The following table shows all relations present in CoDEx ordered by ratio as described in 5. On the right-hand side, it is detailed which relations are present in which dataset variant. The respective right side shows whether it is selected to remain in the dataset. The left side marks whether the relation was selected to pick concept entities.
Heads
Tails
Triples P1412 P1303 P140 P27 P30 P509 P172 P2348 P102 P106 P495 P136 P641 P19 P69 P463 P264
P20 P1050 P101 P2283 P135 P119 P108 P37 P840 P17 P50 P452 P551 P749 P407 P361 P57 P159 P161 P1056 P740 P131 P737 P138 P112 P40 P451 P530 P3373
P26 P3095
P54 P113 P780
Relation languages spoken (...) instrument religion country of citizenship continent cause of death ethnic group time period member of political party occupation country of origin genre sport place of birth educated at member of record label place of death medical condition ifeld of work uses movement place of burial employer oficial language narrative location country author industry residence parent organization language of work or name part of director headquarters location cast member product (...) produced location of formation located in (...) influenced by named after founded by child unmarried partner diplomatic relation sibling spouse practiced by member of sports team airline hub symptoms
Ratio
All models are trained on Nvidia GTX 2080TI or RTX A6000 graphics cards using PyTorch version 1.11. KGC models use the 1.8 implementation of PyKEEN and the text encoder uses Huggingface’s transformer implementation 4.19. The amount of text samples per epoch varies between diferent model configurations and is a combination of “max contexts” (the total amount of text contexts associated with an entity during training) “max contexts per sample” (the number of text contexts used by a model per training step). Model training is stopped if the performance did not increase on the target metric (i.e. hits@10) on the validation split by more than 0.001 for a few epochs. In the following, the hyperparameters for the models presented in 6 are enumerated.
Ranking
Linking JOINT (single context) - best model selected after random search.
Ranking
Linking Embedding Dims.
Unfrozen Layer Regularizer Weight Contexts per Sample Maximum Contexts Masked Batch Size Learning Rate Weight Decay Seed Embedding Dims.
Unfrozen Layer Regularizer Weight Contexts per Sample Maximum Contexts Masked Batch Size Subbatch Size Learning Rate Weight Decay Seed
Tiny JOINT (multi-context) - best model selected after grid search. OWE (single context) - best model selected after grid search. OWE (multi-context) - best model selected after grid search. OWE reference embeddings - best model selected after grid search.
[1] A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. d. Melo, C. Gutierrez, S. Kirrane, J. E. L.
Gayo, R. Navigli, S. Neumaier, et al., Knowledge graphs, Synthesis Lectures on Data, Semantics, and Knowledge 12 (2021) 1–257. [2] R. Hoehndorf, P. N. Schofield, G. V. Gkoutos, The role of ontologies in biological and biomedical research: a functional perspective, Briefings in Bioinformatics 16 (2015) 1069–1080. URL: https://doi.org/10.1093/bib/bbv011. doi:10.1093/bib/bbv011. [3] M. Ali, M. Berrendorf, M. Galkin, V. Thost, T. Ma, V. Tresp, J. Lehmann, Improving inductive link prediction using hyper-relational facts, in: International Semantic Web [4] B. Shi, T. Weninger, Open-world knowledge graph completion, arXiv preprint arXiv:1711.03438 (2017). [5] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Taylor, Freebase: A collaboratively created graph database for structuring human knowledge, in: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, ACM, New York, NY, USA, 2008, pp. 1247–1250. URL: http://doi.acm.org/10.1145/1376616.1376746. doi:10.1145/1376616.1376746. [6] D. Vrandečić, M. Krötzsch, Wikidata: a free collaborative knowledgebase, Communications of the ACM 57 (2014) 78–85. [7] M. Nickel, V. Tresp, H.-P. Kriegel, A three-way model for collective learning on multirelational data., in: ICML, volume 11, 2011, pp. 809–816. [8] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, in: Advances in neural information processing systems, 2013, pp. 2787–2795. [9] B. Yang, W.-t. Yih, X. He, J. Gao, L. Deng, Embedding entities and relations for learning and inference in knowledge bases, arXiv preprint arXiv:1412.6575 (2014). [10] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, in: International Conference on Machine Learning, 2016, pp. 2071–2080. [11] T. Dettmers, P. Minervini, P. Stenetorp, S. Riedel, Convolutional 2d knowledge graph embeddings, in: Thirty-Second AAAI Conference on Artificial Intelligence, 2018. [12] Z. Sun, Z.-H. Deng, J.-Y. Nie, J. Tang, Rotate: Knowledge graph embedding by relational rotation in complex space, arXiv preprint arXiv:1902.10197 (2019). [13] D. Nathani, J. Chauhan, C. Sharma, M. Kaul, Learning attention-based embeddings for relation prediction in knowledge graphs, arXiv preprint arXiv:1906.01195 (2019). [14] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Advances in neural information processing systems, 2013, pp. 3111–3119. [15] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information. corr abs/1607.04606, URL http://arxiv. org/abs/1607.04606 (2016). [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [17] R. Socher, D. Chen, C. D. Manning, A. Ng, Reasoning with neural tensor networks for knowledge base completion, in: Advances in neural information processing systems, 2013, pp. 926–934. [18] R. Xie, Z. Liu, J. Jia, H. Luan, M. Sun, Representation learning of knowledge graphs with entity descriptions, in: Thirtieth AAAI Conference on Artificial Intelligence, 2016. [19] C. Fu, Z. Li, Q. Yang, Z. Chen, J. Fang, P. Zhao, J. Xu, Multiple interaction attention model for open-world knowledge graph completion, in: International Conference on Web Information Systems Engineering, Springer, 2020, pp. 630–644. [20] X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, J. Tang, Kepler: A unified model for knowledge embedding and pre-trained language representation, Transactions of the Association for Computational Linguistics 9 (2021) 176–194. [21] H. Shah, J. Villmow, A. Ulges, U. Schwanecke, F. Shafait, An open-world extension to knowledge graph completion models, in: Thirty-Third AAAI Conference on Artificial Intelligence, 2019. [22] H. Shah, J. Villmow, A. Ulges, Relation specific transformations for open world knowledge graph completion, in: Proceedings of the Graph-based Methods for Natural Language Processing (TextGraphs), Association for Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 79–84. URL: https://www.aclweb.org/anthology/2020.textgraphs-1.9. [23] Y. Zhou, S. Shi, H. Huang, Weighted aggregator for the open-world knowledge graph completion, in: International Conference of Pioneering Computer Scientists, Engineers and Educators, Springer, 2020, pp. 283–291. [24] D. Daza, M. Cochez, P. Groth, Inductive entity representations from text via link prediction, in: Proceedings of the Web Conference 2021, 2021, pp. 798–808. [25] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, Z. Ives, Dbpedia: A nucleus for a web of open data, in: The semantic web, Springer, 2007, pp. 722–735. [26] K. Toutanova, D. Chen, Observed versus latent features for knowledge base and text inference, in: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, Association for Computational Linguistics, Beijing, China, 2015, pp. 57–66. URL: https://www.aclweb.org/anthology/W15-4007. doi:10.18653/v1/ W15-4007. [27] T. Safavi, D. Koutra, CoDEx: A Comprehensive Knowledge Graph Completion Benchmark, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 8328–8350. [28] F. Hamann, A. Ulges, D. Krechel, R. Bergmann, Open-world knowledge graph completion benchmarks for knowledge discovery, in: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Springer, 2021, pp. 252–264. [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. [30] S. E. Robertson, S. Walker, Some simple efective approximations to the 2-poisson model for probabilistic weighted retrieval, in: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, Springer-Verlag New York, Inc., 1994, pp. 232–241. [31] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization., Journal of machine learning research 12 (2011). [32] M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, S. Sharifzadeh, V. Tresp, J. Lehmann, Pykeen 1.0: A python library for training and evaluating knowledge graph emebddings, arXiv preprint arXiv:2007.14175 (2020). [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035. [34] W. Falcon, The PyTorch Lightning team, PyTorch Lightning, 2019. URL: https://github.
com/Lightning-AI/lightning. doi:10.5281/zenodo.3828935. [35] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [36] B. Mitra, N. Craswell, Neural models for information retrieval, arXiv preprint arXiv:1705.01509 (2017). [37] J. Lin, R. Nogueira, A. Yates, Pretrained transformers for text ranking: Bert and beyond, Synthesis Lectures on Human Language Technologies 14 (2021) 1–325.