Introduction

Identifying the relation that is expressed between entities is a very important subproblem of various downstream tasks. For instance, it is critical to handle semantic-web-related tasks such as knowledge graph question answering or knowledge graph population. Usually, it is assumed that the encountered relations are known before. Zero-shot relation extraction breaks with this assumption. During inference time, the goal is to extract entirely new relations not seen before during training time.

With the establishment of pre-trained models, this goal becomes achievable. Those models are trained on large corpora of textual data in an unsupervised way. In zero-shot relation extraction, one assumes that some information on the new relations is available. The simplest kind of information is a label describing the relation. But this only works if the relation label co-occurs with a similar context as encountered during the training of the pre-trained models. If this is not the case, using additional information such as a description of the relation is necessary.

In this work, we analyse the impact of combining fine-grained type information and the relation description on the relation extraction performance. This is based on the assumption that the descriptions contain valuable information on the types of the involved entities. For example, the description of the relation director states director(s) of film, TV-series, stageplay, video game or similar. Therefore it is clear, that the relation should not be used when talking about board members of a company, also sometimes referred to as directors. We incorporate fine-grained type information extracted from Wikidata together with the relation descriptions in the relation extraction process. 1 The contributions are:

• Zero-shot relation extraction model using fine-grained type information and relation descriptions • Ablation study on the impact of fine-grained type information and relation descriptions on the performance

Methodology

Problem Definition

Method

To study the impact of fine-grained type information, we opt to extend a simple but powerful model introduced by Lan et al. [1]. Hence, we cross-encode the information of the text and the relation information in a single input. Different from their work, we do not solely rely on the relation label but also include the relation description. Additionally, we assume the existence of fine-grained types for both, the head and the tail relation, extracted using the P31 relation in Wikidata. We include the relation description under the assumption that it contains valuable ontological information referring to the fine-grained types of the considered entities. For example, for the relation shipping port, the description is shipping port of the vessel (if different from "ship registry"): For civilian ships, the primary port from which the ship operates ...

We denote the types of the head entity by 𝒯 ℎ and the types of the tail entity by 𝒯 𝑡 . Additionally, for each type of an entity, we extract the label describing the type (e.g., human for Q5). The input 𝑥 to the model then consists of four different segments. The first segment describes the head entity:

Head Entity : {𝑙 ℎ } with Types : {𝑇 ℎ } and the second segment describes the tail entity:

Tail Entity : {𝑙 𝑡 } with Types : {𝑇 𝑡 } where 𝑙 □ denotes the label of the head entity ℎ or tail entity 𝑡. 𝑇 □ is the concatenation of the labels of the types of the head or tail entity 𝑇 □ = ⨁︀ 𝑢∈𝒯 □ 𝑙 𝑢 . The third segment gives information on the input text:

Context : {𝑐}

The final segment gives information on the relation:

{𝑙 𝑟 } defined as {𝑑 𝑟 }

where 𝑙 𝑟 denotes the label of the relation 𝑟 and 𝑑 𝑟 is the description of the relation 𝑟.

All segments are then combined into a single coherent text as follows: The whole text is then fed into an encoder-only model 𝑓 (𝑥) which returns a sequence of vectors 𝑒 [𝐶𝐿𝑆] . . . 𝑒 [𝑆𝐸𝑃 ] . The vector 𝑒 [𝐶𝐿𝑆] is then taken and fed a input to a linear layer which then returns a final score.

𝑠 𝑟 = 𝑙(𝑒 [𝐶𝐿𝑆] ) An overview of the model can be found in Figure 1.

Evaluation

We evaluate the model on two popular datasets, FewRel and Wiki-ZSL. Both datasets were annotated on Wikipedia article texts. FewRel is originally a few-shot relation extraction dataset annotated by Han et al. [2]. The dataset was modified for zero-shot purposes by Chia et al. [3]. They split the training, validation and test examples by their relations into disjoint sets. Wiki-ZSL is a zero-shot relation extraction dataset created by Chen et al. [4] based on the Wiki-KB [5]. As the entities and relations in both datasets are linked to Wikidata, we focus on it as the knowledge graph providing the fine-grained entity types.

In each dataset, the set of relations in the training and test dataset is disjoint and randomly assigned. Three different settings are examined per dataset. Each setting considers a different number of relations in the train, validation and test set. The number of relations in the validation/test set varies between 𝑚 = 5, 𝑚 = 10 and 𝑚 = 15 relations. These relations are randomly picked and the remaining relations are assigned to the training set.

To handle the considerable noise induced by the random selection of the relations, the dataset for 𝑚 = 5, 𝑚 = 10 and 𝑚 = 15 were randomly split into train, validation and test sets for five times. A method is evaluated on each split and the results are averaged.

As metrics, precision, recall and F1 are calculated. All metrics are computed in a macro setting which means that for each relation the precision, recall and F1 are calculated and then averaged over all relations.

We compare our method, called TMC-BERT, against several methods: CIM [6] solves the task as a textual entailment problem where the relation descriptions and the input sentence are given to a Natural Language Inference model to classify whether the input sentence entails the relation description. This is done for all potential relations and the highest scoring is taken. ZS-BERT [7] encodes the input sentence as well as the relation descriptions into a dense vector space. An nearest neighbor search is conducted over all the encodings of the relation descriptions given the input sentence. The closest relation encoding is the final relation. Tran et al. (2022) [8] again encode the input sentence and relation descriptions into a dense vector space. They additionally employ a contrastive-learning inspired loss on the input sentence and relation encodings. The final scoring is achieved by concatenating the relation encoding and the sentence encoding and feeding it into a linear layer. RE-Matching [9] encodes the input sentence and relation descriptions as well but uses feature distillation to calculate a similarity score based on more fine-grained feature interactions. RelationPrompt [3] relies on a generative model to generate synthetic data as additional training samples. At the same time, the generative model is also used to generate a relation given the sentence and the two entities as input. We compare against the model with (RelationPrompt) and without (RelationPrompt NG) synthetic training data. MC-BERT [1] models the relation extraction similar to us as a multiple-choice problem where the input sentence and the relation label are rearranged together in a natural sentence, encoded and scored. DSP-ZRSC [10] solves the problem via Discriminative Soft Prompting where the input text, the entities and all relation labels are concatenated, fed into a prompt discriminative language model and each relation label is scored. Tran et al. (2023) [11] solve it as a representation learning problem and introduce a second loss term incorporating the degree of correlation between sentences and relations.

BERT-base-case was used as the model to stay comparable to MC-BERT. The model was fine-tuned on two NVIDIA A6000s with a batch size of 48 and a learning rate of 5𝑒 − 5.

Results

As can be seen in Table 1, the incorporation of type-related information leads to a large increase in performance on several datasets in comparison to regular MC-BERT. On Wiki-ZSL, the performance increases vary between 6 and nearly 8 F1 points. The type-related information has a great impact on, both, recall and precision. On FewRel, the performance increases when considering 5 or 15 unseen relations. However, the performance increases are less pronounced. In comparison to the current SOTA method by Tran et al. [11], TMC-BERT considerably surpasses its performance when confronted with 15 unseen relations. This is the most complex setting as much fewer examples and relations are available during training while more potential relations are encountered during inference. Here, the additional type information helps a lot. Furthermore, the inclusion of fine-grained type information is orthogonal to the properties of the method by Tran et al. [11]. Their method could benefit from it as well.

Ablation study

To examine what changes lead to the large increase in performance, we conducted an ablation study on the incorporation of different kinds of information. Here, we differentiated between three cases:

1. TMC-BERT 2. TMC having the types of the subject and object entity removed (TMC-BERT w/o types) 3. TMC having the description of the relation removed (TMC-BERT w/o desc.)

As can be seen in Table 2, the addition of the relation description alone was the least beneficial type of information. Adding information on relation types leads to a larger improvement, probably as the pre-trained model already associates specific types with certain relation labels. Finally, the ablation study shows that the relation description and fine-grained entity type information complement each other, as using each separately does not lead to as large a decrease in performance as using them together.

Entity Linking impact

As it is not realistic that fine-grained type information is always available, we also evaluate the model when identifying entity types using an entity linker (EL). For that, we train the model with known entity types but evaluate with the entity types as retrieved by an entity linker. As an entity linker, we use ReFinED [12]. As can be seen in Table 3, the performance diminishes when using types identified through entity linking. On Wiki-ZSL, the performance is still surpassing the existing SOTA results at all times. On FewRel, the performance is still greater when only confronted with five relations but decreases more when having to predict 10 or 15 relations. One reason might be that the entity linking performance is lower on FewRel than on Wiki-ZSL.

Table 3

Results on FewRel and Wiki-ZSL when using an entity linker

Case study

Related Work

Commonly, relation extraction is tackled as classification problem. Usually, the input text is encoded and a classification head is attached. To encode text, CNNs [13], RNNs [14] or transformers [15] are usually employed. Recently, pre-trained models have been fine-tuned on the relation extraction task. Due to the fixed classification head, such trained models are not flexible enough to handle new relations [16]. Hence, when targeting zero-shot relation extraction other methods are necessary. Representation-learningbased methods [11,8,4,9,11] try to embed the textual information and the relational information in the same vector space. For that, relational information such as labels or descriptions are usually used to get a representation of the relation. The goal is to learn representations such that the representation of the true relation resides close to a representation of the text in the vector space while the representation of false relations is far away. Recently, generative language models have been increasingly utilized for the task [17,18,3,10]. Here, the model is prompted with the input text as well as information on the potential relations. The model is then fine-tuned to either generate the relation as expressed in the input text or a full triple consisting of the two entities and relation. For example, Chen et al. [18] model it as solving a Masked Language Modelling problem. Also, generative models were applied to generate synthetic training data for relation extraction [3]. Type information was considered in previous works focusing on relation extraction but these works either used very broad types or did not tackle zero-shot relation extraction [19,20,21].

Some methods model the problem as a textual entailment problem [22,23,24]. Here, the idea is that a model that is pre-trained on the textual entailment task is directly applied to the relation extraction task. The assumption is that the model can identify whether the textual information entails the relation description.

The method by Lan et al [1] models relation classification as a multiple-choice problem where the text is encoded with relation information and a score is calculated. This is done for all relations and the relation with the highest score is taken. We extend this approach.

Conclusion and Future Work

In this work, we examined the impact of fine-grained type information on the zero-shot relation extraction problem. Different from past methods, we employed fine-grained type information as additional information and showed that combining this information with the description of the relation leads to a synergistic effect, improving the performance overall. We believe that this is the case because the description information provides valuable ontological information on the domain and range of a relation. This domain and range are then compared against the fine-grained type information of the interacting entities. Furthermore, we validated whether the increase in performance did indeed spring from the combination of type and relation description information which is indeed the case. Finally, we studied the impact of using an entity linker to retrieve the entity types. While it leads to a decrease, the performance often still surpasses the current SOTA in the most complex setting considerably.

In future works, we want to tackle multiple problems. First, it is not certain that one has access to finegrained type information during inference. Therefore, we want to examine, whether the performance of a trained entity typer is sufficient to produce similar results. Secondly, the current architecture follows a cross-encoding approach. While this is not a problem when one encounters only a few relations during inference, in real-world use cases this is not typically the case. There are hundreds of potential different relations that could be encountered during inference. Cross-encoding the text with each one leads to a substantial computational effort. We want to examine whether the relation candidate generation module might also benefit from fine-grained type information. Also, the training process currently only trains the model by randomly sampling other relations. Choosing the relationships in a smarter way might lead to additional improvement. Finally, the impact of fine-grained entity types from other knowledge graphs needs to be evaluated.