The task of zero-shot relation extraction focuses on the extraction of relations not seen during training time. Commonly, additional information about the relation such as the relation name or a description of the relation is utilised. In this work, we analyze whether a relation extractor can benefit from the inclusion of fine-grained type information about the involved entities. This is based on the intuition that relation descriptions might contain ontological information on the domain and range of the entity types that are usually put into relation. For that, we follow a cross-encoding setup where we encode both, the entity information and relation information, as one sequence and learn to score the representation. We examine this method on several datasets and show that the inclusion of the fine-grained type information leads to an improvement in performance.
The problem of relation extraction can be defined as follows: Given an input text , an annotated head ℎ and tail entity , identify the correct relation as expressed in the text. Zero-shot relation extraction separates the set of relations encountered during training from the ones encountered during inference. Hence, during training time, the set of available relations is train, while during test time, the set is test. It holds that train ∩ test = ∅. Also, no annotated examples containing any relations in set test are available during inference time. Additional information defining the relation is available. We assume labels, descriptions and type information on entities to be available.
To study the impact of fine-grained type information, we opt to extend a simple but powerful model
introduced by Lan et al. [
We denote the types of the head entity by ℎ and the types of the tail entity by . Additionally, for each type of an entity, we extract the label describing the type (e.g., human for Q5). The input to the model then consists of four diferent segments. The first segment describes the head entity:
Head Entity : {ℎ} with Types : {ℎ} and the second segment describes the tail entity:
Tail Entity : {} with Types : {} where □ denotes the label of the head entity ℎ or tail entity . □ is the concatenation of the labels of the types of the head or tail entity □ = ⨁︀∈□ .
The third segment gives information on the input text:
Context : {}
{} defined as {} where denotes the label of the relation and is the description of the relation . All segments are then combined into a single coherent text as follows: [CLS] Given the Head Entity : {ℎ} with Types : {ℎ}, Tail Entity : {} with Types : {} and Context : {}, the context expresses the relation [SEP] {} defined as {} [SEP]
The whole text is then fed into an encoder-only model () which returns a sequence of vectors [] . . . [ ]. The vector [] is then taken and fed a input to a linear layer which then returns a ifnal score.
= ([])
This is done for each potential relation, which gives us |test| scores. The highest score is taken as the predicted relation. All potential relations are known beforehand. During training, the model is optimized using cross-entropy loss. Each example contains a single positive relation. The model trains to diferentiate it against other relations by comparing it against incorrect relations. For that, other relations are sampled and used as negative examples. 2
An overview of the model can be found in Figure 1.
We evaluate the model on two popular datasets, FewRel and Wiki-ZSL. Both datasets were annotated
on Wikipedia article texts. FewRel is originally a few-shot relation extraction dataset annotated by Han
et al. [
In each dataset, the set of relations in the training and test dataset is disjoint and randomly assigned. Three diferent settings are examined per dataset. Each setting considers a diferent number of relations in the train, validation and test set. The number of relations in the validation/test set varies between = 5, = 10 and = 15 relations. These relations are randomly picked and the remaining relations are assigned to the training set.
To handle the considerable noise induced by the random selection of the relations, the dataset for = 5, = 10 and = 15 were randomly split into train, validation and test sets for five times. A method is evaluated on each split and the results are averaged.
As metrics, precision, recall and F1 are calculated. All metrics are computed in a macro setting which means that for each relation the precision, recall and F1 are calculated and then averaged over all relations.
We compare our method, called TMC-BERT, against several methods: CIM [
BERT-base-case was used as the model to stay comparable to MC-BERT. The model was fine-tuned on two NVIDIA A6000s with a batch size of 48 and a learning rate of 5 − 5.
As can be seen in Table 1, the incorporation of type-related information leads to a large increase in
performance on several datasets in comparison to regular MC-BERT. On Wiki-ZSL, the performance
increases vary between 6 and nearly 8 F1 points. The type-related information has a great impact on,
both, recall and precision. On FewRel, the performance increases when considering 5 or 15 unseen
relations. However, the performance increases are less pronounced. In comparison to the current SOTA
method by Tran et al. [
To examine what changes lead to the large increase in performance, we conducted an ablation study on the incorporation of diferent kinds of information. Here, we diferentiated between three cases:
2. TMC having the types of the subject and object entity removed (TMC-BERT w/o types) 3. TMC having the description of the relation removed (TMC-BERT w/o desc.)
As can be seen in Table 2, the addition of the relation description alone was the least beneficial type of information. Adding information on relation types leads to a larger improvement, probably as the pre-trained model already associates specific types with certain relation labels. Finally, the ablation study shows that the relation description and fine-grained entity type information complement each other, as using each separately does not lead to as large a decrease in performance as using them together.
As it is not realistic that fine-grained type information is always available, we also evaluate the model when identifying entity types using an entity linker (EL). For that, we train the model with known
CIM
ZS-BERT
Tran et al. (2022) RelationPrompt NG
RelationPrompt
RE-Matching
DSP-ZRSC Tran et al. (2023)
MC-BERT TMC-BERT
CIM
ZS-BERT
Tran et al. (2022) RelationPrompt NG
RelationPrompt
RE-Matching
DSP-ZRSC Tran et al. (2023)
MC-BERT TMC-BERT
CIM
ZS-BERT
Tran et al. (2022) RelationPrompt NG
RelationPrompt
RE-Matching
DSP-ZRSC Tran et al. (2023)
MC-BERT
TMC-BERT Model TMC-BERT w/o desc.
TMC-BERT w/o types TMC-BERT TMC-BERT w/o desc.
TMC-BERT w/o types TMC-BERT TMC-BERT w/o desc.
TMC-BERT w/o types TMC-BERT 49.63 71.54 87.48 51.78 70.66 78.19 94.1 94.50 80.28 90.11 46.54 60.51 71.59 54.87 68.51 74.39 80.0 85.43 72.81 81.21
P 58.05 76.96 87.11 72.36 90.15 92.82 93.4 96.36 90.82 93.94 47.39 56.92 64.41 66.47 80.33 83.21 80.7 81.13 86.57 84.42
P
entity types but evaluate with the entity types as retrieved by an entity linker. As an entity linker, we
use ReFinED [
5 10 15
person’s voice type. expected values: so- main use of the subject (includes current prano, mezzo-soprano, contralto, coun- and former usage) tertenor, tenor, baritone, bass (and derivatives)
Commonly, relation extraction is tackled as classification problem. Usually, the input text is encoded and
a classification head is attached. To encode text, CNNs [
Type information was considered in previous works focusing on relation extraction but these works
either used very broad types or did not tackle zero-shot relation extraction [
Some methods model the problem as a textual entailment problem [
The method by Lan et al [
In this work, we examined the impact of fine-grained type information on the zero-shot relation extraction problem. Diferent from past methods, we employed fine-grained type information as additional information and showed that combining this information with the description of the relation leads to a synergistic efect, improving the performance overall. We believe that this is the case because the description information provides valuable ontological information on the domain and range of a relation. This domain and range are then compared against the fine-grained type information of the interacting entities. Furthermore, we validated whether the increase in performance did indeed spring from the combination of type and relation description information which is indeed the case. Finally, we studied the impact of using an entity linker to retrieve the entity types. While it leads to a decrease, the performance often still surpasses the current SOTA in the most complex setting considerably.
In future works, we want to tackle multiple problems. First, it is not certain that one has access to finegrained type information during inference. Therefore, we want to examine, whether the performance of a trained entity typer is suficient to produce similar results. Secondly, the current architecture follows a cross-encoding approach. While this is not a problem when one encounters only a few relations during inference, in real-world use cases this is not typically the case. There are hundreds of potential diferent relations that could be encountered during inference. Cross-encoding the text with each one leads to a substantial computational efort. We want to examine whether the relation candidate generation module might also benefit from fine-grained type information. Also, the training process currently only trains the model by randomly sampling other relations. Choosing the relationships in a smarter way might lead to additional improvement. Finally, the impact of fine-grained entity types from other knowledge graphs needs to be evaluated.
This project was supported by the House of Computing and Data Science (HCDS) of the Hamburg University within the Cross-Disciplinary Lab programme. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Association for Computational Linguistics, 2021, pp. 1199–1212. doi:10.18653/V1/2021.EMNLP-MAIN.92. [24] A. Obamuyide, A. Vlachos, Zero-shot relation classification as textual entailment, in: Proceedings of the first workshop on fact extraction and VERification (FEVER), 2018, pp. 72–78.