Introduction

Relation Extraction (RE) is an essential task in information extraction (IE) that aims to comprehend relational facts between entities in natural language texts [1]. For the first example in Table 1, given an original input and an entity pair of interest (Apple Inc., iPhone), an RE model should be able to predict the relationship between them, i.e., Apple Inc.

𝑝𝑟𝑜𝑑𝑢𝑐𝑡 𝑝𝑟𝑜𝑑𝑢𝑐𝑒𝑑

− −−−−−−−−− → iPhone. The structured knowledge obtained from RE models can support a variety of downstream applications, such as knowledge graph construction or completion [1], question answering [2], and dialogue systems [3].

Previous research usually formulates RE as a pairwise classification task with pre-trained language models (PLMs), in which novel methods have been proposed [4,5]. Recently, large language models (LLMs) demonstrate promising performance in a variety of downstream tasks [6,7] across several paradigms, such as in-context learning (ICL) [8], chain-of-thought (CoT) prompting [9], and fine-tuning. However, they fall short in multiple specification-heavy tasks, including RE, whose performance under particularly ICL is much behind state-of-the-art PLM-based methods [10]. Table 1 gives some examples of mispredicted entity relationships using LLMs. Overall, the reasons why LLMs cannot perform well in RE include their context unawareness, schema misalignment, and world knowledge ignorance:

1. Context Unawareness. The completion of RE requires a thorough and fine-grained comprehension of the information in given contexts. However, LLMs with ICL usually lack fine-grained context awareness, which results in disregarded or erroneous relation prediction [10]. In the first example in Table 1, LLMs should first thoroughly appreciate the context and the connection between "Apple Inc.", "device", and "iPhone"; otherwise, they are unable to determine the relationship between "Apple Inc." and "iPhone".

Table 1

Examples of mispredicted relationships by large language models (LLMs), consisting of three categories: context unawareness, schema misalignment, and world knowledge ignorance.

Error Type Example

Context Unawareness

Input: Apple Inc. is an American multinational corporation [. 2. Schema Misalignment. RE models are required to predict the relationships between entities from a human-labeled, pre-defined schema. However, the number of candidate relationships is typically lengthy, and some relation types are misaligned between LLMs and human expectations [10,11]. In the second example in Table 1, LLMs may confuse the two relation types, "work for" and "part of ", and make incorrect predictions on the relationship between "Armstrong" and "NASA Astronaut Corps". 3. World Knowledge Ignorance. World knowledge usually plays a vital role in RE, particularly in understanding implicit relationships [12] and domain-specific knowledge [13]. However, LLMs suffer in tasks that require rich world knowledge [14] and solely rely on their internal parametric knowledge [10]. In the third example in Table 1, LLMs may predict the relationship as "inventor" rather than "discoverer" without thoroughly understanding the knowledge of "Albert Einstein" and the "theory of relativity".

Knowledge bases (KBs) have been extensively employed in previous RE research. For example, researchers leverage the relationships obtained from Freebase [15] and Wikipedia infoboxes [16] to classify the relationships between entities in texts. However, such relationships are typically noisy and are not faithful to what is described in the given contexts [17]. The following research focuses on denoising and learning context-dependent relationships, such as utilizing natural language inference (NLI) with entailment prediction [18]. Nevertheless, as LLMs have demonstrated their abilities in NLI [19] and natural language reasoning [20], the capability of the combination of KBs and LLMs requires further exploration to design contextual, aligned, and knowledgeable RE models. Moreover, previous research on knowledge-enhanced RE primarily focuses on the English corpus, which limits the adaptability of RE models to different linguistic contexts. This shortage hinders the development of comprehensive IE systems in the multilingual setting.

In this paper, we propose a novel framework for RE to address the aforementioned challenges by making the process contextually aware, schema-aligned, world knowledge-considered. The framework consists of two stages, entity linking and relation inference, that fully leverage the efficacy of KBs and LLMs in this task. As shown in Figure 1, given an original document and two entities of interest, we first link the entities to Wikidata [21], a large-scale multilingual KB, to ascertain the relationship between the entities in the world knowledge and regard it as the candidate relationship in the document. Subsequently, in the second stage, we use the ICL strategy on LLMs to determine whether the candidate relationship actually takes place in the given context.

We conduct extensive experiments in a multilingual setting using three widely used RE datasets: DocRED [22], REBEL [18], and REDFM [23], with three LLMs: GPT-3.5, Llama 2 [24], and Flan-T5-XL [25]. Experimental results demonstrate the effectiveness of our framework on all datasets, where the performance of zero-shot RE on the models outperforms the cases without knowledge by a significant margin. Additionally, it also achieves state-of-the-art performance on all three datasets and outperforms fine-tuned PLM-based methods, validating the efficacy of our proposed framework. We also conduct additional analysis on the effectiveness of knowledge, the impact of scaling up model parameters, and the coverage of knowledge in multilingualism to further demonstrate the effectiveness and generalizability of our proposed method.

The key contributions of this work are summarized as follows:

• We review the key literature on LLM-based RE thoroughly, and we argue that well-behaved RE models should be contextually aware, schema-aligned, and world knowledge-considered. • We propose a novel framework for RE, consisting of two stages: entity linking and relation inference, to fully leverage the efficacy of KBs and LLMs in the RE task. • Experimental results under a multilingual setting demonstrate the effectiveness and generalizability of our method across diverse linguistic contexts with substantial improvements over state-of-the-art baselines.

Related Work

Relation Extraction

RE has been extensively studied over the past years due to its potency in various downstream applications.

Early research in RE focuses on sentence-level RE [26,27], while some later approaches shift to the document level, aiming to comprehend the relationships between entities across multiple sentences [22]. The most commonly used methods for RE are sequence-based techniques, which essentially rely on LSTM-or Transformer-based architectures [4,5], modeling complicated interactions between entities while implicitly capturing long-distance relationships. Furthermore, graph neural networks (GNNs) are also employed in RE due to their efficacy in representing and interacting with structured data. In this process, researchers construct relevant graphs using words, mentions, entities, or sentences as nodes and predict relationships by reasoning on the graph [28,29].

Knowledge-enhanced RE

Knowledge-enhanced RE incorporates external knowledge information to comprehensively understand the relations between entities. Some existing work utilizes external knowledge bases like Freebase and Wikidata to improve the representation by using entity and relation information. Liu et al. [30] injects triples from knowledge graphs into texts, transforming sentences into knowledge-enhanced sentence trees. Chen et al. [31] proposes a knowledge-aware prompt-tuning approach with synergistic optimization that incorporates knowledge from relation labels into RE. External knowledge can bridge the gap between general domain data and domain-specific data, while general domain RE methods are applied in specific domains. Roy and Pan [32] uses an entity-level knowledge graph in pre-trained BERT for clinical RE, integrating medical information.

LLM-based RE

LLM-based RE has also been studied by researchers motivated by the generalized intelligence of LLMs in various downstream tasks, such as information extraction [33], machine translation [7], and adversarial attacks [6]. However, previous research concludes that LLMs typically fall short in the RE task, whose performance is much behind PLM-based approaches [10,34,35]. To overcome this, Zhang et al. [36] proposes QA4RE, a framework to improve the performance of LLM by aligning RE with question

Entity Linking Relation Inference

Yes.

Question: Is there such a relationship "product produced" between "Apple Inc." and "iPhone"? answering (QA) tasks. Wan et al. [37] proposes GPT-RE that utilizes task-aware representations and reasoning logic to improve entity-relationship relevance and the capability of explaining input-label mapping. Li et al. [38] suggests integrating LLM with an NLI module to construct relation triples in response to the abundance of pre-defined relation types and the uncontrollability of LLMs.

Methodology

Problem Formulation

We define our RE task as follows: Given a document 𝐷 consisting of 𝑁 sentences {𝑠 1 , 𝑠 2 , ..., 𝑠 𝑁 } (𝑁 is the number of sentences within the document, and 𝑁 = 1 indicates sentence-level RE) and an entity pair of interest (𝑒 ℎ , 𝑒 𝑡 ), in which 𝑒 ℎ represents the head entity and 𝑒 𝑡 refers to the tail entity, the RE model aims to determine the potential relationship between 𝑒 ℎ and 𝑒 𝑡 from a pre-defined schema. In our task, a KB 𝒦 is leveraged with world knowledge, and an LLM is utilized to identify the existence of the relationship 𝑟 𝒦 retrieved form 𝒦 between 𝑒 ℎ and 𝑒 𝑡 in the given document.

Entity Linking and Querying

In the first stage of our proposed framework, we conduct entity linking and querying to obtain the candidate relationships between the entities of interest, which are regarded as supervision of world knowledge to the given entity pair. Entity linking is the process of linking recognized entity words to an entity in a KB, which is a pioneering step in extracting construction information from unstructured text [39]. In our framework, we link the labeled entity mentions to Wikidata [21], a large-scale multilingual KB. Once the entities are linked, we introduce a query based on SPARQL 1 to retrieve the relationships between the linked entities and regard it as the candidate relationship between them. For the datasets whose entities are annotated with coreference chains, we iterate the head and tail entities until a pair of entities can be linked to Wikidata.

Relation Inference using LLMs

After obtaining the candidate relationship between the entity pair of interest, in the second stage of our proposed framework, we adopt LLMs to identify whether the relationship actually occurs in the given context. Specifically, we leverage the ICL strategy [9] that conditions LLMs on a natural language instruction and formulate the task as a QA task due to the capacity of LLMs to answer natural questions.

In accordance with the entity linking results in the first stage, we design separate prompts for the entity pairs that have or have not been found potential relationships, and the prompts with separate examples are illustrated in Tables 2 and 3. For the entity pairs that have been found candidate relationships in the KB, we ask LLMs to determine whether they actually exist in the given context. Otherwise, we ask the LLMs to classify the relationships between the entities from the schemas directly. This framework enables us to carry out a contextual, aligned, and knowledgeable RE process: it regards the knowledge in KBs as supervision, and the inference with LLMs makes the predictions with respect to the given contexts. Furthermore, since KBs are human-constructed world knowledge, their candidate knowledge also conforms to human-defined schemas.

Experiments and Analysis

Datasets

We conduct our experiments on the following three datasets, whose dataset statistics are organized in Table 4: • DocRED [22] is a document-level human-annotated RE dataset constructed from Wikipedia and Wikidata. Since at least 40.7% of relational facts in DocRED can only be extracted from multiple sentences, it requires models to comprehensively model the whole document to determine the relationships between entities. • REBEL [18] is a distantly supervised dataset, hyperlinking with Wikidata and Wikipedia for relation extraction. It employs an NLI model to filter noise and address relations that are not entailed by the Wikipedia text through entailment prediction.

• REDFM [23] is constructed for multilingual RE that involves seven languages. Different from the REBEL dataset, REDFM not only applies NLI to filter noise but also conducts manual filtering to ensure the annotation quality. We select the English (EN), Spanish (ES), and German (DE) subsets to validate the performance of our framework in a multilingual setting.

Following the previous work in LLM-based RE [10,34], we sample a subset from the validation set of DocRED and the test set of REBEL and REDFM to validate the performance of our method against baselines. We evaluate the performance of the experimented models using micro F1-score.

Baselines

We compare the performance of our proposed method on RE against the following baselines:

• KD-DocRE [4] is a semi-supervised framework for document-level RE that incorporates axial attention, adaptive focal loss, and knowledge distillation to capture the interdependency among entity-pairs. It addresses the class imbalance problem and the differences between human annotated and distantly supervised data in document-level RE. • DREEAM [5] is a memory-efficient approach for improving document-level RE by incorporating evidence and offering a self-training strategy, addressing high memory consumption and limited annotated data availability in document-level RE.

Experimental Setup

We conduct our experiments on three commonly used multilingual LLMs: GPT-3.5, Llama 2 [24], and Flan-T5 [25], and we access the models with different approaches and settings. For GPT-3.5, we call the API by OpenAI2 and select the gpt-3.5-turbo-instruct checkpoint due to its ability to interpret and execute human instructions seamlessly. For Llama 2 (Llama-2-7b-chat-hf) and Flan-T5 (flan-t5-xl), the models are retrieved from the HuggingFace repository3 . To mimic the randomness of human reasoning and produce relatively stable outputs, we set the temperature of GPT-3.5 and Llama 2 as 0.2.

All experiments are conducted on a single NVIDIA GeForce RTX 4090 graphics card.

Main Results

The experimental results of our proposed framework under different LLMs with and without external knowledge are given in Table 5. From the table, we make the following observations: First, without the incorporation of external knowledge, LLMs have been shown to fall short in the RE task and their performances are much behind those of the state-of-the-art baseline models. The results are also consistent with the previous work [10], indicating the correctness of our implementation. Among the three LLMs, Flan-T5 achieves the best performance and is remarkably close to the deliberated baseline models, indicating its excellent document-level understanding and relation reasoning ability. Llama 2 achieves the worst performance, with its results close to zero. We sample 50 outputs of Llama 2 and compare them with the ground truths. We conclude that this phenomenon is attributed to the excessively uncontrollable and flexible nature of its output compared to the rest of the models.

Second, after incorporating the external knowledge into the models, LLMs exhibit remarkable performance across all datasets, in which the average improvements of GPT-3.5, Llama 2, and Flan-T5 are 52.90, 45.90, and 11.82, respectively. Notably, the performance of Flan-T5 under the zero-shot setting achieves state-of-the-art results on all datasets, which is also better than the deliberated, finetuned PLM-based methods. GPT-3.5 improves the most among the models, but there is still room between the performance and the PLM-based methods. These results demonstrate that the performances of LLMs with external knowledge in a zero-shot setting can be comparable to or even surpass the finetuned PLM-based method on the RE task. They also underscore the effectiveness of our approach in multilingual settings, which is not limited to the English context.

Finally, the performance of the experimented models is consistent regardless of the language and the existence of external knowledge. Flan-T5 consistently achieve the best performance across all datasets, and Llama 2 exhibits comparatively lower performance, indicating that Flan-T5 has a better performance and a robust generalization advantage when dealing with the RE task and can be regarded as an ideal model in real-world application, while Llama 2 requires additional improvements for higher performance.

Additional Analysis

Effectiveness of External Knowledge First, we analyze the effectiveness of the external KB in our proposed method. Since not all entity pairs can be linked to Wikidata, we calculate the percentage of correct prediction of LLMs with and without the incorporation of external knowledge, denoted as 𝑃 𝑤/𝑘𝑛𝑜𝑤 and 𝑃 𝑤/𝑜𝑘𝑛𝑜𝑤 , calculated as:

𝑃 𝑤/𝑘𝑛𝑜𝑤 = # of Correct Prediction # of Entity Pairs Linked to Wikidata ,(1)

We visualize the calculation in Figures 2 and 3, respectively. Our findings show a significant difference in performance with and without incorporating knowledge across all datasets and LLMs. Specifically, the correct prediction of LLMs with external knowledge is as high as more than 80%, while the results without knowledge are inferior, among which only Flan-T5 can exceed 40%. Because of the flexible and uncontrollable nature of Llama 2, its correct predictions without external knowledge are nearly zero, while after incorporating knowledge, its results improve to over 80%. The performance difference indicates that a performance gap exists across different models-while all models can achieve similar performance with external knowledge, their results are dominated by the relation classification result without external knowledge, indicating that LLMs are good inferencers but not classifiers for entity relationships. Moreover, although the performance of LLMs is better on the REBEL dataset with the incorporation of external knowledge, it becomes worse on the models without knowledge due to the large relation schema of the dataset. This remains a challenge for future research to design better methods to deal with the entity pairs that cannot link to the KBs.

Scaling Law

We also analyze whether the performance of LLM-based RE can benefit from scaling up the model parameters. Specifically, we select the Flan-T5 series models with four different model sizes: Flan-T5-Small (80M), Flan-T5-Base (250M), Flan-T5-Large (780M), and Flan-T5-XL (3B) and evaluate the performance of the models with and without external knowledge. As shown in Figure 4, a clear positive scaling effect exists in LLM-based RE, i.e., fine-tuned larger models achieve better performance in the RE task. We can also observe the role of external knowledge. After incorporating external knowledge into the LLM, the increase in the number of model parameters has a smaller impact on the results. Moreover, with external knowledge, Flan-T5-Small can surpass Flan-T5-Large, and Flan-T5-Base can exceed Flan-T5-XL's performance without external knowledge. This validates the effectiveness of both LLMs and external knowledge when handling the RE task.

Coverage of Knowledge in Multilingualism

Given the multilingual support of the chosen LLMs, we extend our investigation to include multilingual RE experiments using the REDFM dataset. The experimental outcomes, as summarized in Table 5, reveal subpar performance when the LLMs attempt multilingual RE tasks directly. However, integrating external knowledge significantly enhances performance, prompting us to explore the coverage of Wikidata for the selected multilingual dataset. To this end, we conduct supplementary experiments on REDFM-EN, REDFM-DE, and REDFM-ES to assess the percentage of samples that could be linked to Wikidata for external knowledge, as illustrated in Figure 5. The results indicate that a relatively high proportion of samples across the three languages could be covered by Wikidata, with coverages nearing or exceeding 85%, specifically 85.5%, 84.5%, and 84.1% for REDFM-EN, REDFM-DE, and REDFM-ES, respectively. The remaining 14.5%, 15.5%, and 15.9% are attributed to entries not indexed by Wikidata, with a small fraction being inaccessible due to unstable network connections.

Conclusion and Future Work

In this paper, we propose a novel framework to address the current challenges of LLMs falling short in RE tasks because of their context-unawareness and schema-misalignment, with world knowledge ignorance. It consists of two stages: entity linking and relation inference, fully leveraging the efficacy of KBs and LLMs in this task. We conduct experiments in a multilingual setting using three datasets and three LLMs to validate the effectiveness of our framework, where the zero-shot RE with world knowledge outperforms those without that by a significant margin and achieves state-of-the-art performance on all experimental datasets, even better than fine-tuned PLM-based methods, indicating the effectiveness of our proposed framework. We also conduct additional analysis on the effectiveness of knowledge, the impact of scaling up model parameters, and the coverage of knowledge in multilingualism to further demonstrate the effectiveness and generalizability of our proposed method. In the future, we will conduct more detailed analysis on other related tasks, such as event relation extraction, to further validate the effectiveness and generalizability of our proposed method.