1. Introduction

Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema

Xiaohan Feng

Xixin Wu

Helen Meng

0 0 Department of System Engineering and Engineering Management, Chinese University of Hong Kong

We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base. An ontology is authored by generating Competency Questions (CQ) on knowledge base to discover knowledge scope, extracting relations from CQs, and attempt to replace equivalent relations by their counterpart in Wikidata. To ensure consistency and interpretability in the resulting KG, we ground generation of KG with the authored ontology based on extracted relations. Evaluation on benchmark datasets demonstrates competitive performance in knowledge graph construction task. Our work presents a promising direction for scalable KG construction pipeline with minimal human intervention, that yields high quality and human-interpretable KGs, which are interoperable with Wikidata semantics for potential knowledge base expansion.

eol>Knowledge Graph Relation Extraction Large Language Model Wikidata Interpretable AI

1. Introduction

Knowledge Graphs (KGs) are structured representations of information that capture entities and their relationships in a graph format. By organizing knowledge in a machine-readable way, KGs enable a wide range of intelligent applications, such as semantic search, question answering, recommendation systems, and decision support [ 1 ]. The ability to construct highquality, comprehensive KGs is thus critical for harnessing the power of these technologies across various domains.

Traditionally, the process of constructing KGs has relied heavily on manual efort by domain experts to define the relevant entities and relationships, populate the graph with valid facts, and ensure logical consistency [ 2 ]. However, this manual curation approach is time-consuming, expensive, and dificult to scale to large, evolving domains. There is a strong need for (semi)automatic methods that can aid the KG construction process by extracting structured knowledge from unstructured data sources such as text.

Recent years have seen growing interest in leveraging Large Language Models (LLMs) for various knowledge capture and reasoning tasks [ 3 ]. Pre-trained on vast amounts of text data, LLMs can generate fluent natural language and have been shown to memorize and recall factual knowledge [ 4 ], [5]. However, directly applying LLMs to KG construction still faces several challenges. First, LLMs may generate inconsistent or redundant facts due to the lack of an explicit, unified schema [ 6 ]. Second, the generated KGs may be incomplete or biased towards the knowledge present in the LLM’s training data, which may not fully cover the target domain, especially for proprietary documents not included in pre-training set. Finally, it can be challenging to integrate LLM-generated KGs with existing knowledge bases due to misalignment with standard ontologies.

In this work, we propose a novel approach that harnesses the reasoning power of LLMs and the structured schema of Wikidata to construct high-quality KGs for proprietary knowledge domains. Our approach begins by discovering the scope of knowledge through the generation of Competency Questions (CQ) and answers from unstructured documents.

We then summarize the relations and properties from these QA pairs into an ontology, matching candidate properties against those defined in Wikidata and extending the schema as needed. Finally, we use the resulting on- Figure 1: Flowchart of proposed approach. tology to ground the transformation of CQanswer pairs into a structured KG. By incorporating the Wikidata schema into our pipeline and grounding generation of KG on the same ontology, we aim to reduce redundancy, leverage the implicit knowledge captured during LLM pretraining while improving interpretability, and ensure interoperability with public knowledge bases. The generated KGs could be parsed with RDF parsers and used in downstream applications, or audited for correctness.

The main contributions of this work are as follows: 1. We propose a novel ontology-grounded approach to LLM-based KG construction that leverages ontology based on Wikidata schema to guide the extraction and integration of knowledge from unstructured text. 2. We introduce a pipeline that combines competency question generation, ontology alignment, and KG grounding to systematically construct high-quality KGs that are consistent, complete, and interoperable with existing knowledge bases. 3. We demonstrate the efectiveness of our approach through experiments on benchmark datasets, showing improvements in KG quality compared to traditional methods alongside with interpretability and utility of generated KGs.

2. Literature Review

Knowledge graph construction has been an active area of research in recent years, with a wide range of approaches proposed for extracting structured knowledge from unstructured data sources [ 2 ]. Early methods relied heavily on rule-based systems and hand-crafted features to identify entities and relations in text [ 7 ]. With the advent of deep learning, neural networkbased approaches have become increasingly popular, enabling more flexible and scalable KG construction [ 8 ].

One prominent line of work focuses on using distant supervision to automatically generate training data for relation extraction [ 9 ]. These methods assume that if two entities are mentioned together in a sentence and also appear in a knowledge base as subject and object of a relation, then that sentence is likely to express the relation. While distant supervision has been shown to be efective at scale, it often sufers from noise and incomplete coverage.

Another important direction is the development of unsupervised and semi-supervised methods for KG construction [ 10 ]. These approaches aim to reduce the reliance on large amounts of labeled data by leveraging techniques such as bootstrapping, graph-based inference, and representation learning. However, they often struggle with consistency and quality control issues. More recently, there has been growing interest in using large language models for KG construction [ 11 ], [ 12 ], [ 13 ]. These methods take advantage of the vast knowledge captured in pretrained Language Models (LM) to generate KG triples through prompt engineering and fine-tuning. While promising, these LM-based approaches only produces triplets without canocalization, which makes portability and interoperability dificult. Additionally, some methods rely on vector-based similarity measures to deduce relationships between entities in KG, which yields good performance but falls short in interpretability [ 14 ].

As mentiond in Introduction, despite the significant progress in KG construction and LLM applications, performance, interpretability, coverage of proprietary documents, and interaction with other knowledge base remain issues. Our pipeline address these by grouding KG generation on ontology based on Wikidata schema, which ensures that output KG is human-readable and makes integrating with Wikidata or other KG easier; In the experiments below we show that these benefits can be also achieved on private documents with decent performance.

3. Method: Ontology-grounded KG Construction

Our proposed approach for ontology-grounded KG construction using LLMs consists of four main stages: 1) Competency Question Generation, 2) Relation Extraction and Ontology Matching, 3) Ontology Formatting, and 4) KG Construction. Figure 1 provides an overview of the pipeline.

3.1. Competency Question (CQ)-Answer Generation

The first step in our pipeline is to generate a set of competency questions (CQs) and answers that capture the key information needs of the target domain. We employ an LLM to generate CQs based on the input documents. The LLM is provided with a set of instructions and examples to guide the generation process, encouraging the creation of well-formed, relevant questions that can be answered using the given documents. This step helps to scope the KG construction task within the knowledge domain, and ensure that the resulting KG aligns with the intended use cases. This also allows further ontology expansion by incorporating user-submitted domaindefining questions when interacting with the knowledge base, which serves as a user friendly interface of refining ontology by submitting new CQs and use our proposed pipeline to attach the incremental knowledge scope to existing ontology.

3.2. Relation Extraction and Ontology Matching

During our preliminary experiments of prompting LLMs to directly generate ontology on documents, we noted that the LLM spontaneously recalled Wikidata knowledge in response, consistent with previous works [ 15 ]. In our preliminary experiments, this behaviour also transfers to small 7B/14B models.

Following this direction, in the second step we extract relations from CQs and match them against Wikidata properties to better elicit model memories on Wikidata when constructing and using ontology. We first prompt LLMs to extract properties from CQ and write brief description on usage of extracted properties, including their domain and range, following editing guidelines of Wikidata. To match these properties against existing entries in Wikidata ontology, we prepopulate a candidate property list with all Wikidata properties after filtering out properties related to external database/knowledge base IDs. These extracted properties are then matched against the candidate list by a vector similarity search between description of properties. The representation for property is sentence embedding constructed from description of properties, and the top 1 closest candidate is retrieved for each extracted property. This matching result between each pair of extracted property and matched top 1 candidate is then vetted by LLM to see if they are really semantically similar as a final deduplication step. If a match is validated, the candidate property is added to the final property list; otherwise, the newly minted property is kept in the final list if we allow expansion from the candidate property list derived from Wikidata, and discarded when the final property list is required to be a subset of candidate property list. The first scenario is suitable for cases when no prior schema is known for the domain and some new properties outside of common ontology are expected, whereas the latter is for a known target list of possible properties.

3.3. Ontology Formatting

In the third stage, we use LLM to generate an OWL ontology based on the matched and newly created properties. We copy the description, domain and range field from all properties under Wikidata semantics. For new properties, LLM is prompted to infer and summarize classes for the domain and range of the relations to output a complete OWL ontology, following the format of copied Wikidata properties. This step ensures that the resulting KG is grounded in a formal, machine-readable ontology that captures relationships between entities, and close to the semantics of Wikidata for interoperability.

3.4. KG Construction

In the final stage , we use the LLM to construct a KG based on the CQs and related answers grounded by the generated ontology in the previous stage. For each (CQ, answer) pair, LLM extracts relevant entities and maps them to the ontology using the defined properties. The output is a set of RDF triples that constitutes the final KG.

4. Experiments and Discussion 4.1. Experiment settings

We evaluate our ontology-grounded approach to KG construction (KGC) on three datasets for KGC datasets: Wiki-NRE [ 16 ], SciERC [ 17 ], and WebNLG [ 18 ]. As Wiki-NRE and WebNLG are partially based on Wikidata and DBpedia (derived from Wikipedia contents), and in our proposed pipeline, Wikidata schema is utilized, we include SciERC for a more robust evaluation, since SciERC contains relation types that are not equivalent by nature to properties in Wikidata.

We used a subset Wiki-NRE’s test dataset containing 1,000 samples with 45 relation types following the split in [ 19 ], due to cost constraints. SciERC’s test set contain 974 samples under a schema with 7 relation types. For WebNLG, we used test set in Semantic Parsing (SP) task, with 1,165 samples and 159 relation types. For evaluation, we adopt partial F1 on KG triplets based on standards in [ 18 ]. All experiments are conducted for one-pass

We note in the previous reports that annotation in KGC reports may be incomplete in terms of both possible relation types and KG triplets [ 19 ], [ 20 ].

As our pipeline is designed to autonomously uncover knowledge structure with no prior assumption on knowledge schema, we report our result in two ways, corresponding to the two configurations of final de-duplication step in Section 3.2: 1. Target schema constrained: In this setting, we match all relation types in test sets to its closest equivalent in Wikidata and constrict ontology to the relation universe in test set. 2. No schema constraint: In this setting, we do not filter matched ontology, even if they are not in schema of test dataset. This setting is close to real-life applications when processing documents with unknown schema.

For property conjunction, evaluate for, compare, feature of in SciERC, we select the closest properties proposed by LLM based on our subjective opinion.

To highlight our system’s competency, rather than directly prompting triplets, we parse output KG with RDF parser and extract all valid RDF triples for KG related to each document in test set, and present triplets to evaluation script for assessment. This ensures that our evaluation is on the generated KG ready to be consumed in downstream application.

We test our pipeline on both Mistral-7B-instruct [ 21 ] and GPT-4o1. Due to cost constraints, we have only tested GPT-4o on target schema constrained setting. For embedding property usage comment, we select bge-small-en [ 22 ]. We use GenIE [ 23 ], PL-Marker [ 24 ], and ReGen [ 25 ] as fine-tuned baseline for Wiki-NRE, SciERC, and WebNLG dataset, respectively (collectively named Non-LLM Baseline). For LLM-based systems, we use results reported in [ 19 ] for WikiNRE and WebNLG on the same Mistral model, and GPT-4 results in [ 3 ] for SciERC. (collectively named LLM Baseline). We note that it is highly unlikely that Mistral-7B poses an advantage over an earlier version of GPT-4, when interpreting result of SciERC.

4.2. Result 1https://openai.com/index/hello-gpt-4o/

Table 1 shows the performance of our Method Wiki-NRE SciERC method compared to state-of-the-art Non-LLM Baseline 0.484 0.532 baselines on this subset. Our proposed LLM Baseline 0.647 0.07 approach exceeds all baseline under tar- Proposed (Mistral) 0.66/0.60 0.73/0.58 get schema constrained setting on Wiki- Proposed (GPT-4o) 0.71/N/A 0.77/N/A NRE and SciERC datasets, while displaying a small performance regression when Table 1: Partial F1 scores on test datasets. Best result without schema constraint. On WebNLG is bolded. Results of proposed pipeline under dataset, our pipeline maintained compet- two settings are presented as Target schema itiveness against fine-tuned SOTA when constrained/no schema constraint. constrained on target schema. These results validate the quality of KG generated by our pipeline, especially SciERC, whose semantics contains properties that are not native to Wikidata. We also note performance improvement when using GPT-4o.

4.3. Discussion

4.3.1. Performance discrepancy on diferent grounding ontology It is worth noting that the relatively lower performance on no schema constraint setting across all datasets is due to the fact that the LLM discovers a richer ontology than the predefined target schema. While this expanded schema may capture additional relevant information, it can hinder extraction performance when evaluated solely against the limited target schema. This showcases the trade-of between schema completeness and strict adherence to a predefined ontology, and our pipeline performs best on a large set of documents with a limited scope of knowledge, requiring a concise schema.

Furthermore, the flipside of performance deficit in an absence of schema constraints, i.e. additional ontology entries outside of dataset-defined properties, cannot be evaluated against the dataset directly, as the ontology is not entirely covered by test set annotations. Hence, the virtue of no schema constraint setting is to demonstrate that our pipeline can indeed provide a coverage of the properties in test set, though somewhat limited compared to baselines, when also capturing ontology outside test set schema, which is potentially more useful when discovering ontology on a novel document set with no expert knowledge in its schema conposition. This ability may be validated by manual evaluation on the full set of captured ontology in a future work down the line.

Nevertheless, the marginal performance deficit leaves room for improvement. Recent reports explored that long input context may pose challenge to LLMs even if such long context length is technically supported [ 26 ]. We conjecture that aside from trimming grounding ontology, which hinders the knowledge coverage of our pipeline, few-shot fine-tuning on the new ontology or general pretraining in KG construction task may be helpful. We leave these as possible future directions. 4.3.2. Utility of generated KG It should be emphasized that, while the selected evaluation tasks evaluate the correctness of extracted triplets, the extracted knowledge graph can do more than that. With ongoing discussion related to grounding LLM knowledge on trusted knowledge sources to reduce hallucination [ 6 ], explicitly generating KG provides a path to audit knowledge elicited when interacting with LLM, and with evidence demonstrating that LLM has the potential to reason on graph and generate an explicit path to retrieve required knowledge [ 27 ], our pipeline may serve as a foundation for an interpretable QA system, where an LLM autonomously extracts ontology and deduces correct retrieval query based on the ontology when handling a set of unstructured document. The interpretability arise from the fact that KG and query could be understood and verified by users. Moreover, our usage of Wikidata schema ofers potential interoperability with the whole Wikidata knowledge base, which safely expands the knowledge scope of QA system. We propose to continue research on this significant direction. 4.3.3. Computational resources We note the growing concern of sustainability in LLM applications due to intensive requirement on computational resources. This pipeline consumes three separate LLM calls per document, plus one call per extracted relation. It is not straight forward to compare the carbon footprint of our approach compared to Non-LLM baselines, as our work at this stage does not require model fine-tuning, whereas all of the Non-LLM baselines employed various tuning techniques when producing the result. On the other hand, our smallest model adopted, Mistral-7B, is more than 10x larger in terms of parameter size compared to T5 models used in Non-LLM baselines. Larger models naturally require more powerful GPU clusters in terms of both GPU quantity and capability, but our zero-shot approach may provide an advantage in terms of resource cost compared to Non-LLM baselines when processing a small number of documents with no training requirement.

When comparing with LLM baselines, we note that the approach by [ 14 ], [ 3 ] consumes 1 and 2 LLM calls per document, respectively. However, we note that these baselines treat knowledge triplet as evaluation target, while we generate a formatted ontology at the end, which is more useful. Nevertheless, we recognize the performance burden and propose to explore techniques in fine-tuning and guided decoding to achieve better performance with smaller model and better reproducibility.

5. Conclusion

We have demonstrated the efectiveness of our ontology-grounded approach to KG construction using LLMs. By leveraging the structured knowledge in Wikidata, pretrained on LLM, and grounding KG construction with generated ontology, our pipeline is able to construct highquality KGs across various domains while maintaining competitive performance with state-ofthe-art baselines. Generated KGs that are conformant with Wikidata schema leaves possibly wide open, of building an interpretable QA system that has robust access to both common knowledge and proprietary knowledge base.

Acknowledgments

This work is supported by Centre for Perceptual and Interactive Intelligence (CPII) Ltd, a CUHK-led InnoCentre under InnoHK scheme of Innovation and Technology Commision.

A. Sample generated KG

This KG was generated under no schema constraint setting for this document: Mohammad Firouzi ( Born 1958 Tehran ) is a prolific Iranian musician , whose primary instrument is the barbat . < P r e f i x e s and d e f i n i t i o n o f d e p e n d e n c i e s o m i t t e d > wd : Mohammad_Firouzi a wd : human ; r d f s : l a b e l " Mohammad F i r o u z i " @en ; wdt : o c c u p a t i o n wd : M u s i c i a n ; wdt : C o u n t r y O f C i t i z e n s h i p wd : I r a n ; wdt : P l a c e O f B i r t h wd : T e h r a n ; wdt : D a t e O f B i r t h " 1 9 5 8 " ^ ^ x s d : d a t e .

Note that in oficial annotation, only triplets related to place of birth and nationality exist, hence the evaluation will be penalized with low precision.

B. Preprocessing of Wikidata schema

To save space in LLM input context and mitigate performance drop on selected target schema when ontology is large, we only include commonly used properties by restricting data type on item, quantity, string, monolingual text, point in time.2 To align with common pretraining objectives of LLM, we substitute entity identifiers (e.g. P19) with its literal label (rdfs:label in PascalCase (e.g. PlaceOfBirth).

2https://www.wikidata.org/wiki/Help:Data_type C. Prompts All prompts are reused across all datasets. C.1. CQ generation

We prompt LLM to generate up to 3 CQs per document for eficiency considering nature of test datasets, but note that this may be adjusted.

Write competency q u e s t i o n s b a s e d on t h e a b s t r a c t l e v e l c o n c e p t s i n t h e document . Write q u e s t i o n s t h a t can be answered u s i n g t h e document o n l y .

Write up t o 3 q u e s t i o n s p e r document .

Below a r e t h e examples and f o l l o w t h e same f o r m a t when g e n e r a t i n g competency q u e s t i o n s : #### Document : Douglas Noel Adams ( 1 1 March 1952 − 11 May 2 0 0 1 ) was an E n g l i s h author , humourist , and s c r e e n w r i t e r , b e s t known f o r The H i t c h h i k e r ’ s Guide t o t h e Galaxy (HHGTTG) .

O r i g i n a l l y a 1978 BBC r a d i o comedy , The H i t c h h i k e r ’ s Guide t o t h e Galaxy d e v e l o p e d i n t o a " t r i l o g y " o f f i v e books t h a t s o l d more than 15 m i l l i o n c o p i e s i n h i s l i f e t i m e . I t was f u r t h e r d e v e l o p e d i n t o a t e l e v i s i o n s e r i e s , s e v e r a l s t a g e p l a y s , comics , a v i d e o game , and a 2005 f e a t u r e f i l m . Adams ’ s c o n t r i b u t i o n t o UK r a d i o i s commemorated i n The Radio Academy ’ s H a l l o f Fame . #### Q u e s t i o n s : CQ1 . What i s t h e d a t e o f b i r t h o f Douglas Noel Adams? CQ2 . What i s t h e d a t e o f d e a t h o f Douglas Noel Adams? CQ3 . What i s t h e o c c u p a t i o n o f Douglas Noel Adams? CQ4 . What i s t h e c o u n t r y o f c i t i z e n s h i p o f Douglas Noel Adams? CQ5 . What i s t h e most n o t a b l e work o f Douglas Noel Adams? CQ6 . What i s t h e o r i g i n a l medium o f The H i t c h h i k e r ’ s Guide t o t h e Galaxy ? CQ7 . I n what y e a r was The H i t c h h i k e r ’ s Guide t o t h e Galaxy o r i g i n a l l y b r o a d c a s t ? CQ8 . How many books a r e i n The H i t c h h i k e r ’ s Guide t o t h e Galaxy " t r i l o g y " ? CQ9 . What o t h e r media a d a p t a t i o n s were c r e a t e d b a s e d on The

H i t c h h i k e r ’ s Guide t o t h e Galaxy ? #### Document : { document t o be p r o c e s s e d }

C.2. CQ answering

Use t h e p r o v i d e d document t o answer u s e r query . I f you don ’ t know t h e answer , j u s t say t h a t you don ’ t know , don ’ t t r y t o make up an answer .

P a s s a g e : { doc } Query : { query }

C.3. Relation extraction

You a r e an a s s i s t a n t i n b u i l d i n g a knowledge graph . Analyze t h e f o l l o w i n g competency q u e s t i o n s and i d e n t i f y a l l r e l a t i o n s h i p s and c o n c e p t s c o n c e p t s mentioned i n t h e q u e s t i o n .

E x t r a c t r e l a t i o n f i r s t , then d e s c r i b e t h e usage o f each r e l a t i o n b a s e d on your u n d e r s t a n d i n g g i v e n t h e c o n t e x t o f competency q u e s t i o n s .

A ft e rw a rd s , e x t r a c t a l l r e l a t i o n − r e l a t e d c o n c e p t s .

You s h o u l d only e x t r a c t p r o p e r t i e s between e n t i t i e s and l i t e r a l s , not e n t i t i e s t h e m s e l v e s , or c l a s s e s o f e n t i t i e s .

T h e r e f o r e , not a l l CQs c o n t a i n v a l i d p r o p e r t i e s .

I f you don ’ t know t h e answer , j u s t say t h a t you don ’ t know , don ’ t t r y t o make up an answer .

Merge a l l r e l a t i o n s i n t o one l i s t and a l l c o n c e p t s i n t o one l i s t .

Do not r e p l y u s i n g a c o m p l e t e s e n t e n c e , and only g i v e t h e answer i n t h e f o l l o w i n g f o r m a t .

Below a r e t h e examples and f o l l o w t h e same f o r m a t t o e x t r a c t t h e r e l a t i o n s : #### Document : Douglas Noel Adams ( 1 1 March 1952 − 11 May 2 0 0 1 ) was an E n g l i s h author , humourist , and s c r e e n w r i t e r , b e s t known f o r The H i t c h h i k e r ’ s Guide t o t h e Galaxy (HHGTTG) .

H i t c h h i k e r ’ s Guide t o t h e Galaxy ? #### R e l a t i o n s : ( d a t e o f b i r t h , The d a t e on which t h e s u b j e c t was born . ) ( d a t e o f death , The d a t e on which t h e s u b j e c t d i e d . ) ( o c c u p a t i o n , The o c c u p a t i o n o f a p e r s o n . ) ( c o u n t r y o f c i t i z e n s h i p , The c o u n t r y o f which t h e s u b j e c t i s a c i t i z e n . ) ( n o t a b l e work , The most n o t a b l e work o f a p e r s o n . ) ( genre , The g e n r e or t y p e o f work . ) ( p u b l i c a t i o n date , The d a t e or p e r i o d when a work was f i r s t p u b l i s h e d or r e l e a s e d . ) ( has p a r t , I n d i c a t e s t h a t t h e s u b j e c t has a c e r t a i n p a r t , component , or e l e m e n t . ) ( s e r i e s , I n d i c a t e s t h a t t h e s u b j e c t i s p a r t o f a s e r i e s , such a s a book s e r i e s , f i l m s e r i e s , or t e l e v i s i o n s e r i e s . ) #### Document : { document t o be p r o c e s s e d } #### Q u e s t i o n s : #### R e l a t i o n s :

C.4. Ontology matching

De cide i f t h e two p r o p e r t i e s a r e s e m a n t i c a l l y s i m i l a r i n an o n t o l o g y .

You s h o u l d say yes i f you d e c i d e t h a t t h e s e p r o p t i e s a r e s i m i l a r , or i f t h e y a r e i n v e r s e p r o p e r t i e s .

Answer i n " yes " or " no " o n l y .

P r o p e r t y 1 : { p1 } P r o p e r t y 2 : { p2 }

C.5. Ontology formatting

For properties under Wikidata schema, we retrieve schema:description, rdfs:domain, rdfs:range for each property and include it in resulting ontology. Otherwise LLM is prompted to author ontology as so: #### R e l a t i o n s : ( r e s u l t s , r e s u l t s : r e s u l t s o f a c o m p e t i t i o n such a s s p o r t s or e l e c t i o n s ) #### Ontology : wdt : R e s u l t s a w i k i b a s e : P r o p e r t y ; schema : d e s c r i p t i o n " r e s u l t s o f a c o m p e t i t i o n such a s s p o r t s or e l e c t i o n s " ; r d f s : l a b e l " r e s u l t s " ; r d f s : domain wd : referendum , wd : c o m p e t i t i o n , wd : p a r t y

c o n f e r e n c e , wd : s p o r t i n g e v e n t ; r d f s : range wd : e l e c t o r a l r e s u l t , wd : v o t i n g r e s u l t , wd : s p o r t r e s u l t , wd : r a c e r e s u l t . #### R e l a t i o n s : { r e l a t i o n }

C.6. KG generation

Your t a s k i s t o c o n s t r u c t a knowledge graph b a s e d on t h e p r o v i d e d o n t o l o g y .

Focus on u n d e r s t a n d i n g r e l a t i o n s h i p s from t h e q u e s t i o n answer p a i r and document , and e x t r a c t r e l a t e d e n t i t i e s , then mapping them t o t h e o n t o l o g y u s i n g t h e p r o p e r t i e s d e f i n e d i n t h e o n t o l o g y .

Do not i n c l u d e new p r o p e r t i e s o t h e r than t h o s e i n o n t o l o g y .

Only use t h o s e p r o p e r t i e s i n t h e o n t o l o g y .

Output i n t u r t l e f o r m a t f o l l o w i n g t h e o n t o l o g y p r o v i d e d . You s h o u l d o n l y i n c l u d e knowledge i n q u e s t i o n answer p a i r s and t h e document .

Do not make up answers .

Use t h i s o n t o l o g y b a s e d on W i k i d a t a a s t h e s t a r t i n g p o i n t : { ont } #### Document : Douglas Noel Adams ( 1 1 March 1952 − 11 May 2 0 0 1 ) was an E n g l i s h author , humourist , and s c r e e n w r i t e r , b e s t known f o r The H i t c h h i k e r ’ s Guide t o t h e Galaxy (HHGTTG) . O r i g i n a l l y a 1978

BBC r a d i o comedy , The H i t c h h i k e r ’ s Guide t o t h e Galaxy d e v e l o p e d i n t o a " t r i l o g y " o f f i v e books t h a t s o l d more than 15 m i l l i o n c o p i e s i n h i s l i f e t i m e . I t was f u r t h e r d e v e l o p e d i n t o a t e l e v i s i o n s e r i e s , s e v e r a l s t a g e p l a y s , comics , a v i d e o game , and a 2005 f e a t u r e f i l m . Adams ’ s c o n t r i b u t i o n t o UK r a d i o i s commemorated i n The Radio Academy ’ s H a l l o f Fame . #### Q u e s t i o n answer p a i r s : Q : What i s Douglas Adams an i n s t a n c e o f ? A : Douglas Adams i s an i n s t a n c e o f human .

Q : What i s Douglas Adams ’ s e x or gender ? A : Douglas Adams ’ s e x or gender i s male . Q : Where was Douglas Adams born ? A : Douglas Adams was born i n Cambridge . Q : Where d i d Douglas Adams d i e ? A : Douglas Adams d i e d i n S a n t a Barbara , C a l i f o r n i a . Q : When was Douglas Adams born ? A : Douglas Adams was born on 1952 −03 −11. Q : On what d a t e d i d Douglas Adams d i e ? A : Douglas Adams d i e d on 2001 −05 −11. Q : What o c c u p a t i o n d i d Douglas Adams have ? A : Douglas Adams was a w r i t e r , comedian , and d r a m a t i s t . Q : What l a n g u a g e s d i d Douglas Adams speak , w r i t e , or s i g n ? A : Douglas Adams spoke , wrote , or s i g n e d E n g l i s h . Q : Where was Douglas Adams e d u c a t e d ?

A : Douglas Adams was e d u c a t e d a t S t John ’ s C o l l e g e , Cambridge and Brentwood School , E s s e x .

Q : What i n s t i t u t i o n i s Douglas Adams an alumni o f ? A : Douglas Adams i s an alumni o f S t John ’ s C o l l e g e . Q : What a r e some n o t a b l e works by Douglas Adams? A : Some n o t a b l e works by Douglas Adams i n c l u d e The H i t c h h i k e r ’ s

Guide t o t h e Galaxy and Dirk Gently ’ s H o l i s t i c D e t e c t i v e Agency .

Q : Was Douglas Adams a member o f any n o t a b l e o r g a n i z a t i o n s ? A : Yes , Douglas Adams was a member o f Monty Python and The

I n d e p e n d e n t on Sunday .

Q : What award d i d Douglas Adams r e c e i v e ?

A : Douglas Adams r e c e i v e d t h e Locus Award f o r B e s t S c i e n c e

F i c t i o n Novel .

Q : What i s t h e Commons C a t e g o r y f o r Douglas Adams? A : The Commons C a t e g o r y f o r Douglas Adams i s " Douglas Adams " . wdt : MemberOf wd : Monty_Python ; wdt : MemberOfOrganization wd : The_Independent_on_Sunday ; wdt : Award wd : L o c u s _ A w a r d _ f o r _ B e s t _ S c i e n c e _ F i c t i o n _ N o v e l ; wdt : CommonsCategory " Douglas Adams "@en .

[1]

Hogan , E. Blomqvist,

Cochez , C. D'amato , G. D. Melo , C.

Gutierrez , S.

Kirrane , J. E. L.

Gayo , R.

Navigli , S.

Neumaier , A.-C. N.

Ngomo , A.

Polleres , S. M.

Rashid , A.

Rula , L.

Schmelzeisen , J.

Sequeda , S.

Staab , A.

Zimmermann , Knowledge graphs , ACM Comput. Surv . 54 ( 2021 ). URL: https://doi.org/10.1145/3447772. doi: 10 .1145/3447772.

[2]

Ji ,

Pan , E. Cambria,

Marttinen ,

P. S.

Yu , A survey on knowledge graphs: Representation, acquisition, and applications , IEEE Transactions on Neural Networks and Learning Systems 33 ( 2020 ) 494 - 514 . URL: https://api.semanticscholar.org/CorpusID:211010433.

[3]

Zhu ,

Wang ,

Chen ,

Qiao ,

Ou ,

Yao ,

Deng ,

Chen , N. Zhang, LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and

Future

Opportunities , 2024 . doi: 10 .48550/arXiv.2305.13168. arXiv: 2305 . 13168 .

[4]

Petroni ,

Rocktäschel ,

Lewis ,

Bakhtin ,

Wu ,

A. H.

Miller ,

Riedel , Language models as knowledge bases? , in: Conference on Empirical Methods in Natural Language Processing , 2019 . URL: https://api.semanticscholar.org/CorpusID:202539551.

[6]

Agrawal ,

Kumarage ,

Alghamdi , H. Liu, Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey , 2024 . doi: 10 .48550/arXiv.2311.07914. arXiv: 2311 . 07914 .

[7]

Agichtein , L. Gravano, Snowball: extracting relations from large plain-text collections , in: Proceedings of the Fifth ACM Conference on Digital Libraries , DL '00, Association for Computing Machinery, New York, NY, USA, 2000 , p. 85 - 94 . URL: https://doi.org/10.1145/ 336597.336644. doi: 10 .1145/336597.336644.

[8]

Zhang ,

Zhong ,

Chen , G. Angeli,

C. D.

Manning , Position-aware attention and supervised data improve slot filling , in: M. Palmer , R. Hwa , S. Riedel (Eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Copenhagen, Denmark, 2017 , pp. 35 - 45 . URL: https:// aclanthology.org/D17-1004. doi: 10 .18653/v1/ D17 -1004.

[9]

Mintz ,

Bills ,

Snow ,

Jurafsky , Distant supervision for relation extraction without labeled data , in: K. -Y. Su , J.

Su , J.

Wiebe , H. Li (Eds.), Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP , Association for Computational Linguistics , Suntec, Singapore, 2009 , pp. 1003 - 1011 . URL: https://aclanthology.org/P09-1113.

[10]

X. L.

Dong , E. Gabrilovich, G. Heitz,

Horn ,

Lao ,

Murphy ,

Strohmann ,

Sun , W. Zhang, Knowledge vault: A web-scale approach to probabilistic knowledge fusion , in: The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD ' 14 , New York, NY, USA - August 24 - 27 , 2014 , 2014 , pp. 601 - 610 . URL: http: //www.cs.cmu.edu/~nlao/publication/2014.kdd.pdf, evgeniy GabrilovichWilko HornNi LaoKevin MurphyThomas StrohmannShaohua SunWei ZhangGeremy Heitz.

[11]

Bosselut ,

Rashkin ,

Sap ,

Malaviya ,

Celikyilmaz ,

Choi , COMET: Commonsense transformers for automatic knowledge graph construction , in: A. Korhonen , D. Traum , L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Florence, Italy, 2019 , pp. 4762 - 4779 . URL: https://aclanthology.org/P19-1470. doi: 10 .18653/v1/ P19 -1470.

[12]

Chen ,

Liu ,

Dong ,

Wang ,

Zhu ,

Zeng , Y. Zhang, AdaPrompt: Adaptive model training for prompt-based NLP , in: Y. Goldberg , Z. Kozareva , Y. Zhang (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2022 , Association for Computational Linguistics , Abu Dhabi, United Arab Emirates, 2022 , pp. 6057 - 6068 . URL: https://aclanthology.org/ 2022 .findings-emnlp. 448 . doi: 10 .18653/v1/ 2022 . findings-emnlp. 448 .

[13]

Yu ,

He , J. Glass, AutoKG: Constructing Virtual Knowledge Graphs from Unstructured Documents for Question Answering , 2021 . doi: 10 .48550/arXiv. 2008 . 08995 . arXiv: 2008 .08995.

[14]

Chen ,

A. L.

Bertozzi , AutoKG: Eficient Automated Knowledge Graph Generation for Language Models , 2023 . doi: 10 .48550/arXiv.2311.14740. arXiv: 2311 . 14740 .

[15]

S. J.

Semnani ,

V. Z.

Yao ,

H. C.

Zhang , M. S. Lam, WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia , 2023 . arXiv: 2305 . 14292 .

[16]

B. D.

Trisedya , G. Weikum,

Qi ,

Zhang , Neural relation extraction for knowledge base enrichment , in: A. Korhonen , D. Traum , L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Florence, Italy, 2019 , pp. 229 - 240 . URL: https://aclanthology.org/P19-1023. doi: 10 .18653/v1/ P19 -1023.

[17]

Luan ,

He ,

Ostendorf , H. Hajishirzi, Multi-task identification of entities, relations, and coreferencefor scientific knowledge graph construction , in: Proc. Conf. Empirical Methods Natural Language Process . (EMNLP) , 2018 .

[18]

T. Castro

Ferreira ,

Gardent ,

Ilinykh ,

C. van der

Lee ,

Mille ,

Moussallem ,

Shimorina , The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020 ), in: T. Castro Ferreira,

Gardent ,

Ilinykh ,

C. van der

Lee ,

Mille ,

Moussallem , A . Shimorina (Eds.), Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) , Association for Computational Linguistics , Dublin, Ireland (Virtual), 2020 , pp. 55 - 76 . URL: https://aclanthology.org/ 2020 .webnlg- 1 .7.

[19]

Zhang , H. Soh, Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction , 2024 . arXiv: 2404 . 03868 .

[20]

Han , T . Peng,

Yang ,

Wang ,

Liu ,

Wan , Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors, 2023 . arXiv: 2305 . 14450 .

[21]

A. Q.

Jiang ,

Sablayrolles ,

Mensch ,

Bamford ,

D. S.

Chaplot , D. de las Casas,

Bressand , G. Lengyel,

Lample ,

Saulnier ,

L. R.

Lavaud , M. -

A. Lachaux , P.

Stock , T. L.

Scao , T.

Lavril , T.

Wang , T.

Lacroix , W. E.

Sayed , Mistral 7b, 2023 . arXiv: 2310 . 06825 .

[22]

Chen ,

Xiao ,

Zhang ,

Luo ,

Lian ,

Liu , Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation , 2023 . arXiv: 2309 . 07597 .

[23]

Josifoski , N. De Cao,

Peyrard ,

Petroni , R. West, GenIE: Generative information extraction, in: M. Carpuat , M.-C. de Marnefe , I. V. Meza Ruiz (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics , Seattle, United States, 2022 , pp. 4626 - 4643 . URL: https://aclanthology.org/ 2022 .naacl-main. 342 . doi: 10 .18653/v1/ 2022 .naacl-main. 342 .

[24]

Ye ,

Lin ,

Li ,

Sun , Packed levitated marker for entity and relation extraction , in: S. Muresan,

Nakov , A . Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long

Papers)

, Association for Computational Linguistics , Dublin, Ireland, 2022 , pp. 4904 - 4917 . URL: https://aclanthology. org/ 2022 . acl-long . 337 . doi: 10 .18653/v1/ 2022 . acl-long . 337 .

[25]

Dognin , I. Padhi , I. Melnyk , P. Das, ReGen: Reinforcement learning for text and knowledge base generation using pretrained language models , in: M. -

F. Moens , X.

Huang , L.

Specia , S. W.-t. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Online and

Punta

Cana , Dominican Republic, 2021 , pp. 1084 - 1099 . URL: https://aclanthology.org/ 2021 . emnlp-main. 83 . doi: 10 .18653/v1/ 2021 .emnlp-main. 83 .

[26]

Li ,

Zhang ,

Q. D.

Do ,

Yue , W. Chen, Long-context llms struggle with long in-context learning , 2024 . arXiv: 2404 . 02060 .

[27]

Brei ,

Frey , L.-P. Meyer, Leveraging small language models for text2sparql tasks to improve the resilience of ai assistance , 2024 . arXiv: 2405 . 17076 .