Knowledge Graphs (KGs) are structured representations of information that capture entities and their relationships in a graph format. By organizing knowledge in a machine-readable way, KGs enable a wide range of intelligent applications, such as semantic search, question answering, recommendation systems, and decision support [1]. The ability to construct highquality, comprehensive KGs is thus critical for harnessing the power of these technologies across various domains.
Traditionally, the process of constructing KGs has relied heavily on manual effort by domain experts to define the relevant entities and relationships, populate the graph with valid facts, and ensure logical consistency [2]. However, this manual curation approach is time-consuming, expensive, and difficult to scale to large, evolving domains. There is a strong need for (semi-)automatic methods that can aid the KG construction process by extracting structured knowledge from unstructured data sources such as text.
Recent years have seen growing interest in leveraging Large Language Models (LLMs) for various knowledge capture and reasoning tasks [3]. Pre-trained on vast amounts of text data, LLMs can generate fluent natural language and have been shown to memorize and recall factual knowledge [4], [5]. However, directly applying LLMs to KG construction still faces several challenges. First, LLMs may generate inconsistent or redundant facts due to the lack of an explicit, unified schema [6]. Second, the generated KGs may be incomplete or biased towards the knowledge present in the LLM's training data, which may not fully cover the target domain, especially for proprietary documents not included in pre-training set. Finally, it can be challenging to integrate LLM-generated KGs with existing knowledge bases due to misalignment with standard ontologies. In this work, we propose a novel approach that harnesses the reasoning power of LLMs and the structured schema of Wikidata to construct high-quality KGs for proprietary knowledge domains. Our approach begins by discovering the scope of knowledge through the generation of Competency Questions (CQ) and answers from unstructured documents. We then summarize the relations and properties from these QA pairs into an ontology, matching candidate properties against those defined in Wikidata and extending the schema as needed. Finally, we use the resulting ontology to ground the transformation of CQanswer pairs into a structured KG. By incorporating the Wikidata schema into our pipeline and grounding generation of KG on the same ontology, we aim to reduce redundancy, leverage the implicit knowledge captured during LLM pretraining while improving interpretability, and ensure interoperability with public knowledge bases. The generated KGs could be parsed with RDF parsers and used in downstream applications, or audited for correctness.
The main contributions of this work are as follows:
1. We propose a novel ontology-grounded approach to LLM-based KG construction that leverages ontology based on Wikidata schema to guide the extraction and integration of knowledge from unstructured text. 2. We introduce a pipeline that combines competency question generation, ontology alignment, and KG grounding to systematically construct high-quality KGs that are consistent, complete, and interoperable with existing knowledge bases. 3. We demonstrate the effectiveness of our approach through experiments on benchmark datasets, showing improvements in KG quality compared to traditional methods alongside with interpretability and utility of generated KGs.
Knowledge graph construction has been an active area of research in recent years, with a wide range of approaches proposed for extracting structured knowledge from unstructured data sources [2]. Early methods relied heavily on rule-based systems and hand-crafted features to identify entities and relations in text [7]. With the advent of deep learning, neural networkbased approaches have become increasingly popular, enabling more flexible and scalable KG construction [8].
One prominent line of work focuses on using distant supervision to automatically generate training data for relation extraction [9]. These methods assume that if two entities are mentioned together in a sentence and also appear in a knowledge base as subject and object of a relation, then that sentence is likely to express the relation. While distant supervision has been shown to be effective at scale, it often suffers from noise and incomplete coverage.
Another important direction is the development of unsupervised and semi-supervised methods for KG construction [10]. These approaches aim to reduce the reliance on large amounts of labeled data by leveraging techniques such as bootstrapping, graph-based inference, and representation learning. However, they often struggle with consistency and quality control issues. More recently, there has been growing interest in using large language models for KG construction [11], [12], [13]. These methods take advantage of the vast knowledge captured in pretrained Language Models (LM) to generate KG triples through prompt engineering and fine-tuning. While promising, these LM-based approaches only produces triplets without canocalization, which makes portability and interoperability difficult. Additionally, some methods rely on vector-based similarity measures to deduce relationships between entities in KG, which yields good performance but falls short in interpretability [14].
As mentiond in Introduction, despite the significant progress in KG construction and LLM applications, performance, interpretability, coverage of proprietary documents, and interaction with other knowledge base remain issues. Our pipeline address these by grouding KG generation on ontology based on Wikidata schema, which ensures that output KG is human-readable and makes integrating with Wikidata or other KG easier; In the experiments below we show that these benefits can be also achieved on private documents with decent performance.
Our proposed approach for ontology-grounded KG construction using LLMs consists of four main stages: 1) Competency Question Generation, 2) Relation Extraction and Ontology Matching, 3) Ontology Formatting, and 4) KG Construction. Figure 1 provides an overview of the pipeline.
The first step in our pipeline is to generate a set of competency questions (CQs) and answers that capture the key information needs of the target domain. We employ an LLM to generate CQs based on the input documents. The LLM is provided with a set of instructions and examples to guide the generation process, encouraging the creation of well-formed, relevant questions that can be answered using the given documents. This step helps to scope the KG construction task within the knowledge domain, and ensure that the resulting KG aligns with the intended use cases. This also allows further ontology expansion by incorporating user-submitted domaindefining questions when interacting with the knowledge base, which serves as a user friendly interface of refining ontology by submitting new CQs and use our proposed pipeline to attach the incremental knowledge scope to existing ontology.
During our preliminary experiments of prompting LLMs to directly generate ontology on documents, we noted that the LLM spontaneously recalled Wikidata knowledge in response, consistent with previous works [15]. In our preliminary experiments, this behaviour also transfers to small 7B/14B models.
Following this direction, in the second step we extract relations from CQs and match them against Wikidata properties to better elicit model memories on Wikidata when constructing and using ontology. We first prompt LLMs to extract properties from CQ and write brief description on usage of extracted properties, including their domain and range, following editing guidelines of Wikidata. To match these properties against existing entries in Wikidata ontology, we prepopulate a candidate property list with all Wikidata properties after filtering out properties related to external database/knowledge base IDs. These extracted properties are then matched against the candidate list by a vector similarity search between description of properties. The representation for property is sentence embedding constructed from description of properties, and the top 1 closest candidate is retrieved for each extracted property. This matching result between each pair of extracted property and matched top 1 candidate is then vetted by LLM to see if they are really semantically similar as a final deduplication step. If a match is validated, the candidate property is added to the final property list; otherwise, the newly minted property is kept in the final list if we allow expansion from the candidate property list derived from Wikidata, and discarded when the final property list is required to be a subset of candidate property list. The first scenario is suitable for cases when no prior schema is known for the domain and some new properties outside of common ontology are expected, whereas the latter is for a known target list of possible properties.
In the third stage, we use LLM to generate an OWL ontology based on the matched and newly created properties. We copy the description, domain and range field from all properties under Wikidata semantics. For new properties, LLM is prompted to infer and summarize classes for the domain and range of the relations to output a complete OWL ontology, following the format of copied Wikidata properties. This step ensures that the resulting KG is grounded in a formal, machine-readable ontology that captures relationships between entities, and close to the semantics of Wikidata for interoperability.
In the final stage, we use the LLM to construct a KG based on the CQs and related answers grounded by the generated ontology in the previous stage. For each (CQ, answer) pair, LLM extracts relevant entities and maps them to the ontology using the defined properties. The output is a set of RDF triples that constitutes the final KG.
We evaluate our ontology-grounded approach to KG construction (KGC) on three datasets for KGC datasets: Wiki-NRE [16], SciERC [17], and WebNLG [18]. As Wiki-NRE and WebNLG are partially based on Wikidata and DBpedia (derived from Wikipedia contents), and in our proposed pipeline, Wikidata schema is utilized, we include SciERC for a more robust evaluation, since SciERC contains relation types that are not equivalent by nature to properties in Wikidata.
We used a subset Wiki-NRE's test dataset containing 1,000 samples with 45 relation types following the split in [19], due to cost constraints. SciERC's test set contain 974 samples under a schema with 7 relation types. For WebNLG, we used test set in Semantic Parsing (SP) task, with 1,165 samples and 159 relation types. For evaluation, we adopt partial F1 on KG triplets based on standards in [18]. All experiments are conducted for one-pass
We note in the previous reports that annotation in KGC reports may be incomplete in terms of both possible relation types and KG triplets [19], [20].
As our pipeline is designed to autonomously uncover knowledge structure with no prior assumption on knowledge schema, we report our result in two ways, corresponding to the two configurations of final de-duplication step in Section 3.2:
1. Target schema constrained: In this setting, we match all relation types in test sets to its closest equivalent in Wikidata and constrict ontology to the relation universe in test set.
In this setting, we do not filter matched ontology, even if they are not in schema of test dataset. This setting is close to real-life applications when processing documents with unknown schema.
For property conjunction, evaluate for, compare, feature of in SciERC, we select the closest properties proposed by LLM based on our subjective opinion.
To highlight our system's competency, rather than directly prompting triplets, we parse output KG with RDF parser and extract all valid RDF triples for KG related to each document in test set, and present triplets to evaluation script for assessment. This ensures that our evaluation is on the generated KG ready to be consumed in downstream application.
We test our pipeline on both Mistral-7B-instruct [21] and GPT-4o 1 . Due to cost constraints, we have only tested GPT-4o on target schema constrained setting. For embedding property usage comment, we select bge-small-en [22]. We use GenIE [23], PL-Marker [24], and ReGen [25] as fine-tuned baseline for Wiki-NRE, SciERC, and WebNLG dataset, respectively (collectively named Non-LLM Baseline). For LLM-based systems, we use results reported in [19] for Wiki-NRE and WebNLG on the same Mistral model, and GPT-4 results in [3] for SciERC. (collectively named LLM Baseline). We note that it is highly unlikely that Mistral-7B poses an advantage over an earlier version of GPT-4, when interpreting result of SciERC. Table 1 shows the performance of our method compared to state-of-the-art baselines on this subset. Our proposed approach exceeds all baseline under target schema constrained setting on Wiki-NRE and SciERC datasets, while displaying a small performance regression when without schema constraint. On WebNLG dataset, our pipeline maintained competitiveness against fine-tuned SOTA when constrained on target schema. These results validate the quality of KG generated by our pipeline, especially SciERC, whose semantics contains properties that are not native to Wikidata. We also note performance improvement when using GPT-4o.
It is worth noting that the relatively lower performance on no schema constraint setting across all datasets is due to the fact that the LLM discovers a richer ontology than the predefined target schema. While this expanded schema may capture additional relevant information, it can hinder extraction performance when evaluated solely against the limited target schema. This showcases the trade-off between schema completeness and strict adherence to a predefined ontology, and our pipeline performs best on a large set of documents with a limited scope of knowledge, requiring a concise schema. Furthermore, the flipside of performance deficit in an absence of schema constraints, i.e. additional ontology entries outside of dataset-defined properties, cannot be evaluated against the dataset directly, as the ontology is not entirely covered by test set annotations. Hence, the virtue of no schema constraint setting is to demonstrate that our pipeline can indeed provide a coverage of the properties in test set, though somewhat limited compared to baselines, when also capturing ontology outside test set schema, which is potentially more useful when discovering ontology on a novel document set with no expert knowledge in its schema conposition. This ability may be validated by manual evaluation on the full set of captured ontology in a future work down the line.
Nevertheless, the marginal performance deficit leaves room for improvement. Recent reports explored that long input context may pose challenge to LLMs even if such long context length is technically supported [26]. We conjecture that aside from trimming grounding ontology, which hinders the knowledge coverage of our pipeline, few-shot fine-tuning on the new ontology or general pretraining in KG construction task may be helpful. We leave these as possible future directions.
It should be emphasized that, while the selected evaluation tasks evaluate the correctness of extracted triplets, the extracted knowledge graph can do more than that. With ongoing discussion related to grounding LLM knowledge on trusted knowledge sources to reduce hallucination [6], explicitly generating KG provides a path to audit knowledge elicited when interacting with LLM, and with evidence demonstrating that LLM has the potential to reason on graph and generate an explicit path to retrieve required knowledge [27], our pipeline may serve as a foundation for an interpretable QA system, where an LLM autonomously extracts ontology and deduces correct retrieval query based on the ontology when handling a set of unstructured document. The interpretability arise from the fact that KG and query could be understood and verified by users. Moreover, our usage of Wikidata schema offers potential interoperability with the whole Wikidata knowledge base, which safely expands the knowledge scope of QA system. We propose to continue research on this significant direction.
We note the growing concern of sustainability in LLM applications due to intensive requirement on computational resources. This pipeline consumes three separate LLM calls per document, plus one call per extracted relation. It is not straight forward to compare the carbon footprint of our approach compared to Non-LLM baselines, as our work at this stage does not require model fine-tuning, whereas all of the Non-LLM baselines employed various tuning techniques when producing the result. On the other hand, our smallest model adopted, Mistral-7B, is more than 10x larger in terms of parameter size compared to T5 models used in Non-LLM baselines. Larger models naturally require more powerful GPU clusters in terms of both GPU quantity and capability, but our zero-shot approach may provide an advantage in terms of resource cost compared to Non-LLM baselines when processing a small number of documents with no training requirement.
When comparing with LLM baselines, we note that the approach by [14], [3] consumes 1 and 2 LLM calls per document, respectively. However, we note that these baselines treat knowledge triplet as evaluation target, while we generate a formatted ontology at the end, which is more useful. Nevertheless, we recognize the performance burden and propose to explore techniques in fine-tuning and guided decoding to achieve better performance with smaller model and better reproducibility.
We have demonstrated the effectiveness of our ontology-grounded approach to KG construction using LLMs. By leveraging the structured knowledge in Wikidata, pretrained on LLM, and grounding KG construction with generated ontology, our pipeline is able to construct highquality KGs across various domains while maintaining competitive performance with state-ofthe-art baselines. Generated KGs that are conformant with Wikidata schema leaves possibly wide open, of building an interpretable QA system that has robust access to both common knowledge and proprietary knowledge base.
This KG was generated under no schema constraint setting for this document: Mohammad Firouzi ( Born 1958 Tehran ) is a prolific Iranian musician , whose primary instrument is the barbat . Note that in official annotation, only triplets related to place of birth and nationality exist, hence the evaluation will be penalized with low precision.
To save space in LLM input context and mitigate performance drop on selected target schema when ontology is large, we only include commonly used properties by restricting data type on item, quantity, string, monolingual text, point in time. 2 To align with common pretraining objectives of LLM, we substitute entity identifiers (e.g. P19) with its literal label (rdfs:label in PascalCase (e.g. PlaceOfBirth).
All prompts are reused across all datasets.
We prompt LLM to generate up to 3 CQs per document for efficiency considering nature of test datasets, but note that this may be adjusted.
W r i t e competency q u e s t i o n s b a s e d on t h e a b s t r a c t l e v e l c o n c e p t s i n t h e document . W r i t e q u e s t i o n s t h a t can be a nswered u s i n g t h e document o n l y . W r i t e up t o 3 q u e s t i o n s p e r document . Below a r e t h e e x a m p l e s and f o l l o w t h e same f o r m a t when g e n e r a t i n g competency q u e s t i o n s :
Use t h e p r o v i d e d document t o answer u s e r q u e r y . I f you don ' t know t h e answer , j u s t s a y t h a t you don ' t know , don ' t t r y t o make up an answer . P a s s a g e : { doc } Query : { q u e r y }
T h e r e f o r e , n o t a l l CQs c o n t a i n v a l i d p r o p e r t i e s . I f you don ' t know t h e answer , j u s t s a y t h a t you don ' t know , don ' t t r y t o make up an answer . Merge a l l r e l a t i o n s i n t o one l i s t and a l l c o n c e p t s i n t o one l i s t . Do n o t r e p l y u s i n g a c o m p l e t e s e n t e n c e , and o n l y g i v e t h e answer i n t h e f o l l o w i n g f o r m a t .
For properties under Wikidata schema, we retrieve schema:description, rdfs:domain, rdfs:range for each property and include it in resulting ontology. Otherwise LLM is prompted to author ontology as so:
Use t h e r e l a t i o n s ( p r o p e r t i e s ) and t h e i r u s a g e comments t o b u i l d an o n t o l o g y i n RDF f o r m a t . I f you don ' t know t h e answer , j u s t s a y t h a t you don ' t know , don ' t t r y t o make up an answer . Don ' t p r o v i d e a n y t h i n g o t h e r t h a n an o n t o l o g y i n RDF f o r m a t . un d e r t h e s e p r e f i x e s i f n e c e s s a r y : @ p r e f i x r d f : < h t t p : / / www. w3 . o r g / 1 9 9 9 / 0 2 / 2 2 − r d f − s y n t a x − ns #> . @ p r e f i x x s d : < h t t p : / / www. w3 . o r g / 2 0 0 1 / XMLSchema#> . @ p r e f i x r d f s : < h t t p : / / www. w3 . o r g / 2 0 0 0 / 0 1 / r d f − schema #> . @ p r e f i x owl : < h t t p : / / www. w3 . o r g / 2 0 0 2 / 0 7 / owl #> . @ p r e f i x w i k i b a s e : < h t t p : / / w i k i b a . s e / o n t o l o g y #> . @ p r e f i x schema : < h t t p : / / schema . o r g / > . @ p r e f i x wd : < h t t p : / / www. w i k i d a t a . o r g / e n t i t y / > . @ p r e f i x wdt : < h t t p : / / www. w i k i d a t a . o r g / prop / d i r e c t / > . Use t u r t l e s y n t a x . # # # # O n t o l o g y : @ p r e f i x r d f : h t t p : / / www. w3 . o r g / 1 9 9 9 / 0 2 / 2 2 − r d f − s y n t a x − ns # . @ p r e f i x r d f s : h t t p : / / www. w3 . o r g / 2 0 0 0 / 0 1 / r d f − schema # . @ p r e f i x wdt : h t t p : / / www. w i k i d a t a . o r g / prop / d i r e c t / . @ p r e f i x wd : h t t p : / / www. w i k i d a t a . o r g / e n t i t y / . @ p r e f i x x s d : h t t p : / / www. w3 . o r g / 2 0 0 1 / XMLSchema# .
This work is supported by Centre for Perceptual and Interactive Intelligence (CPII) Ltd, a CUHK-led InnoCentre under InnoHK scheme of Innovation and Technology Commision.