This study presents a benchmark proposal designed to enhance knowledge engineering tasks through the use of large language models (LLMs). As LLMs become increasingly pivotal in knowledge extraction and modeling, it is crucial to evaluate and improve their performance. Building on prior work aiming at reverse generating competency questions (CQs) from existing ontologies, we introduce a benchmark focused on specific knowledge modeling tasks including ontology documentation, ontology generation, and query generation. In addition, we propose a baseline evaluation framework that applies various techniques, such as semantic comparison, ontology evaluation criteria, and structural comparison, using both existing ground truth datasets and newly proposed ontologies with corresponding CQs and documentation. This rigorous evaluation aims to provide a deeper understanding of LLM capabilities and contribute to their optimization in knowledge engineering applications.
The knowledge engineering and semantic web communities are increasingly experimenting with Large
Language Models (LLMs) to build ontologies and knowledge graphs. Key tasks being explored include:
creating views on heterogeneous data lakes [
In previous work [
In this paper, we propose to develop a benchmark to systematically compare the performance of LLMs for knowledge engineering tasks, specifically focusing on the stages of specification, conceptualization, and validation of an ontology. The core of our proposal is to leverage ontologies that have been published alongside a set of CQs and have been evaluated through the corresponding authoring tests expressed in SPARQL. The goal of the LLM will typically be to understand user intents expressed in natural language
CEUR
ceur-ws.org regarding a given domain of discourse and to produce the axiomatization of that domain in an ontology. Typical knowledge engineering tasks in the scope of the proposal are: 1) conceptualize an ontology from CQs; 2) generate CQs from an ontology (e.g. for evaluating its completeness or discovering new usages); 3) produce the documentation of an ontology; 4) implement queries corresponding to CQs; 5) verbalize knowledge graph excerpts using instance data and an ontology.
Beyond providing a comparison of the performance of LLMs to perform these tasks, we hypothesize
that the implementation of this benchmark will yield the necessary insights to understand to what
extent does the language bias of a LLM [
In this section, we further detail the elements of this benchmark: the use cases and tasks to be evaluated (Section 2.1), the curated datasets (Section 2.2), and the evaluation metrics for improving LLM performance (Section 2.3).
Conceptualize an ontology. This task aims to conceptualize an ontology given a set of competency questions and a domain scope. A variant of it can start from a partial ontology, and the task of the LLM is to complete the ontology by adding the missing classes and properties. In this case, we will evaluate the LLM’s performance in accurately completing an ontology based solely on the upper-level structure. This will help understanding the depth to which the LLM can reach with and without CQs that could assist in clarifying user intent.
Generate Competency Questions. This task focuses on generating competency questions given an
ontology or specific parts of it, similar to the approach outlined in [
Produce the ontology documentation. In this task, the LLM should produce a human-readable
documentation of the ontology, emphasizing on its main classes and properties. This can be further
expanded in generating useful API calls, e.g. following the SPARQL Transformers approach [
Implement queries corresponding to CQs. This task will assess the LLM’s ability to generate autonomously relevant queries given the ontology structure and the user intent expressed with competency questions.
Verbalize Knowledge Graph Excerpts. The task involves generating human-readable interpretations of a dataset, using an ontology as a guide for the LLM to structure the information. The goal is to go beyond the verbalization of all possibles triples (subject, verb, object) and to generate paragraphs that summarize the graphs.
We propose to consider the following criteria to include ontologies in the benchmark: the availability of competency questions that have lead to their conceptualization, comprehensive documentation of the ontology, FAIR-related scores attached to the ontology, SPARQL queries demonstrating the usage of the ontology and/or SHACL [13] shapes constraining its usage. Furthermore, we advocate for a selection that covers diverse domain of discourses. From a conceptualization point of view, the selected ontologies should exhibit diferent structure (e.g. relatively flat ontologies such as schema.org 2 versus nested models such as FRBR [14]) and make use of well-known ontology design patterns (e.g. event-based modeling like CIDOC-CRM3). Although not all ontologies provide the requisite information for every criteria, they should still be suitable for evaluation in specific tasks.
Domain of Discourse. The benchmark should cover as many domains as possible to be able to draw the line between the performance of LLMs and a given domain. Examples include: general purpose ontologies, IT and sensors, creative industries and medias, cultural heritage and museums, healthcare and medicine, biology and life sciences, education and e-learning, e-commerce and retail, finance and banking, and legal sectors.
Candidate ontologies. The following ontologies fulfill the selection criteria discussed above: • DOREMUS [15]: related to music and cultural heritage domains, the ontology comes with a documentation, competency questions, SPARQL queries and APIs and a large knowledge graph. • Polifonia [16]: related to music and cultural heritage, the ontology enables to capture musical and historical knowledge. In addition to CQs and a comprehensive documentation, the authors provides a set of queries that we can leverage as ground truth and a knowledge graph. • DemCare [17]: related to the medical domain, specifically tailored for dementia care and monitoring. Provided with CQs, a dataset, and well-structured documentation. • Odeuropa [18]: related to sensory experiences and cultural heritage, focusing on olfactory experiences in historical contexts. Provided with CQs, documentation, queries and a dataset. • NORIA-O [19]: related to the IT domain, and designed for network monitoring and performing anomaly detection. Provided with CQs, a documentation, queries, and a knowledge graph. • FIBO [20]: related to the financial domain, this ontology is composed of numerous very specialized terms and comes with a set of SPARQL queries aimed at testing its compliance.4 We plan to incorporate additional ontologies from other domains using well-known catalogs such as Linked Open Vocabulary (LOV) [21] and the Industry Portal [22]. We can also rely on [23] which provides a dataset of competency questions for diferent ontologies and domains (e.g. the African wildlife ontology (AWO) [24] , the Software Ontology (SWO) [ 25], the Generic Ontology of Datatypes (OntoDT) [26]) together with their translation into SPARQL queries.
In this section, we outline the process for evaluating and improving LLM performance using a factorial experiment design based on the following factors: 1) Prompting strategies: presence or absence of partial knowledge of the competency questions, taxonomy, and documentation in the LLM’s context, depending on the specific task similar to an ablation study. 2https://schema.org/ 3https://cidoc-crm.org/ 4https://shorturl.at/rwAPc 2) Data instance: presence or absence of instances from a knowledge graph structured by a given ontology to guide the LLM.
The evaluation process will be iterative, refining the integration of the knowledge graph with the
LLM at each step and assessing performance improvements along the way. Multiple iterations will be
conducted and the results will be analyzed using statistical methods to quantify progress. To facilitate
comparison between diferent models or methods, we propose to rely on a CI/CD-enabled pipeline
based on the tools developed in [
Table 1 summarizes the evaluation techniques intended for each of the benchmark tasks (Section 2.1).
• Ontology Evaluation Criteria. Using an existing ontology as the gold standard, we can assess the accuracy, completeness, and conciseness of the generated ontology [28]. This serves the tasks of ontology generation and ontology enrichment. However, adaptability, clarity, and computational eficiency are not addressed in this research, as they depend on the ground truth ontology. • Logical Consistency. This enables us to validate the semantic formalization of an ontology, typically using tools such as the Hermit reasoner [29]. • Structure Comparison. When evaluating the generation of queries, this measure will compare the structure of the generated query with a ground truth query. We can leverage the RTED algorithm, which calculates the Tree Edit Distance (TED) [30] for this purpose. • Fluency and Coherence. When verbalizing and summarizing knowledge graph excerpts (instance data guided by an ontology), this metric will assess the fluency (grammatical correctness) and adequacy (referring to the accurate integration of triples [31]) of the generated text.
In this work, we propose a comprehensive benchmark for knowledge engineering tasks utilizing large
language models (LLMs), specifically focusing on the knowledge conceptualization aspect. Building on
our previous research [
Furthermore, we advocate for the inclusion of a broader range of ontologies, extending beyond our initial focus on the cultural heritage, education, network operations, and medical domains. By incorporating ontologies from other sectors, we aim to establish a robust foundation for comparing LLM performance across diverse domains of discourse. This expansion would enhance our understanding of LLM capabilities and facilitate their fine-tuning within the context of knowledge engineering. Additionally, we propose a baseline evaluation framework for the various tasks, which includes semantic comparisons, ontology evaluation criteria, logical consistency, and structure comparison. We detail the evaluation technique for each task. Moreover, the diverse range of proposed datasets will provide valuable insights into how LLM performance correlates with the size, complexity, and domain of the ontology.
This work has been partially supported by the French National Research Agency (ANR) within the kFLOW project (Grant n°ANR-21-CE23-0028).