In this paper, we present a domain-independent ontology extension workflow supported by LLMs. Ontology Engineering (OE) is a complex field that requires combining technical skills with domain expertise across multiple disciplines. Despite numerous attempts at automation, most of the processes are still manual. Diferent ontology engineering methodologies coexist, but none is a standard. These challenges, together with the lack of highly skilled workers in the sector, increase the entry barriers to the field. In parallel, Large Language Models (LLMs) are becoming prominent in ontology development due to their natural language processing and coding capabilities and their reportedly emergent abilities. In this paper, we focus on human-LLM collaboration for ontology extension. Following a Design Science Research approach, we interviewed 11 experts and modeled the current process of ontology extension to disclose its main issues. We analyzed the concerns and opportunities perceived by ontology engineers for using LLMs. Based on our insights and previous work, we designed a process framework for ontology extension that combines human expertise with LLMs capabilities, providing customizable prompt templates, OE tools, and guidelines. We tested our methodology with an existing greenhouse ontology using GPT-4o. Finally, we qualitatively evaluated the results against a manually crafted extension we use as our gold standard. The results show that the proposed approach holds the potential to (1) get inspiration for adding new entities, (2) deal with complex syntax definitions and repetitive tasks, and (3) verify whether the extended ontology conforms to the requirements and competency questions.
Ontologies are increasingly developed and used in many domains and sectors to unambiguously define
the semantics of concepts and their relations [
Since many domains already have an ontology defined, the next challenge lies in managing its
changes and extensions. In addition to dificulties in defining new concepts and relations, there are also
questions about where and how these new elements should be integrated within the existing ontology.
This makes the engineering process even more complex, as definitions of new concepts now have to
adhere to already existing definitions in the ontology. An unfortunate side trend is that there is a lack
of highly skilled ontology engineers, so tackling this laborious job becomes even more dificult [
To deal with this challenge, one of the solution directions is to make use of Large Language Models
(LLMs) that have arisen in the last few years as a potentially helpful tool for complex language tasks
[
To answer this question, we applied the Design Science Research (DSR) approach to (1) investigate the main challenges in ontology extension, (2) gather requirements for a human-LLM collaborative ontology extension process framework, (3) determine which tasks can best be supported by an LLM, (4) build a prototype that implements the application of an LLM in the ontology extension process framework and (5) evaluate its usability with a specific use case for an existing ontology in the greenhouse sector.
The main contributions of our work are (1) An analysis of the complexity of the process of ontology extension, the main concerns and identified opportunities for the use of LLMs for this process as perceived by the ontology engineers; (2) a set of 22 high-level requirements for the design of an LLMassisted process framework for ontology extension; (3) a domain-independent workflow that integrates LLMs in the process of ontology extension combining the human expertise with the LLM’s capabilities; (4) a qualitative evaluation of LLM performance across various ontology extension tasks, highlighting those with the best outcomes and the greatest potential for practical implementation. Supplementary material, including the data gathered through the interviews, the process framework, the prompt templates, and all the inputs and outputs from the demonstration and evaluation with GPT-4o can be found on GitLab1.
In Section 2, we outline related work on OE in combination with the field of Natural Language Processing. Next, in Section 3 we describe our methodology. In Section 4, we introduce our results and the domain-independent ontology extension workflow. In Section 5, the framework is evaluated. Finally, we summarize our work and explore future directions in Section 6.
Ontologies are formal models that describe concepts and relations of a domain, and are traditionally
created and maintained manually by domain experts and ontology developers [
In recent years, advances in Natural Language Processing (NLP) have accelerated the possibilities of
automating parts of the OE process, with techniques combining linguistic and statistical methods [
Since the introduction of LLMs, there have been many attempts to apply them to the OE process
[
Examples of the first are recent works that input raw text, prompt an LLM for extraction, and evaluate
the resulting model against a ground truth [
Examples of the second approach are the generation of Competency Questions [
Additionally, pipelines and frameworks that combine prompting and other techniques are introduced.
For instance, [
A common aspect of recent studies is that LLMs do not create ontologies that are of suficient quality
[
For the development of the process framework that can reduce the complexity in the ontology extension process by using LLMs, we followed the Design Science Research approach (DSR). The DSR approach has gained common ground in the information systems domain through the seminal works of [29], [30], and, more recently, [31], who provide a practical, phased way of working on designing artifacts. The DSR encompasses the phases described below.
In our research, the aim of the first DSR phase of problem explication was to analyze the current manual process that ontology engineers follow to extend existing ontologies and to uncover the issues they experience during their activities. We followed the Human Research Ethics Research Design Plan of a known research institution, which is in line with European guidance on research ethics. This included a Data Management Plan, a Risk Assessment and Mitigation Plan, and an Informed Consent Procedure. The design plan was submitted and approved before starting the interview process.
We conducted semi-structured interviews with 11 professionals in applied research in OE and LLMs to map their daily practices and to analyze the root causes of dificulties they encounter during the ontology extension process. From the 11 interviewees, 10 are ontology engineers and 1 is mainly focused on LLMs and NLP. From the ontology engineers, some have a background in Artificial Intelligence and NLP, others have a background in formal logic, and some are more focused on operational ontologies and consultancy in the semantics and standardization sector. As reported by the interviewees, they do not normally use OE methodologies in practice, although some of them used SABiO [32] before. They have a customized OE approach and set of best practices, and 6 out the 11 often use Competency Questions.
Throughout these interviews, we explored how the integration of LLMs in the ontology extension workflow can be established. Their input led to the decision to design a human-LLM collaboration framework, emphasizing that LLMs can provide added value for certain OE tasks but not fully automate them. This requires a critical assessment of the LLMs along the way and an approach that is informative but not normative.
The second phase in the DSR approach was to develop a set of functional and non-functional requirements for the process framework design. As the academic literature was still too limited to elicit requirements for our framework, we used the input from the interviews in the previous phase to elicit functional and non-functional requirements and to understand the current process of ontology extension.
We modeled the current ontology extension process (i.e., the manual extension process without the use of LLMs) based on the responses to the interviewees in the previous phase. They were asked several questions about their current OE process, including they steps they execute, the tools and methods they use, and the stakeholders they normally collaborate with. The result of this modeling process was a lfowchart with OE phases, including the actual activities that ontology engineers conduct to develop an extension, the OE tools they use, and several stakeholders involved in some of the activities of the process.
After conducting the interviews, eliciting an initial list of requirements, and modeling the current ontology extension process, we organized a focus group session. All the interviewees that previously participated were invited to the focus group session (but not all of them were present). This session was aimed to validate both the previously elicited requirements and the ontology extension workflow generated. To validate the requirements, we conducted a live survey where we asked “What are the requirements for a human-LLM collaboration framework for ontology extension?”. The participants’ responses where shown in the session and we asked them to vote for their preferred answers. We saved the answers and votes in a table and used this table to validate the requirements previously elicited from the interviews transcripts and to elicit new requirements. The final list of requirements is provided in Table 2 (Appendix A.1).
In the third phase of the DSR approach, we designed a prototype of the process framework, taking the requirements into account. As we wanted the framework to support ontology engineers by not only guiding the ontology extension process steps and activities but also integrating the LLMs for each activity, we mapped the NLP capabilities of LLMs to the downstream tasks in the current ontology extension process. We also analyzed recent literature on the application of LLMs in OE (which, during our research, has been increasingly growing). Our analysis provided an overview of the current use of LLMs for OE tasks, existing LLM-based methods and tools, configurations, and prompt engineering techniques. We arrived at a prototype of the comprehensive process framework for ontology extension by integrating these findings into a process framework design. 3.4. Fourth and fifth phases: Demonstrate and evaluate artifact In our last research phase in the DSR approach, we chose the use case Semantic Explanation and
about autonomous systems working and navigating in the greenhouse. The SENS extension to the CGO was previously derived manually without using LLMs or LLM-based tools. Therefore, we used the manually generated extension as the gold standard for comparison with the extension produced by using our prototype. 2https://appl-ai-tno.nl/projects/sens 3https://gitlab.com/ddings/common-greenhouse-ontology
Recent work on the application of LLMs for OE tasks evaluates the performance of LLMs with
quantitative metrics such as precision and recall against a ground truth dataset [
In this section, we first present the results of the interviews, covering diferent topics regarding the complexity of the ontology extension process and the use of LLMs, in Section 4.1. Next, in Section 4.2, we show the prototype design produced based on the list of requirements elicited from the interviews.
lfowchart diagram and visualization of the themes analyzed in the interviews.
Throughout the interviews, we explored several themes about OE and LLMs. First, we asked all interviewees about the processes or methods they follow when extending an existing ontology, the tools they most often use, and the stakeholders normally involved. Because there is no standard process for OE, nor for extending an existing ontology, we used this information to map as precisely as possible the current ontology extension process. This is described below in Section 4.1.1. In addition, we asked the interviewees about the problems they experience when extending an ontology, the opportunities and the concerns they perceive about the application of LLMs to OE. The results are explained below in Section 4.1.2. 4.1.1. The current ontology extension process There are multiple methodologies for developing ontologies, but there is no consensus on which one should be the standard. In fact, OE methodologies have the highest impact on the relevance and (re)use of ontologies when these are adapted to the needs of the ontology engineers and the requirements of the project [40].
One of the outcomes of the interviews described in Section 3 is that the process of ontology extension is neither static nor linear. It heavily varies depending on diferent factors such as time and budget constraints for the project, the type of ontology (i.e., reference or operational ontology), the availability of standards or structured documentation for the domain or specific use case (“Bottom-up approach”) or the need to extract the knowledge from the domain experts (“Top-down approach”), and even personal preferences such as the choice of using Compentency Questions (CQs). 4https://gitlab.com/eswc2025/ontology-extension-with-llms/-/tree/main/Interviews
Aspect Summary Existing ontologies are too Ontologies grow in size and complexity over time, making big/complex governance and management challenging.
Long discussions with stake- Reaching agreement often involves lengthy debates, especially holders when domain experts lack technical knowledge.
High expertise required Extending ontologies demands high abstraction and familiarity with complex or unfamiliar domains.
Manual maintenance OE processes are largely manual; existing tools are often inadequate for eficient workflows.
Hallucinations LLMs sometimes generate plausible but incorrect responses, which can be hard to detect in formal OE.
Environmental and ethical LLMs have high energy usage; proprietary models pose risks concerns to data privacy and intellectual property.
Loss of enriching human OE fosters collaboration and shared understanding, which process could diminish if fully replaced by LLMs.
Creativity and inspiration LLMs can generate out-of-the-box ideas, especially during initial phases in OE.
Entity extraction LLMs can extract concepts and relationships from unstructured text, aiding information extraction.
Suggestions for best prac- LLMs could provide real-time advice on best practices and tices and syntax checking validate syntax during manual OE tasks.
SPARQL query generation LLMs can generate SPARQL queries from natural language, bridging gaps in human-machine communication.
Small and simple OE tasks LLMs can assist with repetitive or straightforward tasks, reducing manual efort.
To illustrate the process of ontology extension, we modeled the diferent phases, tasks, stakeholders, tools, and decisions to be made within a flowchart diagram. The phases, inspired by the methodology and set of best practices followed by the interviewees and by well-known methodologies such as SABiO [32] and HCOME [41], are Preparation; Conceptualization; Implementation; Verification; Exploitation; and Validation. Stakeholders include the ontology engineer, the domain experts, and the knowledge worker (here defined as the one responsible for importing and integrating the extended ontology within the information system that makes use of it). According to the interviewees, the most used tools are Protégé and TopBraid, as well as generic code editors and diagramming tools for ad-hoc visualizations of the ontology. 4.1.2. Problems in Ontology Engineering, and challenges and opportunities for using LLMs The main challenges and opportunities identified in OE can be divided in three categories: OE problems, concerns about LLMs, and opportunities for LLMs in OE. We provided an overview of the aspects discussed within each category in Table 1 below.
An overview of the causes that make the ontology extension process complex is shown in Figure 1. Key problems include the increasing size and complexity of existing ontologies, which complicates governance and system integration. Additionally, lengthy discussions with stakeholders often arise due to dificulties in translating domain knowledge into reusable and standardized ontology elements, particularly when domain experts lack technical expertise. High levels of abstraction and familiarity with complex or unfamiliar domains are required, making the process demanding for engineers. Moreover, OE tasks remain largely manual, with limited availability of robust tools to streamline workflows.
LLMs can provide opportunities for reducing the complexity in the ontology extension process, but it also raised concerns with the interviewees. They include technical, environmental, and ethical dimensions. A significant technical challenge that the interviewees identified is the issue of hallucinations, where LLMs generate responses that seem correct but include inaccuracies. This is particularly problematic in the field of OE, where logical consistency and factual accuracy are essential. Users may struggle to detect them, increasing the risk of flawed ontologies.
Environmental and ethical concerns were also raised by the interviewees, such as the substantial energy and water consumption associated with LLM training and maintenance. Additionally, the use of proprietary LLMs introduces risks related to data privacy and intellectual property, particularly when handling sensitive information. Finally, there is apprehension about the potential loss of the collaborative human process integral to OE. Figure 2 shows a visualization of the concerns mentioned by the interviewees about the application of LLMs to OE.
Despite the concerns, the interviewees saw several promising opportunities for enhancing ontology extension with LLMs. During the initial stages, they can ofer creative and out-of-the-box suggestions for identifying domain concepts, especially when ontology engineers are unfamiliar with the domain or the required extension. LLMs can also aid in identifying concepts and relationships from unstructured text. Other opportunities are giving advice, SPARQL query generation, and repetitive tasks such as populating an ontology. 4.2. The human-LLM collaboration ontology extension workflow Stemming from the problems in the ontology extension process and from the challenges and opportunities for LLMs to be introduced into this process, discussed above, a set of 22 high-level requirements for a human-LLM collaboration framework for ontology extension were identified. The complete list can be found in Table 2 in the Appendix. This output has been used to design a process framework for ontology extension using LLMs. In our process framework, we aim at a step-by-step interaction of the user with the LLM using simple prompts.
The human-LLM collaboration process framework for ontology extension (Figure 4) is an augmented version of the current process outlined above in Section 4.1.1. We have chosen the current ontology extension process as a template because it already maps the downstream ontology extension tasks in the diferent phases of the process, namely Preparation; Conceptualization; Implementation; Verification; Exploitation; and Validation. In addition, the flowchart format provides flexibility since the ontology engineer can choose the sequence of tasks (transparent rounded boxes in Figure 4) to be executed depending on the specific needs, represented as questions in the diagram (gray rectangular boxes in Figure 4). The preparation phase focuses on gathering documentation about the ontology to be extended and the domain. In the conceptualization phase, the flowchart outlines the steps to reuse concepts from other ontologies (left-hand side), and to build a sub-ontology guided by CQs to then align it with the ontology to be extended (right-hand side). In the implementation and validation phases, the ontology extension is coded in a formal language and validated by using the CQs, visualizations of the ontology, and existing OE tools such as OOPS! [42].
To map the downstream tasks in the ontology extension process to the NLP capabilities of LLMs, we used the categorization of NLP tasks proposed in [43]. Based on the definition of the ontology extension task and its mapping to an NLP task. As an example, “Define ontology extension modules” can be mapped to “Text Classification”, and “Formulate CQs” can be mapped to “Keyphrase Generation”. We indicate which tasks can be assisted by LLMs with a white star-shaped icon with red text and a purple tag with a number in the flowchart diagram in Figure 4.
As seen the figure, most of the LLM-assisted tasks belong to the Preparation and Conceptualization phases. The interviews show that ontology engineers acknowledge the high complexity in these initial phases of the ontology extension process and the capabilities of LLMs to produce relevant ideas for augmenting human inspiration. For example, for task T-1.1 Research about the domain, LLMs can generate a well structured overview of the main aspects in a specific domain, including the relevant terminology [44]. In the Conceptualization phase, task T-2.5 consists of aligning the ontology extension with the ontology to be extended. As demonstrated by Amini et al. [36], LLMs can provide relevant suggestions for manual alignment by proposing 1-to-1 mappings.
As a result of the analysis, we propose a total of 15 ontology extension tasks could be facilitated by LLMs. In Figure 5 we provide a zoomed-in version of Figure 4 for better readability, specifically for phases 2) Conceptualization (Figure 5-a) and 4) Verification (Figure 5-b). For each task, we created a prompt template. As an example, Figure 6 shows (part of) a prompt template for task T-1.6 Create glossary of terms. All the prompt templates can be found in GitLab5.
Beyond the NLP capabilities of LLMs, we also examined recent publications such as the OntoChat
framework [
In addition to OOPS! [42] and FOOPS! [45], we integrated other OE tools in the ontology extension framework for human-LLM collaboration that are not currently used by the ontology engineers that we interviewed. These are Grafo6; OntoEditor [46]; OntoMetrics [47] and its updated version NEOntometrics [48]; WIDOCO [49]; and OnToology [50]. We selected these tools because they are still available and maintained [December 2024]. Although these tools will not eliminate the complexity inherent in the ontology extension process, they can assist with certain downstream tasks.
To evaluate the process framework presented in Figure 4, we executed the diferent LLM-assisted steps and compared them against a manually created ground truth. In this section, we discuss the manually created extension, in Section 5.1. Next, we present highlights from the human-LLM collaboration process framework in Section 5.2. All the results from the demonstration and evaluation, including the generated code, can be found in GitLab7.
With the goal of enabling semantic communication and interoperability between robots from diferent manufacturers in the greenhouse, the Common Greenhouse Ontology (CGO)8 [33] was extended with new concepts and relations within the Semantic Explanation and Navigation System (SENS)9 use case. The CGO is a public domain ontology focused on high-tech greenhouse infrastructure and the systems measuring the necessary data to efectively monitor and control the crops’ growth inside the greenhouse, such as climate data [33, 34]. Figure 7 illustrates the manual SENS extension to the CGO developed and used here as our gold standard. The visualization has been generated with TopBraid, using the code for the use case SENS in the public GitLab repository of the CGO10.
SENS assumes that the use of autonomous systems in high-tech greenhouses will grow in the near future. Consequently, heterogeneous autonomous systems (modeled as “cgo:MovingRobot”) will coexist in the greenhouse, together with human workers, such as growers, pickers, technicians, or managers (modeled under “dul:NaturalPerson”). The greenhouse is a tight space composed of narrow paths and dense vegetation in which human workers and autonomous systems or robots must perform diferent tasks concerning the crops and use specific materials and tools (modeled under “cgo:Tool”, extended from “dul:PhysicalObject”). These objects can be in motion (e.g., carts), or could potentially be hazardous in specific situations (e.g., weeding knives or scissors). Currently, robots can detect obstacles obstructing their way, but will only stop without notifying the human operator. This may lead to unnecessary idle times and delays. The goal of SENS is to improve this scenario concerning foreseeable but unexpected situations (modeled taxonomy under “cgo:UnexpectedButForeseeableSituation”) by enabling the robots to semantically communicate and explain to the human operator the obstacle encountered and the level of urgency of the situation (“cgo:hasSeverity” and “cgo:Severity”). With this information, the human operator can locate the robot in the greenhouse, judge the situation, and act accordingly (e.g., removing the obstacle from the robot’s way).
The manual extension to the CGO (i.e., without using LLMs) was conceptualized in 2023 by an experienced ontology engineer. The SENS extension development was based on the requirements of a dashboard in which notifications could be shown to the user, based on the knowledge about types of 7https://gitlab.com/eswc2025/ontology-extension-with-llms/-/tree/main/Demonstration_and_Evaluation?ref_type=heads 8https://gitlab.com/ddings/common-greenhouse-ontology 9https://appl-ai-tno.nl/projects/sens 10https://gitlab.com/ddings/common-greenhouse-ontology/-/tree/sens?ref_type=heads obstacles found in the greenhouse and types of situations and actions related to the characteristics of the obstacles or objects. The resulting extension is shown in Figure 7. The ontology engineer executed the following tasks: find reusable existing ontologies, define some competency questions, define new concepts/properties, find the best spot to place them in the existing CGO and add them using Topbraid Composer. From four ontologies identified as potentially reusable for the SENS use case extension, two ontologies were selected and reused: the SOMA ontology [51] and the Dolce/DUL ontology [52]. The CGO was extended by adding the concepts and relations for SENS one by one to the ontology. The process of manually extending the CGO with the SENS use case took approximately 40 hours.
To demonstrate and evaluate the human-LLM collaboration framework for ontology extension prototype we performed a walk-through of all the tasks in the process framework (Figure 4) that can be assisted by LLMs. All the prompt templates have been adjusted to the SENS use case, replacing the placeholders by the corresponding information concerning the CGO and the greenhouse domain, and using the description of the use case SENS. We opted to use a custom GPT Assistant11 using OpenAI’s GPT-4
(text of 639 words in PDF format), taken from its public GitLab repository, and a text file containing the CGO code (the full ontology) formatted as a Turtle triples file. We used a version of the CGO code before the implementation of SENS (i.e., the CGO code uploaded to the file store of the GPT Assistant does not contain any data related to the SENS use case).
Both manual and automated ontology extensions have distinct strengths and weaknesses (see Table 5
in the Appendix). The manual approach achieves a richer taxonomy and better reuse of other ontologies
compared to the LLM-assisted approach, where the taxonomy is simpler and GPT hallucinates ontologies
to reuse. However, the extension generated using the framework proposes an original model for the
robot’s decisions when encountering obstacles not considered in the gold standard’s development. For
each output of the tasks executed using the LLM (in this case the GPT Assistant), we carefully observed
the results, reflecting on the output of the LLM and focusing on the correctness and usefulness of the
task for the ontology engineer. All the inputs prompted to the LLM and all the outputs generated are
publicly available in our GitLab repository13. The main insights are:
• Preparation phase – Getting acquainted with the domain, the ontology to be extended,
and the extension: Tasks T-1.1, T-1.2, and T-1.3 produced relevant results that can be used
by the ontology engineer to get acquainted with the domain, the ontology to be extended and
the ontology extension. The outputs can serve as inspiration to the ontology engineer, but the
information must be reviewed and checked, for example by opening the ontology file in Protégé
or TopBraid. The LLM can complement the information provided by these tools, though not
replace them.
• Preparation phase – Gathering existing standards: The output of task T-1.4 was fully
hallucinated. Thus, this indicates that the ontology engineer should use conventional search
engines instead of an LLM for this task.
• Preparation phase – Defining business scenarios, creating a glossary of terms, and
extracting concepts and relations from existing standards: tasks T-1.5, T-1.6, and T-1.7
produced relevant suggestions with few-shot prompting and by making the instructions in the
prompts very specific.
• Conceptualization phase – Formulating Competency Questions: Though also highly
dependent on the quality of the documentation of the use case provided, the output of task T-2.1
was surprisingly relevant. We provided the complete list of CQs generated by the LLM in Table 3
of Appendix A.2. The quality achieved was not expected after examining the results of previous
11https://platform.openai.com/docs/assistants/overview
12https://openai.com/index/hello-gpt-4o/
13https://gitlab.com/eswc2025/ontology-extension-with-llms/-/tree/main/Demonstration_and_Evaluation?ref_type=heads
research on the topic [
After executing and assessing the results of these ontology extension activities, as guided by the process framework (Figure 4) we can conclude that, overall, the generated SENS extension as a result of the human-LLM collaboration process was correct and useful. The major issues are lack of depth and complexity (a human can generate more sub-classes and more and more sophisticated axioms) and the need to integrate the ontology to be extended with the ontology extension manually. The proposed process framework can be specially useful for beginner ontology engineers who are familiar with the basic concepts in ontology engineering (including domain experts with little technical experience in OE). Our framework supports them by providing a step-by-step guide based on best practices and automating a set of tasks so that the user has a starting point to further develop the extension.
It is important to highlight that, during the execution of the LLM-assisted tasks in the ontology extension process, the LLM hallucinated in 2 tasks: when asked about existing standards covering the extension use case; and when asked about existing ontologies to be reused. In our proposed framework, hallucinations are managed manually by the user of the framework. The user must be able to spot the hallucinations and manually correct the answer of the LLM, before continuing to the next task in the process.
The generation of the SENS extension using the process framework and the GPT Assistant took approximately 16 hours of a beginner ontology engineer. As previously mentioned, the manual SENS extension (the gold standard) was developed in approximately 40 hours by an experienced ontology engineer
In this paper, we present a human-LLM collaboration process framework for ontology extension. This framework supports the human ontology engineer and/or domain expert with an LLM in multiple tasks. To evaluate its qualitative performance, we applied the framework to extend an existing greenhouse ontology with new concepts and properties of the domain and compared the result to a manually generated extension, our gold standard. Furthermore, we evaluated each task’s output to determine how efectively an LLM reduces manual efort and enhances human creativity.
The main conclusion of our evaluation is that LLMs are a useful tool for the human ontology engineer to (1) get inspiration on where and how to add new concepts and properties, (2) deal with complex syntax definitions and repetitive tasks, and (3) verify whether the extended ontology conforms to the initially defined requirements and competency questions. Our experiments with the greenhouse ontology show that the proposed framework can lower the entry barriers to the field of ontology engineering, because it guides the ontology engineer and reduces manual efort in some of the tasks. However, due to the problematic hallucinations and because not all ontology engineers are familiar with the use of LLMs, additional training including critical thinking would be necessary for efective interaction of the user of the process framework with the LLM.
We noticed that important preconditions for successful usage of LLMs are (1) specific fine-tuning of the prompt inputs to the LLM, (2) a user-friendly interface with the LLM that provides task-specific support, and (3) last but not least, expert involvement to check the LLM output for correctness and completeness, and to mitigate hallucinations. Furthermore, our interviews demonstrate that more general aspects, that we did not study, such as transparency, trustworthiness, security and environmental impact, should be taken into account when deciding to use an LLM. Previous work on this topic shows that the full automation of the ontology engineering process is currently not possible due to the limitations of LLMs. Our work further indicates that ontology engineers do not favor full automation, as they view this process as inherently human and highly enriching.
In this research, we manually evaluated the process framework design using only the common greenhouse ontology. Since our framework is applicable to any ontology, we plan to further assess the approach with other ontologies across various domains to enhance the generalizability of our results. LLMs may lack the specialized knowledge required for fields like biomedical or legal domains, and thus more expert intervention might be needed when extending ontologies in these domains using our framework. In addition, future work on this topic includes a more extensive evaluation of the framework by ontology engineers with diferent levels of experience; a study the potential of applying ifne-tuned open-source LLMs for specific and smaller tasks such as creating a glossary of terms from unstructured text sources; and the development a user-friendly ontology engineering tool based in our framework, to seamlessly integrate LLMs within the ontology engineering toolkit.
This research was supported by the project NXTGEN Hightech. Our gratitude goes to the Common Greenhouse Ontology (CGO) team and the Semantic Explanation and Navigation System (SENS) project for providing us with the use case and material to conduct our research. Profound gratitude is extended to the participants in the interviews. The authors have no competing interests to declare that are relevant to the content of this article.
During the preparation of this work, the author(s) used Microsoft Copilot in order to: Grammar, spelling check, and improving the readability of some sentences. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. A scenario-based methodology for ontology development, Applied Ontology 10 (2015) 107–145. doi:10.3233/AO-150145, publisher: IOS Press. [29] K. Pefers, T. Tuunanen, M. A. Rothenberger, S. Chatterjee, A Design Science Research Methodology for Information Systems Research, Journal of Management Information Systems 24 (2007) 45–77. doi:10.2753/MIS0742-1222240302. [30] A. Hevner, S. Chatterjee, Design Science Research in Information Systems, in: A. Hevner, S. Chatterjee (Eds.), Design Research in Information Systems: Theory and Practice, Springer US, Boston, MA, 2010, pp. 9–22. doi:10.1007/978-1-4419-5653-8_2. [31] P. Johannesson, E. Perjons, An Introduction to Design Science, Springer International Publishing,
Cham, 2021. doi:10.1007/978-3-030-78132-3. [32] R. Falbo, SABiO: Systematic approach for building ontologies, in: 1st Joint Workshop ONTO.COM/ODISE on Ontologies in Conceptual Modeling and Information Systems Engineering, Co-located with FOIS, CEUR, Rio de Janeiro, RJ, Brazil, 2014. URL: https://www. semanticscholar.org/paper/SABiO%3A-Systematic-Approach-for-Building-Ontologies-Falbo/ 2660676b0d784d46d3c407793c0cee0c10e76c5b. [33] R. M. Bakker, R. van Drie, C. Bouter, S. van Leeuwen, L. van Rooijen, J. Top, The Common Greenhouse Ontology: An Ontology Describing Components, Properties, and Measurements inside the Greenhouse, Engineering Proceedings 9 (2021) 27. doi:10.3390/engproc2021009027, publisher: Multidisciplinary Digital Publishing Institute. [34] J. Verhoosel, B. Nouwt, R. M. Bakker, A. Sapounas, B. Slager, Digitizing Agriculture: A Datahub for Semantic Interoperability in Data-Driven Integrated Greenhouse Systems, ResearchGate, Rhodes Island, Greece, 2023. [35] S. Hertling, H. Paulheim, OLaLa: Ontology Matching with Large Language Models, in: Proceedings of the 12th Knowledge Capture Conference 2023, K-CAP ’23, Association for Computing Machinery, New York, NY, USA, 2023, pp. 131–139. doi:10.1145/3587259.3627571. [36] R. Amini, S. S. Norouzi, P. Hitzler, R. Amini, Towards Complex Ontology Alignment using Large
Language Models, arXiv preprint arXiv:2404.10329 (2024). doi:10.48550/arXiv.2404.10329. [37] N. F. Noy, D. L. McGuinness, Ontology Development 101: A Guide to Creating Your First Ontology (2001). URL: http://protege.stanford.edu/publications/ontology_development/ontology101.pdf. [38] M. McDaniel, V. C. Storey, V. Sugumaran, Assessing the quality of domain ontologies: Metrics and an automated ranking system, Data & Knowledge Engineering 115 (2018) 32–47. doi:10.1016/j. datak.2018.02.001. [39] R. M. Bakker, M. H. T. De Boer, Dynamic Knowledge Graph Evaluation (2024). URL: https: //www.techrxiv.org/users/791451/articles/1070833-dynamic-knowledge-graph-evaluation. [40] K. I. Kotis, G. A. Vouros, D. Spiliotopoulos, Ontology engineering methodologies for the evolution of living and reused ontologies: status, trends, findings and recommendations, The Knowledge Engineering Review 35 (2020) e4. doi:10.1017/S0269888920000065. [41] K. I. Kotis, G. A. Vouros, Human-centered ontology engineering: The HCOME methodology,
Knowledge and Information Systems 10 (2006) 109–131. doi:10.1007/s10115-005-0227-4. [42] M. Poveda-Villalón, A. Gómez-Pérez, M. C. Suárez-Figueroa, OOPS! (OntOlogy Pitfall Scanner!): An On-line Tool for Ontology Evaluation, International Journal on Semantic Web and Information Systems (IJSWIS) 10 (2014) 7–34. doi:10.4018/ijswis.2014040102. [43] K. S. Kalyan, A survey of GPT-3 family large language models including ChatGPT and GPT-4,
Natural Language Processing Journal 6 (2024) 100048. doi:10.1016/j.nlp.2023.100048. [44] R. M. Bakker, D. L. Di Scala, From Text to Knowledge Graph: Comparing Relation Extraction Methods in a Practical Context, in: First International Workshop on Generative Neuro-Symbolic AI, co-located with ESWC 2024, Hersonissos, Crete, Greece, 2024. [45] D. Garijo, O. Corcho, M. Poveda-Villalón, Foops!: An ontology pitfall scanner for the fair principles 2980 (2021). URL: http://ceur-ws.org/Vol-2980/paper321.pdf. [46] A. Hemid, W. Shabbir, A. Khiat, C. Lange, C. Quix, S. Decker, OntoEditor: Real-Time Collaboration via Distributed Version Control for Ontology Development, in: A. Meroño Peñuela, A. Dimou, R. Troncy, O. Hartig, M. Acosta, M. Alam, H. Paulheim, P. Lisena (Eds.), The Semantic Web,
A.1. Requirements for the design of the human-LLM collaboration process framework To extract the requirements inductively and directly from the users’ needs and values, we have used the information gathered through the interview process for the exploration of the problem. As exemplified by Johannesson and Perjons [31], the requirements for the outlined artifact can follow from the root causes of the problem that the artifact will try to address and, ultimately, solve. As illustrated with the Ishikawa diagram presented in Section 4.1.2 (Figure 1), the problem addressed within this research is the complexity in the ontology extension process, and the root causes fall within diferent categories. However, solving these problems is not enough if the concerns and the opportunities that the ontology engineers perceive are not considered. Thus, based on the three themes explored within the content analysis of the interviews, the requirements aim to: 1. Address the complexity of ontology extension (Theme 1 - Figure 1). 2. Reduce the concerns about the use of LLMs for Ontology Engineering (Theme 2 - Figure 2). 3. Leverage the opportunities for the use of LLMs for Ontology Engineering (Theme 3 - Figure 3).
Table 2 shows the complete list of high-level requirements elicited from the interviews with the 11 professionals in OE and LLMs.
The transcripts of the interviews were analyzed using content analysis, following Erlingsson and Brysiewicz [54] and Seljemo et al. [55]. For each interview transcript, we extracted meaning units, condensed meaning units, and grouped these into categories, for each theme. An example of the analysis is provided below in Figure 11. We used the condensed meaning units to elicit the requirements. The resulting design requirements were validated within a focus group session with ontology engineers. From all requirements, only 1 requirement was not fulfilled: Requirement 1.5 (”Support backward compatibility of ontology changes”). Although crucial, its implementation is highly complex and challenging in practice.
A.2. Additional results from the demonstration and evaluation of the process framework Table 3 below shows the Competency Questions generated by the GPT Assistant (based on GPT-4o) using our human-LLM collaboration process framework (Figure 4). This is the output of the task T-2.1: Formulate Competency Question in the process.
As an input for this task, the business scenarios output provided in task 1.5 was homogenized by restructuring the information exchanged between the Robot and the Human Operator and replacing the actions of the Human Operator by Receives the notification and monitors the robot’s decision for all the business scenarios to reflect that the communication is only one-way, from the autonomous system to the human operator. Suggestions of new concepts to add to the ontology given in the input of task 1.1 were also added to the prompt.
As shown in Figure 5-a, after formulating the CQs, these shall be reviewed with the domain experts and other stakeholders such as the knowledge worker and the knowledge user. In this case, we compared these 17 CQs to the SENS gold standard (Figure 7) and we decided that 16 of them are relevant to the use case.
To verify that the generated ontology extension can answer the generated CQs (after populating the
ontology in task T-4.1: Populate ontology extension with instances) we propose task T-4.2: Verify all
CQs can be answered according to test cases. This task has been proposed previously in OntoChat [
Finally, in Table 5 we provide a comparison summary between the ontology extension generated manually (SENS Gold Standard) and the ontology extension generated using our human-LLM collaboration framework, with the model GPT-4o (SENS-GPT).
Competency Question 1. What type of obstacle has the robot detected?
GPT: Answered
Yes 2. Where is the robot currently located? 3. What is the severity level of the detected obstacle? 4. At what time was the obstacle detected? 5. What decision has the robot made regarding the detected obstacle? 6. Why has the robot made the specific decision for obstacle avoidance? 7. Is the detected obstacle a human? 8. Is the detected obstacle a struc- Yes tural component of the greenhouse? 9. What is the new location of the No robot after obstacle avoidance? 10. How much time did the robot take to avoid the obstacle? 11. Were there any deviations from the original task after obstacle avoidance? 12. What static obstacle has been detected that requires human intervention? 13. How many obstacles have Yes been detected within a specified time-frame? 14. What are the possible actions Yes the robot can take after detecting an obstacle? 15. What are the recommenda- Yes tions from the robot for human intervention in case of a static obstacle? 16. What unexpected movable ob- Yes stacle has been detected?
No Yes Yes Yes Yes Yes Yes Yes Yes
GPT: Explanation The ontology includes the class Situation with properties like hasObstacleType and hasDetectedSituation which indicate the type of the detected obstacle.
The ontology does not specify a property for the current location of a robot.
The ontology includes a Situation class with property hasSeverityLevel which can indicate the severity of a detected obstacle.
The ontology includes a Situation class with property hasDetectedTime which specifies the detection time of an obstacle.
The ontology includes properties hasOutcomeAction, hasRecommendedAction within Situation and linked to Robot through actions.
The ontology includes the hasExplanation, hasOutcomeAction, hasRecommendedAction within the Situation class.
The property hasObstacleType in Situation can be used to specify if the obstacle is a Human.
The property hasObstacleType in Situation can be used to specify if the obstacle is a GreenhousePart.
The ontology does not include properties for specifying the new location after obstacle avoidance.
The Situation class includes hasDetectedTime and hasOutcomeAction which further have time-related properties for calculating duration.
The hasTaskDeviation property within Action or OutcomeAction can be used to indicate deviations.
The ontology includes StaticObstacleAlert subclass of Situation and can be linked to hasObstacleType and hasRecommendedAction.
The hasDetectedTime property within Situation can be used to filter based on time frames.
The hasRecommendedAction property within Situation can be used for this purpose.
The hasRecommendedAction property within StaticObstacleAlert can be used to specify recommendations.
The ontology includes a UnexpectedMovableObstacle subclass of Situation and properties like hasObstacleType for unexpected movable obstacles.
Correct / Incorrect Correct Incorrect Correct Correct Correct Correct Correct Correct Correct Correct Correct Incorrect Correct Correct Correct Correct Situations Obstacles Severity level of situations Possible recognitions Decision made by the robot when detecting an obstacle
SENS Gold Standard Taxonomy with 7 sub-classes including diferent situations depending on the position of the human when detected as an obstacle Extended from DUL ontology, including self-moving objects (such as humans) and movable objects (tools found in the greenhouse), and using the FixedObject already existing in the CGO Object property of an unexpected situation. Severity is a class with instances (Alarm, Info, Warning) that depend on the characteristics of the object that have an impact in the robot’s operation and the safety of the human Additional class to diferentiate recognitions from reasoned situations in the SENS dashboard Made outside the ontology, in the SENS dashboard using the data provided by the ontology model
SENS-GPT Taxonomy with 5 sub-classes (position of human not considered) Obstacle class created, with three subclasses: GreenhousePart (existing in the CGO), Human, and Object. Object has several sub-classes representing common tools found in the greenhouse Data property of Situation. The data type is a string that can be "High", "Medium", or "Low" depending on the obstacle found and its impact on the operation of the robot and the safety of the human (reasoned by GPT) Not included Explicitly modeled within the ontology extension using the object properties hasAction and hasRecommendedAction and data properties such as hasExplanation and hasTaskDeviation Only 1 ontology identified for reuse