Ontology engineering is often a complex, time-consuming and costly process that relies heavily on expert engineers. Even experienced ontology engineers introduce errors, such as incompleteness in terms of requirements, and fall into common ontology pitfalls, underscoring the challenge of producing high-quality ontologies. This PhD proposal aims to address these issues by creating an LLM-based assistant for both ontology development and ontology evaluation. The envisioned assistant will ofer suggestions during conceptual modelling, patternbased suggestions for class and property definitions, and real-time validation checks to identify modelling errors. By embedding these capabilities into a unified tool, the research seeks to reduce dependence on expert intervention, enabling mid-level and novice ontology engineers and organisations to develop reliable ontologies more independently, while simultaneously accelerating the workflow of expert ontologists. The outcome of this work will be a software tool that supports and streamlines the ontology engineering lifecycle-facilitating creation, error detection, and quality assessment-thereby making ontology creation faster, less error-prone, and more accessible to non-experts.
Ontology engineering is a challenging task that relies heavily on domain experts for both the creation
and the evaluation of ontologies. Even when developed by ontology experts, ontologies frequently
exhibit errors—ranging from pitfalls flagged by OOPS! (OntOlogy Pitfall Scanner!) [
Meanwhile, many real-world tasks have been (semi-)automated through large language models
(LLMs), such as GitHub Copilot [
Based on eXtreme Design (XD), ontology creation is usually an incremental process of developing
and evaluating [
In this work, we propose the development of an ontology engineering assistant that ofers contextaware suggestions during conceptual modelling and semi-automates CQ verification by providing suggestions as shown in Figure 1. By embedding LLMs guidance and validation into a single pipeline, the tool is designed to accelerate ontology creation, reduce dependence on expert intervention, and make ontology engineering more accessible to novice ontology engineers.
The complexity of ontology engineering, from creation to maintenance, coupled with the necessity for expert knowledge engineers, often makes it challenging for organisations to use semantic web technologies and holds back novice ontology engineers, resulting in increased costs for organisations and creating an obstacle when it comes to adapting semantic web technologies.
By introducing an ontology engineering assistant that incorporates LLMs to create suggestions for ontologists during the development and evaluation phase, this work stands to benefit multiple stakeholders. Expert ontologists can streamline repetitive processes and reduce the risk of common errors, while novices can leverage guidance to accelerate their learning and contribute more efectively to ontology projects. Organisations gain the ability to develop cleaner, more maintainable ontologies with reduced expert involvement, thereby lowering overhead and fostering a wider adoption of semantic technologies. Ultimately, this research makes ontology engineering more accessible to companies by making this task simpler, faster and less costly.
LLMs for ontology development/generation. Recent studies have used LLMs to draft OWL
ontologies from requirements. For example, Lippolis et al. [
LLMs for ontology evaluation. While the work mentioned in the previous paragraph has
incorporated evaluation for their generated ontologies, evaluation has rarely been the central concern. For
example, in Lippolis et al. [
In our recent study [
Thus, despite promising results, the use of LLMs for ontology evaluation, especially concerning functional adequacy, remains underdeveloped. In this context, Garijo et al. [20] leave the categorisation of existing resources on LLM use for ontology evaluation blank, highlighting this gap and pointing to directions where further research, with appropriate setup configurations, could yield more conclusive suggestions.
This proposal to create an ontology creation assistant has two parts: (i) ontology development assistance (ii) ontology evaluation assistance. The main research questions related to the first component of the tool—ontology development assistance—are related to the capability of LLMs in creating ontologies by themselves or assisting ontology engineers in the development phase. They are as follows: • RQ1.1 To what extent can LLMs be used to support the generation of ontologies that meet a predefined set of requirements? Which LLMs and what prompting techniques are more efective? • RQ1.2 What evaluation criteria are suitable for evaluating LLM-generated ontologies? • RQ1.3 What are the strengths and weaknesses of ontologies generated using LLMs? • RQ1.4 To what extent can LLMs assist ontology engineers in ontology development, and what are the benefits and drawbacks of a hybrid approach combining LLM suggestions with expert validation compared to traditional human-only methods?
Similarly to the development phase, the following research questions examine the capabilities of LLMs in supporting ontology engineers during the evaluation phase: • RQ2.1 To what extent can LLMs evaluate ontologies using CQ verification? • RQ2.2 To what extent can LLMs assist ontology engineers in evaluating ontologies through CQ verification, and what are the benefits and drawbacks of a hybrid approach combining LLM suggestions with expert validation compared to traditional human-only methods? This work hypothesises that LLMs can efectively assist ontology engineers in developing and evaluating ontologies.
There have been several experiments done to answer the research questions to some extent with reproducible results [21], and some work in the future work section will try to give more complete answers. The results are divided into two sections: (i) ontology development and (ii) ontology evaluation.
The research questions RQ1.1–RQ1.3 have been answered partially based on our previous work.
In our first work, Saeedizade and Blomqvist [
(a) Initial Experiment: Finding the best LLMs and prompting techniques [
(b) Main experiment, evaluating the generated ontologies manually.
To extend this work and provide a better answer for RQ1.1 and also answer RQ1.2 and RQ1.3, we
performed (To be presented in ESWC 2025) [
There were some risks related to the generalizability of LLMs on domain-specific tasks, which could
result in LLMs performing well on one domain but poorly on another. We performed [
We developed a prototype assistant for ontology evaluation, as described in our recent work [
In the upper part of Figure 4, a CQ, its context, and an ontology are provided to an LLM, which is then prompted to assess whether the ontology correctly models the given CQ. The LLM responds with a binary answer (yes or no), which is subsequently compared against a gold standard to evaluate performance. Using the o1-preview model, this automated approach achieved a macro-F1 score of 0.68.
The lower part of the figure illustrates the semi-automated evaluation process. In this setup, we presented the output generated by the LLM to ontology engineers, who were then asked to determine whether the ontology adequately models the CQ, using the LLM’s suggestion as guidance with the ontology opened in Protégé. We deliberately mixed both correct and incorrect LLM suggestions across CQs that were either correctly or incorrectly modelled in the ontology. The results reveal that participants’ performance was significantly afected by the accuracy of the LLM’s predictions. Specifically, correct LLM suggestions improved human performance by 13%, while incorrect suggestions caused a decline of 26%. However, because the LLM provided more correct suggestions than incorrect ones, the opposing efects largely cancelled each other out, resulting in no net improvement in human performance.
In future work, we plan to extend this tool by incorporating visualisations and additional interactive features, guided by the feedback received during our experimental evaluation.
To evaluate the hypothesis in our chosen setup, the usefulness of LLMs in assisting ontology engineers in developing and evaluating ontologies should be assessed by means of a user-based study. We conducted some experiments to partially answer the mentioned research questions through a manual evaluation of LLMs’ outputs by ontology engineers. Furthermore, we created a prototype to measure users’ performance only in ontology evaluation with CQ verification. Overall, the main evaluation of tools for ontology engineering with LLMs as assistants should be done by measuring users’ performance in ontology engineering, with and without LLMs’ suggestions, and comparing their performance in each task.
After creating a tool for ontology engineering, we should hire/invite ontology engineers to develop and evaluate ontologies following our setup. To evaluate the users’ performance using the tool, we will measure how accurately and eficiently ontology engineers perform the task of ontology engineering with the tool when (i) LLMs’ suggestions are available to a user, and (ii) without LLMs’ suggestions. The performance can be measured by measuring time to show how long it took users to complete the task, and assessing how accurate users were in each setting regarding ontology development and evaluation metrics in line with the selected CQ, such as measuring the proportion of correctly modelled CQs, quality of the final ontology concerning OOPS! warnings and expert evaluation.
In this paper, I propose creating an ontology engineering assistant tool. This tool helps ontologists to develop and evaluate ontologies by leveraging LLMs. Specifically, the approach relies on suggestions from LLMs to address relevant CQ throughout the ontology development process. For the ontology development part of the tool, the next step involves designing the appropriate setup and creating a tool, followed by defining a task that systematically measures its efectiveness via human-in-the-loop evaluation. Furthermore, during the ontology evaluation phase, we will capitalise on the user feedback we received to further enhance the tool by incorporating additional features and refined suggestions.
The author declares the use of generative AI tools in preparing this manuscript. Specifically, ChatGPT were used for sentence polishing, paraphrasing, and grammar fixing; all AI-generated text was reviewed, edited, and approved by the author, who takes full responsibility for the final content.
I would like to express my gratitude to my supervisors, Eva Blomqvist and Robin Keskisärkkä, for their guidance, encouragement, and expertise throughout this work. Their insights and support have been instrumental in shaping both the direction and the quality of this work. [20] D. Garijo, M. Poveda-Villalón, E. Amador-Domínguez, Z. Wang, R. García-Castro, O. Corcho, Llms for ontology engineering: A landscape of tasks and benchmarking challenges (2022). [21] M. J. Saeedizade, R. Alharbi, H. B. Giglou, A. S. Lippolis, E. Blomqvist, V. Tamma, F. Grasso, T. R. Payne, J. D’Souza, S. Auer, et al., A framework for assessing llm consistency in knowledge engineering, 2025.