Large Language Models (LLMs) have shown significant potential for ontology engineering. However, it is still unclear to what extent they are applicable to the task of domain-specific ontology generation. In this study, we explore the application of LLMs for automated ontology generation and evaluate their performance across diferent domains. Specifically, we investigate the generalizability of two state-of-the-art LLMsDeepSeek and o1-preview, both equipped with reasoning capabilitiesby generating ontologies from a set of competency questions (CQs) and related user stories. Our experimental setup comprises six distinct domains carried out in existing ontology engineering projects and a total of 95 curated CQs designed to test the models reasoning for ontology engineering. Our findings show that with both LLMs, the performance of the experiments is remarkably consistent across all domains, indicating that these methods are capable of generalizing ontology generation tasks irrespective of the domain. These results highlight the potential of LLM-based approaches in achieving scalable and domain-agnostic ontology construction and lay the groundwork for further research into enhancing automated reasoning and knowledge representation techniques.
The rapid evolution of large language models (LLMs) has transformed numerous areas of natural
language processing as well as knowledge engineering, and ontology engineering is no exception.
Ontologiesformal representations of knowledge that enable semantic interoperabilityhave long
been essential in domains ranging from healthcare to environmental science. However, traditional
ontology engineering, as shown by requirement-based methods such as eXtreme Design [
Building on these earlier studies that showed the feasibility of LLM-supported ontology generation, this paper extends the investigation by suggesting an ontology generation prompt for reasoning models and examining the generalizability of automated ontology generation across diverse and more specific domains. Neither of these aspects were so far addressed in previous work. In particular, we explore the performance of two state-of-the-art LLMsDeepSeek and OpenAI o1-previewwhich are equipped with reasoning capabilities. By leveraging a dataset
(CC BY 4.0). CEUR
ceur-ws.org comprising 95 carefully curated competency questions paired with corresponding user stories, from specific knowledge domains, our approach systematically tests whether these models can generate good ontology drafts based on requirements that capture domain-specific needs.
The contributions of our study are the following: First, we present an automated pipeline for ontology generation, including a prompt strategy, that harnesses the advanced reasoning capabilities of the most recent commercial and open-source LLMs to interpret natural language requirements and suggest corresponding ontological modules. Second, we evaluate this approach across six distinct knowledge domains, providing empirical evidence of its consistent performance and domain-agnostic potential1.
The remainder of this paper is organized as follows. Section 2 surveys the relevant literature in LLM-based knowledge engineering. Section 3 establishes the preliminary definitions and concepts used in the remainder of the paper. In Section 4, we describe our methodology, including dataset creation and details about the prompt. Section 5 shows our experimental setup and evaluation framework. Section 6 presents and discusses our results, Section 7 addresses the discussion, and Section 8 focuses on limitations and potential risks. Finally, Section 9 concludes the paper and outlines directions for future research.
Ontology generation based on requirements using large language models has been recently
studied. In Benson et al.[
In this section, we present and define terminology that will be used throughout this paper,
building on the one already outlined in Lippolis et al. [
Ontology Generation: We define ontology generation as the process of creating formal representations of knowledge by identifying the necessary classes, properties, and axioms to capture domain-specific concepts and their relationships. This process can be performed manually or supported by automated techniques, such as LLMs, to draft ontologies from natural language inputs in the form of requirements.
Modelling Competency Question: Following the definitions outlined in previous work [
Minor issue: Similar to [
Reasoning Capability of LLMs: The reasoning capability of LLMs refers to their ability to generate logically structured and coherent responses by recognizing and using patterns learned from large amounts of text. This definition includes simulating various forms of reasoningsuch as deducing, generalizing from examples, and drawing analogiesoften by processing information step-by-step (a process sometimes called chain-of-thought, which is natively employed in models like o1-preview). Importantly, LLM reasoning is based on statistical patterns rather than true human understanding or deliberate logic.
In this section, we outline the methodology employed in this study. We begin by describing the dataset creation process and its statistical properties, followed by the explanation of the prompting technique used in this work. Finally, we present the pipeline for ontology generation.
To evaluate ontologies generated by LLMs with reasoning capabilities, we constructed a dataset comprising 95 competency questions (CQs) with 12 ontology stories across 6 distinct domains. Each dataset entry includes: (i) a CQ, (ii) a user story (providing the contextual background), (iii) the binary label of the dificulty level of the CQ (easy or hard), and (iv) the domain of the ontology. Table 1 shows statistics for the dataset.
The CQs were extracted from four sources: Onto-DESIDE 2 (domain: Circular Economy), the Polifonia Project (domains: Music and Events)[13], the WHOW Project (domain: Water and Health) [14] and AquaDiva (domains: Microbe Habitat and Carbon and Nitrogen Cycling)[15]. In particular, these derive from carefully curated large-scale projects, with the fourth one being a semi-automatically annotated dataset.
To ensure balanced and comparable evaluations, each CQ was classified into one of two categories, “Easy” and “Hard”, based on the complexity required for its formal representation. An LLM, o1-preview, was used to assign these labels by estimating the dificulty level of classes and properties needed to model the CQ. Specifically, if a CQ required at most 2 classes and 1 property (either a data or an object property), it was classified as “Easy”; otherwise, it was classified as “Hard”. The labels were then checked manually by an ontology engineer. This labelling strategy was implemented to maintain a consistent distribution of CQ dificulties across all domains.
The labelling of the dataset for the verification of CQs has been carried out manually by two
ontology engineers. Following the error classification outlined in previous work [
A prompt serves as an instruction for LLMs to generate responses to a given input. The prompt
employed in this study follows a few-shot prompting technique, with some examples, building on
and distinguishing it from previous approaches which mainly employed a decomposed prompting
technique [
We used the Independent Ontology Generation method described in [
In this section, we describe the experimental setup. To run the experiments, we used Azure API to call o1-preview and the original DeepSeek API with default hyperparameters. The prompt with a fixed example (few-shot) is appended with ontology story and CQ and sent to the API to get the resulting output file. The results are saved in a Turtle file.
LLMs: In the present study, we have selected two large language models with reasoning capabilities: DeepSeek and OpenAI o1-preview. This selection was motivated by the desire to perform a comparative analysis between the current state-of-the-art closed-source and opensource LLMs. The OpenAI o1-preview model was accessed through the Microsoft Azure API, whereas DeepSeek was used according to the specifications detailed in its oficial documentation 3.
Prompt: The prompt used in this work is available in the supplementary materials on GitHub4. Initially, several prompt configurations were evaluated, including versions with and without examples (i.e., few-shot and zero-shot settings). The results demonstrated that prompts incorporating examples consistently outperformed those without, justifying their use. Furthermore, the impact of varying the complexity of the examples was examined in a separate experiment involving multiple ontology generation tasks. In the interest of conciseness, the detailed results of these additional experiments are not reported here.
Hyperparameters: The default hyperparameters for the LLMs, including temperature and penalty, were employed throughout this study.
Overall, both the o1-preview and DeepSeek models exhibited high and comparable accuracy. Initially, nine and ten CQs remained unmodeled for the o1-preview and DeepSeek models, respectively, out of a total of 95 CQs. However, when minor modelling issues are excluded, the number of unmodeled CQs decreases to eight for the o1-preview model and five for the DeepSeek model.
Table 2 provides a breakdown of mistakes on both easy and hard CQs by domain. Regarding the o1-preview scores, performance was consistent across all domains, suggesting that the ontology generation method is broadly applicable rather than being limited to a specific domain. With slight diferences, although DeepSeek generally demonstrated similar consistency, the scores for the Events domain were noticeably lower, indicating a need for further investigation into the model’s performance in that particular domain.
Our results reveal a performance trade-of between the two models. Specifically, while o1-preview
tends to commit more “minor” errors, DeepSeek is prone to a higher incidence of incomplete
modellings. When we consider cases with “minor” mistakes equivalent to complete modellings,
both models demonstrate similar performance across the domains. This finding suggests that
diferences in error types may not translate into significant disparities in overall outcomes.
3https://api-docs.deepseek.com/
4https://github.com/dersuchendee/Domain-OntoGen
Domain generalizability. Our work extends previous findings by Lippolis et al. [
Reasoning models architecture. One possible explanation for the similar outputs observed from both DeepSeek and o1-preview is their reasoning capabilities. It is plausible that these models benefit from mutual fine-tuning or shared architectural strengths. The collective evidence points to a significant enhancement in reasoning with respect to models with non-reasoning capabilities, which appears to underlie their comparable performance.
Dificulty levels of the requirements. Furthermore, the analysis indicates that both models handle hard and easy CQs with similar performance. This observation challenges the common assumption that increased question complexity necessarily results in lower scores. Instead, our ifndings suggest that the models are robust enough to process questions of varying complexity without a marked performance penalty. Nonetheless, this is limited to only one evaluation criterion.
Error analysis. Finally, a closer look at the errors reveals that the majority occur solely within the Events domain. The corresponding CQs in this category are much shorter than those in other domains, but there are multiple user stories that serve as requirements. This pattern could suggest that longer, more descriptive CQs could provide additional context that aids the models in processing and reasoning, leading to improved performance, especially if multiple scenarios are provided. Therefore, the lack of standardization in requirements is also fundamental when dealing with automated techniques.
Overall, these insights highlight the importance of both model architecture and the design and selection of requirements in achieving balanced and generalizable performance.
One potential risk associated with this study is dataset leakage, which may impact the quality and reliability of the LLM-generated ontology. Specifically, in certain domains, an LLM may have encountered parts of the dataset during pretraining, while in others, it relies solely on its reasoning capabilities. To the best of our knowledge, the CQs and ontology modules were not publicly available in a single dataset, reducing the likelihood that LLMs were trained on these elements in direct succession as consecutive tokens. Another limitation of this work is the exclusive use of reasoning-based LLMs. These models, while beneficial for ontology generation tasks, are generally resource-intensive and less widely available compared to standard LLMs. Additionally, the scope of this study is restricted to only two LLMs, which may limit the generalizability of the findings.
In this study, we investigated the potential of two LLMs, DeepSeek and o1-preview, with reasoning capabilities for automated ontology generation across six diverse domains. Our experiments, conducted on a dataset of 95 competency questions paired with user stories, reveal that both LLMs similarly exhibit robust and consistent performance across all domains and dificulty levels of the CQs. Overall, this work lays a foundation for further advancements in automated ontology engineering, highlighting the potential of LLMs to deliver scalable, domain-agnostic knowledge representation. Future research will build on these results to explore additional LLMs.
This project has received funding from the European Unions Horizon Europe research and innovation programme under grant agreements no. 10105- 8682 (Onto-DESIDE) and 101070588 (HACID), and is supported by the strategic research area Security Link. Additional financial support to this project was provided by NextGenerationEU under NRRP Grant agreement n. MUR IR0000008 - FOSSR ( CUP B83C220- 03950001 ). This work was also supported by the PhD scholarship “Discovery, Formalisation and Re-use of Knowledge Patterns and Graphs for the Science of Science”, funded by CNR-ISTC through the WHOW project (EU CEF programme grant agreement no. INEA/CEF/ICT/ A2019/2063229). Finally we thank OpenAI’s Researcher Access Program Grant for the API credits.
The authors have no competing interests to declare that are relevant to the content of this article.
During the preparation of this work, the author(s) used GPT-4 and Grammarly in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publications content. with medical context, in: 15th International Semantic Web Applications and Tools for Healthcare and Life Sciences (SWAT4HCLS 2024) (to appear), 2024. [12] M. Val-Calvo, M. Egaña Aranguren, J. Mulero-Hernández, G. Almagro-Hernández, P. Deshmukh, J. A. Bernabé-Díaz, P. Espinoza-Arias, J. L. Sánchez-Fernández, J. Mueller, J. T. Fernández-Breis, Ontogenix: Leveraging large language models for enhanced ontology engineering from datasets, Information Processing & Management 62 (2025) 104042. URL: https://www.sciencedirect.com/science/article/pii/S0306457324004011. doi:https: //doi.org/10.1016/j.ipm.2024.104042. [13] J. de Berardinis, V. A. Carriero, N. Jain, N. Lazzari, A. Meroño-Peñuela, A. Poltronieri, V. Presutti, The polifonia ontology network: Building a semantic backbone for musical heritage, in: International Semantic Web Conference, Springer, 2023, pp. 302–322. [14] A. S. Lippolis, G. Lodi, A. G. Nuzzolese, The water health open knowledge graph, Scientific
Data 12 (2025) 274. [15] A. Algergawy, H. Hamed, S. Thiel, B. König-Ries, Towards semantic annotation for scientific datasets, in: European Semantic Web Conference, Springer, 2024, pp. 164–167.