Generative AI (GenAI) solutions are likely to have a profound impact on the Knowledge Engineering (KE) field. Considerable research is needed to understand the extent to which various KE tasks can be performed with GenAI, how the performance of these tasks compares to human baselines, and how to efectively adapt KE workflows to make the best use of GenAI methods. To conduct such research, there is a need of collections of corpora of ontologies with a range of diverse characteristics to support systematic experimentation covering a broad variety of ontology types. We propose collecting such corpora and describe our ongoing eforts to collect ontologies created by students, as representative for the work of junior ontology engineers (beginners level knowledge engineering skills). We also create an ontology analysis workflow to extract key metadata from ontologies and associated reports, which we share with the community.
Recent reports of the success of Generative AI techniques, and in particular, large language models
(LLMs) on several natural language processing and other tasks [
For KE as a field, several exciting research questions await answering, just to name a few: • To what extent and with which performance level can LLMs perform KE tasks? As apparent from the very brief literature overview in the introductory paragraph above, investigation of this question has been pioneered for several KE tasks already. The community would further benefit from: (i) identifying the right granularity of KE tasks when investigating the use of LLMs; and subsequently (ii) agreeing on a catalogue of such tasks that can then be systematically investigated. • How does LLM performance compare to human performance? In KE, there are significant diferences between the characteristics of information artifacts created by engineers with diverse levels of expertise. Therefore, interesting insights could be won by comparing against knowledge artifacts created by contributors with various levels of expertise starting from junior experts (such as students) to experienced ontology engineers. • How can LLMs be best included in KE workflows?
This requires answering the previous research questions, i.e., understanding how well LLMs perform on individual KE tasks, which of these tasks can be fully covered by them and which still require human involvement. (M. Sabou)
CEUR
ceur-ws.org • How to support KE education with LLMs? Could LLMs give reliable feedback on modeling errors typical for junior ontology engineers? How can they be integrated in the process of teaching ontology engineering? • What are cognitive diferences between humans and LLMs during KE? Which human cognitive processes occurring during KE tasks are best reproducible with LLMs? And in which case do LLMs function fundamentally diferently from human agents?
Research Need. A pre-requisite for answering research questions such as the ones above in a systematic manner is the availability of corpora of ontologies that vary across several key dimensions in order to ofer a broad coverage of possible ontology types. For example, ontologies should difer in terms of the domains they cover, their size, their logical complexity, as well as the expertise level of their authors.
To partially address the research need described above we propose to collect ontologies created by junior ontology engineers, for example, students, as part of their training in ontology/knowledge engineering courses. Such ontologies are valuable as examples of domain models created by junior experts. As such, they could shed light on typical modeling errors made by beginners as a basis to assess whether LLMs make similar mistakes as junior experts, as well as to what extent they can identify these mistakes.
Therefore, our proposal is to collect corpora of such ontologies. We next describe our ongoing work in this direction, primarily focusing on a corpus of ontologies created by junior ontology engineers.
Following suit from the proposed idea, we have been collecting ontologies from student assignments from knowledge engineering courses that we have taught during the last decade. Next, we describe the current status of this corpus, as well as a processing workflow that extracts form ontologies their characteristics, competency questions and potential pitfalls. This extraction workflow is made publicly available and can already be used by fellow-researchers.
Corpus source. Ontologies were collected from student projects in ontology development courses at two institutions: the Vienna University of Technology (TU) and the Vienna University of Economics and Business Administration (WU). For the ontology development, students were instructed to model a domain of their choice (with suggestions including Music, Movies, University, or Tourism), create an ontology with a certain minimum number of concepts and relations, and extend it with specific constraints. They were required to formulate competency questions for their envisioned application. The courses were taught over a decade (2014-2024). TU courses were at the master’s level, with TU-ISS designed for beginners and TU-SAIKS for intermediate to advanced students. The WU course was at the bachelor’s level. We also collected the student report associated with each ontology as a source for identifying the ontology’s domain and competency questions. Each ontology and report was processed with the workflow described in Sect. 3.2 to extract metadata as exemplified in Table 2. Table 1 captures statistics about the corpus based on this metadata.
Corpus size. The corpus comprises 504 ontologies from 16 diferent classes across the three courses, with a significant majority originating from the TU-ISS course. The ontologies are in the range of small (avg. 26-27 classes/250-300 axioms) size for beginner courses or medium size (avg. 50 classes/2400 axioms) for the advanced course.
Domains covered by the ontologies are shown in Figure 1. TU-ISS and WU-K2 share a big focus on the movie domain, and each has a big occurrence in books, music, and fashion. In contrast, TU-SAIKS, the intermediate to advanced course, exhibits a more diverse range of domains, including energy, university management, banking, and various services. Although individual counts for these domains in TU-SAIKS are relatively low, they collectively represent a broad spectrum of interests.
Competency questions. TU-ISS and WU-K2 gathered more competency questions (CQs) due to higher enrollment in their introductory courses. Despite running the longest, TU-SAIKS, an advanced course with lower enrollment, contributed fewer CQs. However, the average number of CQs per ontology remained consistent across all courses.
To further expand research on LLM-enabled KE we argue for the need of collecting corpora of ontologies exhibiting variety in their key properties. Future work for our community could include: • Collecting more ontology corpora. We focus on collecting student-authored ontologies. We plan to expand our ontology corpus by processing the remaining ontologies from existing courses and collaborating with other institutions teaching ontology-related classes. Our ontology-metadata extraction workflow could enable fellow-researchers that collect ontologies as a by-product of their work to create their own ontology corpus either for internal experimentation or for sharing those with the community. • Extending ontology analysis workflows. The metadata collected about the ontologies is important for being able to select suitable ontologies for experimentation. In our work, we will enhance 1https://ollama.com/library/gemma2 2Workflow for metadata extraction from ontologies and reports: https://github.com/wu-semsys/ontology-analysis the current analysis workflow by integrating additional evaluation tools and developing an anonymization pipeline to facilitate ontology sharing while protecting privacy. • Establishing systematic evaluation campaigns. As ontology corpora become available, they could enable the community to establish benchmarks for ontology engineering tasks and launch evaluation campaigns to systematically study the impact of GenAI methods on knowledge engineering.
This work was funded by the Austrian Science Fund (FWF) Bilateral AI (Grant Nr. 10.55776/COE12) and HOnEst (V 745) projects. [15] Y. Rebboud, L. Tailhardat, P. Lisena, R. Troncy, Can LLMs Generate Competency Questions?, in:
The Semantic Web: ESWC Satellite Events, 2024. [16] M. Poveda-Villalón, A. Gómez-Pérez, M. C. Suárez-Figueroa, Oops!(ontology pitfall scanner!): An on-line tool for ontology evaluation, International Journal on Semantic Web and Information Systems (IJSWIS) 10 (2014) 7–34. [17] A. Prock, Hybrid Human-Machine ontology verification: Identifying common errors in ontologies by integrating human computation with ontology reasoners, Master’s thesis, TU Vienna, 2021.