As generative AI tools become more widespread, students are increasingly using them for assistance with complex tasks such as modeling ontology constraints. While the success of large language models have been widely explored, in-use applications remain underdeveloped, and experimental findings are often inaccessible to students or novice engineers. As a result, learners do not fully benefit from AI-assisted support or fail to critically engage with AI generated outputs. To bridge this gap, we propose a transparent, research-informed AI Assistant framework that follows hybrid intelligence principles and aims to support Knowledge Engineering education, with a focus on modeling logical ontology constraints. Preliminary results suggest that such a system can improve the accuracy of student-generated ontology models by over 10%.
Knowledge Engineering (KE) comprises a range of activities such as knowledge acquisition and its
representation in semantic models, such as ontologies [
While large language models (LLMs) can provide support, they also come with inherent challenges
and limitations such as lack of reasoning skills [
In the context of KE, the usage of an LLM which lacks the necessary capabilities to perform a concrete
KE task, can fail to improve the quality of the developed resource and may even degrade it [
To address these limitations and extend the state of the art, we propose an AI Assistant framework relying on hybrid (Human-AI) intelligence principles, designed to support students in ontology creation and evaluation with focus on correct constraint (i.e., cardinality, universal and existential quantifiers) modeling. By combining multiple LLMs and assigning them to sub-tasks according to their experimentally observed strengths, the application ofers context-sensitive support throughout diferent stages of the modeling process. Moreover, we aim to build trace-based transparency into the framework by advanced system tracing. Such a setup would allow for contextual explanations of generated outputs, a computation of custom confidence scores based on past performance and collection of traces which can support future adaptations of the system. The envisioned framework would allow students to benefit from AI-generated assistance while minimizing the risk of incorrect or misleading outputs. Ultimately, the proposed approach seeks to improve both the quality of student-generated ontologies and the learning experience itself. Preliminary result suggest that the AI-assisted approach can increase the accuracy of created ontology models by over 10%.
We propose an AI Assistant designed to (1) support students in detecting and classifying constraintrelated modeling errors, (2) explain detected mistakes in an accessible and educational manner and (3) generate possible corrections. The system, illustrated in Fig. 1, comprises several key components: First, multiple LLMs are integrated to create a multi-functional AI Assistant that leverages the complementary strengths of diferent models. Second, an audit layer is incorporated to ensure system transparency and trustworthiness. Third, a feedback loop involving both student and instructor input fosters a hybrid human–AI co-learning process and supports continuous system adaptation.
adopt workflow based on LLM performance & availability
inform initial workflow design
multi-LLM workflow
frequent mistakes LLM availability Ontology models constraints
provides assistance
AI assistance metadata AI assistance ratings LLM capability scores
A u d it L a y e r interaction details AI assistance annotations LLM availability LLM updated capability scores
invoke review
Multi-LLM Workflow. The system distributes KE task responsibilities according to research-reported
strengths of individual models in [
Trace-based Transparency. To mitigate the challenges of transparency and traceability associated with
multi-agent workflows, we integrate a Semantic-Web-based audit layer into the system following the
AuditMAI methodology [
These audit traces can be leveraged to provide contextual information for each AI suggestion, such as which LLM generated the output and why it was chosen (e.g., superior past performance, fallback in case of unavailability). In addition, the audit layer can be utilized to compute custom confidence scores, based on model consensus and performance history, ofering users insight into the reliability of AI outputs. This design encourages critical engagement with AI-generated content, rather than passive acceptance.
Co-learning and adaptation. Building on ideas from human–AI delegation frameworks [14], the system includes a Workflow Manager that operates outside the human–AI team, consisting of the student and AI Assistant. When recurring issues or misclassifications are flagged by the auditing layer, workflows can be adapted, such as switching LLMs for specific tasks, or escalated to instructors for review. This enables targeted intervention and the resulting feedback loop fosters co-learning: students gain from AI guidance, while the system improves through ongoing human oversight and real-world educational interaction.
We leverage findings and collected annotations from a recent experimental assessment of LLM
capabilities [
In comparison, AI-assisted workflows using GPT-4o, a model most familiar to students [
It should be noted that the simulated workflows do not yet incorporate AI-generated mistake explanations and do not consider cases in which students would revise their own models instead of fully relying on AI-generated suggestions. The LLM-generated explanations and contextual information provided by the auditing layer have the potential to further improve ontology modeling accuracy by fostering deeper understanding and enabling informed student revisions. In future work, we will implement the proposed framework and utilize it in Knowledge Engineering university courses. To assess the efectiveness and usability of the system, we plan a number of comprehensive user studies, including feedback surveys and interviews with students and instructors. This evaluation will inform future refinements of the framework and provide empirical insights into how LLM-based AI Assistants can be responsibly and meaningfully integrated into Knowledge Engineering education. This research was funded in whole or in part by the Austrian Science Fund (FWF) BILAI (10.55776/COE12) and HOnEst (V 745) projects. For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. Additionally, the work was supported by the PERKS (101120323) project, co-funded by the European Union. Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.
During the preparation of this work the authors used ChatGPT4 in order to suggest improvements to the readability and language of the manuscript. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article. [14] A. Fuchs, A. Passarella, M. Conti, Optimizing delegation in collaborative human-ai hybrid teams,
ACM Trans. Auton. Adapt. Syst. 19 (2024). [15] H. Johnston, R. F. Wells, E. M. Shanks, T. Boey, B. N. Parsons, Student perspectives on the use of generative artificial intelligence technologies in higher education, International Journal for Educational Integrity 20 (2024) 2.