Competency Questions (CQs) are used guide ontology development, yet formulating them in such a way as to align them to the stakeholder needs remains challenging. This paper presents a comparative analysis of three CQ elicitation methods: manual authoring by ontology engineers; template-based instantiation; and automated generation using diferent LLMs (GPT-4.1, Gemini 2.5). Each CQ is evaluated across dimensions of suitability, readability, and complexity. To facilitate this evaluation we introduce AskCQ, a dataset of 204 CQs derived from a shared user story in the cultural heritage domain. Our results show that manually authored CQs are consistently more acceptable, readable, and concise. LLM-generated CQs are more complex and diverse but require refinement. These findings highlight the importance of human expertise and suggest potential hybrid approaches.
To investigate the impact of diferent Competency Question (CQ) elicitation strategies on the characteristics of the resulting questions, we conducted a comparative analysis across three representative
CEUR
ceur-ws.org approaches: fully manual authoring; template-based instantiation; and automatic generation via Large Language Models (LLMs). Each approach was applied independently to the same ontology requirement source to ensure a fair and controlled comparison.
1. Manual (Human-Authored): Two ontology engineers (HA-1 and HA-2), each with over five
years of professional experience in ontology design and requirement engineering, independently
read and interpreted the same user story. Based solely on their expert understanding of the
personas, goals, and informational needs described therein, each formulated a set of CQs without
constraints on format or style. This condition serves as the expert-driven baseline and reflects
common manual practice in ontology development.
2. Template-Based (Pattern Instantiation): An ontology engineer with similar domain
experience instantiated a curated set of 19 CQ patterns derived from Ren et al. [
All three approaches were applied to the same textual requirement source: a detailed user story developed
for a cultural heritage ontology use case. The story, adapted from the methodology proposed by de
Berardinis et. al. [
To assess the quality and characteristics of the generated CQs, we adopted a multi-dimensional,
mixedmethods evaluation framework encompassing both qualitative expert judgment and quantitative feature
analysis: CQ suitability, structural and semantic properties, and inter-method agreement.
1. Suitability (Expert Evaluation): Each CQ was independently reviewed by three ontology experts,
who rated its acceptability for guiding ontology engineering in the context of the user story. Scores
ranged from -3 (unanimous rejection) to +3 (unanimous acceptance). The experts were not provided
with explicit criteria to preserve their interpretive autonomy, analogous to the elicitation setup. A
Fleiss’ Kappa of = 0.35 indicated fair inter-expert agreement.
2. Readability: Each CQ was assessed to gauge its ease of understanding. We assess readability in a
similar way to Ciroku, et al. [
• Flesch-Kincaid Grade Level (FKGL) — Estimates the education level (U.S. grade) required for
comprehension [
Overall, these four dimensions are expected to provide complementary perspectives on CQ complexity.
A CQ might be semantically complex (e.g., requiring navigation of intricate partonomy or causality
relations) yet linguistically simple (e.g., “What caused this event?”), scoring high on requirement metrics
but low on linguistic/syntactic ones. Conversely, a CQ might lack ontological complexity but is phrased
using complex sentence structures, thereby scoring high on syntactic metrics but low on semantic ones.
5. Semantic Overlap: To analyse the semantic characteristics of CQ sets generated by diferent
approaches, we conducted a study on their embeddings. This utilised Sentence-BERT embeddings from
the all-MiniLM-L6-v2 model [
This was performed in both directions, i.e. for the coverage of Set A by Set B (Set A ← Set B) we determine: – Mean Maximum Similarity (MMS). For each CQ embedding e, in Set A, its maximum cosine similarity to any CQ embedding in Set B, → = max cos(e, , e, ), was identified. The mean of these → scores (and the standard deviation) indicates how well each CQ in Set A is semantically represented by its closest counterpart in Set B. A higher mean suggests stronger semantic parallels ofered by Set B. 2Scores were computed using the textstat Python library and interpreted comparatively, given the short, interrogative nature of CQs.
The same metrics were computed for the coverage of Set B by Set A. • Bidirectional coverage: This symmetric metric quantifies the overall mutual semantic overlap.
cov + →co v , where → It was calculated as → cov is the number of CQs in Set A covered by Set B (i.e., → ≥ ), and →co v is t+h e number of CQs in Set B covered by Set A. Hence, a higher percentage indicates greater shared conceptual space between the two sets.
Together, these dimensions provide a comprehensive, multidimensional view of the suitability, expressiveness, and diversity of CQs produced by diferent elicitation methods, grounded in both expert assessment and computational analysis.
The results for the expert evaluation clearly favoured manually authored CQs. The manual method (HA-1 and HA-2) achieved a mean suitability score of 2.65, with 94.5% of questions accepted by a majority of annotators. This indicates that domain experts are highly efective at producing suitable CQs. The LLM-based methods (GPT-4.1 and Gemini) achieved an average score of 1.24 with 76.0% acceptance, suggesting moderate reliability. Pattern-based CQs scored lowest, with a mean suitability of 0.11 and only 50% acceptance. For readability, human-authored CQs had the lowest FKGL and DCR scores, indicating clearer phrasing. GPT-4.1 generated the most complex and least readable CQs (FKGL 11.64). LLM-generated CQs were also significantly longer (c0), richer in ontological references (c1), and more syntactically complex (c3) than manual or pattern-based ones. Figure 1 consolidates our findings, showing distinct feature profiles from the CQs generated by each elicitation approach. 3
The pairwise comparison results (Table 1) show that the cosine similarities between the centroids of most pairs are relatively high (typically ranging from 0.61 to 0.85). This suggests that, at a high level, all sets tend to address the same core thematic area defined by the user story. The lowest centroid similarities were observed in comparisons involving Gemini (e.g., 0.61 with HA-2), indicating its central theme might be slightly more distinct than the other sets.
CQ Feature Profile
Linguistic (c2) Syntactic (c3)
Requirement (c1) 0.25 0.50
Despite these relatively high centroid similarities, the specific semantic coverage between sets is low, denoting high degrees of novelty, i.e. a high number of CQs not previously generated. The percentage of CQs in one set covered by another (i.e., having a CQ in the other set with similarity ≥ 0.75) is consistently below 21%, and often below 10%. Between the two human annotators ( − 1 ↔ − 2 ), who shared a high centroid similarity (0.82), HA-2 covered 20.5% of HA-1’s CQs, and HA-1 covered 11.1% of HA-2’s CQs (HA-2 has 10 more CQs than HA-1), yielding a bidirectional coverage proportion of 15.3%.
Our findings suggest that CQs manually crafted by ontology engineers tend to demonstrate the highest suitability for OE, due to achieving better readability, lower complexity, and uniquely capturing inferential requirements (implicit functional requirements) essential for robust ontology design. While LLMs can produce relevant and thematically coherent outputs, the resulting CQs exhibited higher complexity, lower readability, and their semantic coverage, though broad, exhibits limited overlap with human-generated CQs and amongst each other. These results suggest that while LLMs can provide reasonable CQs, these are not comparable to expert authored ones, and that human expertise still remains critical in Ontology Engineering. Crucially, our insights on CQ characteristics and limitations of current automated approaches can be leveraged to directly inform and improve their elicitation methods, aiming to better align their outputs with the desiderata of ontology engineers.
Generative AI was only used in the experiments described in the paper, and no Gen AI tool was used to
compose or edit the text.
3A full discussion of the results and analysis for this evaluation is available in [