Non-technical users often face a significant barrier when first attempting to explore complex knowledge graphs (KGs). We define this challenge as the Initial Exploration Problem, characterised by three interrelated barriers: ontology opacity, query incapacity, and scope uncertainty. This paper investigates how large language models (LLMs) can support domain experts in addressing this problem by automatically generating template-style competency questions (CQs) for the Virtual Record Treasury of Ireland (VRTI) KG. These templates are not user-facing themselves, but serve as scafolding for creating curated questions (CuQs), expert-validated, natural language questions that help new users begin meaningfully exploring the graph. We evaluate two LLMs (GPT-4o and Gemini 2.0 Flash) across twelve prompt configurations varying in scope and framing, and assess question quality using both semantic similarity to expert-authored CQs and detailed expert review. Our findings highlight how prompt design influences LLM output, and underscore the value of combining automated generation with expert curation. Ultimately, we propose a practical pipeline to support the creation of exploratory entry points tailored to user needs, helping domain experts craft better questions, and helping users take their first steps into meaningful KG exploration.
A Knowledge Graph (KG), as defined by Hogan et al. in 2021 [
However, a significant challenge arises when non-technical people attempt to explore an unfamiliar
humanities KG [
CEUR
ceur-ws.org
The Initial Exploration Problem is characterised by three interrelated dificulties:
1. Ontology Opacity: Inability to interpret class/property terms and hierarchies (e.g., abstract event
types such as E13_Attribute_Assignment or E65_Creation from the CIDOC CRM ontology
[
This gap presents a substantial barrier to new users engaging with KGs like the Virtual Record
Treasury of Ireland (VRTI) [
Tús Maith (/tuːs ˈmˠah/), coming from the Irish saying “Tús Maith Leath na hOibre” (“a good start is
half the work”), is a framework designed to address the Initial Exploration Problem. A key component of
Tús Maith is the concept of curated questions (CuQ). In this context, CuQs are a set of predefined natural
language questions curated by a domain expert to guide new users in beginning their exploration of
the KG. These questions act as a set of exploration starting points for new users while also giving them
an understanding of the types of questions the KG is able to answer. As will be explained in Section 2, a
past experiment carried out by the authors found that CuQs show promise when used in this context.
These CuQs are conceptually linked to competency questions (CQs), which are traditionally used in
ontology engineering to define the requirements a knowledge base should be able to meet [
This paper presents an experiment on the automatic generation of template-based CQs from the VRTI
KG using two popular Large Language Models (LLMs): ChatGPT-4o [
As seen in Figure 1, the template-style CQs are intended to support domain experts in crafting CuQs that help users explore the scope and potential of the KG. The lifecycle of a question in this process involves three stages: 1. Template-style competency question (CQ): A general template question generated by the
LLM based on class/property input, e.g. ”Where was a person born?”. 2. Filled-in/generated question: A template-style CQ filled in automatically with popular entities within the graph and/or entities the domain expert feels may be of interest, e.g. ”Where was Oscar Wilde born?”. 3. Curated question (CuQ): The filled-in question accepted or refined by a domain expert for presentation to users.
Prior to the experiment outlined in this paper, a small-scale evaluation involving 12 participants was
conducted by the authors to assess the current user-facing components of Tús Maith. These components,
which were implemented within the VRTI Explorer interface [
This section outlines the design of the experiment and the two-step evaluation process that was followed in order to explore the extent to which LLMs at default settings (temperature=1, top_p=1) can be used to generate template-style CQs for the VRTI KG. The design was motivated by the methodology presented in [22], which evaluated six LLMs across five RDF-based ontologies using zero- and few-shot prompting. While our experiment involved only two LLMs, a greater variety of prompts was used (12 distinct prompts as opposed to 6), and crucially, domain experts evaluated the output. The full results and code are openly available at [21], [23] ensuring transparency and reproducibility.
Twelve separate runs were conducted for each LLM, varying by prompt framing, shot type (one-shot or few-shot), and input scope as seen in Table 1. Input scope options are discussed in section 4.2. 1. Prompt 1: Simply instructs the LLM to generate questions from provided classes and properties. 2. Prompt 2: Includes the assigning of a role to the LLM (historical knowledge graph exploration assistant) to guide its generation strategy.
Listing 1 shows the base prompt used for all LLM runs, with conditional sections annotated according to experimental configurations (e.g., +Role, +Examples).
Listing 1: Template-style competency question prompt with condition annotations. # Template-style competency question prompt. # Modular components annotated with experimental conditions. # ------------------------------------------------------{INCLUDE_ROLE? You are a historical knowledge graph exploration assistant. Your task is to generate natural language, template-style competency questions that ordinary users might ask when exploring a historical knowledge graph.} [Condition(s): Prompt 2, +Role] I am providing you with a set of knowledge graph classes, properties {INCLUDE_RELATIONSHIPS? , and relationships}. Using only the provided information, generate 20 template-style competency questions that could be asked about this ontology. [Condition(s): All; relationships only in Subset and CIDOC CRM Relationships, +Relationships] A competency question in this context is defined as a natural language question that helps convey the scope and potential of a KG to users unfamiliar with its structure with the aim of supporting exploration. {INCLUDE_EXAMPLES? Examples of the type of questions I am looking for are: "Where was a person born?", "When did a person die?", "What type of place is this?".} [Condition(s): Few Shot, +Examples] Phrase questions in natural, user-friendly language, avoiding technical or ontology-specific terms e.g. use "was born" instead of "birth event".
Use template-style phrasing: general questions with placeholders like "a person", "a place" etc. Focus on creating general, reusable question templates that can later be filled with specific instance data.
Only use the given classes, properties {INCLUDE_RELATIONSHIPS? , and relationships}; do not invent additional concepts. [Condition(s): All; relationships only in Subset and CIDOC CRM Relationships, +Relationships] You may combine classes and properties if it helps express a useful question.
Phrase each competency question naturally, as if it were intended for a user exploring the knowledge graph.
The VRTI KG integrates multiple ontologies, including CIDOC CRM [
This subsection outlines the two-step evaluation process undertaken by the authors. The first step involved using SentenceBERT [24] to identify, for each LLM, the set of generated questions most semantically similar to a ground truth set of CQs in order to filter the question sets from 24 to 2. In the second step, those two question sets were analysed by domain experts with intimate knowledge of the VRTI KG, evaluating them across three dimensions: relevance, clarity, and answerability. They were also asked to evaluate whether the generated set of CQs contained questions that, in their opinion, could be added to the ground truth question set.
There is no existing benchmark CQ set for the VRTI KG. To establish a minimal ground truth, a historical domain expert with direct involvement in the KG’s development was asked to create a set of CQs [21]. Generated template-style CQs were first evaluated for semantic similarity to the expert-authored ones with SentenceBERT using the all-mpnet-base-v2 sentence-transformer model [25] and compared pairwise using cosine similarity.
Let ∈ ℝ ×
represent the similarity matrix, where is the number of generated questions and the number of expert-authored ones. Each element corresponds to the similarity between the -th generated question and the -th expert question.
For each generated question, the maximum similarity to any expert question was calculated: The average of these maximum similarity scores across all generated questions was then computed as: max = max
1≤≤ max = 1 ∑ max ∑ ( max ≥ ) where is the indicator function. The percentage of generated questions exceeding the threshold is In this experiment, a threshold of = 0.6 was selected, following precedent established in prior work evaluating LLM-generated CQs using SentenceBERT[22]. This value reflects a moderate level of semantic alignment, capturing cases where generated questions are meaningfully similar to expertauthored ones while avoiding overly lenient matches.
While semantic similarity provides an automated means of filtering and ranking generated questions, it does not assess important qualitative aspects such as interpretability, alignment with the KG’s intended usage, or answerability using only the information found within the KG. To address this, a human expert evaluation was conducted on a subset of the generated questions. Two historical domain experts, both very familiar with the content and structure of the VRTI KG, were asked to evaluate a sample of the LLM-generated questions. This sample comprised of one set of template-style CQs per LLM; the sets of questions that achieved the highest average semantic similarity to the expert-authored ground truth questions for each LLM out of the 12 prompt configurations described in Section 4.1. Each question was evaluated independently by both experts using a 5-point Likert scale [26] across three dimensions: 1. Relevance: How well does the question align with the purpose of a CQ in this context? (1 = Not relevant at all, 5 = Highly relevant) = Very unclear or confusing, 5 = Very clear) 2. Clarity: Is the question phrased in a clear, unambiguous, and grammatically correct manner? (1 3. Answerability: Can the question be reasonably answered using information contained within the VRTI KG? (1 = Not answerable at all, 5 = Fully answerable)
These three dimensions were selected to reflect the practical requirements of template-style CQs within the context of the Tús Maith pipeline; question templates that highlight the scope and potential of a KG that can later be filled in with specific entity information.
Relevance considers whether the question aligns well with the definition of a template-style CQ. Clarity addresses interpretability, particularly for non-technical users. Answerability considers whether the question can reasonably be answered using only the information currently present in the KG.
The experts were instructed to evaluate each question as a template, not a fully instantiated question. In addition to scoring individual questions, the experts were also invited to reflect on whether any questions present in the generated set but not in the original ground truth set could, in hindsight, be considered template-style CQ candidates. This qualitative feedback was intended to identify whether the LLM could highlight potential blind spots in the expert-authored ground truth CQs. Expert scores were aggregated to provide average scores per dimension across each LLM-generated set. This enabled comparison not only between individual questions but also between the two diferent LLMs used. This evaluation stage was essential in order to identify whether either/both/or neither LLM could produce not only semantically aligned questions with the ground-truth CQs, but ones that were perceived by experts as meaningful, understandable, and practically useful in the context of supporting exploration within a complex KG.
We first evaluated the similarity between 20 questions generated in each run and a set of 10 domain expert-created ground truth questions using SentenceBERT. The metric used was maximum similarity per question, with average values reported per run (Table 2). Across all runs, the highest-performing runs were Run 6 (the few-shot prompt with a subset of the classes, properties and CIDOC CRM relationship information) for GPT-4o, and Run 11 (the few-shot prompt with a role assigned and just the subset of classes and properties) for Gemini 2.0 Flash. The best run for GPT achieved an average max similarity of 0.418 (20% ≥ 0.6), while the best for Gemini achieved 0.480 (25% ≥ 0.6), indicating a closer lexical and semantic match to the expert-authored questions.
Notably, runs with access to CIDOC CRM relationship data (runs 3, 6, 9, and 12) did not consistently outperform others, and runs provided with the full set of classes and properties tended to underperform. This may suggest that narrowing the input scope to a manually filtered subset, without CIDOC CRM relationship data, helps steer LLM generation toward more meaningful or semantically focused questions.
For step two, the domain expert evaluation, question sets generated by runs 6 (GPT) and 11 (Gemini), the highest-scoring runs in Table 2, were used. Two domain experts independently expressed their judgment on these question sets, rating each individual question via an online questionnaire1 on relevance, clarity, and answerability using a 1-5 Likert scale. They were also asked whether any questions in the generated sets could be added to the ground truth set, to select which question set they preferred overall, and to provide free-text comments. The experts were not told which question set was generated by which LLM. The average scores and quantitative analysis of these scores across both experts are summarised in Table 3. The full set of results can be found here [21], and the code used to analyse the results can be found here [23].
While the GPT-generated question set scored slightly higher in average relevance and clarity, the Gemini-generated question set outperformed it in answerability, based on mean ratings across both experts. Inter-rater reliability varied significantly across dimensions and LLMs. For relevance, both models showed extremely low agreement between raters, with intraclass correlation coeficients (ICCs) close to zero or even negative (GPT ICC3k = –0.015, Gemini ICC3k = 0.144), indicating substantial subjectivity in how relevance was interpreted. This is further reflected in the high standard deviations 1Full questionnaire: https://forms.gle/SPcFTR8ruxp1DfUD7. This demo version no longer collects responses. (up to 2.038 for GPT), suggesting that domain experts varied widely in their opinion on how well the questions align with the purpose of a CQ in the context of Tús Maith (relevance). In contrast, GPT achieved strong inter-rater agreement for both clarity (ICC3k = 0.839) and answerability (ICC3k = 0.839), accompanied by comparatively lower standard deviations, indicating more consistent scoring and tighter consensus among experts. Gemini exhibited moderate reliability (clarity ICC3k = 0.539; answerability ICC3k = 0.671) but showed greater variability in scores, especially for answerability, where standard deviations reached 2.159 for domain expert 1. These combined metrics, ICCs capturing rating agreement between the domain experts, the average rating given by them to the generated question sets across the three criteria, and standard deviations indicating score dispersion, highlight the nuanced trade-ofs between average performance and consistency of expert assessments, which is critical in qualitative evaluations with subjective criteria.
Expert ratings reflected common patterns: questions rated poorly typically contained ambiguous wording or technical jargon that obscured their intent and reduced accessibility for non-technical users. In contrast, highly rated questions were clear, focused on meaningful historical information, and aligned well with the VRTI KG, making them both understandable and answerable. However, these findings also highlight the need to analyse questions comprehensively across all three dimensions rather than focusing on any single criterion. For example, some questions, such as “What is the relationship between a group and a person?”, were rated highly for relevance and clarity by both domain experts but scored lower on answerability due to it referring to information not explicitly contained within the VRTI KG. This underscores that a balance across all dimensions is essential for a question to function well as a template-style CQ. Table 4 presents paired examples of questions across all three evaluation dimensions (Relevance, Clarity, Answerability), showing examples of both the lowest-rated (≤2/5) and highest-rated (≥4/5) questions where both domain experts agreed in their assessments.
Expert comments ofer important qualitative insights. While both models generated syntactically valid questions, they were often overly complex or used domain-specific jargon unsuited to non-technical users, for example, “What type of attribute assignment was carried out?”. One expert observed that Gemini’s questions, although still technical, were easier to interpret and more closely matched the structure and content of the VRTI KG. This likely contributed to both experts ultimately preferring the Gemini-generated set. A notable outcome of the evaluation is that each expert identified at least one Gemini question that could be added to the ground truth CQ set, and one expert did so for GPT. An example of a question a domain expert that found could be added is ”What title does a person have?”. This indicates that LLM-generated questions, despite the current limitations, can reveal potential gaps in manually constructed question templates.
The comparison between SentenceBERT similarity and expert judgment reveals both alignment and divergence. GPT, despite slightly lower SentenceBERT similarity than Gemini, was rated more relevant and clear but less answerable. This suggests that it may generate more ambitious or abstract questions. Conversely, Gemini’s higher similarity and answerability scores indicate that it may produce questions closer to the current graph content and better suited to near-term user exploration needs. An important confounding factor may be KG class coverage. 7 of 10 (70%) expert ground truth questions were person entity related, whereas only 9 of 20 (45%) Gemini questions and 10 of 20 (50%) GPT questions in their respective best runs referred to people. Future work could investigate whether providing the LLM with class coverage ratios that reflect the most frequent entity types in the KG leads to improved question generation results.
Overall, the results highlight that while LLMs can generate semantically rich and partially relevant CQs, there is a trade-of between semantic similarity, readability, and groundedness in the graph. Narrowing input scope and including few-shot examples improved performance, and SentenceBERT evaluation aligned in part with expert assessments. However, inter-rater reliability metrics revealed variation in how experts interpreted key criteria like relevance, underscoring the need for clearer evaluation rubrics for domain experts to follow. While some subjectivity is inevitable, even among domain experts working in the same area provided with a clearer rubric, relevance still remains a valuable criterion as it is one that situates the question within the specific context of template-style CQs, in ways that clarity and answerability alone cannot capture. Further tuning is also needed to ensure the LLM output is suitable for non-expert end users and targeted toward the graph content as it exists currently.
This experiment presents an initial investigation into the use of LLMs for generating template-style CQs to support exploration of the VRTI KG. However, several limitations must be acknowledged.
Firstly, only two LLMs, GPT-4o and Gemini 2.0 Flash, were tested, both using their default settings. As these are commercial models, the prompts were executed via standard API access. Ethical and licensing considerations for such systems are relevant but were not a concern in this case due to the public nature of the VRTI KG. While these are among the most capable publicly available models, future work could explore open-source alternatives such as Mistral [28] or LLaMA 4 [29], which may ofer greater customisability or domain-specific fine-tuning. Expanding the range of models evaluated would provide a clearer picture of how model architecture and access conditions influence output quality.
Secondly, the evaluation involved only two domain experts. Although their feedback was detailed and constructive, a larger and more diverse pool of reviewers would allow for a more reliable assessment of the questions’ practical relevance, clarity, and answerability. Scaling up this aspect of the study would strengthen the validity of the findings and help identify consistent patterns in expert preference. Additionally, while the clarity criterion, like the others, implicitly includes multiple subcomponents (e.g., grammar, ambiguity, phrasing), no formal rubric was used to disaggregate or define these dimensions. The evaluation instead reflects common humanities practice, where expert interpretation and contextual judgment are often more appropriate than rigid scoring frameworks [ 30]. Nonetheless, future work may benefit from developing standardised evaluation rubrics to support clearer comparisons across datasets, models, and evaluators.
Thirdly, further analysis needs to be carried out on whether or not the generated questions can be translated into SPARQL without augmentation. Fourthly, while this study focuses on the early stages of the Tús Maith pipeline, namely, the automatic generation and expert review of question templates, it does not yet evaluate how these questions, when instantiated and validated by domain experts, perform when integrated into the VRTI Explorer interface. A natural next step is to assess the efectiveness of these questions in situ, especially in supporting non-technical users’ early-stage exploration of the graph. Measuring user engagement, comprehension, and exploratory behaviour in response to these questions could ofer valuable insights into their practical impact.
Finally, while the VRTI KG ofers a rich and historically grounded test case, generalisability remains an open question. Future work could explore the application of Tús Maith to other types of KGs. Generic KGs such as DBpedia [31] would ofer breadth, but a more meaningful test of the framework’s relevance would involve similarly complex and semantically rich graphs in the humanities domain. This would better reflect the kinds of exploratory challenges Tús Maith is designed to address.
A number of recent studies have investigated the use of LLMs in order to generate CQs from KGs, though often with diferent aims and assumptions than those guiding this work.
RevOnt [32] explores the reverse engineering of CQs from KGs such as Wikidata, drawing on a set of human-authored CQs from existing corpora as ground truth for evaluating output quality. Their definition of a CQ, “a typical query that an expert might want to submit to a knowledge base of its target domain, for a certain task”, reflects a focus on ontology modelling and evaluation. In contrast, this experiment defines a CQ as a natural language question that helps convey the scope and potential of a KG to users unfamiliar with its structure, with the aim of supporting exploration rather than formal assessment. Moreover, RevOnt assumes the availability of triple verbalisations, a condition that does not hold for the VRTI KG, which contains over 2.7 million non-verbalised triples and makes the RevOnt pipeline non-transferable to this context.
Other work [33] focuses on generating CQs from KGs constructed from PDF documents. After clustering similar entities and generating summaries, CQs are produced from these summaries and evaluated using zero- and few-shot prompting with cosine similarity scores. However, this introduces a degree of circularity by using the summary to both generate and evaluate the questions. Their work also took an LLM-as-a-judge evaluation approach without the involvement of domain experts, while in contrast, this experiment employs human expert evaluation to assess the quality, relevance, and answerability of generated CQs.
A related line of work by Alharbi et al. spans two papers: the first retrofits CQs to existing ontologies by prompting LLMs to generate multiple questions per triple [34], and the second extends this with an analysis of model settings and prompt design strategies across LLMs [35]. Both papers are motivated by the needs of ontology engineers, aiming to support ontology testing, reuse, and design evaluation rather than exploratory search. While methodologically thorough, both studies are situated within ontology engineering workflows and do not integrate expert-led curation or address the needs of non-technical users. However, their exploration of varied prompt formulations supports the hypothesis also under investigation in this paper, that LLMs may behave diferently when cast into specific roles.
While surveys such as [
Lastly, recent work on CQ benchmarking [37] provides criteria for evaluating CQ quality across syntactic, semantic, and domain-specific dimensions, including SPARQL transformability. While useful as background for designing evaluation strategies, these frameworks are not yet widely adopted and assume the existence of a ground truth CQ set, which, up until this experiment, did not exist for the VRTI KG.
This paper has presented Tús Maith as a potential solution to the Initial Exploration Problem, a challenge formally defined here as the set of barriers non-technical users face when first attempting to explore complex KGs. It has also explored the application of LLMs to the generation of CQ templates for a complex historical KG, with the aim of integrating them into the Tús Maith pipeline. KGs such as the VRTI KG, which contains over 2.7 million RDF triples and has been constructed using a variety of often complex ontologies, are typically too large and intricate for domain experts to reasonably recall all relevant classes, properties, and relationships needed to formulate a comprehensive set of template-style CQs manually. In this context, LLMs serve as a cognitive aid, helping experts identify plausible CQ templates that they feel reflect the structure and content of the graph. By combining prompt engineering with structured experimentation across two state-of-the-art LLMs, we have assessed to what extent LLMs can generate CQs to support Tús Maith. Our findings show that, with minimal input, LLMs can generate a promising number of relevant, clear, and answerable questions, particularly when provided with a reduced number of classes and properties and few-shot examples. However, diferences in model behaviour, input scope, and prompt design influence output quality. Domain expert evaluation highlighted the potential of LLM-assisted question generation as a curatorial aid, but also underscored the need for expert oversight in validating and refining outputs. Future work will include extending the CQ evaluation using a clearer criteria rubric across a broader pool of experts, performing an end-to-end evaluation of the Tús Maith pipeline in practice, and investigating its generalisability to other similarly complex KGs. Ultimately, this work supports a vision of a semi-automated curation pipeline that blends LLM-generated scafolding with domain expertise, enabling non-technical end users access to exploration starting points to complex KGs in the humanities domain.
The research conducted in this publication was funded by the Irish Research Council under project ID GOIPG/2021/1165 and supported by the Research Ireland ADAPT Centre for Digital Content Technology (grant number 13/RC/2106_P2).
During the preparation of this work, the author(s) used Chat-GPT-4 in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.