<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>U. Ahmed);</journal-title>
      </journal-title-group>
      <issn pub-type="ppub">1613-0073</issn>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Speech to Semantics: Enabling Conversational Access to Scholarly Knowledge Graph</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Umair Ahmed</string-name>
          <email>umair.ahmed@unicam.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andrea Polini</string-name>
          <email>andrea.polini@unicam.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nicolas Ferranti</string-name>
          <email>nicolas.ferranti@wu.ac.at</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Workshop</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="editor">
          <string-name>LanguageL Models, GPT</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Knowledge Graphs</institution>
          ,
          <addr-line>SPARQL, Natural Language Queries, Automated Query Generation, Semantic Search, Large</addr-line>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>University of Camerino (UNICAM)</institution>
          ,
          <addr-line>Via Andrea D'Accorso, 16, 62032 Camerino MC</addr-line>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>WU (Vienna University of Economics and Business)</institution>
          ,
          <addr-line>Welthandelsplatz 1, 1020, Vienna</addr-line>
        </aff>
      </contrib-group>
      <pub-date>
        <year>1987</year>
      </pub-date>
      <volume>000</volume>
      <fpage>0</fpage>
      <lpage>0002</lpage>
      <abstract>
        <p>Recent advancements in the representation of knowledge via knowledge graphs have paved an intuitive way to build scholarly knowledge for users and for artificial agents. Despite the expressiveness of knowledge graphs, accessing these knowledge graphs requires proficiency in a query language such as SPARQL, presenting a barrier for a multitude of users. In this study, we present a hybrid, end-to-end framework that (i) interprets user questions expressed in natural language, (ii) classifies each query into one of four target categories (conferences, authors, organizations, or papers) using a fine-tuned RoBERTa-Large model, (iii) synthesizes candidate SPARQL queries via a large language model (GPT-4o-mini) augmented with few-shot examples, and (iv) refines the raw query results by reranking either the SPARQL output or, when necessary, fallback candidate items retrieved through vector-space embeddings (indexed with FAISS). On a testbed of 92 manually crafted “gold-standard” SPARQL queries, our automated pipeline achieved over 96% overlap with expert results (≥ 70% overlap in 89/92 cases), with perfect consistency on conference, author, and organization queries and 90% coverage on paper queries given the semantic nature of queries. Moreover, our query-type classifier achieved 99% accuracy, demonstrating the reliability of schema selection. These results indicate that combining LLM-driven query synthesis with embeddings-based reranking delivers a robust, user-centric interface to scholarly knowledge graphs, enabling complex information retrieval without SPARQL expertise.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        The contemporary development and widespread adoption of knowledge graphs have transformed the
way scholarly knowledge is represented and retrieved. These knowledge graphs, in the context of
scholarly knowledge, ofer powerful tools for organizing and interpreting the vast and growing body of
scholarly content [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ]. Initiatives such as DBpedia [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] and Wikidata [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], along with more
domainspecific eforts like OpenResearch [
      </p>
      <sec id="sec-1-1">
        <title>5] and ScholarlyData [6], underscore the potential of knowledge</title>
        <p>graphs to harmonize diverse metadata sources, enable sophisticated analytical queries, and foster deeper
insights into the structure and dynamics of academic research.</p>
        <p>
          Despite their evident potential, these scholarly KGs primarily rely on SPARQL, a standardized query
language for RDF data, which is inherently complex and poses a significant barrier for widespread
adoption [
          <xref ref-type="bibr" rid="ref7 ref8">7, 8</xref>
          ]. Constructing efective SPARQL queries demands extensive knowledge of RDF schema
structure, semantic data models, and query syntax. These skills are rarely possessed by non-expert
users such as researchers, librarians, or policy-makers who often represent the primary beneficiaries of
such graphs [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ].
        </p>
        <p>
          In an efort to bridge this accessibility gap, various methods have been proposed to translate natural
language (NL) queries directly into structured query formats [
          <xref ref-type="bibr" rid="ref10 ref11">10, 11</xref>
          ]. Earlier approaches relied heavily
on manually crafted linguistic rules or structured templates, providing limited flexibility and requiring
        </p>
        <p>CEUR</p>
        <p>
          ceur-ws.org
extensive maintenance [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]. Recent advancements leveraging machine learning, particularly deep
learning and transformer-based language models, have opened promising avenues for more adaptive, scalable
query-generation methods [
          <xref ref-type="bibr" rid="ref13">13, 14</xref>
          ]. For instance, models like SPBERT and modern text-to-SPARQL
pipelines have significantly improved performance by learning syntactic and semantic representations
of SPARQL queries and domain knowledge jointly [
          <xref ref-type="bibr" rid="ref12">12, 14</xref>
          ]. Nonetheless, purely automated methods
still face challenges in consistently producing accurate and syntactically correct queries, particularly
for complex or nuanced user questions [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ].
        </p>
        <p>Furthermore, complementary research employing embedding-based retrieval techniques has
demonstrated significant success in handling semantic search and retrieval problems within knowledge bases
[15, 16]. Embeddings ofer powerful mechanisms to approximate semantic similarity, enabling rapid
approximate-nearest-neighbor searches using high-dimensional vector representations, thus
circumventing the strict matching constraints of purely symbolic query methods [17]. Tools such as FAISS
exemplify these advances by eficiently managing large-scale embedding indexes and enabling hybrid
or fallback retrieval strategies when symbolic queries fail [18].</p>
        <p>
          Motivated by these recent developments, this paper introduces a novel hybrid approach designed
explicitly to simplify access to scholarly KGs for non-expert users. Our method synergistically combines
multiple contemporary techniques:
• Natural-language classification leveraging RoBERTa-Large [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], fine-tuned to classify queries
into four target categories: conferences, authors, organizations, and papers.
• Large Language Model (LLM)-based SPARQL generation, using GPT-4o-mini [19], augmented
with type-specific few-shot examples, significantly improving the semantic precision of the
generated queries.
• Embedding-based reranking or fallback, utilizing vector-space embeddings (indexed via FAISS)
[18] to refine or directly retrieve results when symbolic SPARQL queries fail to return satisfactory
outputs.
        </p>
        <p>We empirically demonstrate the efectiveness of our hybrid system through rigorous evaluations
against a set of 92 manually curated (”gold-standard”) SPARQL queries. Our pipeline achieves over 96%
overlap with expert-generated results, surpassing the 70% overlap threshold consistently, and
demonstrating perfect consistency in queries targeting conferences, authors, and organizations, alongside
robust performance on queries targeting research papers.</p>
        <p>The subsequent sections are organized as follows: Section 2 presents related literature on Natural
Language-to-SPARQL translation, embedding-based retrieval, and hybrid approaches. Section 3 dives
into the background associated with this study, section 4 details our hybrid system architecture. Section
5 outlines the experimental setup, datasets, and evaluation metrics, followed by a detailed discussion of
results. Section 6 discusses limitations and future extensions. Section 7 concludes the study with an
intuitive summarization of findings.</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>2. Related Works</title>
      <sec id="sec-2-1">
        <title>2.1. Natural-Language Interfaces to Knowledge Graphs</title>
        <p>
          Early approaches to natural-language interfaces for querying structured knowledge graphs primarily
relied on rule-based grammars, templates, and manually engineered mappings between language
patterns and ontological constructs [
          <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
          ]. Systems such as Sparklis [20] facilitated interactive query
formulation, allowing users to iteratively refine queries through structured templates. Despite their
efectiveness, these systems required extensive manual maintenance, making them dificult to scale
across large, evolving knowledge bases.
        </p>
        <p>Recent advancements have shifted toward leveraging large language models (LLMs) to enhance
natural language interfaces for knowledge graphs. For instance, the EDGE system integrates LLMs to
facilitate natural language interactions with educational knowledge graphs, improving user experience
in data exploration [21]. Additionally, following framework utilizes Retrieval-Augmented Generation
(RAG) to enhance SPARQL query generation, reducing semantic errors and improving robustness [22].</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. Large Language Models (LLMs) for Query Generation</title>
        <p>The advent of pre-trained LLMs such as GPT-3 and GPT-4 has significantly reshaped the landscape of
semantic parsing and query synthesis. These models can translate natural-language queries into formal
queries like SPARQL with minimal annotated examples [23, 24]. Few-shot prompting techniques have
been shown to generalize query generation across diverse knowledge schemas, reducing the need for
extensive domain-specific annotations [ 24].</p>
        <p>To address challenges like hallucinations and out-of-distribution errors in LLM-generated queries,
Sharma et al. introduced PGMR (Post-Generation Memory Retrieval), a modular framework that
separates query structure generation from knowledge retrieval. This approach significantly reduces the
incidence of hallucinated URIs in SPARQL query generation [25]. Frameworks like FRASE leverage
frame-semantic structured representations to improve generalization in SPARQL query generation,
particularly in scenarios involving naturally phrased, template-free questions [26]. Moreover, SPARKLE
integrates knowledge graph structures directly into the decoding process of LLMs, reducing the
occurrence of inoperative query generations [27].</p>
      </sec>
      <sec id="sec-2-3">
        <title>2.3. Embedding-Based Retrieval and Reranking</title>
        <p>
          Embedding-based retrieval methods provide an efective complementary mechanism for structured
queries by representing entities and their relationships within vector spaces [
          <xref ref-type="bibr" rid="ref11 ref12">11, 12</xref>
          ]. These techniques
allow rapid approximate-nearest-neighbor search, exemplified by frameworks such as Facebook’s FAISS
library [28]. Embeddings have proven highly efective for entity retrieval and semantic search tasks
[
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], enabling eficient retrieval of relevant entities even in the absence of explicit symbolic matches.
        </p>
        <p>Hybrid approaches combining symbolic and neural methods have recently emerged, demonstrating
superior performance by reranking symbolic query results using embeddings [29, 30]. Graph-based
reranking methods have been explored to enhance the selection of optimal query graphs in Knowledge
Base Question Answering systems. Jia and Chen proposed a two-step approach involving initial
ranking followed by reranking of query graphs, leading to improved retrieval accuracy [31]. The KGR3
framework integrates retrieval, reasoning, and reranking components to enhance knowledge graph
completion tasks, using context-enriched modules to improve prediction accuracy [32]. Additionally,
ReranKGC introduces a cooperative retrieve-and-rerank framework for multi-hop knowledge graph
completion, improving accuracy and eficiency [ 33].</p>
      </sec>
      <sec id="sec-2-4">
        <title>2.4. Positioning Our Work</title>
        <p>Our research advances the state-of-the-art by integrating several successful strategies into a unified
pipeline tailored specifically for scholarly knowledge graphs. Specifically, our hybrid architecture
synthesizes:
• Natural-language classification using RoBERTa-Large, achieving near-perfect query-type
identification accuracy.
• LLM-driven (GPT-4o-mini) SPARQL generation augmented by few-shot examples, significantly
reducing reliance on manually annotated datasets.
• Embedding-based refinement (FAISS) employed both as a fallback mechanism and a reranking
technique to enhance retrieval accuracy.</p>
        <p>• LLM-based summarization of the results as according to the user query
Our work represents the systematic efort to integrate LLM-based SPARQL generation with
embeddingbased reranking specifically targeted at scholarly knowledge graphs, achieving expert-level retrieval
quality without extensive manual annotations or user SPARQL expertise.</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Background</title>
      <sec id="sec-3-1">
        <title>3.1. Scholarly Data Management: An Evolving Landscape</title>
        <p>
          The past decade has witnessed substantial evolution in scholarly data management, marked by a shift
from isolated bibliographic databases to richly structured, interconnected knowledge graphs (KGs).
Unlike traditional relational databases, scholarly knowledge graphs utilize semantic web standards such as
RDF (Resource Description Framework) and OWL (Web Ontology Language) to integrate heterogeneous
academic metadata seamlessly [34, 35, 36]. These structured graphs empower researchers, libraries,
and institutions to discover relationships among authors, publications, conferences, and institutions
previously concealed in disparate databases [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]. By enabling queries that traverse multiple dimensions,
such as author afiliations, citation patterns, and co-authorship networks, scholarly knowledge graphs
have profoundly transformed the capabilities of bibliometric analysis and academic discovery [35].
        </p>
        <p>
          Yet, the complexity inherent in these rich semantic structures introduces a notable barrier:
traditional querying mechanisms, particularly SPARQL (the standardized language for querying RDF-based
knowledge graphs), are notoriously dificult to master without extensive training [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ]. This limitation
confines scholarly graph utilization largely to data engineers or semantic web specialists, excluding the
broader academic community who stand to benefit most.
        </p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Towards Democratizing Access to Scholarly Knowledge</title>
        <p>
          The growing recognition of this limitation has prompted eforts toward developing intuitive,
userfriendly query interfaces. Central to this vision is the concept of natural language querying, a paradigm
that seeks to leverage users’ inherent linguistic capabilities to interact with complex data structures
without explicit technical knowledge [37]. However, natural language interfaces introduce their own
set of challenges: ambiguity, variability in linguistic expression, and dificulty translating informal
questions into precise symbolic queries [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ].
        </p>
        <p>Simultaneously, advancements in deep learning and natural language processing have opened
pathways toward bridging this gap. Particularly transformative are pre-trained large language models
(LLMs), capable of generating structured queries such as SPARQL from plain-language user prompts,
thereby drastically simplifying user interactions with semantic web resources [38]. Nonetheless, the
efective application of LLMs to scholarly data introduces practical concerns, especially regarding
accuracy, consistency, and the need for validation mechanisms ensuring generated queries yield meaningful,
high-quality results [38].</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. The Complementary Role of Embeddings</title>
        <p>A parallel and complementary technological advancement has emerged through embedding-based
retrieval methods. Embeddings provide semantic vector-space representations of entities (authors,
papers, conferences, organizations), capturing subtle semantic relationships beyond direct symbolic
matches [35]. By transforming entities into continuous vector representations, embeddings facilitate
rapid approximate nearest-neighbor searches, thus enabling eficient retrieval of semantically similar
results without rigid symbolic constraints [28]. This technology, operationalized through libraries like
FAISS, has become essential in scenarios demanding real-time semantic retrieval at scale [28].</p>
        <p>However, embedding-based approaches alone cannot fully leverage structured relationships explicitly
encoded in scholarly KGs. Thus, an ideal query mechanism combines symbolic querying (SPARQL) and
embedding-based semantic retrieval into a unified, hybrid framework that maximizes the strengths of
both paradigms: symbolic precision with embedding-driven semantic flexibility.</p>
      </sec>
      <sec id="sec-3-4">
        <title>3.4. Research Gap and Motivation</title>
        <p>Existing methods in scholarly KG querying predominantly focus either on precise symbolic querying
(via SPARQL) or purely semantic embedding-based retrieval, with relatively limited exploration of
how best to integrate these paradigms efectively in scholarly contexts. The complexity, diversity, and
specialized nature of scholarly metadata call for a bespoke hybrid solution specifically designed to
address domain-specific queries reliably and intuitively [ 36].</p>
        <p>The necessity for a robust hybrid approach is driven by practical considerations: researchers require
both accurate and comprehensive query results that align closely with their semantic intent. This
demands a mechanism that dynamically utilizes the strengths of symbolic SPARQL querying for
structured precision, complemented by embeddings to maintain semantic coherence and manage cases
where symbolic retrieval falls short or returns inadequate results [37].</p>
      </sec>
      <sec id="sec-3-5">
        <title>3.5. Our Approach: Bridging Symbolic and Semantic Retrieval</title>
        <p>Responding to this pressing need, our work presents a tailored hybrid architecture explicitly designed
for scholarly knowledge graphs. We incorporate advanced machine learning methods, specifically
a RoBERTa-Large classifier for robust query categorization, GPT-4o-mini for reliable generation of
SPARQL queries from natural language inputs, and FAISS-based embeddings for semantic refinement
and fallback retrieval. This synthesis uniquely positions our work, ofering a comprehensive solution
that addresses existing gaps by seamlessly integrating symbolic query precision and embedding-based
semantic flexibility in scholarly knowledge graphs.</p>
        <p>The subsequent sections of this paper detail this integration and empirically validate its efectiveness,
demonstrating that such a hybrid approach significantly enhances the accessibility and usability of
scholarly knowledge graphs for non-technical users.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Methodology</title>
      <p>This study proposes a comprehensive methodology to introduce a natural language interface for
querying the scholarly knowledge graph. It employs natural language processing (NLP) techniques,
large language models (LLM), semantic embeddings, and symbolic query execution to constitute into a
GraphRAG system underlying the natural language query interface.</p>
      <sec id="sec-4-1">
        <title>4.1. Knowledge Graph Preparation and Embeddings Generation</title>
        <p>4.1.1. Dataset Acquisition and Graph Setup
We initiated the study by acquiring the ScholarlyData knowledge graph, a prominent RDF-based dataset
encompassing structured metadata on conferences, research papers, authors, and organizations. The
ScholarlyData graph was imported into a semantic graph database environment (e.g., Apache Jena,
GraphDB), enabling eficient storage, indexing, and SPARQL querying.
4.1.2. Semantic Embeddings Creation
To support embedding-based retrieval, we computed separate semantic embeddings for each of the four
primary entity categories within the scholarly knowledge graph:
• Conferences
• Authors
• Organizations
• Papers</p>
        <p>Embeddings were generated using a pre-trained transformer-based language model, specifically
Sentence-BERT, to convert textual descriptions (e.g., titles, abstracts, names, organizational descriptions)
into continuous, dense vector representations capturing semantic meaning. These embeddings were
indexed using Facebook’s FAISS library, allowing eficient approximate nearest-neighbor retrieval at
query time.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Query Type Classification using RoBERTa-Large</title>
        <p>To determine the category of entities targeted by user-submitted natural-language queries, we finetuned
a supervised classification method based on RoBERTa-Large, a robust transformer-based language
model optimized for text classification tasks.
4.2.1. Automated Federated Dataset Construction
Training data for query-type classification was auto generated using GPT-4. We created a federated
training/evaluation dataset containing diverse natural-language query examples synthetically paired
with their corresponding entity types (conference, author, organization, or paper). This automated
process ensured extensive linguistic coverage without manual annotation eforts. Following is the
distribution of the dataset:</p>
        <p>Model
RoBERTa-large (fine-tuned)</p>
        <p>Training Samples
772</p>
        <p>Test Samples
100
4.2.2. Fine-Tuning RoBERTa-Large Classifier
The RoBERTa-Large model was fine-tuned on the generated dataset using a supervised training approach,
employing standard hyperparameters (learning rate: 2 × 10−5, batch size: 16, maximum token length:
128). RoBERTa-Large was chosen for its strong performance on a wide range of natural language
understanding tasks, particularly in scenarios requiring nuanced contextual representations, making it
well-suited for accurately interpreting diverse scholarly queries. This resulted in a high-performance
model capable of reliably classifying user queries into the defined entity types, facilitating targeted
downstream retrieval.</p>
      </sec>
      <sec id="sec-4-3">
        <title>4.3. SPARQL Query Generation via GPT-4o-mini</title>
        <p>Following query-type identification, the natural-language queries were transformed into executable
SPARQL queries using the GPT-4o-mini language model.
4.3.1. Few-Shot Prompt Engineering
To guide GPT-4o-mini in generating accurate SPARQL queries, we implemented a few-shot learning
approach tailored for each query type. Specifically, we provided carefully curated examples of
naturallanguage queries paired with correctly structured SPARQL queries corresponding to scholarly KG
schemas. These prompts ensured GPT-4o-mini could efectively generalize from limited examples,
generating syntactically valid and semantically accurate SPARQL queries.
4.3.2. Query Execution
The SPARQL queries generated by GPT-4o-mini were directly executed against the semantic knowledge
graph database. The results retrieved from these symbolic queries formed the primary candidate set for
answering user queries.</p>
      </sec>
      <sec id="sec-4-4">
        <title>4.4. Hybrid Retrieval and Reranking Approach</title>
        <p>To ensure semantic relevance and retrieval quality, we incorporated a hybrid retrieval and
reranking strategy that dynamically integrated symbolic SPARQL results with embedding-based semantic
similarity computations.
4.4.1. Embedding-based Result Reranking
When SPARQL queries returned a suficient number of results ( ≥ 5), the retrieved entities were
semantically reranked using cosine similarity scores calculated between their embeddings and the embedding
of the user’s natural-language query. This step enhanced semantic coherence and reduced ambiguity
inherent in purely symbolic query results.
4.4.2. Embedding-only Retrieval (Fallback Mechanism)
If SPARQL queries returned insuficient results ( &lt; 5) or no results, indicating either semantic mismatches
or query complexity, our system defaulted directly to embedding-based retrieval. Specifically, we queried
the FAISS-indexed embeddings of the identified entity type using the original natural-language query,
retrieving the top- nearest entities based on semantic similarity scores.
4.4.3. GraphRAG-based Answer Generation
Following the reranking and retrieval of the final results, the conclusive step of the GraphRAG was
triggered. The top- embedding results and the original natural-language query were provided back into
GPT-4o-mini. Leveraging Retrieval-Augmented Generation, GPT-4o-mini synthesized these retrieved
entities into a coherent, contextually relevant, natural-language summary that directly addressed the
user’s query.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Evaluation</title>
      <p>To validate the efectiveness and accuracy of our proposed hybrid query system, we designed a
comprehensive evaluation involving rigorous experimental setups, comparative analyses against manually
constructed ground-truth queries, and a detailed assessment of results across multiple dimensions.</p>
      <sec id="sec-5-1">
        <title>5.1. Experimental Setup</title>
        <p>5.1.1. Dataset and Gold-Standard Queries
We prepared a dataset comprising 92 manually crafted natural-language queries, each paired with
expertly constructed corresponding SPARQL queries, hereafter referred to as the gold-standard queries.
These queries were selected to reflect realistic academic information needs and were distributed across
four distinct categories:
• Conferences (22 queries)
• Authors (25 queries)
• Organizations (20 queries)
• Papers (25 queries)</p>
        <p>The diversity and representativeness of the queries ensured robust coverage of typical scenarios
encountered in scholarly querying tasks.
5.1.2. Evaluation Metrics
To quantitatively measure retrieval efectiveness, we employed the following metrics:</p>
        <p>Overlap Percentage: Defined as the intersection of retrieved entities by our automated pipeline and
the manually retrieved entities (gold-standard), normalized by the size of the gold-standard result set:
Overlap = | auto ∩  manual| × 100%</p>
        <p>| manual|</p>
        <p>Threshold Accuracy: We established a performance threshold at 70% overlap, considering queries
achieving this threshold as successfully answered. This threshold was selected to balance recall and
precision while remaining practically useful for typical scholarly use cases.
5.1.3. Comparative Baselines
For thoroughness, our automated approach was compared primarily against:
• Manually constructed SPARQL queries.</p>
        <p>• A pure embedding-based retrieval baseline using FAISS without symbolic query generation.</p>
        <p>This allowed us to highlight the advantages of our hybrid model in both structured and semantic
contexts.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Results and Analysis</title>
        <p>5.2.1. Overall Performance
Out of the 92 queries evaluated, our hybrid retrieval system achieved the following outcomes:
• Successful Retrieval (≥70% overlap): 89 queries (96%)
• Mean Overlap Percentage (all queries): 92.5%</p>
        <p>These results indicate that the vast majority of automated queries closely approximated expert-level
query quality.
5.2.2. Performance by Entity Type
We further analyzed performance separately for each query category to assess consistency across
diferent scholarly query contexts.</p>
        <p>These breakdowns illustrate particularly robust consistency in conference, author, and organization
queries, with only slightly lower performance on paper-related queries, possibly due to higher semantic
complexity inherent in publication metadata.
5.2.3. Query Type Classification Accuracy
The RoBERTa-Large classifier demonstrated exceptional efectiveness, achieving an accuracy of 99%
in predicting query categories on the federated validation set. This high classification accuracy was
instrumental in routing each query to the correct generation and retrieval pipeline.</p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. Comparative Analysis: Symbolic vs. Embedding-only Retrieval</title>
        <p>To better understand the benefit of our hybrid strategy, we compared it with a baseline using FAISS
embeddings only (no symbolic query generation):</p>
        <p>Our hybrid approach significantly outperformed the pure embedding baseline, underscoring the value
of combining symbolic SPARQL generation with embedding-based reranking and fallback mechanisms.</p>
      </sec>
      <sec id="sec-5-4">
        <title>5.4. Discussion and Interpretation</title>
        <p>The results clearly demonstrate that our hybrid retrieval method efectively bridges the gap between
SPARQL expertise and user-friendly natural-language interfaces. Key findings include:
• Robust query-type classification: The 99% classification accuracy ensured precise identification
of query intent, enabling targeted downstream processing.
• Balanced hybrid strategy: The integration of symbolic querying with embedding-based
reranking delivered consistent results, addressing weaknesses of either method alone.
• Semantic flexibility: Embedding-based reranking improved semantic coherence and
compensated for variability in query phrasing.</p>
      </sec>
      <sec id="sec-5-5">
        <title>5.5. Limitations and Error Analysis</title>
        <sec id="sec-5-5-1">
          <title>Despite the strong overall performance, several limitations were observed:</title>
          <p>• The number of test dataset was short, it comprised of 92 manually curated examples.
• The federated dataset created for the training of RobertaLarge might contain some bias or lack of
proper generalization when tested over a huge number of queries.</p>
          <p>Future work will include having a larger evaluation dataset curated by experts, improving entity
embeddings, expanding few-shot prompt coverage for LLMs and explore other finetuned llms to
generate SPARQL query, and exploring advanced reranking strategies to further enhance the accuracy
and robustness of results across all entity types.</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusion</title>
      <p>In this study, we presented a robust hybrid framework that enables natural-language querying of
scholarly knowledge graphs by combining the precision of symbolic SPARQL generation with the
lfexibility of embedding-based retrieval. Our system leverages a fine-tuned RoBERTa-Large classifier
for query type prediction, GPT-4o-mini for SPARQL generation guided by few-shot prompting, and
FAISS-indexed semantic embeddings for result reranking and fallback retrieval. This architecture
bridges the gap between expert-level structured querying and accessible, intuitive user interfaces.</p>
      <p>Through extensive evaluation against a curated set of 92 gold-standard SPARQL queries, our approach
achieved a 96% success rate in matching expert results, with a mean overlap of 92.5%, demonstrating
its efectiveness in replicating expert performance. The integration of GraphRAG further ensured
meaningful answers even in cases where symbolic retrieval alone was insuficient.</p>
      <p>Our findings highlight the practical potential of combining LLM-driven query generation with
embedding-based semantic reasoning to improve the accessibility and usability of scholarly knowledge
graphs for non-expert users. Future work will explore multilingual query support, dynamic schema
adaptation, user feedback integration, and expansion to other domains beyond scholarly metadata.</p>
    </sec>
    <sec id="sec-7">
      <title>7. Acknowledgments</title>
      <p>This project has received funding from the European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No 955569. The opinions expressed
in this document reflect only the author’s view and in no way reflect the European Commission’s
opinions. The European Commission is not responsible for any use that may be made of the information
it contains.</p>
    </sec>
    <sec id="sec-8">
      <title>8. Supplementary Materials</title>
      <sec id="sec-8-1">
        <title>The resources for this study are placed at the following repository: https://anonymous.4open.science/r/scholarlyGraphRAG-BD22/README.md</title>
      </sec>
    </sec>
    <sec id="sec-9">
      <title>Declaration on Generative AI</title>
      <p>During the preparation of this work, the author(s) used GPT-4 and Grammarly to assist with
paraphrasing, grammar correction, and spelling checks.
[14] A. Meloni, D. R. Recupero, F. Osborne, A. Salatino, E. Motta, S. Vahadati, J. Lehmann, Assessing
large language models for sparql query generation in scientific question answering, in: CEUR
Workshop Proceedings, volume 3953, 2025.
[15] H. Peng, H. Li, Y. Song, V. Zheng, J. Li, Diferentially private federated knowledge graphs
embedding, in: Proceedings of the 30th ACM international conference on information &amp; knowledge
management, 2021, pp. 1416–1425.
[16] L. Yue, Y. Zhang, Q. Yao, Y. Li, X. Wu, Z. Zhang, Z. Lin, Y. Zheng, Relation-aware ensemble learning
for knowledge graph embedding, arXiv preprint arXiv:2310.08917 (2023).
[17] H.-J. Cha, S.-W. Choi, E.-B. Lee, D.-M. Lee, Knowledge retrieval model based on a graph database
for semantic search in equipment purchase order specifications for steel plants, Sustainability 15
(2023) 6319.
[18] I. A. Ebeid, Medgraph: A semantic biomedical information retrieval framework using knowledge
graph embedding for pubmed, Frontiers in big Data 5 (2022) 965619.
[19] J. Qi, C. Su, Z. Guo, L. Wu, Z. Shen, L. Fu, X. Wang, C. Zhou, Enhancing sparql query generation
for knowledge base question answering systems by learning to correct triplets, Applied Sciences
14 (2024) 1521.
[20] S. Ferré, Sparklis: An expressive query builder for sparql endpoints with guidance in natural
language, Semantic Web 8 (2016) 405–418.
[21] N. Afreen, G. Balloccu, L. Boratto, G. Fenu, F. M. Malloci, M. Marras, A. G. Martis, Edge: A
conversational interface driven by large language models for educational knowledge graphs
exploration, in: Proceedings of the 33rd ACM International Conference on Information and
Knowledge Management, 2024, pp. 5159–5163.
[22] M. Arazzi, D. Ligari, S. Nicolazzo, A. Nocera, Augmented knowledge graph querying leveraging
llms, arXiv preprint arXiv:2502.01298 (2025).
[23] D. Bustamante, H. Takeda, Sparql generation with entity pre-trained gpt for kg question answering,
arXiv preprint arXiv:2402.00969 (2024).
[24] S. Xu, S. Liu, T. Culhane, E. Pertseva, M.-H. Wu, S. J. Semnani, M. S. Lam, Fine-tuned llms know
more, hallucinate less with few-shot sequence-to-sequence semantic parsing over wikidata, arXiv
preprint arXiv:2305.14202 (2023).
[25] A. Sharma, L. Lara, C. J. Pal, A. Zouaq, Reducing hallucinations in language model-based sparql
query generation using post-generation memory retrieval, arXiv preprint arXiv:2502.13369 (2025).
[26] P. A. K. Karou Diallo, A. Zouaq, Frase: Structured representations for generalizable sparql query
generation, arXiv e-prints (2025) arXiv–2503.
[27] J. Lee, H. Shin, Sparkle: Enhancing sparql generation with direct kg integration in decoding,</p>
      <p>Expert Systems with Applications (2025) 128263.
[28] J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with gpus, IEEE Transactions on</p>
      <p>Big Data 7 (2019) 535–547.
[29] J. Lu, K. Hall, J. Ma, J. Ni, Hyrr: Hybrid infused reranking for passage retrieval, arXiv preprint
arXiv:2212.10528 (2022).
[30] M. Acosta, C. Qin, T. Schwabe, Neuro-symbolic query optimization in knowledge graphs, arXiv
preprint arXiv:2411.14277 (2024).
[31] Y. Jia, W. Chen, A method of query graph reranking for knowledge base question answering,
arXiv preprint arXiv:2204.12808 (2022).
[32] M. Li, C. Yang, C. Xu, X. Jiang, Y. Qi, J. Guo, H.-f. Leung, I. King, Retrieval, reasoning, re-ranking:
A context-enriched framework for knowledge graph completion, arXiv preprint arXiv:2411.08165
(2024).
[33] M. Gao, Y. Xie, W. Chen, F. Zhang, F. Ding, T. Wang, J. Yao, J. Zheng, K.-F. Wong, Rerankgc: A
cooperative retrieve-and-rerank framework for multi-modal knowledge graph completion, Neural
Networks 188 (2025) 107467.
[34] S. Verma, R. Bhatia, S. Harit, S. Batish, Scholarly knowledge graphs through structuring scholarly
communication: a review, Complex &amp; intelligent systems 9 (2023) 1059–1095.
[35] A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. D. Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo,</p>
      <p>R. Navigli, S. Neumaier, et al., Knowledge graphs, ACM Computing Surveys (Csur) 54 (2021) 1–37.
[36] D. Fensel, U. Şimşek, K. Angele, E. Huaman, E. Kärle, O. Panasiuk, I. Toma, J. Umbrich, A. Wahler,
D. Fensel, et al., Introduction: what is a knowledge graph?, Knowledge graphs: Methodology,
tools and selected use cases (2020) 1–10.
[37] R. Usbeck, A.-C. Ngonga Ngomo, M. Röder, D. Gerber, S. A. Coelho, S. Auer, A. Both,
Agdistisgraph-based disambiguation of named entities using linked data, in: The Semantic Web–ISWC
2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014.</p>
      <p>Proceedings, Part I 13, Springer, 2014, pp. 457–471.
[38] H. Chen, Large knowledge model: Perspectives and challenges, arXiv preprint arXiv:2312.02706
(2023).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>V.</given-names>
            <surname>Lopez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Uren</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. R.</given-names>
            <surname>Sabou</surname>
          </string-name>
          , E. Motta,
          <article-title>Cross ontology query answering on the semantic web: an initial evaluation</article-title>
          ,
          <source>in: Proceedings of the fith international conference on Knowledge capture</source>
          ,
          <year>2009</year>
          , pp.
          <fpage>17</fpage>
          -
          <lpage>24</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>C.</given-names>
            <surname>Unger</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Bühmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Lehmann</surname>
          </string-name>
          , A.
          <string-name>
            <surname>-C. Ngonga Ngomo</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Gerber</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Cimiano</surname>
          </string-name>
          ,
          <article-title>Template-based question answering over rdf data</article-title>
          ,
          <source>in: Proceedings of the 21st international conference on World Wide Web</source>
          ,
          <year>2012</year>
          , pp.
          <fpage>639</fpage>
          -
          <lpage>648</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>J.</given-names>
            <surname>Lehmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Isele</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Jakob</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Jentzsch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kontokostas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. N.</given-names>
            <surname>Mendes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Hellmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Morsey</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Van Kleef</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Auer</surname>
          </string-name>
          , et al.,
          <article-title>Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia</article-title>
          ,
          <source>Semantic web 6</source>
          (
          <year>2015</year>
          )
          <fpage>167</fpage>
          -
          <lpage>195</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>D.</given-names>
            <surname>Vrandečić</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Krötzsch</surname>
          </string-name>
          ,
          <article-title>Wikidata: a free collaborative knowledgebase</article-title>
          ,
          <source>Communications of the ACM</source>
          <volume>57</volume>
          (
          <year>2014</year>
          )
          <fpage>78</fpage>
          -
          <lpage>85</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>S.</given-names>
            <surname>Vahdati</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Arndt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Auer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Lange</surname>
          </string-name>
          ,
          <article-title>Openresearch: collaborative management of scholarly communication metadata, in: Knowledge Engineering and Knowledge Management: 20th International Conference</article-title>
          ,
          <string-name>
            <surname>EKAW</surname>
          </string-name>
          <year>2016</year>
          , Bologna, Italy,
          <source>November 19-23</source>
          ,
          <year>2016</year>
          , Proceedings 20, Springer,
          <year>2016</year>
          , pp.
          <fpage>778</fpage>
          -
          <lpage>793</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A. G.</given-names>
            <surname>Nuzzolese</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. L.</given-names>
            <surname>Gentile</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Presutti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Gangemi</surname>
          </string-name>
          ,
          <article-title>Conference linked data: the scholarlydata project</article-title>
          ,
          <source>in: The Semantic Web-ISWC</source>
          <year>2016</year>
          : 15th International Semantic Web Conference, Kobe, Japan,
          <source>October 17-21</source>
          ,
          <year>2016</year>
          , Proceedings,
          <source>Part II 15</source>
          , Springer,
          <year>2016</year>
          , pp.
          <fpage>150</fpage>
          -
          <lpage>158</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>S.</given-names>
            <surname>Harris</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Seaborne</surname>
          </string-name>
          ,
          <string-name>
            <surname>E.</surname>
          </string-name>
          <article-title>Prud'hommeaux, Sparql 1.1 query language</article-title>
          .
          <source>w3c recommendation</source>
          (
          <year>2013</year>
          ), URL https://www. w3. org/TR/sparql11-query (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>A.</given-names>
            <surname>Zaveri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Rula</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Maurino</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Pietrobon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Lehmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Auer</surname>
          </string-name>
          ,
          <article-title>Quality assessment for linked data: A survey: A systematic literature review and conceptual framework</article-title>
          ,
          <source>Semantic web 7</source>
          (
          <year>2015</year>
          )
          <fpage>63</fpage>
          -
          <lpage>93</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9] W.-t. Yih,
          <string-name>
            <given-names>M.</given-names>
            <surname>Richardson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Meek</surname>
          </string-name>
          , M.-
          <string-name>
            <given-names>W.</given-names>
            <surname>Chang</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. Suh,</surname>
          </string-name>
          <article-title>The value of semantic parse labeling for knowledge base question answering</article-title>
          ,
          <source>in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume</source>
          <volume>2</volume>
          :
          <string-name>
            <surname>Short</surname>
            <given-names>Papers)</given-names>
          </string-name>
          ,
          <year>2016</year>
          , pp.
          <fpage>201</fpage>
          -
          <lpage>206</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>J. D'Abramo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Zugarini</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Torroni</surname>
          </string-name>
          ,
          <article-title>Investigating large language models for text-to-sparql generation</article-title>
          ,
          <source>in: Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing</source>
          ,
          <year>2025</year>
          , pp.
          <fpage>66</fpage>
          -
          <lpage>80</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>D.</given-names>
            <surname>Banerjee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. A.</given-names>
            <surname>Nair</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. N.</given-names>
            <surname>Kaur</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Usbeck</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Biemann</surname>
          </string-name>
          ,
          <article-title>Modern baselines for sparql semantic parsing</article-title>
          ,
          <source>in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>2260</fpage>
          -
          <lpage>2265</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>H.</given-names>
            <surname>Tran</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Phan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Anibal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. T.</given-names>
            <surname>Nguyen</surname>
          </string-name>
          , T.-S. Nguyen,
          <string-name>
            <surname>Spbert:</surname>
          </string-name>
          <article-title>An eficient pre-training bert on sparql queries for question answering over knowledge graphs</article-title>
          ,
          <source>in: Neural Information Processing: 28th International Conference, ICONIP</source>
          <year>2021</year>
          , Sanur, Bali, Indonesia, December 8-
          <issue>12</issue>
          ,
          <year>2021</year>
          , Proceedings,
          <source>Part I 28</source>
          , Springer,
          <year>2021</year>
          , pp.
          <fpage>512</fpage>
          -
          <lpage>523</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>S.</given-names>
            <surname>Auer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Stocker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Vogt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Fraumann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Garatzogianni</surname>
          </string-name>
          ,
          <article-title>Orkg: Facilitating the transfer of research results with the open research knowledge graph</article-title>
          ,
          <source>Research Ideas and Outcomes</source>
          <volume>7</volume>
          (
          <year>2021</year>
          )
          <article-title>e68513</article-title>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>