Question answering over scholarly knowledge graphs involves many challenges: complex graph patterns, long-tail distributed data, revision and evolution of the scholarly ontologies, and knowledge graphs incompleteness due to constant research dynamics. In this work, we present an LLM-based approach for SPARQL query generation over Open Research Knowledge Graph (ORKG) for the ISWC SciQA Challenge. Our approach proposes a couple of improvements to the recently published SPARQLGEN approach, that performs one-shot SPARQL query generation by augmenting Large Language Models (LLMs) with the relevant context within a single prompt. Similar to SPARQLGEN, we include heterogeneous data sources in the SPARQL generation prompt: a question itself, an RDF subgraph required to answer the question, and an example of a correct SPARQL query. In the current work, we focused on designing subgraph extraction algorithms, that are close to real-life scenarios of generative KGQA, and replaced the random choice of example question-query pair with similarity scoring.
CEUR ceur-ws.org
Scholarly knowledge graphs has become a recent trend and inspired the adaptation of KGQA
systems to new complex domains, that are constantly evolving, bringing new facts and concepts
to the knowledge graph. Currently, there is a number of scholarly knowledge graphs, that difer
in metadata and coverage, being built on top of various ontologies and data sources [
In the current paper, we propose further improvements to SPARQLGEN, a recent one-shot
approach for generating SPARQL queries with prompting LLMs [
Main contributions of the paper are the following: • improvement of SPARQLGEN, a one-shot method for SPARQL query generation with prompting LLMs, that can be quickly adapted to other datasets and knowledge graphs, • subgraph extraction algorithm, that can be used in LLM-augmented SPARQL query generation scenarios, • evaluation of the improved SPARQLGEN method on the ORKG benchmark.
Fine-tuning LLMs to generate SPARQL queries has already a proven track record with
topleaderboard results on QALD-9 with SGPT [
Rony et al. [
The recent approach, which we are extending upon, is SPARQLGEN [
The findings of SPARQLGEN approach show that the model struggles to deal with an unknown knowledge graph. Namespace errors, incomplete triples and ignoring KG structure errors occur more frequently for the unseen dataset and it seems beneficial to introduce the model to the KG in pre-training already (see supplementary material in in the repository: https: //github.com/danrd/sparqlgen).
This section presents an overview of the SciQA Challenge and the datasets used in this study.
The primary objective of this research endeavor was to participate in the Scholarly QuestionAnswering over Linked Data (Scholarly QALD) challenge. This challenge is centered around Knowledge Graph Question Answering utilizing the ORKG Scholarly Knowledge Graph. The challenge was structured into two distinct stages.
The first stage aimed to develop a model and conduct validation using a held-out subset with known true labels, thereby establishing an initial leaderboard. The second stage focused on assessing the model’s quality and generating final results for comparison among the various solutions developed. The central challenge task involved transforming a user’s natural language question into a SPARQL query and executing this query on the provided knowledge graph to furnish a response. The eficacy of the proposed solutions was evaluated through a set of 200 questions, employing the F1-score metric in both stages.
To train and evaluate the developed models, the SciQA dataset was designated as the target data source. This dataset is publicly available in the repository: https://zenodo.org/record/7744048 . The SciQA dataset encompasses a rich assortment of records. Each record comprises a question expressed in natural language, the corresponding SPARQL query, and a list of knowledge graph triples that represent the answer to that question.
Notably, the SciQA dataset is characterized by a significant diversity of questions, spanning various domains, response types, and sizes. The questions also exhibit structural and computational complexity and a degree of ambiguity. In addition to the SciQA dataset, the repository contains a link to the dump of the knowledge graph designed for question answering. This knowledge graph is primarily structured around scholarly data, including papers, contributions, and connections between them, accompanied by pertinent metadata.
The knowledge graph contains a total of 1,133,217 triples and 21,243 contributions, with an overall dump size of 152 megabytes. This knowledge graph is inherently heterogeneous, encompassing diverse data pertaining to papers from an array of research fields. Its structure is notably intricate, rendering the challenge particularly demanding. Specifically, the graph incorporates both discrete and continuous data entries, encompassing a wide range of primitive types such as numbers in various formats, dates, strings, and binary labels.
Our approach is implemented as a modular architecture with the following components: (1) subgraph retrieval (2) example selection context to populate prompt elements, (3) prompt building, (4) prompt execution, (5) removing hallucinations and validating the query, (6) query execution.
During steps (1) and (2), each data point in the SciQA test set was augmented with the context, represented as (i) the subgraph, required to execute the query (see Sec. 5), and (ii) a question-query pair, similar to the question (see below). Then the prompt builder (3) constructs a prompt from the augmented SciQA datapoint and a guiding example with the order of the prompt elements, defined in the experiment config, as shown in Fig. 1), and the serialized prompt is sent to the GPT-3.5 completions endpoint. The resulting query is validated (5) and executed (6). Validation includes removing hallucinated symbols (i.e. generated text prior the query, like System:, Query:, randomly inserted newlines, etc.) The pipeline architecture is shown in Fig. 1. Subgraph extraction is described in more detail in Sec. 5.
Following the SPARQLGEN approach, to combine heterogeneous data sources in the prompt, we used an abstract structure, called prompt element, that is instantiated during the experiment. Each prompt element has fields description and source, as well as methods for pre-processing the source data (see prompt elements Example, Instruction, Question and Knowledge Graph in Fig. 1). A prompt in SPARQLGEN is a serialized sequence of prompt elements. The implemented structure allows to configure experiments with minimal changes in the code structure and quickly design custom prompt templates.
Example Selection For one-shot prompting, the example was sampled from the train subset of the proposed dataset by computing Levenstein distance between input question and every question from the train subset. Subsequently the list of questions was sorted by Levenstein distance, and record with the lowest distance was selected for prompt generation. The approach based on Levenstein distance instead of semantic similarity has been selected due to the low diversity of the dump dataset, and a lot of instances in the test subset which had at least one training sample with the identical structure except a few property values.
Our approach is based on finding for each question semantically related objects in ORKG, namely
resources, properties, papers and contributions. For this purpose, ’all-mpnet-base-v2’ sentence
transformer [
Essentially, the process of subgraph retrieval comprises the following steps. (1) question processing: filtering out parts of speech, except nouns, adjectives and verbs; bigrams generation and encoding. (2) relevant objects extraction: defining most relevant to the question ORKG objects based on cosine similarity between its bigrams embeddings and embeddings of ORKG object labels leveraging initially constructed with HNSW indexes. (3) subgraphs retrieval: deriving 2-hop paths containing given similar objects. (4) subgraphs merging and converting into string with postprocessing for prompting.
In our experiments we implemented two approaches for involving derived triples in subgraph construction process as part of step (3). The first approach considers papers and contributions as a main source of structured information for LLMs. Therefore, it relies on retrieving a number of papers and contributions with related titles from ORKG, excluding triples with irrelevant predicates. However, in some cases titles are not specific enough and do not contain needed keywords. To address this issue, the second approach is to directly extract triples containing resources and properties with high similarity to the question, though in some cases it requires particular triple patterns. More detailed description of the implemented approaches is provided with Figure 2, whereas the whole process of subgraph retrieval in general is presented in Fig. 3:
During the experiments, we evaluated, how the subgraph extraction algorithm influences the SPARQL generation quality. In the baseline version, no subgraph was provided, only an example. The results are summarized in Table 1.
According to the experimental results, even without involving subgraph extraction algorithms the proposed architecture is capable to obtain relatively decent result with 0.922 F1 score. Moreover, the table shows that the second approach to subgraph extraction leads to minor F1 score decreasing. This probably means that structured information, extracted from ORKG, predominantly contributes additional noise to the prompts. Consequently, this approach needs further revision or adaptation to the given KG. On the other hand, the first approach fosters slight F1 metric increase demonstrating its usefulness in general that requires investigation in future works.
The idea behind hyperparameters selection for the experiments on the one hand was to comply with 4096 tokens limit for GPT-3.5 and at the same time to measure the contribution changing the balance between diferent parameters. Table 2 indicates that the results mostly robust to variation in hyperparameters that at least partly could be caused by relatively low impact of the subgraph extraction algorithms on the system performance.
Using the optimal configuration of the subgraph extraction algorithm, all generated queries (100%) contained one or more missing prefixes, necessitating their manual addition at the beginning of each query to obtain meaningful results from the database. Two generated queries (1%) were entirely incorrect, specifically AQ1806 and AQ1787. These queries consisted solely of lists of nodes from the graph database. This could be attributed to the lack of examples in the training set that exhibit semantic similarity to the desired questions, with a similarity score high enough to facilitate the reuse of provided examples for new query generation. Consequently, the model might have attempted to deduce the values that the target query should have returned or replicated elements of the provided subgraph.
However, upon a more detailed investigation, the evidence suggests that the training subset contains relevant examples with Levenshtein similarity scores of 0.944 and 0.8947, indicating very high similarity. The significantly high similarity and structural homogeneity were also confirmed through manual evaluation. Therefore, the model’s inability to utilize these examples and the provided subgraph to generate a correct SPARQL query should be attributed to its stochastic nature.
Furthermore, regarding other configurations of the subgraph retrieval algorithm, the following types of inconsistencies were observed:
1. Incorrect namespace prefixes (such as myontology, ex, foaf) appeared in 2 instances, possibly due to the model’s bias;
2. Instances, in which the model requested additional data using natural language for query generation, occurred once. Notably, in this case, the structure of the provided example closely resembled the format of the input question, and thus, the model should have been capable of handling this case correctly, given that it processed most of the dataset without errors. Such behavior can also be attributed to the inherent stochasticity of the model’s behavior, and an additional request should be suficient to resolve the observed issue in most cases; 3. Hallucinated node identifiers occurred in one instance, likely due to the complexity of the provided query. In this case, the question was particularly challenging to handle without extensive reference to database content, as the node identifiers could not be straightforwardly derived from text labels. The model essentially had to guess, as the required instances arguably were absent from the provided subgraph.
For more details regarding the classification system used for categorizing errors refer to the supplementary materal for SPARQLGEN paper 4.
As a result, the majority of the model’s errors were a consequence of its random nature, with only a negligible fraction of cases (0.25% of the total amount of handled questions) in which the model has completely failed to respond correctly. The residual error should be studied with greater attention. Evidently, in these cases, the model has failed to reproduce the query
structure, translate the necessary parameter values to corresponding query fields, or eficiently utilize the provided subgraph due to incompleteness of the train corpus, insuficient architecture complexity and other constraints.
The presented work shares results on one-shot SPARQL query generation with large language
models for the fraction of the ORKG, proposed at the SciQA challenge. The results, which
were obtained, require further assessment. First of all, the performance of all participating
systems, evaluated with F1-score, looks very high for such a complex domain as scholarly
knowledge graphs. Our approach was ranked last with 0.935 F1-score, and the winner has
the F1-score of 0.99, which is not that far for the one-shot approach without pre-training or
ifne-tuning. For other benchmarks (i.e. QALD series), the leaderboard is much lower 5. This
might be the result of the lack of structural diversity in the test set. Second, bringing the
subgraph to the prompt doesn’t result in that considerable performance increase, as it was
shown for SPARQLGEN [
Given the ongoing research activity on enhancing LLMs with KGs, future work is quite extensive. First of all, we would like to proceed with fine-tuning open source LLMs on multiple knowledge graphs and compare the performance vs. the connecting knowledge graphs to the LLM in a retrieval fashion, as well as evaluate the generalization of the fine-tuned LLM on more datasets.
Developing robust subgraph extraction algorithms, that produce relevant knowledge snippets to augment the LLMs across diferent tasks, remains another perspective research direction. The requirement to have only relevant triples in the subgraph is also quite challenging to achieve with a single algorithm. Thus, some other subgraph reduction procedures can be added: i.e. instructing the LLM to prune some of the irrelevant triples6, using KG embeddings-based classifier to estimate the relevance of the triple, or some actor-critic approach to iteratively refine the subgraph content.
6This was suggested by one of the reviewers.