The Scholarly Question Answering over Linked Data (Scholarly QALD) 1 at The International Semantic Web Conference (ISWC) 2023 challenge presents two sub-tasks to tackle question answering (QA) over knowledge graphs (KGs). We answer the KGQA over DBLP (DBLP-QuAD) task by proposing a neurosymbolic (NS) framework based on PSYCHIC 2, an extractive QA model capable of identifying the query and entities related to a KG question. Our system achieved a F1 score of 00.18% on question answering and came in third place for entity linking (EL) with a score of 71.00%.
Knowledge veracity has always been one of the key topics of the Web [
One way the Semantic Web community addresses this issue is by constructing KGs which
are structured repertoires of entities and relationships [
Aside from consolidating bodies of information, KGs provide a platform for information
retrieval [
Integrating LLMs and KGs has emerged as a solution to mitigate the shortcomings of large
language models [
In this paper, we show how a NS system can handle the tasks of QA and EL over a KG. The rest of the paper is organized as follows. In section 2, we discuss some of the related work. In section 3, we present the experimental setup. In section 4, we discuss the results. Finally, we present our conclusions in section 5.
This section reviews some of the proposed systems designed for QA and EL over KGs.
Several techniques have emerged to tackle the problem of QA over KGs. Zheng and Zhang make
use of structured query patterns which involve identifying query graph candidates in the KG
using EL and disambiguation and transforming them to SPARQL queries [
Nikas et al. combine graph-based techniques with neural-based methods to create a NS
QA system [
Diomedi and Hogan propose a diferent approach by using neural machine translation to map the questions in natural language to SPARQL templates directly. They show that this method, coupled with tree-based entity disambiguation techniques, turns the QA problem to a slot-filling task and outperforms vanilla deep learning (DL) models [ 11]. Aghaei et al. leverage a similar slot-filling pipeline on a domain-specific KG to demonstrate that it can reliably learn the pattern structures of the domain queries [12].
In their work, Mavromatis and Karypis and Li et al. leverage graph networks to compute graph embeddings and compare them with the input question embeddings to target the correct answer entities [13, 14]. Dutt et al. utilize graph convolution networks to score questions based on similarity and derive their corresponding graph paths to answer new questions [15].
Saxena et al. and Zuo et al. demonstrate how using KG embeddings can help solve multi-hop questions [16, 17]. Rony et al. propose a system whereby KG embeddings are stored in a vector database for faster retrieval and improved performance in answering complex questions [18].
Shi et al. present a survey on the techniques seen in EL over KGs. They regroup approaches into rule-based, machine learning (ML) and DL EL methods [19]. In their work, Dubey et al. use rule-based methods by leveraging global traveling salesman approximate solver algorithms to disambiguate entities in large KGs [20]. Steinmetz employ abstract meaning representation, a series of text dependency parsing rules, to identify and group sentences having the same meaning but diferent structures [ 21]. Using this technique, they perform data augmentation to enrich entity representation and achieve better performance on the EL task [21]. In a similar fashion, Radhakrishnan et al. use co-occurrences from large corpora to enrich existing KGs with entity information and create dense KGs as a basis for EL [22].
Thawani et al. employ ML techniques by combining TF-IDF with Wikidata entries to generate feature vectors and score entity candidates accordingly [23]. Li et al. make use of translating embeddings to encode entity-entity relationships and combine them with entity-relation embeddings to get better performance over KGs [24].
From a DL perspective, Huang et al. show that using BERT to calculate sentence embeddings over entities outperforms rule-based entity enrichment approaches [25]. Banerjee et al. improve on this approach by concatenating entity embeddings with context using FastText embeddings in a pointer network entity linker structure designed to represent entities in a dense vector space and identify the correct one for any input entity [26].
Finally, Ding et al. propose a NS approach based on EL using SpaCy embeddings in a casebased reasoning by computing the cosine score between input entity embeddings and KG entity embeddings [27]. The final entity is computed automatically according to the best matching embedding or manually labeled based on a threshold cosine score [27]. Diomedi and Hogan experiment with a pipeline that integrates rule-based entity matching over DBpedia and Wikidata KGs and neural network models to associate the resulting entities to a fixed set from a designed KG [28].
This section describes the framework for our experiments in terms of data, system and training process.
The dataset considered for this shared task is divided in two parts: the DBLP-QuAD 2 dataset which consists of 10000 question-SPARQL pairs and is answerable over the DBLP KG, and a dataset of 500 questions retaining the same format provided by the task organizers which will be referred to as seed data. We provide details for both datasets in the following subsections.
DBLP-QuAD is a scholarly KGQA dataset with 10,000 question-SPARQL query pairs targeting the DBLP KG. DBLP-QuAD was created using the OVERNIGHT approach where logical forms are first generated from a KG. Canonical questions are then generated from these logical forms. The dataset is split into 7,000 training, 1,000 validation and 2,000 test questions. Each questionSPARQL pair consists of the following data fields: the id of the question ( id), a string containing the question (question), a paraphrased version of the question (paraphrased_question), a SPARQL query that answers the question (query), the type of the query (query_type), the template of the query (template_id), a list of entities in the question (entities), a list of relations in the question (relations), a boolean indicating whether the question contains a temporal expression (temporal) and a boolean indicating whether the question is held out from the training set (held_out). Sample data is shown in Figure 1. 3.1.2. Seed data Participants are provided with seed data to evaluate the performance of their systems on the QA and EL sub-tasks. This data is curated by the task organizers and consists of 500 random question-SPARQL pairs that comply with the same format as the DBLP-QuAD data. For the final evaluation, the organizers provided an additional set of 500 random questions containing only the question and its paraphrase. This dataset was used exclusively to score the performance of the participant systems in both sub-tasks.
This section introduces our proposed system. It presents the system architecture and describes the training process in our experiments.
Since the challenge presents itself as a QA task, we selected the DistilBERT-base uncased 3 model to tackle it in an extractive QA setting. Extractive QA confines a model to selecting the appropriate answer chunk from a context given as input with the question. It also leverages some control over the model answers as opposed to generative QA which makes models generate the answer. 2https://huggingface.co/datasets/awalesushil/DBLP-QuAD 3https://huggingface.co/distilbert-base-uncased
Training a QA model involves giving the model an input composed of the question to be answered and a context that should contain the answer. In the scope of this shared task, we targeted two types of answers: the SPARQL query which should directly answer the given question and the list of entities for EL. We constructed the context to include both pieces of information and added information to guide the model regarding the nature of the question asked and the expected SPARQL answer: the query type and the template id provided in the DBLP-QuAD data. We also introduced symbolic information through a symbolic rule engine that inserts the special tokens [CLS] at the start of the context string and [SEP] between the diferent pieces of information in the context. These markers were added to ground the output of the model and enable it to discriminate between the diferent pieces of context information. The aim is to help the model learn the types of query structures and entities it will be asked to retrieve.
On the output side, we fine-tuned our PSYCHIC ( Pre-trained SYmbolic CHecker In Context) model to return both the SPARQL query and the list of entities. We constructed the output as a string containing both pieces of information separated by the [SEP] token. We also introduced another symbolic rule engine which is a programmatic function designed to split the output string based on the [SEP] symbol to return the query and entities chunks separately. The PSYCHIC model architecture is presented in Figure 2.
To answer both parts of the shared task, we need to return a result from a SPARQL query for the QA challenge and an entity list for the EL challenge. In that respect, the PSYCHIC model alone is not enough. The QA challenge defines an answer as the result of a SPARQL query, whereas PSYCHIC returns the queries themselves. This is a deliberate design choice since teaching a model to recognize patterned query structures is still an easier problem than teaching it to learn dynamic answers ranging from boolean values to lists of elements. Being able to correctly return the right query is essentially the hard part of the challenge, and the additional step to get the final answer consists in running the query returned by PSYCHIC using the SPARQL endpoint provided by the task organizers.
As such, we built the NS framework shown in Figure 3 to leverage the power of LLMs and symbolic reasoning. It takes as input the question-SPARQL pairs, processes them and prepares the question-context dataset needed for the PSYCHIC model. The model predicts the output in the form of a string containing the query, the separator [SEP] and the entity list. Two additional modules, the query and entity sanitizers, extract the relevant pieces of information from the output, namely the query and the entity list, by splitting the output string and validating the query and entity elements by matching them to their respective patterns. Through this sanitization process, these modules can perform error-correction to account for malformed strings returned by the model, e.g., transforming ’select distinct? answer where {? answer < https : / / dblp. org / rdf / schema # authoredby > < https : / / dblp. org / pid / 00 / 2941 > }’ to ’select distinct ?answer where { ?answer <https://dblp.org/rdf/schema#authoredBy> <https://dblp.org/pid/00/2941> }’ for the query and [’< https : / / dblp. org / pid / 00 / 2941 >’] to [’<https://dblp.org/pid/00/2941>’] for the entity list.
The resulting output from the query sanitizer module is a correctly-formed SPARQL query that is run using the SPARQL endpoint to return the expected final answer. The output of the entity sanitizer is the correctly-formed entity list.
We divided our experiment into two phases: training and inference or evaluation phase.
For the training phase, we treated the question and paraphrase as distinct questions to efectively double our overall training, validation and test sets. We trained the PSYCHIC model on the training and validation splits and evaluated on the test split. We set the following training hyperparameters for the model: learning rate = 2e− 05, training batch size = 16, evaluation batch size = 16, seed = 42, optimizer = Adam with betas = (0.9, 0.999) and epsilon = 1e− 08, and number of epochs = 3.
For the inference step, we distinguished two phases: the dev phase and the final phase. The dev phase corresponds to the phase when the first 500 random question-SPARQL pairs are made available by the organizers. The final phase represents the true system evaluation phase whereby the dataset used is the set of 500 random questions (and their paraphrases). The phases difer by the nature of the input given to the PSYCHIC model. In the dev phase, the context was constructed the same way as for DBLP-QuAD, i.e., following the [CLS] + QUERY_TYPE + [SEP] + TEMPLATE_ID + [SEP] + QUERY + [SEP] + ENTITIES pattern. In the final phase, since the only available information is the question and its paraphrase, we used an entity linker provided by the task organizers that leverages a t5-base language model with translating embeddings to return a list of predicted entities for each provided question. These results were concatenated using the symbolic rule engine to form a diferent context pattern, i.e., [CLS] + EL_RESULT1 + [SEP] + EL_RESULT2 + [SEP] + ... + [SEP] + EL_RESULTN for N returned entity results.
We used the F1-QA and F1-EL metrics, representing the F1 scores on each of the QA and EL sub-tasks respectively, to evaluate our system. All experiments were performed on a Dell G15 Special Edition 5521 hardware with 14 CPU Cores, 32 GB RAM and NVIDIA GeForce RTX 3070 Ti GPU.
In this shared task, we propose a NS framework to tackle QA and EL sub-tasks over a KG. We explore the efects of including symbolic learning in the context of LLMs and evaluate the overall performance on the sub-tasks for a changing context. In the future, we plan to extend these symbolic mechanisms to generative models such as retrieval augmented generation (RAG) pipelines. The Semantic Web–ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings 20, Springer, 2021, pp. 235–251. [10] S. Cm, J. Prakash, P. K. Singh, Question answering over knowledge graphs using bert based relation mapping, Expert Systems (2022) e13456. [11] D. Diomedi, A. Hogan, Question answering over knowledge graphs with neural machine translation and entity linking, 2021. doi: 10.48550, arXiv preprint arXiv.2107.02865 (2021). [12] S. Aghaei, E. Raad, A. Fensel, Question answering over knowledge graphs: A case study in tourism, IEEE Access 10 (2022) 69788–69801. [13] C. Mavromatis, G. Karypis, Rearev: Adaptive reasoning for question answering over knowledge graphs, arXiv preprint arXiv:2210.13650 (2022). [14] S. Li, K. W. Wong, C. C. Fung, D. Zhu, Improving question answering over knowledge graphs using graph summarization, in: Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part IV 28, Springer, 2021, pp. 489–500. [15] R. Dutt, K. Bhattacharjee, R. Gangadharaiah, D. Roth, C. Rose, Perkgqa: Question answering over personalized knowledge graphs, in: Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 253–268. [16] A. Saxena, A. Tripathi, P. Talukdar, Improving multi-hop question answering over knowledge graphs using knowledge base embeddings, in: Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 4498–4507. [17] Z. Zuo, Z. Zhu, W. Wu, W. Wang, J. Qi, L. Zhong, Improving question answering over knowledge graphs with a chunked learning network, Electronics 12 (2023) 3363. [18] M. R. A. H. Rony, D. Chaudhuri, R. Usbeck, J. Lehmann, Tree-kgqa: an unsupervised approach for question answering over knowledge graphs, IEEE Access 10 (2022) 50467– 50478. [19] J. Shi, Z. Yuan, W. Guo, C. Ma, J. Chen, M. Zhang, Knowledge-graph-enabled biomedical entity linking: a survey, World Wide Web (2023) 1–30. [20] M. Dubey, D. Banerjee, D. Chaudhuri, J. Lehmann, Earl: joint entity and relation linking for question answering over knowledge graphs, in: The Semantic Web–ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8–12, 2018, Proceedings, Part I 17, Springer, 2018, pp. 108–126. [21] N. Steinmetz, Entity linking for kgqa using amr graphs, in: European Semantic Web
Conference, Springer, 2023, pp. 122–138. [22] P. Radhakrishnan, P. Talukdar, V. Varma, Elden: Improved entity linking using densified knowledge graphs, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 1844–1853. [23] A. Thawani, M. Hu, E. Hu, H. Zafar, N. Divvala, A. Singh, E. Qasemi, P. Szekely, J. Pujara, Entity linking to knowledge graphs to infer column types and properties, 2019, semantic web challenge on tabular data to knowledge graph matching, iswc, 2019. [24] Q. Li, F. Li, S. Li, X. Li, K. Liu, Q. Liu, P. Dong, Improving entity linking by introducing knowledge graph structure information, Applied Sciences 12 (2022) 2702. [25] B. Huang, H. Wang, T. Wang, Y. Liu, Y. Liu, Entity linking for short text using structured knowledge graph via multi-grained text matching (2020). [26] D. Banerjee, D. Chaudhuri, M. Dubey, J. Lehmann, Pnel: Pointer network based endto-end entity linking over knowledge graphs, in: The Semantic Web–ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020, Proceedings, Part I 19, Springer, 2020, pp. 21–38. [27] W. Ding, V. K. Chaudhri, N. Chittar, K. Konakanchi, Jel: applying end-to-end neural entity linking in jpmorgan chase, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 15301–15308. [28] D. Diomedi, A. Hogan, Entity linking and filling for question answering over knowledge graphs, in: Natural Language Interfaces for the Web of Data (NLIWOD) Workshop, 2022.