In this paper we propose the task of list verbalisation within a Knowledge Graph Question Answering system. Inspired by the Gricean Maxims of Quantity, Relation, and Manner we show a proof of concept ranking answer candidates through graph-based and language model-based measurements for on the one hand popularity and on the other hand a more pragmatically informed context. Our finding show that in our current set-up graph-based measures work best, while language model-based systems need further refinement and may benefit from approaches such as fine-tuning or prompting. We evaluate our approach with a user study and give insights into promising future directions of the task.
Question Answering (QA) systems are becoming more prevalent in both research and
realworld applications such as virtual assistants like Siri or Alexa. With vast amounts of structured
knowledge available in ontologies, they have been a key element in furthering QA development
and adaptation. Up until recently their answers were however limited to formal query responses,
which limit usefulness for conversational systems as well as non-expert users. To increase
naturalness and understanding [
Responses consisting of a whole list present the user with information overload, in particular
when questions result in a large number of answers. They will most likely not succeed in
ifnding an answer that is informative to them. From a pragmatic perspective, we can analyse
the likely success of a question-answer exchange through the lens of the co-operative principle
consisting of four maxims proposed by Grice [
Our approach combines summarisation and ranking, to find the most informative answer for the user. It is a first step to, on the one hand, the verbalisation of more complex formal query responses, and on the other hand the inclusion of context in the summarisation and verbalisation of structured data.
Our contributions in this paper are as follows:
1. Task We create the new task of list verbalisation, which was previously neglected
(VQuAnDa [
As our task and evaluation approach are influenced by a range of research fields, we will describe previous approaches from a range of fields including KGQA systems, data-to-text generation, summarisation, ranking, and their evaluations. To the best of our knowledge, this paper is the ifrst exploration of list verbalisation in the context of Gricean maxims.
Most QA systems can be divided into two groups. The first group works with unstructured data
such as SQuAD [
In a KGQA task the question is given in natural language, gets translated into a query, queries a database and retrieves an answer. The most common data in KGQA sets are the editions of QALD [11] and LC-QuAD [12]. The former’s focus has over the years shifted towards multi-linguality [13]. The latter provides a data set of 5000 complex question-query pairs over DBpedia.
Our focus lies on the verbalisation of the formal query response, turning the problem into a data-to-text task. Most approaches for this task are based on WebNLG [14] and more recently AGENDA [15] and use either transformers [15, 16] or Graph Neural Nets (GNN) [17]. Most of the previously described tasks are however too domain specific to be applied to an open-domain knowledge graph like DBpedia or Wikidata.
Data sets that have specifically explored verbalisation in a KGQA context are VQuAnDa [
List verbalisation can be approached from the perspective of ranking with respect to relevance.
Approaches considering the ranking of multiple query response options have been extensively
studied in the field Information Retrieval, ranking from early symbolic approaches [
Context in Information Retrieval and Summarisation systems is mostly based on
personalisation of the results for the user based on their previous interaction with the system [
Our evaluation is guided by the Gricean maxims [
We propose a the new task of list verbalisation with respect to the Gricean maxims of Quantity, Relation, and Manner. Given a long answer list, the task is to select a small number of items that serve to illustrate the most relevant information conveyed by the list.
What is relevant to the user can be highly situation-dependent and depend on individual factors specific to the user. In this first approach, we define a number of specific contextual cues that define what is relevant for the user. In the simplest scenario, we assume that the user is simply interested in the most popular examples from the entire list.
Another reason to not only work with the highest scores is that they enforce popularity bias. DBpedia is based on Wikipedia which is predominantly white, western and male [33, 34, 35, 36]. Cues are an option to diversify or personalise results and increase the exposure of non-popular items.
The contextual cues should reflect relatively open secondary factors. We currently use the following: location (place of birth, place of publication, nationality), time (recency, time of publication, date of birth), genre, and gender.
We illustrate the task using the following question / verbalisation pair: Question: ”Whose work is in the Musée d’Orsay?”
VQuAnDa answer template: ”The artists of the artworks located in the Musée d’Orsay are [answer].”
The place holder [answer] is filled with 30 names (see step 1 Figure 1), which is still on the lower end, as the number of items in such lists can exceed 1000. Giving an answer consisting of 30 names would be a violation of the Gricean Maxim of Quantity since it presents more information than can be the assumed was intent of the user’s question. This can lead to an information overload and the relevant information being buried. An answer consisting of three very popular artists (Vincent van Gogh, Paul Gauguin, Henri Matisse), however, would not violate the maxim and can be considered relevant. If we want to consider a particular context (e.g. a previous conversational context of female artists), another subset of artists can provide a relevant answer (e.g. examples of female artists).
We explore two approaches to list verbalisation: A language-model based approach and a
knowledge-graph based approach. In both approaches, we aim to rank the answer list with
respect to popularity or a particular contextual cue to retrieve the most relevant answers. After
a brief outline of the data we work with (Section 4.1), we introduce the two ranking approaches
we use to generate answers (Section 4.2). Finally, we describe the set-up of our human evaluation
(Section 4.3).
4.1. Data
As described in section 2, available verbalisation data sets and systems do not currently deal
with long list answers in an adequate manner. VQuAnDa [
We construct list verbalisations containing the top- list items retrieved by our raking approaches. The final verbalisation is presented through a template. Consider the example below showing the answer to the questions about artists in the Musée d’Orsay (introduced in Section 3). New answer template: ”The artists of the artworks located in the Musée d’Orsay are for example [top-] and [n - top-] others.” Our template is only going to verbalise the top- answers and enumerates the rest of the list, to still convey the information of more answers being present.
We use two diferent techniques to identify the top- answer candidates, the probability given to an answer candidate by a language model (GPT-2 [37], LM) and the density of a constructed graph for an answer candidate. Both ranking approaches only consider the items already provided in the list.
We can expect that language model prediction can, at least to some degree, reflect the relevance of an answer-candidate with respect to a particular context. To score the answer options, we ifrst create a template to fill with singular answers instead of plural. This is done by converting the subject and predicate of the sentence into singular. For the museum example this means: Singular answer template: ”An artist of the artworks located in the Musée d’Orsay is [answer].” When ranking answers with respect to a contextual cue, we currently add the cue to the template sentence, after which the answer candidates are ranked: Singular answer template with gender cue: ”A female artist of the artworks located in the Musée d’Orsay is [answer].” We fill the answer slot for each answer candidate and use GPT-2 1 to calculate the answer’s log-likelihood and perplexity. Upon initial qualitative exploration, we noticed that log-likelihood provided better answers than expected, despite its tendency to penalise longer answers (e.g. names consisting of multiple tokens). As expected, perplexity also resulted in high-quality answers. When investigating the correlation between answer length and LM scoring more closely we found that while maximising for log-likelihood significantly penalises answers for being long, minimising for perplexity overly rewards them (see Figure 2a). We include both metrics in our experiments to establish whether humans may have a preference for shorter or longer list items in the answers. (a) Influence of answer length on log-likelihood (b) Influence of answer length on perplexity
To retrieve relevant answers from a graph, we construct the corresponding graph of each answer candidate containing all outgoing nodes. To identify the most relevant graph we use density (D): || =
| |(| | − 1) where is the number of edges and is the number of nodes in the graph. Density reflects how well connected a graph is and can thus be seen as an approximation of popularity. More recently it has also been shown to correlate well with human judgements in the evaluation of open-domain dialogue systems [38]. The higher the density of the graph, the higher we rank the corresponding answer option. For an example result see step 2 in Figure 1. To calculate density for lists with specific contextual cures, we add an additional restriction containing the context cue.
While we considered a more commonly used metric such as PageRank, the implementation would have led to the loss of examples since DBpedia does not currently provide a PageRank measurement in its SPARQL endpoint. Reformatting to for example the Wikidata format would have lead to either loss of examples and answer options due to an imperfect mapping or labour intensive human input.
The goal of our evaluation is to establish whether potential users prefer the verbalisations created by our ranking approaches over the original, long lists. In addition, we check whether the ranked examples in our verbalisations are perceived as more helpful than random examples. We thus create verbalised answer options with rankings according to randomness (unranked), log-likelihood, perplexity and density.
In order to verify our ranking we conducted a survey with 10 participants. For each of the selected examples they were presented with five verbalised answer rankings: Random order, ranked by log-likelihood, by perplexity, and by density. As a fith alternative we included an option to indicate if they thought the full list would have been more suitable. They then ranked the options from most (1rst choice) to least (5th choice) relevant. Context cues were included as a highlighted word below the question and participants were instructed to include them in their ranking. Participants were told to not rank questions if they were too unfamiliar with the topic in question2.
In our experiments we are taking the first steps in providing a proof of concept for our extractive summarisation and context cues. For each approach, we fill our answer template (see Section 4.2) with the top- =3 ranking results. The results of the human ratings are summarised in Table 1 (general popularity ranking) and Table 2 (ranking with respect to context cues). The Tables show how many times an answer created through the diferent ranking methods was placed on a particular rank by the human participants.
The participants clearly favoured density over the other measures for both popularity and context cue options. Log-likelihood outperforms perplexity in case of popularity, but both are ranked lower than random when including the context cue. A promising finding is that even for shorter lists, the adjusted verbalisation was preferred over the full list. The low performance of the LM-based measurements could be improved by fine-tuning on the task or using prompting, which we are considering as next steps. 2One participant accidentally left out a question in the popularity ranking option leading to one score less than in the context cue option.
We identified the task of list verbalisation as a subtask of KGQA Answer Verbalisation. We show a first proof of concept ranking answers incorporating both graph-based and language model-based measurements to identify the most informative answer candidates. We base our approach on the Gricean Maxims and show that popularity alone might not sufice to create an informative answer. We are currently including only the Maxims of Quantity, Relation, and Manner, but hope to expand to the Maxim of Quality in the future. We will do this by moving from our current extractive summaries to more abstractive renditions. Instead of slot filling we will work on full pipeline or end-to-end verbalisation systems.
This research was funded by the Vrije Universiteit Amsterdam and the Netherlands Organisation for Scientific Research (NWO) through the Hybrid Intelligence Centre via the Zwaartekracht grant (024.004.022), and the Spinoza grant (SPI 63-260) awarded to Piek Vossen. We would also like to thank the reviewers for their excellent feedback that enhanced this paper. All remaining errors are our own. tics, Melbourne, Australia, 2018, pp. 784–789. URL: https://aclanthology.org/P18-2124. doi:10.18653/v1/P18- 2124. [8] S. Reddy, D. Chen, C. D. Manning, CoQA: A Conversational Question Answering Challenge, Transactions of the Association for Computational Linguistics 7 (2019) 249–266. URL: https://aclanthology.org/Q19-1016. doi:10.1162/tacl_a_00266, place: Cambridge, MA Publisher: MIT Press. [9] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, Z. Ives, DBpedia: A Nucleus for a Web of Open Data, in: K. Aberer, K.-S. Choi, N. Noy, D. Allemang, K.-I. Lee, L. Nixon, J. Golbeck, P. Mika, D. Maynard, R. Mizoguchi, G. Schreiber, P. Cudré-Mauroux (Eds.), The Semantic Web, Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, 2007, pp. 722–735. doi:10.1007/978- 3- 540- 76298- 0_52. [10] D. Vrandečić, M. Krötzsch, Wikidata: a free collaborative knowledgebase, Communications of the ACM 57 (2014) 78–85. Publisher: ACM New York, NY, USA. [11] V. Lopez, C. Unger, P. Cimiano, E. Motta, Evaluating question answering over linked data, Web Semantics Science Services And Agents On The World Wide Web 21 (2013) 3–13. doi:10.1016/j.websem.2013.05.006, publisher: Elsevier. [12] P. Trivedi, G. Maheshwari, M. Dubey, J. Lehmann, LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs, in: C. d’Amato, M. Fernandez, V. Tamma, F. Lecue, P. Cudré-Mauroux, J. Sequeda, C. Lange, J. Heflin (Eds.), The Semantic Web – ISWC 2017, volume 10588, Springer International Publishing, Cham, 2017, pp. 210–218. URL: http: //link.springer.com/10.1007/978-3-319-68204-4_22. doi:10.1007/978- 3- 319- 68204- 4_ 22, series Title: Lecture Notes in Computer Science. [13] A. Perevalov, D. Diefenbach, R. Usbeck, A. Both, QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers, in: 2022 IEEE 16th International Conference on Semantic Computing (ICSC), 2022, pp. 229–234. doi:10.1109/ICSC52841.2022.00045, iSSN: 2325-6516. [14] T. Castro Ferreira, D. Moussallem, E. Krahmer, S. Wubben, Enriching the WebNLG corpus, in: Proceedings of the 11th International Conference on Natural Language Generation, Association for Computational Linguistics, Tilburg University, The Netherlands, 2018, pp. 171–176. URL: https://aclanthology.org/W18-6521. doi:10.18653/v1/W18- 6521. [15] R. Koncel-Kedziorski, D. Bekal, Y. Luan, M. Lapata, H. Hajishirzi, Text Generation from Knowledge Graphs with Graph Transformers, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 2284–2293. URL: https://aclanthology.org/ N19-1238. doi:10.18653/v1/N19- 1238. [16] M. Schmitt, L. F. R. Ribeiro, P. Dufter, I. Gurevych, H. Schütze, Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs, in: Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs15), Association for Computational Linguistics, Mexico City, Mexico, 2021, pp. 10–21. URL: https://aclanthology.org/2021.textgraphs-1.2. doi:10.18653/v1/2021.textgraphs- 1.2. [17] D. Marcheggiani, L. Perez-Beltrachini, Deep Graph Convolutional Encoders for Structured Data to Text Generation, in: Proceedings of the 11th International Conference on Natural Language Generation, Association for Computational Linguistics, Tilburg University, The
Graph, 2018. doi:10.1109/ICALIP.2018.8455241, pages: 83. [29] P. N. Bennett, F. Radlinski, R. W. White, E. Yilmaz, Inferring and using location metadata to personalize web search, in: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, SIGIR ’11, Association for Computing Machinery, New York, NY, USA, 2011, pp. 135–144. URL: https://doi.org/10.1145/2009916. 2009938. doi:10.1145/2009916.2009938. [30] S. G. Sripada, E. Reiter, J. Hunter, J. Yu, Generating English summaries of time series data using the Gricean maxims, in: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’03, Association for Computing Machinery, New York, NY, USA, 2003, pp. 187–196. URL: https://doi.org/10.1145/956750. 956774. doi:10.1145/956750.956774. [31] V. Setlur, M. Tory, How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behavior, in: CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 2022, pp. 1–17. URL: https: //dl.acm.org/doi/10.1145/3491102.3501972. doi:10.1145/3491102.3501972. [32] E. Lloret, L. Plaza, A. Aker, The challenging task of summary evaluation: an overview, Language Resources and Evaluation 52 (2018) 101–148. URL: https://doi.org/10.1007/ s10579-017-9399-2. doi:10.1007/s10579-017-9399-2. [33] P. Konieczny, M. Klein, Gender gap through time and space: A journey through Wikipedia biographies via the Wikidata Human Gender Indicator, New Media & Society 20 (2018) 4608–4633. URL: https://doi.org/10.1177/1461444818779080. doi:10.1177/ 1461444818779080, publisher: SAGE Publications. [34] M. Hinnosaar, Gender inequality in new media: Evidence from Wikipedia, Journal of Economic Behavior & Organization 163 (2019) 262–276. URL: https://www.sciencedirect. com/science/article/pii/S0167268119301234. doi:10.1016/j.jebo.2019.04.020. [35] J. M. Ezell, Empathy plasticity: decolonizing and reorganizing Wikipedia and other online spaces to address racial equity, Ethnic and Racial Studies 44 (2021) 1324–1336. URL: https://www.tandfonline.com/doi/full/10.1080/01419870.2020.1851383. doi:10.1080/ 01419870.2020.1851383. [36] L. M. Bridges, R. Pun, R. A. Arteaga (Eds.), Wikipedia and Academic Libraries, Michigan Publishing, 2021. URL: https://hdl.handle.net/2027/fulcrum.cv43p013f. doi:10.3998/mpub. 11778416. [37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language Models are
Unsupervised Multitask Learners (2019) 24. [38] S. Báez Santamaría, P. Vossen, T. Baier, Evaluating Agent Interactions Through Episodic Knowledge Graphs, 2022. URL: http://arxiv.org/abs/2209.11746. doi:10.48550/arXiv. 2209.11746, arXiv:2209.11746 [cs].