With the rapid advancement of semantic web technologies, various Knowledge Graphs have been created by various institutes and organizations to preserve data. What is challenging is how humans can intuitively access and learn from such large-scale knowledge. In this paper, to achieve a meaningful, engaging, and enjoyable interaction between visitors and large-scale knowledge graphs, we will propose a knowledge graph-based conversational agent that selects and packages user-interested information into narratives, models user interests, and takes initiative (e.g. answering user questions, providing recommendations, or requesting feedback or clarification) to convey knowledge. As an application, we will focus on the cultural heritage domain and create a museum guide.
processes that result in massive digital collections in the form of large-scale KGs. This not only ensures the long-term preservation of cultural artifacts in their digital form but also allows online instant access to the resources that otherwise require physical presence and fosters the development of applications like virtual exhibitions and online museums. We will describe how these KGs can be accessed by visitors through a conversational museum guide in a meaningful, engaging, and enjoyable interaction. The key contributions of this research are: • Extracting triples from KGs, ordering and packaging them according to the interests of the visitors, and generating narratives from them. • Modeling visitors’ interests during the conversation for personalized and context-aware access to information. • Forming a conversational museum guide that creates meaningful, engaging and enjoyable interaction between visitors and KGs based on its own initiatives.
Lately, knowledge graphs have been used for the representation of knowledge in ML-based
chatbots in fields such as e-commerce [ 7], retrieving domain-specific context information [ 8],
disaster support [9], or cross-language communication [10]. With explicitly modeled knowledge
and the ability to connect to other KGs, integrating the KG with the chatbot helps enhance
the “intelligence” of the conversational agent when the connection of diferent contextual
information is require [11, 9, 12] or better detecting user intent [10]. KGs are also used in
combination with social robotics (e.g. Furhat1) and conversational AI platform (e.g. Rasa2) [
So far, there has not been much work exploring the use of KGs for museum guides. Varitimiadis
et al. [
In summary, most chatbots do not ofer human-like conversations, lack meaningful interaction, and do not provide the complete requested knowledge. They do not model the user’s interest or give personalized recommendations. They do not have information ordering and packaging strategies, narrative generation capabilities, and suitable strategies to convey knowledge; they respond mainly through rule-based methods and are unable to answer factoid questions 1https://furhat.io/ 2https://rasa.com/ 3https://cloud.google.com/dialogflow 4https://flask.palletsprojects.com 5https://www.europeana.eu/en beyond a set of questions. In this research, we plan to develop a KG-based conversational museum guide that is knowledgeable about the museum collections and takes the initiative of answering questions, ofering recommendations and asking for feedback or clarifications. We also investigate how to formally model user interest and incorporate it into knowledge extraction and delivery.
Our basic assumption is that our conversational agent plays a role of a museum guide that guides the visitor through the collection and delivers as much knowledge about the collection as possible while keeping the visitor as engaged as possible. The agent focuses on the KGs about some specific exhibitions or CH institutions. Additionally, the agent can access some external KGs such as Europeana, Wikidata6, or DBpedia7 but such expeditions should be brief to keep the whole interaction as focused as possible.
The main research question of our work is “How to shape knowledge-centered humanmachine conversations and meaning-making in an engaging and intuitive manner?” This question generates many other research questions which are described in the following: • RQ1: How should the information in the KG be extracted according to the interests of the visitors? How should this information be arranged (i.e. size of the information, the order of the delivery, etc.) and presented to visitors? – The KG contains a lot of information, but to present this information in an appealing and intuitive manner to the visitor, it is necessary to extract an as complete as possible and as concise as possible subgraph containing the user’s requested entities with their attributes and the entities that they are closely associated with. On the other hand, the user interest should be taken into account when ordering and packaging certain triples into a natural language response to keep the user engaged. • RQ2: How to formally model user interests during the conversation so that user interests can be reasoned directly with knowledge extraction and delivery for personalized and context-aware access to the information? – To improve the user experience, it is necessary to model the user’s interests during the interactions in order to extract user-specific information from the KG and to provide personalized recommendations. By formally representing user interest, the communication between the user and the knowledge graphs could become more meaningful and seamless. • RQ3: How to form a conversational museum guide that creates meaningful, engaging, and enjoyable interaction between visitors and KGs based on its own initiatives? – The CA should be proactively involved in the conversation to meet the information needs of visitors by answering their questions, providing recommendations, or requesting feedback or clarification. 6https://www.wikidata.org/wiki/Wikidata:Main Page 7https://www.dbpedia.org/,
The main idea behind our approach is to explore how KGs can be leveraged to create a museum guide with extended question-answering and recommendation capabilities based on user interest modelling. The proposed approach is shown in Figure 1. It comprises three main components which are Knowledge Extraction, Ordering and Packaging, Conversational Agent, and User Modeling.
This section is about our RQ1. We plan to extract knowledge from the KG for a user’s requested information by generating as complete as possible and as concise as possible subgraphs and then order and package certain triples into natural language responses.
Classifying intents is usually the first step in conversational systems. This process maps a user’s requested information to a predefined class with the aim of facilitating the understanding of it. This classification helps identify user intention before generating SPARQL queries. Tokenization and part-of-speech tagging approaches will be used to extract information about diferent types of entities. Additionally, entity linking is needed to recognize whether the entities mentioned by the user are actual entities in the KG. We intend to handle both intent classification and entity recognition using Dual Intent Entity Transformer (DIET) [14]. We expect users to perform the following dialogue acts [15] (Note that the exact list of acts will be decided based on data collected from real conversations between visitors and museum guides): • A request about the topic (e.g., I’d like to know about ”The Night Watch”). • A request about an aspect (e.g., Could you tell me about its creator?). • A request about a mentioned concept (e.g., Do you know more about Rembrandt?). • A requests about an unmentioned concept (What is there to know about oil paint?). • Provide positive feedback (e.g., That’s quite interesting!). • Provide negative feedback (e.g., That’s pretty boring). • Accept ofer of information (e.g., I’d love to learn about its creator).
• Decline ofer of information (e.g., Sorry, I’m not interested in that.).
The main purpose is to create a series of queries to extract a subgraph from the KG containing information about the user request. We intend to generate SPARQL queries from the user request eficiently and query the KG to retrieve a sub-graph containing information about the user request. How to decide the boundary of such a subgraph, how complete and how concise such subgraph should be, etc. are the questions we want to address.
Once the subgraph is extracted, we need to consider how to package multiple triples in the optimal order to generate short utterances, instead of generating a long text which is not suitable for conversations. The order of information should reflect the user’s interest and the agent’s own goal so that a suficient amount of knowledge is transferred to the user in a personalized and engaging order.
Generating natural language texts from a set of triplets is a challenging task in NLG [16, 17, 18]. We plan to use pre-trained language models [19] for this purpose.
In order to answer RQ2, we plan to formally model and represent user interests and incorporate them while extracting and delivering knowledge. Based on the Cultural Heritage KG, we will build a basic KG of user interests that will be updated and extended dynamically during the conversation. We intend to assign weights to the user’s interest in each type of entity in the KG and dynamically update them as the user expresses interest in a particular entity. With such weighted information, the user interest KG can be incorporated into the knowledge extraction, ordering, and packaging step (see Section 4.1) that makes sure the extracted subgraph is of the most interest to the user and the knowledge is ordered and delivered in a more personalized manner.
This section is about our RQ3. The focus is on how to make the agent self-driven with its own purpose, decide which actions to take given the user’s input, and not only answer questions, but also recommend or ask for feedback or clarification. Additionally, how to make sure the conversations are engaging and enjoyable.
The conversational AI platform is used for managing interaction with the user. Dialogflow and Rasa are the most popular conversational AI platforms, that use advanced AI techniques and can integrate large sources of knowledge including KGs.
The conversational museum guide on top of the usual acts (e.g. greetings, ...) should have the ability to answer visitors’ questions, make recommendations, or request feedback or clarification to satisfy visitors’ information needs. Overall, we expect the CA to perform the following dialogue acts: • Directly answer an info request (”The Night Watch” painting is ...). • Provide related info (e.g., I do not know, but. . .). • Ask for feedback (e.g., Do you find (info) interesting?). • Ofer to discuss topic (e.g., Want to learn about ”The Night Watch”?). • Ofer to discuss aspect (e.g., Do you want to know about its creator?).
• Ofer to discuss mentioned concept (e.g., I could say more about the Rembrandt?).
To the best of our knowledge, there is no available dataset containing conversations between users and CA in the museum domain. Therefore, we plan to interview museum guides to understand “What is the dialogue between visitors and guides”, set up a small-scale data collection to record such dialogues, and potentially apply existing algorithms to recognize prototypical dialog acts in the museum context.
For evaluation, we plan to create a user study as an assessment approach. To make this evaluation interesting for the user, we plan to integrate our CA into a virtual reality environment with multi-modal input (e.g. user’s utterances, eye-gaze, speech emotion) and output (e.g. text, voice). This work is in line with our previous work8, a conversational museum guide in a web-based virtual exhibition based on a previous physical exhibition of the Het Rembrandthuis in Amsterdam.
One evaluation metric in the user studies could be asking users and domain experts to give a rating on a scale of 1 to 5, based on whether the information extracted from the KG is concise and complete and properly transformed into the natural language, whether the extracted information is interesting, whether users’ interests been taken into account in the conversation, whether the CA’s responses make sense with respect to the user’s query, etc. Another evaluation could be a user study between our CA and an existing deployed chatbot (e.g. Google Arts9 and Google 8https://www.hhai-conference.org/demos/pd_paper_3267/ 9https://artsandculture.google.com/ Assistant10). Additionally, for evaluation of the generated text from the KG, we plan to calculate the BLEU score [20].
To conclude, we have seen from the state-of-the-art approaches that research on KG-based conversational agents is still in its early stages. With this research, we intend to push this forward by proposing a KG-driven museum guide that takes initiative to create a meaningful, engaging, and enjoyable interaction between visitors and large-scale knowledge graphs. It particularly will have the ability to extract sub-graphs from a KG based on users’ requested information, order and package information, generate natural language from KG, answer questions, model user interest, and make recommendations. In the future, we plan to integrate it into a virtual reality environment with multi-modal input for detecting user interest and multi-modal responses in virtual reality. Given this, we expect the outcome of this research to help shape knowledge-centered human-machine conversations and meaning-making in an engaging and intuitive manner.
I would like to thank my supervisors Prof. Dr. Dirk Heylen and Dr. Shenghui Wang for their support, guidance, and constructive suggestions for this research proposal. Further thanks to Dr. Mariet Theune for invaluable feedback on this work. [7] A. Vegesna, P. Jain, D. Porwal, Ontology based chatbot (for e-commerce website), International Journal of Computer Applications 179 (2018) 51–55. [8] A. Quamar, C. Lei, D. Miller, F. Ozcan, J. Kreulen, R. J. Moore, V. Efthymiou, An ontologybased conversation system for knowledge bases, in: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, 2020, pp. 361–376. [9] J. Boné, J. C. Ferreira, R. Ribeiro, G. Cadete, DisBot: a Portuguese disaster support dynamic knowledge chatbot, Applied Sciences 10 (2020) 9082. [10] J. Shabbir, M. U. Arshad, W. Shahzad, NUBOT: Embedded knowledge graph with RASA framework for generating semantic intents responses in Roman Urdu, arXiv preprint arXiv:2102.10410 (2021). [11] A. Ait-Mlouk, L. Jiang, KBot: a knowledge graph based chatbot for natural language understanding over linked data, IEEE Access 8 (2020) 149220–149230. [12] S. Yoo, O. Jeong, Auto-growing knowledge graph-based intelligent chatbot using BERT (2020). [13] O.-M. Machidon, A. Tavčar, M. Gams, M. Duguleană, CulturalERICA: A conversational agent improving the exploration of European cultural heritage, Journal of Cultural Heritage 41 (2020) 152–165. [14] T. Bunk, D. Varshneya, V. Vlasov, A. Nichol, DIET: Lightweight language understanding for dialogue systems, arXiv preprint arXiv:2004.09936 (2020). [15] P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, M. Gardner, A dataset of information-seeking questions and answers anchored in research papers, arXiv preprint arXiv:2105.03011 (2021). [16] H. Gao, L. Wu, P. Hu, F. Xu, RDF-to-text generation with graph-augmented structural neural encoders, in: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 3030–3036. [17] Z. Li, Z. Lin, N. Ding, H.-T. Zheng, Y. Shen, Triple-to-text generation with an anchor-toprototype framework, in: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 3780–3786. [18] G. Zhou, G. Lampouras, WebNLG challenge 2020: Language agnostic delexicalisation for multilingual RDF-to-text generation, in: Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), 2020, pp. 186–191. [19] L. F. Ribeiro, M. Schmitt, H. Schütze, I. Gurevych, Investigating pretrained language models for graph-to-text generation, arXiv preprint arXiv:2007.08426 (2020). [20] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.