Recent years have seen a huge increase in the popularity of information retrieval(IR) systems, which enable users to hold natural language conversations. IR Systems such as conversational agents are typically goal-oriented and use predefined queries to retrieve information from backend systems. Researchers have improved these agents to adapt to diferent modalities, such as images, sound, and video, to enhance the conversational experience. Though these systems efectively address users' information requirements, there is a need for an approach that can easily adapt to diverse use cases and meet all user's information needs without the user being aware of the backend system. In this work, we propose a novel approach called Multimodal Information Retrieval System using Knowledge Graph (MIR-KG) to address the information requirement of the user. In the proposed approach, the data surfaced through the conversation agents are stored in a backend database called knowledge graphs (KG). The approach takes multimodal input, uses an ofline representation of KG called ontology to identify entities and relations, and generates dynamic KG queries. The paper introduces a context-building technique called Multimodal Context Builder(MCB) to preserve user-provided entities in long conversations and use the ontology to build the KG queries over the context information. We compared our results with a Multi-headed Hierarchical Encoder-Decoder with attention approach and found that the proposed approach gives a more detailed response to user queries. The Training Data Generator (TDG) generates the base training set for setting up the conversation agent. This approach eliminates the time required to collect question and answer pairs needed in the case of goal-based modules. The proposed approach is demonstrated using an already constructed KG with data from the MMD[1] website. The approach can also be applied to other domains..
Conversational agents(CA) enable users to post questions and get answers. Organizations
have recently concentrated on enabling people to engage with these conversational agents
Georgia
∗Corresponding author.
†These authors contributed equally.
through multimodal interactions. The approach described in this research, MIR-KG (Multimodal
Information Retrieval System utilizing Knowledge Graph), uses ontologies and knowledge
graphs to power the question-answering process. The proposed approach combines
transformerbased intent and multimodal entity detection from user questions with procedural dynamic
query creation. The multimodal entities provided as input to the conversational agent are
progressively collected into the conversation context. The dynamic query generation engine
uses the context to generate multimodal knowledge graph queries. Multimodal Knowledge
Graphs (MMKG)[
There has been considerable work in question answering systems that use knowledge graphs.
There has also been research on multimodal conversational IR systems. However, little work has
been done using KG and transformer-based intent and entity identification models to generate
responses in multimodal IR systems. Earlier works in literature [
A Multimodal Knowledge Graph(MMKG) is a KG where the attribute set will have values, which are a multimodal representation of data such as text, image, video, sound, human expression, and so on. The representation of data will include binary data, references (image URL, video URL), and embeddings(image embeddings[16], sentence embeddings[17]). Figure 2(a) shows an example of a multimodal knowledge graph where the entities E include Product, Sleeves, Type, Color, and, Gender. Relation R includes HAS_SLEEVES, HAS_TYPE, HAS_COLOUR, FOR_GENDER. The attribute A includes both text and image data. The Product entity has text attributes Title, Price, and Description and Image attributes ImageURL, ImageData, and ImageEmbeddings
The ontology acts as an ofline schema definition of the MMKG database. It holds enough information necessary for generating the dynamic query. Ontology is defined as a set = { , , , } where ⊆ is a set of entities, ⊆ is a set of attributes. = { , , } is a set of relation paths where and are entities in and ∈ is the relation between and . ={’Text’, ’Numeric’, ’Image’, ’Embeddings’, ’Date’} is a set of data types. Each attribute
∈ will have a type ∈ . Figure 2(b) shows the entities, attributes, and types.
CA Framework[18] provides a platform for developing and hosting conversational agents, or BOTS. A CA Framework contains templates, standard development kits(SDK), and tools to facilitate the creation of conversational agents. The SDK provides a dialogue or conversation lfow feature to manage a long-running conversation. A dialog flow is configurable to perform tasks such as sending messages to users, asking users to enter questions, providing more context, and calling backend API. The conversation flow can be limited to a single or many turns, as shown in Fig 1. CA Framework has a persistent store that allows developers to store userprovided information in user variables. User variables are configurable to persist though out the chat session, initialized, reinitialized, or destroyed depending on the user input during the dialogue session. In the example in Figure 1, the dialogue flow is configured to ask the user an initial question(Q1). After Q1 results are provided, the user is asked to provide more context in Q2 and Q3. The dialogue is configured to request more context until the user is satisfied with the results or signals to close the chat.
This section captures the implementation approach for our proposed Multimodal Information Retrieval System using Knowledge Graph (MIR-KG) and Multimodal Context Builder(MCB) using modalities of text and image. Figure 3 shows the representation of the process flow. 1. The user enters the question or provides more context in a CA that is hosted on a web application or an app 2. CA framework receives the question provided. The dialogue flow triggers the NLU service to determine the intent and entities and produce logical forms. Intents are matched with the utterances generated using the training data generator. (Refer to Sec 4.1) 3. Depending on the identified intent, the dialogue flow either a) Initiates the dynamic query generation module providing the context object collected in user variables, or b) stores the intents and entities identified in the user variable and requests more context by sending a message to the user, such as ”can you provide more information.” 4. The context object and the ontology are provided as input to the dynamic query generation module, which creates the query to execute against the MMKG. 5. The results are returned to the CA framework. The CA framework formats the results to show results in a user-friendly format.
The problem with CA based on trained models is obtaining or producing the necessary data set for training the backend models. Though there are approaches such as synthetic data generation[19] used for generating the training data, there is still a requirement of ground truth question-answer pairs which may exist or may take time to gather from subject matter experts(SME). The paper proposes a novel Training Data Generator(TDG) approach to create training data containing question-answer pairs using ontology to address this problem.
The proposed TDG approach generates a set of tuples = { , , } where I is the set of intents, Q is the set of questions, and is the entity in the ontology that will be provided as a query output of question Q. The approach uses a common phrase set = { , } where P is the set of common phrases to generate the training data for questions related to entity . The algorithm for TDG is shown in Algorithm 1. For every entity in , the algorithm look for common phrases in P and generates the question obtained by concatenating the string with the entity name of and a where clause . The where clause is obtained by concatenating the string ”where” with every combination of attribute { , } ∈ where is the set of attributes defined in the ontology.
The following scenario explains the TDG process for the ontology shown in Figure 2(b). Consider = {“ ”, “ ”} , attribute set = {“ ”, “ ”} and = “ ” and = {”I am looking for a”,”show me ”}. The training tuples generated for this setup are {”getProduct”, ”I am looking for a product where product price is #ent and color title is #ent”,” Product”}, {”getProduct”, ”I am looking for a product where product description is #ent and color title is #ent”,” Product”}, {”getProduct”, ”show me product where product price is #ent and color title is #ent”,” Product”}, {”getProduct”, ”show me product, where product description is #ent and color title, is #ent”,” Product”}. It can be noted from the example that the TDG approach Algorithm 1 Training data generator Input: , from Ontology file Output: List of utterances ( ) 1: for each ∈ do for each ∈ do for each ∈ do for each ∈ do if
≠ then
The NLU service converts the natural language question into a logical form used by the dynamic query generator to generate MMKG queries. The NLU service requires certain intent and entity configurations to perform the following activities 1) Intent identification, 2) Entity identification, and 3) Logical form generation. The NLU service is built on a standard NLP framework such as spacy[20]. Figure 4 shows the process flow of NLU service.
This process involves configuring the intents and entities the CA system should support. The NLP pipeline will use the configuration to identify entities and intent from the question. The configuration process involves updating the intent(TDG output) and entity information into the NLU service configuration. Entity information include 1. Synonym Entities: Configuration to identify entities that have discrete values. The configuration information includes tuples { , , } where ∈ and are the entities in ontology, is the attribute of with value ∈ where V is the set of distinct values in MMKG for entity . For the MMKG in Figure 2, the distinct values for Color include {Color, Blue},{Color, Green}. The distinct values for gender include {Gender, Male},{ Gender
, Female}, {Gender , Male | Men},{Gender , Female | Ladies}. It can be noted that {Gender,
Female | Ladies} also captures synonyms for the value ”Female” as ”Ladies.”
2. Regular Expression Entities: Similar to the Synonym entities, this configuration
identifies entities with continuous values. In Figure 2, the ”Price” attribute in the ”Product”
entity is a continuous variable. ”250$” can be identified using the regular expression
[
3. Phrase Entities: Configuration to identify entities that are embedded in phrases.
Entity recognition uses the NLP pipeline and entity configuration to identify the entities provided as part of the query. The NLP pipeline is a feature provided by the NLP framework and consists of Entiry Ruler, Named Entity Recognition(NER) and Entity Relation Linker. Entity Ruler identifies and marks tokens in the supplied question as entities after comparing them with the configured entities. NER assigns labels to the identified entities, and the Entity relation linker identifies the relationship between the entities and the entity values depending on the relation terms such as ”greater than”, ”lesser than”, ”similar to”, ”having a value”, and so on. The entity recognition module generates quadruple = { , , , } where R is the relation value. For the question, ”can you filter these sweatshirts for men and price less than 250 dollars,” the word ”Male” is identified using Synonym entities, and ”250 dollars” is identified using Regular Expression entities. NER maps ”Male” with the ”Gender.Title” attribute and ”250” with the ”Product.Price” attribute. The Entity relation linker maps entities and attributes with values and generate quadruple {”Gender”,” Title”,” Male”,” =”} and {”Product”,” Price”,” 250”,” <”}.
The question sent to the NLU service matches the predefined questions generated by TDG. The proposed approach uses transformer-based models for sentence similarity and identifying the most relevant TDG utterance matching the query passed. If Q is the question passed, the entity vector masking modules mask the entities identified by the NLP pipeline and generate sentence embedding model[17] generates the sentence vector for is compared for similarity with all the questions in tuples = { , , . The . The sentence } generated by TDG using the algorithm mentioned in Algorithm 2. The tuple with the maximum similarity score and greater than the minimum threshold is identified as the intent of the question.
is usually set to 0.75 or above. If question Q = ”I am looking for t-shirts in red color and price less than 250 dollars”, the entity masking module generates = ”I am looking for #ent in #ent color and price less than #ent dollars.” The intent identification algorithm identifies the TDG tuple = {”getProduct”, ”I am looking for a product where product price is #ent and color title is #ent”,” Product”} as the intent tuple.
Algorithm 2 Intent Identification Input:
, Output: TDG tuple, = { , , 1: for each ∈ do ← ( ← || ∗ || ∗ | |
= 2: 3: 7: 9: 4: end for 5: 6: if 8: else ← (
≥ 10: end if
) then return = { , , } return unknown intent } ) ∑=1
√∑=1
∗ 2
√∑=1 2
and question.
A logical form is generated containing the output from entity and intent identification. The logical form generated is a set = {, , , }. Q is the question asked, ER is the set of all the recognized quadruples, is the entity tuple identified by the intent identification algorithm is the maximum similarity score of the identified intent question with the supplied
The MCB technique uses ontology and entity lists provided by the NLU to generate the context objects. The context object is a collection of entities recognized by the NLU in multiple turns. The CA framework(Refer 3.3) plays a key role in creating and updating the context object. Table 1 shows the working of the MCB approach for the sample conversation shown in Figure 1. The conversation. It can be noted from the table that the CA Framework is configured to control the flow by asking questions shown in the ”CA Response” column. The user-provided response is shown in ”User Question/Context”. Once the intent I is identified in the first turn, the CA framework considers only the entities recognized by the NLU engine to build the context object in subsequent turns. If no entities are identified, the CA framework creates the context object by mapping the question provided with the product ”Description” column. The results of the dynamic query generated using the CO will be shown to the user at each turn. If unsatisfied, the user provides more context to filter the results further. The MCB approach thus incrementally builds the context in multiple turns and becomes a convenient option to maintain the context. This approach is not limited by the total number of turns in the conversational flow.
The dynamic query generation module processes the context object stored in the CA Framework and builds the KG query. Figure 5(a) shows the query mapping between entity fields, return entity, ontology fields, and the query components for Neo4J and SQL. Figure 5(b) shows the dynamic query generation for query Q1 in Figure 1. The dynamic query generation module contains procedural code for handling data types such as embeddings, text, numeric, and date. These rules can be altered depending on the target platform. The query object is the entity whose attributes will be displayed in the results, and this value is fetched from in the context object. The query’s return fields are obtained from the entity ′ attribute set in the ontology. The ”where” clause for the question is populated by the entities listed in the context object. Since the dynamic query generator contains generic rules depending on the data types, the approach becomes extensible. This approach requires the creation of ontology to apply to a diferent domain saving significant efort in training and testing.
The proposed approach was tested by ingesting 1743 JSON records from the MMD[
Conversation agents enable users with IR by querying the underlying systems. This paper presents a novel predictive and rule-based approach that reduces training time and supports multimodal conversations using a multimodal knowledge graph. Domain ontology defines entities, attributes, and rules and drives the dynamic query generation module, which helps create conversational IR systems for a new domain with minimal efort. [15] A. Paranjape, A. See, K. Kenealy, H. Li, A. Hardy, P. Qi, K. R. Sadagopan, N. M. Phu, D.
Soylu, and C. D. Manning. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. Stanford NLP, 3rd Proceedings of Alexa Prize, arXiv:2008.12348, 2020 [16] Vision Transformer -https://en.wikipedia.org/wiki/Vision_transformer [17] Sentence Embedding -https://en.wikipedia.org/wiki/Sentence_embedding [18] Bot Framework SDK -https://docs.microsoft.com/en-us/azure/bot-service/bot-serviceoverview?view=azure-bot-service-4.0 [19] Synthetic Data -https://en.wikipedia.org/wiki/Synthetic_data [20] Language Processing Pipelines · spaCy -https://spacy.io/usage/processing-pipelines