Despite the recent advances in Natural Language Processing (NLP) techniques many issues and ine ciencies arise when it comes to creating a system capable of interacting with the users by means of text conversations. The current techniques rely on the development of chatbots that, however, require designing the conversation ow, de ning training questions and associating the expected responses. Even though this process allows the creation of e ective question-answering systems, this methodology is not scalable, especially when the answers are to be found in documents. Other approaches, instead, rely on graph embedding techniques and graph neural networks to de ne the best answer given a question. These methods, however, require set-up training routines and to dispose of the ground truth that, in general, is di cult to retrieve or create for real industrial applications. In this paper we introduce a conversational framework for semantic question answering. Our work relies on knowledge graphs and the use of machine learning for determining the best answer given a question associated with the content of the knowledge graph. In addition, by leveraging text mining techniques we are able to identify the best set of answers that suit the question that are further ltered by means of deep learning algorithms.
In recent years the advances in chatbot development have given rise to e cient frameworks for virtual assistants creation. Even though it's possible to design complex conversational ows, these frameworks have not yet solved the core aspect of the technology: the question/answer pair de nition. It's indeed mandatory not only to design the conversational ow, but also to de ne a set of questions associated with the response that the virtual assistant, i.e. the chatbot, has to return. This process exposes the technology to ine ciencies mainly due to the manual process of analysing the use case, de ning the conversational ow, de ning the conversational steps, associating the training examples and the answers to each conversational steps.
In industrial contexts it's often the case where the set of questions and the respective answers are already de ned into documents that, however, are often unstructured because designed for human readers. In these situations, especially if the number of documents is non negligible, to retrieve the questions and answers pair becomes impossible, requiring the business owners to de ne manually a subset of the original questions or to avoid adopting this technology.
The framework we introduce in this paper proposes to solve the aforementioned problem. Thanks to a proprietary technology we map the content of arbitrarily shaped documents into a knowledge graph and by means of the synergy between text mining techniques and deep learning algorithms we are able to identify the best answer without any manual setup required nor training of custom models.
This research is driven by the business needs of a partner company that aims at creating an application to continuously improve an existing conversational customer service agent able to assist customers to manage and maintain the machines and software they produce. Indeed, they produce two main kinds of machines and a management software. For each product, they carefully write an user manual that contains all the necessary information to understand, use and maintain that product. However, at every new release of the products they need to update the manual, thus possibly changing also the intents, the training phrases and the answers of the chatbot system designed for customer service. For this reason we designed the framework presented in this paper, which can manage this issue by directly creating the query-answer engine using the product's manuals.
The rest of this paper is organized as follows. Chapter 2 presents an overview of the state of the art, with a focus on the methodologies for graph question answering, in Chapter 3 the knowledge graph schema and the framework logic is de ned and in Chapter 4 the experimental results obtained are shown. Finally, Chapter 5 draws some considerations about the infrastructural setup and in Chapter 6 some conclusions and future directions are mentioned. 2
For a long time has been aimed the use of systems capable of assisting users
by providing instant responses to their questions. Indeed such systems, today
called chatbots, have been studied and investigated since 1960. Many proposals
have been given to birth, such as Eliza [
Human knowledge provides a formal understanding of the world. Knowledge
graphs have become an increasingly popular research direction towards
cognition and human-level intelligence. A knowledge graph is a representation of the
knowledge contained in an ontology, a structured approach to represent concepts
and relations that belong to a shared conceptualization of a speci c domain [
The main research topics related to knowledge graphs cover four main
aspects: knowledge graph representation learning, knowledge acquisition and
completion, temporal knowledge graph and knowledge-aware applications. Among
knowledge-aware applications, an extremely active area of research is represented
by knowledge-graph-based question answering that answers natural language
questions with facts from knowledge graphs. There exist several proposals for
question answering systems based on knowledge graphs. For example, in [
Instead in [
Other approaches make use of graph neural networks, such as [17{19], to
reason over the constructed graph. In [
The framework that is presented in this paper has the objective of being a reliable system to associate user questions to the content of arbitrarily structured documents. With this objective in mind the design choices led to a solution that is general enough to map the content of documents and that requests no training because, in general, the business applications for which this framework has been developed almost never comes with the ground truth labels. For such reasons, the main drivers that led the design choices are: { The de nition of a knowledge graph schema that is able of mapping the content of arbitrarily structured documents { The adoption of text mining techniques and the use of pre-trained deep learning models on relevant content in order to avoid further training { The execution of iterative searches with increasingly less search constraints and the de nition of acceptance thresholds In section 3.1 will be presented the graph schema as well as will be described the entities and relations involved. In section 3.2 will be presented the search logic, the text mining techniques, the deep learning algorithms adopted and the iterative constraint relaxation process that has the objective of increasing the number of candidate answers to be associated with the user question. 3.1
The knowledge graph is de ned as G = (A; E; R) where A = fa1; : : : ; aAg is the set of attributes, E = fe1; : : : ; eE g is the set of entities and R = fr1; : : : ; rRg is the set of relations. Its schema has been designed with the aim of mapping the content of arbitrarily structured documents. Knowledge graph schema is depicted in Figure 1.
The entities (purple rounded rectangles) represent the domain concepts and are characterized by one or more attributes (blue ellipses). The relations (green hexagons) link two or more entities and can have one or more attributes.
As it may be noticed, some contains relations own the attribute index. This has been added to reconstruct the content of each chapter /paragraph by ordering the contained elements. In addition, any paragraph may contain an arbitrary number of paragraphs to allow replicating the structure of complex documents, often characterized by several nested levels. It's also worth mentioning the role of owner played by the entity document in relations contains. This has been added because if there exists more than a sentence or paragraph with the same text associated to more than one document, without the owner role it would be impossible to distinguish which document contains the aforementioned entity and which does not.
Finally, entity token owns attributes lemma and token-type. Each token corresponds to one of the results of the tokenization process and is further characterized by its lemma and its token type, i.e. whether it is a verb, a noun, an adjective, etc. 3.2
Given the populated graph G and a user query q, the objective of identifying the best answer is reached, in some cases, with an iterative process that involves the relaxation of the constraints involved in candidate selection.
Intuitively, user query q is analyzed to extract the set of tokens, each
composed of its lemma and the type (e.g. verb, name, adjective). Then the iterative
process of searching the best answer begins by selecting all and only the
sentences that contains all the tokens extracted from query q. If exist any sentences
that contain all the tokens, for each of them and for the query is computed the
embedding using Universal Sentence Encoder Multilingual QA [
More speci cally, rst of all the candidate answers without constraint relaxation (relaxation case I) are searched. The user query q is used to extract and select a subset, that depends on the token-type, of the corresponding set of tokens:
T q;I = ftq1; : : : ; tqT q;I g : tiq = (tiq;lemma; tiq;type)8i = f1; : : : ; T q;I g where tiq = (tiq;lemma; tiq;type)
S = fs1; : : : ; sSg 2 E is the query dependent pair composed of the token lemma and the token type obtained with respect to the user question. Then, given the set of all the sentence entities (1) (2) given the set of the contains relations that must exist between a candidate answer and the tokens selected from user query
Cq;I (s) = f(s; tq1); : : : ; (s; tqT q;I )g; s 2 S; tiq 2 T q;I ; 8i = f1; : : : ; T q;I g (4) given the set of all the relations of type contains that belong to the knowledge graph and that involve entities token and sentence SI (Cq;I ) = fs1; : : : ; sS g 2 S : 8si 2 SI (Cq;I ) ) 8cq 2 Cq;I (si)9c^ 2 C : cq = c^ (6)
The set of candidates of Equation 6 is then served to the pre-trained deep
learning model Universal Sentence Encoder Multilingual QA [
II All the tokens selected from the user query are used but is only considered attribute lemma III The combination of all the tokens selected from the user query is used considering both attributes: lemma and token-type IV The combination of all the tokens selected from the user query is used, considering only attribute lemma In particular, the set of tokens for cases II, III and IV is:
T q;II = ftq1; : : : ; tqT q;II g : tiq = (tiq;lemma)8i = f1; : : : ; T q;II g T q;III = ftq1; : : : ; tqT q;III g : 9i 2 f1; : : : ; T q;III g : tiq = (tiq;lemma; tiq;type)
T q;IV = ftq1; : : : ; tqT q;IV g : 9i 2 f1; : : : ; T q;IV g : tiq = (tiq;lemma) In addition, for cases III and IV that involve the combinations of tokens, the set of candidate answers corresponds to the union of candidate answers obtained with respect to each combination.
Finally, given the results of the constraint relaxation case II and the maximum similarity score, if the value does not exceed the threshold, then the search with constraint relaxation case III is executed. An analogous logic happens after case III that continues with constraint relaxation case IV if the threshold is not exceeded. (7) (8) (9) 4
The knowledge graph content corresponds to the content of three technical manuals about industrial machines produced for the business need of the partner company. The main topics concern document management, the resolution of problems encountered with machinery, contract management and restoration operations.
The content of such documents has been rst extracted by means of a
proprietary technology, then has been ingested into the database in accordance with
the schema described in Section 3.1. The graph database chosen for such tests
is Vaticle [
The same dataset has also been used for the development of a virtual agent structured in intents, according to the business need of the partner company. The dataset is composed of 1297 phrases, used by the conversational engine for training the model, representative of 140 intents with an average number of training phrases for each intent equal to 9:26.
The objective of the virtual agent is to answer questions about the content of the documents. For such reasons, the answers of the chatbot are similar to those that are retrieved from the framework proposed in this paper. Therefore, the evaluation process consists in a qualitative comparison of the two kinds of answers. These, in general, may di er because the answers retrieved from the framework correspond to the sentences of the document whereas the virtual assistant answers have been speci cally designed to be coherent with the conversational ow.
For such reasons, before proceeding with the evaluation analysis it has been executed a preprocessing and ltering operation of the dataset. More speci cally, the chatbot intents have been reduced from 140 to 24, namely those that return as an answer a string similar to some sentences of the documents, i.e. that have not been speci cally created as a result of a synthesis or rework of document content. In addition, a non negligible number of intents with answers like \yes", \no", \indeed" or that refer to customer service emails or phone numbers have been removed. After the aforementioned reduction of the dataset, the number of training phrases used to test the performance of the proposed framework is 205.
The qualitative evaluation process consisted in using each virtual assistant training phrase as a user query and to propose it to the presented framework to compare the responses. In particular, the service that implements the framework returns the top 10 answers that, in general, may be less than 10 depending on the number of candidate sentences eligible as answers. Each sentence has been qualitatively compared with the chatbot response and if the content of a candidate answer corresponded to the chatbot response (by means of a qualitative analysis, focusing on the semantic of the sentences), then the candidate answer was considered as correct with respect to the user query. The precision@1 and precision@3 have been computed: the results show that the precision@1 is 0:42 whereas the precision@3 is 0:89. Even though the precision@1 is not high, we can see that the precision@3 reaches really good results, especially considering that this framework does not require any kind of training. Moreover, each answer provided by the framework is associated with the paragraph and the title of the document that contains it. This is extremely useful in detecting the best answer, because many paragraphs of the same document or even of di erent documents contain similar phrases, that could match with the input question, especially if the question is not that speci c (as an example, the question "what do I have to do if the machine suddenly stop working?" does not specify which machine and given that there are two di erent manuals related to di erent machines the ranking of the answers will not take into account this information, thus the results could not be properly ranked based on the machine type but the fact that the answer is associated with the title of the manual that contains that phrase allows to easily detect the proper answer.). Thus, thanks to the fact that the framework shows all the answers detected with some informative metadata, it is easy to properly select the best answer even though this is not the rst in the ranking. This is the reason why precision@3 is a good indicator of the performance of the framework.
In order to enhance its usability, the framework is also provided with a
userfriendly interface for interaction, as depicted in Figure 2, that allows to insert
the query, to consult the answers and the graph associated to the answers.
To conclude, we present some considerations related to the infrastructure needed
to run such framework. We deployed the services using Google Cloud
Platform [
As depicted in Figure 3, we served the front-end web application using Google
App Engine [
One main value of the proposed work is that it doesn't require any training. Ultimately, the developed framework shows good performance when compared to the answer of a trained virtual assistant developed using the same dataset. In conclusion, the methodology is e ective in answering questions related to the content of documents mapped into the knowledge graph. In general, it appears to be a performing system to be used in an industrial context, especially when the source of information is contained in documents not designed for questionanswering tasks.
Some future works to enhance the performance of the proposed framework may involve investigating the use of graph machine learning techniques for node embeddings and link prediction as well as using synonyms of the tokens to augment the set of candidate sentences. Moreover, one of the most challenging tasks in the eld of knowledge graphs is represented by knowledge graph acquisition, especially because this task enables all the knowledge-aware applications. Indeed, even though we have designed an ad hoc ontology to manage unstructured documents, this ontology does not take into account speci c concepts contained in the documents. Thus, enriching the schema with concepts, relations and entities can enhance the performance of the proposed solution, especially when considering the use of graph machine learning techniques. For this reason, to enhance the framework, an important area of research to investigate is that of knowledge graph acquisition, especially knowledge graph completion, to be able to enrich the ontological schema with useful concepts, relations and entities to be leveraged for detecting the best answer.