Question answering (QA) over large scale knowledge graphs has been the focus of much NLP research and in this paper, we focus on natural language questions that are taken from the SimpleQuestions [7] and WebQSP [3] Datasets, that contain tuples of the form (subject, relation, object, question). We tackle the problem of QA in 3 steps : 1) Extraction of mentioned entities from the question and linking to entities in the Knowledge Graph. 2) Detecting the domain of the Object (answer). 3) Prediction of the most relevant relation for answering the question.
Prior deep learning approaches use relation as a class label only and hence don't capture the semantic level correlation between the question and the relation. To overcome this limitation, we propose a Triple Input Siamese Metric Learning Model (TISML), that scores similarity between questions and candidate relations, and thereby indirectly predicts the relation most relevant to a given question. But this approach was observed to have failed at times when words of candidate relations are highly similar to the words present in the question (discussed in section 8 Type-A). As a result, this tends to mislead the model to predict the relation incorrectly. We, therefore, propose that if the broad domain of the expected answer is also input to the model, the model tends to select relevant relations improving the relation prediction model and this results in state of the art performance. Consider this question from SimpleQuestions Dataset, i.e., "who is a production company that performed Othello".
Here we first extract the mentioned entity "Othello", using a model (referred to as Entity Tagging Model), and identify all the relations of this entity in the knowledge graph as candidate relations. Considering two of the candidate relations as ("theater/ theater_production/ producing_company", "film/ film/ production_companies"). A model that takes only question and the candidate relation as input predicts "film/ film/ production_companies" as the correct relationship, which is actually wrong. However, if we also input the domain of the answer, "theater", it helps the model to score the candidate relations appropriately and predicts "theater/ theater_production/ producing_company" as the correct relation. The main contributions of this paper are : 1) We demonstrate that a metric learning similarity scoring network along with the injected domain knowledge, enhances Question Answering over the Knowledge Graph. 2) We release the SimpleQuestions and WebQSP datasets1 created for our experiments to carry out further research.
Terms mentioned entity, and subject name mean the same thing, and may be used interchangeably.
We assume that a background Knowledge Graph comprising of a set of triples (π = {π‘ 1 , ..., π‘ π }) is available, here each triple π‘ π is represented as a set of three terms {Subject, Relation, Object}, also referred to as {π, π , π}. We are concerned with natural language questions (π π β π), which mention an entity of the knowledge graph (π). We also assume that such questions can be answered using single triple (for single relation questions) or multiple triples (for multi-relation questions) of the knowledge graph. For this example, the ground truth triple comprises of subject π π ="Othello", relation π π ="theater/ theater_production/ producing_company", and object π π ="National Theatre of Great Britain". In this context, the objective of the Question Answering task is to retrieve the appropriate answer ("National Theatre of Great Britain") from the knowledge graph.
We formulate this problem as a supervised learning task. We assume that a set of questions π π = {π 1 , ..., π π } and corresponding ground truth triples π π = {π‘ 1 , ..., π‘ π } (with π‘ π = (π π π , π π π , π π π )) are available as training data. The underlying knowledge graph for our work is Freebase [6]. For Simple Questions dataset we have used a smaller version of Freebase i.e., FB2M [8] and for WebQSP dataset we have used the full Freebase.
Mapping a natural language question to a knowledge graph is a well studied task and a significant amount of work has been done on this topic over the last two decades [ [4], [23], [7], [19]]. As per recent trends, answering natural language queries via knowledge graphs follow two broad approaches namely, "Semantic Parsing based" and "Information Extraction based" which are further explained below.
β’ Semantic Parsing based : These approaches [ [4], [5], [13]] involve expressing natural language queries into SPARQL queries (logical forms) and then project these queries to a knowledge base to extract relevant facts. The advent of deep learning approaches, which captured the semantics of a natural language query helped to further improve the performance of these systems. The semantics captured through these deep learning approaches are encoded in a fixed-length vector and are projected on a knowledge graph representation to extract relevant facts. β’ Information Extraction based : Work by [22] 2 claimed that by using a simple RNN, they are able to obtain better results for both entity tagging and relation detection on SimpleQuestions dataset. Another work by [24] used Hierarchical BiLSTM based Siamese network for relation prediction and claimed that relation detection task has a direct impact on Question Answering task on both the datsets. Using attention with RNN along with a similarity matrix based CNN has been able to achieve superior results in [1]. [20] used a BiLSTM-CRF tagger followed by a BiLSTM to capture mention detection and relation classification respectively. [17] were among the first ones to apply BERT [10] for this task but did not get any improvement over the previous state-of-art. [12] proposed a similar approach to ours using similaritybased network for relation detection, however, they have removed about 2% of data from the test set. To the best of our knowledge, none of the cited approaches utilize domain information to predict relations.
the sub-type of the subject and place_of_birth is the property or the attribute of that person. So for extracting the domain of the subject for triple (S, R, O), we have to look at the first component of the relation (R). Since we are tagging the domain of the "answer" to every question in the dataset, we search for the domains of the "object" in a (subject, relation, object, question) tuple by finding reverse relation between the subject and object. The process of domain data creation is depicted in figure 1. Questions which are tagged by single domain, we refer to them as unambiguous questions and questions tagged by None3 or multiple domains are referred as ambiguous questions. In SimpleQuestions Dataset there were 57,421 unambiguous questions and 16,432 (None) and 2,057 (multiple domain) ambiguous questions. Domain Tagging for ambiguous questions: In order to tag such questions with the appropriate domain, we referred to the tagged domains of unambiguous questions, in the following steps :
(1) Create a One to One mapping between relations and domains:
β’ Create a mapping table between a relation and a domain for every unambiguous question. β’ Select the most frequently occurring domain for a relation among all the tagged domains. β’ update the mapping table with a unique domain for a relation (2) Tag domain to ambiguous questions from the mapping table.
We present a schematic diagram of the proposed approach in figure (2). Here, given a question π with ground truth triples as (π , π, π), we first find the mentioned entity or the subject of the question via an Entity Tagging Model. From the identified entity, we obtain all the candidate subjects S = {π 1 , ..., π π } (in figure 2, S = {π 1 , π 2 }) and also extract all the candidate relations R = {π 1 , ..., π π } connected to π from the Knowledge Graph (in figure 2, R = {π 1 , π 2 }). We also input this question q to another model which predicts the domain of the expected answer for that question, this model is hereafter referred to as Domain Prediction Model. Further, the question that is input to the TISML Model is modified by inserting a string < π > in place of the mentioned entity and yielding a formatted question q'. This was done to ensure that the Siamese Model is agnostic to the specific mentioned entity in the question while predicting the triplet score and could also give the positional information to the neural networks [24].
In this section, we discuss about all the three individual models in detail.
(1) Entity Tagging Model: It is a sequence labelling task (IO tagging) which uses a BiLSTM and a Conditional Random Field layer [20] for detecting the mentioned entity in the question. K entity candidates are predicted using the top-K Viterbi algorithm. Further, candidate aliases are extracted from the Freebase SQL table by querying it using predicted K candidates. Candidates having minimum Levenshtein distance (between aliases and the detected mentioned entity) will be the predicted subject names and their corresponding machine ids will be retrieved as candidate machine ids 4 . This 4 While creating SQL tables for Freebase, along every machine id (MId) different string aliases are mapped. for example:(MId| alias| alias-normalized-punctuation| alias-normalized-punctuation-stem| alias-preprocessed):::
model is used by [20] 5 , which is the state-of-the-art algorithm in Question Answering task over Knowledge Graph and is used as the baseline algorithm for comparing our results, hence, we also used the same model for our task.
(2) Domain Prediction Model: It is a supervised classification task, where the input is a question q and the output is the predicted domain of the answer type of the question. For this task, we use a pre-trained BERT Large [10] classification model and fine tune on SimpleQuestions dataset by adding an additional fully connected layer on top of BERT and learn the weights of this layer to predict the correct domain for the question. We fine-tune the model for 5 epochs and keep sequence length as 40 and batch size as 64. This model outperforms other classification models, namely, LSTM [14], CNN [16], BiLSTM with attention [11], Capsule Network [15], results for domain prediction are presented in table 1. This is because of the fact that BERT is trained on a huge corpus (Wikidata (2.5 billion words), BookCorpus (800 million words)) and can thus leverage the knowledge it has learned which results in better prediction of the domains.
(3) Triple Input Siamese Metric Learning Model: In order to select the correct relation for the question q, we use a TISML Model6 (refer to figure 3) which captures the semantics between all the inputs (question, relation, domain). This network consists of 3 different embedding generator networks -Glove Embedding Layer, 1D-CNN Layer and an LSTM Layer. Each input is passed through these networks which further generate their respective embeddings. These embeddings are then concatenated through a Merge layer followed by multiple dense layers. The final embedding computed is used to calculate a score between 0 to 1, that indicates whether the triple has correct relation or not. [24] 77 63.9 GRU [18] 71.2 -BiGRU-CRF & BiGRU [19] 73.7 -BiLSTM & BiGRU [19] 74.9 -Attentive RNN & Similarity Matrix based CNN [1] 76.8 -BiLSTM-CRF & BiLSTM [20] 78.
We have compared our approach with previous Deep Learning approaches mentioned in Section 3. Evaluation metrics used are same from the baseline approach [20] for SimpleQuestions dataset (i.e. accuracy). For WebQSP dataset evalution is done similar to [24], where Top-1 accuracy is reported for answer prediction, i.e., among multiple predicted relations we pick the top scored relation and use it for answer prediction 7 . We have also compared with our approach but without using domain information. According to Table 2, it can be seen that augmenting domain knowledge along with 7 For Web Question dataset only 64 questions in test data have multiple answers rest others have multiple relations with single tagged answers We have shown in Table 3 a few examples from SimpleQuestions dataset in case of which the relations were predicted wrongly by the baseline approach, however, with our approach such questions could be answered correctly. In our analysis, we found out that such errors were improved in our approach, due to 2 main reasons, which are -
(1) Type-A (Improvement due to Domain Prediction Model):
In the question "What high school is located in Hugo", the baseline model predicts the relation as location/ location/ containedby which is not correct, this could be because of the word "located" in the query or due to similar pattern questions belonging to this relation, and hence their model, which is a relation classification model, predicts a relation containing "location". However, our model predicts the domain of this question as "education", since the question is essentially asking about the "high school" which belongs to the education domain. This information pushes the Triple Input Siamese Metric Learning Model to select the relation similar to the education domain which is education/ school_category/ schools_of_this_kind.
(2) Type-B (Improvement due to Similarity Model): Another type of error that is improved by our approach belongs to the case when the relation predicted by both the approaches is from the same domain, but still different due to varying sub-domain. For instance, for the question "what is a chinese album", the baseline model detects music/ release_track/ release, however our model predicts music/ album_release_type/ albums as the correct relation. This is because of the fact that our model exploits the semantic level correlation between the question and the relation and is able to match the two at a literal level, which can be seen from the fact that there is a presence the word "album" in both the question as well as the relation. (3) Category-3 Error: There are 386 questions in the test set that do not contain a head_entity. Previous work done by [12] have removed such questions from the evaluation of their model, we, however, did not remove these questions from the dataset. For example, the question "Who is an alumni involved in IT" does not contain a mentioned entity, which is a data creation error and cannot be solved and predicted as None.
(4) Category-4 Error: These errors occur because Entity Tagging Model is not able to identify the subject present in the question correctly which results into a selection of wrong candidate relations set from the knowledge graph. For example, given a question "what's the name of a popular Japanese to Portuguese dictionary", has the ground truth mentioned entity as "dictionary", however, the Entity Tagging Model predicts "Portuguese" as the subject, which leads to a wrong set of candidate relations and hence results in wrong answer prediction.
In this paper, we propose the use of domain information as an additional information for predicting the correct relation for both single relation and multi-relation datasets. Such information is predicted from the question using a Domain Prediction model and helps in strengthening the outcome of the TISML Model to select the most appropriate relation for the question. Our proposed approach outperforms previous approaches on Question Answering over Knowledge Graph and achieves a new state-of-the-art results on SimpleQuestions and WebQSP datasets. For future work we will also explore datasets like GraphQuestions [21] and ComplexQuestions [2] to deal with more aspects of general Question Answering.