This work focuses on the relation matching problem in knowledge based question answering systems. Finding the right relation a natural question asks is a key step in current knowledge based question answering systems, while also being the most difficult one, because of the mismatch between natural language question and formal relation type definitions. In this paper, we present two approaches to tackle this problem. The first approach tries to directly learn the soft match between the question and the relations from the training data using neural networks. The second approach enriches the relation name with natural language support sentences generated from Wikipedia, which provide additional matches with the question. Experiments on the WebQuestions dataset demonstrate that both of our approaches improve the relation matching accuracy of a prior state-of-the-art. Our further analysis reveals the high quality of support sentences and suggests the rich potential of support sentences in question answering and semantic parsing tasks.
Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
In: L. Dietz, C. Xiong, E. Meij (eds.): Proceedings of the First Workshop on Knowledge Graphs and Semantics for Text Retrieval and Analysis (KG4IR), Tokyo, Japan, 11-Aug-2017, published at http://ceur-ws.org use neural networks to select the most related (best) NLSS for the relation. The best NLSS of the relation is then matched with the question via another LSTM, and provide a richer and more natural description of the relation.
Our experiments on the WebQuestion [
The system architecture is illustrated in Figure 1. The process of answering each question can be described with the following four steps:
Entity Linking: Identify entities from the question [
Answer Ranking: Rank the fact candidates using Learning to rank [
According to our error analysis on one of previous state-of-the-art OpenQA system, Aqqu, [
Ground Truth error (30:0%): The dataset gives the wrong answer or no answers are provided.
Relation Matching error (41:96%): Wrong relation is matched by the top fact candidate.
Entity Linking error (9:82%): Wrong entities or no entities identified.
Other (18:22%): Knowledge base inconsistency, invalid matching patterns, and combinations of several error types.
The relation matching is the major challenge of the system: most of the errors the system made are due to relation
matching mistakes. Thus this work mainly focuses on this step and follows the implementation of the Aqqu [
The goal of the semantic parsing stage is to find a matching score between the question q, and a fact candidate c. A fact candidate is the triple of subject s, relation r, and object o . Because the object mostly does not appear in the question, the semantic parser’s goal is to match q with the candidate facts’ subject and relation (s; r).
We employ two LSTM models to perform this relation matching: LSTM-Seq and LSTM-Match.
LSTM-Seq takes the question q and fact pair (subject s and relation r) as the input. It first concatenates them to a sequence q + s + r, and maps their words to embeddings. It then uses a Bi-LSTM to map the input sequence q + s + r to hidden representations, and employ a dense layer to predict the ranking score:
fLSTM-Seq(q; c) = Dense(Bi-LSTM(Embedding(q + s + r))):
LSTM-Match takes the question q and the relation r as the input. Instead of concatenating them into one word sequence, LSTM-Match maps q and r to embeddings, uses two Bi-LSTM’s to map the q and r to their hidden representations, and computes the cosine similarity of the Bi-LSTM’s outputs.
fLSTM-Match(q; c) = cosine(Bi-LSTM(Embedding(q)), Bi-LSTM(Embedding(r)))
Training: The two LSTM’s are trained using standard pairwise learning to rank:
= argmin max(0; 1
f (q; c+) + f (q; c ));
(1)
(2)
(3)
The parameters to learn include the word embeddings, LSTM parameters, and the Dense layer’s parameters. The two
LSTM’s and their embedding layers are trained independently with separate parameters. The training pairs c+; c are
formed by their F1 score [
The short natural language question and the formally defined relation names lead to vocabulary mismatch problems. For instance, the question “how was pluto discovered?” should be matched with the Freebase relation “discovery technique”, which is hard to distinguish from the relation “discoverer”. This section introduces our approaches that enrich the relation description with natural language support sentences (NLSS) generated from Wikipedia. 4.1
For each candidate fact triple (s; r; o), we extract all possible support sentences in Wikipedia that contain both the root entity s and the answer entity o. Specifically, We scan through the entire enwiki data dump and parse each Wikipedia page into a list of sentences. Wikipedia has manual annotations for each Wikipedia entity, and we can map them to Freebase entities. For all sentences in Wikipedia that contain both s and o, we add them to the NLSS pool for the candidate fact triple (s; r; o). If the answer node o represents an attribute, not an entity, then all sentences contain the root entity are included as its NLSS. 4.2
We first select the best support sentences for each candidate fact, given there could be many sentences mentioning the subject and object entities. Then we use these best support sentences to enrich the semantic parsing, which provides the ‘relation expansion’ effect.
Support Sentence Selection. There are two ways to select the best support sentence: (1) the one with the highest similarity with the question; (2) the one with the highest similarity with the fact candidate. Both similarity scores are determined by the trained LSTM-Seq model.
Semantic Parsing with Support Sentences. We use another LSTM-Seq model to match the best support sentences
with the question. It is trained and used the same as described in the last section, only that the fact candidate is replaced by
its best support sentences. This generates two additional ranking scores, which are then combined with the other semantic
parsers’ (for example, LSTM-Seq and LSTM-Match) using RankSVM [
WebQuestions [
The performances of our relation matching methods are shown in Table 1 shows the performance results. The first column lists the semantic parsers used, and the ‘+’ sign refers to the combination of multiple parsers using RankSVM.
Effectiveness of LSTM: LSTM-Seq and LSTM-Match alone are able to achieve similar performance with the stateof-the-art relation matching model CDSSM. Combining LSTM-Seq and LSTM-Match together outperforms CDSSM by 5% absolute F1 score. Combining the three neural networks together leads to another 2 3% relative improvements in precision and recall, and greatly improves the robustness of the system: 5% more questions are correctly answered in Top 10. This shows that LSTM-Seq and LSTM-Match cover different aspects of the relation matching and can be further combined with the previous states-of-the-art.
Effectiveness of Support Sentences: Adding support sentences further improves the recall from 53:1% to 57:2%, and precision from 37:2% to 39:6%. Intuitively, the support sentences provide additional expressions of the candidate fact’s relation, mitigate the vocabulary mismatch problem, and thus are more effective in improving recall.
Here is an example question that is answered correctly after using the support sentences:
Question: who was vp for lincoln? Matched fact candidate: [Abraham Lincoln] ! government/us president/vice president ! Andrew Johnson, Hannibal Hamlin Fact candidate matched previously: [Abraham Lincoln] ! person.profession ! Lawyer, Politician, Statesman Top support sentence: Also in 1864, Brandegee was a member of the Connecticut delegation to the [[1864 Republican National Convention—National Republican Convention]] in [[Baltimore]], which re-nominated President [[Abraham Lincoln—Lincoln]], and nominated [[Andrew Johnson]] for the [[Vice President of the United States—Vice Presidency]].
From this example, we can see that the QA system seems to confuse the question “who was vp for lincoln” with “who was lincoln”. The matched relation is “person.profession” before using the support sentences. The support sentences enrich the vocabulary of the fact candidate, and increases the matching score with the question term ‘vp’.
We further studied the per question win/loss of the support sentences. We found that adding support sentences gives different answers for a total of 234 questions in testing, with 87 wins, 33 losses, and other ties. The high win/loss ratio (87/33) demonstrates the advantage of the support sentences, but the small number of influenced questions (234/2032) limits the overall improvements. The reason is that many fact candidate sentences (75%) have no support sentences at all, as shown in Table 2. Our support sentence finding method is rather simple and may miss too many possible support sentences. How to align more natural language sentences to Freebase triples while maintaining their quality is an important task in the future work.
Error Analysis: We further performed an error analysis of our system. Among the 58% errors our system made, 38%(771) of the questions are answered with wrong relations, 12% are due to entity linking error, and the rest 8% are either because the answer or the entities cannot be found in Freebase. The distribution shows that the core problem of the system is still relation matching. Many such errors are rather subtle and would require more fine-grained language understanding. For example, in the example below the system mistakenly ranked ‘place of birth’ before ‘nationality’ while the question wants the country.
Question: where is jason mraz from? Correct fact candidate: [Jason Mraz] ! people.person.nationality ! United States of America (ranked at 3rd) Matched fact candidate: [Jason Mraz] ! people.person.place of birth ! Mechanicsville (ranked at 1st) 7
This work focuses on the relation matching problem in knowledge based question answering systems. This task is challenging due to the different characteristics of natural language questions and formal relation type schema: Our manual error analysis of a prior state-of-the-art system, Aqqu, suggests that most of the errors the QA system made are because of relation matching mistakes. We developed two different approaches to address this problem. The first one uses two BiLSTM’s to directly learn the match between the question and the candidate fact triples, while the second one enriches the relation descriptions with support sentences found in Wikipedia, and feed the support sentences into the LSTM to address the vocabulary mismatches.
Our experiments demonstrate that both of our approaches provide similar or better relation match accuracy compared to
the CDSSM used in a prior state-of-the-art knowledge based QA system [
Possible future work includes experimenting with better support sentence generating techniques, hoping to improve the
coverage while maintaining the high quality. Another direction is to incorporate the presented relation matching methods
into state-of-the-art QA systems, for example, Aqqu [
Perhaps the most interesting finding of our exploration is the promising potential of the joint utilization of the formal
semantics in the knowledge graphs and the rich natural language support sentences. Previously such support sentences
were mostly used during the extraction of the knowledge base facts, for example, in OpenIE [