This report is about our participation in the Answer Validation Exercise (AVE2008). Our system casts the AVE task into a Recognizing Textual Entailment (RTE) problem and uses an existing RTE system to validate answers. Additional information from named-entity (NE) recognizer, question analysis component, and so on, is also considered as assistances to make the final decision. In all, we have submitted two runs, one run for English and the other for German. They have achieved f-measures of 0.64 and 0.61 respectively. Compared with our system last year, which purely depends on the output of the RTE system, the extra information does show its effectiveness.
as the Root Node. Together with the dependency paths in between, they form a subtree of the original
dependency structure, which can be viewed as an extended version of Predicate-Argument Structure
On top of the tree skeleton of H, the tree skeleton of T can also be extracted. We assume that if the entailment holds from T to H, at least, they will share the same topics. Since in practice, there are different expressions for the same entity, we have applied some fuzzy matching techniques to correspond the topic words in T and H, like initialism, partial matching, etc. Once we successfully identify the topic words in T, we trace up along the dependency parse tree to find the lowest common parent node, which will be marked as the root node of the tree skeleton of T1.
After some generalizations, we merge the two tree skeletons by 1) excluding the longest common prefixes for
left spines and 2) excluding the longest common suffixes for right spines. Finally, we will get the dissimilarity of
the two tree skeletons and we call it Spine Differences, i.e. Left Spine Difference (LSD) and Right Spine
Difference (RSD). Then, since all the remaining symbols are POS tags and (generalized) dependency relation
tags, they altogether form a Closed-Class Symbol (CCS) set. The spine difference is thus a sequence of CCSs. To
represent it, we have utilized a Subsequence Kernel and a Collocation Kernel
We have also considered the comparison between root nodes and their adjacent dependency relations. We have observed that some adjacent dependency relations of the root node (e.g. <SUBJ>or <OBJ>) can play important roles in predicting the entailment relationship. For instance, the verb “sell” has a direction of the action from the subject to the object. In addition, the verb “sell” and “buy” convey totally different semantics. Therefore, we assign them two extra simple kernels named Verb Consistence (VC) and Verb Relation Consistence (VRC). The former indicates whether two root nodes have a similar meaning, and the latter indicates whether the relations are contradictive (e.g. <SUBJ> and <OBJ> are contradictive).
Finally, the main approach is assisted by two backup strategies: one is called the Triple Similarity and the other is called the BoW Similarity. Chief requirements for the backup strategy are robustness and simplicity. Accordingly, we construct a similarity function, which operates on two triple (dependency structure represented in the form of <head, relation, modifier>) sets and determines how many triples of H are contained in T. The core assumption here is that the higher the number of matching triple elements, the more similar both sets are, and the more likely it is that T entails H. The function uses an approximate matching function. Different cases (i.e. ignoring either the parent node or the child node, or the relation between nodes) might provide different indications for the similarity of T and H. We then sum them up using different weights and divide the result by the cardinality of H for normalization. The BoW similarity score is calculated by dividing the number of overlapping words between T and H by the total number of words in H after a simple tokenization according to the space between words. 1 The Root Node of T is not necessary to be a verb, instead, it could be a noun, a preposition, or even a dependency relation.
Since the input of the AVE task is a list of questions, their corresponding answers and the documents containing these answers, we need to adapt them into T-H pairs for the RTE system. For instance, the question is, How many "Superside" world championships did Steve Webster win between 1987 and 2004? (id=87)2 The QA system gives out several candidate answers to this question, as follows, ten (id=87_1) 24 (id=87_2) … Each answer will have one supporting document where the answer comes from, like this,
The most successful sidecar racer in Superside has been Steve Webster MBE, who has won ten world championships between 1987 and 2004. (id=87_1)
The assumption here is that if the answer is relevant to the question, the document which contains the answer should entail the statement derived by combining the question and the answer. This section will mainly focus on the combination of the question and the answer and in the next sections the RTE system and how to deal with the output of the system will be described.
In order to combine the question and the answer into a statement, we need some language patterns. Normally, we have different types of questions, such as Who-questions asking about persons, What-questions asking about definitions, etc. Therefore, we manually construct some language patterns for the input questions. For the example given above (id=87), we will apply the following pattern,
Steve Webster won <Answer> "Superside" world championships between 1987 and 2004. (id=87)
Consequently, we substitute the <Answer> by each candidate answer to form Hs – hypotheses. Since the supporting documents are naturally the Ts – texts, the T-H pairs are built up accordingly,
Id: 87_1 Entailment: Unknown Text: The most successful sidecar racer in Superside has been Steve Webster MBE, who has won ten world championships between 1987 and 2004.
Hypothesis: Steve Webster won ten "Superside" world championships between 1987 and 2004.
These T-H pairs can be the input for any generic RTE systems. In practice, after applying our RTE system, if the T-H pairs are covered by our main approach, we will directly use the answers; if not, we will use a threshold 2 The “id” comes from AVE 2008 test data, i.e. “AVE2008-annotated-test-EN.xml”. to decide the answer based on the two similarity scores. Therefore, every T-H pair has a triple similarity score and a BoW similarity score, and for some of the T-H pairs, we directly know whether the entailment holds. The post-processing is straightforward, the “YES” entailment cases will be validated answers and the “NO” entailment cases will be rejected answers. In addition, the selected answers (i.e. the best answers) will naturally be the pairs covered by our main approach or (if not,) with the highest similarity scores. 3.2
Additional Components
The RTE system is used as a core component of the AVE system. Based on the error analysis of last year’s
results, this year we use additional components to filter out noisy candidates. Therefore, two extra components
are added to the architecture, the NE recognizer and the question analyzer. For NE recognition, we use
StanfordNER
NE list: Steve Webster (person), 1987 (date), 2004 (date); A_Type: Number
A’_Type: Number
Then, heuristic rules are straightforward to be applied, e.g. checking the consistence between A_Type and A’_Type, checking whether all (or how many of) the NEs also appear in the documents, etc. All these results together with the outputs of the RTE system will be synthesized to make the final decision. 4
We have submitted two runs for this year’s AVE tasks, one for English and one for German. In the following, we will first show the table of the results and then present an error analysis.
In the table, we notice that both for English and German, our validation system outperforms the best QA systems, which suggests the necessity of the validation step. Although there is a gap between the system performance and the perfect selection, the results are quite satisfactory. If we compare this year’s results with last year’s, the additional information does improve the results significantly.
Comparing the recall and precision, for both languages, the latter is worse. Therefore, we did some error analysis to see whether there is still some space for improvements. An interesting example in the English data is as follows,
Question: What is the name of the best known piece by Jeremiah Clarke? (id=0011) Answer: a rondo (id=0011_7) Document: The most famous piece known by that name, however, is a composition by Jeremiah Clarke, properly a rondo for keyboard named Prince of Denmark's March.
Our system wrongly validated this answer, because “a rondo” is not the name of that music work. In fact, what we need here is a special proper name recognizer which can differentiate whether the noun is a name for a music work.
In the German data, other kinds of errors occur. For instance,
Question: Wer war Russlands Verteidigungsminister 1994? (id=00203) Answer: Pawel Gratschow (id=0020_6) Document: Wie der russische Verteidigungsminister Pawel Gratschow am Mittwoch in Tiflis weiter bekanntgab, will Rußland insgesamt fünf Militärstützpunkte in den KaukasusRepubliken Georgien, Armenien und Aserbaidschan einrichten. 1994-02-02
The key problem here is that the year “1994” in the document might not be the year when the event happened, but the year of the report. This asks us to further synthesize the information we have, i.e. NE annotation and dependency parsing, to make better use of them. 5
To sum up, in this paper, we described our participation of AVE 2008. Based on the experience of last year’s participation, apart from the RTE core system, we add two extra components, NE recognizer and question analyzer, to further improve the results. The strategy is quite successful according to the comparison of system performances.
However, the problem has not been fully solved. Due to the noisy web data, filtering some documents in the preprocessing step could be even more effective than working on the post-processing phase. Another direction considered by us is to take a closer look at the different performances between different languages.