This paper introduces the new INAOE's answer validation method. This method is based on supervised learning approach that uses a set of attributes that capture some lexical-syntactic relations among the question, the answer and the given support text. In addition, the paper describes the evaluation of the proposed method at both the Spanish Answer validation Exercise (AVE 2008) and the Spanish Question Answering Main Task (QA 2008). The evaluation objectives were twofold. One the one hand, evaluate the ability of our answer validation method to discriminate correct from incorrect answers, and on the other hand, measure the impact of including an answer validation module in our QA system. The evaluation results were encouraging; the proposed method achieved a 0.39 F-measure in the detection of correct answers, outperforming the baseline result of the AVE 2008 task by more than 100%. It also enhanced the performance of our QA system, showing a gain in accuracy of 22% for answering factoid questions. Furthermore, when there were evaluated three candidate answers per question, the answer validation method increased the MRR of our QA system by 40%, reaching a MRR of 0.28.
Latest evaluations of question answering (QA) systems evidenced two important facts about the
state of the art of this ¯eld. First, they indicated that it already does not exist any system
capable of answering all types of questions with similar precision rates, and second, they revealed
that most current QA systems are complementary (see for instance the Spanish QA evaluation
overview at CLEF 2005 [
In line with these recent e®orts, in this paper we describe a new AV method. This method,
similar to our previous year work [
In order to evaluate the proposed method we considered two di®erent scenarios: the Answer
Validation Exercise (AVE 2008) and the Question Answering Main Task (QA 2008). The objective
of the ¯rst scenario was to evaluate the ability of our AV method to discriminate correct from
incorrect answers as well as its capacity to combine the answers from several QA systems. In
contrast, the goal of the second evaluation scenario was to measure the impact of including an
answer validation module in our QA system [
The evaluation results were encouraging; the proposed method achieved a 0.39 F-measure in the detection of correct answers, outperforming the baseline result of the AVE 2008 task by more than 100%. It also enhanced the performance of our QA system, producing a gain in accuracy of 22% for answering factoid questions. Furthermore, when there were evaluated three candidate answers per question, the answer validation method increased the MRR of our QA system by 40%, reaching a MRR of 0.28.
The rest of the paper is organized as follows. Section 2 describes our answer validation method. Section 3 presents the evaluation results of the proposed method in both the Answer Validation Exercise and Spanish QA Main Task. Finally, Section 4 exposes our conclusions and outlines some future work directions. 2
Given a question (Q ), a candidate answer (A) and a support text (S ), this method returns a con¯dence value (¯) that allows deciding whether to accept or reject a candidate answer. In other words, it helps to determine if the speci¯ed answer is correct and if it can be deduced from the given support text.
Like other previous answer validation methods evaluated in the last AVE [
The returned con¯dence value ¯ is generated by means of a supervised learning approach that considers three main processes: preprocessing, attribute extraction and answer classi¯cation. The following sections describe each of these processes. 2.1
The objective of this process is to extract the main content elements from the question, answer and support text, which will be subsequently used for deciding about the correctness of the answer. This process considers two basic tasks: on the one hand, the identi¯cation of the main constituents from the question-answer pair, and on the other hand, the detection of the core fragment of the support text as well as the consequent elimination of the unnecessary information.
1The entailment between the pair (T, H ) occurs when the meaning of H can be inferred from the meaning of T. 2.1.1
We detect three basic constituents from the questions: its main action, the action actors, and if exist, the action restriction. As an example, consider the question in Table 1. In this case, the action is represented by the verb invade, its actors are the syntagms Which country and Iraq, and the action restriction is described by the propositional syntagma in 1990.
In order to detect the question constituents we ¯rstly apply a shallow parsing to the given question2. Then, from the resulting syntactic tree (Qparsed), we construct a new representation of the question (called Q' ) by detecting and tagging the following elements: 1. The action constituent. It corresponds to the syntagm in Qparsed that includes the main verb. 2. The restriction constituent. It is represented by the prepositional syntagm in Qparsed having at least one explicit time expression (e.g., in 1990), or including a preposition such as after or before. 3. The actors constituents. These constituents are formed by the rest of the elements in Qparsed.
It is commonly divided in two parts. The ¯rst one, henceforth called hidden actor constituent, corresponds to the syntagm that includes the interrogative word and it is generally located at the left of the action constituent. The second part, which we call the visible actor constituent, is formed by the rest of the syntagms, generally located at the right of the action constituent.
Finally, we also consider an answer constituent, which is simply the lemmatized candidate answer (denoted by A' ). 2.1.2
Commonly, the support text is a short paragraph |of maximum 700 bytes according to CLEF evaluations| which provides the context necessary to support the correctness of a given answer. However, in many cases, it contains more information than required, damaging the performance of RTE methods based on lexical-syntactic overlaps. For instance, the example of Table 1 shows that only a part of the last sentence in the support text (i.e., Iraqi invasion of Kuwait) is useful for validating the given answer, whereas the rest of the text only contribute to produce an irrelevant overlap (e.g., Kuwait was a close ally of Iraq).
In order to reduce the support text to the minimum useful text fragment according to the candidate answer validation, we proceed as follows: ² First, we apply a shallow parsing to the support text, obtaining the syntactic tree (Sparsed). 2In all the text processing used in our method (i.e., lemmatization, part of speech tag, named entities recognition and classi¯cation, and shallow parsing), we employed the open source tool called Freeling (http://garraf.epsevg.upc.es/freeling/).
² Second, we match the content terms (nouns, verbs, adjectives and adverbs) from the question constituents against the terms from Sparsed. In order to avoid some minimal writing di®erences of the same concept not solved by the morphological analysis (e.g., Iraq against Irak or Iraqi), we compare the terms using the Levenshtein edition distance3. Mainly, we consider that two di®erent words are equal if their distance value is less than 0.4. ² Third, based on the number of matched terms, we align the question constituents with the syntagms from the support text. ² Forth, we match the answer constituent against the syntactic tree (Sparsed). The idea is to ¯nd all occurrences of the answer in the given support text. ² Fifth, we determine the minimum context of the answer in the support text that contains all matched syntagms. This minimum context (represented by a sequence of words around the answer) is what we call the core fragment (denoted by T' ). In the case that the support text includes several occurrences of the answer, we select the one with the smallest context.
Applying the procedure described above we determine that the core fragment of the support text showed at Table 1 is in an Iraqi invasion of Kuwait. 2.2
This stage gathers a set of processes that allow extracting several attributes from the question, the answer and the support text. These attributes can be categorized in two di®erent groups: the attributes that indicate the relation between the question and the answer, and the attributes that measure the entailment relation between the question-answer pair and the support text.
The following sections describe both kinds of attributes and explain the way they are calculated from Q', A' and T'. 2.2.1
We consider four di®erent attributes from the question: the question word (what, how, where, etc.), the question category (factoid or de¯nition), the expected answer type (date, quantity, name or other), and the type of question restriction (date, period, event, or none).
The question word, question category, and the expected answer type are determined using a set of simple lexical patterns. Some of these patterns are showed below. It can be observed that each of them includes information about the question category and the expected answer type. (WHAT OR WHO) is [whatever ]
HOW many [whatever ]
WHEN [whatever ] ! ! !
DEFINITION { OTHER FACTOID { QUANTITY FACTOID { DATE
On the other hand, the value of the question restriction (date, period, event or none) depends on the form of the restriction constituent. If this constituent contains only one time expression, then this value is set to \date". In the case the restriction constituent includes two time expressions, it is set to \period". If the restriction constituent does not include any time expression, then the question restriction is de¯ned as \event". Finally, when the question does not have any restriction constituent, the value of the question restriction is set to \none".
3The Levenshtein edition distance has been previously used in other works related to answer validation in Spanish
language, see for instance [
This attribute indicates if the question and answer types are compatible. The idea of this attribute is to capture the situation where the semantic class of the evaluated answer does not correspond to the expected answer type. For instance, having the answer yesterday for the question How many inhabitants are there in Longyearbyen?.
This is a binary attribute: it is equal to 1 when the answer corresponds to the expected answer type, and it is equal to 0 if this correspondence does not exist.
Taking into account the idea of \considering candidates as allies rather than competitors" [
Di®erent from other redundancy methods (like the one present in [
The edition distance strategy allows dealing with the great language variability and also with the presence of some typing errors. In this way, an answer X contributes to the redundancy rate of another answer Y and vice versa, even though X and Y are not exactly the same (e.g. spacial telescope and Hubble telescope). 2.2.2
The attributes of this category are of two main types: (i) attributes that measure the overlap between the support text and the hypothesis (an a±rmative sentence formed by combining the question and the answer); and (ii) attributes that denote the di®erences (non-overlap) between these two components.
It is important to explain that, di®erent from other RTE methods, we do not use the complete support text, instead we only use its core fragment T'. In addition, we neither need to construct an hypothesis text, instead we use as hypothesis the set of question-answer constituents (the union of Q' and A', which we call H' ).
These attributes express the degree of overlap |in number of words| between T' and H'. In particular, we compute an overlap attribute for each one of the fourth types of content terms (nouns, verbs, adjectives and adverbs) as well as for each one of the six types of named entities (names of persons, places, organizations, and other things, as well as dates and quantities). We generate these ten di®erent overlap attributes for each one of the ¯ve constituents in H' (the action constituent, the restriction constituent, the hidden actor constituent, the visible actor constituent, and the answer constituent). In this way, we get a total of ¯fty attributes that represent the overlap characteristics.
Similar to the calculation of the answer redundancy attribute, in this case we also apply the edition distance to evaluate the overlap between the terms of H' and T'.
These attributes indicate the number of non-overlapped terms from the core fragment of the support text, that is, they indicate the number of terms from T' that are not present in any of the detected constituents. Mainly, we measure the non-overlap between the answer constituent and each one of the other constituents, and compute this non-overlap for each type of content term as well as for each type of named entity. In total we generate forty di®erent non-overlap attributes. This ¯nal process generates the answer validation decision by means of a supervised learning approach. In particular, it applies a boosting ensemble formed by ten decision tree classi¯ers4.
The constructed classi¯er decides whether to accept or reject the candidate answer based on the ninety-six attributes described in the previous section. In addition, it also generates a validation con¯dence (¯) that indicates how reliable is the given answer in accordance to the support text. 3
As we previously mentioned, we evaluated the proposed AV method in two di®erent scenarios: the
Answer Validation Exercise (AVE 2008) and the Question Answering Main Task (QA 2008). The
objective of the ¯rst scenario was to evaluate the ability of the AV method to discriminate correct
from incorrect answers as well as its capacity to combine the answers from several QA systems.
In contrast, the goal of the second evaluation scenario was to measure the impact of including an
answer validation module in our QA system [
This evaluation exercise focuses on analyzing two di®erent aspects of the AV methods: on the one hand, their ability to discriminate correct from incorrect answers (answer validation evaluation ), and on the other hand, their capacity to select the correct answer from a pool of candidate answers returned by diverse QA systems (stream fusion evaluation ).
From previous experiments [
Table 3 shows the answer validation results corresponding to our two submitted runs. It also shows (in the last row) the results for a 100% VALIDATED baseline (i.e., an answer validation system that accepted all given answers). The results indicate that reducing the acceptance threshold (RUN 1) our method achieved a high recall but a low precision, which means that it correctly accepts must correct answers (there are a few false negatives), but it also incorrectly accepts many wrong responses (there are several false positives). In contrast, the second run (RUN 2) got a worst recall, but achieved a major precision and F-measure, outperforming the baseline result in more than 100%.
Complementary to the previous data, Table 4 shows the evaluation results for the QA stream fusion. These results indicate that the QA-accuracy of RUN 1 is 19% better than the accuracy of RUN 2. Given that RUN 2 clearly outperformed the answer validation result of RUN 1 (see Table 3), these results con¯rm our observation that the best answer validation method not necessary produces the best QA stream fusion performance.
Besides the traditional QA-accuracy measure, this year the AVE organizers included a new evaluation measure called QA-performance. This measure allows evaluating the in°uence into the question answering task of not only correctly accept those right answers but also to correctly reject those wrong ones. The results in Table 4 indicate that, because of the better capacity of RUN 2 to reject wrong answers, the QA-performance of both runs were very similar.
Finally, it is important to comment that, in order to understand the behavior of the proposed
method, we carried out a deep analysis of the usefulness of each one of the used characteristics.
Table 5 resumes the information gain values of the main kinds of attributes used by our supervised
AV method. Surprisingly, these values show that the proposed non-overlap characteristics are more
discriminative than the traditional overlap features.
In order to evaluate the e®ect of including an AV module in a QA system, this year we submitted
two di®erent runs at the Main QA task. The ¯rst run (inao081eses) was the original output of our
QA system (refer to [
Results from Table 6 indicate that the AV module helped increasing the number of right answers for factoid questions, improving the accuracy of our QA system by 22%. In contrast, the AV module damaged the treatment of de¯nition question since it incorrectly rejected three right answers. In this case, taking into account that our QA system is very accurate for answering de¯nition questions, our conclusion is that it is better not to include the AV module.
Given that this year was allowed to deliver three candidate answers per question, we not only evaluated the e®ect of the AV module in the answer accuracy but also in rank of the correct answers. For achieving this objective, we included more that one answer for some questions; the ¯rst run (inaoe081eses) consisted of 422 answers, whereas the second run (inaoe082eses) included a total of 343 answers. The di®erence in the number of answers between both runs was caused because many candidate answers were rejected in the second run during the validation process. It is important to mention that the AV module not only eliminated some tentative incorrect answers, but also modi¯ed their ¯nal order. In particular, the rank of the answers in the second run was determined by means of their con¯dence values. The evaluation results indicated that the second run outperformed by 40% (with a Mean Reciprocal Rank of 0.28) the output of the ¯rst run (with a Mean Reciprocal Rank of 0.20). 4
This paper presented a new AV method based on a supervised textual entailment approach. This method mainly di®ers from previous ones in the kind of used attributes. In particular, it considers some novel attributes that characterize: (i) the compatibility between question and answer types; (ii) the redundancy of answers across streams; and (iii) the overlap as well as the non-overlap between the question-answer pair and the core fragment of the support text. Regarding these attributes, it is important to mention that an analysis about their usefulness showed that the proposed non-overlap characteristics are more discriminative than the traditional overlap features.
The proposed method was evaluated in two di®erent scenarios: the Spanish Answer Validation Exercise (AVE 2008) and the Spanish QA Main Task (QA 2008). The objective of the ¯rst scenario was to evaluate the ability of our AV method to discriminate correct from incorrect answers as well as its capacity to combine the answers from several QA systems. In contrast, the goal of the second evaluation scenario was to measure the impact of including an answer validation module in a QA system. The evaluation results were encouraging; the proposed method achieved a 0.39 F-measure in the detection of correct answers, outperforming the baseline result of the AVE 2008 task by more than 100%. It also enhanced the performance of our Spanish QA system, producing a gain in accuracy of 22% for factoid questions. Furthermore, when there were evaluated three candidate answers per question, the AV method allowed increasing the MRR of our QA system by 40%, reaching a MRR of 0.28.
Finally, it is important to comment that this year our best results in the AVE (a F-measure of 0.39 and a qa-accuracy of 0.32) were very distant from those corresponding to a perfect validation. Ac We presume that this situation was caused by the decreasing number of right answers together with the increasing number of relevant support passages related to the wrong answers. In order to tackle these problems, and based on the fact that non-overlap attributes were the most discriminative, we plan to include more elements (such as prepositions, conjunctions, and some punctuation marks) for their computation.
This work was done under partial support of CONACYT (project grants 43990 and 61335, and scholarships 171610 and 165499). We also want to thank CLEF organizers for the provided resources. [12] Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M., eds.: Evaluation of Multilingual and Multi-modal Information Retrieval, 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, September 20-22, 2006, Revised Selected Papers. In Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M., eds.: CLEF. Volume 4730 of Lecture Notes in Computer Science., Springer (2007)