The first Answer Validation Exercise (AVE) has been launched at the Cross Language Evaluation Forum 2006. This task is aimed at developing systems able to decide whether the answer of a Question Answering system is correct or not. The exercise is described here together with the evaluation methodology and the systems results. The starting point for the AVE 2006 was the reformulation of the Answer Validation as a Recognizing Textual Entailment problem, under the assumption that hypothesis can be automatically generated instantiating hypothesis patterns with the QA systems' answers. 11 groups have participated with 38 runs in 7 different languages. Systems that reported the use of logic have obtained the best results in their respective subtasks. The first Answer Validation Exercise (AVE 2006) was activated to promote the development and evaluation of subsystems aimed at validating the correctness of the answers given by QA systems. This automatic Answer Validation is expected to be useful for improving QA systems performance, help humans in the assessment of QA systems output, improve systems confidence self-score, and to develop better criteria for collaborative systems.
As a difference with the previous campaigns of the QA track, a text snippet was requested to support the correctness of the answers. The QA assessments were done considering the given snippet, so the direct relation between QA assessments and RTE judges was preserved: Pairs corresponding to answers judged as Correct have an entailment value equal to YES; pairs corresponding to answers judged as Wrong or Unsupported have an entailment value equal to NO; and pairs corresponding to answers judged as Inexact have an entailment value equal to UNKNOWN and are ignored for evaluation purposes. Pairs coming from answers not evaluated at the QA Track are also tagged as UNKNOWN and they are also ignored in the evaluation.
The evaluation is based on the detection of the correct answers and only them. There are two reasons for this. First, an answer will be validated if there is enough evidence to affirm its correctness. Figure 2 shows the decision flow that involves an Answer Validation module after searching for candidate answers: In the cases where there is not enough evidence of correctness (according to the AV module), the system must request another candidate answer. Thus, the Answer Validation must focus on detecting that there is enough evidence of the answer correctness.
Second, in a real exploitation environment, there is no balance between correct and incorrect candidate answers, that is to say, a system that validates QA responses does not receive correct and incorrect answers in the same proportion. In fact, the experiences at CLEF during the last years showed that only 23% of all the answers given by all the systems were correct (results for the Spanish as target, see [6]). Although numbers are expected to change, the important thing is that the evaluation of Answer Validation modules must consider the real output of Question Answering systems, which is not balanced. We think this leads to different development strategies closer to the real AV Exercise that, anyway, must be evaluated with this unbalanced nature.
Therefore, instead of using an overall accuracy as the evaluation measure, we proposed to use precision
(
In the other hand, the higher the proportion of YES pairs is, the higher the baselines are. Thus, results
can be compared between systems and always taking as reference the baseline of a system that accept all
answers (return YES in 100% of cases). Since UNKNOWN pairs are ignored in the evaluation (though they
were present in the test collection), the precision formula (
Eleven groups have participated in seven different languages at this first AVE 2006. Table 2 shows the participant groups and the number of runs they submitted per language. Al least two different groups participated for each language, so the comparison between different approaches is possible. English and Spanish were the most popular with 11 and 9 runs respectively. Only 3 of the 12 groups (FUH, LCC and ITC-IRST) have participated in the Question Answering Track showing the chance for new-comers to start developing a single QA module and, at the same time, open a place for experienced groups in RTE and KR to apply their research to the QA problem. We expect that in a near future the QA systems will take advantage of this communities working in the kind of reasoning needed for the Answer Validation.
Tables 3-9 show the results for all participant system in each language. Since the number of pairs and the proportion of the YES pairs is different for each language (due to the real submission of the QA systems), results can’t be compared between languages. Together with the systems precision, recall and F-measure, two baselines values are shown: the results of a system that always accept all answers (returns YES in 100% of the pairs), and the results of a hypothetical system that returns YES for the 50% of pairs.
In the languages where at least one system reported the use of Logic (Spanish, English and German) the best performing system was one of them. Although the use of Logic doesn’t guarantee a good result, the best systems used it. However, the most extensively used techniques were Machine Learning and overlapping measures between text and hypothesis.
System Id Group COGEX LCC ZNZ – TV_2 U. Rome itc-irst ITC-irst ZNZ – TV_1 U. Rome MLEnt_2 U. Alicante uaofe_2 U. Alicante MLEnt_1 U. Alicante uaofe_1 U. Alicante utwente.ta U. Twente utwente.lcs U. Twente 100% YES Baseline 50% YES Baseline ebisbal U.P. Valencia
The starting point for the AVE 2006 was the reformulation of the Answer Validation as a Recognizing Textual Entailment problem, under the assumption that hypothesis can be automatically generated instantiating hypothesis patterns with the QA systems answers. Thus, the collections developed in AVE are specially oriented to the development and evaluation of Answer Validation systems. We have also proposed a methodology for the evaluation in chain with a QA Track.
11 groups have participated with 38 runs in 7 different languages. Systems that reported the use of logic have obtained the best results in their respective subtasks.
Future work aims at developing an Answer Validation model where the hypotheses can include the type of answer requested by the question in order to reformulate the Answer Validation Exercise for the next campaign. Finally, we want to quantify the gain in performance that the Answer Validation systems give in chain with the Question Answering ones.
This work has been partially supported by the Spanish Ministry of Science and Technology within the R2D2SyEMBRA project (TIC-2003-07158-C04-02). We are grateful to all the people involved in the organization of the QA track (specially to the coordinators at CELCT, Danilo Giampiccolo and Pamela Forner) and to the people that built the patterns for the hypotheses: Juan Feu (Dutch), Petya Osenova (Bulgarian), Christelle Ayache (French), Bodgan Sacaleanu (German) and Diana Santos (Portuguese).