The Answer Validation Exercise at the Cross Language Evaluation Forum (CLEF) is aimed at developing systems able to decide whether the answer of a Question Answering (QA) system is correct or not. We present here the exercise description, the changes in the evaluation with respect to the last edition, and the results of this third edition (AVE 2008). Last year's changes allowed us to measure the possible gain in performance obtained by using AV systems as the selection method of QA systems. In this edition we wanted to reward AV systems able to detect if all the candidate answers to a question are incorrect. 9 groups have participated with 24 runs in 5 different languages, and compared with the QA systems, the results show an evidence of the potential gain that more sophisticated AV modules might introduce in the task of QA. The first Answer Validation Exercise (AVE 2006) [6] was activated two years ago in order to promote the development and evaluation of subsystems aimed at validating the correctness of the answers given by Question Answering (QA) systems. In some sense, systems must emulate human assessment of QA responses and decide whether an answer is correct or not according to a given supporting text. This automatic Answer Validation is expected to be useful for improving QA systems performance [4]. However, the evaluation methodology in AVE 2006 did not permit to quantify this improvement and thus, the exercise was modified in AVE 2007 [8], where the problem of Automatic Hypothesis Generation was also opened. In AVE 2007 participant systems had to emulate QA systems selecting one answer per question from a set of candidate ones. These candidate answers were the ones given by QA systems participating at the QA main track at CLEF. This allowed us to study the use of Answer Validation (AV) systems as the answer selection method used by a QA system. Nevertheless, it was not acknowledged the ability of an AV system detecting if all the candidate answers to a question were incorrect. Systems with this ability could ask for new answers to the QA systems, opening the possibility of obtaining a correct answer to the question. Besides, NIL answers could be detected. Then, we have studied this behaviour in AVE 2008. Following the format proposed in AVE 2007, in this edition participant systems received a set of triplets (Question, Answer, Supporting Text) and they had to return a value for each triplet rejecting or accepting it. More in detail, the input format was a set of pairs (Answer, Supporting Text) grouped by Question (see Figure 1 for an example). Systems must consider the Question and validate each of these (Answer, Supporting Text) pairs. The number of answers to be validated per question depends on the number of participant systems at the QA main track.
<q id="0001" lang="EN"> <q_str>What was the nationality of Jacques Offenbach?</q_str> <a id="0001_1" value=""> <a_str>Germany</a_str> <t_str doc="Offenbach">Offenbach Offenbach Offenbach can refer to: The city Offenbach in Hesse,
Germany.</t_str> </a> <a id="0001_2" value=""> <a_str>France</a_str> <t_str doc="Jacques Offenbach">His son received the name "Jakob Offenbach" at birth, though he changed it to Jacques when he settled in France.</t_str> </a> <a id="0001_3" value=""> <a_str>Thousand Oaks</a_str> <t_str doc="LA111794-0288">Ventura College's production of George Bernard Shaw's "Arms and the Man" and Moorpark College's version of the Jacques Offenbach operetta "La Vie Parisienne" are the costume shows; in Thousand Oaks, Cal Lutheran University is mounting the contemporary drama "Minor
Demons."</t_str> </q> Participant systems must return one of the following values for each answer according to the response format (see Figure 2): • • •
VALIDATED indicates that the answer is correct and supported by the given supporting text. There is no restriction in the number of VALIDATED answers returned per question (from zero to all). SELECTED indicates that the answer is VALIDATED and it is the one chosen as the output to the current question by a hypothetical QA system. The SELECTED answers are evaluated against the QA systems of the Main Track. No more than one answer per question can be marked as SELECTED. At least one of the VALIDATED answers must be marked as SELECTED.
REJECTED indicates that the answer is incorrect or there is not enough evidence of its correctness. There is no restriction in the number of REJECTED answers per question (from zero to all). q_id a_id [SELECTED|VALIDATED|REJECTED] confidence
In the exercise we want to promote the development of AV systems that perform an analysis beyond the use of redundancies in answers. Since the fact of grouping all the answers to the same question could lead to provide extra information based on counting answer redundancies, like in AVE 2007, if an answer is contained in another answer, we remove the shorter one. Besides, with this processing no extra information is given with respect to QA participant systems at the main track, allowing the comparison with them. Furthermore, NIL and void answers were discarded for building the collections. This processing lead to a reduction in the number of answers initially available in the collections (see Tables 1 and 2): from 13.79% in the Italian development collection to 78.57% in the Bulgarian test collection.
Like in the past edition of QA@CLEF [3], questions were grouped by topic. In this organization by topics, the first question of each topic is self contained in the sense that there is no need of information outside the question to answer it. However, the rest of the topic questions can refer to implicit information linked to the previous questions and answers of the topic group (anaphora, co-reference, etc.). Therefore, for the AVE 2008 test collections we only made use of the self-contained questions (the first one of each topic group) and their respective answers given by the participant systems in QA.
For the assessments, we reused the QA judgements because they were done considering the supporting snippets in a similar way the AV systems must do. The relation between QA assessments and AVE judgements was the following: • Answers judged as Correct in QA have a value equal to VALIDATED in AVE • Answers judged as Wrong or Unsupported in QA have a value equal to REJECTED in AVE • Answers judged as Inexact in QA have a value equal to UNKNOWN in AVE and are ignored for evaluation purposes.
• Answers not evaluated at the QA main track (if any) are also tagged as UNKNOWN in AVE and they are also ignored in the evaluation.
Development collections were obtained from the QA@CLEF 2006 [5] and 2007 [3] main track questions and answers. Table 1 shows the number of questions and answers for each language together with the percentage that these answers represent over the number of answers initially available, and the number of answers with VALIDATED and REJECTED values.
These collections were available for participants after their registration at CLEF at http://nlp.uned.es/clefqa/ave/
Questions n a m r e
G 108 h s i l g n E 67 h s i n a p
S 169 h c n e r
F 118 n a i l a t
I 100 h c t u D 78
Test collections were obtained from the runs sent to QA@CLEF 2008 main track [2]. In this edition, there were runs in 9 languages: German, English, Spanish, French, Bulgarian, Dutch, Portuguese, Romanian and Basque. Thus, a test collection in AVE was generated for each of these languages.
Table 2 shows the number of questions and the number of answers to be validated (or rejected) in the test collections together with the percentage that these answers represent over the answers initially available.
Answers (final) 1027 1055 1528 199 27 228 1014 % over available answers 39.61% 57.37% 49.98% 60.30% 21.43% 42.54% 43.63% VALIDATED 111 79 153 52 12 44 208
REJECTED 854 940 1354 126 9 177 747 UNKNOWN 62 36 21 21 6 7 59
Table 2. Number of questions and answers in the AVE 2008 test collections 497 48.58% 52 406 39
In order to evaluate systems' performance, we used two groups of measures. In [7] was argued why the AVE
evaluation is based on the detection of correct answers. Then, instead of using an overall accuracy, the first
group of measures is composed by precision (
Results can be compared between systems but always taking as reference the following baselines: 1. 2.
A system that accepts all answers (returns VALIDATED or SELECTED in 100% of cases) A system that accepts 50% of the answers (random) precision = | det ected _ as _ SELECTED _ or _ VALIDATED | | predicted _ as _ SELECTED _ or _ VALIDATED | recall = | det ected _ as _ SELECTED _ or _ VALIDATED |
| CORRECT _ answers |
F =
2 * recall * precision
recall + precision
(
Then, the second group of measures aims at comparing QA systems performance with the potential gain that
AV systems could add to them. The first of these measures is qa_accuracy (
This measure has an upper bound given by the proportion of questions that have at least one correct answer
(in its corresponding group). This upper bound corresponds to a perfect selection of the correct answers given by
all the QA systems at the main track. The normalization of qa_accuracy with this upper bound is given by
%_best_combination (
Besides the upper bound, results of qa_accuracy can be compared with the following baseline system: a
system that validates 100% of the answers and selects randomly one of them. Thus, this baseline can be seen as
the average proportion of correct answers per question group. We called this baseline random_qa_accuracy (
qa _ accuracy = | answers _ SELECTED _ correctly |
| questions | % _ best _ combination = | answers _ SELECTED _ correctly | | questions _ with _ correct _ answers | *100 random _ qa _ accuracy = 1
∑
| questions | q∈questions
| correct _ answers _ of (q) |
| answers _ of (q) |
(
Therefore, we proposed the use of qa_rej_accuracy (
An estimation of the value obtained in this range is given by estimated_qa_performance (9). This measure considers that the questions accounted by qa_rej_accuracy are answered with the accuracy given by qa_accuracy. Therefore, this measure acknowledges a higher precision of AV systems detecting incorrect answers.
qa _ rej _ accuracy = | questions _ REJECTED _ correctly |
| questions |
qa _ accuracy _ max = qa _ accuracy + qa _ rej _ accuracy
estimated _ qa _ performance = qa _ accuracy + qa _ rej _ accuracy * qa _ accuracy
(
Nine groups (the same number that in the last edition) have participated in five different languages (German, English, Spanish, French and Romanian) with 24 runs. Table 3 shows the participant groups and the number of runs they submitted per language. Again, English and Spanish were the most popular with 8 and 6 runs respectively.
Tables 5-9 in the appendix show the results of Precision, Recall and F measure over correct answers for all participant systems in each language. Results cannot be compared between languages since the number of answers to be validated and the proportion of the correct ones are different for each language (due to the real submission of QA systems). However, they can be compared in each language with two baselines values that are given: the results of a system that always accepts all answers (validates 100% of the answers), and the results of a hypothetical system that validates the 50% of answers. In our opinion, F-measure is an appropriate measure to identify the systems that perform better, measuring their ability to detect the correct answers and only them. However, it is also important to try to obtain some evidences about the improvement that AV systems could provide to QA systems. Tables 10-14 in the appendix show the rankings of systems (merging QA and AV systems) according to estimated_qa_performance calculated only over the subset of questions considered in AVE 2008. The tables contain also the information about the results of QA and AVE systems using the measures qa_accuracy, %_best_combination, qa_rej_accuracy and qa_accuracy_max. The values of qa_accuracy and estimated_qa_performance are the same in QA systems. Again, results cannot be compared between different languages, but they can be compared with the random baselines and with the results of the best QA system (which is marked with a shadow).
The graphic interpretations of these tables are shown in Figures 3-7 in the appendix. In these graphics the value of qa_accuracy is 1 in the perfect selection baseline. This corresponds to a perfect selection of a correct answer (if any) per question and the detection of all the questions with no correct answers (qa_rej_accuracy). However, the value of estimated_qa_performance in this baseline is not 1 because it is assumed that the questions detected in qa_rej_accuracy will be answered with a precision value equal to the qa_accuracy of the perfect selection baseline. This value represents the accuracy of the best combination of the QA systems involved, which is not perfect.
In three languages (German, English and Romanian) there has been at least one AV system performing better than the best QA system. In the languages where the best value of qa_accuracy was not obtained by an AV system, the best QA system outperforms in more than a 50% the following QA systems. If we see an AV system as a multi-stream selector of candidate answers, then AV systems follow a behaviour similar to an ensemble of classifiers. In Machine Learning (ML), an ensemble of classifiers is likely to be more accurate than an individual classifier except in the case of an element of the assemble outperforms in a high percent the rest of the classifiers [1]. Therefore, it seems obvious that there must be more work focused in performing a better selection in this kind of situations.
Regarding the use of the new measure estimated_qa_performance, the rankings are very similar to the ones obtained ranking by qa_accuracy. In fact, there have been only two changes, which are located in the English ranking (see Table 13 in the appendix). Firstly, the system uaic_2 obtains a better performance than ofe according to qa_accuracy (0.24 against 0.19). However, according to estimated_qa_performance, ofe is better than uaic_2 (0.27 against 0.24). This means that uaic_2 is better selecting correct answers. Nevertheless, if we consider the possible gain in performance that might be obtained detecting that all the answers to a question are incorrect and asking for new ones to the QA systems, then ofe is better. Therefore, the system ofe may help to obtain better results in QA than the system uaic_2. Besides, it can be seen how the ranking according to estimated_qa_performance is more similar to the one given by F-measure, which in some way, also considers the precision of a system detecting incorrect answers.
The second change in the rankings involves the QA system dfki081deen, which has a better performance than the AVE system jota_2 according to qa_accuracy. However, according to estimated_qa_performance, the two systems have the same performance. Again, this indicates that AV systems detecting incorrect answers could lead to a better performance in QA.
Thus, it seams that estimated_qa_performance is a better measure for AV systems than qa_accuracy because it takes into account the ability of a system rejecting incorrect answers. Thus, it is given a better estimation of the performance obtained by using AV systems in QA. Furthermore, the rankings are more similar to the ones obtained by using F-measure.
All the participants have reported the use of textual entailment in their systems except two groups (LINA and LIMSI). However, while in the past edition the half of the participants reported the use of automatic hypothesis generation, in this edition only two participants (U. Iasi and U. Alicante) have used it. 6 of the 9 groups (FUH, U. Iasi, INAOE, DFKI, U. Alicante and LIMSI) have also participated in the QA main track, showing that there is a growing interest in using AV in QA participant systems at CLEF.
Table 4 shows the techniques used by AVE participant systems. Following the tendency showed in the past edition, all the systems have reported the use of lexical processing. Moreover, this year there are more groups using syntactic processing, mainly chunking or dependency analysis. Except in Spanish, where none system reported the use of syntactic processing, the system with the best result in each language performed some kind of syntactic processing, mainly by means of dependency parsing. However, the use of semantic analysis has decreased while the use of WordNet has been increased (50% of participants used it). Furthermore, there has been a high increase in the use of Named Entities, with 7 of 9 groups considering them. Therefore, it seems that it can be an important information to be taken into account in AV.
All the participants except two systems (U. Iasi and LINA) have used ML for taking the validation decision, following the tendency of the last edition. Besides, ML was used by the participants with the best score in each language. While lexical similarity was the most common feature used, syntactic similarity was included by the half of the participants. However, semantics features were taken into account by very few participants. Only one participant (FUH) reported the use of a theorem prover this year. Support vector machines (SVM) and decision trees were the most used classifiers. Nevertheless, there are not evidences about the best performance of one or another of these classifiers.
Finally, after a comparison between the resources taken into account and the results obtained, it seems that more resources do not imply better performance. In fact, systems performing semantic analysis have not achieved the best results in their languages.
i s a I .
U
In AVE 2008 there has been the same number of participants of last year (9) in 5 different languages. However, 8 more runs have been sent, showing a growing interest in the task.
Results show that AV systems could improve the performance of current QA systems. This improvement comes when AV systems are used for selecting the final answer from a set of candidate ones. In fact, according to the results, except in the languages where the best QA system outperforms the others QA systems in more than a 50%, there was an AV system with better performance than QA systems.
In this edition new measures have been introduced in order to obtain a more informative estimation of the potential of AV systems in QA performance. These new measures reward the ability of some systems detecting if all the candidate answers to a question are incorrect. These measures have shown to be very useful when two systems have a similar performance according to qa_accuracy. In this situation, the new measure estimated_qa_performance have indicated that AV systems with a better precision detecting incorrect answers would be more useful in QA because more answers could be asked to QA systems when all the candidate answers to a question are incorrect. Then, a correct answer might be found.
The most used technique continues being lexical processing while the use of syntactic analysis has grown. Nevertheless, very few systems have performed semantic analysis. Besides, a high percent of participants have combined different features using ML. Finally, the best systems performed both lexical and syntactic analysis, and they consider NE. This work has been partially supported by the Spanish Ministry of Science and Innovation within the project QEAVis-Catiex (TIN2007-67581-C02-01), the Education Council of the Regional Government of Madrid and the European Social Fund. We are grateful to all the people involved in the organization of the QA track (specially to the coordinators at CELCT, Danilo Giampiccolo and Pamela Forner).
The following tables show the values of Precision, Recall and F measure over correct answers of AVE participant systems in different languages.
DFKI FUH FUH
UA INAOE UA INAOE
ltqa 0,61 0,54 glockner_1 0,39 0,33 glockner_2 0,29 0,25 100% VALIDATED 0,21 0,12 50% VALIDATED 0,19 0,12 Table 5. Precision, Recall and F measure over correct answers for German
0,71 0,49 0,34 1 0,5 The following tables and graphics show the comparison of AV systems performance with QA systems of AVE participant systems in different languages. qa_ accuracy_max
1 0,29 0,22 0,19 0,17 0,17 0,10 0,08