A Pilot Question Answering Task has been activated in the Cross-Language Evaluation Forum 2004 with a twofold objective. In the rfist place, the evaluation of Question Answering systems when they have to answer conjunctive lists, disjunctive lists and questions with temporal restrictions. In the second place, the evaluation of systems' capability to give an accurate self-scoring about the confidence on their answers. In this way, two measures have been designed to be applied on all these different types of questions and to reward systems that give a confidence score with a high correlation with the human assessments. The forty eight runs submitted to the Question Answering Main Track have been taken as a case of study, conrfiming that some systems are able to give a very accurate score and showing how the measures reward this fact.
The QA Pilot Task followed the rules stated in the QA Main Track guidelines except for the source and the target languages, the type and number of questions, and the evaluation measure.
One hundred of questions were posed in Spanish and the corpus used was the EFE Spanish press agency collection of news from 1994 and 1995. The questions of this Pilot Task were distributed throughout the following types: factoid (18), denfiition (2), conjunctive list (20), temporally restricted by date (20), temporally restricted by period (20), and temporally restricted by event (20 nested questions). A little amount of questions had no answer in the document collection (2 NIL factoid questions). As usual, a question was assumed to have no answer when neither human assessors nor participating systems could nfid one.
Ideally, QA systems should tend to give a unique answer for each question but, however, there exist some questions whose answer depends on the context or evolves in time. In these cases, disjunctive lists are obtained, that is, lists of different and correct items representing a disjunction of concepts. The decision of which one of them is the most correct is strongly dependant on the user’s information need, text errors, consistency between different texts (specially in the news domain), etcetera. Therefore, being able to obtain all the possible correct and distinct answers for a question seems to be a desirable feature for open domain QA systems.
For this reason, there was no limit for the number of answers at the Pilot Task, but one answer for each question must be given at least. If systems believed that it was no response to a question in the corpus, they had to answer NIL.
In the conjunctive list type of questions, a determined or undetermined quantity of items is required for conforming an only answer. A conjunctive list is a series of items representing a conjunction of concepts. For the Pilot Task, the goal was to obtain the largest amount of different items within each answer.
Three subtypes of temporally restricted questions have been proposed at the Pilot Task, and three moments with regard to the restriction (before, during or after the temporal restriction): • Restriction by Date, where a precise date contextualises the question, which can refer either to a particular moment, before or after. A date could consist in a day, a month, a year, etcetera, depending on the question. Examples: - T ES ES 0011 ¿Que´ sistema de gobierno teına´ Andorra hasta mayo de 1993? - T ES ES 0014 ¿Quie´n visito´ Toledo el 22 de febrero de 1994? • Restriction by Period. In this case, questions are referred explicitly to a whole period or range of time. A period could be expressed by a pair of dates delimiting it, or by a name accepted as designation of some important periods as, for example, Cuaresma4. Examples: - T ES ES 0086 ¿Quie´n reino´ en Espa~na durante el Siglo de Oro de su literatura? de agosto de 1981? - T ES ES 0037 ¿Quie´n goberno´ en Bolivia entre el 17 de julio de 1980 y el 4 • Event restriction, that implies an embedded or implicit extra question because it is necessary to answer the nested question to determine the temporal restriction. Then, the temporal restriction refers to the moment in which the second event occurred. For example: - T ES ES 0098 ¿Quie´n fue el rey de Be´lgica inmediatamente antes de la - T ES ES 0079 ¿Que´ revolucio´n estudiantil surgio´ en Francia al a~no coronacio´n de Alberto II? siguiente de la Guerra de los Seis ıDa´s? 4Cuaresma is the Spanish word standing for Lent.
The degree of inference necessary to solve the temporal restrictions was not the same for all the questions. In some questions a reference to the temporal restriction could be found in the same document, while in other questions it was necessary to accede to other documents to temporally locate the question. 3
The evaluation measure has been designed in order to reward systems that return as many different and correct answers as possible to each question but, at the same time, punishing incorrect answers. Two reasons motivate the negative adding for the incorrect answers: First, it is assumed that a user of a QA system would prefer a void answer rather than an incorrect one. Systems must validate their answers and must give an accurate confidence score. Second, since there was no limit in the number of answers, systems must calibrate the risk of giving too much incorrect ones. The effect was that no more than three answers per question were given.
In order to evaluate systems’ self-scoring, a mandatory confidence score given by means of a real number ranged between 0 and 1, was requested. 0 meant that the system had no evidence on the correctness of the answer, and 1 meant that the system was totally sure about its correctness.
The evaluation measure has been designed to reward systems that: • answer as many questions as possible, • give as many different right answers to each question as possible, • give the smaller number of wrong answers to each question, • assign higher values of the score to right answers, • assign lower values of the score to wrong answers, • give answer to questions that have less known answers. 3.1
According to the criteria above, the evaluation measure is defined as follows: K(sys) =
1 #questions ·
X i∈questions
P r∈answers(sys,i) max {R(i), answered(sys, i)}
score(r) · eval(r) where R (i) is the total number of known answers to the question i that are correct and distinct; answered(sys,i) is the number of answers given by the system sys for the question i ; score (r) is the condfience score assigned by the system to the answer r ; eval (r) depends on the judgement given by a human assessor.
; K(sys) ∈ R∧K(sys) ∈ [
When K (sys) equals 0 it matches with a system without knowledge that assigns 0 to the condfience score of all their answers. Therefore, K (sys) = 0 is established as a baseline and K -measure gives an idea about the system’s knowledge.
The answers nfiding process, accomplished by human assessors, is strongly determined by the evaluation measure. In the case of K -measure the parameter R(i) requires a knowledge of all the correct and distinct answers contained in the corpus for each question. This fact introduces a very high cost in the pre-assessment process because it is not easy to ensure that, even with a human search, all distinct answers for each question have been found in a very large corpus. One alternative is to relax the pre-assessment process and consider only the set of different answers found by humans or systems along the process. Another alternative is to request only one answer per question and ignore recall. 3.2
A second measure, derived from the K -measure, is proposed to evaluate exercises when just one answer per question is requested (number of questions equals number of answers) or when the achievement of all the possible answers by the system is not outstanding for the exercise. That measure has been called K1 -measure (K -measure for systems giving 1 answer per question) and it is defined as follows:
K1(sys) = r∈answers(sys)
#questions
P
• Accuracy measure, commonly used in all evaluations [
Mean F-measure, used in the QA Track at TREC 2003 [
Mean Reciprocal Rank, used in the QA Track at TREC [
Since the confidence score has been included in the K -measure, a high correlation between
selfscoring and correctness is expected to produce higher values of K. However, it is interesting to
know separately the quality of the scoring given by every system. Hence, it is proposed the use of
the correlation coefficient ( r ) between self-scoring value (in range [
σ assess(sys)score(sys) σ assess(sys) · σ score(sys)
; r(sys) ∈ R ∧ r(sys) ∈ [
When a system assigns a score = 1 to its correct answers and score = 0 to the rest, it obtains a correlation coefficient r = 1, meaning that such a system has a perfect knowledge about the correctness of its response. A correlation coefficient equal to 0 indicates that score and correctness have no correlation. A negative value indicates that there is a certain correlation but in the other direction. 3.5
In the QA 2004 Main Track [
The data from the assessment process for the Pilot Task are shown in Table 2. Only one run
from the University of Alicante (UA) [
The UA system has correctly answered 15% of the questions. The best result corresponds to
factoid questions with a 22.22% of questions with a correct answer. However, in the past edition
of QA at CLEF, this team obtained better results (up to 40% of questions with a correct answer)
[
The UA system never gave more than three answers per question, independently of the type of formulated question. It seems an heuristically established limit for the system that has affected the achievement of good conjunctive and disjunctive list answers.
41 questions got NIL as an answer, with a confidence score of 0 for all them. Unfortunately, these 41 questions had at least one answer in the corpus. On the other hand, the UA system did not identify the 2 posed NIL questions.
Finally, it seems that the UA system did not play with the score value in the best way. The maximum value given for the condfience score was 0.5002 and several questions with only one correct answer in the corpus had associated several different answers with similar confidence score. The K -measure for the UA’s exercise was K = − 0.086 with a correlation coefficient of r = 0.246 between self-scoring and real assessment. 5
Questions whose answer is a conjunctive or a disjunctive list, and questions with temporal restrictions, still remain a challenge for most QA systems. However, these are only a few types of dicffiult questions which QA systems will have to manage in the near future. A specialization and further collaboration among teams could be expected in order to achieve QA systems with higher accuracy and coverage for different types of questions. In fact, the QA Main Track at CLEF shows that different participant systems answer correctly different subsets of questions.
Two measures have been proposed in order to reward systems that give a confidence score with a high correlation with human assessments and, at the same time, return more correct answers and less incorrect ones. The case of study shows that systems are able to give very accurate self-scoring, and that the K and K1 measures reward it. However, systems don’t need to respond all the questions to obtain good results, but to find a good balance between the number of correct answers and the accuracy of their condfience score.
On the one hand, this seems a good way to promote the development of more accurate systems with better answer validation. On the other hand, it is a good way to permit some specialization, to open the possibility of posing new types of questions and, at the same time, to leave the door open for new teams starting to develop their own systems.
This work has been partially supported by the Spanish Ministry of Science and Technology within the following projects: TIC-2002-10597-E Organization of a Competitive Task for QA Systems; TIC-2003-07158-104-01 Answer Retrieval from Digital Documents, R2D2; and TIC-2003-07158C04-02 Multilingual Answer Retrieval Systems and Evaluation, SyEMBRA.
We are grateful to Julio Gonzalo, from UNED-NLP Group, and Alessandro Vallin, from ITCIrst (Italy), for their contributions to this work. In addition, we would like to thank the University of Alicante team for their effort in participating in the Pilot Task. MITRE,