The general aim of the third CLEF Multilingual Question Answering Track was to set up a common and replicable evaluation framework to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages. Nine target languages and ten source languages were exploited to enact 8 monolingual and 73 cross-language tasks. Twenty-four groups participated in the exercise.Overall results showed a general increase in performance in comparison to last year. The best performing monolingual system irrespective of target language answered 64.5% of the questions correctly (in the monolingual Portuguese task), while the average of the best performances for each target language was 42.6%. The cross-language step instead entailed a considerable drop in performance. In addition to accuracy, the organisers also measured the relation between the correctness of an answer and a system's stated confidence in it, showing that the best systems did not always provide the most reliable confidence score.
Question answering
The CLEF QA evaluation campaign conducted in 2005 1 was the result of the experience acquired during the two previous campaigns and of the proposals suggested in last year’s workshop in order to make the track more challenging and realistic.
At a first look one realizes that over the years the series of QA competitions at CLEF has registered a steady increment in the number of participants and languages involved, which is particularly encouraging as multilinguality is one of the main characteristic of these exercises. In fact, in the first campaign, which took place in 2003, eight groups from Europe and North America participated in nine tasks, three monolingual -Dutch, Italian and Spanish- and five bilingual, where questions were formulated in five source languages -Dutch, French, German, Italian- and answer were searched in an English corpus collection. In 2004 eighteen groups took part to the competition, submitting 48 runs. Nine source languages -Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese and Spanish- and 7 target languages -all the source languages but Bulgarian and Finnish, which had no corpus available- were considered in the task. In 2005 the number of participants rose to twenty-four, 67 runs were submitted, and 10 source languages -the same as those used in the previous year plus Indonesian- and 9 source languages -the same used as sources, except Indonesian which had no corpus available- were exploit in 8 monolingual and seventy-three cross-language tasks. Moreover, some innovation was introduced concerning the type of questions proposed in the exercise and the metrics used in the evaluation. Despite the expectation of some the organisers for more radical innovations were partially disappointed, this edition of QA@CLEF was altogether successful and can be considered a good starting point for next campaigns.
A preliminary overview of the 2005 QA track is presented in this paper, explaining more in details the procedure followed to build the test sets and providing a preliminary analysis of the results. 2
The tasks proposed in 2005 QA campaign were characterised by a basic continuity with what
had been done in 2004. In fact, to the demand for more radical innovation a more conservative
approach was preferred, as most organizers opted to further investigate the procedures
consolidated in the last two campaigns before moving to the next stage. As a matter of fact, the task
remained basically the same as that proposed in 2005, although some minor changes were actually
introduced, i.e. a new type of questions, and two new measures, namely K1 measure and r value.
Both question type and measure were borrowed from the Spanish pilot task proposed at CLEF
2004 [
Ten source languages -Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese, Spanish and, as an experiment, Indonesian- and 9 target languages -all the source languages except Indonesian- were considered at the 2005 CLEF QA track. Eighty-one tasks were setup, 8 monolingual -Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese, Spanishand 73 bilingual. In this way, all the possible combinations between source and target languages were exploited, but for two exceptions: Indonesian, being included in a cross-language QA competition, was used only as a source in the Indonesian-English task, meanwhile the monolingual English task was discarded as it has been abundantly tested in TREC campaigns, according to the decision taken in the previous competition.
1For more information about QA@CLEF campaigns visit http://clef-qa.itc.it. 2.2
As in the previous campaign, 200 questions were provided as an input in all tasks, and exact answer-strings were required as an output. The target corpora in all the languages were collections of newspapers and news agencies’ articles, whose texts had been SGML tagged. Each document had a unique identifier (docid) that systems had to return together with the answer, in order to support it. The corpora, released by ELRA/ELDA, were large, unstructured, opendomain text collections.
Although the numbers of questions was the same as last year, there were changes regarding the type of questions and their distribution. Meanwhile How- and Object questions were not included in 2005 task, since they were considered particularly problematic in the evaluation phase, a new subtype of factoid questions was introduced, called temporally restricted questions, which constrained by either an event -e.g. Who was Uganda’s President during Rwanda’s war? -, a date -e.g. Which Formula 1 team won the Hungarian Grand Prix in 2004? - or a period of time-e.g. Who was the President of the European Commission from 1985 to 1995?. Up to 30 temporally restricted questions could be included in each task.
As far as the three major type of questions -Factoids (F), Definition (D) and NIL (N)- are concerned, the breakdown, both suggested and real, is shown in Table 1.
In order to increase the overlap between the test sets of different target languages, this year a certain number of topics were assigned to each language and a particular effort was made in order to get general questions, which could easily find an answer also in the other corpora. As a result, no question was actually answered in all 9 languages, but the inter-language partial overlap was increased anyway, as 21 questions appeared in 5 target languages, 66 questions in 4 target languages, 159 questions in 3 target languages and 265 in 2 target languages. 3
The procedure for question generation was the same as that adopted in the previous campaigns. Nine groups were involved in the generation, translation and manual verification of the questions: the Bulgarian Academy of Science, Sofia, Bulgaria (CLPP) was in charge for Bulgarian; the Deutsches Forschungszentrum fr Knstliche Intelligenz Saarbrcken, Germany, (DFKI) for German; the Evaluations and Language Resources Distribution Agency Paris, France(ELRA/ELDA) for French; the Center for the Evaluation of Language and Communication Technologies Trento, Italy (CELCT) for Italian; Linguateca, Oslo (Norway), Braga, Lisbon& Porto for Portuguese; the Universidad Nacional de Educacin a Distancia Madrid, Spain (UNED) for Spanish, the University of Amsterdam, The Netherland for Dutch; the University of Helsinki, Finland for Finnish; the University of Limerick, Ireland for English; and the Department of Computer Science of University of Indonesia joined the activity translating 200 English questions into Indonesian, in order to set up the cross-language Indonesian- English task. 3.1
The questions in the test sets addressed large open domain corpora, mostly represented by the same comparable document collections used last year: NRC Handelsblad (years 1994 and 1995) and Algemeen Dagblad (1994 and 1995) for Dutch; Los Angeles Times (1994) and Glasgow Herald (1995) for English; Le Monde (1994) and SDA French (1994 and 1995) for French; Frankfurter Rundschau (1994), Der Spiegel (1994 and 1995) and SDA German (1994 and 1995) for German; La Stampa (1994) and SDA Italian (1994 and 1995) for Italian; PUBLICO(1994 and 1995) for Portuguese and EFE (1994 and 1995) for Spanish. This year two new corpora were added, Aamulehti (1994-1995) for Finnish, and Sega and Standard for Bulgarian. Unfortunately the Bulgarian corpora dated back to 2002, so that the information contained in it was difficulty comparable with the that of the other corpora.
According to the consolidate procedure, 100 questions were produced in each target language (except Indonesian), manually searching relevant documents for at least one answer. The questions were then translated into English, so that could be understood and reused by all the other groups. Answers were not translated this year, as it was a time-consuming and basically useless activity. The co-ordinators attempted to balance the difficulty the test sets according to the different answer types of the questions already used in the previous campaigns, i.e. TIME , MEASURE, PERSON, ORGANISATION, LOCATION, and OTHER. HOW and OBJECT questions were however inserted in this exercise because generate ambiguous responses, which are quite difficult to be assessed.
As said, up to thirty temporally restricted questions were allowed, and were themselves classified
according to the above mentioned types, ie, time, measure, etc Particular care was taken this year
in choosing 10% of NIL questions. In fact, some organizers realised that in the previous campaigns
NIL questions were quite easily identified by systems, as they were manually generated searching
for named entities which were not in the corpora. On the contrary, this time NIL questions were
selected randomly from those that seemed to have no answer in the document collections, and were
double-checked Once the 900 questions were formulated in the original source languages, translated
into English and collected in a common XML format, native speakers of each source language,
with a good command of English were recruited to translate the English version of all the other
questions trying to adhere as much as possible to the original. This process was as challenging as
any translation job can be, since many cultural discrepancies and misunderstanding easily creep
in. Anyway, as was already pointed out in 2004 “he fact that manual translation captured some
of the cross-cultural as well as cross-language problems is good since QA systems are designed to
work in the real world” [
Once all the 900 questions were translated into ten source languages -the Indonesian group translated only the final 200 English question-, 100 additional questions for each target language were selected from the other source languages, so that at the end each language had 200 questions. The added questions were manually verified and searched for answers in the corpus of the respective language. The collection was called Multi9-05, and was presented in the same XML format adopted in 2004. The entire collection is made up of 205 definition questions and 695 factoid , which are quite well balanced according to their types, being divided as follows: 110 MEASURE; 154 PERSON; 136 LOCATION; 103 ORGANISATION, 107 OTHER, 85 TIME. The total number of temporally restricted questions is 149.
Although this new kind of questions appeared to be quite interesting, no comprehensive analysis of the results in this group of questions have been made so far, and the experiment requires to be furtherly investigated. The Multi9-05 can now be added to the previous campaigns’ collections, which already represent a useful reusable benchmark resource. The proposal to integrate the missing answers with the correct results provided by the systems during the exercise has remained unanswered. The positive trend in terms of participation registered in 2004 was confirmed in the last campaign. From the original 8 groups who participated in 2003 QA task, submitting a total of 19 runs in 9 tasks, the number of competitors has raised to twenty-four, which represent an increase of 33% respect to last year, when 18 groups took part in the exercise. The total of submitted runs was sixty-seven. All the participants in 2005 competition were from Europe, with the exception of group from University of Indonesia which tried the experimental cross-language task IndonesianEnglish.
As shown in table 2, the systems were tested only against 22 of the 81 activated tasks. Monolingual English was discarded this year, as it was in last competition, because the task has been sufficiently investigated in TREC campaigns, and as far as Indonesian is concerned, only the task with English as a target was set up. The non-activated tasks are represented by a blank cell in table 2.
All nine monolingual tasks (in bold in the table) were tested at least by 1 systems, being French (FR) and Spanish (ES) the most chosen languages. As far as bilingual tasks are concerned, 15 participants altogether chose to test their systems in a cross-language task. English was as usual the most frequent target language, being involved in 8 cross-lingual tasks completed by 9 participants; Spanish was chosen as a target in a cross-language task by three groups, and so was French, meanwhile only one system tried a cross-language task with Portuguese (PT) as a target, i.e. EN-PT. All the other languages was not considered as a target in bilingual tasks. 4
The procedure adopted to assess the systems’s outputs was practically the same as the last year. Participants were allowed to submit just one response per question and up to two runs per task, which were judged by human assessors according to correctness and exactness -where correctness expresses whether the answer is clear and pertinent, while exactness evaluates whether the information is either too much or too less. Like in 2004 only exact answers were allowed, and the responses were judged as Right, Wrong, ineXact or Unsupported (when the answer-string contained a correct answer but the returned docid did not support it). As a partial analysis of the inter-tagger agreement has shown, the exactness is still a major problem in evaluation, as most disagreement between judges concerns this parameter.
Definition questions, which were introduced last years, and were considered particularly difficult also because they could raise problems in assessing their exactness, generally scored quite well, proving that as they are now they are less challenging than one thought. In fact, the answer often consists in the solution of an acronym, when they concern organisation, or is expressed as an apposition of the proper name, when persons are concerned. As said, the introduction of Temporal Restricted Questions hasn’t been properly analysed yet. As a general remark, it must be said that their number in the test sets was probably too small to provide significative data on their impact on systems’ results. Furthermore, some of them were ”false temporally restricted”, as said above, and a system could retrieve an answer without even considering the temporal restriction. The main measure used for the evaluation was the accuracy, i.e the fraction of right answers. The answers were returned unranked (i.e. in the same order as in the test set), but a confidence value, that could range between 0 and 1, could be added to each string and be considered to calculate an additional Confidence-weighted Score (CWS)[NOTA OVERVIEW 2004]. This year two additional evaluation measures, i.e. the K1 value and r coefficient, borrowed by [VEDERE NOTA], were experimentally introduced, in order to find a comprehensive measure which takes into account both accuracy and confidence. Anyway, being confidence an additional and optional value, only some systems could be assigned the CWS, and consequently the K1 and r coefficient; therefore an analysis based on this measures is not very significant at the moment. In comparison to last year, the performances of the systems in this campaign show a general improvement, although a significant variation remains among target languages. In fact, in 2004 the best performing monolingual system irrespective of target language (henceforth ’best overall’) answered 45.5% of the questions correctly, while the average of the best performances for each target language (henceforth ’average of best’) was 32.1%. In 2005 the best overall and average of best figures were 64.5% (in the monolingual Portuguese task)-representing an increase of 19 pointand 42.6% respectively. As far as bilingual tasks are concerned, as usual the cross-lingual step generically entailed a considerable drop in performance.
In the following nine sections the results of the runs for each target language are thoroughly discussed. For each target language two kinds of results are given, summarized in two tables. One presents the overall performance, giving the number of right (R), wrong (W), inexact (X), and unsupported (U) answers; the accuracy, in general and on Factoids (F), Definitions (D) and Temporal (T); Precision (P), Recall (R) and F measure for NIL questions; and finally CWS, K1 and r of each run. The second table shows the accuracy of the systems with respect to the answer types, i.e Definition, sub-classified as Organisation (Or) and Person (Pe), and Factoid and Temporally Restricted, sub-classified as location (Lo), measure (Me), organisation (Or), other (Ot), person (Pe) and time (Ti). Below each answer type, the number of posed questions of that type is shown in square brackets.
The last row of the second table shows a virtual run, called Combination, in which the classification ”right answer” is assigned to a question if any of the participating systems found it. The objective of this combination run is to show the potential achievement if one merged all answers and considered the set of answers right, provided one answer were right. 4.1
For the first time Bulgarian was addressed as a target language at CLEF 2005. Thus, no comparison can be made with previous results from the same task, but some comments on the present ones are in order.
This year two groups participated in monolingual evaluation tasks with Bulgarian as a target language: IRST, Trento and BTB, LML, IPP, Sofia. Two runs were submitted for BulgarianBulgarian. Both results are below the desired figures (27.50 % and 18.50 % correct answers), but they outperform their own results from the last year where Bulgarian was used as a source language and English - as a target. Obviously, the Inexact and Unsupported value metrics do not have substantial impact over the final estimations. It seems that as a group the definition questions are the best assessed type (40 % and 42 %). Then come the factoid ones. The worst performance goes to the temporally restricted questions. Then, NIL questions exhibit better recall than precision. It might be explained by the fact that the systems return NIL when they are not sure in the answer. Only IRST group results provide a confidence weighted score.
It is interesting to discuss the results according to the answer types. Recall that definitions did well as a group. However, when divided further into Organization and Person types, it turns out that the Organization type was better handled by one of the participants, while the Person type was better handled by the other. From non-temporally restricted factoids Organizations and Other have been the most problematic types. From temporally restricted factoids Measure was unrecognized, but the number of these questions was not so high anyway. Person subtype was not detected as well, which is a bit surprising fact.
Most of the problems concerning assessors’ agreement were in one ‘green area’: between Wrong and Inexact. Recall that it was also a problem at CLEF 2004. Here we do not have in mind easy cases, such as: What is FARC? The system answered ‘Columbia’ instead of answering ‘Revolutionary Armed Forces of Colombia’ or at least ‘Revolutionary Armed Forces’. We have in mind subtle cases as follows: (1) too general answers, but still correct (Q: What is ESA? A: ‘agency’ instead of ‘(European) space agency’), and (2) partial answers, but still correct (Q: Who was proclaimed patron of Europe by the Pope on 31 December 1980? A: ‘St. Cyril’ instead of ‘St. Cyril and Methodius’). Under the former type we consider answers that are given only some ‘top ontological’ categorization. Under the latter we consider cases, in which part of the answer is presented, but the other part is missing. Very often it concerns questions of measure (Q: How much did Greenpeace earn in 1999? A: ‘134’ instead of ‘$ 134 mln.’).
This year for the first time Bulgarian was tested as a target language at the CLEF track. Two groups made runs on Bulgarian-Bulgarian task. The results are promising in spite of being lower than the half of the correctly recognized answers. So, we consider this a good start. The two extraction systems will be improved on the evaluation feedback. They need to handle better local contexts as well as to try to handle non-local support information.
In the evaluation phase the most problematic still seems to be the definition of the Inexact answer. Inexactness exhibit gradability. In this respect it either should be defined in a more elaborate way (concerning generality and partiality, and per answer type), or there should be introduced a more objective system of final evaluation. Our suggestion is that inexact answers have to contain the head noun of the correct answer. The degree of inexactness depends on the recognized modifiers of the head. If the correct answer is a coordination, then the inexactness is determined also by presence of each coordinates. 4.2
There were three research groups that took part in this year’s evaluation for the QA-track having German as target language. The number of total system runs submitted by the participants was six, with three runs for every of the two source languages: German and English. The results of evaluation for every participant group are shown in the tables below.
For the monolingual German runs the results for definition and temporal questions are better then those for factoid questions. As table 6 shows, within the definition questions, results are better for ORGANIZATION as for PERSON answer types. For factoid questions, best results were attained for TIME, PERSON, LOCATION and ORGANIZATION answer types, in order of their mention, while for temporal questions, results were equally good for PERSON, MEASURE and ORGANIZATION answer types.
For the cross-lingual English-German runs, best results were registered for definition questions, followed by factoid questions, and with poor results by temporal questions. Again, best results for definition questions were for ORGANIZATION answer types and for factoid questions the order of accuracy remains unchanged with respect to the monolingual runs.
Results computed for a ”virtual” system, through aggregation of all existing results, show an increase of almost 35% for the monolingual task, and 20% for the cross-lingual task, in accuracy over the best results achieved by participating systems. Overall, twelve cross-lingual runs with English as a target were submitted. The results are shown in Tables 7 and 8.
The best scoring system overall was DFKI DEEN Run 1 with 25.5%. This score includes all three types of question, i.e. Factoid, Definition and Temporal. For Factoid questions alone, the highest scoring was DLTG FREN Run 1 (20.66%). For Definition questions alone, the highest scoring was DFKI DEEN Run 1 (50%). For Temporal question alone, three systems had an equal top score, DLTG FREN Run 2, IRST BGEN Run 1 and LIRE FREN Run 2 (all 20.69%). DFKI’s main advantage over other systems was their ability to answer definition questions - their score of 50% was well ahead of the next best score of 38% achieved by IRST ITEN Run 1 and IRST ITEN Run 2.
Last year, results were only single-judged with all answers to a given question being judged by one assessor using an adapted version of the NIST software. Four assessors each did 50 questions, there being 200 in all. Any issues found by assessors were then discussed and resolved at a series of plenary sessions. This year, all results were double-judged using the same software and with six assessors: Two independently judged questions 1-66, two judged 67-133 and two judged 134-200, there being 200 questions in total once again. The judgements were then automatically compared using the diff utility. A list of variant judgements was then prepared and presented to each pair of assessors for resolution.
The degree of agreement between assessors was found to range between 91.41% and 94.90%, computed as follows: For questions 1-66 there were 66 questions and 12 runs, 792 judgements in all. 68 differences were recorded, so the level of agreement is (792-68)/792, i.e. 91.41%. For questions 67-133, there were 804 jusgements with 69 differences recorded, i.e. 91.42% agreement. Finally, for questions 134-200 there were again 804 judgements with 41 differences recorded, i.e. 94.90% agreement.
In almost all cases, points of disagreement could be tracked down to problematic questions which either had no clear answer (but several vague ones) or which had several possible answers depending on the interpretation of the question.
Definition questions were once again included this year but a method of assessing them was not decided upon prior to the competition. In other words, participants did not really know what sort of system to build for definitions and we as assessors were unsure how to go about judging the answers. In consequence we used the same approach as last year: If an answer contained information relevant to the question and also contained no irrelevant information, it was judged R if supported, and U otherwise. If both relevant and irrelevant information was present it was judged X. Finally, if no relevant information was present, the answer was judged W. Two main types of system were used by participants, those which attempted to return an exact factoid-style answer to a question, and those which returned one or more text passages from documents in the collection. Generally, the former type of system is attempting a harder task because it is returning more concise information than is the latter type of system. For this reason, our evaluation method is designed to favour the former type. This was an arbitrary decision, taken in the absence of further guidelines. Our judgements are as accurate as we can make them within our own criteria but we should point out that different criteria could produce different results. Concerning the overall assessment process, we had no procedural difficulties as the format of the data was the same as last year and Michael Mulcahy in particular had already devoted a great deal of time to the adaptation of the software and the development of additional utilities in 2004. Also, most of the assessors were familiar both with the software and with the judgement criteria. We arrived at two conclusions during the assessment process. Firstly, the main points of difference between assessors in judging answers can be traced back to intrinsic problems associated with certain questions. In other words we need to devote more time to the problem of generating good questions which on the one hand are of the kind which potential users of our systems might pose, and on the other hand have clear answers. We should arrive at objective tests which can be applied to a candidate question and its answers to enable its suitability for use in CLEF to be assessed. Secondly, the situation in respect of definition questions was not ideal for either participants or assessors. This could affect our results for the EN target language as well as their relationship to the results for other target languages. 4.4
Seven groups submitted 18 runs having Spanish as target language: 13 of them had also Spanish as source language, 2 had Italian and 3 had English. Notice that is the first time that bilingual runs were submitted.
Table 9 shows the number of correct answers, CWS, K1 and correlation coefficient for all systems. Table 10 shows the number of correct answers for each type of question. Table 11 shows the number of correct answers for each type of temporal restriction. Table 12 shows the evolution of the most important criteria in the systems performance for the last three years.
run inao051esesM tova051esesM inao052esesM tova052esesM upv051esesM alia051esesM aliv051esesM alia052esesM talp051esesM talp052esesM mira051esesM mira052esesM upv052esesM upv051enesC mira052enesC mira051enesC mira051itesC mira052itesC
The virtual combination run was able to answer correctly 73.50% of the questions. The best performing system achieved an overall accuracy of 42% but it only gave a right answer for the 56% of the questions correctly answered by the combination run. Thus, we can expect improvements of the systems in a short term.
As shown in Table 10, systems generally behaved better with questions about definitions, locaIn the factoids without temporal restrictions, the best performing system answered correctly 29.66% of the questions, a very similar accuracy comparing with the results in 2004 (see Table 12).
Concerning questions with temporal restriction, the systems with the best behavior answered correctly 34.38% of the questions, a similar result comparing with overall accuracy. run
As shown in Table 10, when considering the question type, the accuracy scores present small differences. Nevertheless, when the restriction type (date, event and period) is taken into account, the differences are more important (see Table 11). It is worth mentioning that for questions restricted by event, the virtual combination run clearly outperforms individual systems separately (low overlapping on correct answers).
In definition questions the best performing system obtained 80% of accuracy. The improvement is remarkable considering that in the 2004 track the best systems answered correctly 70% of the questions.
Regarding NIL questions, the best systems achieved a recall of 0.80. F-measure improvements are also remarkable, with an increase of about 26% with respect to last year (0.30 in 2004 vs. 0.38 in 2005).
Systems have also clearly improved their confidence self-score. While in 2004 the system with
higher correlation coeficience (r) reached 0.17 [
In order to analyze the interannotator agreement, we have randomly selected 4 out of 18 runs which have been judged by two assessor with different levels of expertise. Most of the differences among assessors can be found when judging an answer as Right or as ineXact. In many cases, an assessor without experience assess as Right an answer that an experienced assessor would judge as ineXact. Table 14 shows the maximun variation of correct answers for these four runs (average = ± 2.9%).
Finally we can conclude that both the improvement in systems’ self-evaluation, the scores obtained by the participanting systems (73.50% in combination, 42% individually), and the systems’ evolution during the last three years, let us expect a significant improvement in Spanish question answering technologies in the near future. The year 2005 was the first year when Finnish existed as a target language. Only one group submitted runs for this task, and both of the runs were monolingual. The artificial combination run presented in table 16 shows that the upper bound on the performance of a system that would merge the results of the existing runs and somehow select the right answers from the combined pool of candidate answers is 26.50%. This is by far the lowest monolingual combination run score among the participating languages. The next one is bulgarian with a combination score of 36.00 % (see Table 4). However, when we calculate the average score for the monolingual runs of each target language, we can see that Finnish is not very far behind, for the average accuracy of the Finnish runs is 21.00%, that of the Bulgarian ones is 23.00%, that of the Italian ones is 24,08%, that of the French ones is 25,20%, and so on. The confidence scores that the systems having Finnish as target assign to the answers only very faintly reflect the assessor’s opinion on the correctness of the answer, as can be seen from the correlation coefficient between the system’s score and correctness (r) in Table 15.
The evaluation of the Finnish answers was not straightforward because the evaluation
guidelines [
Seven research groups took part in evaluation tasks using French as target language: Synapse D´eveloppement (France), CEA-LIST/LIC2M (France), LIMSI-LIR (France), Universit´e de Nantes, LINA (France), Helsinki University (Finland), Universitat Polit`ecnica de Val`encia, UPV (Spain) and TOVA, a joint system between UPV and the Instituto Nacional de Astrof´ısica O´ ptica y Electro´nica (Mexico). All participating groups took part in the monolingual task: four groups submitted one run and three groups submitted two runs FR-FR. Only Synapse D´eveloppement took part in the bilingual tasks. This group submitted three runs, one run per source language: Italian, English and Portuguese. Table 17 shows the results of the assessment of the thirteen submitted runs. This year, many groups participated in the Question Answering tasks with French as a target. It appears that the number of participants for the French task has increased significantly: seven this year as opposed to one last year. The best results were obtained by Synapse D´eveloppement for one of the monolingual runs (syna051frfr). This group ranked 2nd and 3rd in the two English-French and Portuguese-French runs which is better than all the other monolingual French runs. The two monolingual runs by the spanish TOVA group reached the 4th and 5th positions.
The correct answers given for all the runs are presented in table 18, sorted by type of answer (location, measure, organization, etc.). The results show the limits of the system developed by Synapse D´eveloppement, which obviously lie in factoid-other (9/20), factoid-measure (10/20) and factoid-time (11/20), whereas results are much better for definition and factoid-person questions. The aim of the virtual run called combination is to provide an upperbound on the possible performance of a system that would merge the existing runs and somehow select the right answers from the combined pool of candidate answers. The best run (syna051frfr) is able to supply 76.19% of the correct answers of combination. This ratio could be enhanced if results for factoid-measure or factoid-time questions were better.
2The question requires the head noun of the answer to be in the nominative case - der K¨onig - instead of the dative case - dem K¨onig.
3The question requires the answer to be in the nominative case - Mika Kaurism¨aki - instead of the genetive case - Mika Kaurism¨aen. * Results calculated over 197 questions. The main problem encountered during the assessment of answers was related to the temporally restricted factoid questions. This year and for the first time in CLEF this kind of questions was included in the test sets. We thought that the generation of this kind of questions would be relatively easy, but did not foresee that the assessment on those questions would be so difficult.
In fact, many temporally restricted factoid questions have not been built properly as there was no logic of restriction at all. The question ”In which famous capital was the Eiffel Tower built in 1889?” is a good example. Here, ”in 1889” is a redundant information rather than a temporally restriction and will be ignored by the system: the correct answer returned with a document associating the Eiffel Tower to Paris will be a right answer even if it does not specify that the Eiffel Tower was built in 1889.
Therefore, from the beginning of the assessment phase on, many questions arise such as ”Should the date be included in the document joined to the answer?”, ”Should all the items included in the question be found in the document in order to consider the answer as correct?”. Now we know how to handle those temporally restricted factoid questions and such problems should not occur next year.
This year, as far as French language is concerned, the best system obtained very good results : 128 correct answers out of 200. In all the QA@CLEF tracks, these are the best results ever obtained for the French used as target language. Moreover, we could see a growing interest in Question Answering from the European research community: the QA@CLEF-2005 attracted more participants in evaluation tasks using French as target language than the previous editions. In addition, the benchmark resources built for these evaluations contributed to the development and the improvment of systems, and could be used again as training resources in the next edition. 4.7
Three groups participated in the Italian monolingual task, and no one in the other bilingual tasks with Italian as target. A total of six runs were submitted, two each research group: ITC-Irst, the Universidad Polit´ecnica de Valencia (UPV) and a joint experiment by UPV and the Mexican INAOE (Instituto Nacional de Astrof´ısica, O´ ptica y Electro´nica). As table 19 shows the best system (the one developed by UPV and INAOE) answered correctly to 27.5% of the questions, and the other two systems achieved similar results.
In 2004 two teams had participated in the Italian monolingual task, submitting a total of 3 runs. The best performer had an overall accuracy of 28%, while the average performance was 25.1%. In 2005 the task itself attracted more research groups, and though the best system was approximately as good as the one of last year, the average overall accuaracy is slightly worse (i.e. 24%), which probably means that the Italian monolingual test set was more challenging in 2005. As far as the types of questions are concerned, it is interesting to notice that definitional questions proved to be easier than factoids. Between 38 and 50% of definitional got a correct answer, while temporally restricted questions were tougher for the three participating systems. Eleven questions (no. 3, 20, 30, 60, 65, 84, 85, 107, 113, 116 and 124) received a correct answer in all the six submitted runs, and five among them are definition questions referred to a person. This suggests that this type of questions have often a straightforward answer that appears between brackets or in appositive form within the text. Table 20 shows that the factoids with location, person and time as answer type were the easiest for systems, and if the three systems had worked together, they could have achieved an overall accuracy of 46.5%, which encourages research groups to share tools and resources in the future.
run tova052ititM tova051ititM upv 051ititM upv 052ititM irst051ititM irst052ititM combination
The manual assessment procedure was the same as it was in 2004. Two assessors had a brief training session (based on the 2004 submissions) that aimed at making them familiar with the evaluation tool interface and at solving preliminary doubts. Both assessors judged all the six runs and then the answers with different judgments were double-checked and received a third, final judgment. Table 21 gives the number of different judgments per run and the inter-assessor kappa coefficient, which is quite high (average value is 0.874). run tova052itit tova051itit upv 051itit upv 052itit irst051itit irst052itit
A total of 70 disagreement cases were registered, most of them involved the judgment couples RX (11 cases), R-W (13 cases), U-W (10 cases) and above all X-W (31 cases). Clearly, the evaluation guidelines did not deal extensively with answer exactness, so assessors had some difficulties in deciding which portion of an answer-string was acceptable and which not. In most of the cases (i.e. 26) where an assessor assigned X and the other W, the third and final judgment was W. This year two teams that took part in the QA@CLEF track used Dutch as their target language: the University of Amsterdam and the University of Groningen. In total, three runs were submitted, all using Dutch as the source language. All runs were assessed by two assessors, with very interassessor agreement (0.950 for gron051nlnl, and 0.976 for uams051nlnl and uams052nlnl). The results of the evaluation for all runs are provided in Tables 22 and 23. run gron051nlnlM uams051nlnlM uams052nlnlM #
r
When scored in terms of the percentage of correct (i.e, correct and exact and supported) answers, the run labeled gron051nlnl (submitted by the University of Groningen) clearly outperforms the two runs submitted by the University of Amsterdam: 49.50% vs. 44% and 44%. When compared to the correct answers in the Groningen run, many of the inexact answers in the Amsterdam runs are caused by incorrect definitions; here’s an example.
0094 NLNL Wat is Eyal? gron051nlnl: militante joodse groep uams051nlnl: leider van de extreem-rechtse groep This observation is confirmed if we take a closer look. In the 200 questions, six initial words occur more than ten times: Wie (Who), Wat (What), Hoe (How), Welke (Which), Waar (Where) and In (In). The performance of the questions with four of the six initial words is similar for the three runs. For Wat, Groningen obtains 67% right and Amsterdam 39%. This difference is mainly caused by the problem with the definition answers just mentioned. For Hoe, Groningen obtains 63% and Amsterdam 36%. Seven of the eight Hoe questions for which only Groningen found the answer, were of the format Hoe heet DEFINITION? (What is the name of DEFINITION?).
All in all, the Groningen run performs noticeably better than the Amsterdam runs in terms of precision—this is clear from the differences in answers labeled X (inexact): only 18 for Groningen, and as many as 28 and 29 for Amsterdam.
If we drill down a bit further, and consider the detailed results in Table 23, we see that Groningen outperforms Amsterdam on Organisations in the Definitions category, and on Other questions in the Factoid category; Amsterdam is slightly better in Person definitions. On other categories, the differences are very minor or non-existent. There is, however, a noticeable difference in performance on NIL questions, with Amsterdam achieving far higher F-scores than Groningen.
To conclude, let’s adopt a somewhat alternative perspective. The differences between the
Groningen run and the Amsterdam are mainly in the number of inexact answers; in terms of the
number of unsupported or wrong answers the differences are negligible. Put differently, in terms
of the number of answers that are “helpful” [
run In 2005 there were five runs with Portuguese as target, submitted by three different research teams: In addition to the two participants from last year, SINTEF with the Esfinge system and the University of E´vora, we had a newcomer from industry, Priberam, a Portuguese company specialized in NLP products. Although a collection of Brazilian Portuguese news was added to the CLEF collection, no Brazilian participants turned up as yet for CLEF. run prib051ptptM ptue051ptptM esfg051ptptM esfg052ptptM esfg051enptC # #
Another relevant remark is that definitions do not seem to be more difficult on average than factoid questions, as was the case last year. We believe, however, that this is due to a considerable simplification of precisely what “definition questions” are, where they boil down to mainly ask for a person’s profession or title. We did some further analysis of the results in order to have other measures of confidence in the systems, which are displayed in table 26. We looked specifically at (i) the cases where no answer was given (null answer), which keep the user in a state of ignorance, no matter the system was right in providing the null answer or wrong because it could not find it; (ii) the cases where any user could at once see the answer was rubbish (rubbish); and (iii) the cases where the wrong answers could be misleading (dangerous). Of course it depends on the ignorance of the questioner, and we were very conservative in imagining total ignorance. Probably most of the “dangerous” questions would at once be spotted as system’s mistakes by an ordinary user – or at least arise some suspicion.
The results show that the PTUE system is both the most reliable (less non-NIL wrong answers) and the most conservative system (most empty answers), the more “dangerous” one being Esfinge. 5
This paper presented the multilingual Question Answering evaluation campaign organized at CLEF 2005. QA@CLEF considerably increased both in number of participants -we are now closer to the Question Answering track at TREC- and also in the number of languages involved. It is also relevant that this year we were able to activate a task with Bulgarian as a target, a language of a new EU member country. A pilot cross-language task with Indonesian as source and English as target has been also activated.
Being the organization of the task at its third year, is now well tested, although involving nine different Institutions of as many different countries, and showed to be able to support the high number of exchanges required by the organization of the task. This is particularly significative considering that all the organizations involved in QA@CLEF guarantee their support on a voluntary basis.
The increased number of participants allowed to carried out a number of interesting comparisons among systems participating at the same task (this was one of the drawback of the 2004 campaign). In addition, it is worth to mention that Question Answering techniques for European languages, being mainly based on NLP tools and resources for the respective languages, demand for better tool and resources. In a cross-language perspective the integration of such resources is also crucial.
Finally, having (at least partially) achieved its goal to promote Question Answering for European languages, there is now a quite large scientific community in Europe on Question Answering, QA@CLEF is now ready to propose its own view on QA, designing a roadmap for the next years multilingual QA systems.