We describe WiQA 2006, a pilot task aimed at studying question answering using Wikipedia. Going beyond traditional factoid questions, the task considered at WiQA 2006 was to return-given an source page from Wikipedia-to identify snippets from other Wikipedia pages, possibly in languages different from the language of the source page, that add new and important information to the source page, and that do so without repetition. A total of 7 teams took part, submitting 20 runs. Our main findings are twofold: (i) while challenging, the tasks considered at WiQA are do-able as participants achieved impressive scores as measured in terms of yield, mean reciprocal rank, and precision, (ii) on the bilingual task, substantially higher scores were achieved than on the monolingual tasks.
CLEF 2006 featured a pilot on Question Anwering Using Wikipedia, or WiQA [
In this overview we first provide a description of the tasks considered and of the evaluation and assessment procedures (Section 2). After that we describe the runs submitted by the participants (Section 3 and detail the results (Section 4). We end with some preliminary conclusions (Section 5).
The WiQA 2006 task deals with access to Wikipedia’s content, where access is considered both from a point of view and from an author point of view. 2.1
As our user model we take the following scenario: a reader or author of a given Wikipedia article (the source page) is interested in collecting information about the topic of the page that is not yet included in the text, but is relevant and important for the topic, so that it can be used to update the content of the source article. Although the source page is in a specific language (the source language), the reader or author would also be interested in finding information in other languages (the target languages) that he explicitely specifies.
With this user scenario, the task of an automatic system is to locate information snippets in Wikipedia which are: • outside the given source page, • in one of the specified target languages, • substantially new w.r.t. the information contained in the source page, and important for the topic of the source page, in other words, worth including in the content of (the future editions of) the page.
Participants of the WiQA 2006 pilot could take part in two flavors of the task: a monolingual one (where the snippets to be returned are in the language of the source page) and a multilingual (where the snippets to be returned can be in any of the languages of the Wikipedia corpus used at WiQA). 2.2
The corpus used at WiQA 2006 consists of XML-ified dumps of Wikipedia in three language:
Dutch, English, and Spanish. The dumps are based on the XML version of the Wikipedia
collections [
For each of the three WiQA 2006 languages (Dutch, English, and Spanish) a set of 50 topics correctly tagged as PERSON, LOCATION or ORGANIZATION in the XML data collections was released, together with other topics, announced as optional. These optional topics either did not fall into these three categories, or were not tagged correctly in the XML collections. The optional topics could be ignored by systems without penalty. In fact, the submitted runs provided responses for optional topics as well as for the main topics.
When selecting Wikipedia articles as topics, we included articles explicitely marked as stubs, as well as other short and long articles.
In order to create the topics for the English-Dutch bilingual task, 30 topics were selected from the English monolingual topic set and 30 topics from the Dutch monolingual topic set. The bilingual topics were selected so that the corresponding articles are present in Wikipedias for both languages.
In addition to the test topics, a set of 80 (English language) development topics was released. 2.4
Given a source page, automatic systems return a list of short snippets, defined as sequences of at
most two sentences from a Wikipedia page. The ranked list of snippets for the topic were manually
assessed using the following binary criteria, largely inspired by the TREC 2003 Novelty task [
One of the purposes of the WiQA pilot task was to experiment with different measures for evaluating the performance of systems. WiQA 2006 used the following simple principal measure for accessing the performance of the systems: • yield : the average (per topic) number of supported, novel, non-repetitive, important target snippets.
• mean reciprocal rank of the first supported, important, novel, non-repeated snippet, and • overall precision: the percentage of supported, novel, non-repetitive, important snippets among all submitted snippets. 2.5
To establish the ground truth, an assessment environment was developed by the track organizers. Assessors were given the following guidelines. For each system and each source article P the ordered list of the returned snippets was be manually assessed with respect to importance, novelty and non-repetition following the procedure below: 1. Each snippet was marked as supported or not. To reduce the workload on the assessors, this aspect was checked automatically. Hence, unsupported snippets were excluded from the subsequent assessment. 2. Each snippet was marked as important or not, with respect to the topic of the source article.
A snippet is important if it contains information that a user of Wikipedia would like to see in P or an author would consider worth to be present in P. Snippets were assessed for importance independently of each other and regardless of whether the important information was already present in P (in particular, presence of some information in P does not necessarily imply its importance). 3. Each important snippet was marked as novel or not. It was to be considered novel if the important information in the snippet is substantially new with respect to the content of P. 4. Each important and novel snippet was marked as repeated or non-repeated, with respect to the important snippets higher in the ranked list of snippets. Following this procedure, snippets were assessed along four axes (support, importance, novelty, non-repetition). Assessors were not required to judge novelty and non-repetition of snippets that are considered not important for the topic of the source article. The reason for this was to avoid spending much time on assessing irrelevant information. Assessors provided assessments for the top 20 snippets for each result list returned. Figure 1 contains a screen shot of the assessment interface.
A total number of 14203 snippets had to be assessed; the number unique snippets assessed is 4959. Of these, 3396 were assessed by at least two assessors.
The results of the assessments for all submitted runs (anonymized) will be made available to all the participants for further analysis and experiments. 2.6
For each task (three monolingual and one bilingual), participating teams were allowed to submit up to three runs. For each topic of a run, the top 20 submitted snippets were manually assessed as described above. 3
Table 1 lists the runs submitted to WiQA 2006: 19 for the monolingual task (Dutch: 3, English: 12, Spanish: 4) and 1 for the bilingual task (English-Dutch). In Table 2 we present the aggregate results of the assessment of the runs submitted to WiQA 2006. Columns 3–7 show the following aggregate numbers: total number of snippets (with at most 20 snippets considered per response for Group English monolingual task LexiClone Inc.
Universidad Polit´ecnica de Val`encia University of Alicante University of Essex/Limerick University of Amsterdam
Universidad Polit´ecnica de Val`encia University of Alicante Daedalus consortium
UA-DLSI-2 dltg061 dltg062 uams-linkret-en uams-link-en uams-ret-en WLV-one-old WLV-two WLV-one rfia-bow-es
Near phrase temporal Limit of ten snippets per topic Limit of twenty snippets per topic Cross-links and IR for snippet ranking Only cross-links for snippet ranking Only IR for snippet ranking No coreference, link analysis Coreference no coreference, version 2 simple “bag of words” submission
InLink sentence retrieval, rank by novelty InLink passage retrieval, combine cosine and novelty in ranking, no threshold uams-linkret-nl uams-link-nl uams-ret-nl
a topic); total number of supported snippets; total number of important supported snippets; total number of novel and important supported snippets; and the total number of novel and important supported with repetition.
The results indicate that the task of detecting important snippets is a hard one: for most submissions, only 50–60% of the found snippets are judged as important. The performance of the systems for detecting novel snippets has a substantially higher range: between 50% and 80% of the found important snippets are judged as novel with respect to the topic article. 4
Table 3 shows the evaluation results for the submitted runs: total yield (for a run, the total number of “perfect” snippets, i.e., supported, important, novel and not repeated), the average yield per topic (only topics with at least one response are considered), the mean reciprocal rank of the first “perfect” snippet and the precision of the systems’ responses.
Clearly, most systems cope well with the pilot task: up to one third of the found snippets are assessed as “perfect” for the English and Spanish monolingual tasks, and up to one half for the Dutch monolingual and the English-Dutch bilingual task. Quite expectedly, the relative ranking of the submitted runs is different for different evaluation measures: as in many complex tasks, the best yield (a recall-oriented measure) does not necessarily lead to the best precision and vice versa.
Run name
An interesting aspect of the results is that the performance of the systems differs substantially for the four tasks. This can be due to the fact that the submissions for tasks were assessed by different assessors (native speakers of the corresponding languages), as well as due to the differences in the sizes and structures of the Wikipedias in these languages. It is worth pointing out that the highest scores were achieved on the English-Dutch bilingual task; this may suggest that different language versions of Wikipedia do indeed present different material on a given topic.
Finally, a more detailed analysis of this issue, as well as the analysis of the inter-annotator agreement will be presented by the time of the CLEF workshop. 5
We have described the first installment of the WiQA—Question Answering Using Wikipedia— task. Set up as an attempt to take question answering beyond the traditional factoid format and to one of the most interesting knowledge sources currently available, WiQA had 8 participants who submitted a total of 20 runs for 4 tasks. The results of the pilot are very encouraging. While challenging, the task turned out to be do-able, and in cases several participants managed to achieve impressive yield, MRR, and precision scores. Surprisingly, the highest scores were achieved on the bilingual task. As to the future of WiQA, as pointed out before we aim to take a close look at our assesssments, perhaps add new assessments, and analyse inter-assessor agreement along various dimensions. The WiQA 2006 pilot has shown that it is possible to set up tractable yet challenging information access tasks involving the multilingual Wikipedia corpus—but this was only a first step. In the future we would like to consider additional information access scenarios, all centered around Wikipedia. 6
We are very grateful to the following people and organizations for helping us with the assessments: Jos´e Luis Mart´ınez Fern´andez and C´esar de Pablo from the Daedalus consortium; Silke Scheible and Bonnie Webber at the University of Edinburgh; Udo Kruschwitz and Richard Sutcliffe at the University of Essex; and Bouke Huurnink and Maarten de Rijke at the University of Amsterdam.
Valentin Jijkoun was supported by the Netherlands Organisation for Scientific Research (NWO) under project numbers 220-80-001, 600.-065.-120 and 612.000.106. Maarten de Rijke was supported by NWO under project numbers 017.001.190, 220-80-001, 264-70-050, 354-20-005, 600.065.-120, 612-13-001, 612.000.106, 612.066.302, 612.069.006, 640.001.501, 640.002.501, and and by the E.U. IST programme of the 6th FP for RTD under project MultiMATCH contract IST-033104.