The Robust-WSD at CLEF 2009 aims at exploring the contribution of Word Sense Disambiguation to monolingual and multilingual Information Retrieval. The organizers of the task provide documents and topics which have been automatically tagged with Word Senses from WordNet using several state-of-the-art Word Sense Disambiguation systems. The Robust-WSD exercise follows the same design as in 2008. It uses two languages often used in previous CLEF campaigns (English, Spanish). Documents were in English, and topics in both English and Spanish. The document collections are based on the widely used LA94 and GH95 news collections. All instructions and datasets required to replicate the experiment are available from the organizers website (http://ixa2.si.ehu.es/clirwsd/). The results show that some top-scoring systems improve their IR and CLIR results with the use of WSD tags, but the best scoring runs do not use WSD.
The Robust-WSD task at CLEF 2009 aims at exploring the contribution of Word Sense Disambiguation to monolingual and multilingual Information Retrieval. The organizers of the task provide documents and topics which have been automatically tagged with Word Senses from WordNet using several stateof-the-art Word Sense Disambiguation systems. The task follows the same design as in 2008.
The robust task ran for the fourth time at CLEF 2009. It is an Ad-Hoc
retrieval task based on data of previous CLEF campaigns. The robust task
emphasizes the di cult topics by a non-linear integration of the results of individual
topics into one result for a system, using the geometric mean of the average
precision for all topics (GMAP) as an additional evaluation measure [
For the second year, the robust task also incorporated word sense
disambiguation information provided by the organizers to the participants. The task follows
the 2007 joint SemEval-CLEF task [
The experiment involved both monolingual (topics and documents in
English) and bilingual experiments (topics in Spanish and documents in English).
In addition to the original documents and topics, the organizers of the task
provided both documents and topics which had been automatically tagged with
word senses from WordNet version 1.6 using two state-of-the-art word sense
disambiguation systems, UBC [
In addition, the participants could use publicly available data from the English and Spanish wordnets in order to test di erent expansion strategies. Note that given the tight alignment of the Spanish and English wordnets, the wordnets could also be used to translate directly from one sense to another, and perform expansion to terms in another language.
The datasets used in this task can be used in the future to run further experiments. Check http://ixa2.si.ehu.es/clirwsd for information of how to access the datasets. Topics and relevance judgements are freely available. The document collection can be obtained from ELDA purchasing the CLEF Test Suite for the CLEF 2000-2003 Campaigns { Evaluation Package. As an alternative, the website o ers the unordered set of words in each document, that is, the full set of documents where the positional information has been eliminated to avoid replications of the originals. Lucene indexes for the later are also available from the website.
In this paper, we rst present the task setup, the evaluation methodology and the participation in the di erent tasks (Section 2). We then describe the main features of each task and show the results (Sections 3 - 5). The nal section provides a brief summing up. For information on the various approaches and resources used by the groups participating in this task and the issues they focused on, we refer the reader to the rest of the papers in the Robust-WSD part of the Ad Hoc section of these Proceedings. 2
The Ad Hoc task in CLEF adopts a corpus-based, automatic scoring method
for the assessment of system performance, based on ideas rst introduced in
the Cran eld experiments in the late 1960s [
The Documents. The robust task used existing CLEF news collections but
with word sense disambiguation (WSD) information added. The word sense
disambiguation data was automatically added by systems from two leading research
laboratories, UBC [
The document collections were o ered both with and without WSD, and included the following1: { LA Times 94 (with word sense disambiguated data); ca 113,000 documents, 425 MB without WSD, 1,448 MB (UBC) or 2,151 MB (NUS) with WSD; { Glasgow Herald 95 (with word sense disambiguated data); ca 56,500 documents, 154 MB without WSD, 626 MB (UBC) or 904 MB (NUS) with WSD.
The Topics. Topics are structured statements representing information needs.
Each topic typically consists of three parts: a brief title statement; a one-sentence
description; a more complex narrative the relevance assessment criteria. Topics
are prepared in xml format and identi ed by means of a Digital Object Identi er
(DOI)2 of the experiment [
The WSD robust task used existing CLEF topics in English and Spanish as follows: { CLEF 2001; Topics 10.2452/41-AH { 10.2452/90-AH; LA Times 94 { CLEF 2002; Topics 10.2452/91-AH { 10.2452/140-AH; LA Times 94 { CLEF 2003; Topics 10.2452/141-AH { 10.2452/200-AH; LA Times 94, Glasgow Herald 95 { CLEF 2004; Topics 10.2452/201-AH { 10.2452/250-AH; Glasgow Herald 95 { CLEF 2005; Topics 10.2452/251-AH { 10.2452/300-AH; LA Times 94, Glasgow Herald 95 { CLEF 2006; Topics 10.2452/301-AH { 10.2452/350-AH; LA Times 94, Glasgow Herald 95
Topics from years 2001, 2002 and 2004 were used as training topics (relevance assessments were o ered to participants), and topics from years 2003, 2005 and 2006 were used for the test.
All topics were o ered both with and without WSD. Topics in English were
disambiguated by both UBC [
<WF>for</WF> </TERM> <EN-desc> <TERM ID="10.2452/141-WSD-AH-5" LEMA="find" POS="VBP"> <WF>Find</WF> <SYNSET SCORE="0" CODE="00658116-v"/> ... </EN-title>
WordNet version 1.6. A large-scale disambiguation system for Spanish was not available, so we used the rst-sense heuristic, yielding senses from the Spanish wordnet, which is tightly aligned to the English WordNet version 1.6 (i.e., they share synset numbers or sense codes). An excerpt from a topic is shown in Figure 1, where each term in the topic is followed by its senses with their respective scores as assigned buy the automatic WSD system3.
Relevance Assessment. The number of documents in large test collections such as CLEF makes it impractical to judge every document for relevance. Instead approximate recall values are calculated using pooling techniques. The robust WSD task used existing relevance assessments from previous years. The 3 Full sample and dtd are available at http://ixa2.si.ehu.es/clirwsd/ relevance assessments regarding the training topics were provided to participants before competition time.
The total number of assessments was 66,441 documents of which 4,327 were relevant. The distribution of the pool according to each year was the following: { CLEF 2003: 23,674 documents, 1,006 relevant; { CLEF 2005: 19,790 document, 2,063 relevant; { CLEF 2006: 21,247 document, 1,258 relevant;
Seven topics had no relevant documents at all: 10.2452/149-AH, 10.2452/161AH, 10.2452/166-AH, 10.2452/186-AH, 10.2452/191-AH, 10.2452/195-AH, 10.2452/321-AH. Each topic had an average of about 28 relevant documents and a standard deviation of 34, a minimum of 1 relevant document and a maximum of 229 relevant documents per topic. 2.2
Evaluation campaigns such as TREC and CLEF are based on the belief that
the e ectiveness of Information Retrieval Systems (IRSs) can be objectively
evaluated by an analysis of a representative set of sample search results. For
this, e ectiveness measures are calculated based on the results submitted by the
participants and the relevance assessments. Popular measures usually adopted
for exercises of this type are Recall and Precision. Details on how they are
calculated for CLEF are given in [
The robust task emphasizes the di cult topics by a non-linear integration of
the results of individual topics into one result for a system, using the geometric
mean of the average precision for all topics (GMAP) as an additional evaluation
measure [
The individual results for all o cial Ad Hoc experiments in CLEF 2009 are
given in the one of the Appendices of the CLEF 2009 Working Notes [
As shown in Table 1, 10 groups submitted 89 runs for the Robust tasks: { 8 groups submitted monolingual non-WSD runs (25 runs out of 89); { 5 groups also submitted bilingual non-WSD runs (13 runs out of 89). All groups submitted WSD runs (51 out of 89 runs): { 10 groups submitted monolingual WSD runs (33 out of 89 runs) { 5 groups submitted bilingual WSD runs (18 out of 89 runs)
Table 2 provides a breakdown of the number of participants and submitted runs by task. Note that jaen submitted a monolingual non-WSD run as if it was a WSD run, and that alicante missed to send their non-WSD run on time. The gures below are the o cial gures. participant task alicante AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 darmstadt AH-ROBUST-MONO-EN-TEST-CLEF2009 darmstadt AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 geneva AH-ROBUST-MONO-EN-TEST-CLEF2009 geneva AH-ROBUST-WSD-BILI-X2EN-TEST-CLEF2009 geneva AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 ixa AH-ROBUST-BILI-X2EN-TEST-CLEF2009 ixa AH-ROBUST-MONO-EN-TEST-CLEF2009 ixa AH-ROBUST-WSD-BILI-X2EN-TEST-CLEF2009 ixa AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 jaen AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 know-center AH-ROBUST-BILI-X2EN-TEST-CLEF2009 know-center AH-ROBUST-MONO-EN-TEST-CLEF2009 know-center AH-ROBUST-WSD-BILI-X2EN-TEST-CLEF2009 know-center AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 reina AH-ROBUST-BILI-X2EN-TEST-CLEF2009 reina AH-ROBUST-MONO-EN-TEST-CLEF2009 reina AH-ROBUST-WSD-BILI-X2EN-TEST-CLEF2009 reina AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 ufrgs AH-ROBUST-BILI-X2EN-TEST-CLEF2009 ufrgs AH-ROBUST-MONO-EN-TEST-CLEF2009 ufrgs AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 uniba AH-ROBUST-BILI-X2EN-TEST-CLEF2009 uniba AH-ROBUST-MONO-EN-TEST-CLEF2009 uniba AH-ROBUST-WSD-BILI-X2EN-TEST-CLEF2009 uniba AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009 valencia AH-ROBUST-MONO-EN-TEST-CLEF2009 valencia AH-ROBUST-WSD-MONO-EN-TEST-CLEF2009
When the goal is to validate how well results can be expected to hold beyond a particular set of queries, statistical testing can help to determine what di erences between runs appear to be real as opposed to di erences that are due to sampling issues. We aim to identify whether the results of the runs of a task are signi cantly di erent from the results of other tasks. In particular, we want to test whether there is any di erence between applying WSD techniques or not. Signi cantly di erent in this context means that the di erence between the performance scores for the runs in question appears greater than what might be expected by pure chance. As with all statistical testing, conclusions will be quali ed by an error probability, which was chosen to be 0.05 in the following. 90% 80% 70% 60%
We have designed our analysis to follow closely the methodology used by similar analyses carried out for Text REtrieval Conference (TREC) [23].
We used the MATLAB Statistics Toolbox, which provides the necessary functionality plus some additional functions and utilities.
Two tests for goodness of t to a normal distribution were chosen using the MATLAB statistical toolbox: the Lilliefors test and the Jarque-Bera test. In the case of the CLEF tasks under analysis, both tests indicate that the assumption of normality is not violated for most of the data samples (in this case the runs for each participant).
The two tests were: { Robust Monolingual vs Robust WSD Monolingual; { Robust Bilingual vs Robust WSD Bilingual.
In both cases, the t-test con rmed that the mean of the two distributions are di erent and, in particular, the mean of the monolingual distribution is greater than the mean of the robust monolingual WSD, and the same happens for the bilingual. This suggests some loss of performances due to the e ect of the word sense disambiguation in both monolingual and bilingual tasks. However, there 90% 80% 70% 60% 0%0% 10% 20% 30% 40% are a few topics where the WSD techniques signi cantly improve the e ectiveness of the retrieval; these are the cases worth studying from a WSD point of view. In this section we focus on the comparison between WSD and non-WSD runs. Overall, the best MAP and GMAP results in the monolingual system were for two distinct runs which did not use WSD information. Several participants were able to obtain their best MAP and GMAP scores using WSD information. In the bilingual experiments, the best results in MAP was for non-WSD runs, but two participants were able to pro t from the WSD annotations. As it is di cult to summarize the behavior of all participants below, we will only mention the performance of the best teams, as given in Tables 3 and 4. The interested reader is directed to the working notes of each participant for additional details.
In the monolingual experiments, cf. Table 3, the best results overall in MAP
was for darmstadt. Their WSD runs scored very similar to the non-WSD runs,
with a slight decrease of MAP (0.09 percentage points) and a slight increase
of GMAP (0.07 percentage points) [15]. The second best MAP score and best
GMAP was attained by reina [16] without WSD, with their WSD systems
show90%
80%
70%
60%
0%0%
10%
20%
30%
40%
ing a considerable performance drop. The third best MAP and second GMAP
where obtained by uniba [
In the bilingual experiments, cf. Table 4, the best results overall in MAP
were for reina with a system which did not use WSD annotations [16]. The
best GMAP was for geneva using WSD [
GMAP). Ufrgs [
All in all, the exercise showed that some teams did improve results using WSD (close to 1 MAP point and more than 1 GMAP point in monolingual, and below 1 MAP/GMAP point in bilingual), but the best results for both monolingual and bilingual tasks were for systems which did not use WSD. 4
This new edition of the robust WSD exercise has measured to what extent IR systems could pro t from automatic word sense disambiguation information. The conclusions on the monolingual subtask are similar to the conclusions of 2008. The evidence for using WSD in monolingual IR is mixed, with some top scoring groups reporting small improvements in MAP and GMAP, but with the best overall scores for systems not using WSD.
Regarding the cross-lingual task, the situation is very similar, but the improvements reported by using WSD are smaller.
Instructions and datasets to replicate the results (including Lucene indexes) are available from http://ixa.si.ehu.es/clirwsd.
The robust task was partially funded by the Ministry of Education (project KNOW TIN2006-15049) and the European Commission (project KYOTO ICT2007-211423). We want to thank Oier Lopez de Lacalle, who runs the UBC WSD system, and Yee Seng Chan, Hwee Tou Ng and Zhi Zhong, who run the NUS WSD system. Their generous contribution was invaluable to run this exercise. 15. Wolf, E., Bernhard, D., Gurevych, I.: Combining Probabilistic and TranslationBased Models for Information Retrieval based on Word Sense Annotations Information Retrieval. In this volume. 16. Zazo, A., Figuerola, C.G., Alonso Berrocal, J.L., Gomez, R.: REINA at CLEF 2009 Robust-WSD Task: Partial Use of WSD Information for Retrieval. In this volume.