We describe the objectives and organization of the CLEF 2008 ad hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was changed considerably this year with the introduction of new document collections consisting of library catalog records derived from The European Library, with a non-European target language, and with a task offering word sense disambiguated data for groups interested in the impact of natural language processing on the performance of information retrieval systems. The track was thus structured in three distinct streams denominated: TEL@CLEF, Persian@CLEF and Robust WSD. The results obtained for each task are presented and statistical analyses are given.
The ad hoc retrieval track is generally considered to be the core track in the Cross-Language Evaluation Forum (CLEF). The aim of this track is to promote the development of monolingual and cross-language textual document retrieval systems. From 2000 - 2007, the track used exclusively collections of European newspaper and news agency documents1. This year the focus of the track was considerably widened: we introduced very different document collections, a nonEuropean target language, and an information retrieval (IR) task designed to attract participation from groups interested in natural language processing (NLP). The track was thus structured in three distinct streams:
The first task offered monolingual and cross-language search on library catalog records and was organized in collaboration with The European Library (TEL)2. The second task resembled the ad hoc retrieval tasks of previous years but this time the target collection was a Persian newspaper corpus.
The third task was the robust activity which this year used word sense disambiguated (WSD) data, and involved English documents and monolingual and cross-language search in Spanish.
In this paper we first present the track setup, the evaluation methodology and the participation in the different tasks (Section 2). We then describe the main features of each task and show the results (Sections 3 - 5). Statistical testing is discussed in Section 6 and the final section provides a brief summing up.
For information on the various approaches and resources used by the groups participating in this track and the issues they focused on, we refer the reader to the other papers in the Ad Hoc section of these Working Notes. 2
The ad hoc track in CLEF adopts a corpus-based, automatic scoring method for the assessment of system performance, based on ideas first introduced in the Cranfield experiments in the late 1960s [13]. The tasks offered are studied in order to effectively measure textual document retrieval under specific conditions. The test collections are made up of documents, topics and relevance assessments. The topics consist of a set of statements simulating information needs from which the systems derive the queries to search the document collections. Evaluation of system performance is then done by judging the documents retrieved in response to a topic with respect to their relevance, and computing the recall and precision measures. The distinguishing feature of CLEF is that it applies this evaluation paradigm in a multilingual setting. This means that the criteria normally adopted to create a test collection, consisting of suitable documents, sample queries and relevance assessments, have been adapted to 1 Over the years, this track has built up test collections for monolingual and crosslanguage system evaluation in 13 European languages (see the Introduction to this volume for more details) 2 See http://www.theeuropeanlibrary.org/ satisfy the particular requirements of the multilingual context. All language dependent tasks such as topic creation and relevance judgment are performed in a distributed setting by native speakers. Rules are established and a tight central coordination is maintained in order to ensure consistency and coherency of topic and relevance judgment sets over the different collections, languages and tracks. 2.1
Each of the three ad hoc tasks this year used a different set of documents.
The TEL task used three collections: – British Library (BL); 1,000,100 documents, 1.2 GB; – Biblioth´eque Nationale de France (BNF); 1,000,100 documents, 1.3 GB; – Austrian National Library (ONB); 869,353 documents, 1.3 GB.
We refer to the three collections (BL, BNF, ONB) as English, French and German because in each case this is the main and expected language of the collection. However, each of these collections is to some extent multilingual and contains documents (catalog records) in many additional languages.
The TEL data is very different from the newspaper articles and news agency dispatches previously used in the CLEF ad hoc track. The data tends to be very sparse. Many records contain only title, author and subject heading information; other records provide more detail. The title and (if existing) an abstract or description may be in a different language to that understood as the language of the collection. The subject heading information is normally in the main language of the collection. About 66% of the documents in the English and German collection have textual subject headings, in the French collection only 37%. Dewey Classification (DDC) is not available in the French collection; negligible (¡0.3%) in the German collection; but occurs in about half of the English documents (456,408 docs to be exact).
Whereas in the traditional ad hoc task, the user searches directly for a document containing information of interest, here the user tries to identify which publications are of potential interest according to the information provided by the catalog card. When we designed the task, the question the user was presumed to be asking was “Is the publication described by the bibliographic record relevant to my information need?”
The Persian task used the Hamshahri corpus of 1996-2002 newspapers as the target collection. This corpus was made available to CLEF by the Data Base Research Group (DBRG) of the University of Tehran. Hamshahri is one of the most popular daily newspapers in Iran. The Hamshahri corpus is a Persian test collection that consists of 345 MB of news texts for the years 1996 to 2002 (corpus size with tags is 564 MB). This corpus contains more than 160,000 news articles about a variety of subjects and includes nearly 417000 different words. Hamshahri articles vary between 1KB and 140KB in size3. 3 For more information, see http://ece.ut.ac.ir/dbrg/hamshahri/
The robust task used existing CLEF news collections but with word sense
disambiguation (WSD) added. The word sense disambiguation data was
automatically added by systems from two leading research laboratories, UBC [
The document collections were offered both with and without WSD, and included the following: – LA Times 94 (with word sense disambiguated data); ca 113,000 documents, 425 MB without WSD, 1,448 MB (UBC) or 2,151 MB (NUS) with WSD; – Glasgow Herald 95 (with word sense disambiguated data); ca 56,500 documents, 154 MB without WSD, 626 MB (UBC) or 904 MB (NUS) with WSD.
An excerpt for a document4 is shown in Figure 1, where each term in the document is followed by its senses with their respective scores as assigned by the automatic WSD system. 2.2
Topics in the CLEF ad hoc track are structured statements representing information needs. Each topic typically consists of three parts: a brief “title” statement; a one-sentence “description”; a more complex “narrative” specifying the relevance assessment criteria. Topics are prepared in xml format and identified by means of a Digital Object Identifier (DOI)5 of the experiment [35] which allows us to reference and cite them.
For the TEL task, a common set of 50 topics was prepared in each of the 3 main collection languages (English, French and German) plus Dutch and Spanish in response to demand. Only the Title and Description fields were released to the participants. The narrative was employed to provide information for the assessors on how the topics should be judged. The topic sets were prepared on the basis of the contents of the collections.
In ad hoc, when a task uses data collections in more than one language, we consider it important to be able to use versions of the same core topic set to query all collections. This makes it easier to compare results over different collections and also facilitates the preparation of extra topic sets in additional languages. However, it is never easy to find topics that are effective for several different collections and the topic preparation stage requires considerable discussion between the coordinators for each collection in order to identify suitable common candidates. The sparseness of the data made this particularly difficult for the TEL task and tended to lead to the formulation of topics that were quite broad in scope so that at least some relevant documents could be found in each collection. A result of this strategy is that there tends to be a considerable lack of evenness of distribution in relevant documents. For each topic, the results 4 Full sample and dtd are available at http://ixa2.si.ehu.es/clirwsd/ 5 http://www.doi.org/ <DOC> <DOCNO>GH950102-000000</DOCNO> <DOCID>GH950102-000000</DOCID> <HEADLINE> <TERM ID="GH950102-000000-1" LEMA="alien" POS="JJ"> <WF>Alien</WF> <SYNSET SCORE="0.6" CODE="01295935-a"/> <SYNSET SCORE="0.4" CODE="00984080-a"/> </TERM> <TERM ID="GH950102-000000-2" LEMA="treatment" POS="NN"> <WF>treatment</WF> <SYNSET SCORE="0.827904118008605" CODE="00735486-n"/> <SYNSET SCORE="0" CODE="03857483-n"/> <SYNSET SCORE="0.172095881991395" CODE="00430183-n"/> <SYNSET SCORE="0" CODE="05340429-n"/> </TERM> <TERM ID="GH950102-000000-3" LEMA="be" POS="VBZ"> <WF>is</WF> <SYNSET SCORE="0.0113384126222329" CODE="01787769-v"/> <SYNSET SCORE="0.181174635551023" CODE="01784339-v"/> <SYNSET SCORE="0.644489771431999" CODE="01775973-v"/> <SYNSET SCORE="0.00515927770112184" CODE="01666138-v"/> <SYNSET SCORE="0.0420541124242606" CODE="01775163-v"/> <SYNSET SCORE="0.00347951286819845" CODE="01840295-v"/> <SYNSET SCORE="0.0540524326594277" CODE="01811792-v"/> <SYNSET SCORE="0" CODE="01843641-v"/> <SYNSET SCORE="0.000119983202351671" CODE="01552250-v"/> <SYNSET SCORE="0.0418741376207331" CODE="01781222-v"/> <SYNSET SCORE="5.99916011758354e-05" CODE="01782836-v"/> <SYNSET SCORE="0.0161977323174756" CODE="01817610-v"/> </TERM> expected from the separate collections can vary considerably, e.g. in the case of the TEL task, a topic of particular interest to Britain, such as the example given in Figure 2, can be expected to find far more relevant documents in the BL collection than in BNF or ONB.
For the Persian task, 50 topics were created in Persian by the Data Base Research group of the University of Tehran, and then translated into English. The rule in CLEF when creating topics in additional languages is not to produce literal translations but to attempt to render them as naturally as possible. This was a particularly difficult task when going from Persian to English as cultural differences had to be catered for.
For example, Iran commonly uses a different calendar from Europe and reference was often made in the Persian topics to events that are well known to Iranian society but not often discussed in English. This is shown in the example of Figure 3, where the rather awkward English rendering evidences the uncertainty of the translator. <?xml version="1.0" encoding="UTF-8" standalone="no"?> <topic> <identifier>10.2452/451-AH</identifier> <title lang="en">Roman Military in Britain</title> <title lang="de">Römisches Militär in Britannien</title> <title lang="es">El ejército romano en Britania</title> <title lang="fr">L'armée romaine en Grande-Bretagne</title> <title lang="nl">Romeinse Leger in Groot-Brittannie</title> <description lang="en">Find books or publications on the Roman invasion or military occupation of Britain.</description> <description lang="de">Finden Sie Bücher oder Publikationen über die römische Invasion oder das
Militär in Britannien.</description> <description lang="es">Encuentre libros o publicaciones sobre la invasión romana o la ocupación militar romana en Britania.</description> <description lang="fr">Trouver des livres ou des publications sur l'invasion et l'occupation de la Grande-Bretagne par les Romains.</description> <description lang="nl">Vind boeken of publicaties over de Romeinse invasie of bezetting van
Groot-Brittannie.</description>
Topics from years 2001, 2002 and 2004 were used as training topics (relevance assessments were offered to participants), and topics from years 2003, 2005 and 2006 were used for the test.
All topics were offered both with and without WSD. Topics in English were
disambiguated by both UBC [
</title> <description lang="en">Find documents that include information about the 2nd of Khordad
presidential elections.</description> <description lang="fa"> ;?+@> </description>
76 -(. /(0 !"!#$ %&! '()(*+," !120 1! !(345" 60(7 89 :9 ";<= "1 "(>;?. <narrative lang="en">Any information about candidates and their sayings, Khatami's unexpected
winning in the 2nd of Khordad 1376 presidential election is relevant.</narrative> <narrative lang="fa"> AB&#= C($D A(> 8+EF & (>!G0(, !120 1! !(345" 60(7 H2)#0 A(>;?.
I." 76 -(. /(0 !"!#$ %&! 1! A12J% I.(K1 '()(*+," 1! LM($ /#N+?0#O </narrative> participating in the ad hoc tasks are used to form a pool of documents for each topic and language by collecting the highly ranked documents from selected runs according to a set of predefined criteria. Traditionally, the top 100 ranked documents from each of the runs selected are included in the pool; in such a case we say that the pool is of depth 100. This pool is then used for subsequent relevance judgments. After calculating the effectiveness measures, the results are analyzed and run statistics produced and distributed.
The stability of pools constructed in this way and their reliability for
postcampaign experiments is discussed in [
The main criteria used when constructing the pools were: – favour diversity among approaches adopted by participants, according to the descriptions of the experiments provided by the participants; – choose at least one experiment for each participant in each task, chosen among the experiments with highest priority as indicated by the participant; – add mandatory title+description experiments, even though they do not have high priority; – add manual experiments, when provided; – for bilingual tasks, ensure that each source topic language is represented.
One important limitation when forming the pools is the number of documents to be assessed. Last year, for collections of newspaper documents, we estimated that assessors could judge from 60 to 100 documents per hour, providing binary <top> <num>10.2452/141-WSD-AH</num> <EN-title> <TERM ID="10.2452/141-WSD-AH-1" LEMA="letter" POS="NNP"> <WF>Letter</WF> <SYNSET SCORE="0" CODE="05115901-n"/> <SYNSET SCORE="0" CODE="05362432-n"/> <SYNSET SCORE="0" CODE="05029514-n"/> <SYNSET SCORE="1" CODE="04968965-n"/> </TERM> <TERM ID="10.2452/141-WSD-AH-2" LEMA="bomb" POS="NNP"> <WF>Bomb</WF> <SYNSET SCORE="0.888888888888889" CODE="02310834-n"/> <SYNSET SCORE="0" CODE="05484679-n"/> <SYNSET SCORE="0.111111111111111" CODE="02311368-n"/> </TERM> <TERM ID="10.2452/141-WSD-AH-3" LEMA="for" POS="IN">
<WF>for</WF> </TERM> <EN-desc> <TERM ID="10.2452/141-WSD-AH-5" LEMA="find" POS="VBP"> <WF>Find</WF> <SYNSET SCORE="0" CODE="00658116-v"/> ... </EN-title> judgments: relevant / not relevant. Our estimate this year for the TEL catalog records was higher as these records are much shorter than the average newspaper article (100 to 120 documents per hour). In both cases, it can be seen what a time-consuming and resource expensive task human relevance assessment is. This limitation impacts strongly on the application of the criteria above - and implies that we are obliged to be flexible in the number of documents judged per selected run for individual pools.
This meant that this year, in order to create pools of more-or-less equivalent size (approx. 25,000 documents), the depth of the TEL English, French, and German pools and of the Persian pool was 607. 7 Tests made on NTCIR pools in previous years have suggested that a depth of 60 in normally adequate to create stable pools, presuming that a sufficient number of runs from different systems have been included
Table 1 reports summary information on the 2008 ad hoc pools used to calculate the results for the main monolingual and bilingual experiments. In particular, for each pool, we show the number of topics, the number of runs submitted, the number of runs included in the pool, the number of documents in the pool (relevant and non-relevant), and the number of assessors.
The box plot of Figure 5 compares the distributions of the relevant documents across the topics of each pool for the different ad hoc pools; the boxes are ordered by decreasing mean number of relevant documents per topic.
As can be noted, TEL English, French and German distributions appear similar and are asymmetric towards topics with a greater number of relevant documents. Both the English and French distributions show some upper outliers, i.e. topics with a greater number of relevant document with respect to the behaviour of the other topics in the distribution. These outliers are probably due to the fact that CLEF topics have to be able to retrieve relevant documents in all the collections; therefore, they may be considerably broader in one collection compared with others depending on the contents of the separate datasets.
For the TEL documents, we judged for relevance only those documents that are written totally or partially in English, French and German (and Spanish for searches on the English collection as we expected this language to be used only for ES to EN runs), e.g. a catalog record written entirely in Hungarian was counted as not relevant as it was of no use to our hypothetical user; however, a catalog record with perhaps the title and a brief description in Hungarian, but with subject descriptors in French, German or English was judged for relevance as it could be potentially useful. Our assessors had no additional knowledge of the documents referred to by the catalog records (or surrogates) contained in the collection. They judged for relevance on the information contained in the records made available to the systems. This was a non trivial task due to the lack of information present in the documents. During the relevance assessment activity there was much consultation between the assessors for the three TEL collections in order to ensure that the same assessment criteria were adopted by everyone.
As shown in the box plot of Figure 5, the Persian distribution presents a greater number of relevant documents per topic with respect to the other distributions and is more symmetric between topics with lesser or greater number of relevant documents. This greater symmetry in distribution of relevant documents is probably due to the fact that the topic set was created just on the basis of the contents of the Persian collection, rather than needing to reflect the contents of multiple collections. In addition, as can be seen from Table 1, it has been possible to sample all the experiments submitted for the Persian tasks. This means that there were fewer unique documents per run and this fact, together with the greater number of relevant documents per topic suggests either that all the systems were using similar approaches and retrieval algorithms (however this is not so - see Section 4 below) or that the systems found the Persian topics quite easy.
TEL English Pool (DOI 10.2454/AH-TEL-ENGLISH-CLEF2008) 28,104 pooled documents – 25,571 not relevant documents – 2,533 relevant documents 50 topics 21 out of 61 submitted experiments TEL French Pool (DOI 10.2454/AH-TEL-FRENCH-CLEF2008) 3 assessors 24,530 pooled documents – 23,191 not relevant documents – 1,339 relevant documents 50 topics 14 out of 45 submitted experiments 28,734 pooled documents – 27,097 not relevant documents – 1,637 relevant documents 50 topics 16 out of 47 submitted experiments
Persian Pool (DOI 10.2454/AH-PERSIAN-CLEF2008) 26,814 pooled documents – 21,653 not relevant documents – 5,161 relevant documents 50 topics 66 out of 66 submitted experiments Pool size Assessors Pool size Pool size Assessors Pool size Pooled Experiments – monolingual: 13 out of 37 submitted experiments – bilingual: 8 out of 24 submitted experiments Pooled Experiments – monolingual: 9 out of 29 submitted experiments – bilingual: 5 out of 16 submitted experiments Assessors
TEL German Pool (DOI 10.2454/AH-TEL-GERMAN-CLEF2008) Pooled Experiments – monolingual: 10 out of 30 submitted experiments – bilingual: 6 out of 17 submitted experiments Pooled Experiments – monolingual: 53 out of 53 submitted experiments – bilingual: 13 out of 13 submitted experiments Assessors 22 assessors P − 8 0 0 2 F E L C − S I L G E − L E T − A / 4 5 4 2 . 0 1 H
N H
H 8 0 0 2 F E L C − A M R E G − L E T − A / 4 5 4 2 . 0 1 0 6 2 0 4 2 0 2 2 0 0 2 0 8 1 0 6 1 s t n e D t n a v e l e e b m u 40 cum 1 o 120 froR 8 0 0 2 F E L C − 0 8 0 6 0 4 0 2 0
The relevance assessment for the Persian results was done by the DBRG group in Tehran. Again, assessment was performed on a binary basis and the standard CLEF assessment rules were applied.
As has already been stated, the robust WSD task used existing relevance assessments from previous years. The relevance assessments regarding the training topics were provided to participants before competition time.
This year, we tried a slight improvement with respect to the traditional pooling strategy adopted so far in CLEF. During the topic creation phase, the assessors express their opinion about the relevance of the documents they inspect with respect to the topic. Although this opinion may change during the various discussions between assessors in this phase, we consider these indications as potentially useful in helping to strengthen the pools of documents that will be judged for relevance. They are thus added to the pools. However, the assessors are not informed of which documents they had previously judged in order not to bias them in any way.
Similarly to last year, in his paper, Stephen Tomlinson, has reported some sampling experiments aimed at estimating the judging coverage for the CLEF 2008 test collections. He finds that this tends to be lower than the estimates he produced for the CLEF 2007 collections. With respect to the TEL collections, the implication is that at best 50% to 70% of the relevant documents are included in the pools - and that most of the unjudged relevant documents are for the 10 or more queries that have the most known answers [40]. For Persian the coverage seems to be lower; this could be a result of the fact that all the Persian topics tend to be relatively broad. It is our intention to look more closely into the question of coverage of these pools by performing some post-workshop stability tests. The results will be reported in our Proceedings paper. 2.4
Evaluation campaigns such as TREC and CLEF are based on the belief that
the effectiveness of Information Retrieval Systems (IRSs) can be objectively
evaluated by an analysis of a representative set of sample search results. For
this, effectiveness measures are calculated based on the results submitted by the
participants and the relevance assessments. Popular measures usually adopted
for exercises of this type are Recall and Precision. Details on how they are
calculated for CLEF are given in [
The individual results for all official ad hoc experiments in CLEF 2008 are given in the Appendices at the end of these Working Notes [17,18,19]. 2.5
As shown in Table 2, a total of 24 groups from 14 different countries submitted official results for one or more of the ad hoc tasks - a slight increase on the 22 participants of last year. Table 3 provides a breakdown of the number of participants by country8
A total of 289 runs were submitted with an increase of about 22% on the 235 runs of 2007. The average number of submitted runs per participant also increased: from 10.6 runs/participant of 2007 to 12.0 runs/participant of this year.
Participants were required to submit at least one title+description (“TD”) run per task in order to increase comparability between experiments. The large majority of runs (215 out of 289, 74.40%) used this combination of topic fields, 27 (9.34%) used all fields9, 47 (16.26%) used the title field. The majority of experiments were conducted using automatic query construction (273 out of 289, 94.47%) and only in a small fraction of the experiments (16 out 289, 5.53%) were queries been manually constructed from topics. A breakdown into the separate tasks is shown in Table 4(a).
Seven different topic languages were used in the ad hoc experiments. As always, the most popular language for queries was English, with Farsi second. The number of runs per topic language is shown in Table 4(b). 8 Two additional Spanish groups presented results after the deadline for the robust tasks; their results were thus not reported in the official list but their papers are included in this volume [30], [32]. 9 The narrative field was only offered for the Persian and Robust tasks.
Participant Institution Country chemnitz Chemnitz University of Technology Germany cheshire U.C.Berkeley United States geneva University of Geneva Switzerland imag Inst. For Infocomm Research France inaoe INAOE Mexico inesc INESC ID Portugal isi Indian Statistical Institute India ixa Univ. Basque Country Spain jhu-apl Johns Hopkins University Applied Physics Lab United States karlsruhe University of Karlsruhe Germany know-center Knowledge Relationship Discovery Austria opentext Open Text Corporation Canada tehran-IRDB IR-DB Research Group Iran tehran-NLP NLP-Software Engineering Grad. Lab Iran tehran-NLPDB NLP-DB Research Group Iran tehran-NLPDB2 NLP-DB Group Iran tehran-SEC School of Electrical Computing-1 Iran twente Univ. of Twente Netherlands ucm Universidad Complutense de Madrid Spain ufrgs Univ. Fed. do Rio Grande do Sul Brazil uniba Universita’ di Bari Italy unine U.Neuchatel-Informatics Switzerland xerox Xerox Reseearch - Data Mining France xerox-sas Xerox SAS Italy The objective of this activity was to search and retrieve relevant items from collections of library catalog cards. The underlying aim was to identify the most effective retrieval technologies for searching this type of very sparse data. Two subtasks were offered: Monolingual and Bilingual. In both tasks, the aim was to retrieve documents relevant to the query. By monolingual we mean that the query is in the same language as the expected language of the collection. By bilingual we mean that the query is in a different language to the expected language of the collection. For example, in an EN → FR run, relevant documents (bibliographic records) could be any document in the BNF collection (referred to as the French collection) in whatever language they are written. The same is true for a monolingual FR → FR run - relevant documents from the BNF collection could actually also be in English or German, not just French.
In CLEF 2008, the activity we simulated was that of users who have a working knowledge of English, French and German (plus wrt the English collection also Spanish) and who want to discover the existence of relevant documents that can be useful for them in one of our three target collections. One of our suppositions was that, knowing that these collections are to some extent multilingual, some systems may attempt to use specific tools to discover this. For example, a system trying the cross-language English to French task on the BNF target collection but knowing that documents retrieved in English and German will also be judged for relevance might choose to employ an English-German as well as the probable English-French dictionary. Groups attempting anything of this type were asked to declare such runs with a ++ indication. 13 groups submitted 153 runs for the TEL task: all groups submitted monolingual runs (96 runs out of 153); 8 groups also submitted bilingual runs (57 runs out of 153). Table 4(a) provides a breakdown of the number of participants and submitted runs by task. Table 5 shows the top five groups for each target collection, ordered by mean average precision. The table reports: the short name of the participating group; the mean average precision achieved by the experiment; the DOI of the experiment; and the performance difference between the first and the last participant.
Figures 7, 9, and 11 compare the performances of the top participants of the TEL Monolingual tasks. – X → EN: 90.99% of best monolingual English IR system; – X → FR: 56.63% of best monolingual French IR system; – X → DE: 53.15% of best monolingual German IR system.
While the best result for English, obtained with German topics, is very good and can be considered as state-of-the-art for a good cross-language system running on well-tested languages with reliable processing tools and resources such as English and German, the results for the other two target collections are fairly disappointing. We have no explanation for this at the present. In the TEL experiments, all the traditional approaches to monolingual and crosslanguage retrieval were attempted by the different groups. Retrieval algorithms Ad−Hoc TEL Monolingual English Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% unine [Experiment UNINEEN3; MAP 37.54%; Not Pooled] inesc [Experiment RUN3; MAP 36.23%; Not Pooled] chemnitz [Experiment CUT_SIMPLE; MAP 35.62%; Pooled] jhu−apl [Experiment JHUMOEN4RF; MAP 35.31%; Not Pooled] cheshire [Experiment BKAHTELMENTDT2F; MAP 34.66%; Pooled] Ad−Hoc TEL Bilingual English Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% chemnitz [Experiment CUT_SIMPLE_DE2EN; MAP 34.16%; Pooled] cheshire [Experiment BKAHTELBFRENTDT2FB; MAP 28.25%; Not Pooled] ufrgs [Experiment UFRGS_BI_SP_EN2; MAP 23.16%; Not Pooled] twente [Experiment FCW; MAP 22.78%; Pooled] jhu−apl [Experiment JHUBIDEEN5; MAP 21.11%; Not Pooled] 0% 0% 10% 20% 30% 40% 50% Recall 60% 70% 80% 90%
100% 50%
Recall 10% 20% 30% 40% 60% 70% 80% 90% 100%
Fig. 8. Bilingual English Ad−Hoc TEL Monolingual French Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% unine [Experiment UNINEFR3; MAP 33.27%; Not Pooled] xerox [Experiment J1; MAP 30.89%; Not Pooled] jhu−apl [Experiment JHUMOFR4; MAP 29.50%; Pooled] opentext [Experiment OTFR08TD; MAP 25.24%; Not Pooled] cheshire [Experiment BKAHTELMFRTDT2FB; MAP 24.38%; Pooled] Ad−Hoc TEL Bilingual French Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% cheshire [Experiment BKAHTELBDEFRTDT2FB; MAP 18.85%; Pooled] chemnitz [Experiment CUT_SIMPLE_EN2FR; MAP 17.54%; Pooled] jhu−apl [Experiment JHUBINLFR5; MAP 17.46%; Not Pooled] xerox [Experiment GER_FRE_J; MAP 11.63%; Not Pooled] xerox−sas [Experiment CACAOENGFREPLAIN; MAP 6.78%; Pooled] 50%
Recall 10% 20% 30% 40% 60% 70% 80% 90% 100% Ad−Hoc TEL Monolingual German Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% opentext [Experiment OTDE08TDE; MAP 35.72%; Not Pooled] jhu−apl [Experiment JHUMODE4; MAP 33.77%; Pooled] unine [Experiment UNINEDE1; MAP 30.13%; Pooled] xerox [Experiment T1; MAP 27.37%; Pooled] inesc [Experiment RUN3; MAP 22.98%; Not Pooled] Ad−Hoc TEL Bilingual German Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% jhu−apl [Experiment JHUBIENDE5; MAP 18.99%; Not Pooled] chemnitz [Experiment CUT_MERGED_SIMPLE_EN2DE; MAP 18.52%; Pooled] cheshire [Experiment BKAHTELBENDETDT2FB; MAP 15.56%; Pooled] xerox [Experiment FRE_GER_J; MAP 12.06%; Not Pooled] karlsruhe [Experiment AIFB_ONB_EN; MAP 6.67%; Pooled] 0% 0% 10% 20% 30% 40% 50% Recall 60% 70% 80% 90%
100% 50%
Recall 10% 20% 30% 40% 60% 70% 80% 90% 100%
Track LF TEL@CLEF Monolingual English 10 TEL@CLEF Monolingual French 0 TEL@CLEF Monolingual German 4 TEL@CLEF Bilingual English 0 TEL@CLEF Bilingual French 0 TEL@CLEF Bilingual German 0 PERSIAN@CLEF Monolingual Persian 39 PERSIAN@CLEF Bilingual Persian 4 Robust Monolingual English 1 Robust WSD Monolingual English 1 Robust Bilingual English 0 Robust WSD Bilingual English 0
In the following sections, two different graphs are presented to summarize the results of this test. All experiments, regardless of topic language or topic fields, are included. Results are therefore only valid for comparison of individual pairs of runs, and not in terms of absolute performance. Both for the ad-hoc and robust tasks, only runs where significant differences exist are shown; the remainder of the graphs can be found in the Appendices [17,18,19].
The first graph shows participants’ runs (y axis) and performance obtained (x axis). The circle indicates the average performance (in terms of precision) while the segment shows the interval in which the difference in performance is not statistically significant.
The second graph shows the overall results where all the runs that are included in the same group do not have a significantly different performance. All runs scoring below a certain group perform significantly worse than at least the top entry of the group. Likewise all the runs scoring above a certain group perform significantly better than at least the bottom entry in that group. To determine all runs that perform significantly worse than a certain run, determine the rightmost group that includes the run, all runs scoring below the bottom entry of that group are significantly worse. Conversely, to determine all runs that perform significantly better than a given run, determine the leftmost group that includes the run. All runs that score better than the top entry of that group perform significantly better.
UNINEEN3 UNINEEN2
RUN3
UNINEEN1 CUT_SIMPLE
JHUMOENS
JHUMOEN4RF BKAHTELMENTDT2F
M UNINEEN4 JHUMOEN4 JHUMOEN5 OTEN08TDE
RUN4 RUN1
RUN2 OTEN08TD
F
J1 OTEN08T
S1 BKAHTELMENTT2FB UFRGS_MONO_EN2
IN_EXPC2C10
IPAL02 IPAL01 IPAL03
IPAL04
UFRGS_MONO_EN1 CUT_MULTI10_WX_PLUSPLUS
OTEN08TDZ CACAOENGENGPLAIN
AIFB_BL_EN
CACAOENGENGEXPANDED CUT_MULTI10_W1MINUSX_PLUSPLUS
CUT_MULTI10_W1_PLUSPLUS
AF 0.1 0.1 0.2 Fig. 20. Ad-Hoc TEL Bilingual English. Experiments grouped according to the Tukey T Test.
Ad−Hoc TEL Monolingual French Task − Tukey T test with "top group" highlighted
UNINEFR1 UNINEFR3 UNINEFR2
J1 JHUMOFR4 UNINEFR4
JHUMOFR5 JHUMOFR4RF
F
S1 OTFR08TD
AF
OTFR08TDE BKAHTELMFRTDT2FB
RUN3
OTFR08T IN_EXPC2C10
RUN4 JHUMOFRS
RUN1 OTFR08TDZ
RUN2
CUT_SIMPLE BKAHTELMFRTT2FB
CACAOFREFREPLAIN CUT_MULTI10_WX_PLUSPLUS
CACAOFREFREEXPANDED
CUT_MULTI10_W1_PLUSPLUS CUT_MULTI10_W1MINUSX_PLUSPLUS 0.1 0.2 −0.1 0 0.1 Fig. 22. Ad-Hoc TEL Bilingual French. Experiments grouped according to the Tukey T Test.
Ad−Hoc TEL Monolingual German Task − Tukey T test with "top group" highlighted
OTDE08TDE JHUMODE4 OTDE08TD
JHUMODE5 JHUMODE4RF
UNINEDE1 OTDE08T UNINEDE3 UNINEDE2
T1 UNINEDE4 OTDE08TDZ
RUN3
AF F
S CUT_MERGED_SIMPLE BKAHTELMDETDT2FB
JHUMODES CUT_MULTI10_WX_PLUSPLUS
BKAHTELMDETT2FB
RUN4 RUN2
RUN1 CACAOGERGERPLAIN
IN_EXPC2C10 CACAOGERGEREXPANDED
AIFB_ONB_DE
CUT_MULTI10_W1_PLUSPLUS CUT_MULTI10_W1MINUSX_PLUSPLUS 0.1 Fig. 23. Ad-Hoc TEL Monolingual German. The figure shows the Tukey T Test. s t n e imCUT_MERGED_SIMPLE_MULTI10_W1_EN2DE r e p x E
PRF2
F AIFB_ONB_EN AIFB_ONB_FR
ENG_F
CACAOENGGERPLAIN CACAOENGGEREXPANDED
CACAOFREGERPLAIN CACAOFREGEREXPANDED 0 Fig. 24. Ad-Hoc TEL Bilingual German. Experiments grouped according to the Tukey T Test.
UNINEPE2 UNINEPE4 UNINEPE3
UNINEPE1 JHUFASK41R400
JHUFA5R100 JHUFAWR50 JHUFAMR50 OTFA08T OTFA08TD
OTFA08TDE UTNLPDB3INEXPC2
UTNLPDB3PL2 UTNLPDB3INEXPB2
UTNLPDB3OWA UTNLPDB3NOWA UTNLPDB3BB2 UTNLPDB3DFR UTNLPDB1MT UTNLPDB2TON UTNLPDB3IFB2 UTNLPDB3INL2
KMEANS5
KMEANS2 PDDP_2MEANS2
KMEANS4 KMEANS1
STN MLUSR
KMEANS3 UTNLPDB2TDN UTNLPDB1M5G PDDP_2MEANS4 PDDP_2MEANS3 PDDP_2MEANS5 PDDP_2MEANS6 UTNLPDB1M4G PDDP_2MEANS1
MLSR STD
PDDP UTNLPDB3TFIDF UTNLPDB3BM25 UTNLPDB1M3G OTFA08TDNZ
TTO
TTD UTNLPDB2TD3 UTNLPDB2TDW UTNLPDB2TNA UTNLPDB2TA3 UTNLPDB2TOT UTNLPDB2OTT 0.2
JHUENFA5R100 JHUENFAWR50 JHUENFAMR50 UTNLPDB1BT5G UTNLPDB1BT4G
CLDTDR UTNLPDB1BA10 UTNLPDB1BT1 UTNLPDB1BT5
CLQTR UTNLPDB1BA
CLQTDR 0.1 0.2 0.3 0.6 0.7
0.8 0.4 0.5 arcsin(sqrt(Average Precsion)) Fig. 26. Ad-Hoc Bilingual Persian. Experiments grouped according to the Tukey T Test.
Ad−Hoc Robust Monolingual English Test Task − Tukey T test with "top group" highlighted UNINEROBUST4 UNINEROBUST1
ISILEMTDN EN2ENNOWSDPSREL
BM25_BO1_AVICTF
BM25_BO1 BM25_KLD
ISILEMTD
EN2ENNOWSD UFRGS_R_MONO2_TEST UFRGS_R_MONO1_TEST
INAOEF
ASSO
INAOEV MONO11NUS2F MONO1TDNUS2F
MONO13NUS2F MONO12NUS2FOUT
MONO14NUS2F 0.1 0.2 0.3
0.4 0.5 arcsin(sqrt(Average Precsion)) 0.6 0.7 0.8 Fig. 27. Robust Monolingual English. Experiments grouped according to the Tukey T Test.
Ad−Hoc Robust Bilingual English Test Task − Tukey T test with "top group" highlighted UFRGS_R_BI3_TEST
ISIESENTD
UFRGS_R_BI2_TEST tsn UFRGS_R_BI1_TEST e m i r e xpES2ENNOWSDPSREL E
ES2ENNOWSD CROSS1TDNUS2F
CROSS1NUS2F 0 0.1 0.2
Experiment DOI Groups 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.UFRGS.UFRGS R BI3 TEST X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.GENEVA.ISIESENTD X X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.UFRGS.UFRGS R BI2 TEST X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.UFRGS.UFRGS R BI1 TEST X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.IXA.ES2ENNOWSDPSREL X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.IXA.ES2ENNOWSD X 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.UNIBA.CROSS1TDNUS2F 10.2415/AH-ROBUST-BILI-X2EN-TEST-CLEF2008.UNIBA.CROSS1NUS2F Fig. 28. Robust Bilingual English. Experiments grouped according to the Tukey T Test.
Ad−Hoc Robust Word Sense Disambiguation Monolingual English Test Task − Tukey T test with "top group" highlighted 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 arcsin(sqrt(Average Precsion)) Fig. 29. Robust WSD Monolingual English. The figure shows the Tukey T Test.
Ad−Hoc Robust Word Sense Disambiguation Bilingual English Test Task − Tukey T test with "top group" highlighted UFRGS_R_BI_WSD1_TEST ES2EN1STTOPSNUSDOCSPSREL ES2EN1STTOPSUBCDOCSPSREL
ES2ENNUSDOCSPSREL ES2ENUBCDOCSPSREL
ISIESPWSDTDN
ISIESPWSDTD CROSSWSD12NUS2F CROSSWSD1NUS2F CROSSWSD11NUS2F The ad hoc task this year has been almost completely renovated with new collections and new tasks. For all three tasks, we have been very happy with number of participants. However, it is really too soon to be able to provide any deep analysis of the results obtained. This is left to the post-workshop proceedings. In any case, it is our intention to run all three tasks for a second year both in order to provide participants with another chance to test their systems after refinement and tuning on the basis of this year’s experiments and also to be able to create useful and consolidated test collections. In particular, for both the TEL and Persian tasks, we intend to perform some experiments on this year’s test collections in order to verify their stability. The results will be reported in the Proceedings.
From our first impressions of the results of the TEL task, it would appear that there is no need for systems to apply any dedicated processing to handle the specificity of these collections (very sparse, essentially multilingual data) and that traditional IR and CLIR approaches can perform well with no extra boosting. However, we feel that it is too early to make such assumptions; many more experiments are needed.
The Persian task continued in the tradition of the CLEF ad hoc retrieval tasks on newspaper collections. The first results seem to confirm that the traditional IR/CLIR approaches port well to ”new” languages - where by ”new” we intend languages which have not been subjected to a lot of testing and experimental IR studies previously.
The robust exercise had, for the first time, the additional goal of measuring to what extent IR systems could profit from automatic word sense disambiguation information. The conclusions are mixed: while some top scoring groups did manage to improve the results using WSD information by aprox. 1 MAP percentage point (aprox. 4 MAP percentage points in the cross-language exercise) and the best monolingual GMAP score was for a WSD run (0.27 percentage points), the best scores for the rest came from systems which did not use WSD information. Given the relatively short time that the participants had to try effective ways of using the word sense information we think that these results are positive, but we think that a subsequent evaluation exercise would be needed for participants to further develop their systems. 8
The TEL task was studied in order to provide useful input to The European Library (TEL); we express our gratitude in particular to Jill Cousins, Programme Director, and Sjoerd Siebinga, Technical Developer of TEL. Vivien Petras, GESIS-IZ Social Science Information Centre, Germany, and Nicolas Moreau, Evaluation and Language Resources Distribution Agency, France, were responsible for the creation of the topics and the supervision of the relevance assessment work for the ONB and BNF data respectively. We thank them for their valuable assistance.
We should also like to acknowledge the enormous contribution to the coordination of the Persian task made by the Data Base Research group of the University of Tehran and in particular to Abolfazl AleAhmad and Hadi Amiri. They were responsible for the preparation of the set of topics for the Hamshahri collection in Farsi and English and for the subsequent relevance assessments.
The robust task was partially funded by the Ministry of Education (project KNOW TIN2006-15049) and the European Commission (project KYOTO ICT2007-211423). We would like to acknowledge the invaluable suggestions and practical assistance of Arantxa Otegi, German Rigau and Piek Vossen. In particular, Arantxa was responsible for the XML coding of the word sense disambiguation results. We also want to thank Oier Lopez de Lacalle, who runs the UBC WSD system, and Yee Seng Chan, Hwee Tou Ng and Zhi Zhong, who run the NUS WSD system. Their generous contribution was invaluable to run this exercise. 13. Cleverdon, C.: The Cranfield Tests on Index Language Devices. In Sparck Jones, K., Willett, P., eds.: Readings in Information Retrieval, Morgan Kaufmann Publisher, Inc., San Francisco, California, USA (1997) 47–59 14. Clinchant, S., Renders, J.-M.: XRCE’s Participation to CLEF 2008 Ad-Hoc Track.
In this volume. 15. Conover, W.J.: Practical Nonparametric Statistics. 1st edn. John Wiley and Sons,
New York, USA (1971) 16. Costa Acosta, O., Geraldo, A.P., Orengo, V.M., Villavicencio, A.: UFRGS@CLEF2008: Indexing Multiword Expressions for Information Retrieval.
In this volume. 17. Di Nunzio, G.M., Ferro, N.: Appendix A: Results of the TEL@CLEF Task. In Borri, F., Nardi, A., Peters, C., eds.: Working Notes for the CLEF 2008 Workshop, http://www.clef-campaign.org/ [last visited 2008, September 5] (2008) 18. Di Nunzio, G.M., Ferro, N.: Appendix B: Results of the Persian Task. In Borri, F., Nardi, A., Peters, C., eds.: Working Notes for the CLEF 2008 Workshop, http: //www.clef-campaign.org/ [last visited 2008, September 5] (2008) 19. Di Nunzio, G.M., Ferro, N.: Appendix C: Results of the Robust Task. In Borri, F., Nardi, A., Peters, C., eds.: Working Notes for the CLEF 2008 Workshop, http: //www.clef-campaign.org/ [last visited 2008, September 5] (2008) 20. Dolamic, L., Fautsch, C., Savoy, J.: UniNE at CLEF2008: TEL, Perisan and Robust
IR. In this volume. 21. Geraldo, A.P., Orengo, V.M.: UFRGS@CLEF2008: Using Association Rules for
Cross-Language Information Retrieval. In this volume. 22. Guyot, J., Falquet, G., Radhouani, S., Benzineb, K.: UNIGE Experiments on
Robust Word sense Disambiguation. In this volume. 23. Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In Korfhage, R., Rasmussen, E., Willett, P., eds.: Proc. 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993), ACM Press, New York, USA (1993) 329–338 24. Jadidinejad, A.H:, Mohtarami, M., Amiri, H.: Investigation on Application of Local
Cluster Analysis and Part of Speech Tagging on Persian Text. In this volume. 25. Judge, G.G., Hill, R.C., Griffiths, W.E., Lu¨tkepohl, H., Lee, T.C.: Introduction to the Theory and Practice of Econometrics. 2nd edn. John Wiley and Sons, New York, USA (1988) 26. Karimpour, R., Ghorbani, A., Pishdad, A., Mohtarami, M., AleAhmad, A., Amiri, H., Oroumchian, F.: Using Part of Speech tagging in Persian Information Retrieval.
In this volume. 27. Kuersten, J., Wilhelm, T., Eibl, M.: CLEF 2008 Ad-Hoc Track: On-line Processing
Experiments with Xtrieval. In this volume. 28. Larson, R.: Logistic Regression for Metadata: Cheshire takes on Adhoc-TEL. In this volume. 29. Machado, J., Martins, B., Borbinha, J.: Technical University of Lisbon CLEF 2008
Submission (TEL@CLEF Monolingual Task). In this volume. 30. Mart´ınez-Santiago, F., Perea-Ortega, J.M., Garc´ıa-Cumbreras, M.A.: SINAI at Robust WSD Task @ CLEF 2008: When WSD is a Good Idea for Information Retrieval Tasks. In this volume. 31. MacNamee, P.: JHU Ad Hoc Experiments at CLEF 2008. In this volume. 32. Navarro, S., Llopis, F., Mun˜oz, R.: IRn in the CLEF Robust WSD Task 2008. In this volume.