This paper describes our work at CLEF 2007 Robust Task. We have participated in the monolingual (English, French and Portuguese) and the bilingual (English to French) subtask. At CLEF 2006 our research group obtained very good results applying local query expansion using windows of terms in the robust task. This year we have used the same expansion technique, but taking into account some criteria of robustness: MAP, GMAP, MMR, GS@10, P@10, number of failed topics, number of topics bellow 0.1 MAP, and number of topics with P@10=0. In bilingual retrieval experiments three machine translation programs were used to translate topics. For the target language, translations were merged before performing a monolingual retrieval. We also applied the same local expansion technique. This year the results were disappointing. We think out that the reason is the difficulty to select the best measurement for robustness. Perhaps the problem is that all measurements are average results over all topics, but the hard topics are inherently hard and must be analyze separately. This year all our runs also ends up in good ranking, both base runs and expanded ones. We think that the reason is that we used a good information retrieval system, and the expansion technique is robust because it does not deteriorate significantly the retrieval performance.
Robust retrieval tries to obtain stable performance over all topics by focusing on poorly performing
topics. Robust tracks were carried out at TREC 2003, 2004 and 2005 for monolingual retrieval [
The system’s robustness ensures that all topics obtain minimum effectiveness levels. In
information retrieval the mean of the average precision (MAP) is used to measure systems’
performance. But, poorly performing topics have little influence on MAP. At TREC, geometric average
(GMAP), rather than MAP, turned out to be the most stable evaluation method for robustness
[
Our main focus was monolingual retrieval. The steps followed are explained below. For bilingual retrieval experiments we used machine translation (MT) programs to translate topics into document language, and then we performed a monolingual retrieval. 2
For the monolingual experiments we used the well-known vector space model, using the dnu-ntc term weighting scheme. For documents, letter u stands for the pivoted document normalization: we adjusted pivot to the average document length and slope set to 0.1 for all the collections. We decided to remove the terms present in more than 25 percent of documents. For the English and French languages we verified that stemming improve retrieval. Last year we saw that stemming does not deteriorate the retrieval performance of hard topics, so we also decided to apply stemming for the Portuguese language. For English we used the Porter stemmer, and for French and Portuguese the stemmers from the University of Neuchatel in the web page http://www.unine.ch/info/clef/. From the descriptions and narratives of the topics we automatically removed certain phrases such as “Find documents that . . . ”, “Les documents pertinents relatent . . . ” or “Encontrar documentos sobre . . . ”.
At CLEF 2006 Robust Task our research group obtained very good results applying local query
expansion using windows of terms [
For Portuguese we decided to use the best combination obtained last year for the Spanish experiments, due two reasons. First, the Portuguese language is more similar to Spanish than English or French are. Second, the average number of terms per sentence in the Portuguese collection is very similar to the Spanish one. We use a distance value of 2, taking the first 10 documents, and adding 30 terms to the original query.
For the bilingual experiments the CLIR system was the same as that used in monolingual retrieval. A previous step was carried out before searching, to translate English topics into French. We used three MT programs: L&H Power Translator Pro 7.0, Systran1 and Reverso2. For each topic we combined the terms of the translations in a single topic: this is another expansion process, 1http://www.systransoft.com 2http://www.reverso.net
Basis
t 0.3226 0.1190 0.5602 0.7613 0.3200 5 16 16 0.3382 0.0940 0.5749 0.7555 0.3710 9 18 23 0.3387 0.0825 0.5711 0.7307 0.3013 15 28 36 0.3035 0.0821 0.5819 0.7555 0.3242 9 16 22 although in most cases the three translations were identical. Finally, a monolingual retrieval was performed. The local query expansion using co-occurrence based thesauri built with terms windows was also applied.
For each subtask and topic language five runs were submitted for the test and training topics. The name of the run begins with “reina”, follows the abbreviation of the language (EN, FR or PT for the monolingual runs, and E2F to indicate the English to French bilingual runs), follows the fields of the topics used in the run (t: title, td: title and description, tdn: title, description and narrative), follows with the letter “e” to indicate if expansion of terms was used and/or the letter “T” to indicate if the run is a test run. For example, the run “reinaENtdeT” stands for the test run submitted for the English collection using the title and descriptions fields of the topics, and applying term expansion. We send the “tdn” runs only for internal testing purposes. 3
We only analyze results of our test runs, i.e., for the test topics of the robust task. Table 1 shows the results of the runs. We can see that term expansion no improves performance for all measurements. At CLEF 2006 Robust Task our research group obtained very good results applying local query expansion using windows of terms in the robust task. This year at CLEF 2007 the results were disappointing. We think out that the reason is the difficulty to select the best measurement for robustness. Perhaps the problem is that all measurements are average results over all topics, but the hard topics are inherently hard and must be analyse separately. When a topic becomes hard depends on the document collection, the topic collection, the information retrieval system and the topic itself. Therefore general directives to improve performance of hard topics are difficult to suggest.
This year all our runs also ends up in good ranking, both base runs and expanded ones. We think that the reason is that we used a good information retrieval system, and the expansion technique is robust because it does not deteriorate significantly the retrieval performance.