This article describes our participation at the Domain-Speci c track. We used the Xtrieval framework [2], [3] for the preparation and execution of the experiments. The translation of the topics for the cross-lingual experiments was realized with a plug-in to access the Google AJAX language API2. This year, we submitted 20 experiments in total. In all our experiments we applied a standard top-k pseudo-relevance feedback algorithm. Also, all of our submissions were merged experiments, where multiple stemming approaches for each language were combined to improve retrieval performance. The evaluation of the experiments showed that the combination of stemming methods works very well. Translating the topics for the bilingual experiments deteriorated the retrieval e ectiveness only between 8 and 15 percent in comparison to our best monolingual experiments.
The approach we used for this years participation is mainly based on the following ideas. At rst we combine
several stemming methods for each language in the retrieval stage. The combination of the results was done
by our implementation of the Z-Score operator [
The detailed setup of our experiments and the results of the evaluation are presented in the following subsections. 3.1
We submitted 5 monolingual experiments in total, 2 for the English and the German subtasks and 1 for the
Russian subtask. For all experiments a language-speci c stopword list was applied4. We used di erent
stemmers for each language: Porter5 and Krovetz [
We submitted 12 experiments in total for the bilingual subtask, i.e. 4 experiments were submitted for each target language collection. We compared the translation from di erent source languages and the performance of pure topic translation with combined translation. For the combined translation we used the pure topic translation and tried to improve the translation with the help of the bilingual thesauri, i.e. for every term occurring in the bilingual thesauri we added its provided translation to the topic. Again, we used a standard pseudo-relevance feedback algorithm to improve retrieval e ectiveness. In Table 2 we compare each of the bilingual experiments with respect to the performance of the corresponding monolingual experiment. Probably due to the quality of Google's translation service and the strong performance of our monolingual runs the retrieval e ectiveness of our bilingual experiments is also very good. Surprisingly one of our bilingual experiments on the Russian target collection performed best, although our monolingual experiment had the worst overall performance. This is thought to be due to the smaller number of submissions for the bilingual 4http://members.unine.ch/jacques.savoy/clef/index.html 5http://snowball.tartarus.org 6http://www-user.tu-chemnitz.de/~wags/cv/clr.pdf The following list provides a summary of the analysis of our retrieval experiments for the Domain-Speci c track at CLEF 2008:
Monolingual: The performance of our monolingual experiments was very good for the German and English collections and worse for the Russian collection. Interestingly, the retrieval e ectiveness could not be improved by utilizing the provided domain-speci c thesauri for query expansion. 7http://www.clef-campaign.org/2005/working notes/workingnotes2005/appendix a.pdf - p. 61 8http://www.clef-campaign.org/2006/working notes/workingnotes2006/Appendix Domain Speci c.pdf - p. 63 9http://www.clef-campaign.org/2007/working notes/AppendixC.pdf - p. 206
Bilingual: Probably due to the used translation service our bilingual experiments performed very well and achieved the best results on each target collection. Astonishingly, we could not improve the retrieval performance by using the provided bilingual thesauri.
Multilingual: Again, mainly due to the quality of the translation and the result list combination capabilities of the Xtrieval framework we achieved very impressive results in term of retrieval e ectiveness.
There was no signi cant di erence between the experiments with English and German topics. 5
This year, we achieved very good retrieval performance in almost all subtasks of the Domain-Speci c track. Since our main research focus shifted to Multimedia Information Retrieval there were no interesting contributions to retrieval community in this work, except for the fact that combining di erent stemming approaches helped to improve retrieval performance. Another important observation in all our experiments for this years CLEF campaign was that the translation service provided by Google seems to be extremely superior to any other approach or system. This should motivate the cross-language community to investigate and improve their current approaches.
We would like to thank Jaques Savoy and his co-workers for providing numerous resources for language processing. Also, we would like to thank Giorgio M. di Nunzio and Nicola Ferro for developing and operating the DIRECT system10.
This work was partially accomplished in conjunction with the project sachsMedia, which is funded by the Entrepreneurial Regions 11 program of the German Federal Ministry of Education and Research.