In this year's participation we continued to evaluate open source information retrieval software. We used mainly the system Lucene and experimented with some of the most effective optimization strategies applied in CLEF. The effectiveness of open source and other free tools can be enhanced by these optimization strategies. For most languages, blind relevance feedback leads to considerable improvement. Indexing strategies with n-grams have not led to improvements within Lucene. In the CLEF 2004 campaign, we tested an adaptive fusion system based on the MIMOR model with several mono- and multi-lingual tasks. As a basic retrieval system we employed the open source system Lucene. Our main goal is to measure the quality of open source product in comparison to the best systems at CLEF. We exploit some of the most promising optimization techniques applied at CLEF in order to observe the potential for improvement of standard IR systems like Lucene. This work contributes to the practical application of the results from CLEF. Lucene has proved to be very efficient in CLEF as well as in other projects (e.g. cf. Hackl, Mandl & Schwantner 2004) and is becoming increasingly popular. We expect Lucene to be employed in many more contexts. Therefore, we intend to continue testing its effectiveness within the CLEF campaign. Our basic fusion approach MIMOR is described in more detail in Womser-Hacker 1997 and Mandl & WomserHacker 2004a. MIMOR has already been applied to CLEF experiments (Hackl, Kölle et al. 2004).
The tools we employed this year include Lucene 1.4-final1 and JavaTM-based snowball2 language analyzers. Last year we had also evaluated the MySQL’s full text indexing and search module, but due to their poor performance they were excluded this year. For this year’s participation we focussed on different indexing methods such as different stemmers and n-gram-techniques.
We took part in the monolingual tracks for Russian and Finnish, the bilingual track English to Russian and the multilingual track.
Firstly, we ran some preliminary monolingual experiments on the collections from 2003 without query expansion (Table 1). Note that we did not index the LA Times 1994, as well as the 1994 volumes of the French collections as they were not needed for this year.
Secondly, we had planned to try out every combination of indexing methods for a language, to find out whether there is some fusion potential. This can be seen in the table for Finnish (“Finnish all”), where all result lists from the different indices were merged into a single one.
Russian character handling in Java led to problems which caused a very low performance.
For evaluation of the test runs we used our beta stage Java clone of the official trec_eval program.
For the submitted runs we used the title and descriptor topic fields, which were also mandatory. We applied pseudo-relevance feedback for all tasks. For runs involving Russian, we also created one run without BRF. To translate the queries we used the internet service freetranslation.com3 which provided some surprisingly good translations from English to Russian. The results can be seen in Table 2.
Runs UHImlt1 UHImlt2 UHIenru1 UHIenru2 UHIenru3 UHIru1 UHIru2 UHIru3 UHIfi1 UHIfi2 For Finnish, the performance is quite high. The snowball stemmer works very well.
For Russian, our results are very bad due to some encoding problems. BRF still worked well for Russian under these circumstances. Also, the test runs had indicated that the Lucene stemmer seems very capable of dealing with Russian and it held up to that expectation.
The multilingual runs suffered severely from the obstacles that led to the bad results for Russian. We do also have only limited insight into the usefulness of intertran.com as a translation tool for Finnish.
This year’s Russian tracks posed some challenges we could not easily overcome. Despite working with Java only and unicode-based character sets, the Russian stopwords could not be eliminated. We did not have the resources to work on a more sophisticated approach. 4
Our system is far from well adapted the task. It has about 30 weighting parameters. This year, we could only experiment with a few ones.
Furthermore, in future years we intend to exploit the observed relation between the number of named entities in
topics and retrieval performance
We would like to thank the Jakarta and Apache projects’ teams for sharing Lucene with a wide community as well as the providers of Snowball. Furthermore, we acknowledge the work of several students from the University of Hildesheim who implemented MIMOR as part of their course work.