Fusion and optimization based on relevance judgements have proven to be successful strategies in information retrieval. In this year's CLEF campaign we applied these strategies to multilingual retrieval with four languages. Our fusion experiments were carried out using freely available software. We used the snowball stemmers, internet translation services and Lucene's and the new MySQL's text retrieval tools. In the CLEF 2002 campaign, we tested an adaptive fusion system based on the MIMOR model within the GIRT track (Hackl et al. 2002). For CLEF 2003, we applied the same model to multilingual retrieval with four languages. We chose English as our source language because most of the web based translation services offer translations to and/or from English. Our experiments were carried out fully automatically.
to evaluate within evaluation initiatives. MIMOR could train an individual or group based optimization of the fusion. However, in evaluation studies, a standardized notion of relevance exists. 4
The tools we employed this year include Lucene 1.31, MySQL 4.0.122 and JavaTM-based snowball3 analyzers. Most of the data pre-processing was carried out by Perl-scripts. In a first step, customized snowball stemmers were used to stem the collections. Stopwords were also eliminated4. Then, the collections were indexed by Lucene and MySQL. Lucene needed less than half the time that MySQL needed for indexing the collections of 1321 MB. A second step involved the translation of the English topics into French, German and Spanish. The translation was carried out with the free internet services FreeTranslation, Reverso and Linguatec5. The decision to select these tools, was based on a heuristic evaluation of several services. The queries of CLEF 2001 were used to gather data for a comparison of the translations. Examining the different translations, it became apparent that the quality of the machine translations is certainly not quite satisfying. At the same time, the translation systems usually exhibited different weaknesses. Because of that, we decided to use more than one translation system and merge the results. The tools which performed best and showed significantly different results at our evaluation were chosen.
The topics were also stemmed with snowball and stopwords were removed. The translated and processed queries for each language were then merged by joining the three translations while eliminating dublettes. We did not try to identify any phrases. Before working on the official runs, both retrieval systems employed were tested. Using the data (collections and relevance assessments) from 2001 we carried out several runs. Despite their dissimilar stand-alone performances, the systems were granted equal weights for the fusion process at first (1:1). After four runs, the weights strongly favoured Lucene and we went on experimenting with the 2002 data. The peak performance of the fusion was reached at a ratio of 7:1 (= 0.875:0.125) favouring Lucene's results. This suggests that some of MySQL’s best relevant results helped the overall precision. (cf. table 1). Despite the low retrieval quality of MySQL, it still contributed to the fusion. Note, however, that we did not include the Italian collections and that we used the “perfect”, that is, monolingual, queries in our tests, so there may be some bias. Italian was part of the 2001 and 2002 campaign, but it is not part of the multilingual-4 track in CLEF 2003. 1 Lucene: http://jakarta.apache.org/lucene/docs/index.html 2 MySQL: http://www.mysql.com/ 3 Snowball: http://jakarta.apache.org/lucene/docs/lucene-sandbox/snowball/ 4 We employed the stopword lists at http://www.unine.ch/Info/clef/ and manually added some words. 5 Linguatec Personal Translator: http://www.linguatec.net/online/ptwebtext/index.shtml
Reverso: http://www.reverso.net/, Free Translation: http://www.freetranslation.com/ According to our analysis, MySQL’s weak performance can be partly explained by the following factors. When building a fulltext index, MySQL automatically filters the content of the table column(s) to be indexed, i.e. it also tries to remove stopwords and words with a length of three or less characters. If not explicitly changed, an English stopword list is used by default. This signifies that the indices of the English collections might have been altered slightly compared to the indices of other collections. The removal of very small words might have had a bigger impact on all collections. Because stemming was carried out separately before importing the data into the database, it is very likely that several stemmed words did have a length of three characters or less and therefore were not included in the index. As we later found out, the variable controlling the length of the words to be discarded can be changed, in which case the indices would have to be rebuild. We estimate that MySQL ignored approximately 5 to 10 % of the query and document terms due to this setting.
BLIND RELEVANCE FEEDBACK
Multilingual
merged
To further improve retrieval quality, blind relevance feedback (BRF) was implemented. We selected expansion
terms with either the Robertson selection value (RSV) or the Kullback-Leibler (KL) divergence measure
Queries
German merged English merged French merged Spanish merged 5059 5216 5157 5227
Due to time constraints, we could not determine the best parameters for BRF. A sample run without BRF took 4+ hours on our dual Pentium III 800Mhz, 1GB RAM, SCSI 160 HDD machine. A run with BRF taking the top five documents and adding ten terms commonly took more than twelve hours. Unfortunately, some instabilities in MySQL-DB caused further delay for our experiments. 0.3017 0.3138 0.3049 0.3127 All our submitted runs apply BRF KL 5 20, on behalf of the multilingual task, R1 uses the 7:1 merging scheme, yet R2 is a lucene-only run. Both monolingual runs are rather a by-product obtained in the course of our main (multilingual) task. The processing sequence chosen allowed for a efficient extraction of the monolingual data. In our test runs, we were able to show that fusion helped raise at least the recall, although the results for 2003 could not confirm this finding. The Lucene-based runs generally outperform the fusion runs, except for a marginally better recall in the merged monolingual run (Table 3).
5
951 / 1006 945
0.2849 0.3057 As in 2002, we included stand-alone retrieval systems in our MIMOR fusion approach. Again, one system performed much worse than the other. Next year, we would like to expand to the multilingual-8 track and continue to work on merging schemes, query expansion and term weighting.
We would like to thank the Jakarta and Apache projects’ teams for sharing Lucene with a wide community as well as the providers of MySQL and snowball. Furthermore, we acknowledge the work of several students from the University of Hildesheim who implemented MIMOR as part of their course work.