This paper reports on our participation in CLEF 2005's ad-hoc multi-lingual retrieval track. The ad-hoc task introduced Bulgarian and Hungarian as new languages. Our experiments focus on the two new languages. Naturally, no relevance assessments are available for these collections yet. Optimization was mainly based on French data from last year. Based on experience from last year, one of our main objectives was to improve and refine the n-gram-based indexing and retrieval algorithms within our system.
In the CLEF 2004 campaign, we tested an adaptive fusion system based on the MIMOR model
The optimization of the retrieval system parameters was based on the French corpus of last year. The tools
employed this year include Lucene and JavaTM-based snowball2 analyzers as well as the Egothor3 stemmer. In
previous CLEF results it has been pointed out, that a tri-gram index does not produce good results for French
Document fields: only officially permitted document fields were indexed. These were indexed as they were as well as in an extra Field FULLTEXT enclosing all the contents from the other fields.
Origin of query terms: query terms could come from either title or description fields or both.
Phrase queries of ngram terms: length of phrases, Boolean operators for concatenating terms • • •
Rigidity of phrase queries: non-exact phrase queries Blind relevance feedback (BRF): relevant documents/expansion term configuration (with Robertson Selection Value as term weighting scheme) and origin of expansion terms
Weighting for all parameters mentioned above The search field FULLTEXT provided best performance overall. Searching on the other fields by themselves or in combination and with weighting did not yield as good results as the simple full-text approach. The document field employed for BRF mattered much more. Here, best results were obtained with the TEXT field. For all runs, we used the stopword lists made available by the University of Neuchatel and made a few minor changes. 2.1
Query Construction for N-Gram Indexing For phrase queries, the approach that worked best was one that constructed queries as follows: Given a query with 3 grams NG1, NG2, NG3 build the query so that q = “NG1“ OR “NG2“ OR “NG3“ OR “NG1 NG2“ OR “NG2 NG3“ OR “NG1 NG2 NG3“. Of course, such a query is likely to retrieve a lot of documents. Effectively, in almost all cases the system retrieved between 80% and 95% of all documents in the database. As Table 1 shows, these results can greatly be improved by applying a more sophisticated configuration on top of the depicted query construction. One means is to allow phrases to be non-exact match phrases, i.e. allow WITHIN or NEAR-like operations, denoted by slop in the table. Here, the best setting was five, values started getting visibly worse from 10 up. Table 3 gives optimized boost values for the n-gram retrieval experiments. The ratio of these figures has been determined experimentally. It can be seen that title terms are more important than description terms. Moreover, longer phrases are better than short ones, limited by the fact that starting with phrases of length 4, performance began to drop. The single most important issue though are short terms. Phrase queries with only one term are of course just plain term queries. If, however, such a term query contains a term that has a smaller word length than the gram size, and taking into account that stopwords are eliminated, there is strong evidence that that term is highly important. In fact, most of these terms were acronyms or foreign words, e.g. in 2004 topics “g7“, “sida“ (French acronym for AIDS), “mir“ (Russian space station), “lady“ (Diana).
Blind relevance feedback had little impact on n-gram retrieval performance. For some queries, good short terms like those mentioned above were added to the query. However, terms selected by the algorithm received no special weight, i.e. they received a weight of one. Higher weights worsened the retrieval results. Furthermore, considering more than the top five documents for blind relevance feedback did not improve performance. Table 4 summarizes the results the best configurations achieved. Queries that contained terms from both title and description fields from the topic files performed better than those that were based on only one source. The weighting of these terms, however, was a major impact factor. Several experiments with different boost values and blind relevance feedback parameters were carried out for each stemmer. The following tables 6, 7 and 8 show the results for the three stemmers. The parameters settings optimized with the French collection of CLEF 2004 were applied to the multi-lingual collection in 2005. We submitted monolingual runs for Bulgarian, French, Hungarian and domain specific (GIRT), bilingual runs for French and GIRT. For Bulgarian and Hungarian we employed the setting outlined above for two runs each – 4-gram and 5-gram: searching on full text representations, boosting single terms which were shorter than the grams length, using BRF (5 docs, 30 terms), and a slop of 5.
For French, we used the Lucene-stemmer and the settings derived above. Additionally, we carried out a 5gram based run as a comparison to Bulgarian and Hungarian. Both of these monolingual were then reshaped by 4 adding terms tentatively derived from the multilingual European terminology database Eurodicautom . We extracted additional terms from the top three hits from the database, if they were available. At least one of the query terms had to be present in the resulting term list, no special subject domain was chosen. These terms were assigned a weight of one.
In the ad-hoc task, we submitted two English-to-French runs, one of which was enhanced by additional Eurodicautom terms, and one Russian-to-French run, all translated by ImTranslator5. The settings were the same as for the monolingual runs. 4 http://europa.eu.int/eurodicautom/Controller 5 http://freetranslation.paralink.com/ For the participation in CLEF 2005, we could stabilize the n-gram indexing and search. The performance remains worse than for stemming based runs. We compared three stemmers with different parameter settings.
For future participations in ad-hoc tasks, we intend to apply the RECOIN (REtrieval COmponent
INtegrator)6 framework
We would like to thank Nina Kummer and Sarah Risse for including the Egothor stemmer into our system. We also acknowledge the work of Viola Barth and Joachim Pfister who ported the trec_eval tool to Java.