Corpus-based approaches to CLIR have been studied for many years. However, using commercial MT systems for CLEF has been considered easier and better performing. Our goal is to be one of the CLEF participants who show that the hypothetical performance drop is not large enough to justify the loss of control and transparency, especially for research systems. We participated in two bilingual runs and the small multilingual run using software and data that are free to obtain, transparent and modifiable. Over the past years, a necessary condition for a good cross- or multi-lingual performance in CLEF appeared to be the use of commercial MT systems, be it purchased or freely available online (Systran etc.)[1,3,11]. While using black boxes to cross the language barrier allowed researchers to concentrate on important issues such as stemming, query pre- and post-processing, combining black boxes outputs, and multilingual merging, [1,3,11] we believe that query translation does play an essential role in CLIR, and that understanding, control and transparency are crucial in a research system. Online MT systems can be upgraded, lose their free status, or change parameters at will, making past experiments irreproducible. If such a dependence is permitted, research in IR in general can be similarly reduced to pre- and post- processing of Google I/O. Our goal is to attempt to move away from basing the core of our CLIR research system on a module that cannot be fully understood and modified, to which future access might not be guaranteed, and in which external changes are allowed and sometimes not even detected. The main challenge, however, is to do so while sacrificing as little performance as possible. Our initial attempt to reach this goal (CLEF 2001) was disappointing in this respect, mainly because we disallowed using translation resources entirely and relied on the temporal correspondence between CLEF documents to produce a “parallel” corpus. In CLEF 2003 we relaxed the independence requirement to using transparent data and code, freely available or available for a modest one-time fee, which we can store locally, easily modify, recompile and process, and which cannot change in uncontrollable or undetectable ways. We participated in two bilingual tasks (DE->IT, IT->ES), and the small multilingual task, which involved four languages. Our general approach was to rely on parallel corpora and GIZA++ [8] for query translation, and on Lemur [9] for retrieval. All these resources (as well as the stemmers we used where applicable) fulfill the criteria outlined above. Moreover, with the exception of LDC data, which we did not use in the official runs but did use in preliminary experiments, all these resources are free of charge and publicly available. In section 2 we discuss the parallel data and preprocessing (stemming, stopping etc.). In section 3 we discuss our approach to bilingual retrieval in general as well as approaches for situations where a parallel corpus between the two languages does not exist.
the English versions. Our main focus in bilingual retrieval has been query translation without the use of commercial MT systems, including Systran. In this section we will discuss our bilingual retrieval system using a parallel corpus, as well as the challenge of handling language pairs for which parallel corpora do not exist.
Conceptually, our approach consists of several steps: 1. Parallel corpora and test documents preprocessing 2. Dictionary generation from parallel corpora 3. Pseudo-Relevance Feedback in the source language 4. Query translation 5. Pseudo-Relevance Feedback in the target language 6. Retrieval 3.1
We have used GIZA++ [
Query translation was done on a word-by-word basis. A significant difference from MT or online dictionary based approaches is that instead of using a rank-based cutoff (i.e. the first or first two variants for each word) we are using all translations weighted by their translation probability:
qt = qs•Mst where qt is the query in the target language, qs is the query in the source language, and M st is the translation matrix. M was pruned to 50 translations per word for efficiency reasons.
This is similar to IBM and BBN CLIR approaches [
PRF and retrieval parameters we tuned include the number of documents to be considered relevant, the number of new query words added, the relative weight of added queries (usually 0.5) and term weighting method. There is one such parameter set for each pre- and post- translation query expansion, and for each language pair. However, experiments on CLEF 2001 and 2002 indicated that post-translation query expansion hurts performance by diluting the query in some languages, so the second set of parameters were set to 0 for the bilingual runs. 3.3
The bilingual task this year was more challenging, in that we were aware of no Italian-Spanish or German-Italian parallel corpora. However, since most parallel corpora have English as one of the languages we had the option of using English as a pivot language in two ways: 1. to create a new parallel corpus if there is significant overlap (as described in Section 2). This is the least likely situation, but it does happen in the case where there is an underlying text translated in multiple languages, as it happened with the European Parliament corpus. 2. to translate first to English, then from English. This is where keeping and using translation probabilities is very useful. In traditional MT approaches, where the query is translated as a sentence twice, the (binary) mistakes accumulate, making the original meaning difficult to preserve. We believe the original meaning is easier to preserve when the entire query vector is translated, taking into account the translation probabilities:
qt = qs•Ms2EN • MEN2t where qt is the query in the target language, qs is the query in the source language, and M X2Y is the translation matrix for language X to language Y. 3.4
All our official runs use the Title and Description fields. Relevant parameters are pre-translation feedback documents/terms, whether a new parallel corpus was created or if English was used as a pivot language during translation. It is hard to draw conclusions from the official runs without more extensive experimentation on CLEF 2003 data (to be completed in the final version of the working notes). In particular, we are seeking an explanation for the extremely low relative performance of cmuI2Sparafb. If the run is not buggy, feedback performance is very unstable from one translation method to the other, and from language to language. This would not be a complete surprise, since feedback performance varied dramatically for French and Spanish on our system for CLEF 2001 and 2002 , and is the main reason why our runs are duplicated with their “low feedback” alternative in both bilingual and multilingual tasks. 4
By using English as the query language we have leveraged the parallel corpora that had English as one of the languages. We have experimented with several parallel corpora, but chose the European Parliament proceedings as the corpus for our CLEF submission. We performed bilingual retrieval as described in Section 3, and we used Lemur for English monolingual retrieval. We then merged the results using the two methods described in Section 4.1. The number of feedback documents and words were tuned for each language. 4.1
We examined two simple merging strategies: normalizing the individual scores and two step RSV [
Two-step RSV is a reindexing-based method: top ranked documents from each collection are translated to the topic language, then reindexed. Note that this is fundamentally different from translating the test collection, which we would like to avoid. Only top documents are translated, instead of a large test collection. However, the disadvantage of this method is that translation and reindexing need to be done online. Document caching can somewhat alleviate this problem when there are many queries.
Translation is done on a word-by-word basis, using the translation matrix built from the parallel corpus. We use only the first two translations for efficiency; however, we allocate S slots to each untranslated word and distribute the translated words proportionally to their normalized translation probabilities. Due to lack of running time, official runs had S=3. 4.2
All our official runs use the Title and Description fields. Our main goal in participating in this year’s CLEF was to prove that freedom from opaque, uncontrollable commercial systems does not have to mean poor CLIR performance for European languages. Many conceptual or implementation-related improvements can be made. They include better solutions for using a pivot language, especially when the domains do not match; better morphological processing, pseudo-relevant regression for merging etc. 6