This year, we have participated on multilingual two years on and Multi-8 merging-only CLEF task. Our main interest has been to test several usual CLIR tasks and investigate how they a®ect to the ¯nal performance of the multilingual system. Speci¯cally, we have evaluated the information retrieval model used to obtain each monolingual result, the merging algorithm, the translation approach and the application of query expansion techniques. The obtained results show that by means of improving merging algorithms and translation resources we reach better results than improving other CLIR modules such as IR engines or the expansion of queries.
1.1
The merging algorithm In short, the basic 2-step RSV idea is straightforward: given a query term and their translations into the other languages, their document frequencies are grouped together. Therefore, the method requires recalculating the document score by changing the document frequency of each query term. Given a query term, the new document frequency will be calculated by means of the sum of the monolingual retrieved document frequency of the term and their translations. In the ¯rst step the query is translated and searched on each monolingual collection. This phase produces a T0 vocabulary made up by \concepts". A concept consists of each term together with its corresponding translation. Moreover, we obtain a single multilingual collection D0 of preselected documents as result of the union of the ¯rst 1000 retrieved documents for each language. The second step consists of creating a dynamic index by re-indexing the multilingual collection D0, but considering solely the T0 vocabulary. Finally, a new query formed by concepts in T0 is generated and this query is carried out against this dynamic index.
Thus, the ¯rst step of 2-step RSV consists of retrieving relevant documents for each language
and the alignment of the query and its translations. This year we have tested the performance
of the algorithm by testing several information retrieval engines, used for retrieving relevant
documents for each monolingual collection, and then applying the second step of the merging
algorithm over the retrieved documents. To sum up, we have tested ZPrise with OKAPI weighting
function[6], IRn passage system[
The basic process is the following. In a ¯rst step each monolingual collection is preprocessed as
usual (tokens extraction, stopper, stemmer). In addition, compound words are decompounded
as possible to the German, Swedish, Finnish and Dutch languages. We use the decompounding
algorithm depicted in [
Since we have used machine translation for several languages, and 2-step RSV requires to
group together the document frequency for each term and its own translations, and MT translates
the whole of the phrase better than word-by-word, 2-step RSV merging algorithm is not directly
feasible with MT (given a word of the original query, its translation to the rest of languages must
be known). Thus, we propose in [
In spite of the proposed algorithm to align phrases and translations at term level works ¯ne, it does not obtain fully aligned queries. In order to improve the system performance when some terms of the query are not aligned, we make two subqueries. The ¯rst one is made up by the aligned terms only and the other one is formed with the non-aligned terms. Thus, for each query every retrieved document obtains two scores. The ¯rst score is obtained with 2-step RSV merging algorithm over the ¯rst subquery. On the other hand, the second subquery is used in a traditional monolingual system with the respective monolingual list of documents. Therefore, we have two scores for each query, the ¯rst one is calculated by using the dynamic and global index created by 2-step RSV for all languages and the other one is calculated locally for each language. Thus we have to integrate both values. As a way to deal with partially aligned queries (i.e. queries with some terms not aligned), we have implemented several ways to combine the aligned and non-aligned score in a only score per query and retrieved document: 1. Raw mixed 2-step RSV. Combining the RSV value of the aligned words and not aligned words with the formula: 0:6¤ < RSValigneddoc > +0:4¤ < RSVnotaligned > 2. Mixed 2-step RSV by using Logistic Regression. The formula: e(alpha¤ < RSValigneddoc > +beta¤ < RSVnotaligned >) 3. Mixed 2-step RSV by using Logistic Regression and local score. The last one also uses Logistic Regression but include a new component, the ranking of the doc. It applies the formula: e(alpha¤ < RSValigneddoc > +beta¤ < RSVnotaligned > +gamma¤ < rankingdoc >) 4. Mixed 2-step RSV by using Bayesian Logistic Regression and local score. The last one is very similar to the previous approach, but it is based on bayesian logistic regression instead of logistic regression.
Versions two, three and four require training set (topics and their relevance assessments), and it must be available for each monolingual collection. We have used CLEF queries(140-160) relevance assessments available this year for training purposes. Thus, twenty ¯rst queries have been used as training and the other forty have been used for evaluation. 3
Some experiments based on ZPrise use pseudo-relevance feedback technique. We have adopted
Robertson-Croft's approach [
The second step of the merging method does not make use of automatic query expansion techniques such as relevance feedback (RF) or pseudo-relevance feedback (PRF) applied to monolingual queries. Since RF and PRF extend every monolingual query with collection-dependent words, the reindexing process (second step of 2-step RSV) will not take into account all of these words. Because such words are not the same for each monolingual collection, and the translation to the other languages is unknown, 2-step RSV method ignores these new terms for the second step. However, the overall performance will also improve since PRF and RF improve on monolingual experiments and usually some extended terms coincide with terms of the original query, and such terms will be aligned. Rest of expanded terms are integrated as non-aligned terms by using the approaches depicted in the section 2 for mixed 2-step RSV. Of course, the percentage of non-aligned words is increased because of the application of PRF. Table 2 shows the percentage of aligned words for expanded queries by using PRF and Machine Translation.
The tables 3, 4, 5 show the o±cial results. In order to evaluate the translation approach e®ect in the multilingual result, we have recovered some old experiments from CLEF 2003 for 161-200 CLEF queries(experiment ujarsv2 2003). Such experiments are based on Machine Dictionary Readable resources, and we have compared it with results of this year (experiment UJARSV2), based on Machine Translation. In order to evaluate the e®ect of query expansion we have developed experiments ujaprfrsv2 and UJAPRFRSV2RR. Finally, experiments UJARSV2RR, UJAUARSV2RR, UJAMENEOKRR or UJAMENEDERR use several IR systems and models to obtain the list of retrieved documents. This table shows some interesting results: ² Note that the improvement for this year is considerable respect of 2003, mainly because of a best translation strategy. ² In spite of the very di®erent performance of the bilingual experiments (Table 6), ¯nal multilingual average precision is very similar independently of the selected documents for each IR system. ² Since the simultaneous application of PRF and Machine Translation decreases dramatically the percentage of aligned words, the application of PRF improves very slightly the ¯nal result. ² Good performance of raw-mixed 2-step RSV, obtaining a result very near to the result reached by means of logistic regression and neural networks. This result is counterintuitive since the method adds two values which are not directly comparable: the score obtained by both aligned and non-aligned terms. Some of the reasons for this good result are: { ® parameter limits the weight of the unaligned factor. { Not all the terms to be added to the original query are new terms since some terms obtained by means of pseudo-relevance feedback are in the initial query. Thus, these terms are aligned terms. In the same way this explains the good performance of 2-step RSV original method with expanded queries. { Only 20 queries available for training. { CLEF document collections are highly comparable (news stories from the same period).
The results might be di®erent if collections have vastly di®erent sizes and/or topics.
Thus, 2-step RSV reaches the same precision in spite of using di®erent IR systems. This is a drawback if the IR system used for the ¯rst step implements a IR model more sophisticated than the IR model implemented for the second step of the algorithm. In such situation, the improvement is not fully exploited by 2-step RSV merging algorithm because 2-step RSV creates a dynamic index based on classic document retrieval models (more precisely the dynamic index is created by using a document-based OKAPI weighting schema). So, what should we do to improve these results?. Since the second step is basically an OKAPI IR engine, we could improve such engine by using better IR models, and improving the translation and alignment processes. 5
In this work, we have tested the merging algorithm 2-step RSV in several ways. We have compared the CLEF 2003 and CLEF 2005 Multi-8 results, by using CLEF 160-200 queries. This year we have obtained better results than the 2003 edition. We think that the main reason is a better translation approach and a more re¯ned version of the merging algorithm.
The obtained results show that the improvement of merging algorithms and translation resources are higher than the improvement obtained by expanding the query by means of pseudorelevance feedback.
In the same way, the improvement of the monolingual IR System used to retrieve each monolingual list of documents obtains very slightly better results in the ¯nal multilingual system. In order to evaluate the impact of the monolingual IR system, we have evaluated several lists of retrieved documents by using two IR systems and some of the retrieved documents available for the Multi-8 Merging-only task, but holding the same translation approach and merging algorithm. Results show that the precision is very similar independently of the monolingual IR engine. We conclude that improvements in the selection of documents by using some monolingual IR engine is not fully exploited by 2-step RSV merging algorithm since this algorithm creates a dynamic index based on classic document retrieval models.
When pseudo-relevance feedback and machine translation is applied in the same experiment, the percentage of aligned words is too low to apply optimally some mixed variant of 2-step RSV. Thus, a more e®ective word alignment algorithm must be developed, especially for the new terms added to the query by means of PRF.
Finally, we think that the overall performance of the CLIR system will be overcome if we develop better translation strategies and we improve the IR model used for the creation of the dynamic index for the second step of the algorithm. 6
Thank you very much to Fernando LLopis and Elisa Noguera from University of Alicante for their support with the IRn package.
Thank you very much to Jacques Savoy and Gareth Jones to supply the results of several bilingual experiments for the Multi-8 Merging-only task and the administration of the web place.
This work has been supported by Spanish Government (MCYT) with grant TIC2003-07158C04-04. [6] S.E. Robertson and S.Walker. Okapi-Keenbow at TREC-8. In Proceedings of the 8th Text Retrieval Conference TREC-8, NIST Special Publication 500-246, pages 151{162, 1999.