This year, we have participated on multilingual two years on and Multi-8 merging-only CLEF task. Our main interest has been to test several usual CLIR tasks and investigate how they affect to the final performance of the multilingual system. Specifically, we have evaluated the information retrieval model used to obtain each monolingual result, the merging algorithm, the translation approach and the application of query expansion techniques. The obtained results show that by means of improving merging algorithms and translation resources we reach better results than improving other CLIR modules such as IR engines or the expansion of queries.
In order to evaluate the relevance of several usual CLIR modules, we have made a combination between the collection fusion algorithm 2-step RSV and several IR systems. 2-step RSV collection fusion algorithm is detailed in [4,5]. We depict briefly this algorithm above.
In short, the basic 2-step RSV idea is straightforward: given a query term and their translations into the other languages, their document frequencies are grouped together. Therefore, the method requires recalculating the document score by changing the document frequency of each query term. Given a query term, the new document frequency will be calculated by means of the sum of the monolingual retrieved document frequency of the term and their translations. In the first step the query is translated and searched on each monolingual collection. This phase produces a T 0 vocabulary made up by "concepts". A concept consists of each term together with its corresponding translation. Moreover, we obtain a single multilingual collection D 0 of preselected documents as result of the union of the first 1000 retrieved documents for each language. The second step consists of creating a dynamic index by re-indexing the multilingual collection D 0 , but considering solely the T 0 vocabulary. Finally, a new query formed by concepts in T 0 is generated and this query is carried out against this dynamic index.
Thus, the first step of 2-step RSV consists of retrieving relevant documents for each language and the alignment of the query and its translations. This year we have tested the performance of the algorithm by testing several information retrieval engines, used for retrieving relevant documents for each monolingual collection, and then applying the second step of the merging algorithm over the retrieved documents. To sum up, we have tested ZPrise with OKAPI weighting function [6], IRn passage system [2], and several relevant documents list available from Multi-8 Merging-only task.
The basic process is the following. In a first step each monolingual collection is preprocessed as usual (tokens extraction, stopper, stemmer). In addition, compound words are decompounded as possible to the German, Swedish, Finnish and Dutch languages. We use the decompounding algorithm depicted in [3]. The preprocessed collections have been indexed by using the passage retrieval system IRn and ZPrise. The IRn system has been modified in order to return a list of relevant documents, the documents that contain the relevant passages. Given a query and its translations into the other languages, each query is searched in the corresponding monolingual collection.
Since we have used machine translation for several languages, and 2-step RSV requires to group together the document frequency for each term and its own translations, and MT translates the whole of the phrase better than word-by-word, 2-step RSV merging algorithm is not directly feasible with MT (given a word of the original query, its translation to the rest of languages must be known). Thus, we propose in [3] an straightforward and effective algorithm in order to align the original query and its translation at term level. It aligns about 80-85% of non-empty words (Table 1). In spite of the proposed algorithm to align phrases and translations at term level works fine, it does not obtain fully aligned queries. In order to improve the system performance when some terms of the query are not aligned, we make two subqueries. The first one is made up by the aligned terms only and the other one is formed with the non-aligned terms. Thus, for each query every retrieved document obtains two scores. The first score is obtained with 2-step RSV merging algorithm over the first subquery. On the other hand, the second subquery is used in a traditional monolingual system with the respective monolingual list of documents. Therefore, we have two scores for each query, the first one is calculated by using the dynamic and global index created by 2-step RSV for all languages and the other one is calculated locally for each language. Thus we have to integrate both values. As a way to deal with partially aligned queries (i.e. queries with some terms not aligned), we have implemented several ways to combine the aligned and non-aligned score in a only score per query and retrieved document: Versions two, three and four require training set (topics and their relevance assessments), and it must be available for each monolingual collection. We have used CLEF queries(140-160) relevance assessments available this year for training purposes. Thus, twenty first queries have been used as training and the other forty have been used for evaluation.
Some experiments based on ZPrise use pseudo-relevance feedback technique. We have adopted Robertson-Croft's approach [1] where the system expands the original query generally by 10-15 search keywords, extracted from the 10-best ranked documents. We have chosen this configuration because empirically we have obtained better results than with other configurations available at ZPrise system.
The second step of the merging method does not make use of automatic query expansion techniques such as relevance feedback (RF) or pseudo-relevance feedback (PRF) applied to monolingual queries. Since RF and PRF extend every monolingual query with collection-dependent words, the reindexing process (second step of 2-step RSV) will not take into account all of these words. Because such words are not the same for each monolingual collection, and the translation to the other languages is unknown, 2-step RSV method ignores these new terms for the second step. However, the overall performance will also improve since PRF and RF improve on monolingual experiments and usually some extended terms coincide with terms of the original query, and such terms will be aligned. Rest of expanded terms are integrated as non-aligned terms by using the approaches depicted in the section 2 for mixed 2-step RSV. Of course, the percentage of non-aligned words is increased because of the application of PRF. Table 2 shows the percentage of aligned words for expanded queries by using PRF and Machine Translation. This table shows some interesting results:
• Note that the improvement for this year is considerable respect of 2003, mainly because of a best translation strategy.
• In spite of the very different performance of the bilingual experiments (Table 6), final multilingual average precision is very similar independently of the selected documents for each IR system.
• Since the simultaneous application of PRF and Machine Translation decreases dramatically the percentage of aligned words, the application of PRF improves very slightly the final result.
• Good performance of raw-mixed 2-step RSV, obtaining a result very near to the result reached by means of logistic regression and neural networks. This result is counterintuitive since the method adds two values which are not directly comparable: the score obtained by both aligned and non-aligned terms. Some of the reasons for this good result are:
-α parameter limits the weight of the unaligned factor.
-Not all the terms to be added to the original query are new terms since some terms obtained by means of pseudo-relevance feedback are in the initial query. Thus, these terms are aligned terms. In the same way this explains the good performance of 2-step RSV original method with expanded queries.
-Only 20 queries available for training.
-CLEF document collections are highly comparable (news stories from the same period). The results might be different if collections have vastly different sizes and/or topics.
Thus, 2-step RSV reaches the same precision in spite of using different IR systems. This is a drawback if the IR system used for the first step implements a IR model more sophisticated than the IR model implemented for the second step of the algorithm. In such situation, the improvement is not fully exploited by 2-step RSV merging algorithm because 2-step RSV creates a dynamic index based on classic document retrieval models (more precisely the dynamic index is created by using a document-based OKAPI weighting schema). So, what should we do to improve these results?. Since the second step is basically an OKAPI IR engine, we could improve such engine by using better IR models, and improving the translation and alignment processes.
In this work, we have tested the merging algorithm 2-step RSV in several ways. We have compared the CLEF 2003 and CLEF 2005 Multi-8 results, by using CLEF 160-200 queries. This year we have obtained better results than the 2003 edition. We think that the main reason is a better translation approach and a more refined version of the merging algorithm.
The obtained results show that the improvement of merging algorithms and translation resources are higher than the improvement obtained by expanding the query by means of pseudorelevance feedback.
In the same way, the improvement of the monolingual IR System used to retrieve each monolingual list of documents obtains very slightly better results in the final multilingual system. In order to evaluate the impact of the monolingual IR system, we have evaluated several lists of retrieved documents by using two IR systems and some of the retrieved documents available for the Multi-8 Merging-only task, but holding the same translation approach and merging algorithm. Results show that the precision is very similar independently of the monolingual IR engine. We conclude that improvements in the selection of documents by using some monolingual IR engine is not fully exploited by 2-step RSV merging algorithm since this algorithm creates a dynamic index based on classic document retrieval models.
When pseudo-relevance feedback and machine translation is applied in the same experiment, the percentage of aligned words is too low to apply optimally some mixed variant of 2-step RSV. Thus, a more effective word alignment algorithm must be developed, especially for the new terms added to the query by means of PRF.
Finally, we think that the overall performance of the CLIR system will be overcome if we develop better translation strategies and we improve the IR model used for the creation of the dynamic index for the second step of the algorithm.
Thank you very much to Fernando LLopis and Elisa Noguera from University of Alicante for their support with the IRn package.
Thank you very much to Jacques Savoy and Gareth Jones to supply the results of several bilingual experiments for the Multi-8 Merging-only task and the administration of the web place.
This work has been supported by Spanish Government (MCYT) with grant TIC2003-07158-C04-04.