This year, the University of Exeter participated in both the CLEF 2002 monolingual and bilingual task for two languages: Italian and Spanish. We submitted 4 ranked results each for both Italian and Spanish Monolingual tasks and 5 each for the bilingual tasks. We report experimental results from our investigations of merging topic translations from two machine translation (MT) systems and recent experimental results for query expansion and term weighting from alternative collections. Our results show that although, query expansion and term weighting from a pilot collection has been shown to be effective in improving retrieval performance in information retrieval, the performance can be affected negatively if the lexicon of the pilot and the test collection differ. The main objective of our participation in CLEF 2002 was to test the effectiveness of some of our methods developed after CLEF 2001, and also to investigate the retrieval behavior of document collection in Italian and Spanish using topic sets in several other languages. Our official submissions included Italian and Spanish monolingual tasks as well as bilingual tasks using topic sets in German, French, English, Italian and Spanish. To present a fair comparison of results across all language pairs and methods for the bilingual runs, we used the same translation resources for each pair and we also present results for each retrieval method for each pair. Both the collections and the topics were translated from the source language into English. Firstly, because we had intended to participate in the multilingual task and secondly because the retrieval system we used could not deal with accented words. We were unable to submit results for the multilingual task because of time constraints. Our general approach was to use the collection and topic translation strategy for CLIR. The document collection and the topic statements were submitted to the selected MT system, the output was then collected and applied on the information retrieval (IR) system. For all our submissions and subsequent runs presented in this paper, we used both the Systran Version: 3.0 and the Globalink Power Translation Pro Version: 6.4 MT systems for topic translation. It should be noted that the two collections used in our experiments were translated using only Systran Version 3.0. Pseudo-relevance feedback (PRF) has been shown to be an effective approach to improving retrieval performance in IR and also in CLIR [1][2][3]. In our experimental work in [4][5] we demonstrated the effectiveness of a new PRF method using the Okapi BM25 probabilistic model [6]. In this work we investigated the idea of selecting expansion terms for document summaries and found this method to be more reliable than query expansion from full documents. Since CLEF 2001, we have also explored data combination techniques that merge the output of the two MT systems for the topics, and use this as the initial query set. Furthermore, we have also been investigating the use of a comparable collection (pilot) for generating expansion terms and term weighting. The method is described fully below. Our experiments for CLEF 2002 explore the effectiveness of these methods with automatically translated documents and topics. The remainder of this paper is organized as follows: Section 2 reviews the information retrieval methods used, Section 3 gives a brief description of the data processing techniques used, Section 4 describes the different methods of PRF, Section 5 gives the experimental results and section 6 concludes the paper.
cfw(i) × tf (i, j) × ( K1 + 1)
K1 × ((1 − b) + (b × ndl( j))) + tf (ij) where cw(i,j) = the weight of term i in document (j), cfw(i) = the standard collection frequency weight tf(i,j) = the document term frequency ndl(j) = the normalized document length calculated as follows ndl( j) =
dl( j)
where dl(j) = the length of j K1 and b are empirically selected tuning constants for a particular collection. K1 modifies the effect of term frequency and b modifies the effect of document length. All our experiments were done with K1and b set to 1.4 and 0.6. The parameters were set using the CLEF 2001 data sets.
Relevance feedback is a method used to improve retrieval effectiveness by either improving the query terms (Query modification) or the term weights (term-reweighting). All our experiments used query expansion to modify the query to attempt to improve the quality of the initial query by adding new terms selected from a pool of potential expansion terms from the initial retrieval run.
Our query expansion method selects terms from summaries of the top 5 ranked documents. The summaries
were generated using the method described in [
rw(i) is the standard Robertson/Sparck Jones relevance weight [12] reproduced below rw(i) = log (r(i) + 0.5)( N − n(i) − R + r(i) + 0.5)
(n(i) − r(i) + 0.5)(R − r(i) + 0.5) where n(i) = the total number of documents containing term i r(i) = the total number of relevant documents term i occurs in R = the total number of relevant documents for this query
N = the total number of documents In our modified version, although potential expansion terms are selected from the summaries of the top 5 ranked documents, they are ranked using the top 20 ranked documents from the initial run. 3
The two document collections used in our experiments, Italian and Spanish were translated to English using the Systran Translation Software version 3.0. This was necessitated by the inability of the retrieval system (Okapi) used for our experiments to deal with accented terms as well as languages other than English. All queries were translated from the source language into English using both the Systran Version 3.0 and Globalink Power Translation Pro version 6.4 MT software. All our experiments were done using both the title and description fields of the CLEF topics.
This method is the same as that used in our CLEF 2001 official submissions [
Query expansion is aimed at improving initial search topic in order to make it a better expression of user’s information need. This is normally achieved by adding terms selected from assumed relevant document retrieved from the test collection, to the initial query. Another approach that has been shown to be effective is the selection of expansion terms from a larger collection, a subset of which would be the test collection. Based on the assumption that if additional documents from the same corpus as the test collection are available, these can be used for improved query expansion, we explore the idea of pilot searching [13]. The larger data collection is likely to enable more accurate parameter estimation and hopefully better retrieval and document ranking. The Okapi submissions for the TREC-7 [13] adhoc tasks used the TREC disks 1-5 of which the TREC-8 data is a subset, for parameter estimation and query expansion. The method was found to be very effective. Our post CLEF 2001 results for bilingual English also demonstrated the effectiveness of this approach [14]. The TREC-8 data collection consisting of more than half a million documents was used as “pilot collection” in our experiments. The CLEF 2002 English collection is a subset of the TREC-8 data collection. Two different approaches were taken to the pilot searching procedure. They are as follows 1 2
Apply the original query terms on the pilot collection using the Okapi system without feedback. Extract
terms from the top R assumed relevant documents; rank the extracted terms and select the desired number of
expansion terms from the top of the list. The corresponding cfw(i) term weights are also stored along with
the expansion terms. The expansion terms are added to the initial query terms and applied on the test
collection (Test coll. weight and Pilot coll. expansion terms). This approach is shown to give an
improvement for the CLEF 2001 bilingual task [
The second method involve using the expansion terms from the pilot collection as above but this time the cfw(i) weights from the pilot collection are used instead of the term weights from the test collection (Pilot coll. terms and weight). This method gave a further improvement for the CLEF 2001 bilingual task.
MT systems sometimes make translation errors due to the limitations of the dictionaries used in them. However, this problem can be tackled by combining the outputs of multiple MT systems. This idea is based on the proven notion that combination of evidence from multiple information sources is beneficial to text retrieval. Thus, in this method, for each untranslated query, two different translations of the query from the two different translators used in these experiments are merged into a single query.
5
In this section we report the results of our investigation for all methods described above. Baseline results without feedback and results after the application of the different methods of feedback are presented for Italian and Spanish Monolingual and Bilingual tasks. All our official submissions are indicated by a *. In all cases the results use the Title and Description fields of the search topics and we present the average precision (Avep), the % change in average precision relative to the baseline for the MT system used (% chg) and the total number of relevant documents retrieved (R-ret). For feedback runs all initial query terms are upweighted by multiplying the original term weights by 3.5.
Run-id
coll. expansion terms
388 *414 *453 376
6.70% 16.75% -3.09%
347 362 373 407 364 379 359
Table 3-8 shows the retrieval behavior for the language pairs before and after various methods of feedback for Italian bilingual runs. Result using pilot collection for query expansion and term weighting again gives the worst result for all pairs overall. This is almost certainly due to the differences in the lexicon used in the pilot and the test collection, which reduces the query-document matching and subsequently result in reduction in retrieval effectiveness. The result for the combined queries further strengthens the idea that combining the output from two MT systems might help in reducing the effect of poor MT systems on retrieval. The best result overall is given by the Italian-English run using test collection weight and pilot collection expansion terms. Globalink translated topics gave better performance overall compared to the results using Systran translated topics although French topics using Systran performed better than that using Globalink.
Run-id
coll. expansion terms
442 *473 *475 420
7.01% 7.46% -4.98%
R-ret 2413 2538 2517 2249 Table 9 and 10 monolingual Spanish results shows the same trend as the Italian monolingual results in Table1-2 above. Results again show that using pilot collection to estimate term weight and query expansion for Systran topics is not very effective, it resulted in about 5% reduction in average precision compared to the baseline. The combined query method (Table 10) is also shown to reduce the negative effect of translation output on retrieval.
Run-id
feedback
318 359 382
334 342 373 2372 387 369 414
350 370 396
Results for Spanish bilingual runs (Table 11-16) show that Globalink translated topics gave better performance overall compared to results using Babelfish translated topics. The Spanish to English pair using summary-based expansion term selection from pilot collection and cfw(i) from test collection gave the best results overall. Combining the queries from the two MT systems (Table 13 and 14) is again shown to be effective in reducing degradation in performance brought about by poor MT output.
Query expansion and term weighting from a pilot collection has been shown to be very effective in information retrieval. The results above however suggests otherwise. This is probably due to the differences in the language of the pilot and the test collection. To test this theory, we did some further runs; the pilot and the test collection were merged to form a single collection. This merged collection was then used as the pilot collection, i.e. for query expansion and term weighting. The expanded query and the corresponding weight is then applied on the test collection. The tables below show the effect of this method on retrieval. In all cases we present the result for both Systran and Globalink translated topics.
Run-id
442 490
346 371 371
18.09% 16.30% 14.51% 372 378 366
21.97% 12.17% 18.06%
R-ret 874 922 970 In this paper we have presented our results for the CLEF 2002 monolingual and bilingual Italian and Spanish retrieval tasks. The results suggest that good retrieval results can be achieved by merging the output of two commercially available MT systems. It also shows that all language pairs behave very differently to different feedback method, this requires further investigation to determine the causes of such behavior and how they can be tackled. The combined query method is very effective in smoothing out the negative effects of bad translations in most cases. Using pilot collection to estimate term weight and for query expansion although shown to be very effective in [13], the results shown here suggests that when there is a difference in the language of the pilot and the test collection the method might not be as effective. We show that further improvements can be achieved by merging the two collections to form a pilot collection.
Further investigation is needed to determine the reason for the slightly poor performance of the Systran translated queries compared to the Globalink translated queries. We also noticed that some terms were left untranslated by the MT systems, this is more predominant in the Systran translations, and might have been reason for the lower performance achieved using the topics translated using Systran MT compared to the performance for Globalink topics in the bilingual results. [6] S.E. Robertson and S. Walker. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 232-241, Dublin, 1994. A.C.M. [7] M.F. Porter. An algorithm for suffix stripping. Program, 14:10-137, 1980. [8] H.P. Luhn. The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development, 2(2):159-165, 1958. [9] H.P. Edmundson. New Methods in Automatic Abstracting. Journal of the ACM, 16(2):264-285, 1969 [10] A. Tombros and M. Sanderson. The Advantages of Query-Biased Summaries in Information Retrieval. In proceedings of the 21st Annual International ACM SIGIR Conference Research and Development in
[11] S.E. Robertson. On term selection for query expansion. Journal of Documentation, 46:359-364, 1990. [12] S.E. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American
[13] S.E. Robertson, S. Walker, and M. M. Beaulieu. Okapi at TREC-7: automatic ad hoc, filtering, VLS and interactive track. In E. Voorhees and D.K. Harman, editors, Overview of the Seventh Text REtrieval Conference (TREC-7), pages 253-264. NIST, 1999. [14] G.J.F. Jones and A.M. Lam-Adesina. Combination Methods for improving the Reliability of Machine Translation Based Cross-Language Information Retrieval. Proceedings of the 13th Irish Conference on