For our fourth participation in the CLEF evaluation campaigns, our objective was to verify whether our combined query translation approach would work well with new requests and new languages (Russian and Portuguese in this case). As a second objective, we suggested a selecting procedure that could extract a smaller number of documents from collections that for the current request seem to contain no or only few relevant items. We also applied different merging strategies in order to obtain more evidence on the respective relative merits.
1. Bilingual Information Retrieval SYSTRAN GOOGLE FREETRANSLATION INTERTRAN REVERSO WORLDLINGO BABELFISH PROMPT ONLINE BABYLON www.systranlinks.com/ www.google.com/language_tools www.freetranslation.com/web.htm intertran.tranexp.com/ www.reverso.fr/url_translation.asp www.worldlingo.com/ babelfish.altavista.com/ webtranslation.paralink.com/ www.online-translator.com/srvurl.asp?lang=en www.babylon.com
When using the Babylon bilingual dictionary to translate an English request word-by-word, usually more than one translation is provided, in an unspecified order. We decided to pick only the first translation available (labeled “Babylon 1”), the first two terms (labeled “Babylon 2”) or the first three available translations (labeled “Babylon 3”).
Table 1 shows the resulting mean average precision using the various translation tools and the Okapi
probabilistic model (see Savoy (2004c) for implementation details). Of course, not all tools can be used for
each language, and thus as shown in Table 1 various entries are missing (indicated with the label “N/A”). From
this data, we can see that the results from the FreeTranslation MT system usually obtain satisfactory retrieval
performances (around 82% of the MAP of the corresponding monolingual search). As another good translation
system, we may mention Reverso or BabelFish for the French, Prompt for the Russian or Online for both the
Russian and Portuguese languages. For the Finnish language we found only two translation tools, but
unfortunately their overall performance levels were not very good (a similar low level performance was also
found when translating English topics into various Asian languages
Systran Google FreeTrans InterTrans Reverso WorldLingo BabelFish Prompt Online Babylon 1 Babylon 2 Babylon 3
French
Okapi 49 queries
0.4685 0.3729 (79.6%) 0.3680 (78.5%) 0.3845 (82.1%) 0.2664 (56.9%) 0.3830 (81.8%) 0.3728 (79.6%) 0.3729 (79.6%)
N/A
N/A
0.3706 (79.1%)
0.3356 (71.6%)
0.3378 (72.1%)
It is known that while a given translation tool may produce acceptable translations for a given set of
requests, it may perform poorly for other queries
Language Combination Comb!1 Comb!2 Comb!3 Comb!4 Comb!5 Best single Comb!1 Comb!2 Comb!3 Comb!4 Comb!5
French
Okapi 49 queries
Bab2+Free Bab2+Reverso Reverso+Systran
Free+Rev Bab2+Free+
Reverso 0.3845 0.3784 0.3857 0.3858 0 . 4 0 6 6 0.3962
Finnish Okapi 45 queries Bab1+Inter
Mean average precision
Finnish Okapi 45 queries
Bab1+Inter 0.2290 0 . 2 5 2 9 0.2653 0 . 3 0 4 2
Russian Okapi 34 queries Bab1+Free Free+Prompt Prompt+Online
Free+Online Bab1+Free+
Online 0.3067 0 . 3 8 8 8 0.3032 0.2964 0.3043 0.3324
Portuguese
Okapi 46 queries Free+Online Bab1+Systran Bab1+Free+Onl Bab1+Free+Sys
Bab1+Free+ Online+Systran 0.4057 0.4072 0.3713 0 . 4 2 0 4 0.3996 0.4070
The resulting retrieval performances shown in Table 2 are sometimes better than the best single translation
scheme indicated in the row labeled “Best single” (e.g., the strategies “Comb 4” or “Comb 5” for French, or
“Comb 1” for Russian, and “Comb 3” for the Portuguese language). Of course, the main difficulty in this
bilingual search was the translation of English topics into Finnish, due to limited number of free translation
tools. When handling those languages less-often speaking around the world, it seems it would be worthwhile
considering other translation alternatives, such as probabilistic translation based on parallel corpora
For monolingual searches, as described in Savoy (2004c), we used a data fusion search strategy that combined the Okapi and Prosit probabilistic models (see details in Section 2). The data shown in Table 3 indicates that our data fusion approaches may result in better retrieval effectiveness (except for the Finnish 4gram indexing scheme or the Russian corpus). Of course before combining the result lists we could also automatically expand the translated queries, using a pseudo-relevance feedback method (Rocchio’s approach in the present case). The resulting mean average precision as shown in Table 4 did not improve the retrieval effectiveness when compared to the best single approach. In Tables!3 and 4, under the heading “Z-scoreW”, we attached a weight of 1.5 to the Prosit model, and 1 to the Okapi model. Finally, Table 5 depicts the parameters used for our official bilingual runs. 2. Multilingual Information Retrieval
Our multilingual information retrieval system is based on the use of a query translation strategy instead of
either translating all documents into a common language (e.g., English), combining both query and document
translations
34 queries Prosit (3/15) Okapi (3/10)
Round-robin Pro-Free-Reverso 0.2962 UniNEBru2
46 queries Prosit (10/20) Okapi (5/15) Norm RSV Onl-Free-Bab1 0.4704 UniNEBpt1
46 queries Okapi (0/0) Prosit (0/0)
Norm RSV Onl-Free-Sys-Bab1
0.4491 UniNEBpt2
To account for the document score computed for each retrieved item (denoted RSVk for document Dk), we
might formulate the hypothesis that each collection is searched by the same or a very similar search engine and
that the similarity values are therefore directly comparable
Unfortunately the document scores cannot always be directly compared, thus as a third merging strategy we normalized the document scores within each collection by dividing them by the maximum score (i.e. the document score of the retrieved record in the first position) and denoted them “Norm Max”. As a variant of this normalized score merging scheme (denoted “Norm RSV”), we could normalize the document RSVk scores within the ith result list, according to the following formula:
Norm RSVk = ((RSVk - MinRSVi) / (MaxRSVi - MinRSVi))
As a fifth merging strategy, we might use logistic regression to predict the probability of a binary outcome
variable, according to a set of explanatory variables
Pr ob [Dk is rel | rank k , rsvk ] =
ea+b1⋅ln(rank k )+b2 ⋅rsv k 1 + ea+b1⋅ln(rank k )+b2 ⋅rsv k (1) (2)
†
IR model 1 (#docs/#terms) IR model 2 (#docs/#terms) Data fusion operator Translation tools Mean average precision
IR model 1 (#docs/#terms) IR model 2 (#docs/#terms) Data fusion operator Translation tools Mean average precision
Parameters of each single run according to each language English French Finnish (4-gram) Russian (word) 42 queries 49 queries 45 queries 34 queries Okapi (3/15) Prosit (3/10)
Z-score 0.5580 Okapi (3/15) Prosit (3/10)
Z-score
0.5580
Prosit (3/10)
Finally, we suggest merging the retrieved documents according to the Z-score, taken from their document
scores
Z-Score RSVk = ai . [((RSVk-MeanRSVi) / StdevRSVi ) + d i] with di = ((MeanRSVi- MinRSVi)/StdevRSVi) (3) within which the value of di is used to generate only positive values, and ai (usually fixed at 1) is used to reflect the retrieval performance of the underlying retrieval model and to account for the fact that pertinent items are not uniformly distributed across all collections.
Table 6 depicts the exact parameters used to search in the four different collections. For the Russian
collection, we only considered the word-based indexing strategy while for the Finnish language we only used
the 4-gram indexing scheme. In the top part of Table 6, it can be seen that we used a combined query
translation strategy for French, Finnish and Russian languages. As described in our monolingual experiments
Table 7 depicts the retrieval effectiveness of various merging strategies using three different bilingual search parameter settings. In this table, the round-robin scheme will be used as a baseline. On the one hand, when different search engines are merged (Condition A and Condition B), the raw-score merging strategy results in very poor mean average precision. On the other hand, when the same search engine is used (Condition C), the resulting performance is better, but this is not the best one we should be able to achieve. The normalized score merging based on Equation 1 shows degradation over the simple round-robin approach when using parameter setting Condition B (0.1042 vs. 0.2340, or -4.9% in relative performance). Applying our logistic model using both the rank and the document score as explanatory variables, the resulting mean average precision is clearly better than the round-robin merging strategy and than other merging approaches (under Condition A or C). Under Condition B, the difference between our logistic model and the Z-score merging strategy is rather small (0.3111 vs. 0.3019, or 3.1% in relative performance).
As a simple alternative, we also suggest a biased round-robin approach which extracts not one document per collection per round but one document for the Russian corpus and two from the English, French and Finnish collection (because the last three represent larger corpora). This merging strategy results in good retrieval performance, better that the simple round-robin approach. Finally, the Z-score merging approach seems to provide generally satisfactory performance. Moreover, we may multiply the Z-score by an a value (performance under the label “ai = 1.5” with the ai values set as follows: EN: 1.5, FR: 1.5, FI: 1.0, and RU: 1.0).
Norm Max Norm RSV (Eq. 1) Logistic reg. (ln(rank), RSV) Biased round-robin Z-score (Eq. 3) Z-score (Eq. 3) ai = 1.5 Logistic reg. & Selection (0) Logistic reg. & Selection (3) Logistic reg. & Selection (10) Logistic reg. & Selection (20) Logistic reg. & Selection (50) Logistic reg. & OptimalSelect
Condition A Condition B Condition C 50 queries 50 queries 50 queries
0.2386 0.2430 0.2358 0.0642 (-73.1%) 0.0650 (-73.2%) 0.3067 (+30.1%) 0.2552 (+7.0%) 0.1044 (-57.0%) 0.2484 (+5.3%) 0.2899 (+21.5%) 0.1042 (-57.1%) 0.2646 (+12.2%) 0.3090 (+29.5%) 0.3111 (+28.0%) 0.3393 (+43.9%) 0.2639 (+10.6%) 0.2683 (+10.4%) 0.2613 (+10.8%) 0.2677 (+12.2%) 0.2903 (+19.5%) 0.2555 (+8.4%) 0.2669 (+11.9%) 0.3019 (+24.2%) 0.2867 (+21.6%) 0.2957 (+23.9%) 0.2959 (+21.8%) 0.3405 (+44.4%) 0.2953 (+23.8%) 0.2982 (+22.7%) 0.3378 (+43.3%) 0.2990 (+25.3%) 0.3008 (+23.8%) 0.3381 (+43.4%) 0.3010 (+26.1%) 0.3029 (+24.7%) 0.3384 (+43.5%) 0.3044 (+27.6%) 0.3064 (+26.1%) 0.3388 (+43.7%) 0.3234 (+35.5%) 0.3261 (+34.2%) 0.3558 (+50.9%)
It cannot be expected however that each result list would always contain pertinent items in response to a given request. In fact, a given corpus may contain no relevant information regarding the submitted request or the pertinent articles cannot be found by the search engine. In a cross-lingual environment we have found an additional problem: important facets of the original request were translated with inappropriate words or expressions. In all these cases, it is not useful to include items provided by such collections (or such search engines) in the final result list. In addition, the number of pertinent documents is usually not uniformly distributed across all four collections. For a given request (e.g., related to a regional or a national event), only one or two collections may contain relevant documents describing this particular event.
To take into account these phenomena, we have designed a selection procedure which works as follows. First, for each result list we normalize the document score according to our logistic regression method (given in Equation 2). After this step, each document score represents the probability that the underlying article is relevant (with respect to the submitted query and the collection). In the second step, for each result list (or language) we sum the document scores of the first 15 top-ranked documents. If this sum exceeds a given threshold (depending on the collection or search engine), we can thus consider that the corresponding collection contains many pertinent documents. Otherwise, we might only include the m best ranking retrieved items from the corpus (with a relatively small m value). We may thus limit the number of items extracted from a given corpus while also taking account of the fact that each collection usually contains few pertinent items. Table 7 lists the mean average precision achieved using this selection strategy under the label “Logistic reg. & Selection (m),” where the value m indicates that we always include the m best retrieved items from each corpus in our final result list. Of course, when we set m = 0, the system will not extract any documents from a collection having a poor overall score. Finally under the label “Logistic reg. & OptimalSelect“, we have computed the mean average precision that can be achieved when the selection is done without any error (with m = 0). When using such an ideal selection system, the mean average precision is clearly better than all other merging strategies (e.g, under Condition C, the MAP is 0.3558 vs. 0.3393 with the logistic regression without selection).
Query lang.
Query type English English English English English
TD TD TD TD TD
Type automatic automatic automatic automatic automatic
Merging logistic Z-scoreW raw-score logistic & select
Z-scoreW
Parameters
Condition A Cond.!A, ai!=!1.5
Condition C Cond. A, m = 20
Condition B
MAP
In multilingual tasks, searching documents written in different languages represents a real challenge. In this case we propose a new simple selecting strategy which will avoid extracting a relatively large number of documents from collections when these documents are of little interest with respect to the current request (see Table 7). In this multilingual task, it is also interesting to mention that combining the result lists provided by same search engine (Condition C in Table 7) may sometimes produce good retrieval effectiveness compared to combining different search models (Condition A in Table 7).
Acknowledgments
The authors would like to thank the CLEF-2004 task organizers for their efforts in developing various European languages test-collections. The authors would also like to thank C. Buckley from SabIR for giving us the opportunity to use the SMART system, together with Samir Abdou for his help in translating the English topics. This research was supported by the Swiss National Science Foundation under Grant #21-66 742.01.
C201 C202 C203 C204 C205 C206 C207 C208 C209 C210 C211 C212 C213 C214 C215 C216 C217 C218 C219 C220 C221 C222 C223 C224 C225
Nick Leeson's Arrest East Timor Guerrillas Victims of Avalanches Tamil Suicide Attacks G7 Summit in Halifax Fireworks Injuries “Sophie's World” Tour de France Winner Nobel Peace Prize Candidates Peru-Ecuador Border Conflict Sportswomen and Doping Papal Travels Multi-billionaires Re-election of Peru's President Glue-sniffing Youngsters AIDS in Africa Andreotti and the Mafia EU Commissioner Candidates European Cars in Russia 2002 Olympic Winter Games Presidential elections in France Chernobyl Disaster outside ex-USSR Woman solos Everest Nuclear Power Plant of Sosnovyi Bor C226 C227 C228 C229 C230 C231 C232 C233 C234 C235 C236 C237 C238 C239 C240 C241 C242 C243 C244 C245 C246 C247 C248 C249 C250
Atlantis-Mir Docking New Portuguese Prime Minister Pension Schemes in Europe Greenhouse Effect Deaf and Society Seal-hunting A typhoon in the Philippines Panchen Lama Lady Diana Mental Health of the Young Sioux Ghost Shirt New political parties Record Permanence in Space Films of Kieslowski Footballer of the Year 1994 Christopher Reeve Castro visits UN Alexander the Great's Tomb Macedonia Name Dispute Women's Ten Thousand Metres Champion Rabies in Humans Title of the queries of the CLEF-2004 test-collection