For our third participation in the CLEF evaluation campaign, our objective for both multilingual tracks is to propose a new merging strategy that does not require a training sample to access the multilingual collection. As a second objective, we want to verify whether our combined query translation approach would work well with new requests.
Based on our experiments of last year [
When translating an English request word-by-word using the Babylon bilingual dictionary, we decided to pick only the ¯rst translation available (labeled "Babylon 1"), the ¯rst two terms (labeled "Babylon 2") or the ¯rst three available translations (labeled "Babylon 3"). Table 1 shows the resulting mean average precision using translation tools, using the Okapi probabilistic model and based on word-based indexing scheme. Of course, not all tools can be used for each language, and thus as shown in Table 1 various entries are missing (indicated with the label "N/A"). From this data, we see that usually the Reverso or the FreeTranslation system produce interesting retrieval performance. We found only two translation tools for the Swedish and the Finnish languages but unfortunately their overall performance levels were not very good.
A particular translation tool may however produce acceptable translations
for a given set of requests, but may perform poorly for other queries. This is
a known phenomenon [
To date we have not been able to detect when a given translation will produce satisfactory retrieval performance and when it will fail. Thus before carrying out the retrieval, we have chosen to generate a translated query by concatenating two or more translations. Table 2 shows the retrieval e®ectiveness for such combinations, using the Okapi probabilistic model (word-based indexing). The top part of the table indicates the exact query translation combination used while the bottom part shows the mean average precision achieved by our combined query translation approach. The resulting retrieval performance is better than the best single translation scheme indicated in the row labeled "Best" (except for the strategy "Comb 1" in Spanish).
As described in [
Using the original and the translated queries, we then search for pertinent items
within each of the four and eight corpora respectively. From each of these result
lists and using a merging strategy, we need to produce a unique ranked result
list showing the retrieved items. As a ¯rst approach, we considered the
roundrobin (RR) approach whereby we took one document in turn from all individual
lists [
To account for the document score computed for each retrieved item (denoted
RSVk for document Dk), we might formulate the hypothesis that each collection
is searched by the same or a very similar search engine and that the similarity
values are therefore directly comparable [
Unfortunately the document scores cannot be directly compared, thus as a third merging strategy we normalized the document scores within each collection by dividing them by the maximum score (i.e. the document score of the retrieved record in the ¯rst position) and denoted them "Norm Max". As a variant of this normalized score merging scheme (denoted "NormN"), we may normalize the document RSVk scores within the ith result list, according to the following formula:
N ormN RSVk =
M inRSV i
M inRSV i (1)
As a ¯fth merging strategy, we might use the logistic regression [
P rob [Dk is rel j rankk; RSVk] =
e®+¯1¢ln(rankk)+¯2¢RSVk 1 + e®+¯1¢ln(rankk)+¯2¢RSVk (2)
As a new merging strategy, we suggest merging the retrieved documents according to the Z-score, taken from their document scores. Within this scheme, we need to compute, for the ith result list, the average of the RSVk (denoted M eanRSV i) and the standard deviation (denoted StdevRSV i). Based on these values, we may normalize the retrieval status value of each document Dk provided by the ith result list, by computing the following formula: N ormZ RSVk = ®i ¢ · RSVk ¡ M eanRSV i
StdevRSV i
¸ + ±i (3) with ±i =
StdevRSV i within which the value of ±i is used to generate only positive values, and ®i (usually ¯xed at 1) is used to re°ect the retrieval performance of the underlying retrieval model.
The justi¯cation for such a scheme is as follows. If the RSVk distribution is linear, as shown in Table 5 and in Figure 1, there is no great di®erence between a merging approach based on Equation 1 or the proposed Z-score merging strategy. It is our point of view (and this point must still be veri¯ed), that such a distribution may appear when the retrieval scheme cannot detect any relevant items. However, after viewing di®erent result lists provided from various queries and corpora, it seems that the top-ranked retrieved items usually provide a much greater RSV values than do the others (see Table 6 and Figure 2). Thus, our underlying idea is to emphasis this di®erence between these ¯rst retrieved documents and the rest of the retrieved items, by assigning a greater normalized RSV value to these top-ranked documents. simple round-robin approach (34.92 vs. 36.71, -4.9% in the small, automatic experiment, and 26.52 vs. 29.81, -11% in the large automatic experiment). Using our logistic model with both the rank and the document score as explanatory variables (row labeled "Logistic"), the resulting mean average precision is better than the round-robin merging strategy.
As a simple alternative, we also suggest a biased round-robin ("Biased RR" or "bRR") approach which extracts not one document per collection per round but one document for the French, English, Italian, Swedish and Finnish corpus and two from the German, Spanish and Dutch collection (representing larger corpora). This merging strategy results in interesting retrieval performance. Finally, the new Z-score merging approach seems to provide generally satisfactory performance. Moreover, we may multiply the normalized Z-score by an ® value (performance under the label "NormZ ®i = 1:25" or "NormZ ®i = 1:5"). Under the label "NormZ coll-d", the ® values are collection-dependant and are ¯xed as follows: en: 1, fr: 0.9, de: 1.2, sp: 1.25, it: 0.9, nl: 1.15, sv: 0.95, and fi: 0.9.
Of course, we may combine the two or three best merging strategies (performance depicted in Table 8, namely the "biased round-robin" (denoted "bRR"), "logistic regression" (or "log.") and the "NormZ ®i = 1:5" (or "Z-1.5")). Using various data fusion operators, the retrieval e®ectiveness of these data fusion approaches are shown in Table 9. Finally, the descriptions of our o±cial runs for the small and large multilingual tracks are shown in Table 10. 4
In this fourth CLEF evaluation campaign, we have evaluated various query translation tools, together with a combined translation strategy, resulting in a retrieval performance that is worth considering. However, while a bilingual search can be viewed as easier for some pairs of languages (e.g., from an English query into a French document collection), this task is clearly more complex for other languages pairs (e.g., English to Finnish). On the other hand, the multilingual, and more precisely the large multilingual task, shows how searching documents written in eight di®erent languages can represent a challenge. In this case, we have proposed a new simple merging strategy based on the Z-score computed from the document scores, a merging scheme that seems to result in interesting performance.
Acknowledgments. The author would like to thank C. Buckley from SabIR for giving us the opportunity to use the SMART system. This research was supported in part by the Swiss National Science Foundation (grant #21-66 742.01).