As in CLEF 2003, Berkeley experimented with the CLEF Russian Izvestia document collection with monolingual and bilingual runs for the Russian collection. For CLEF 2004 we also experimented with Chinese and Japanese as topic languages, using English as the 'pivot' language. For bilingual retrieval our approaches were query translation (for English as a topic language) and 'fast' document translation from Russian to English (for Chinese and Japanese translated to English as the topic language). Chinese and Japanese topic retrieval significantly under-performed English Æ Russian retrieval because of the 'double translation' loss of effectiveness.
log O ( R | D , Q ) = log = log
P ( R | D , Q )
P ( R | D , Q ) = − 3 .51 + 37 .4 ∗ x1 + 0 .330 ∗ x 2 − 0 .1937 ∗ x 3 + 0 .0929 ∗ x 4
P ( R | D , Q ) 1 − P ( R | D , Q ) with respect to a query, and
1 n qtfi x1 = ∑
n + 1 i=1 ql + 35 where O ( R | D , Q ), P ( R | D , Q ) mean, respectively, odds and probability of relevance of a document x2 = x3 = x4 = n
dtfi dl + 80 1 n
∑ log n + 1 i =1 ctfi cl where n is the number of matching terms between a document and a query, and ql : query length dl: document length cl: collection length qtf_i: the within-query frequency of the ith matching term dtf_i: the within-document frequency of the ith matching term ctf_i: the occurrence frequency of the ith matching term in the collection.
This formula has been used since the second TREC conference and for all NTCIR and CLEF cross-language
evaluations [
CLEF 2003 marked the first time a document collection was available and evaluated in the Russian language.
The CLEF Russian collection consists of 16,716 articles from Izvestia newspaper for 1995. This is a small
number of documents by most CLEF measures (the smallest other collection of CLEF 2003, Finnish, has 55,344
documents; the Spanish collection has 454,045 documents). We used the Russian and English indexes
generated for CLEF 2003 for all our CLEF 2004 Russian runs. The collection is also rich in metadata, including
specification of geography for news articles; this can be exploited for mapping and geotemporal querying of
documents relating to place and time [
The Russian document collection was supplied in the UTF-8 unicode encoding, as were the Russian version of
the topics. However, since the stemmer we employ is in KOI8 format, the entire collection was converted into
KOI8 encoding, as with CLEF 2003 [
We submitted two Russian monolingual runs, the results of which are summarized below. As in CLEF 2003, both runs utilized blind feedback, choosing the top 30 terms from the top ranked 20 documents of an initial retrieval run For BKRUMLRR1 and BKRUMLRR2 runs we used TITLE and DESCRIPTION document fields for indexing. The results of our retrieval are summarized in Table 1. Results were reported by the CLEF organizers for 34 topics which had one or more relevant documents.
0.5734 0.5636 0.5506 0.4969 0.4670 0.4526 0.3628 0.2989 0.2839 0.2555 0.2548 123 108 Adding the Narrative section to the query did not significantly improve results because the Narrative section did not contribute additional content terms beyond those found in the Title and Description fields of the topics.
We submitted eight bilingual runs against the Russian document collection, four with English as topic language and two each with Chinese and Japanese as topic languages. These runs used an index in which only the TITLE and TEXT fields of each Russian document was indexed, so are directly comparable to the monolingual runs BKMLRURR1 and BKMLRURR2 above. The four EnglishÆ Russian runs utilized query translation from English topics into Russian. We compared two web-available translation systems, SYSTRAN at http://babelfish.altavista.com/ for the first two runs (BKRUBLER1, BKRUBLER2) and the PROMT system (runs BKRUBLER3, BKRUBLER4) developed in Russia and found at http://www.translate.ru.
Translation Topic fields Retrieved Relevant Rel Ret Precision at 0.00 at 0.10 at 0.20 at 0.30 at 0.40 at 0.50 at 0.60 at 0.70 at 0.80 at 0.90 at 1.00 Avg. Prec.
BKRUBLER1 BKRUBLER2 BKRUBLER3 BKRUBLER4
Babelfish Babelfish PROMT PROMT
TD TDN TD TDN 34000 34000 34000 34000 123 123 123 123 69 85 98 93
The results demonstrate clearly the superiority of the PROMT system for this topic set. 3.4
Because Chinese and Japanese were available as topic languages, we experimented with these languages by
translating the topics to English (i.e. used English as a pivot language). Our approach to translation from
Chinese or Japanese topics to English was to utilize a widely available software package, the SYSTRAN CJK
Personal system available for less than $US100. from www.systransoft.com. However, instead of query
translation a second time, we utilized a technique (also used for Russian in CLEF 2003) developed by Aitao
Chen, called ‘Fast Document Translation’ [
The results, displayed below in Table 3, show that there is considerable loss of performance when using English as a pivot language for these Asian language (we have re-displayed the best EnglishÆ Russian runs for comparison). It may be that this performance was hampered by the reduced utility of the English documents translated from Russian, as was the case for our CLEF 2003 bilingual performance which used this method. We did not try merging of runs from the two methods to see if it would improvement performance.
Our monolingual Russian performance was acceptable but certainly not outstanding. For many topics, TitleDescription runs out-performed Title-Description-Narrative runs, because the Narrative section added no new information and might sometimes add noise terms.
For all our runs our bilingual retrieval results were worse than monolingual (Russian-Russian) retrieval in terms of overall precision. However the translation of English to Russian by the PROMT system achieved 82% of monolingual for the TD runs. One puzzling and interesting topic was number 202 (“Nick Leeson's Arrest”) where our bilingual retrieval out-performed our monolingual runs – it seems that the PROMT translation and transliteration “Арест Ника Лизона” came up with a better spelling of the last name than the Russian topic creator who used “Арест Ника Леесон”, which did not seem to match any relevant documents. According to the summary results for Russian monolingual, at least one run achieved 1.00 precision for this topic; it would be most interesting to see how they modified the topic to match to the three relevant documents. A cautionary note must be made about the CLEF-2004 Russian topic set. The total number of relevant documents was only 123 for the entire topic set, with a mean of 3.6 relevant documents per topic. Because of the nature of the retrieval results by query from the Russian collection (22 of the 34 topics have 2 or fewer relevant documents) one has to be careful about drawing conclusions from any submitted results. 4
For CLEF 2004, we experimented with the CLEF Russian document collection with both monolingual Russian and bilingual to Russian from English, Chinese and Japanese topics In addition to query translation methodology for bilingual retrieval, we tried a fast document translation method of the Russian collection to English and performed English-English monolingual retrieval with the translated topics from Chinese and English to Japanese. ChineseÆ Russian and JapaneseÆ Russian bilingual performance results were significantly worse than query translation from English to Russian.
We would like to thank Aitao Chen for supplying writing the logistic regression ranking software and for performing the fast document translation from Russian to English. 5