As in CLEF 2003, Berkeley experimented with the CLEF Russian Izvestia document collection with monolingual and bilingual runs for the Russian collection. For CLEF 2004 we also experimented with Chinese and Japanese as topic languages, using English as the 'pivot' language. For bilingual retrieval our approaches were query translation (for English as a topic language) and 'fast' document translation from Russian to English (for Chinese and Japanese translated to English as the topic language). Chinese and Japanese topic retrieval significantly under-performed English Russian retrieval because of the 'double translation' loss of effectiveness.
CLEF 2003 was the first time a Russian language document collection was available in CLEF. We had worked for several years with Russian topics in both the GIRT task and the CLEF main tasks, so extension of our techniques to Russian was straightforward No unusual methodology was applied to the Russian collection, however encoding remained an issue and we ended up using the KOI-8 encoding scheme for both Russian documents and topics.
Berkeley has used a monolingual document ranking algorithm which uses statistical clues found in documents and queries to predict a dichotomous variable (relevance) based upon logistic regression fitting of prior relevance judgments. The exact formula is:
mean, respectively, odds and probability of relevance of a document with respect to a query, and
where n is the number of matching terms between a document and a query, and ql : query length dl: document length cl: collection length qtf_i: the within-query frequency of the ith matching term dtf_i: the within-document frequency of the ith matching term ctf_i: the occurrence frequency of the ith matching term in the collection. This formula has been used since the second TREC conference and for all NTCIR and CLEF cross-language evaluations [1].
Russian Retrieval for the CLEF main task CLEF 2003 marked the first time a document collection was available and evaluated in the Russian language.
The CLEF Russian collection consists of 16,716 articles from Izvestia newspaper for 1995. This is a small number of documents by most CLEF measures (the smallest other collection of CLEF 2003, Finnish, has 55,344 documents; the Spanish collection has 454,045 documents). We used the Russian and English indexes generated for CLEF 2003 for all our CLEF 2004 Russian runs. The collection is also rich in metadata, including specification of geography for news articles; this can be exploited for mapping and geotemporal querying of documents relating to place and time [2].
The Russian document collection was supplied in the UTF-8 unicode encoding, as were the Russian version of the topics. However, since the stemmer we employ is in KOI8 format, the entire collection was converted into KOI8 encoding, as with CLEF 2003 [3]. In indexing the collection, we converted upper-case letters to lowercase and applied Snowball's Russian stemmer (http://snowball.tartarus.org/russian/stemmer.html) together with Russian stopword list created by merging the Snowball list with a translation of the English stopword list. In addition the PROMPT translation system would also only work on KOI8 encoding which meant that our translations from English also would come in that encoding.
We submitted two Russian monolingual runs, the results of which are summarized below. As in CLEF 2003, both runs utilized blind feedback, choosing the top 30 terms from the top ranked 20 documents of an initial retrieval run For BKRUMLRR1 and BKRUMLRR2 runs we used TITLE and DESCRIPTION document fields for indexing. The results of our retrieval are summarized in Adding the Narrative section to the query did not significantly improve results because the Narrative section did not contribute additional content terms beyond those found in the Title and Description fields of the topics.
We submitted eight bilingual runs against the Russian document collection, four with English as topic language and two each with Chinese and Japanese as topic languages. These runs used an index in which only the TITLE and TEXT fields of each Russian document was indexed, so are directly comparable to the monolingual runs BKMLRURR1 and BKMLRURR2 above. The four English Russian runs utilized query translation from English topics into Russian.
We compared two web-available translation systems, SYSTRAN at http://babelfish.altavista.com/ for the first two runs (BKRUBLER1, BKRUBLER2) and the PROMT system (runs BKRUBLER3, BKRUBLER4) developed in Russia and found at http://www.translate.ru. The results demonstrate clearly the superiority of the PROMT system for this topic set.
Because Chinese and Japanese were available as topic languages, we experimented with these languages by translating the topics to English (i.e. used English as a pivot language). Our approach to translation from Chinese or Japanese topics to English was to utilize a widely available software package, the SYSTRAN CJK Personal system available for less than $US100. from www.systransoft.com. However, instead of query translation a second time, we utilized a technique (also used for Russian in CLEF 2003) developed by Aitao Chen, called 'Fast Document Translation' [4]. Instead of doing complete document translation using MT software, the MT system is used to translate the entire vocabulary of the document collection on a word-by-word basis without the contextualization of position in sentence with respect to other words. Monolingual retrieval was performed by matching the English versions of the Chinese or Japanese topics against the translated English document collection. More details can be found in our CLEF-2003 final paper [3].
The results, displayed below in Table 3, show that there is considerable loss of performance when using English as a pivot language for these Asian language (we have re-displayed the best English Russian runs for comparison). It may be that this performance was hampered by the reduced utility of the English documents translated from Russian, as was the case for our CLEF 2003 bilingual performance which used this method. We did not try merging of runs from the two methods to see if it would improvement performance.
Our monolingual Russian performance was acceptable but certainly not outstanding. For many topics, Title-Description runs out-performed Title-Description-Narrative runs, because the Narrative section added no new information and might sometimes add noise terms. For all our runs our bilingual retrieval results were worse than monolingual (Russian-Russian) retrieval in terms of overall precision. However the translation of English to Russian by the PROMT system achieved 82% of monolingual for the TD runs. One puzzling and interesting topic was number 202 ("Nick Leeson's Arrest") where our bilingual retrieval out-performed our monolingual runs -it seems that the PROMT translation and transliteration "Арест Ника Лизона" came up with a better spelling of the last name than the Russian topic creator who used "Арест Ника Леесон", which did not seem to match any relevant documents. According to the summary results for Russian monolingual, at least one run achieved 1.00 precision for this topic; it would be most interesting to see how they modified the topic to match to the three relevant documents.
A cautionary note must be made about the CLEF-2004 Russian topic set. The total number of relevant documents was only 123 for the entire topic set, with a mean of 3.6 relevant documents per topic. Because of the nature of the retrieval results by query from the Russian collection (22 of the 34 topics have 2 or fewer relevant documents) one has to be careful about drawing conclusions from any submitted results.
For CLEF 2004, we experimented with the CLEF Russian document collection with both monolingual Russian and bilingual to Russian from English, Chinese and Japanese topics
In addition to query translation methodology for bilingual retrieval, we tried a fast document translation method of the Russian collection to English and performed English-English monolingual retrieval with the translated topics from Chinese and English to Japanese. Chinese Russian and Japanese Russian bilingual performance results were significantly worse than query translation from English to Russian.
We would like to thank Aitao Chen for supplying writing the logistic regression ranking software and for performing the fast document translation from Russian to English.