The Dublin City University group participated in the monolingual, bilingual and multilingual retrieval tasks this year. The main focus of our investigation this year was extending our retrieval system to document languages other than English, and completing the multilingual task comprising four languages: English, French, Russian and Finnish. Results from our French monolingual experiments indicate that working in French is more effective for retrieval than adopting document and topic translation to English. However, comparison of our multilingual retrieval results using different topic and document translation reveals that this result does not extend to retrieved list merging for the multilingual task in a simple predictable way.
Dublin City University's (DCU) participation in CLEF 2004 Monolingual, Bilingual and Multilingual track builds on previous work at the University of Exeter [1]. Previously had focussed on document translation using machine translation using English as a "pivot" language for all tasks. Our work for CLEF 2004 concentrated on extending our retrieval system to work in the document language with topic translation when needed. Our strategy is to use our existing Okapi retrieval system and make use of the Snowball stemmers and stop word lists [2]. Using these tools we completed runs for monolingual French, Russian and Finnish, official bilingual French and Russian, and the multilingual track consisting of English, French, Russian and Finnish, together with the additional monolingual and bilingual runs needed for the multilingual task.
This paper briefly describes our retrieval system and reports results from our runs with analysis of the observed performance levels.
In this section we first give a brief overview of our retrieval system, and then describe how we adapted it to be language independent and the specific detail of preprocessing for our CLEF 2004 runs.
The basis of our experimental retrieval system remains the City University research distribution version of the Okapi system, as used in our previous CLEF participation [1]. The basic Okapi system includes tools for English language preprocessing and can only handle ASCII characters for English characters and punctuation symbols.
For English language runs we continued to use this system. The documents and search topics were processed to remove stopwords from a list of about 260 words; suffix stripped using the Okapi implementation of Porter stemming [3] and terms were indexed using a small set of synonyms.
Terms are weighted using the standard BM25 weighting scheme and all runs use our summarybased pseudo relevance feedback (PRF) method [4]. The summary generation method combines Luhn's keyword cluster method, a title terms frequency method, a location/header method and a query-bias method from to form an overall significance score for each sentence.
In order to use the Okapi software with non-English document we carried out the language preprocessing outside Okapi and then encoded the resulting character strings into ASCII as follows.
Language Preprocessing The documents and topic are prepared using a pipeline of preprocessing components. Firstly, the data is tokenised to isolate the text body from the SGML/XML markup tags. Then, all punctuation characters are deleted from the text body, with the following exceptions: full stops, commas, semi-colons, colons, exclamation marks and question marks. Whitespace is inserted to separate these punctuation characters from word tokens. The third step is the conversion of characters to lower case. A finite set of upper case characters are mapped to lower case equivalents. Distinct mappings must be used for each character set. The Russian characters were converted to KOI-8 character encoding as required by the Snowball tools, while the Finnish and French documents use ISO Latin 1. Conversion of the Russian data loses some data, for example the degree sign prevalent in weather forecasts is lost, further some corruptiion of the original data to "boxdrawing" symbols was observed. The Russian stopword list used here consists only of the simple first part of the Snowball list.
At the next stage stop words are removed. The stop word lists provided by Snowball are used for French and Finnish stop word removal. The words are then passed to the Snowball stemmer. The only alteration to the default stemmer functionality is the conversion of the Russian character encoding from ISO to KOI-8. Finally, the whitespace preceding the maintained punctuation characters is removed1 .
Text Encoding The OKAPI system does not accept special characters that are used in Finnish, French and Russian. All character strings were encoded with just the 26 lowercase letters a to z. The encoding guarantees that different input words are discriminably represented and that the reverse operation (decoding) can be easily performed. However, the encoded form is not readable by humans and string similarities do not stay intact. The latter is not a problem, since we do not want to retrieve fuzzy matches to our queries with OKAPI. Example: for the three words pécheur, pêcheur and pêcheurs are encoded as gropmdpbtfui, cbppmdpbtfui and klcgrwruwanejd.
This section presentation results and analysis of our experimental runs. We report precision at rank 10, average precsion and total number of relevant documents retrieved. System parameters were selected using CLEF 2003 test collections for each language. All runs use the Title and Description CLEF topic fields. In all cases Okapi K1 = 1.0 and b = 0.75. The following PRF summary sentence selection: T = title method, Q = query-bias method, A = linear summ of all methods, L = Luhn method. The 20 top ranked PRF expansion terms were selected from the summaries of the top 5 ranked documents. The original topic terms were upweighted by a factor of 3.5 relative to terms introduced by PRF. documents used for expansion term selection in both cases. It can be seen that working in French produces superior retrieval performance with respect to both precision and recall metrics.
Russian Runs Table 2 shows results for Russian monolingual retrieval. The PRF summary length is 6 sentences here with 20 documents used for expansion term selection. This is a small document collection and the lack of variation in recall for the different summary methods is perhaps not surprising. Further development of our Russian language preprocessing is planned, but these results are generally encouraging.
Finnish Runs Table 3 shows results for Finnish monolingual retrieval. Summary length is 4 sentences with 30 documents used for expansion term selection. Our preprocessing of Finnish here only employs the Snowball stemming. This does not fully address the complex structure of Finnish word compounds, and further work is planned to extend word decompounding. While average precision appears reasonable here, recall appears poor in some cases, probably resulting from the failure to properly address decompounding.
English Runs Table 4 shows English monolingual results. Our retrieval system appears to be performing fairly well on this dataset. Results are included here for analysis of the multilingual retrieval runs.
German to French Runs Table 5 shows results for German to French bilingual retrieval. PRF summary length is 4 sentences with 20 documents used for expansion term selection. Topics were translated directly from German to French using Systran via the Babelfish (http: //www.babelfish.altavista.com) website. We observed about 30% reduction in average precision relative to monolingual our French retrieval accompanied by a large reduction in relevant documents retrieved. Dutch to French Runs Table 6 shows results for Dutch to French bilingual retrieval. PRF parameters are the same as German to French retrieval, topics again being translated directly using Babelfish. In this case we see that average precision is reduced by only 20% with a matching smaller decrease in relevant retrieved relative to monolingual retrieval.
English to Russian Runs Table 7 shows results for English to Russian bilingual retrieval. PRF summary length is 6 sentences with only 6 documents used for expansion term selection. Topics are translated using Systran (http://www.systranbox.com/systran/box), PROMT (http://www. online-translator.com/default.asp?lang=en) and LogoMedia (http://www.logomedia.net/.) Results are shown for Systran and a union merge of the three translations. The merged results show a marginal reduction in performance metrics, this is perhaps a little surprising with respect to the number of relevant retrieved.
English to Finnish Runs and English to French Runs Table 8 and Table 9 show results for English to Finnish and English to French bilingual retrieval respectively. English to Finnish topic translation was carried out using InterTrans2 and topics translated from English to French using Systran. These results are included here for analysis of the multilingual retrieval runs.
Table 10 shows results for our multilingual runs. In all cases PRF used A type summaries. Multilingual output was generated by merging separate lists using data fusion, each document being assigned a score w = ms(j)/gmax ms, where ms(j) is the original document score and gmax ms is the global maximum ms(j) across the lists being merged. The fused lists shown in Table 11 shows results for these runs broken down by the individual languages in the merged lists. It can be seen that the dramatic reduction in performance between schemes 1 and 2 shown Table 10 results entirely from loss in performance for the French documents. Interestingly the combination with the The Times UK data in scheme 3 appears to overcome this problem. Similarly working with the 4 separate lists in schemes 4 and 5 produces better overall results than scheme 1 with the untranslated documents. The dominance of French in scheme 1 needs to be further investigated. The French collection is by far the largest here, which may lead to it dominating the scores, the errors introduced by document translation may help to ameliorate this effect, but this issue needs to be investigated properly. 12 shows English and French retrieval within the merged collection list used for scheme 2 in Table 10 prior to fusion with Russian and Finnish. Comparing these results with those for scheme in Table 11 it can be seen that loss in retrieval in the multilingual fusion is caused mainly by the behaviour of the French documents, presumably because of low matching scores arising from document translation errors. By contrast Table 13 shows corresponding results for the collection merged with The Times UK 1995. While there is no significant change in the results prior to multilingual fusion, scheme 3 shows a good improvement of scheme 2 in Table 11, the additional information from The Times collection may produce more robust matching scores for the translated French documents.
Columns 3 and 4 of Tables 12 and 13 show results for the English and translated French documents with PRF using the respective merged collections. Column 3 can be compared with column A in Table 4 and column 4 with translated documents column A in Table 9. While there is little change to the effectiveness of English document retrieval from using merged PRF, there is an observable improvement in both precision and recall for the translated French documents. Looking at the behaviour of these lists schemes 4 and 5 of Table 11, there is a little effect on the average performance in the merged lists.
Our work for CLEF 2004 has produced a system that can be easily adapted to different document languages. Further work is needed to improved preprocessing for specific languages. While our multilingual experiments show interesting behaviour for individual language components of merged retrieva lists, further investigation is needed to better understand the reasons for these results.