The Dublin City University group participated in the monolingual, bilingual and multilingual retrieval tasks this year. The main focus of our investigation this year was extending our retrieval system to document languages other than English, and completing the multilingual task comprising four languages: English, French, Russian and Finnish. Results from our French monolingual experiments indicate that working in French is more effective for retrieval than adopting document and topic translation to English. However, comparison of our multilingual retrieval results using different topic and document translation reveals that this result does not extend to retrieved list merging for the multilingual task in a simple predictable way.
In this section we first give a brief overview of our retrieval system, and then describe how we adapted it to be language independent and the specific detail of preprocessing for our CLEF 2004 runs.
For English language runs we continued to use this system. The documents and search topics
were processed to remove stopwords from a list of about 260 words; suffix stripped using the Okapi
implementation of Porter stemming [
Terms are weighted using the standard BM25 weighting scheme and all runs use our
summarybased pseudo relevance feedback (PRF) method [
In order to use the Okapi software with non-English document we carried out the language preprocessing outside Okapi and then encoded the resulting character strings into ASCII as follows. Language Preprocessing The documents and topic are prepared using a pipeline of preprocessing components. Firstly, the data is tokenised to isolate the text body from the SGML/XML markup tags. Then, all punctuation characters are deleted from the text body, with the following exceptions: full stops, commas, semi-colons, colons, exclamation marks and question marks. Whitespace is inserted to separate these punctuation characters from word tokens. The third step is the conversion of characters to lower case. A finite set of upper case characters are mapped to lower case equivalents. Distinct mappings must be used for each character set. The Russian characters were converted to KOI-8 character encoding as required by the Snowball tools, while the Finnish and French documents use ISO Latin 1. Conversion of the Russian data loses some data, for example the degree sign prevalent in weather forecasts is lost, further some corruptiion of the original data to “boxdrawing” symbols was observed. The Russian stopword list used here consists only of the simple first part of the Snowball list.
At the next stage stop words are removed. The stop word lists provided by Snowball are used for French and Finnish stop word removal. The words are then passed to the Snowball stemmer. The only alteration to the default stemmer functionality is the conversion of the Russian character encoding from ISO to KOI-8. Finally, the whitespace preceding the maintained punctuation characters is removed 1.
Text Encoding The OKAPI system does not accept special characters that are used in Finnish, French and Russian. All character strings were encoded with just the 26 lowercase letters a to z. The encoding guarantees that different input words are discriminably represented and that the reverse operation (decoding) can be easily performed. However, the encoded form is not readable by humans and string similarities do not stay intact. The latter is not a problem, since we do not want to retrieve fuzzy matches to our queries with OKAPI. Example: for the three words p´echeur, pˆecheur and pˆecheurs are encoded as gropmdpbtfui, cbppmdpbtfui and klcgrwruwanejd. 3
This section presentation results and analysis of our experimental runs. We report precision at rank 10, average precsion and total number of relevant documents retrieved. System parameters were selected using CLEF 2003 test collections for each language. All runs use the Title and Description CLEF topic fields. In all cases Okapi K1 = 1.0 and b = 0.75. The following PRF summary sentence selection: T = title method, Q = query-bias method, A = linear summ of all methods, L = Luhn method. The 20 top ranked PRF expansion terms were selected from the summaries of the top 5 ranked documents. The original topic terms were upweighted by a factor of 3.5 relative to terms introduced by PRF.
French Runs Table 1 shows results for French monolingual retrieval. Separate results are shown for documents and topics in French and translated into English into Systran MT. For French document PRF summary length is 4 sentences and for translated 6 sentences, with 20 1Punctuation must be maintain to facilitate document summarization for PRF.
T 0.361 0.410 844
Q 0.365 0.414 849
Finnish Runs Table 3 shows results for Finnish monolingual retrieval. Summary length is 4 sentences with 30 documents used for expansion term selection. Our preprocessing of Finnish here only employs the Snowball stemming. This does not fully address the complex structure of Finnish word compounds, and further work is planned to extend word decompounding. While average precision appears reasonable here, recall appears poor in some cases, probably resulting from the failure to properly address decompounding.
English Runs Table 4 shows English monolingual results. Our retrieval system appears to be performing fairly well on this dataset. Results are included here for analysis of the multilingual retrieval runs. 3.2
German to French Runs Table 5 shows results for German to French bilingual retrieval. PRF summary length is 4 sentences with 20 documents used for expansion term selection. Topics were translated directly from German to French using Systran via the Babelfish (http: //www.babelfish.altavista.com) website. We observed about 30% reduction in average precision relative to monolingual our French retrieval accompanied by a large reduction in relevant documents retrieved.
T 0.263 0.295 72.0% 727
English to Russian Runs Table 7 shows results for English to Russian bilingual retrieval. PRF summary length is 6 sentences with only 6 documents used for expansion term selection. Topics are translated using Systran (http://www.systranbox.com/systran/box), PROMT (http://www. online-translator.com/default.asp?lang=en) and LogoMedia (http://www.logomedia.net/.) Results are shown for Systran and a union merge of the three translations. The merged results show a marginal reduction in performance metrics, this is perhaps a little surprising with respect to the number of relevant retrieved.
English to Finnish Runs and English to French Runs Table 8 and Table 9 show results for English to Finnish and English to French bilingual retrieval respectively. English to Finnish topic translation was carried out using InterTrans2 and topics translated from English to French using Systran. These results are included here for analysis of the multilingual retrieval runs. 3.3
2Kindly provided by Jacques Savoy.
collection for retrieval, 4: separate English, translated French, Russian and Finnish fused, 5: as 4, expect English and French PRF expansion taken from merged collection from 2.
Table 11 shows results for these runs broken down by the individual languages in the merged lists. It can be seen that the dramatic reduction in performance between schemes 1 and 2 shown Table 10 results entirely from loss in performance for the French documents. Interestingly the combination with the The Times UK data in scheme 3 appears to overcome this problem. Similarly working with the 4 separate lists in schemes 4 and 5 produces better overall results than scheme 1 with the untranslated documents. The dominance of French in scheme 1 needs to be further investigated. The French collection is by far the largest here, which may lead to it dominating the scores, the errors introduced by document translation may help to ameliorate this effect, but this issue needs to be investigated properly.
Merged English and French Collections Table 12 shows English and French retrieval within the merged collection list used for scheme 2 in Table 10 prior to fusion with Russian and Finnish. Comparing these results with those for scheme in Table 11 it can be seen that loss in retrieval in the multilingual fusion is caused mainly by the behaviour of the French documents, presumably because of low matching scores arising from document translation errors. By contrast Table 13 shows corresponding results for the collection merged with The Times UK 1995. While there is no significant change in the results prior to multilingual fusion, scheme 3 shows a good improvement of scheme 2 in Table 11, the additional information from The Times collection may produce more robust matching scores for the translated French documents.
Columns 3 and 4 of Tables 12 and 13 show results for the English and translated French documents with PRF using the respective merged collections. Column 3 can be compared with column A in Table 4 and column 4 with translated documents column A in Table 9. While there is little change to the effectiveness of English document retrieval from using merged PRF, there is an observable improvement in both precision and recall for the translated French documents. Looking at the behaviour of these lists schemes 4 and 5 of Table 11, there is a little effect on the average performance in the merged lists. 2 3 4 5
Prec. 10 docs Av Precision
Rel. Ret.
Prec. 10 docs Av Precision
Rel. Ret.
Prec. 10 docs Av Precision
Rel. Ret.
Prec. 10 docs Av Precision
Rel. Ret.
Prec. 10 docs Av Precision Rel. Ret. 4
Our work for CLEF 2004 has produced a system that can be easily adapted to different document languages. Further work is needed to improved preprocessing for specific languages. While our multilingual experiments show interesting behaviour for individual language components of merged retrieva lists, further investigation is needed to better understand the reasons for these results.