In the CLEF 2005 initiative, multlingual web retrieval was integrated as a task for the first time. This paper describes experiments based on one multilingual index carried out at the University of Hildesheim. Several indexing strategies based on a multi-lingual index have been tested with the EuroGOV corpus. Boosting topic fields with higher weight led to best results during post submission runs. The experiments also led to experiences in working with large test collections and the challenges associated with them.
Web search engines has become a part of every day life for many people. The development of information retrieval systems for the web is faced with many challenges (Arasu et al. 2001). Systems give different answers to these challenges and it is difficult to judge the effect of decisions during the design of search enigne. As a consequence, there is a great need for evaluation in web retrieval (Hawking 2000). The web is also a natural source for multilingual documents.
Within the Cross Language Evaluation Forum (CLEF) the web track has been created (Sigurbjörnsson et al. 2005b). A large multilingual corpus has been collected and distributed (Sigurbjörnsson et al. 2005a). In our first participation, we intended to tune our system to the challanges of a large web corpus. For the experiments, language resources in all languages were not available from ad-hoc retrieval. As a consequence, we considered n-gram indexing for the web retrieval task (McNamee & Mayfield 2004).
Since the files of the EuroGOV corpus were not released in well formed XML, substantial effort for data preprocessing was necessary. A corpus in well formed XML would allow us to use the System implemented during the CLEF 2004 campaign for multilingual ad-hoc tasks (Hackl et al. 2005). The two main items in the EuroGOV files that needed replacing were predeclared entities. This step was required for ampersand with the associated entity reference in the URL fields of the individual documents and all nested CDATA tags. The first attempt to reformat the files has been carried out by a Perl-script. At the first view, it seemed that the Perl-scirpt would work perfectly for our needs. Unfortunately, we realized that during the process of indexing the corpus, the XML parser would frequently report "parser exceptions" that we traced back to the fact that the XML files still contained a couple of not adjusted predeclared entities. Having this in mind, a Java program was developed that worked through the whole corpus perfectly. It seems that Perl is not able to process EuroGOV files bigger than 250 MB since we successfully tested the Perl-script with the small-size files (22 MB, 59 MB & 220 MB) of the corpus.
As mentioned in the introduction, one multilingual index was created. In order to generate a slim index we assembled a multilingual The bases for this list were the stopwordlists supplied by the University of Neuchatel1 and a list developed specifically for the Czech language (Hofman Miquel 2005). All lists were combined and revised into one file. This multilingual stopwordlist covers twelve languages and was used for the indexing process of the corpus.
For our retrieval experiments, we created three different multilingual indexes. Two were created with the Lucene StandardAnalyzer 2 , which does not implement any linguistic processing apart from word segmentation. The first index covered the whole corpus whereas the second index cut off the indexing process after a maximum of 200 characters for each individual document. Due to this approach, the sizes of the indexes varies from 5 GB to 700 MB.
The third index was created with a NGram Analyzer also applied to multilingual ad-hoc retrieval before (Hackl et al. 2005). Because of performance and time restrictions the trigram approach was only applied to the title field of the individual documents in the corpus files. As a result the size of the index is down to 300 MB which led to a very quick and stable performance at retrieval time. These three indexes are the foundation for our experiments. As a main retrieval engine, we used Lucene 1.4 3 . Some of the basic code for retrieval and n-gram analysis was adopted from previous CLEF ad-hoc experiments (Hackl et al. 2005). Six different baseline runs were submitted. We did not use any of the metadata that was supplied by the topics due to time and resource constraints. Our monolingual queries were created with the title field of the topic whereas the multilingual queries were based on the monolingual title field and the translation language English field. Both types of queries were sent to one multilingual index. Results are shown in table 1. Looking at the results of the submitted runs it becomes clear that the trigram index did not confirm the expectations. On average, the monolingual runs differ from the multilingual runs by about 0.0162 MRR points.
Having those results in mind the method of indexing the corpus with the Lucene StandardAnalyzer turned out to be more effective than the trigram strategy. The post experiments will illustrate this effect more clearly.
For our post experiments we decided to generate another trigram index covering the whole corpus. The purpose of this experiment was to confirm the results from the official runs or to improve them by providing a better or more a complete index respectively. We also wanted to see if through boosting of the individual query fields (title & translation language English) the difference between the mono-and multilingual runs could be compensated or even improved. As table 2 shows quite obviously, even a more complete index was not able to improve the MRR of the trigram runs. The results declined by approx. 50 %.
For the first web track at CLEF we intended to tune our system to be able to cope with a large amount of data. We suceeded in returning valid results for several runs.
In future experiments, we intend to step beyond the baseline runs and try to involve the metadata that is being provided by the WebCLEF topics. We also want to include advanced quality measures into consideration. Link based quality measures seem to be integral part of commercial search engines. They have been evaluated at the web track at TREC (Hawking 2000). Advanced quality measures take more features into account, especially information and design aspects (Mandl 2005).