We describe the University of Amsterdam's participation in the WebCLEF track at CLEF 2005. We submitted runs for both the mixed monolingual task and the multilingual task.
In the CLEF 2005 WebCLEF track, we took part in two of the retrieval tasks. We took part in the WebCLEF mixed monolingual task. Our participation here was aimed at evaluating the effectiveness of standard ad hoc retrieval settings for a stream of topics in various languages. Our assumption was that this would shed new light on the robustness of modern information retrieval techniques.
We also took part in the WebCLEF multilingual task. Our participation here was aimed at
evaluating the effectiveness of straightforwardly combining runs using a number of translations of
the original English queries. Such methods have previously been used successfully at the CLEF
multilingual ad hoc retrieval task [
This paper is structured as follows. In Section 2 we describe our retrieval system as well as the approaches used for the two WebCLEF tasks in which we participate. Section 3 describes our official retrieval runs for WebCLEF 2005, and Section 4 discusses the results we have obtained. Finally, in Section 5, we offer some conclusions regarding our multilingual web retrieval efforts.
Our retrieval system is based on the Lucene engine with a number of home-grown extensions [
For our ranking, we used the default similarity measure in Lucene [
|D| freq(t, D) 1 + log p d
| | |q ∩ d| |q| t∈q sX tft,q · idft 2 2.2
We indexed the whole collection by simply extracting the full text from the documents. We did not apply any stemming nor did we use a stopword list. We applied case-folding and normalized marked characters to their unmarked counterparts, i.e., mappingo¨ to o, ae to ae, ˆıto i, etc. The only language specific processing we did was a transformation of the multiple Russian encodings into an ASCII transliteration. 2.3
We used the WorldLingo machine translation [
We combined various ‘base’ runs using the unweighted CombSUM function of Fox and Shaw [
We submitted one run to the mixed-monolingual task. The run uses the short topic statement in the htitlei efild of the WebCLEF 2005 topics. Our run uses Lucene’s standard ranking formula applied on our full-text index (as discussed in Section 2 above). 3.2
We submitted four runs to the multilingual task. All runs use the English version of the short topic statement in the htranslation language=”EN”i field of the WebCLEF 2005 topics, and the translations mentioned in Section 2.3.
We experimented along two dimensions. The rfist dimension is the number of topic languages: All topics Home pages Named pages
Recall that WebCLEF provides a stream of topics, with topics from arbitrary languages. For the multilingual task, we use the English short topic statement. The downside of this is, of course, that finding the targeted page in the source language becomes a formidable problem. The upside is that, at least, the topic language is known, and the same holds for the translations we obtained. The second dimension we experiment with is trying to exploit this knowledge: All results Topics in one language may likely retrieve pages in other languages as well. A case in point is WebCLEF topic WC0014, whose English topic statement (“Chancellery at the Spreebogen”) could still allow us to retrieve German pages targeted by the German topic statement (“Bundeskanzleramt am Spreebogen”). Hence, we may simply use all pages retrieved by a topic of a particular, known language.
Language restricted Since we know the language of the topic in each of the translations, and the intention of the translated topic is to retrieve pages in that language, we may decide to restrict the pages returned by our retrieval system. We do this by restricting retrieved pages to the dominant domains. For example, for a run with the topics translated to Dutch, we restrict pages to come from either the .nl or the .eu.int domain.
Combining the two dimensions naturally suggests the four following cases: 1. using nine topic languages without restriction; 2. using nine topic languages and restricting pages to dominant domains; 3. using five topic languages without restriction; and 4. using five topic languages and restricting pages to dominant domains.
For each of the cases, we obtain vfie to nine different runs, which we combine using unweighted CombSUM. This results in the four runs submitted to WebCLEF 2005. 4 4.1
We submitted a single no-thrills run for the mixed monolingual task, using standard ad hoc document retrieval setting (as discussed in Section 3). Table 1 reports the result of the mixed monolingual run. A number of observations present themselves. First, we see that, on average, the desired page is found in the top three. That is a reassuring result for the mixed monolingual All topics Nine languages Nine languages, restricted Five languages Five languages, restricted
Nine languages Nine languages, restricted Five languages Five languages, restricted
Nine languages Nine languages, restricted Five languages Five languages, restricted MRR 0.0072 0.0157 0.0084 0.0163 MRR 0.0109 0.0158 0.0129 0.0168 S@1 0.0041 0.0124 0.0041 0.0124
S@1 0.0066 0.0066 0.0066 0.0066 S@5 0.0083 0.0165 0.0083 0.0165
S@5
0.0066
0.0230
0.0098
0.0230
S@10
0.0124
0.0165
0.0124
0.0207
S@10
0.0197
0.0262
0.0197
0.0262
task. Somewhat worrying is the success rate at rank 10, with no relevant page found for over
40% of the topics. Second, named page topics score somewhat higher than home page topics, on
all measures. This is well-known from other web retrieval tasks [
We submitted four runs for the multilingual task (as discussed in Section 3). We will rfist look at the overall results, and then focus on the effectiveness for each of the languages in which we translated the English topics.
Table 2 reports the result of the multilingual runs. Again, we make a number of observations. First, we see that scores are substantially lower than for the mixed monolingual task. The complexity of the multilingual task can hardly be overestimated: given an English query we have to guess what page in any language has to be returned to the user. Obvious ways of limiting this wealth of options are the use of topic meta-fields, or of sophisticated techniques to extract target language cues. Second, our experiment with the number of translations to use, points conclusively to the smaller set of vfie language used frequently in the topic set. It is a reassuring fact that the improvement is moderate, and the extended set of translations is far from detrimental to the performance. Note that the extended set includes, for example, Italian, which is not used in any of the topics. Third, our experiment with restricting our intention to pages in the language of the topic translation is clearly successfull. It leads to substantial improvement of the score.
We now zoom in on the effectiveness of the individual translations. Table 3 lists the results of the translated queries, both evaluated against the whole topic set, as well as against all topics targeting a page in the language at hand. We see the following. First, when looking at the restricted topic sets, effectiveness varies from total failure (Greek) to perfection (French). The score for the five frequent languages is reasonable compared to those of the mixed monolingual task. Hence, one may conclude that the automatic topic translations are effective. Second, when looking at all topics, the scores are generally unimpressive and mirroring the frequency with which a topic of the given language appears in the topic set. This comes as no surprise, given that the topic set covers eleven languages, and each of the topic translations will dominantly target only one of them. Third, the translated topics pick up relevant pages in languages other than the target language. In particular, the Italian topics do pick up a relevant page for 35 of the topics. Fourth, the single topic language runs are still much more effective than the combined multilingual runs in Table 2. This is a disappointing result, and clearly indicates that the straightforward run combination is ineffective. On a more positive note, however, the results for the individual translations strongly suggest that more sensible methods are possible. 5
This paper documents the University of Amsterdam’s participation in the CLEF 2005 WebCLEF track. The EuroGOV collection used at WebCLEF is based on a crawl of governmental information from a range of sites. Such a collection of web data is much noisier than traditional collections of newswire and newspaper data originating from a single source. Moreover, the linguistic variety in the collection makes it harder to apply language-specific processing methods such as stemming algorithms. Hence, we simply indexed the collection by extracting the full text from the documents.
For the mixed monolingual task, we submitted a single, standard ad hoc retrieval run. Our main finding is that such a straightforward approach is relatively effective, that uses no web specific settings. Considering the fact that we are dealing with a stream of topics in eleven languages, and with an even greater number of languages in the collection, this sheds new light on the robustness of modern information retrieval techniques.
For the multilingual task, we experimented with different numbers of translations of the English queries, and with restricting the returned pages to the now-known language of the query at hand. Our experiments show benecfiial effects for restricting the number of translation to those occurring frequently in the topic set, as well as for limiting query translations to return only pages in the language of the query. In general, however, the combined results for the multilingual task are unimpressive. The individual query translations, however, seem relatively successful in targeting their share of relevant pages. This casts considerable doubt on the effectiveness of standard combination methods for this particular task. 6
We want to thank Valentin Jijkoun for help with the Russian collection. Jaap Kamps was supported by a grant from the Netherlands Organization for Scienticfi Research (NWO) under project numbers 612.066.302 and 640.001.501. Maarten de Rijke was supported by grants from NWO under project numbers 017.001.190, 220-80-001, 264-70-050, 354-20-005, 612-13-001, 612.000.106, 612.000.207, 612.066.302, and 612.069.006.