In the CLEF 2005 Ad-Hoc Track we experimented with language-specific morphosyntactic processing and light Natural Language Processing (NLP) for the retrieval of Bulgarian, French, Italian, English and Greek.
This paper is organised as follows. Section 2 presents an overview of the linguistic foundations of this work. Section 3 discusses our monolingual and bilingual runs. Section 4 concludes with a summary of the approaches tested and the extent of their success.
The driving force behind our participation in CLEF 2005 has been to explore the effect of diverse languages across a set Information Retrieval (IR) platform. It was anticipated that this effect would be considerable, not only in terms of technical implementation issues, but also in terms of language resources. We used the same retrieval platform as reported in CLEF 2004 [2], on top of which we added Natural Language Processing (NLP).
NLP is assumed to be essential in IR for morphologically rich languages. We tested the validity of this statement for Greek -English retrieval. Also, noun phrase extraction, a popular NLP application, has been tested for monolingual French and bilingual Italian -French retrieval, using our in-house Noun Phrase (NP) extractor. Other NLP applications used in the context of this work include light syntactic analysis, achieved by a probabilistic part-of-speech tagger, lemmatisation and morphological analysis [6,7].
The motivation behind our participation in CLEF 2005 was to examine the performance of a set IR platform across a span of dissimilar languages, and thus reveal the extent to which retrieval models and system tuning issues are accountable for IR performance on a per language basis. We used our existing retrieval platform, which accommodates a range of matching models and a strong query expansion baseline [2,5].
For Bulgarian, the lack of language resources meant that the collection was simply stemmed and indexed, without any supplementary analysis. Stemming was realised using the Russian Snowball stemmer [4]. For the English -Bulgarian retrieval, the Skycode machine translation system was used [3]. Table 2 Table 2 reveals the modifying influence of translation on retrieval performance, which is even stronger for shorter topics. The overall performance of the matching models is highly correlated, with a p-value of 0.00048. This delineates the need for additional language resources for Bulgarian, whose evidence would weigh more heavily on retrieval performance than that of simple stemming.
For Greek we used a POS tagger and morphological analyser developed by Xerox [8]. Closed class terms were rejected to reduce noise, while lemmas were automatically translated into English using Babelfish [1]. The performance of these runs is presented in Table 3. Submitted runs are printed in boldface and optimal runs in italics. 3. English and Greek -English Mean Average Precision (MAP) Table 3 shows that translation has a considerable effect on retrieval performance. The overall scores for Greek -English retrieval are closer to their monolingual equivalents than the English -Bulgarian scores are to the monolingual Bulgarian scores. This underlines the auxiliary service rendered to the Greek topics by the morphological analysis and lemmatisation. Table 3 also reveals that light syntactic analysis and lemmatisation can assist retrieval just as well as stemming.
For French we used a variation of the monolingual French strategy tested in CLEF 2004 [2]. We opted for a less aggressive stemming approach, which targets mainly inflectional variants. A probabilistic POS tagger [6] provided a pellucid syntactic analysis of the topics. Noun phrases were extracted using our in-house NP extractor. For Italian -French, Italian noun phrases were extracted and translated separately into French, using Worldlingo [7]. The collective scores of the above runs are presented in Table 4. Submitted runs are printed in boldface and optimal runs in italics. 4 shows that POS analysis and NP extraction is associated with better retrieval performance, and appears to benefit monolingual retrieval more than it assists bilingual retrieval, as can be deduced by the fact that the difference between the monolingual and bilingual runs is higher when POS NP is used (29.58% on average), than when it is not (26.78% on average). This observation is indicative of the fact that even though light NLP can be of significant assistance to IR, it cannot counter the shortcomings of insufficient translation resources.
Our participation in the CLEF 2005 Ad-Hoc track for Bulgarian, English -Bulgarian, French, Italian -French and Greek -English retrieval was shown to be successful, with a difference from the Median Precision ranging between +1.135 (for Bulgarian) and +7.830 (for English -Greek). On a collective basis, poor or no language resources were at all times associated with consistently low retrieval performance. On an individual basis, lemmatisation was shown to be a satisfactory replacement of stemming for Greek, while noun phrase extraction was shown to benefit retrieval directly and consistently for French and Italian -French. We have shown that light morphosyntactic processing can assist IR for highly inflectional languages, and by doing so, we have carried our initial contention a posse ad esse successfully.