In the CLEF 2005 Ad-Hoc Track we experimented with language-specific morphosyntactic processing and light Natural Language Processing (NLP) for the retrieval of Bulgarian, French, Italian, English and Greek. Categories and Subject Descriptors [H.3 Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing - Linguistic Processing; H.3.3 Information Search and Retrieval - Information Filtering, Retrieval Models.
The motivation behind our participation in CLEF 2005 was to examine the performance of a set IR platform across a span of dissimilar languages, and thus reveal the extent to which retrieval models and system tuning issues are accountable for IR performance on a per language basis. We used our existing retrieval platform, which accommodates a range of matching models and a strong query expansion baseline [2, 5].
For Bulgarian, the lack of language resources meant that the collection was simply stemmed and indexed, without any supplementary analysis. Stemming was realised using the Russian Snowball stemmer [4]. For the English – Bulgarian retrieval, the Skycode machine translation system was used [3]. Table 2 summarises these runs. Submitted runs are printed in boldface and optimal runs in italics.
Table 2 reveals the modifying influence of translation on retrieval performance, which is even stronger for shorter topics. The overall performance of the matching models is highly correlated, with a p-value of 0.00048. This delineates the need for additional language resources for Bulgarian, whose evidence would weigh more heavily on retrieval performance than that of simple stemming.
For Greek we used a POS tagger and morphological analyser developed by Xerox [8]. Closed class terms
were rejected to reduce noise, while lemmas were automatically translated into English using Babelfish [
Table 3 shows that translation has a considerable effect on retrieval performance. The overall scores for Greek – English retrieval are closer to their monolingual equivalents than the English – Bulgarian scores are to the monolingual Bulgarian scores. This underlines the auxiliary service rendered to the Greek topics by the morphological analysis and lemmatisation. Table 3 also reveals that light syntactic analysis and lemmatisation can assist retrieval just as well as stemming.
For French we used a variation of the monolingual French strategy tested in CLEF 2004 [2]. We opted for a less aggressive stemming approach, which targets mainly inflectional variants. A probabilistic POS tagger [6] provided a pellucid syntactic analysis of the topics. Noun phrases were extracted using our in-house NP extractor. For Italian – French, Italian noun phrases were extracted and translated separately into French, using Worldlingo [7]. The collective scores of the above runs are presented in Table 4. Submitted runs are printed in boldface and optimal runs in italics.
Our participation in the CLEF 2005 Ad-Hoc track for Bulgarian, English – Bulgarian, French, Italian – French and Greek – English retrieval was shown to be successful, with a difference from the Median Precision ranging between +1.135 (for Bulgarian) and +7.830 (for English – Greek). On a collective basis, poor or no language resources were at all times associated with consistently low retrieval performance. On an individual basis, lemmatisation was shown to be a satisfactory replacement of stemming for Greek, while noun phrase extraction was shown to benefit retrieval directly and consistently for French and Italian – French. We have shown that light morphosyntactic processing can assist IR for highly inflectional languages, and by doing so, we have carried our initial contention a posse ad esse successfully. [2] Lioma, C., He, B., Plachouras, V., and Ounis, I. "The University of Glasgow at CLEF 2004: French Monolingual Information Retrieval with Terrier". In Multilingual Information Access for Text, Speech and Images: Results of the Fifth CLEF Evaluation Campaign, Volume Editors: C. Peters, P. D. Clough, G. F. J. Jones, J. Gonzalo, M. Kluck and B. Magnini, Lecture Notes in Computer Science, Springer-Verlag, 2005.