Introduction

Terrier (http://ir.dcs.gla.ac.uk/terrier) is a toolkit for the rapid development of large-scale Information Retrieval (IR) applications. It is based on a framework for deriving non-parametric probabilistic models for IR. The framework deploys more than 50 Divergence From Randomness (DFR) models for term weighting [1]. The term weighting models are derived by measuring the divergence of the actual term distribution from that obtained under a random process. Terrier was demonstrated to be highly effective at retrieving Web documents at the recent TREC-11 and TREC-12, and is currently available as the search engine of the Web site of the Department of Computing Science at the University of Glasgow (http://www.dcs.gla.ac.uk/search).

In this paper, we report on our participation in the French Monolingual task. Our main aim was to test to which extent our existing English monolingual Terrier retrieval system could perform French retrieval, simply by changing the stemmer and stopword list from English into French. We chose French in order to test our system on new unfamiliar grounds. We opted for minimal language-specific normalisation changes, namely the use of a French stemmer and stopword list, and chose to exclude other performance enhancing options, such as POS-taggers and morphological analysers. Our secondary aim was to continue and complement our earlier work (TREC-11, TREC-12) on studying the effect of length normalisation on the retrieval performance, through the investigation of its impact on French IR. The outcome of this experimentation is being put to practical use, as we are currently working towards merging our existing English and French monolingual retrieval systems into one, thus extending our system to accommodate Cross Language Information Retrieval.

This paper is organised as follows. Section 2 presents a brief overview of the retrieval approaches adopted for our participation in CLEF 2004. Section 3 presents our official retrieval runs for the French monolingual task. Section 4 analyses the obtained results, along with a series of unofficial runs for the said task. Section 5 concludes with a brief summary of our participation in CLEF 2004 and the direction of our future research work.

System Setup

The following preprocessing steps were applied both to documents and queries. All input was tokenized. Punctuation marks and numbers of more than 4 digits were omitted. Proper nouns, abbreviations, acronyms, multi word units and compounds were not extracted or processed. Accents were preserved. Both queries and documents were stopped using the standard French stopword list available with the Snowball stemming algorithm for French [5]. We did not eliminate topic specific phrases such as "Les documents pertinents devront mentionner/parler de/donner des details sur…" from the queries. We did not use a stop stem list, as we used the stopword list before the stemming stage. We used the French stemmer from the Snowball family of stemmers, developed by Martin Porter [5]. The stemmer striped affixes from the index words in a specific order and applied repair strategies, where applicable, in order to reduce the input into clusters of words sharing the same stem.

We experimented with the PL2 weighting model, one of the Divergence From Randomness (DFR) term weighting models developed within Amati & Van Rijsbergen's probabilistic framework for IR [1]. Using the PL2 model, the relevance score of a document d for a query q is given by: ( , )

t q qtf w t d ∈ ⋅

where qtf is the frequency of term t in the query q, w(t,d) is the relevance score of a document d for the query term t, given by:

2 2 2 1 1 ( , ) ( log log 0.5 log (2 )) 12 1 t w t d tf tf e tf tf tf fn n n n n n λ π λ = ⋅ + + − ⋅ + ⋅ ⋅ ⋅ ⋅ +(1)

where λ is the mean and variance of a Poisson distribution. λ is given by

F N , (F << N),

where F is the term frequency of the term t in the whole collection and N is the number of documents in the collection.

tfn is the normalised within-document frequency of the term t in the document d. It is given by the normalisation 2 [1,3]:

2 log 1 _ ( ), ( 0) avg l tfn tf c c l + = ⋅ ⋅ >

where c is a parameter.

tf is the within-document frequency of the term t in the document d.

l is the document length and _ avg l is the average document length in the whole collection.

We estimated the parameter c of the normalisation 2 by measuring the normalisation effect on the term frequency distribution with respect to the document length distribution [4]. More specifically, our tuning approach automatically adjusted the parameter c to a value dependent on the topic fields used. For the runs submitted to CLEF 2004, we obtained the following values: c=4.83 for short queries (only Title field was used), c=1.56 for long queries (all the three fields were used), c=3.1 for queries using the Title and Description fields, and c=2.6 for queries using the Title and Narrative fields.

We have also used a query expansion mechanism, which follows the idea of measuring divergence from randomness. The approach can be seen as a generalisation of the approach used by Carpineto and Romano in which they applied the Kullback-Leibler divergence to the un-expanded version of BM25 [2,3]. The query expansion formula is given by: where -withinDocumentFrequency is the term frequency in the X top-retrieved documents. X depends on the setting we applied as indicated below.

-termFrequency is the term frequency of the given term in the collection.

-totalDocumentLength is the sum of the length of the X top-retrieved documents.

-collectionLength is the number of tokens in the whole collection.

For short queries, we extracted the 10 most informative terms from the top 3 retrieved documents as the expanded terms. For long queries, we extracted the 100 most informative terms from the top 25 retrieved documents as the expanded terms. For queries using the Title and Description fields we extracted the 10 most informative terms from the top 15 retrieved documents, and for queries using the Title and Narrative fields we extracted the top 15 informative terms from the top 3 retrieved documents. We added these terms to the query and repeated the retrieval stage.

Runs

This section presents our French monolingual retrieval runs submitted to CLEF 2004. We realised our In addition to the above runs, we also undertook further experiments, in order to test additional query length and query expansion settings, varying the number of expanded terms and the used number of top retrieved documents.

Results

This section summarises and discusses the results of our CLEF 2004 participation and of our additional runs. The best run was the one combining the topic fields of Title and Description (UOGTD), which slightly exceeded our best submitted run (UOGLQ). Overall, query length had little impact on the performance of the runs (MAP varied from 0.4237 to 0.4485).

In general, query expansion decreased the mean average precision of all the runs (see Table 2). Table 3 shows that query expansion does not work, independently of the length of the query. In order to analyse the low performance of query expansion, we ran additional experiments with query expansion varying the number of expanded terms (#terms) and the used number of top retrieved documents (#documents) compared to the setting presented in Section 2. Table 4 shows the effect of the said parameter tuning on the performance of the system. Overall, query expansion deteriorated performance, independently of the used parameters. The used parameters in the official submitted runs were the optimal ones. Finally, subsequent experiments revealed that the parameter c of the normalisation which was estimated by our tuning approach automatically (see Section 2) was indeed optimal, proving thus that the parameter tuning approach for term frequency normalisation adopted [4]

Conclusions and Future Work

This paper presented a French monolingual IR system developed at the University of Glasgow. The system was evaluated in the French monolingual track of CLEF 2004.

The experiments on which we briefly reported indicated the following. Our existing Terrier retrieval platform was shown to be truly modular, as it was extended to perform French monolingual IR successfully, simply by changing the stemming and stopword components from English into French, therefore with a very low overhead. Moreover, we found that query expansion performed poorly.

We are now investigating optimal ways to merge the French monolingual retrieval system described in this paper, with our existing English monolingual retrieval system, into a single crosslingual retrieval platform, the performance of which is to be tested in the CLEF 2005 multilingual track.