We describe our participation in the TEL@CLEF task of the CLEF 2008 ad-hoc track, where we measured the retrieval performance of the IR service that is currently under development as part of the DIGMAP project. DIGMAP's IR service is mostly based on Lucene, together with extensions for using query expansion and multinomial language modelling. In our runs, we experimented combinations of query expansion, Lucene's off-the-shelf ranking scheme and the ranking scheme based on multinomial language modelling. Results show that query expansion and multinomial language modelling both result in increased performance. One task of the ad-hoc track at the 2008 edition of the Cross Language Evaluation Forum (CLEF) addresses the problem of searching and retrieving relevant items from collections of bibliographic records from The European Library (TEL@CLEF). Three target collections were provided, each corresponding to a monolingual retrieval task where we participated: • TEL Catalogue records in English. Copyright British Library (BL) • TEL Catalogue records in French. Copyright Bibliothèque nationale de France (BnF) • TEL Catalogue records in German. Copyright Austrian National Library (ONB) The evaluation task aimed at investigating the best approaches for retrieval from library catalogues, where the information is frequently very sparse and often stored in unexpected languages.
2.1
ranking(q, d) = ∑
tft,q ⋅ idft ⋅ tft,d ⋅ idft ⋅ coordq,d ⋅ weightt , t ∈q normq normd where tft,X = idft = 1 + log normq = normd = coordq,d = ∑ tft,q ⋅ idft2 , t ∈q | d |, | q ∩ d |
| q | termFrequency(t, X ),
| D | documentFrequency(t, D)
,
Lucene has been extensively used in previous editions of the CLEF, NTCIR and TREC joint evaluation experiments. € 2.2
We experimented with a Lucene extension that implements a retrieval scheme based on
estimating a language model (LM) for each document, using the formula described by
t ∈q
In the formula, d is a document and t is a term in query q. The probabilities are reduced to rankequivalent logs of probabilities. To account for data sparseness, the likelihood P(t|d) is interpolated using Jelinek-Mercer smoothing.
In the formula, D is the collection and λ is a smoothing parameter (in our experiments set to the default value of 0.15). The model needs to estimate three probabilities: the prior probability of the document, P(d); the probability of observing a term in a document, P(t|d) and the probability of observing the term in the collection, P(t|D). Assuming the query terms to be independent, and using a linear interpolation of a document model and a collection model to estimate the probability of a query term, the probabilities can be estimated using maximum likelihood estimates:
P(t | d) = P(t | D) = P(d) =
t'∈D | d | ∑| d'| d '∈D termFrequency(t, d)
| d | documentFrequency(t, D) ∑ documentFrequency(t', D)
This language modelling approach has been used in past experiments within the CLEF, NTCIR
and TREC joint evaluation campaigns – see for example
The fact that there are frequently occurring spelling variations and synonyms for any query term
degrades the performance of standard techniques for ad-hoc retrieval. To overcome this problem, we
experimented with the method for pseudo feedback query expansion proposed by
Assuming that the top D documents returned for an original query qi are relevant, a better query qi+1 can be given by the terms resulting from the formula bellow: qi+1 = α ⋅ qi + β
⋅ ∑ termWeight(dr ) | D | dr ∈D In the formula, α and β are tuning parameters. In our experiments, they were set to the default values of 1.0 and 0.75. The system was allowed to add up to 200 terms extracted from the 10 highest ranked documents (i.e. the |D| parameter) from the original query qi. The query expansion method was tuned € through experiments with the ad-hoc collections and relevance judgements from previous CLEF editions. 2.4
Before the actual indexing, the document collections (i.e. the bibliographic records) were passed
through the following pre-processing operations:
• Field Weighting - The bibliographic records composing the collections from the TEL@CLEF
experiment contain structured information in the form of document fields such as title or subject. We
use the scheme proposed by
Topic processing was fully automatic and the queries submitted to the IR engine were generated using all parts of the topics (i.e. title, description and narrative). The generation of the actual queries from the query topics was based on the following sequence of processing operations: • • •
Parsing and Normalisation - All characters were reduced to the lowercase unaccented equivalents (i.e. “Ö” reduced to “o” and “É” to “e” etc.) in order to maximise matching.
Stop Word Removal - Stopword lists were used to remove terms that carry little meaning and would
otherwise introduce noise. The considered stop words came from the minimized lists distributed with
Lucene, containing words such as articles, pronouns, prepositions, conjunctions or interjections. For
English, French and German, these lists contained 120, 155 and 231 terms, respectively.
Retrieval – The resulting queries were submitted to the IR system, which had been used to index the
document collections. In some of the submitted runs, variations of the
Lucene internally normalizes documents and queries to lower case, also removing stop-words. However, explicitly introducing these operations when processing the topics has the advantage of facilitating the development of more advanced topic processing (e.g. adding query expansion methods). 3
We submitted 12 official runs to the CLEF evaluation process, a total of 4 runs for each of the languages/collections under consideration in the monolingual task. The conditions under test for each of the submitted runs are as follows: 4 http://lucene-qe.sourceforge.net/ 5 http://snowball.tartarus.org/
Baseline run using the off-the-shelf retrieval model from Lucene.
Lucene with the language modelling extension.
Lucene with the language modelling extension and language-specific stemming algorithms. Lucene’s off-the-shelf retrieval model with the extension for doing Rocchio query expansion.
We also discuss here the results of some unofficial runs that resulted from experiments that we performed with our retrieval engine. The test conditions for these unofficial runs are:
Lucene with the language modelling extension and Rocchio query expansion.
Lucene with the language modelling extension, Rocchio query expansion and stemming. Lucene’s off-the-shelf retrieval model with Rocchio query expansion and stemming.
Results
The charts at Figure 1 show precision-recall curves for the official runs, separating the results according to the language (i.e. English, French and German submissions, from left to right).
The obtained results support the support the hypotheses that using Rocchio query expansion and a ranking scheme based on language modelling can be beneficial to the CLEF ad-hoc task. Our official runs only made use of relatively simple techniques, but we’re now in the process of implementing additional features into our retrieval engine. These include geographic information retrieval extensions with basis on Local Lucene6 and advanced query expansion methods using bibliographic information. 6