This paper presents a report on our participation in the CLEF-2009 monolingual and bilingual ad hoc TEL@CLEF tasks involving three different languages: English, French and German. Language modeling is adopted as the underlying information retrieval model. While the data collection is extremely sparse, smoothing is particular important when estimating a language model. The main purpose of the monolingual task is to compare different smoothing strategies and investigate the effectiveness of each alternative. This retrieval model is then used alongside a document re-ranking method based on Latent Dirichlet Allocation (LDA) which exploits the implicit structure of the documents with respect to original queries for the monolingual and bilingual tasks. Experimental results demonstrated that three smoothing strategies behave differently across testing languages while LDA-based document re-ranking method should be considered further in order to bring significant improvement over the baseline language modeling systems in the cross-language setting.
two models. The main object of smoothing is to adjust the maximum likelihood estimator of a language
model so that it will be more accurate [
However, previous success over news collection data does not necessarily mean it will be efficient over the library data. Firstly the data is actually multilingual: all collections to a greater or lesser extent contain records pointing to documents in other languages. However this is not a major problem because the majority of documents in the test collection are written in main languages of those test collections. Furthermore, documents written in different languages tend not to match the queries in main languages. The main characteristic of the data is that it is very different from the newspaper articles and news agency dispatches previously used in the CLEF. The data tends to be very sparse. Many records contain only title, author and subject heading information; other records provide more detail (see the experiment section on what fields are chosen to include). The average document lengths are 14.66 for BL and 24.19 for BNF collections after pre-processing, respectively.
Recently, there is a trend of exploring the hidden structure of documents to re-rank results [
Smoothing a data set typically means creating an approximating function that attempts to capture important patterns in the data, while leaving out noise or other fine-scale structures/rapid phenomena. In language modeling, the simplest reason to use smoothing is to not assign a zero probability to unseen words. The accuracy of smoothing is directly related to the retrieval performance.
Given a text sequence (either a query or a document), the probability distribution can be regarded
as a probabilistic language model Md or Mq from each document d or each query q. In other words, it
assumes that there is an underlying language model which ”generates” a term (sequence) [
MLEd w =
g(w 2 d) åw0 g(w0 2 d) (1)
We choose to use three representative methods that are widely used in previous research and relatively
efficient to implement. The Jelinek-Mercer method, the Bayesian smoothing using Dirichlet priors, and the
absolute discounting method. The description of the methods could be found in [
The intuition behind the document re-ranking method is the hidden structural information among the documents: similar documents are likely to have the same hidden information with respect to a query. In other words, if a group of documents are talking about the same topic which shares a strong similarity with a query, in our method they will get allocated similar ranking as they are more likely to be relevant to the query. In addition, the refined ranking scores should be relevant to the initial ranking scores, which, in the experiments conducted in this paper, are combined together with the re-ranking score either using a linear fashion.
The distance between a query and a document in this method adopts the KL divergence between the query terms and document terms and through a linear combination of the re-ranking scores based on initial ranker and the latent document re-ranker.
This method could be found in great detail in [
All of the documents in the experiment were indexed using the Lemur toolkit1. Prior to indexing, Porter’s stemmer and a stopword list2 were used for the English documents. We use a French analyzer3and a German analyzer to analyze French and German documents. The query sets consist of 50 topics, all of which were used in the experiment. Each topic is composed of several parts such as: Title, Description, Narrative. We chose to conduct Title+Description runs as queries. The queries are processed similarly to the treatment in the test collections. The chosen fields used in the indexing and searching are shown in the Table 1. 3.2
In order to investigate the effectiveness of various techniques, we performed a retrieval experiment with several permutations. These experimental runs are denoted as follows:
For monolingual retrieval:
LM-DIR: This part of the experiment involved retrieving documents from the test collection using language modeling with Bayesian smoothing method using Dirichlet prior.
LM-ABS: as above, except that the absolute discounting smoothing method was used. LM-JM: as above, except that the Jelinek-Mercer smoothing method was adopted. 1http://www.lemurproject.org 2ftp://ftp.cs.cornell.edu/pub/smart/ 3http://lucene.apache.org/ For bilingual retrieval:
GOOGLETRANS: In this part of the experiment, documents were retrieved from the test collection using the Google Translator for translating the queries. (It is worth to note that due to the submission restrictions this is an unofficial measurement.)
GOOGLETRANS-LDA: Here we retrieved documents from the document collection using query translations suggested by the Google Translator, then directly re-rank the retrieval results using the translated query with the proposed LDA based document re-ranking method.
GOOGLETRANS-SLDA: Here we retrieved documents from the document collection using query translations suggested by the Google Translator, then we conducted a multilingual corpus with documents written in both query and document languages. Re-ranking was performed by apply the LDA based method on this multilingual space (with the translated and the original query). 4 4.1
In this section we compare three smoothing methods across different languages in the library search (Table 2). As we conducted queries using the title and description fields, they could be considered as long informative queries. Previous research on news and web data suggested that on average, Jelinek-Mercer is better than Dirichlet and absolute discounting in metrics like non-interpolated average precision, precision at 10 and 20 documents while both Jelinek-Mercer and Dirichlet clearly have a better average precision than absolute discounting. The German monolingual runs demonstrated the same observation where JelnekMercer is better than Dirichlet, which is subsequently better than absolute discounting.
English and French runs showed a different behaviour. Absolute discounting is a clear winner of
three smoothing methods while Jelinek-Mercer still perform better than Dirichlet. If explained by the two
different roles in the query likelihood retrieval method [
We now turn to the bilingual tasks to study the LDA-based re-ranking method. The main experimental results are presented in Table 3, for all three languages. The first question we are interested in is how the re-ranking method performs directly over the bilingual retrieval results (taken as a whole). It is shown that our methods bring improvements upon the Google translator baselines in all of the 6 relevant comparisons. Another observation is that in many cases, the method can outperform the baselines for all the evaluation metrics.
With respect to the bilingual re-ranking, the method showed some improvements over the Googe translator and direct re-ranking methods in the X2EN and X2DE runs in terms of mean average precision. The performance is somewhat disappoint in the X2FR runs. Furthermore, although there are some improvements, the difference are not large enough in terms of MAP. However, regarding to the precision at 5 documents the method could demonstrate higher performance than simple re-ranking. This shows that the method is a promising direction but need further investigation.
It is worth mentioning that the combination of methods used in this experiment could achieve very good overall performance as nearly all of our selected monolingual and bilingual runs are among top five participants in CLEF 2009 (except in the French monolingual task) such as: TCDENRUN2 absolute discounting, English monolingual TCDDERUN1 Dirichlet prior, German monolingual TCDDEENRUN3 Google translator with SLDA, German-English bilingual TCDDEFRRUN2 Google translator with LDA, German-French bilingual TCDENDERUN3 Google translator with SLDA, English-German bilingual 5
In this paper we have described our contribution to the CLEF 2009 ad hoc monolingual and bilingual tracks. Our monolingual experiment involved the comparison of three different smoothing strategies applied to language modeling approach for library data retrieval. We also made a first attempt to extend the previously proposed document re-ranking method to cross-language information retrieval. Experimental results demonstrated that smoothing methods tend to behave differently in the library search and across testing languages. They also showed that LDA-based document re-ranking method should be considered further in order to bring significant improvement over the baseline language modeling systems in the crosslanguage setting. 6
This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at University of Dublin, Trinity College.