The main focus of the DCU group's participation in the CLEF 2006 Robust Track in CLEF 2006 was not to identify and handle difficult topics in the topic set per se, but rather to explore a new method of re-ranking a retrieved document set. The initial query is used to re-rank documents retrieved using a query expansion method. The intention is to ensure that the query drift that might occur as a result of the addition of expansion terms chosen from irrelevant documents in pseudo relevance feedback (PRF) is minimised. By re-ranking using the initial query, the relevant set is forced to mimic the initial query more closely while not removing the benefits of PRF. Our results show that although our PRF is consistently effective for this task, the application of our re-ranking method generally has little effect on the ranked output.
Robust cross-language information retrieval, Pseudo relevance feedback, Document reranking This paper describes the DCU experiments for the CLEF 2006 Robust Track. Our official submission included monolingual runs for English and for Spanish, Italian and French where topics and documents had been translated into English and a bilingual run for Spanish using English topics. Unfortunately due to errors in our system we were unable to submit result for monolingual and bilingual German.
Our general approach was to translate non-English documents and topics into English for use as a pivot language. Collections and topics were translated into English using the Systran Version: 3.0 Machine Translator (Sys). Pseudo Relevance Feedback (PRF) which aims to expand query by selecting potential useful terms from the top retrieved documents to improve retrieval has been shown to be effective in our previous submissions to CLEF 2001-2005, and also in our other research work outside of CLEF. Therefore, we again use this method with our extended PRF method of term selection from document summaries rather than full documents that has been thoroughly tested in our past research work. In addition, for this task we explored the application of a new post-retrieval re-ranking method that we are developing.
The remainder of this paper is structured as follows: Section 2 covers background to robust information retrieval tasks, Section 3 describes our system setup and the information retrieval (IR) methods used, Section 4 presents our experimental results and section 5 concludes the paper with a discussion of our findings. The robust track was first introduced in the Text Retrieval Conference (TREC) in 2003. The aim was to explore methods of improving retrieval effectiveness for topics that performed poorly using standard generally high performing IR methods, i.e. hard topics. For these topics it is usually the case that although relevant documents exist in the target collection, the topic is not discriminatory enough to find the relevant documents or bring them into the retrieved set of potentially relevant documents. Several approaches have been taken by TREC participants which aim to tackle these hard topics and improve IR effectiveness. This work falls into two main categories: either using a contemporaneous collection (e.g. the web) for query expansion or re-ordering the original ranking of the retrieved relevant documents.
Kwok. et al. used the web as a contemporaneous collection from which terms were selected for
query expansion [
Amati et al. also found that query expansion from the web resulted in better retrieval for hard
topics as long as the queries are short [
Piatko et al. used a re-ranking method that aimed to improve the initial ranking of retrieved
relevant documents using a method called the minimal matching span [
For our experiments we used the City University research distribution version of the Okapi system retrieval
system. Stopwords were removed from both the documents and search topics, and the Okapi
implementation of Porter stemming algorithm [
cfw (i ) × tf (i, j ) × ( K 1 + 1) K 1 * ((1 − b) + (b × ndl ( j ))) + tf (i, j ) (1) where cw(i,j) represents the weight of term i in document j, cfw(i) is the standard collection frequency weight, tf(i,j) is the document term frequency, and ndl(j) is the normalized document length. ndl(j) is calculated as ndl(j) = dl(j)/avdl where dl(j) is the length of j and avdl is the average document length for all documents. k1 and b are empirically selected tuning constants for a particular collection. k1 is designed to modify the degree of effect of tf(i,j), while constant b modifies the effect of document length. High values of b imply that documents are long because they are verbose, while low values imply that they are long because they are multi-topic. In our experiments values of k1 and b are estimated based on the CLEF 2003 ad hoc retrieval task data.
Short and imprecise queries can affect IR effectiveness. To curtail this negative impact, relevance feedback (RF) via query expansion (QE) is often employed. QE aims to improve initial query statements by addition of terms from user assessed relevant documents. These terms are selected using document statistics and usually describe the information request better. Pseudo-Relevance Feedback (PRF) whereby relevant documents are assumed and used for QE is on average found to give improvement in retrieval performance, although this is usually smaller than that observed for true user-based RF.
PRF can result in a query drift if expansion terms are selected from assumed relevant document
which are in fact not relevant. In our past research work [
rsv(i)= r(i)× rw(i) In our modified version, potential expansion terms are selected from the summaries of the top 5 ranked documents, and ranked using statistics from assuming that the top 20 ranked documents from the initial run are relevant.
As part of our investigation for the CLEF 2006 robust track we explored the application of a further novel re-ordering of the retrieved document list obtained from our PRF process. This reordering method attempts to ensure that retrieved documents with more matching query terms have their ranking improved, while not discarding the effect of document weighting scheme used. To this end we devised a document re-ranking formula as follows: where doc_wgt = the original document matching score
b = an empirical value ranging between 0.1 and 0.5
nmt = the number of original topic terms that occur in the document
mmt = the mean of the value nmt for a given query over all retrieved documents
where r(i) = number of relevant documents containing term i
rw(i) is the standard Robertson/Sparck Jones relevance weight [
(n(i) − r(i) + 0.5 )(R − r(i) + 0.5 ) where n(i) = the total number of documents containing term i r(i) = the total number of relevant documents term i occurs in R = the total number of relevant documents for this query N = the total number of documents
doc_wgt (1 − b) + (b*nmt/mmt) (4) (2) (3) In this section we describe our parameter selection and present our experimental results for the CLEF 2006 Robust track. Results are given for baseline retrieval without feedback, after the application of our PRF method and after the further application of our re-ranking procedure.
The CLEF 2006 topics consist of three fields: Title, Description and Narrative. We conducted experiments used the Title and Description (TD) or Title, Description and Narrative (TDN) fields. For all runs we present the precision at both 10 and 30 documents cutoff (P10 and P30), standard TREC average precision results (AvP), the number of relevant documents retrieved out of the total number of relevant in the collection (RelRet), and the change in number of RelRet compared to Baseline runs.
To set appropriate parameters for our runs development runs were carried out using the training topics provided. The topics provided were taken from the CLEF 2003 The Okapi parameters were set as follows k1=1.2 b=0.75. For all our PRF runs, 5 documents were assumed relevant for term selection and document summaries comprised the best scoring 6 sentences in each case. Where the length of sentence was less than 6, half of the total number of sentences were chosen. The rsv values to rank the potential expansion terms were estimated based on the top 20 ranked assumed relevant documents. The top 20 ranked expansion terms taken from these summaries were added to the original query in each case. Based on results from our previous experiments, the original topic terms are upweighted by a factor of 3.5 relative to terms introduced by PRF.
Baseline – baseline results without PRF using Title, Description and Narrative topic fields (TDN) f20narr – feedback results using the Title, Description and Narrative topic fields. 20 terms are added to the initial query. f20re-ranked - same as F20narr, but documents are re-ranked using the formula (4) above. f20desc – feedback results using the Title and Description sections of query. 20 terms are added to the initial query.
Comparing the Baseline and f20narr runs it can be seen that application of PRF improves all the performance measures for all runs with the exception of the RelRet for Spanish monolingual where there is a small reduction. By contrast for the Spanish bilingual run there is a much larger improvement in RelRet than is observed for any of the other runs.
Application of the re-ranking method to the f20narr list produces little change in the ranked output. The only notable change is a further improvement in the RelRet for the Spanish bilingual task. Varying the value of the b factor in equation 4 made only a small difference to the results. We are currently investigating the reasons for this results, and exploring approaches to the re-ranking method which will have a greater impact on the output ranked lists.
This paper has presented a summary of our results for the CLEF 2006 Robust Track. The results show that our summary-based PRF method is consistently effective across this topic set. We also explored the use of a novel post-retrieval re-ranking method. Application of this procedure led to very modification in the ranked lists, and we are currently exploring alternative variations on this method.
English French Spanish Italian Spanish bi f20narr (TDN) P10 P30 Av.P RelRet Chg RelRet f20re-ranked (TDN) P10 P30 Av.P RelRet Chg RelRet f20desc (TD) P10 P30 Avep RelRet Chg RelRet