This paper reports the participation of the XLDB Group from the University of Lisbon in the CLEF 2006 ad-hoc monolingual and bilingual subtasks for Portuguese. We present our IR system and detail both the query expansion strategy and the weighting scheme. In the end, we describe our runs and analyse our performance.
This paper describes the third participation of the XLDB Group from the University of Lisbon in the CLEF
ad-hoc Portuguese monolingual task and English to Portuguese bilingual task. Our main goal is to obtain
a stable platform to test GIR approaches for the GeoCLEF task [
In 2004 we participated with an IR system made from components of tumba!, our web search
engine [
In 2005 we developed QuerCol, a query generator module with query expansion, and we implemented
a tf idf term weighting scheme with a result set merging module for our IR system [
The rest of this paper is organised as follows: in Section 2 we describe our IR system and detail the improvements made to QuerCol and SIDRA, our indexing, retrieval and ranking module. In Section 3 we present our evaluation goals and submitted runs. Section 4 shows the results obtained, and Section 5 summarises our conclusions and refers future work.
QuerCol (Query Collator) was first developed in 2005 for our ad-hoc participation. Quercol processed
the CLEF topics for our IR system, combining the topic terms with their morphological derivatives to
produce boolean query strings in Disjunctive Normal Format, that is, with disjunctive groups of conjunctive
terms [
.
For each topic, the non-stopword terms from the title are extracted and combined into a boolean query with AND operators. The initial query is submitted to SIDRA, generating an initial run. Note that, in our automatic runs, we did not used the description and the narrative fields.
We used the wt (pt -qt ) algorithm to weight the terms for our query expansion algorithm, as represented in
the following equation [
R + r + 0:5) r + 0:5) r R n r N R where N is the total number of documents, n is the number of documents containing the term t, R is the number of relevant documents, r is the number of relevant documents containing the term t. In the end, pt represents the probability of the term t to occur in a relevant document, while qt represents the probability of the term t to occur in a non-relevant document.
QuerCol assumes that all documents above a certain threshold parameter, the top-k documents, are relevant for a given topic, and that the remaining documents are non-relevant. The top-k documents are then tokenised and their terms are weighted, generating a ranked term list that best represent the top-k documents, and thus the information need of the topic.
Finally, QuerCol generates a boolean query string in the Disjunctive Normal Format, using terms from the ranked term list and from the topic title. The terms are combined using one of the two following query construction approaches, the Logical combination and the Relaxed combination.
The Logical combination assumes that all non-stopwords from the topic title represents different
concepts, and that these concepts must be mentioned in the retrieved documents (see Figure 3). The queries
generated by the Logical combination reproduces the boolean constraints described by Mitra et al [
As each concept may be represented by many terms, the Logical combination approach searches the
ranked term list to find related terms for each concept. When found, the terms are moved into the
corresponding concept’s bag of terms, the concept bags. QuerCol relates the term to a concept if they share the
same stem, as given by Porter’s stemming algorithm for Portuguese [
The query generation step produces all possible combinations (nCm) of the m n matrix of m bags n terms in each bag, resulting in m n partial queries containing one term from each concept bag. The terms of the partial queries are joined by the AND operator, with the exception of the terms from the expanded terms bag, that are joined by the OR operator. We made this exception for the expanded terms to avoid query drift. In the Disjunctive Normal Format, the final query string is the following:
Nonetheless, Mitra et al pointed that there are relevant documents that may not mention all concepts
from the topic title [
Weighting and Ranking The SIDRA retrieval and ranking module implemented the BM25 weighting scheme, where the weight for each term is computed as follows:
BM25(ti) = k1 (k1 + 1) ((1 k2) + k2 term_ f req(ti)
doc_len avg_doc_len ) + d log(
N
doc_ f req(ti) + 0:5 doc_ f req(ti) + 0:5 ) The parameters were set to the standard values of k1=2.0 and k2=0.75.
Robertson et al. have recently proposed an extension of the BM25 scheme for structured documents,
suggesting that document elements such as the title could be repeated in a corresponding unstructured
document, so that the title terms can be weighted more important [
For CLEF, we assumed that the first three sentences of each document should be weighted as more important, as the first sentences of news articles often contains a summary of the content. Robertson’s extension was applied in run PT4, giving a weight of 3 to the first sentence, and a weight of 2 to the following two sentences.
SIDRA’s ranking module was also improved to support disjunctive queries more efficiently, so we
discarded the result merging set module that we used in CLEF 2005 [
We submitted 4 runs for the Portuguese ad-hoc monolingual subtask and 4 runs for the English to Portuguese ad-hoc bilingual subtask. The Portuguese monolingual runs evaluated both QuerCol query construction strategies and the BM25 term weigh extension, while the English runs evaluated different values for the top ranked document threshold (top-k documents), and for the size of the expanded terms bag (top-k terms). Table 1 resumes the configuration of the submitted runs.
The PT1 run was created manually from topic terms, their synonyms and morphological expansions. For our automatic runs, we used the CLEF 2005 topics and qrels to find the best top-k term values for a fixed value of 20 top-k documents. The Logical combination obtained a maximum MAP value of 0.2099 for 8 top-k terms. The Relaxed combination did not performed well for low top-k term values, but for higher values it outperformed the Logical combination with a maximum MAP value of 0.2520 for 32 top-k terms.
For the English to Portuguese bilingual subtask, we translated the topics with Babelfish (http:// babelfish.altavista.com), and used different values for top-k terms and top-k documents, to evaluate if they significantly affect the results. 4
Table 2 presents our results. For the Portuguese monolingual subtask, we observe that our best result was obtained by the manual run, but the automatic runs achieved a performance comparable to the manual run. The Relaxed combination produced better results than the Logical combination, generating our best automatic runs. The BM25 extension implemented in the PT4 run did not produce significant improvements.
For the English to Portuguese bilingual task, we observe that the different top-k terms and top-k document values do not affect significantly the performance of our IR system. The PT3 and EN4 runs were generated with the same configuration, to compare our performance in both subtasks. The monolingual run obtained the best result, with a difference of 32% in the MAP value to the corresponding bilingual run. For our participation on the CLEF 2006 adhoc task, we implemented well-known algorithms in our IR system to obtain good results on the ad-hoc task, allowing us to stay focused on GIR approaches for the GeoCLEF task.
Our results show that we improved our monolingual IR performance in precision and in recall. The best run was generated from a query built with a Relaxed combination, with an overall recall value of 84.2%. We can not tell at this time what are the contributions of each module to the achieved improvements of the results.
The English to Portuguese bilingual results show that the topic translation was poor, resulting in a decrease of 0.111 in the MAP values for runs PT3 and EN4. The difference between there two runs show that we need to implement a specific translation module to improve our bilingual results.
We also observe that the top-k terms and top-k document parameters did not affect significantly the performance of the IR system.
Next year, our efforts should focus on improving the query expansion and query construction
algorithms. QuerCol can profit from the usage of the description and narrative fields, producing better query
strings. Also, we can follow Mitra et al. suggestion and rerank the documents before the relevance
feedback, to ensure that the relevant documents are included in the top-k document set [
We would like to thank Daniel Gomes, who built the tumba! repository used on our participation, and to all developers of the tumba! search engine. Thanks also to Alberto Simões for providing the topic translations. Our participation was partly financed by the Portuguese Fundação para a Ciência e Tecnologia through grants POSI / PLP / 43931 / 2001 (Linguateca) and POSI / SRI / 40193 / 2001 (GREASE).