We present a report on our participation in the mixed monolingual web task of the 2006 Cross-Language Evaluation Forum (CLEF). We compared the result of web page retrieval based on the page content, page title, and anchor page. The retrieval effectiveness for the combination of page content, page title, and anchor texts was better than that of the combination of page title and page title only. Applying the pseudo-relevance feedback improved the retrieval performance of the queries.
2.1
The web collection contains over two million documents from the EUROGOV collection. In these experiments, we used Lemur information retrieval system to index and retrieve the documents. Lemur is built based on the language model [3]. We index the webpages according to their content pages, title pages, and anchors. Stopwords were removed from the collection, but word stemming was not applied to the collection. 4
We were very surprised to see the results of our participation this year. All of the results that we submitted are very low compared to our last year’s result. In 2005, we indexed the collection using a different information retrieval system, i.e., Lucene2 which is built based on the vector similarity model [1, 4]. The first result is shown in Table 1. In the retrieval, we compute the total relevance score by summing up the relevance scores based on page content, page title, and anchor texts found on the webpages.
The third technique applies the pseudo-relevance feedback to the retrieval that uses the combined score of page content, page title, and anchor texts. As shown in Table 3, the feedback reduced the performance of the queries where the MRR dropped to 0.0253. The pseudo-relevance feedback was done using the top-5 relevant documents retrieved.
Finally, the last result was obtained by applying the pseudo-relevance feedback to the combined relevance score of page content and page title only. As shown in Table 4, we obtained the highest retrieval performance with MRR of 0.0918. To investigate the cause of our poor retrieval performance, we conducted some further experiments. We used the queries from last year’s task and ran them on the same index that was built using Lemur. The result is as shown in Table 5, which is much better than for this year’s queries. However, we found a sign of indexing error, i.e., there were some domains that Lemur was unable to index. This resulted in Lemur’s not being able to retrieve any documents for a number of queries. We also suspected that the index for documents in languages containing non-latin characters was corrupt, as indicated by the fact that documents in some domains such as Russian and Greek were never retrieved. 5
Our results demonstrate that combining the page content, the page title, and anchor texts resulted in a better mean reciprocal rank (MRR) compared to searching using the page content and page title only. The pseudorelevance feedback that we employed increased the retrieval performance of the queries. However, we had some problems with indexing the collection, which resulted in our poor retrieval performance in our participation this year. We hope to improve our results in the future by exploring still other methods.