For UFRGS's participation on CLEF's Robust task, our aim was to compare retrieval of plain documents to retrieval using information on word senses. The experimental run which used word-sense disambiguation (WSD) consisted in indexing the synset codes of the senses which had scores higher than a predefined threshold. The documents in both baseline and WSD runs were indexed by Zettair. The metric for comparing queries and documents was OkapiBM25. The results of the experiments show that only 47 topics were helped by the strategy, while 103 had their performances worsened. A statistical t-test has shown that the experimental run which did not use WSD information significantly outperformed the one which did. A deeper analysis of our results and a set of further experiments are now under preparation.
This paper reports on experiments submitted to CLEF 2009 Robust track. The aim of the task is to assess the validity of using word-sense disambiguated data for Information Retrieval. 2.1
We worked on the English news collections composed by LA Times 94 and Glasgow Herald 95. There are 169,477 documents in total. Three versions of the collection were available: a “plain” version, and two versions with word-sense disambiguation (WSD) data.
Using the WSD documents (UBC version), we created a document collection composed by the synset codes of all WordNet senses which exceeded an arbitrary threshold (set to 0.30). WordNet is an lexical base, in which nouns, verbs, adjectives and adverbs are grouped in sets called “synsets”. Figure 1 shows an example of an input word found in a document and the result of the processing that extracts the synset codes. If a term did not have a synset code, or a sense scoring higher than the threshold, we kept the original word form (i.e. the contents of the <WF> tag. <TERM ID="C041-27" LEMA="report" POS="VBP"> <WF>report</WF> <SYNSET SCORE="0.393362015980332" CODE="00655029-v"/> <SYNSET SCORE="0" CODE="00653609-v"/> <SYNSET SCORE="0" CODE="00653917-v"/> <SYNSET SCORE="0" CODE="00655324-v"/> <SYNSET SCORE="0.606637984019668" CODE="00653371-v"/> <SYNSET SCORE="0" CODE="00653772-v"/> </TERM>
The IR system we used was Zettair (Zettair), which is a compact and fast search engine developed by RMIT University (Australia) distributed under a BSD-style license. Zettair implements a series of IR metrics for comparing queries and documents. We used Okapi BM25 as some preliminary tests we performed on other data collections showed it achieved the best results.
We have submitted one baseline runs indexing the plain collection and one run using the WSD-annotated documents. There was a bug in the code that generated our WSD run, so we also report on a third (unofficial) run (WSD2) which has the correct data. The details of the runs are shown in Table 1.
The table shows that the number of the total terms in the WSD run was smaller than in the baseline run. However, the opposite has happened with the number of distinct terms. The average number of terms per documents was higher on the WSD run as in many cases, more than one sense was kept for a term. 2.2
Our results are summarised in Table 2 and Figure 3. The baseline run clearly outperformed the WSD run. A ttest using the average precision of the 160 queries has yielded a p-value of 0.0045, showing that the baseline was significantly better than the WSD run. The Recall-Precision curves on Figure 3 also show that the baseline was better in all recall levels. The superiority of the baseline is also reflected on the number of relevant documents retrieved and on precision at different cut-off points.
MAP 0.3160 0.2547
A topic-by-topic analysis has shown that ten queries had the same average precision in both runs, 47 improved with WSD information, and 103 were better in the baseline run. Table 3 shows the top ten topics which were helped by the addition of WSD information and Table 4 shows the ten topics that were most harmed. A more detailed topic-by-topic analysis will be performed so that we can identify common patterns in the topics which had their performances improved and the ones which had their results worsened by the addition of WSD information. The experiments reported here were a starting point and we plan to investigate some aspects further. First, we only worked with the UBC data. It would be interesting also to do experiments with the NUS collection to enable some comparisons.
We arbitrarily chose a threshold of 0.30 for the synset codes to be maintained. The idea is to try different thresholds and assess how they impact the results.
We also plan to investigate different strategies for query expansion using synonyms and related terms extracted from WordNet. 4
This paper described the experiments performed by our group for CLEF 2009 Ad hoc Robust task. We compared an experimental run in which we indexed the plain documents with an experimental run in which we took WSD information into consideration. The results have shown that the baseline (plain) run has outperformed the WSD run.
We plan to do further experiments as there are many issues which are worthy of a more detailed investigation.
This work was partially supported by CNPq. WordNet. Retrieved 01/03/09, 2009, from http://wordnet.princeton.edu/ Zettair. Retrieved 11/06/07, 2007, from http://www.seg.rmit.edu.au/zettair/