This paper describes the participation of the REINA research group at CLEF 2009 Robust-WSD Task. We have participated in both monolingual and bilingual subtasks. In past editions of the robust task our research group obtained very good results for non-WSD experiments applying local query expansion using co-occurrence based thesauri constructed using windows of terms. We applied it again. For WSD experiments, our intention was to use the WSD information and WordNet for expansion, but we did not have time to use them. We only used the lemma proposed by the POS tagger of the WSD collection as a stemmer. In bilingual retrieval experiments, two on-line machine translation programs were used to translate topics, and translations were merged before performing a monolingual retrieval. We also applied the same local expansion technique. Our non-WSD runs obtained the top rank considering the GMAP measure (monolingual and bilingual subtasks). However, regarding expansion we viewed that the settings tuned for a system not always produces retrieval improvement when the conditions change: number of query terms, query language, document or query subject, linguistic approach, etc. Our WSD runs also obtained very good positions in the rankings even thought we only used partial WSD information. However, in comparison with non-WSD runs the retrieval performance made worse. We detected some homonym errors in the POS tagger and probably these errors are worse than the errors carried out by the Porter stemmer used in the non-WSD experiments.
http://reina.usal.es
Robust retrieval tries to obtain stable performance over all topics by focusing on poorly performing topics. Word sense disambiguation (WSD) is the identification process of sense of a word used in a given sentence. The goal of the CLEF 2009 robust task was to test whether WSD can be used beneficially for retrieval systems. For this, the organizers provided document collections annotated with WSD from previous CLEF campaigns. Our research group has participated in the monolingual (English) and the bilingual (Spanish to English) subtasks with non-WSD and WSD experiments. For non-WSD ones, this year we used the same approach of our group at past CLEF robust tasks: an IR system based on the vector space model and applying a query expansion technique that uses co-occurrence based thesauri built with windows of terms. For WSD experiments, our primary approach was to use the WSD information and WordNet for retrieval, but we had no time to use them. Nevertheless, we have used the information of the part-of-speech (POS) tagger as a stemmer, instead of the Porter stemmer for English used in our non-WSD experiments.
Our main focus was monolingual retrieval. The steps followed are explained below. For bilingual retrieval experiments we used machine translation (MT) programs to translate topics into document language, and then a monolingual retrieval was implemented. 2
At past CLEF robust campaigns our non-WSD runs got very good rankings [
The last step was to apply local query expansion using windows of terms. This technique uses co-occurrence relations in windows of terms from the first retrieved documents to build a thesaurus to expand the original query. Terms close to query terms must have higher value of relation than other terms in the document. In this case, it is important to define the distance value between two terms. If distance is zero, both terms are adjacent. If distance is one, then there exists one term between the two terms, and so on. To compute the distance, stop words are removed and sentence or paragraph limits are not taken into account.
To expand the original query, terms with a high co-occurrence value with all terms of the query
must be selected. We use the measurement of scalar product with all query terms to obtain the
terms with highest potential to be added to the original query. A description of this procedure
can be found in [
Taking into account that the geometric average (GMAP), rather than the mean of the average precision (MAP), turned out to be the most stable evaluation method for robustness, several tests were carried out to obtain the best performance using the training topics. The highest improvement achieved with this expansion technique was by using a distance value of 2, taking the first 5 retrieved documents, and adding 10 terms to the original query. For the rest of the experiments we used the same settings when this expansion was applied. Runs without expansion were also submitted at CLEF task.
For the bilingual experiments, the CLIR system was the same as that used in monolingual retrieval. A previous step was carried out before searching, to translate Spanish topics into English. We used two on-line machine translation (MT) programs: Systran1 and Reverso2. For each topic 1http://www.systransoft.com 2http://www.reverso.net We had no time to use all the WSD information and WordNet for retrieval. Nevertheless, we used a piece of the WSD information as a stemmer. The POS tagger proposes a “lema” for each word it analyzes. We use this lemma for indexing instead of the stem returned by the Porter stemmer used in the non-WSD experiments. We also applied the same expansion technique with the same settings used in non-WSD experiments. 4
For monolingual retrieval we used a simple document retrieval system based on the vector space model and we applied a local query expansion technique as a basis of our runs. The use of query expansion can be used to improve retrieval, in fact this is the approach used at TREC and CLEF robust tasks, but we have verified that the settings for a system not always produces retrieval improvement when the conditions change (number of query terms, language, document or query subject, linguistic approach, etc.). The problem is that poorly performing topics behaved differently when changing the retrieval conditions. We think that regarding robustness the objective must be to make good information retrieval systems, rather than to tune some query expansion techniques.
For the bilingual retrieval, the use of on-line MT programs to translate topics is a good approach for CLIR. Collecting terms from some translations of a topic is a technique that also improves the systems performance.
For the WSD experiments we only made a partial use of the WSD information of the topics and documents collection, in spite of this our runs obtained a good positions in the subtasks. In all cases our non-WSD experiments obtained better results than WSD experiments. We think that the reason is the information we used form the POS tagger, owing on the fact that we detected some errors in it, primarily homonym errors, both in Spanish and in English. In Spanish this kind of error is sometimes introduced by the elimination of accent sings in the process. For example, in the Spanish topic “Pesticidas en alimentos para beb´es” (Pesticides in baby food), the word “beb´es” (babies, a noun) was tagged as verb, and the lemma proposed was “beber ” (to drink). The explication is here: in Spanish the word “bebes” (note the missing accent) is the second-person form of the present tense of the verb “beber ” (tu´ bebes agua, you drink water.)
We think that errors based on homographs are the most important ones to deteriorate the retrieval performance. It is very important that the POS tagger works fine, otherwise any process that depends on it will increase the error. Probably these errors are worse than the errors carried out by the Porter stemmer used in the non-WSD experiments.