This task was meant to compare the results of two different retrieval techniques: the first one was based on the words found in documents and query texts; the second one was based on the senses (concepts) obtained by disambiguating the words in documents and queries. The underlying goal was to come up with a more precise knowledge about the possible improvements brought by word sense disambiguation (WSD) in the information retrieval process. The proposed task structure was interesting in that it drew up a clear separation between the actors (humans or computers): those who provide the corpus, those who disambiguate it, and those who query it. Thus it was possible to test the universality and the interoperability of the methods and algorithms involved.
To index the corpus, we chose the IDX-VLI indexer described in [gfb06] because it can gather a wealth of information (positions, etc.), it has built-in operators and it is remarkably fast. Still, we only used the basic version of that indexer, i.e. we did not use any relevance feedback mechanism, context description or any other sophisticated tool of that sort. We thus avoided interfering with the direct results of the experiment and we facilitated the result analysis.
During the indexing process the strategies were applied to all the terms, including numbers, except for the stopwords. Given the poor performance of the POS approach, we quickly gave up this option.
The same translators were applied to the queries, with an extended stop-word list including words such as report, find, etc. For each topic we derived three queries: • • •
T : Include only the title part TD : Include the title and description translated terms
TDN : Include the title, description and narrative translated terms.
In order to come up with a reasonably good base line we tested several approaches to build a Boolean pre-filter from a given topic (results are the mean average precision (MAP) on T): • • • • • • • • • •
OR (25.5%) : The logical OR of the terms (or lemmas) AND (15.8%) : The logical AND of the terms NEAR (15.2%) : The logical OR of all the pairs (ti NEAR tj) where ti and tj are the query terms AND-1 (23.6%): The logical OR of all the possible conjunctions of terms, except for the conjunction of all the terms.
The best results in terms of MAP were produced by the OR filtering, followed by the computation of a relevance score based on the Okapi BM25 weighting model (with default parameters). The test was carried out on the titles (T) of 150 training topics. More restrictive filtering schemes were tried out but did not perform any better, probably because of the relatively small size of the corpus.
Runs with word senses: For the disambiguation-based runs we tried out several other filtering schemes, including:
OR (22.4%) : The logical OR of the best synset corresponding to a topic term AND (15.1%) : The logical AND of the best synset corresponding to a topic term NEAR (12.5%) : The logical OR of all the pairs (si NEAR sj) where si and sj are the best synsets corresponding to a topic term ti and tj AND-1 (18.8%): The logical OR of all the possible conjunctions of synsets, except for the conjunction of all the synsets HYPER (14.3%): The logical AND of each (si OR hi) where si is the best synset corresponding to a topic term ti and hi is the direct hypernym of si in WordNet ORHYPER(18.43%): The logical OR of each (si OR hi) where si is the best synset corresponding to a topic term ti and hi is the direct hypernym of si in WordNet.
However, none of these strategies performed any better than the basic OR filter on terms. 1 This amounts to considering that the disambiguation algorithm is "perfect". Alternatively we could have added all the synsets with a score greater than a given threshold.
The first table below shows the mean average precision (in percent) calculated on the training queries with different query processing options (disambiguation algorithm and part-of-topic selection) and different document processing options (disambiguation and translation). Of course, the <TERM> processing (NAT, LEM, WSD or WSDL) was always the same throughout the queries and the corpus for a given run.
The base line was the run with topic selection TDN and term selection LEM (i.e. the whole topic with stemming).
The second table shows the results of the testing queries, which are slightly better than those of the training queries (maybe the testing queries were somewhat easier).
The base line (LEM) for the Spanish queries was created by automatically translating the queries from Spanish into English.
The tests on the NUS corpus produced better results than those on the UBC one. Therefore most of the runs were performed on the NUS corpus, while the UBC corpus would be used to test the interoperability of the disambiguation processes. avg precision on TRAINING requests (OR strategy)
NAT 25.2% LEM 27.0% 31.9%
NUS WSD avg precision on TESTING requests (OR strategy)
WSD
UBC
Using the D and N parts-of-topics increases the precision in all cases (with and without WSD). This is probably due to the ranking method which benefits from the additional terms provided by D and N. On the test run with UBC disambiguation, the senses alone (WSD) decrease the MAP: -4.6% on T queries and -3.1% on TDN. On training requests, adding the lemmas to the senses (WSDL) slightly improves the MAP (+0.6%). This is the only case where disambiguation brings any improvement. Using different disambiguation algorithms for the queries and the documents noticeably decreases the results. This should not happen if the algorithms were perfect. It shows that disambiguation acts as a kind of encoding process on the words, and obviously the best results are obtained when the same encoding, producing the same mistakes, is applied to both queries and documents. Thus, at this stage, the disambiguation algorithms are not interoperable.
We carefully analyzed about 50 queries to better understand what happened with the disambiguation process. For instance, the query with the title "El Niño and the weather" was disambiguated as follows (NUS): • • •
El was understood as the abbreviation el. of elevation Niño was understood as the abbreviation Ni of nickel, probably because the parser failed on the nonASCII character ñ weather was correctly understood as the weather concept.
Although the disambiguation was incorrect, WSD was as good as LEM because the "encoding" was the same in the collection and in the query and there were few or no documents about nickel that could have brought up noise.
More generally, when the WSD results were better than the LEM ones, it was not due to semantic processing but to contingencies. For instance, the query title "Teenage Suicides" had a better score with WSD because teenage was not recognized! Thus the query became suicides, which is narrower than teenage OR suicide and, on this corpus, avoids retrieving a large amount of irrelevant documents about teenagers.
A few items of the test run are commented in Appendix A.
The poor performance on Spanish queries is due to 1) the above-mentioned lack of interoperability between the different WSD algorithms, and 2) the low quality of the Spanish WSD itself.
This can be illustrated with some examples: On Question 41: "Pesticide in baby food" is translated by "Pesticidas en alimentos para bebes" and is then converted into the FOOD and DRINK (verb) concepts because bebes is a conjugated form of beber, which is the Spanish verb for drink.
On Question 43: "El Niño and the weather" is translated by "El Niño y el tiempo" and is then converted into the CHILD and TIME concepts because Niño is the Spanish noun for child and tiempo is an ambiguous word meaning both time and weather.
Given those difficulties, outstanding results could not be expected.
Looking back on the questions and results, it can be noted that 1'793 documents were retrieved out of the 2'052 relevant ones, i.e. almost 90% of them. The core issue is to sort out the documents so as to reject those whose content does not match users' expectations.
A closer look at our results on the Training corpus showed that we got a pretty good performance on some of the requests. This does not mean that our search engine understood the said requests correctly; it is simply due to the fact that the corpus included only good matches for those requests, so it was almost impossible to find wrong answers.
For instance, on Question 50 about "the Revolt in Chiapas", we retrieved 106 documents out of the 107 relevant ones with an average precision of 87%. This is due to the fact that in the corpus, the Chiapas are only known for their revolt (in fact if we google the word "Chiapas" a good proportion of the results are currently about the Chiapas rebellion).
On the other hand, on Question 59: "Computer Viruses", our search engine retrieved 1 document on 1 with an average precision of 0.3%. This is because the 300 documents retrieved before the one we were looking for were indeed about viruses and computers, but did not mention any virus name or damage as was requested. Therefore term disambiguation does not help the search engine to understand what kind of documents are expected. A question such as the one above requires the text to be read and understood in order to decide whether it is actually a correct match.
Intuitively, Word Sense Disambiguation should improve the quality of information retrieval systems. However, as already observed in previous experiments, this is only true in some specific situations, for instance when the disambiguation process is almost perfect, or in limited domains. The observations presented here seem to support this statement. We propose two types of explanations:
When a query is large enough (more than one or two words), the probability that a document containing these words uses them with a meaning different from the intended one is very low. For instance, it is unlikely that a document containing mouse, cheese and cat is in fact about a computer mouse. This probably makes WSD useless in many situations. Such a request is similar in nature to the narrativebased tests. On the other hand, the WSD approach could make more sense when requests include only one or two words (which is the most frequent case in standard searches).
WSD is a very partial semantic analysis which is insufficient to really understand the queries. For instance, consider the query "Computer Viruses" whose narrative is "Relevant documents should mention the name of the computer virus, and possibly the damage it does". To find relevant documents, a system must recognize phrases which contain virus names ("the XX virus", "the virus named XX", "the virus known as XX", etc.). It should also recognize phrases describing damages ("XX erases the hard disk", "XX causes system crashes" but not "XX propagates through mail messages"). These tasks are very difficult to perform and they are far beyond the scope of WSD. Moreover, they require specific domain knowledge, as shown in [rf06] .
The modifications brought to our stop-word lists showed that our search engine is more sensitive to various adjustments of its internal parameters than to the use of a WSD system. Indeed, when we ran a new series of tests with English-only stop words (which eliminated some terms in the requests, such as "eu" and "un"), our new score for the LEM-TDN (which was our best result in this task) increased from 39.17% to 39.63%. Finally, as we argued in [grf05], conceptual indexing is a promising approach for language-independent indexing and retrieval systems. Although an efficient WSD is essential to create good conceptual indexes, we showed in [grf05] that ambiguous indexes (with several concepts for some terms) are often sufficient to reach a good multilingual retrieval performance, for the reasons mentioned above. [gfb06] Guyot, J., Falquet, G., Benzineb, K. (2006) Construire un moteur d'indexation. Technique et science informatique (TSI), Hermes, Paris. [grf05] Guyot, J., Radhouani, S., Falquet, G. (2005) Conceptual Indexing for Multilingual Information Retrieval. In Accessing Multilingual Information Repositories: 6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, Revised Selected Papers. C. Peters, et al. (Eds.). Lecture Notes in Computer Science, Vol. 4022, Springer. [nus07] Chan, Y. S., Hwee T., Zhong, Z. (2007) NUS-PT: Exploiting Parallel Texts for Word Sense Disambiguation in the English All-Words Tasks. Proc. of the 4th International Workshop on Semantic Evaluations (SemEval 2007). Prague, Czech Republic. pp 253--256. [rf06] Radhouani, S., Falquet, G. (2006) Using External Knowledge to Solve Multi-Dimensional Queries, in Proc. 13th Intl Conf. on Concurrent Engineering Research and Applications (CE 2006), Antibes, Sept. 2006. IOS Press. [ubc07] Agirre, E., Lopez de Lacalle, O. (2007) UBC-ALM: Decombining k-NN with SVD for WSD. Proc. of the 4th International Workshop on Semantic Evaluations (SemEval 2007). Prague, Czech Republic. pp 341--345.
Query 178 Query 178
06417803-n China WSD06404073-n
China-Taiwan OR relation (WSD06417803-n OR WSD00018916-n)