In this paper we calculate and analyse the lexical ambiguity of queries in a crosslingual Image Retrieval (Flickling) and compare it with the results obtained by users. We want to know to what extent the lexical ambiguity of a query in uences the correct localization of an image in a multilingual framework. With this, our nal objective is to determine the necessity of Word Sense Disambiguation systems in Information Retrieval tasks, according to user behaviour. The results show that users really deal with lexical ambiguity: in general terms, users nd the correct image with low ambiguity queries.
In this paper1 we present a preliminary study about how users deal with lexical ambiguity during the interaction with an Information Retrieval system. Taking advantage of Flickling system2, we have analysed user behaviour with lexical ambiguity in a multilingual image retrieval task.
The Natural Language Processing community has not reached an agreement as to whether Word Sense Disambiguation (WSD) systems are useful or not in Information Retrieval (IR) tasks. In other words, there is not an agreement about IR systems improving the retrieval process with a WSD system.
1This paper has been supported by the Spanish Government, project TEXT-MESS TIN-2006-15265-C06-01. Elena Lloret is funded by the FPI grant (BES-2007-16268) from the Spanish Ministry of Science and Innovation, under this project. Marcel Puchol-Blasco is funded by the research grant BFPI06/182 from the Valencian Regional Government (Generalitat Valenciana).
2http://cabrillo.lsi.uned.es/ ickling
In general terms, on the one hand, papers like [
There are di erent approaches to WSD, with di erent disambiguation algorithms and lexical
resources, and di erent problems [
In our opinion, the key question is to know if lexical ambiguity is really a decisive factor in IR tasks for a user: if it is, WSD systems are necessary in IR. If not, they are unnecessary.
Following Robins [
In the next section, we introduce the mathematical formula created to calculate the lexical ambiguity of a query. Then we show some data extracted from the iclef log: the lexical ambiguity and the image retrieved by the system, the lexical ambiguity and the images correctly found by the users, and the lexical ambiguity of a speci c user. Finally we extract some preliminary results and future work. 2
In order to achieve our objective, we need to know the general ambiguity of each query, and then compare it with the success of the query. That is, according to the general ambiguity of a query, we want to know if the image has been located or not, and also if both aspects are related.
In this sense, we must represent each query with a value that shows the general ambiguity of this query.
On the one hand, the ambiguity of each word is the number of senses that it has in WordNet
[
= (1
Q1s ) 1 n where represents the general ambiguity of the query, s the number of senses for each word in WordNet or EuroWordNet, and n the number of words in the query.
Next, we are going to split the formula into di erent parts, in order to explain each in more detail. The ambiguity of one word is uniquely related to the number of associated senses to it in the WordNet or EuroWordNet lexical resource, but the ambiguity of a query is not the sum of the word senses. The user introduces a word in a query with a speci c sense. Therefore, we take into account the probability of each sense for each word. The formula 1s calculates the ambiguity of the word: the probability that the user is using only one sense for a word. The scores obtained by this formula are in the range between 0 (upper ambiguity) and 1 (lower ambiguity).
Moreover, considering that a query is composed of a set of n words, the ambiguity of a query will be associated to the ambiguity of its n related words. The formula that computes this ambiguity is de ned such as s11 s12 ::: s1n that simpli ed is de ned as such Qin=11 si . Finally, with the aim of considering the opposite situation (values near zero indicate lower ambiguity while values near one indicate upper ambiguity), we introduce the opposite element `1' in the formula being 1 Qin=11 si the nal formula that computes the ambiguity of each query.
The second part of the formula ( 1 ) is the reciprocal of the number of words in the query.
n The number of words in a query has a speci c weight during the retrieval process and in the general ambiguity of the query too. The words that make up a query are the context in which each ambiguous word is disambiguated. That is, for a user, each word has a speci c sense in this speci c context (the rest of words in the query). Therefore, the number of words in the context in uences the ambiguity of the query.
From a semantic point of view, the fact that di erent languages are used in the same query is a method of lexical disambiguation. In this sense, users employ translation to deal with the semantic ambiguity. The same word in di erent languages could have a di erent amount of senses (for example, the Spanish word \jugar" -5 senses- and the English translation \to play" -29 senses-). In these cases, we consider this set of words (that is, a word and its translation to another language) as the same word. Considering that users use translation to reduce the ambiguity, we can conclude that the ambiguity of these complex words are the intersection of their senses, that is, we consider the language with less senses for the same word. In the previous example, we considered that the ambiguity of \jugar - to play" is the amount of senses of only \jugar": 5 senses.
There is a general problem with words that do not appear in WordNet or EuroWordNet. In these cases, we assume that that word is either a proper noun or a technical term. In both cases, we consider that these words do not have ambiguity, and assume only one sense for each3.
In a nutshell, from our point of view, a query with only one word and ve senses has more ambiguity that a query with ve words each with one sense. In each of these queries, the number of senses is the same, but not the number of words. This formula tries to represent this fact. 3
For this paper we take into account only three languages: English, Spanish and Italian4. We have calculated the general lexical ambiguity of each query with the previous formula, and we have extracted some data related to the success of these queries. Speci cally, we have extracted data about these three aspects: lexical ambiguity of the queries and images retrieved by the system, lexical ambiguity of the queries and images found by the users, lexical ambiguity of the queries and images found by a speci c user.
3.1
The average number of images returned for a speci c ambiguity in a question has been extracted. Graph 3.1 shows the average of images retrieved by the system according to the level of lexical ambiguity within the queries.
3Another possibility is that a word does not appear in the lexicon due to a typographic error and spelling mistake by the user. This case is not taken into account. However, as we will show later, this assumption has introduced some errors in the results.
4Portuguese and Dutch have not been taken into account. The ambiguity associated with words of these languages will be the minimum number of senses where we have translation. Queries with no results are not relevant for our experiments because the words contained in the queries usually are words with spelling mistakes by the users.
These data show that the amount of image retrieved by the system increases according to the level of the queries' lexical ambiguity. It is meaningful the increment of images retrieved from an ambiguity level of 0.5 to 0.9. With these data, we can say that the system has more precision with low ambiguity queries.
However, it is curious the amount of retrieved images with non-ambiguous queries (0). It is higher than the images retrieved with ambiguity 0.1 or 0.2. This is an exception to the general behaviour of data.
The reason for this exception is the words that do not appear in the lexicon. As we said before, the words that do not appear in the lexicon are considered non-ambiguous words (proper nouns or technical terms). However, it has been a hard decision due to, on one hand, proper nouns and technical terms possibly being ambiguous; and, on the other hand, there are ambiguous words that simply do not appear in the lexicon and words written in other languages. In all these cases, we have considered these words as non-ambiguous words, and, in many times, it is not true.
Images 500 400 300 200 100 0 0 0.1 0.2 0.3 0.4 0.5 Ambiguity 0.6 0.7 0.8 0.9
This is the reason why the amount of queries with no ambiguity is high. However, not all these queries really have no ambiguity. It must be also taken into account that, in a multilingual framework, we depend totally on the completeness of each lexicon. This aspect of the formula must be reviewed.
Independently of this fact, the data show a clear increase in the image retrieved according to the increment of the queries ambiguity. With the maximum ambiguity of the queries (0.8 and 0.9), the system retrieves the maximum amount of images allowed (500). With this we can make a preliminary conclusion: with more lexical ambiguity in the query, a higher number of images are retrieved and the system has less precision. In this sense, a WSD system could be useful for IR tasks. 3.2
Table 1 and graphs 3.2 and 3.2b show the average of images located correctly by all users according to the queries' lexical ambiguity. Images not located by the users are included too.
To clarify the data, we have divided them into two graphs. The rst one shows the data with queries with 0 ambiguity up to 0.5 ambiguity. The second one, from 0.5 ambiguity to 0.9.
+
These data clearly show that with low ambiguous queries, the users correctly nd more much images than with queries with high ambiguity: 71.3% of the images have been correctly located with queries with low ambiguity (between 0 to 0.3). However, an unambiguous query doesn't always mean that the user correctly nds an image, or an ambiguous query signify that the user doesn't nd the image. For example, as shown the table, with queries with ambiguity 0, a 74.4% of the images are correctly located and a 23.5% are not located.
For queries with high ambiguity (second graph) the situation is di erent. Actually, these data do not show relevant behaviour: the amount of images located and not located by the user are more or less the same (approximately 50%), but the total amount of images located and not located by the users with higher ambiguity queries is low.
With these data we can achieve interesting conclusions. The great majority of images correctly located has queries with low ambiguity: 1046 images for queries with ambiguity 0; 963 images for queries with ambiguity 0.1, or 1220 images for queries with ambiguity 0.2, from a total of 4011. 92.37% of the images correctly located have queries with low ambiguity (between 0 and 0.3). However, the 88.93% of the images not located have queries with the same level of ambiguity.
The reasons why an image is not located could be di erent: for example, cases where the user does not know the language of the image or the correct translation, or cases where the user does not nd the correct words.
However, we notice that, in the majority of cases, when a user correctly nd an image, the ambiguity of the query is low. From the total amount of images, 71.3% have been correctly located with queries with low ambiguity (between 0 to 0.3), against the 20.26% of images not located with queries with the same level of ambiguity.
Images 45 40 + 35 30 25 20 15 10 5 0 0.5 0.55 + 0.6
According to these data we can conclude that, in the majority of cases, the query's lexical ambiguity in uences the precision of the user nding images with a Multilingual Image Retrieval system. However, at the same time, due to the users correctly nding some images with queries with high ambiguity and they can not nd correctly some images with queries with low ambiguity, our conclusion can not be categorical: the ambiguity of the query is an important (and maybe decisive) factor, but it is not the unique factor for a correct image retrieval by a user.
In this sense, the use of a WSD system in order to improve the IR tasks is important, and maybe decisive. In any case, it is interesting that, according to these data, it is not necessary to disambiguate with one sense per word. Up a 0.3 level of ambiguity is acceptable for a user to nd images correctly. Therefore, a coarse-grained WSD system5 could be useful for IR tasks. 3.3
Another interesting fact is how a speci c user deals with the lexical ambiguity of the queries. In order to show this, we have extracted the lexical ambiguity of the queries from the best user (the user that has located more images correctly). They are shown in graph 3.3 and table 2. We have taken into account only the last query: the query in which the user has found the image.
5That is, WSD systems that could disambiguate with a low level of granularity, with more than one related sense per word, or with general semantic categories.
Queries100 200 180 160 140 120 80 60 40 20 0 0 0.1 0.2 0.3 0.5 0.6 0.7
0.8 0.4
Ambiguity
In this case, the data are clear: the majority of images has been correctly found with nonambiguous queries (47.9% - 190 images). With low ambiguity (between 0.1 and 0.3) the 27.5% (109 images) has been correctly found. However, 17.9% (71 images) of the images have been found with high ambiguity queries (between 0.5 and 0.9).
The behaviour of data with a speci c user is similar to the average of users: the majority of images has been found correctly with low ambiguity queries (75.4% for queries with ambiguity between 0 and 0.3). However, the user can nd images correctly with high ambiguous queries (17.9%). Therefore, the conclusion is the same. For this user, according to these data, the lexical ambiguity of the query is an important factor: with less ambiguous queries, the user can nd more images correctly. However, it is not the unique factor, because there are some cases in which the user can nd images correctly with high ambiguous queries. 4
According to all these data, we assume the following preliminary conclusions:
In general terms, the lexical ambiguity of the queries in uences the precision of a user nding images: with less ambiguous queries, the user can nd more images correctly.
The user can nd images correctly with low ambiguous queries: ambiguity between 0 and 0.3. For a human, it is not necessary to have a ne grain disambiguation: with an ambiguity of 0 to 0.3, he can nd the majority of images correctly.
The user tries to make up the least ambiguous query to improve the retrieval task. Although the lexical ambiguity of the query is an important factor to nd images correctly, it is not the unique nor absolute factor: they can nd some images correctly with ambiguous queries (on average, less than with low ambiguous queries).
Extrapolating these results to the general case about the usefulness of WSD systems in IR tasks, we think that it is necessary to apply these kind of systems to improve the precision of the IR systems. However, it is not necessary to have a ne-grain disambiguation. Otherwise, coarsegrained disambiguation could be useful and enough. For example, to disambiguate with more than one related sense (in WordNet), to use lexical resources with low ambiguity, to disambiguate with semantic classes, generic concepts or domains, etc.
As a future work, there are some aspects that must be extensively developed:
The formula must be improved in order to correctly represent words that do not appear in the lexical resource. Some options are: to use more than one lexicon (general and technical) or to apply Named Entities Recognition systems to the query.
In order to prove that coarse-grained disambiguation is enough for an IR task, it is necessary to repeat the experiment with other lexical resources with less granularity than WordNet, or to reduce the granularity of WordNet with some well-known technique (domains, clustering of related senses, semantic classes, etc.).
Finally, all these data have been extracted from an Image Retrieval system. It is necessary to complete the analysis with a more generic Text Retrieval system. [8] Robert Krovetz. On the Importance of Word Sense Disambiguation for Information Retrieval.
In Creating and Using Semantics for Information Retrieval and Filtering. State of the Art and Future Research. Third International Conference on Language Resources and Evaluation (LREC) workshop, 2002. [9] Fernando Mart nez-Santiago, Jose M. Perea-Ortega, and Miguel A. Garc a-Cumbreras. SINAI at Robust WSD Task @ CLEF 2008: When WSD is a Good Idea for Information Retrieval tasks? In Working Notes for the CLEF 2008 Workshop, 2008. [10] Sergio Navarro, Fernando Llopis, and Rafael Mun~oz. IRn in the CLEF Robust WSD Task 2008. In Working Notes for the CLEF 2008 Workshop, 2008. [11] Arantxa Otegi, Eneko Agirre, and German Rigau. IXA at CLEF 2008 Robust-WSD Task: using Word Sense Disambiguation for (Cross Lingual) Information Retrieval. In Working Notes for the CLEF 2008 Workshop, 2008. [12] Jose R. Perez-Aguera and Hugo Zaragoza. UCM-Y!R at CLEF 2008 Robust and WSD tasks.
In Working Notes for the CLEF 2008 Workshop, 2008.