The hypothesis which this paper tries to validate is that text based image retrieval could be improved by the use of semantic information, by means of an expansion algorithm and a module specifically designed to exclude common words and negated words from queries. The expansion algorithm applies specification marks to disambiguate words making use of WordNet [13]. An implementation of this algorithm has been developed for these experiments. On the other hand, the module in charge of removing common words and detecting negated words has also been specifically developed for this work. However, after an initial evaluation, none of these modules led to an improvement in the retrieval quality compared to the baseline experiment, which consists on the indexing of nouns present in image captions, without no further preprocessing.
The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM)
along with DAEDALUS, a leading company in linguistic technologies in Spain, spin-off of two of these groups,
and the coordinator of the MIRACLE team. This is our fourth participation in CLEF, after years 2003, 2004 and
2005. As well as bilingual, monolingual and robust multilingual tasks [
Following the structure of previous campaigns, ImageCLEF task has been divided again in several subtasks as
described in [
Topics proposed by the organization are divided in three sections, a title, a narrative, i.e. a longer description of the topic, and a set of images that could be used by a Content Based Image Retrieval (CBIR) system to perform the search. Image captions in the IAPR collection have different fields and, in the MIRACLE approach, only title and description fields have been extracted and indexed separately. Taking into account available elements, several experiments have been executed, which are described in section 4.
Our previous experiences in image retrieval processes [9][10] have shown that, in practical, current indexing techniques have reached their precision and recall upper limit. To surpass this point, we think that semantic information should be considered in the retrieval process. For this reason the system developed this year includes an implementation of a WordNet based semantic expansion method that uses specification marks [13] (adapted to version 2.1 of WordNet) and a topic analysis module, which is intended to detect common words and to filter out words introduced by negation expressions.
The semantic expansion method was defined to disambiguate words appearing in WordNet, using the context of the word to select the correct sense among the set of senses assigned by WordNet. Intuitively, the idea is to select the sense pertaining to the hyperonym of the word that includes the greater number of senses for the words appearing in the context. The main objective is to include only the synonyms corresponding to the correct sense or the word instead of adding all synonyms for all senses of a word. Not every word can be disambiguated by applying this algorithm. Thus, to increase the number of correctly disambiguated words, three heuristics were identified. Only one of this heuristics has been included in our implementation, the Definition Heuristic, which discards senses whose gloss does not include any word present in the context. In CLEF 2005 we tried to use a variation of this algorithm, which was intended to disambiguate word pairs, but no optimal results were obtained and, so, this year we decided to use our implementation of the original algorithm. An example of the result of the expansion process applied to the title of topic 30 is shown in Figure 1.
beds, room
"room with more than two beds"
Semantic Expansion without WSD Semantic Expansion with WSD beds, furniture, "piece of furniture", beds, furniture, "piece of furniture", "article of furniture", plot, "plot of "article of furniture", room, area ground", patch, bottom, "natural depression", depression, stratum, layer, sheet, "flat solid", surface, foundation, base, fundament, foot, groundwork, substructure, understructure, room, area, way, "elbow room", position, "spatial relation", opportunity, chance, gathering, assemblage
Figure 1. Example of the semantic expansion method.
On the other hand, a topic analysis module has been developed to make a deeper selection of terms to search against the index. This module is used to filter out common words and expressions as well as to detect negation structures. Thus, phrases like “Relevant images will show” are excluded and words accompanying phrases like “are not relevant” are expressed with a negation symbol “-“ which is interpreted by the search engine. When negations are interpreted, not every word taking part in the sentence is excluded, only those words that do not appear in an affirmative form are negated. An example of the result is shown in Figure 2.
Relevant images will show an interior view of a church or cathedral. Images showing exterior views of churches or cathedrals are not relevant. Interior views of other buildings than fanes are not relevant.
church, cathedral, view, photos, -images, exterior, -buildings,-fanes
These are the two main components included in the MIRACLE image retrieval system tested in this campaign.
The following section describes system architecture. For both subsystems, a linguistic analysis of the topic text is
needed. For this purpose, the Charniak’s parser [
The software architecture for the text based image retrieval system is depicted in Figure 3. As can be seen, the
approach is based on the integration of modules that can be optionally activated in order to configure the
different experiments to be submitted. The retrieval process is divided in two tasks: the indexing process, in
charge of building the indexes to be searched. As already mentioned the used search engine is Xapian [
Tokenizer
Collection
Morphosyntactic
Analyzer (Charniak) Three different indexes are built: one containing the titles of the image captions, another one containing only the descriptions present in the image captions and the last one mixing titles and descriptions. The search process is devoted to the construction of the query to be executed on the previously built indexes. During this process several tasks are performed:
Tokenization. The LingPipe software [
Topic Analysis. Topic analysis is applied to every input topic. Two options can be selected: noun, to extract only words tagged as nouns during the parsing process, or common, to filter out common expressions like “Relevant images will show” and to exclude words introduced by negation structures like “[...] are not relevant”. Only words tagged as nouns are considered and the “-“ operator available in Xapian to exclude words is used. It is also possible to mark the topic section to be used: title, narrative or both. Semantic Expansion. Optionally, semantic expansion can be applied to the output of the Topic Analysis module. This semantic expansion is based on WordNet and the previously described disambiguation method is applied to select the synonyms to be included in the query. Words to be excluded from the query are ignored by the expansion algorithm and the disambiguation process is performed taking into account the scope defined by one sentence.
Different combinations of these modules configure the experiments that have been performed and submitted.
Different modules have been selected to define experiments. Although the IAPR collection is available in two languages, English and German, only the English target language has been considered. On the other hand, four different query languages have been tested: English, Japanese, Polish, Russian and Traditional Chinese. Names and descriptions of runs are included in Table 1. The column “Topic Part” marks the field of the topic that have been processed: if the label “Title” is included, it means that only the title fragment of the topic has been used, whereas if the label “Title+Narrative” is given, it means that both fields of the topic have been processed. Bilingual runs are always marked with the label “Title” because this is the only field provided in topic in languages distinct than English. Values for the “Topic Analysis” column can be “noun” or ”common” as described in the previous section. The “Expansion” columns takes ”Yes” value if the Semantic Expansion module has been used or “No” in other case. Finally, the “Index” column points out which section of the image caption is used, the title fragment, the description fragment or both.
Run Name miratnntdenen miranntdenen miranctdenen miratnctdenen miratncdenen miratnndenen miranndenen mirancdenen miratctdjaen miratntdjaen miratctdplen miratctdzhsen miratntdzhsen miratntdplen miratctdruen miratnndtdenen miranndtdenen mirannddenen miratnnddenen miratndtdjaen miratndtdplen miratncdtdenen miratncddenen mirancddenen mirancdtdenen miratndtenen miratndtdruen miratndtdzhsen
English English English English English English English English Japanese Japanese Polish Trad. Chinese Trad. Chinese Polish Russian English English English English Japanese Polish English English English English English Russian Trad. Chinese
Obviously, there are more possible combinations, but these ones where considered the most appropriate according to previous experiences. For example, using only image titles of captions is not useful because a poor description of each caption can be obtained by the indexer. Besides, if only topic titles are used as the input for the search process, the characterization of the query built by the search engine is weak and no good results are usually obtained. Next section provides precision and recall measures for these experiments and a qualitative explanation for them.
Obtained results are somehow discouraging. Table 3 shows the MAP measure for the submitted runs.
Run Name miratnntdenen miranntdenen miranctdenen miratnctdenen miratncdenen miratnndenen miranndenen mirancdenen miratctdjaen miratntdjaen miratctdplen miratctdzhsen miratntdzhsen miratntdplen miratctdruen miratnndtdenen miranndtdenen mirannddenen miratnnddenen miratndtdjaen miratndtdplen miratncdtdenen miratncddenen mirancddenen mirancdtdenen miratndtenen miratndtdruen miratndtdzhsen
As can be concluded from these figures, the expansion modules does not produce any improvement, on the contrary, a decrease of 18 % in MAP is observed. Although the application of expansion methods have not been definitely proved to increase precision figures, the great decrease produced in these experiments is likely due to a bug in the implementation. The code and partial evaluations of the expansion algorithm are going to be reviewed to determine if it is working in the proper way. On the other hand, when the topic analysis module is used to analyse negation expressions, a decrease in MAP is measured. This is not a strange result, taking into account the complexity of topics. When the topic and image caption sections used in the retrieval process are regarded, one can conclude that if greater amounts of text are used in both topic and caption better precision is obtained. Finally, as observed in previous bilingual retrieval experiments, when the language of topics is different of the language of the document collection an average 10% decrease in MAP is produced. This is due to the noise introduced by the translation step needed in these situations.
Table 4 shows MAP values for the experiments where content based image retrieval is used to support textual retrieval. As concluded in past experiments, content based partial results have no effect in the retrieval precision. miratnntdienen miratncdtdienen 0,2016 0,0084
One direct conclusion from the previous section is that the experiment considered as the baseline could not be improved. Although a deeper exploration of results and processes have to be carried out, initially seems to be due to a improper operation of the expansion module. Besides, it is worth mentioning that there is an 8% decrease regarding the best MAP obtained last year and in both years experiments were quite similar. This decrease is the effect of changing the image collection used to test both systems and a clear dependency among retrieval techniques and image collections used to test those techniques can be concluded. It will be interesting to compare results for other participants using both test collections.
A conclusive evaluation of the functionality of the implemented expansion algorithm must be performed. The analysis of obtained results has been started but not still concluded by the time of writing this report. Some failures in the code have been already detected and corrected. The final goal is to include results of experiments run with the reviewed expansion algorithm and compared with the actual ones.
Future work in this image retrieval task will try to exploit semantic information obtained from syntactic analysis and from external resources. Text captions present in the IAPR collection are formed by nominal and prepositional phrases that could be analysed to extract relations among concepts represented by the headers of phrases. Some works in this line will be tested in future campaigns.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for Madrid Community), S-0505/TIC/000267.
Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): José Carlos González-Cristóbal, Ana González-Ledesma, José Miguel Goñi-Menoyo, Sara Lana-Serrano, Ángel MartínezGonzález, Antonio Moreno-Sandoval and César de Pablo-Sánchez. line http://ir.shef.ac.uk/cloughie/papers/muscle[6] Lana-Serrano, S.; Goñi-Menoyo, J.M.; and González-Cristóbal, J.C. MIRACLE at GeoCLEF 2005: First Experiments in Geographical IR. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear).