Several researches demonstrate that the use and integration of various knowledge sources improves the quality and efficiency of information systems. This paper presents the system developed by the SINAI research group at the ImageCLEF Wikipedia Retrieval task. Using only the English text associated with each image or its translation from French or German, our system applies several combinations of the textual tags and the combination of information retrieval (IR) systems. The influence of machine translation, the use of the different annotation tags and the retrieval results from different IRs systems are the main aims of this work. The obtained results show that the election of the annotation tags and the IR system have influence in the results, and some fusions of the retrieved lists can improve the performance.
The Wikipedia Retrieval task is an ad-hoc image retrieval task. The aim of
ImageCLEF Wikipedia Retrieval task is to investigate retrieval approaches with a
large and heterogeneous collection of images and their annotations, extracted from
Wikipedia1 [
In 2011 the collection is the same used in the 2010 ImageCLEF Wikipedia task [
Given an English query as an image, sample images and annotations, our textbased system uses the annotations to find relevant images from the Wikipedia image collection. Several combinations of the results obtained with the annotation labels were tested and the fusion of the results obtained with two IRs systems.
The remainder of the paper is organized as follows: in the second section the system is described. In Section 3, the experiments and results are presented. Finally, conclusions and further work are shown in Section 4.
An image from the Wikipedia collection contains some data, such as the name or the file of this image and annotations data. These annotations contain three optional tags (description, comment and caption) in different languages (English, French and German) also optional. Fig. 1 shows an example of an image with metadata from the collection.
The following Fig. 2 shows the general architecture of our system. The collection modules (on the left of the figure) work with the collection of annotations. The first module translates non English data into English, as it has been explained previously. The second module processes the English data, applying stopwords removal and stemming. Caption and description tags are extracted and different combinations are created. Then, the IR systems Lucene and Lemur create different indexes with the combinations of tags.
The queries module (on the right of the figure) works with the queries. The first module preprocesses the query (stopwords removal and stemming), and then the information is run against the indexes, obtaining different lists of retrieved documents. In some cases a fusion module is applied to join the lists from the IR systems.
Translation module (Reverso) Data processing (stopper, stemmer, tags combinations)
Tags combination Collection English data (original or translated) Caption and description tags
Query processing (stopper, stemmer) IR Systems Lucene & Lemur (English)
IRs
Combination of relevant lists
Our text-based system uses English as main language and according with the language distribution of the collection, the 237,434 Wikipedia images cannot be used. Some of the do not have annotations or the language is not determined. The following images were used in our experiments: - English only: 70,127 - English and German: 26,880 - English and French: 20,747
English, German and French: 22,899 Translations from o German only: 50,291 o French only: 28,461 o German and French: 9,646
A total number of 140,653 English annotations were used and 88,398 non-English annotations were translated.
The first module makes the automatic translation for the non-English metadata. For
this purpose we used the online machine translator Reverso2. Captions and
descriptions were translated, and comments were discarded because in previous
experiments we concluded that they were not useful. The second module makes a
typical text preprocessing (text from caption and description tags). English stopwords
removal and the Porter's stemmer [
Then, the third module uses two Information Retrieval (IR) systems, Lemur3 and Lucene4. Lucene is a high performance, full-featured text search engine that has shown to be robust in several text retrieval tasks. The Lemur Toolkit is an opensource toolkit designed to facilitate research in language modelling and information retrieval. It supports different automatic indexing strategies and a variety of retrieval models. In both cases the default parameters were used, the Okapi weighting schema for Lemur and automatic feedback was applied. 2.1 Combining the lists We have implemented two simple methods to combine the lists of relevant documents retrieved by the IR or IRs systems.
The first method is the combination of the lists for each annotation tag. The baseline
experiments were run using only the description and caption tags. The results of these
experiments were two lists of relevant documents for each IR system used. In a
second step we used a simple technique to merge these lists [
The different combinations are showed in the following list: - The use of the description tag only - 10% of caption and 90% of description - 20% of caption and 80% of description - ….. - 80% of caption and 20% of description - 90% of caption and 10% of description - The use of the caption tag only
All these combinations were indexed with Lucene and Lemur separately and then the queries were run against each index. We made previous experiments with the ImageCLEF Wikipedia Retrieval 2010 framework (non-official results) and the best results were obtained with the combination 80% of caption and 20% of description. The next section shows these experiments and results.
The second method fusion the lists of relevant documents obtained the IRs systems used. Lucene and Lemur results were combined for each query, with different weights for each list, as it was explained in the previous paragraph. Based on the experiments made with the 2010 framework, the results show that Lemur works better than Lucene. The best combined result was obtained with the fusion 90% of Lemur and a 10% of Lucene, although the improvement over the Lemur global result was not significant. The next section shows also these experiments and results.
This section describes some previous experiments made in the ImageCLEF Wikipedia framework that were the base of the system developed. The previous results are not shown completely due to the amount of them.
As it was described previously, the dataset used contains 140,653 English annotations and 88,398 non-English annotations translated (229,051 in total). The complete collection contains 237,434 Wikipedia images, so we used a 96% of the total collection. 3.1 Previous experiments - Lemur IR system with Okapi and feedback and Lucene with different weights for each list of relevant documents (from 10% Lemur and 90%Lucene to 90% Lemur and 10% Lucene).
The following Table 1 shows some relevant results obtained with the 2010 framework:
Exp1: Lucene without feedback. Caption label.
Exp2: Lucene without feedback. Description label.
Exp3: Lucene without feedback. Caption and description labels.
Exp4: Lucene without feedback. 80% Caption and 20% description labels. - Exp5: Lemur with Okapi and feedback. Caption label. - Exp6: Lemur with Okapi and feedback. Description label. - Exp7: Lemur with Okapi and feedback. Caption and description labels. - Exp8: Lemur with Okapi and feedback. 80% Caption and 20% description labels. - Exp9-Best combination: 90% Lemur with Okapi and feedback and 10%
Lucene without feedback. Caption and description labels.
All the results are evaluated using the Mean Average Precision (MAP) function.
Last column shows the improvement or decrement in percentage of each MAP result
over the baseline case with Lucene and Lemur.
The best textual run in 2010 was submitted by XRCE with a MAP of 0.2361 [
We analyzed some features of the queries such as the number of words of the query and the relevant documents obtained with each IR system, and some features of the retrieved list of relevant documents such as the RSV (Retrieval Status Value). After this analysis we concluded that these features cannot determine the use of one or another IR system or the use of each tag or its combination. 3.2 Results with the 2011 framework.
Based on the previous results shown we run six experiments with the 2011 framework to test some features of our system. These six official runs are the following: - Exp1: Lucene without feedback. Caption label. - Exp2: Lemur with Okapi and feedback. Caption label. - Exp3: Lucene without feedback. Caption and description labels. - Exp4: Lucene without feedback. 80% Caption and 20% description labels. - Exp5: Lemur with Okapi and feedback. Caption and description labels. - Exp6: 90% Lemur with Okapi and feedback and 10% Lucene without feedback. Caption and description labels.
Table 2 shows the official runs and results for Image Wikipedia retrieval 2011.
Lemur improves Lucene in all the experiments.
Caption tag introduces almost the complete relevant documents, and description tag does not introduce some relevant documents.
The combination of tags improves the results.
The fusion of lists of relevant documents retrieved by the IR systems improves only a bit the MAP result.
In this paper we present the experiments and results obtained with our system developed for the ImageCLEF Wikipedia retrieval environment. We tested some techniques with the 2010 framework, applying them to the official runs in 2011.
In this participation we have tested some textual features such as different IR systems, the importance of the annotation labels and some simple fusion methods to combine these labels and the lists of relevant documents retrieved.
Our results show that the IR systems do not obtain the same results, some labels are more important than others and the combination of them improves the final results.
As further work we want to analyze in depth the results obtained, queries grouped by topics, by length, etc. We will also test the influence of the translation with different automatic translators. An important future aim will be to train a machine learning system with relevant features that decide what to apply to improve the results (combination of IR systems, external knowledge or the annotations from the different tags).
Acknowledgments. This work has been supported by the Regional Government of Andalucía (Spain) under excellence project GeOasis (P08-41999), the Spanish Government under project TEXT-COOL 2.0 (TIN2009-13391-C04-02) and the local project RFC/UJA2009/12/14.