This paper describes the participation of MIRACLE research consortium at the ImageCLEF Photographic Retrieval task of ImageCLEF 2007. For this campaign, the main purpose of our experiments was to thoroughly study different merging strategies, i.e. methods of combination of textual and visual retrieval techniques. While we have applied all the well known techniques which we had already used in previous campaigns, for both textual and visual components of the system, our research has primarily focused on the idea of performing all possible combinations of those techniques in order to evaluate which ones may offer the best results and analyze if the combined results may improve (in terms of MAP) the individual ones. The system includes three main modules. On one hand, apart from the search engine (Xapian or Lucene), the textual retrieval module includes parsers, stemming, stopword filtering, proper noun detection and semantic expansion components. On the other hand, the visual retrieval module is based on two well-known content-based engines: GIFT and FIRE. Finally, the merging module allows to use different operators (AND, OR, LEFT, RIGHT) to combine the outputs of the two previous subsystems and to calculate the result relevance based on different metrics (max, min, avg, max-min). We finally submitted 110 multilingual textual (text-based) runs, 22 visual (content-based) runs and 21 mixed runs. Results in general show a poor performance for all groups, due to the characteristics of the image collection and the difficulty of the defined topics. The most interesting conclusion is that the defined merging strategies are successful as our best mixed experiment outperforms both the textual and visual experiments in which it is based, using the LEFT operator for the combination along with the max-min metric for computing the relevance.
This paper describes our participation at the ImageCLEF Photographic Retrieval task of ImageCLEF 2007.
Briefly, the goal of this task (fully described in [
Participants are provided with a list of 60 topics which include a short textual title representing the research goal in 15 different languages (Danish, Dutch, English, Finnish, French, German, Italian, Japanese, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, and Simplified and Traditional Chinese) and, in addition, three image examples for each topic. The objective is to retrieve as many relevant images as possible from the given visual and multilingual topics.
For this campaign, the main purpose of our experiments was to thoroughly study the different merging strategies, i.e. methods of combination of visual and textual techniques. While we have applied all the well known techniques which we had already used in previous campaigns [11] [12], for both textual and visual components of the system, our research has focused on the idea to perform all possible combinations of those techniques in order to evaluate which ones may offer the best results.
All experiments are fully automatic, thus avoiding any manual intervention. None of the experiments incorporates relevance feedback, due to time constraints. We finally submitted 110 multilingual textual (textbased) runs, 22 visual (content-based) runs and 21 mixed runs (using a combination of both), elaborated on in the following sections.
Based on our experience in previous campaigns, to be able to execute a large number of runs that exhaustively cover all the combinations of the different techniques, we developed a flexible system, composed of a set of small components which can be easily added in different configurations and are executed sequentially to build the final result set. the textual (text-based) retrieval module, which indexes the IAPR TC-12 image descriptions to look for those descriptions that are more relevant to the text of the topic; the visual (content-based) retrieval module, that provides the IAPR TC-12 images which are more similar to the given topic images; and, finally, the merging module, which uses different operators to combine the outputs of the two previous subsystems to provide the final results. 2.1.
Since MIRACLE has taken part in ImageCLEF (or, in general, in CLEF), different linguistic and statistical techniques have been developed to be used in the text-based part of the different tasks [11] [12]. This year, the main goal was to make an exhaustive study of these diverse methods, combining them in all possible ways and testing the results achieved. A flexible and configurable system has been developed for this purpose. The list of components includes:
Proper Noun Detection. A heuristic based module to detect the appearance of entities in the text, based on a finite state automaton.
Linguistic Analyzer. A component to obtain morphosyntactic analysis (POS tagging) and
lemmatization. The target language considered in MIRACLE experiments has been English (although
the IAPR TC-12 collection was also available in German and Spanish) so the Charniak parser [
Stemming. Of course, a component to obtain stems for words has been included as part of the indexing
process. The implementation considered for this component has been the Porter’s stemmer [
Semantic Expansion. Words appearing in the query can be expanded using Wordnet [
Search Engine. Two different search engines have been used, Lucene [1] and Xapian [
The final goal of this system configuration was to evaluate which combination of techniques was the best, considering last year relevance assessments. However, there were many possible combinations (although some of them were illogic, like applying semantic expansion to stems). Therefore only those with the best results on 2006 data where selected. 2.2.
For this part of the system, we resorted to two publicly and freely available Content-Based Information Retrieval
systems: GIFT (GNU Image Finding Tool) [
In the case of GIFT, the complete image database was indexed in a single collection, downscaling each image to 32x32 pixels. For each topic, a visual query is made up of all the images contained in the topic. Then, this visual query was used in GIFT to obtain the list of the most relevant images (i.e., images which are more similar to those included in the visual query), along with the corresponding relevance values.
On the other hand, we used the results of the FIRE system (kindly provided by the organizers), with no further processing. 2.3.
The textual and image result lists are then merged and combined by applying different techniques, which are characterized by an operator and a metric for computing the relevance (score) of the result. Table 1 shows the defined operators: union (OR), intersection (AND) and external join (LEFT/RIGHT JOIN). Each of these operators selects which images are part of the final result set. Then, results are reranked by computing a new relevance measure value based on their corresponding input results, by using different metrics shown in Table 2. score = max(a, b) score = min(a, b) score = avg(a, b) score = max(a, b) + min(a, b) *
min(a, b) max(a, b) + min(a, b)
Experiments are defined by the choice of different combinations of the previously described modules, operators and score computation metrics. All experiments are fully automatic, avoiding any manual intervention. None of the experiments incorporates relevance feedback, due to time constraints.
We finally submitted a wide set of experiments: 110 multilingual textual (text-based) runs, 22 visual (contentbased) runs and 21 mixed runs (using a combination of both). The different name schemas for the run identifiers are described in the following tables: textual retrieval experiments (Table 3), visual retrieval experiments (Table 4) and mixed experiments (Table 5). (2) Method: TI (use image title index), DE (use image description index), NO (use image notes index), LO (use image location index), NP (perform proper noun detection), S (apply stemming) (3) Combination: OR (images in any result set), AND (images in both result sets), max (use maximum value), min (use minimum value), avg (use average value) (4) Topic language: DE (German), ES (Spanish), FR (French), JA (Japanase), PT (Portuguese), RU (Russian), SV (Swedish) (2) Combination: OR (images in any result set), AND (images in both result sets), max (use maximum value), min (use minimum value), avg (use average value) After the evaluation by the task organizers, results for each kind of experiment are presented in the following tables. Each table shows the run identifier, the mean average precision (MAP), the precision at 10, 20 and 30 first results and the number of relevant images which have been retrieved (out of 3,416 relevant images). Regarding text-based experiments (Table 6), the best results are achieved with the experiment where the title (TI), description (DE) and location (LO) fields of the image caption have been indexed using Xapian (Xa) and applying the proper noun detection module (NP) along with the Porter stemming algorithm (S). For this experiment a MAP of 0.1995 is obtained. Last year, our top ranked text-based experiment had a MAP of 0.2005 with a similar combination of techniques. The only difference is that, while in 2006 topic narratives where also used to build the query, this year there was no narrative provided for the topics. Implications of this are that no improvement has been reached. (1) out of 3,416 relevant images In general, our experiments show Lucene turns out to give worse results than Xapian (the first run with Lucene search engine has a MAP of 0.1871), although this conclusion has to be further studied. Moreover, no strategy for merging results from both search engines have led to any improvement in MAP with respect to the baseline experiments, neither with the OR operator (which was supposed to increase the number of results at the expense of a loss of precision) nor the AND operator (which was supposed to increase precision). (1) Only the first two results for each language are shown 1896 1902 1616 1616 1942 1946 1672 1567 1518 1533 1771 1791 1461 1446 The best results correspond to Spanish, our mother tongue in which we have a strong expertise, with a MAP similar to the monolingual experiments. We are negatively surprised by the low precision with French and German languages that may be attributed to a deficient stemming module. However, this issue has to be further researched.
The 2007 collection did not include the description field and our experiments have been produced using 2006 data collection (we did not notice this fact until the submission was over). Therefore our results are not really comparable to the results provided by the rest of participants, shown in Table 8. However, our baseline experiment would be among the best results this year and our group would be ranked 4th out of 16. Our conclusion is that for the IAPR TC-12, textual descriptions of images are very convenient for the retrieval process and should not be dropped out in for the next years. Table 9 in next page shows the results of the visual-based experiments. In this case, MAP is very low due to the complexity of the visual retrieval task in this domain. The best experiment is the combination of results with GIFT and FIRE, and it is better than using GIFT alone. However, we regretfully did not execute the experiment with FIRE alone, which would have been shown whether the combination of both systems improves the final result. Also note that, although the number of relevant images retrieved using AND is lower than when OR is used, the precision is higher. Results from other participants, shown in Table 10, clearly outperform our own results. Our MAP is 56% lower than the MAP of the best experiment and our group is ranked 7th out of 13 participants. The best experiment uses a combination of all textual and visual results obtained in the previous experiments with the LEFT operator along with the Max-Min (mm) scoring metric. Moreover, the mm metric clearly outperforms the others (max, min, avg) even with the same operator over the same result sets, as seen in Mix2LEFT experiments. The most interesting conclusion is that this merging strategy is successful as the MAP is higher than in both the textual and visual partial experiments with which the experiment is built. Finally, although our mixed experiments can really be compared to the others groups’ for the reasons already stated before, our MAP is considerably lower than the MAP of the best mixed experiment (Table 12), with a decrease to 70%. However, MIRACLE ranked 5th (out of over 10 participants), which is indeed considered to be an acceptable position.
cut-EN2EN-F50
Results in general show a poor performance for all groups, due to the characteristics of the image collection and the difficulty of the defined topics.
For the textual retrieval, the experiment where the title, description and location fields of the image caption have been indexed using Xapian and applying the proper noun detection module and the stemming component has produced the best results, with a MAP of 0.1995, which shows that no improvement has been reached with regards to last year experiments when our top ranked text-based experiment had a MAP of 0.2005 with a similar combination of techniques. In general, we obtain worse MAP with Lucene rather than Xapian, although this topic has to be further investigated. Regarding the visual retrieval, our results are not good compared to other groups’, but it is not strange as our areas of expertise do not specifically include image processing and we resorted to publicly available “black-box” engines. In addition, no specific conclusion can be obtained as we omitted to include the result for FIRE itself, so we could not evaluate if the merging strategy is successful or not. However, our main interest was not in experiments where only text or image content is used in the retrieval process. Instead, our challenge was to test whether the text-based image retrieval could improve the analysis of the content of the image, or vice versa. The most interesting conclusion is that merging strategies are successful as our best mixed experiment outperforms both the textual and visual experiments in which it is based, using the LEFT operator for the combination along with the Max-Min (mm) metric for computing the relevance. This result shows that our hypothesis was right. Our combination of a “black-box” search using publicly accessible content-based retrieval engines with a text-based search has turned out to provide better results than other presumably “more complex” techniques. This simplicity may be a good starting point for the implementation of a real system.
We are sure that there may still be some place for improvement with a more careful study of the parameters for the merging strategy (both the combination operator and the score computing metric). In addition, none of the experiments incorporates relevance feedback, due to time constraints. However, we are already carrying out some experiments that incorporate this technique which are showing promising results, overcoming our submissions.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267. [1] Apache Lucene project. On line http://lucene.apache.org [Visited 10/08/2007].
On line