This paper presents the 2005 MIRACLE's team participation in the ImageCLEFmed task of ImageCLEF 2005. This task certainly requires the use of image retrieval techniques and therefore it is mainly aimed at image analysis research groups. Although our areas of expertise don't include image analysis research, we decided to make the effort to participate in this task to promote and encourage multidisciplinary participation in all aspects of information retrieval, no matter if it is text or content based. We resort to a publicly available image retrieval system (GIFT [1]) when needed.
The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM)
along with DAEDALUS, a company founded in 1998 as a spin-off of two of these groups. DAEDALUS is a
leading company in linguistic technologies in Spain and is the coordinator of the MIRACLE team. This is the
third participation in CLEF, after years 2003 and 2004 [
Image or multimedia retrieval is interesting for the domain of cross-language information retrieval as the media
such as images are inherently almost insensitive to language. Many collections exist on the Internet which
contain images as well as multilingual texts. However, the retrieval of images is an often-neglected topic in the
information retrieval domain. In particular, hospitals produce an enormous amount of visual data but tools to
manage these images and videos are scarce and exist currently only as research prototypes [
ImageCLEFmed 2005 extends the 2004 experiments with a larger database and more complex queries. The
database consists of images [
Query tasks have been formulated with example images and a short textual description explaining the research goal. The task organizers provide a list of topic statements in English, French and German, and a collection of images for each topic. Normally one or two example images for the desired result are supplied. One query also contains a negative example as a test. The goal of ImageCLEFmed is to retrieve as many relevant images as possible from the given visual and multilingual topics.
The task organizers have also made available results from a state-of-the-art image retrieval system (medGIFT)
and a state-of-the-art text engine (Lucene [
The next section is devoted to the description of the different experiments which were carried out. 3
We focused our experiments to fully automatic retrieval, avoiding any manual feedback, and submitted runs both using only visual features for retrieval (content-based retrieval) and also runs using visual features and text (combination of content-based and text-based retrieval).
To isolate from the content-based retrieval part of the process, we resorted to GIFT (GNU Image Finding Tool)
[
Our approach is based on the multidisciplinary combination of GIFT content-based searches with text-based retrieval techniques. Our system consists of three parts: the content-based retrieval component (mainly GIFT), the text-based search engine and the merging component, which combines the results from the others to provide the final results.
We finally submitted 13 different runs to be evaluated by the task coordinators, which are explained in the following section.
This experiment consists on a content-based-only retrieval using GIFT. Initially the complete image database was indexed in a single collection using GIFT, down-scaling each image to 32x32 pixels. For each ImageCLEFmed query, a visual query is made up of all the images contained in the ImageCLEFmed query. Then, this visual query is introduced into the system to obtain the list of more relevant images (i.e., images which are more similar to those included in the visual query), along with the corresponding relevance values.. Although different search algorithms can be integrated as plug-ins in GIFT, only the provided separate normalisation algorithm has been used in our experiments.
There is only one submission, with “mirabase.qtop” as its run identifier.
These experiments are similar to the preceding one, also a content-based-only retrieval using GIFT, but incorporating relevance feedback. Each visual query is introduced into the system to obtain the list of images which are more similar to the visual query. Then the top N results are added to the original visual query to build a new visual query which is again introduced into the system to obtain the final list of results. In addition, GIFT allows to build a weighted visual query in which a relevance value may be associated to each included image. There are 3 submissions: • • • mirarf5.qtop This run takes the 5 most relevant images for feedback, each one with a value of 1 for its relevance in the visual query. The relevance in the visual query for the original images remains 1. mirarf5.1.qtop The same as mirarf5.qtop but using a value of 0.5 for the relevance in query of feedback images. The relevance in query for the original images remains 1. mirarf5.2.qtop
The same as mirarf5.qtop but using a value of 0.5 for the relevance in query of the original images. Finally, the different content-based runs are shown in Table 1.
First all the case annotations are indexed using a text-based retrieval engine (explained later). Natural language
processing techniques are applied before indexing. An adhoc language-specific (for English, German and
French) parser is used to identify different classes of alphanumerical tokens such as dates, proper nouns,
acronyms, etc., as well as recognising common compound words. Text is tokenized, stemmed [
Two different text-based retrieval engines were used. One was Lucene [
The combination strategy consists on reordering the results from the content-based retrieval using a text-based retrieval. For each ImageCLEFmed query, a multilingual textual query is build with the English, German and French queries (first processing each one with the language depending parser and then concatenating the three lists), and executed in the search engine to obtain the list of top-1000 cases which are more relevant to the textual query.
The list of relevant images from the content-based retrieval is reordered, moving to the beginning of the list those images which belong to a case that is in the list of top-1000 cases. The rest of the images remain in the end of the list.
There are 10 different submissions: • • • mirabasefil.qtop, mirarf5fil.qtop, mirarf5.1fil.qtop, mirarf5.2fil.qtop These runs consist on the combination as previously described of content-based-only runs with the textbased retrieval obtained with KSite. mirabasefil2.qtop, mirarf5fil2.qtop, mirarf5.1fil2.qtop, mirarf5.2fil2.qtop The same experiment, but using Lucene.
Other runs Two other experiments were developed to test if there was any difference in results when using our own content-based GIFT index or using the medGIFT results provided by the task organizers. So, medGIFT was used as the starting point and then the same combination method as described before was applied. o o mirabase2fil.qtop medGIFT results filtered with text-based KSite results mirabase2fil2.qtop medGIFT results filtered with Lucene results Finally, the different mixed retrieval runs are shown in Table 2.
Relevance assessments have been performed by experienced medical students and medical doctors at OHSU and the University hospitals of Geneva. Submissions from all groups are used to create image pools, which are judged for relevance by assessors, based on using a ternary classification scheme: (1) relevant, (2) partially relevant and (3) not relevant. The aim of the ternary scheme is to help assessors in making their relevance judgments more accurate (e.g., an image is definitely relevant in some way, but maybe the query object is not directly in the foreground; it is therefore considered partially relevant).
The pools are assessed and the end result is a set of relevance assessments called qrels, which are then used to evaluate system performance and compare submissions from different groups.
The results of the content-based runs are shown in the next table, ordered by its mean average precision value. As shown in Table 3, the best result was obtained with the base experiment, which means that relevance feedback has failed to improve the results (neither to worsen them). This may be due to an incorrect choice of the parameters, but this has to be further studied.
Apart from MIRACLE, other 8 groups participated in this year's evaluation in the content-based-only runs. Table 4 compares each group’s best submission. Only one group is above us in the group ranking, although their average precision is much better than ours. Our pragmatic approach using a “standard” publicly available content-based retrieval engine such as GIFT has proved to be a better approach than other presumably more complex techniques. We still have to test if another selection of indexing parameters (different from image down-scaling to 32x32 pixels and separate normalization algorithm) may provide better results. The results of the content-based and text-based mixed retrieval runs are shown in Table 5, ordered by its mean average precision value. In this case, using relevance feedback provides slightly better precision values. Considering the best runs, the optimum choice seems to be to assume 1.0 for the relevance of the top 5 results and reduce the relevance of the images in the original query.
It is interesting to observe that the worst combination is to take both results provided by the task organizers (content-based medGIFT results and text-based Lucene results), with a performance decrease of 15%. Comparing content-based runs with the mixed runs, Table 6 shows that the combination of both types of retrieval offers better performance and even the worst mixed run is better than the best content-based only run. This actually proves that text-based image retrieval can be used to improve the content-based only retrieval, with much superior performance. Apart from MIRACLE, other 6 groups participated in this year's evaluation in the content-based and text-based runs. Table 7 shows the results for each group’s best submission. In this case, our position in the table shows that the submissions from other groups clearly surpassed our results. Anyway, these results are very satisfying for us, considering that we are not a group with expertise in image analysis research.
It is also interesting to note that most groups managed to improve their results with mixed approaches over the content-based only runs. This is especially visible for the NCTU group, with an improvement from 0.06 to 0.23 (+355%) in the average precision. 5
Our main interest is not in experiments where only image content is used in the retrieval process. Instead, our challenge was to test whether the text-based image retrieval could improve the analysis of the content of the image, or vice versa. Results show that this hypothesis was right. Our combination of a “black-box” search using a publicly accessible content-based retrieval engine with a text-based search has turned to provide comparable results to other presumably “more complex” techniques. This simplicity may be a good starting point for the implementation of a real system.
We think that there still may be some space for improvement with a more careful study of the parameters for the relevance feedback and the combination strategy.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01 and also by the European Union with the funding of NEDINE project in the e-Content programme.
Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, Ana González-Ledesma, José José Mª Guirao-Miras, Sara Lana-Serrano, Paloma MartínezFernández, Ángel Martínez-González, Antonio Moreno-Sandoval and César de Pablo Sánchez.