H.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4 Systems and Software; H.3.7 Digital Libraries General Terms
This article describes the use of the medGIFT retrieval system for three of the four ImageCLEF 2005 retrieval tasks. We participated in the ad{hoc retrieval task that was similar to the 2004 ad{hoc task, the new medical retrieval task that required much more semantic analysis of the textual annotation than in 2004 and the new automatic annotation task. The techniques used in 2005 are fairly similar to the 2004 techniques for the two retrieval tasks. For the automatic annotation task, scripts were optimised to allow classi cation with a retrieval system. Unfortunately, an error in the text retrieval system corrupted part of our runs and led to relatively bad results for all runs including text. This error should be xed before the nal proceedings are printed, so correct gures are expected for this.
All retrieval results rely heavily on two retrieval systems: for visual retrieval we use the GNU Image Finding Tool (GIFT ), and for textual retrieval the EasyIR retrieval system. For the ad-hoc retrieval task, two runs were submitted with di erent con gurations of grey levels and the Gabor lters. No textual retrieval was attempted, but only purely visual retrieval, resulting in generally lower scores than text retrieval. For the medical retrieval task, visual retrieval was performed with several con gurations of Gabor lters and grey level and color quantisations as well as several variations of combining text and visual features. Unfortunately, all these runs are broken as the textual retrieval results are almost random. Due to a lack of resources no relevance feedback runs were submitted, which is where medGIFT performed best in 2004. For the classi cation task, a retrieval with the image to classify was performed and the rst N = 1; 5; 10 resulting images were used to calculate scores for the classes by simply adding up the score of the N-images for each class. No machine learning was performed on the data of the known classes, so the results are surprisingly good and were only topped by systems with sophisticated learning strategies optimised for the used data set.
Image retrieval is an increasingly important domain in the eld of information retrieval. Compared
to text retrieval little is known about how to search for images, although it has been an extremely
active domain as well in the eld of computer vision as in information retrieval [
In 2005, the ad{hoc retrieval task created topics that were better adapted for visual systems using the same database as in 2004. The tasks made available contained three images, so more visual information. We submitted two con gurations of our system to this task using visual information only.
The medical retrieval task was performed on a much larger database than in 2004 containing
a total of more than 50.000 images [
The automatic annotation task was very interesting and challenging at the same time [
ImageCLEF gave us the opportunity to compare our system with other techniques which is invaluable and will provide us with directions for future research. 2
For our ImageCLEF participation, we aim at combining content-based retrieval of images with cross{language retrieval applied on the textual annotation of the images. Based on the results from last year (2004), we used parameters that were expected to lead to good results, plus some new combinations. 2.1
The technology used for the content{based retrieval of images is mainly taken from the Viper 1
project of the University of Geneva. Much information about this system is available [
Local color features at di erent scales by partitioning the images successively into four equally sized regions (four times) and taking the mode color of each region as a descriptor; 1http://viper.unige.ch/ 2http://www.gnu.org/software/gift/ global color features in the form of a color histogram, compared by a simple histogram intersection; local texture features by partitioning the image and applying Gabor lters in various scales and directions. Gabor responses are quantised into 10 strengths; global texture features represented as a simple histogram of responses of the local Gabor lters in various directions and scales.
A particularity of GIFT is that it uses many techniques well{known from text retrieval. Visual
features are quantised and the feature space is very similar to the distribution of words in texts,
corresponding roughly to a Zipf distribution. A simple tf/idf weighting is used and the query
weights are normalised by the results of the query itself. The histogram features are compared
based on a histogram intersection [
The medical version of the GIFT is called medGIFT 3 [
The basic granularity of the Casimage and MIR collections is the case. A case gathers a textual report, and a set of images. For the PathoPic and Peir databases annotation exists for every image. The queries contain one to three images and text in three languages. We used all languages as a single query and also indexed all documents in a single index. Case{based annotation was expanded to all images of the case after the retrieval step, so for us the nal unit of retrieval is the image. 2.2.1
Textual experiments were conducted with the easyIR engine4. As a single report is able to contain
written parts in several languages mixed, it would have been necessary to detect the boundaries of
each language segment. Ideally, French, German and English textual segments would be stored in
di erent indexes. Each index could have been translated into the other language using a general
translation method, or more appropriately using a domain-adapted method [
We chose a generally good weighting schema of the term frequency - inverse document frequency family. Following weighting convention of the SMART engine, cf. Table 1, we used atc-ltn parameters, with = = 0.5 in the augmented term frequency.
3http://www.sim.hcuge.ch/medgift/ 4http://lithwww.epfl.ch/~ruch/softs/softs.html
First Letter n (natural) l (logarithmic) a (augmented) Second Letter n(no) t(full) Third Letter
n(no) c(cosine)
Term Frequency f(tf)
tf 1 + log(tf) + ( matxf(tf) ), where = 1 Inverse Document Frequency f( d1f )
1 log( dNf ) Normalisation f(length)
1
qPi=1 wi2;j
t
qPj=1 wj2;q
t
and 0 <
Combinations of visual and textual features for retrieval are rather scarce in the literature [
To combine the two lists, two di erent methods were chosen. The rst one simply combines the list with di erent percentages for visual and textual results (textual= 50, 33, 25, 10%). In a second form of combination the list of the rst 1000 visual results was taken, and then, all those that were in the rst 200 textual documents were multiplied with N{times the value of the textual results. 3
For the ad{hoc retrieval task we submitted results using fairly similar techniques as those in 2004. The 2005 topics were actually more adapted to the possibilities of visual retrieval systems as more visual attributes were taken into account for the topic creation. Still, textual retrieval stays very necessary for good results. It is not so much a problem of the queries but rather a problem of the database containing mostly grey or brown scale images of varying quality where automatic treatment such as color indexing is di cult. This should change in 2006 with a new database using mostly consumer pictures of vacation destinations. Such a database could be better analysed automatically using the available color information
We used the GIFT system in two con gurations, once the normal GIFT engine with 4 grey levels and the full HSV space using the Gabor lter responses in four directions and at three scales. The second con guration took into account 8 grey levels as the 2004 results for 16 grey levels were actually much worse than expected. We also raised the number of directions of the Gabor lters to 8 instead of four. The results of the basic GIFT system were made available to all participants and used by several. Surprisingly the results of the basic GIFT system remain the best in the test with a MAP of 0.0829, being at the same time the best purely visual system participating. The system with eight grey levels and eight directions for the Gabor lters performed slightly worse and a MAP of 0.0819 was reached. Other visual systems performed slightly lower. The best mono{lingual text systems performed at a MAP of 0.41. Several text retrieval systems performed worse than the visual system for a variety of languages. 4
We were new to the automatic annotation task as almost everyone and had mainly used our system for retrieval, so far. Due to a lack of resources no optimisation using the available training data was performed. Still, the tf/idf weighting is automatically weighting rare features higher which leads to a discriminative analysis.
As techniques we performed a query with each of the 1000 images to classify and took into account the rst N = 1; 5; 10 retrieval results. For each of these results images from the training set the correct class was determined and this class was thus augmented with the similarity score of the image. The class with the highest nal score became automatically the nal class selected for the image. For retrieval we used three di erent settings of the features using 4, 8, and 16 grey levels. The runs with 8 and 16 grey levels also had eight directions of the Gabor lters for indexation. Best results obtained in the competition were from the Aachen groups (best run at 12.6% error rate) that have been working on very similar data for several years, now.
The best results for our system were retrieved when using 5NN and eight grey levels (error rate 20.6%), and the next best results using 5NN and 16 grey levels (20.9). Interestingly, the worst results were obtained with 5NN and 4 grey levels (22.1). Using 10NN led to slightly worse results (21.3) and 1NN was rather in the middle (4 grey levels 21.8; 8 grey levels: 21.1; 16 grey levels 21.7).
As a result we can say that all results are extremely close together 20.6-22.1 %, so the di erences do not seem statistically signi cant. 5NN seems to be the best but this might also be linked to the fact that some classes have a very small population and 10NN would simply retrieve too many image of other classes to be competitive. 8 levels of grey and 8 directions of the Gabor lters seem to perform best, but the di erences are still very small.
In the future it is planned to train the system with the available training data using the
algorithm described in [
Unfortunately, our textual retrieval results submitted contained an indexation error, and so the textual results were almost random. We have not identi ed the error yet, but hope to have it found before the nal proceedings are printed. Thus the only textual run that we submitted had only a MAP of 0.0226, whereas the best textual retrieval systems were at 0.2084 (IPAL/I2R). Due to a limitation of resources, we were not able to submit relevance feedback runs, which is the discipline where GIFT usually is strongest. The best feedback system was OHSU with a map of 0.2116 for only textual retrieval.
The best visual system is I2R with a map of 0.1455. Our GIFT retrieval system was made available to participants and was widely used. Again, the basic GIFT system obtained the best results among the various combinations in feature space (map 0.0941), with only i2r having actually better results. The second indexation using 8 grey levels and eight directions of the Gabor lters performs slightly worse at 0.0872.
For mixed textual/visual retrieval, the best results were obtained by IPAL/I2R with map 0.2821. Our best result in this category is using 10% textual part and 90% visual part and obtains 0.0981. These results should be much better when using a properly indexed text base. The following results were obtained for other combinations: 20% visual: 0.0934, 25%: 0.0929, 33%: 0.0834, 50%: 0.044. When using eight grey levels and 8 Gabor directions: 10% visual: 0.0891, 20%: 0.084, 33%: 0.075, 50%: 0.0407. The results could lead to the assumption that visual retrieval is better than textual retrieval in our case, but this holds only true because of our indexation error. We will try to x the error and deliver proper results as soon as possible to have a correct comparison with the other groups.
A second combination technique that we applied used as a basis the results from textual retrieval and then added the visual retrieval results multiplied with a factor N = 2; 3; 4 to the rst 1000 results of textual retrieval. This strategy proved fruitful in 2004 the other way round by taking rst the visual results and then augmenting only the rst N=1000 results. The results for the main GIFT system were: 3 times visual: 0.0471, 4 times visual 0.0458, 2 times visual 0.0358. For the system with 8 grey levels, the respective results are: 3 times visual 0.0436, 4 times visual 0.0431, 2 times visual 0.0237. A reverse order of taking the visual results rst and then augment the textually similar would have led to better results in this case but when having correct results for text as well as for visual retrieval, this needs to be proven.
We cannot really deduct extremely much of our current submission as several errors prevented better results. 6
Although we did not have any resources for an optimised submission we still learned from the 2005 tasks that the GIFT system delivers a good baseline for image retrieval and that it is widely usable for a large number of tasks and di erent images.
More detailed results show that the ad{hoc task is hard for visual retrieval even with a more visually{friendly set of queries as the image set does not contain enough color information or clear objects, which is crucial for fully visual information retrieval.
The automatic annotation or classi cation task proved that our system delivers good results even without learning and shows that information retrieval can also be used well for document classi cation. When taking into account the available training data these results will surely improve signi cantly.
From the medical retrieval task not much can be deduced for now as we need to work on our textual indexation and retrieval to nd the error responsible for the mediocre results. Still, we can say that GIFT is well suited and among the best systems for general visual retrieval. It will need to be analysed which features were used by other systems, especially the few runs that performed better.
For next year we will de nitely have to take into account the available training data and we hope as well to use more complex algorithms for example to extract objects form the medical images and limit retrieval to theses objects. Another strong point of GIFT is the good relevance feedback and this can surely improve results signi cantly as well. Already the fact to have a similar databases for two years in a row would help as such large databases need a large time to be indexed and require human resources for optimisation as well. 7
Part of this research was supported by the Swiss National Science Foundation with grant 632066041. [16] Hemant D. Tagare, C. Ja e, and James Duncan. Medical image databases: A content{based retrieval approach. Journal of the American Medical Informatics Association, 4(3):184{198, 1997.