MedGIFT is a medical imaging research group of the Geneva University Hospitals and the University of Geneva, Switzerland. Since 2004, the medGIFT group has participated in the ImageCLEF benchmark each year, focusing mainly on the medical imaging tasks. For the medical image retrieval task, two existing retrieval engines were used: the GNU Image Finding Tool (GIFT) as image retrieval engine and Apache Lucene as textual retrieval engine. To improve the retrieval performance, automatic query expansion was used. In total, 13 runs were submitted as well for the image-based topics and the case-based topics. The baseline setup used for the past five years already obtained the best result among all our visual submissions. For the medical image annotation task, two approaches were tested. One approach is using GIFT for image retrieval and kNN (k-Nearest Neighbors) for the classification, which was already used for the past 5 years. The second approach used Scale-Invariant Feature Transform (SIFT) technology with a Support Vector Machine (SVM) classifier. Three runs were submitted in total, two with the GIFT-kNN-based approach and one using a combination of the SIFT-SVM-based approach and GIFT-kNN-based approach. For the medical image classification task, the GIFT-kNN-based approach gives stable results, although not in the quality of the best groups. The SIFT-SVMbased approach implementation did not achieve the expected better performance. We think that the SVM kernel may be a key factor that requires further optimization.
Image Retrieval, image classification, medical imaging
these competitions using the same technology as a baseline and trying to improve the performance
of this baseline over time using simple modifications. The GIFT3 (GNU Image Finding Tool,
[
For textual retrieval the Lucene4 system was employed in 2009. The full text of the articles was indexed and the query texts were not treated for the retrieval at all but used as they were given.
More information concerning the setup and the collections of the medical retrieval task of 2009
can be red in [
This section describes the basic technologies used for retrieval. The modifications to the base technologies will be detailed in the results section. 2.1
The text retrieval approach used in 2009 is based on the Apache Lucene retrieval engine. No specific terms such as MeSH (Medical Subject Headings) were used. Only one textual run was submitted. Five mixed visual/textual runs were submitted combining the textual retrieval with visual retrieval in various ways. The texts were indexed entirely from the html which contains the articles used by ImageCLEFmed task, removing all links and metadata keeping only the text. The query text was not modified, either. 2.2
GIFT has been used for the visual retrieval for the past five years. This tool is open source and can be used by other participants of ImageCLEF as well. The goal of using the standard GIFT is also to provide a baseline to facilitate the evaluation of other techniques. GIFT uses a simple partitioning of the image into fixed regions to obtain local features.
The performance obtained by GIFT remained limited over the past years. Various strategies
were applied in order to improve the results. Aspect–ratio was integrated as feature and simple
query expansion with the most similar images was added for the retrieval task [
In this section the results and technical details for the two medical tasks of ImageCLEF 2009 are detailed. 3.1
All the runs submitted for evaluation are based on visual retrieval using GIFT with 8 gray levels as this had consistently given best results in past years. Various strategies to improve the results were tried out. An simple improvement strategy is to separately query the images belonging to one topic as these images are quite often dissimilar. Then, the result can be combined in a late
fusion. Another approach is to supply images of other topics as negative feedback assuming that this can create more discriminative queries.
The following steps were thus performed: • for each topic, the similarities between all query image pairs were calculated; • based on the similarity between the images pairs, queries are treated either as a query of multiple images or as several separate queries; • for each topic, a random selection of 2 images from other topics were added as negative feedback; In total, 16 automatic runs were submitted. 10 runs were submitted for the image–based retrieval topics and 6 for the case–based topics. These runs can be divided into 2 textual, 10 visual, and 4 mixed runs. Run identifiers are composed of the strategies applied to the run. The following list explains the labels and their meaning.
• txt : textual; • vis: visual; • mix : mixed; • sep: separate queries for the various images of a topic; • withAR: aspect ratio added as feature; • withNegImg: query expansion with negative examples of other topics; • sum: simple sum to combine scores of several queries; • max : maximum score of an image is taken to combine several queries; • 0.x for mixed runs, 0.x is the weight for the visual retrieval and (1-0.x) for the textual retrieval; • EN the language used for textual retrieval is English; • BySim For the case–based topics the similarity score of images appearing in the case is used; • ByFreq For the case–based topics the frequency of images of a case in the first 1000 results is used.
Results for the medical retrieval task are shown in two parts. The results of 25 image–based topics are shown in Table 1 and those of the case–based topics (topics 26–30) are shown in Table 2. Mean average precision (MAP), binary preference (Bpref), and early precisions (P10, P30) were selected for our tables. The ImageCLEFmed 2009 retrieval task has an unbalanced number of visual, mixed, and textual runs. We submitted no manual or feedback runs at all. 3.1.1 Image–Based Topics As in past years, most of the submissions are textual or mixed and only few groups worked on improving the visual runs. Only 5 groups submitted a total of 15 visual runs. All our runs and the best runs in each category can be seen in Table 1 Our best visual run is the standard GIFT with 8 gray levels that was not submitted for the competition but has actually the best MAP. The other submitted runs give worse results but in general the performance of all runs is rather low with a MAP below 2%. Also the other measures such as early precision and number of relevant images found are much lower for the purely visual runs. None of the visual runs obtains extremely good results.
Run best textual HES-SO-VS txt EN best visual run medGIFT vis GIFT8 medGIFT vis sep max medGIFT vis sep sum withAR medGIFT vis sep sum medGIFT vis sep max withAR medGIFT vis sum withNegImg medGIFT vis max withNegImg best mixed run medGIFT mix 0.3withNegImg EN medGIFT mix 0.5 EN medGIFT mix 0.5withNegImg EN run type Textual Textual Visual Visual Visual Visual Visual Visual Visual Visual Mixed Mixed Mixed Mixed Run best run HES-SO-VS txt case medGIFT mix 0.5BySim EN medGIFT vis maxBySim withAR medGIFT vis sumBySim withAR medGIFT vis maxByFreq withAR medGIFT vis sumByFreq withAR
A total of 52 textual runs were submitted. We only submitted a single textual run using an extremely simple strategy of Lucene with no pre–treatment of neither queries nor documents. The results looking at MAP are not as good as other approaches but early precision is fairly good and looking at the simplicity of the approach this is rather surprising. The performance is above the average of the textual runs.
In total, 25 mixed runs were submitted. We submitted three mixed runs that have a large variety of combination strategies. It can be seen that the combination strategy has a very strong influence on the results. Our best mixed run is slightly better in early precision than our text runs but worse in all other measures. In early precision our mixed run can rival the best runs submitted and reaches the second overall position. 3.1.2
Case–based topics In Table 2, all our runs for the case–based topics are shown and the best run of the competition for comparison. Again it can be seen that the pure text retrieval has by far the best results, despite the simplicity of our approach. Our mixed approach was worse in all categories. The purely visual runs have a fairly low MAP similar to the image–based topics and even the early precision was rather low.
In both the viaul and mixed categories our system perfromed best but this is also partly due to the low participation. 3.2
For the medical image annotation task 6 groups submitted in total 18 runs. The medGIFT group
submitted 3 runs. Two runs used the same strategy as in the past 2 years using GIFT as a
basic retrieval technology, then adding aspect ratio to the similarity score. For classification the
5 nearest neighbors (5NN) was used. More details on the strategy can also be found in [
Textual runs significantly out–performed visual runs, which was coherent with past experiences. Even using the basic Lucene with no pretreatment obtained fairly good results compared to the much more complex approaches of other groups. Since 2004 GIFT with 8 gray levels has been used as a baseline. Generally, the result obtained were below the average MAP of purely visual runs. The strategy to perform separate queries for each query image of a topic increased early precision but decreased MAP: Using aspect ration for the retrieval has a slight influence on early precision but is otherwise not better. Combinations of text and visual result can obtain a higher early precision.
For the image annotation task a new SVM–based strategy was employed but did not increase results at all. The basic GIFT strategy still had our best results but not comparable with the results of the best groups. The combination of using GIFT features and SIFT did not improve results either. Using a small number of results for the combination does not seem appropriate. 5
The paper summarized the participation of the medGIFT group in ImageCLEF2009. The medical image retrieval and medical image annotation tasks were addressed. Results show that textual retrieval techniques outperfrom visual techniques but that visual retrieval can be an addition to improve early precision of retrieval.
This study was partially supported by the Swiss National Science Foundation (Grant 200020– 118638/1), the HES SO with the BeMeVIS project, and the European Union in the 6th Framework Program through the KnowARC project (Grant IST 032691).