This article presents the participation of the medGIFT group in ImageCLEFmed 2012. Since 2004, the group has participated in the medical image retrieval tasks of ImageCLEF each year. There are three types of tasks for ImageCLEFmed 2012: modality classification, imagebased retrieval and case-based retrieval. The medGIFT group participated in all three tasks. MedGIFT is developing a system named ParaDISE (Parallel Distributed Image Search Engine), which is the successor of GIFT (GNU Image Finding Tool). The alpha version of ParaDISE was used to run most of the experiments in the competition. Results show that our approach using Bag-of-Visual-Words (BoVW), Bag-of-Colors (BoC) and Lucene for the image captions is the best run for mixed modality classification. The same approach is also the best for the image retrieval task in terms of bpref. In the case-based retrieval task, the Lucene baseline is the best run in terms of mean average precision (MAP). We were the only group presenting mixed and visual runs in these tasks.
ImageCLEF is the cross–language image retrieval track1 of the Cross Language
Evaluation Forum (CLEF). ImageCLEFmed is part of ImageCLEF focusing on
medical images [
The Bag–of–Visual–Words (BOVW) and Bag–of–Colors (BOC) approaches
were used for visual retrieval and the textual baseline is based on the open source
Lucene4 system. In ImageCLEFmed 2011, the BoVW approach was applied
using the Scale Invariant Feature Transform (SIFT) [
BoC with BoVW and textual descriptors in order to yield better results. Second, we use training set expansion strategies since for some of the image categories only very few annotated examples were available. This significantly improves classification performance.
The widely used BoVW [
The BoC representation is analogous to the BoVW representation and to the Bag–of–Words representation of text documents. As the two methods are quite complementary and as the representations as histograms are very similar, the two approaches can easily be fused for medical image classification.
In Section 2, we describe the datasets and the techniques used. We evaluate the runs submitted to the ImageCLEFmed 2012 benchmark in Section 3. Finally, conclusions are presented in Section 4. 2
This section describes the basic techniques used in ImageCLEFmed 2012 by the
medGIFT group. For the the majority of the runs, the alpha version of ParaDISE
(Parallel Distributed Image Search Engine) developed by the medGIFT group is
used. This image search engine is built on top of Hadoop [
Image Collection We used the database provided by ImageCLEFmed 2012. The database contains over 300,000 images of 75’000 articles of the biomedical open access literature. This is a subset from the PubMed Central6 database containing over one million
images. This set of articles contains all articles in PubMed that are open access
but the exact copyright for redistribution varies among the journals. A more
detailed description of the ImageCLEFmed 2012 setup is given in [
Textual Techniques For text retrieval the standard settings of the Apache Lucene text retrieval system are used. The documents containing the journal texts are cleaned off their XML elements and only the remaining text is used.
Two indexations were done for ImageCLEF 2012: (1) the full text of all articles was indexed on an article basis, and (2) the captions were indexed on a figure basis. In the past it was shown that for case–based retrieval an indexation of the full text had best results whereas for image–based retrieval the caption text delivered much better results. 2.3
Visual Techniques The baseline for the visual description of the image is the Bag–of–Visual–Words (BoVW) approach. In this approach, local SIFT descriptors are extracted from each image. Then, the descriptors of the image are quantized, assigning each one to its nearest neighbour from a fixed set of local descriptors, called “visual vocabulary”. The image is then represented by a histogram of the frequency of the “visual words“. Similarity between two images can be quantified using a distance metric to measure the distance between the two histograms.
In our runs, the SIFT implementation in the fiji7 image processing
package was used for the extraction of the local descriptors as in our participation
in ImageCLEFmed 2011. In order to create the visual vocabulary, our
implementation of the density–based clustering algorithm DENCLUE [
Based on the BoVW approach, the Bag–of–Colors (BoC) [
CIELab is constructed by finding the most frequently occurring colors in each
image of a sub set of the collection. The modality classification training set was
used for the creation of the color vocabulary. The CIE (International Commission
on Illumination) 1976 L*a*b (CIELab) space was used, being a perceptually
uniform color space. CIELab is a space defined by L for luminance and a, b for
the color–opponent dimensions for chrominance [
np np c¯i = ∑ ∑ gj (pk) 8i 2 f1; : : : ; kcg
k=1 j=1
where
gj (p) =
{ 1 if d"(p; cj )
0 otherwise
d"(p; cl) 8l 2 f1; : : : ; kcg
(1)
From experiments on the modality classification task of ImageCLEFmed 2012,
kc = 100 and kvw = 238 were chosen as the sizes of the color and visual
vocabulary, respectively. For the comparison of the images the histogram
intersection [
Fusion Techniques In image retrieval and classification tasks, often different features, systems and results can be combined to deliver improved results. Moreover, multiple query images can describe in more detail the visual characteristics to be retrieved. In these cases a fusion strategy needs to be used. Two main fusion strategies exist, early and late fusion. In the early fusion, the vectors of the features or the systems are merged into a single vector. For the early fusion of multiple positive and/or negative query images, Rocchio’s algorithm can be used: qm = qo + 1
∑ jDrj dj2Dr dj
1 jDnrj dj2Dnr ∑ dj (2) where ; and are weights, qm is the modified query, qo is the original query, Dr is the set of relevant images and Dnr is the set of non–relevant images. In our scenario there is a lack of non–relevant images, so only the second term of the right part of the equation is used. This algorithm can be applied only for vectors of the same feature spaces so it is not applicable for the fusion of different visual features or retrieval systems, in general.
In the late fusion, the retrieval results of the features or the systems are
fused. Two main categories of late fusion techniques exist, the score–based and
rank–based methods. In 2009, the ImageCLEF@ICPR fusion task was organized
to compare late fusion techniques using the best ImageCLEFmed visual and
textual results [
ScombSUM(i) =
Nk ∑ Sk(i) k=1
ScombMNZ(i) = F (i) ScombSUM(i) where F (i) is the frequency of image i being returned by one input system with a non–zero score, and S(i) is the score assigned to image i.
In general, rank–based fusion (RRF) worked better than score–based fusion. The reciprocal rank fusion [20] is a simple fusion method based on ranks(5). (3) (4) (5) RRF score(d 2 D) = ∑ r2R
1 k + r(d) where, D is the set of documents retrieved, R is the set of rankings of the documents and k = 60. 3
This section details the techniques that were used to produce the runs for ImageCLEFmed 2012 and then evaluates the runs. 3.1
Image and Case Retrieval Techniques Two strategies were compared for the fusion of the multiple queries, early fusion and late fusion (see Section 2.4). For the fusion of visual features and the fusion of visual and textual systems, late fusion was applied. Score–based and rank– based fusion techniques were compared. To summarize, fusion was used in three cases: { fusing multiple visual features to produce visual runs; { fusing textual and visual runs to produce mixed runs; { fusing vectors (early fusion) of query images which belong to the same topic or their results (late fusion).
In the image retrieval task, as the fulltext search retrieved articles instead of images, an article–to–image mapping was used. If an image was contained in multiple articles, only the article with the highest score was taken into account giving its score to the image. If multiple images were contained in the same article, all the images received the common article’s score.
The same strategy was applied in the case–based task, because the image search retrieved images instead of articles. The article received the score of the best scored image that it contained. If multiple articles contained the same image they all received the common image’s score.
Table 1 contains a summary of the techniques used, while Tables 2 and 3
show the details of the submitted runs.
Driven by the good performance of [
Two methods of expanding the training set were used. In the first expansion technique, s images taken randomly of each class were used as queries. As in the original training set the number of images per class varies from 5 to 50, by choosing s = 5; l = 20 we can theoretically obtain a relatively balanced training set (105–150 per class) of 4,100 images. In the second, all the training images were queried, resulting in a larger non–balanced training set. E. g. for l = 20 an expanded training set of 21,000 images can be obtained theoretically. In practice, smaller sizes were obtained, mainly because of two reasons: retrieved images that are already contained in the training set were discarded and images retrieved multiple times by query images of different classes were discarded as well.
For a run to qualify as visual, we considered that the expanded training set used in this run needs to be created only by visual means. This means that the queries on the full data set used only visual features for the retrieval. Similarly, this was repeated using mixed (visual and textual) queries for the mixed runs. This resulted in a final number of 5 training sets (2 balanced, 2 non–balanced and the original training set).
A k nn classifier using weighted voting was used to classify the test images.
For the choice of the classifier parameters the results of [
Table 4 gives the details of the submitted runs. 3.3
Image Retrieval Evaluation Table 5 displays the results of the medGIFT runs for the image retrieval task. ir9, a mixed approach, achieved the highest MAP, GM–MAP and bpref among all our submitted runs in the image–based retrieval task. This result highlights the potential benefits of combining textual and visual features.
For the visual runs, the baseline of BoVW, ir1 and ir2, did not demonstrate good results. However, when BoVW is fused with BoC the results improve significantly. Furthermore, when using RRF for the fusion of techniques the MAP decreases, which contradicts our hypothesis.
Our best results in early precision (P10 and P30) on the other hand were obtained with a text retrieval approach, ir8. This is in contradiction with past results where the best overall results were often textual whereas early precision was often better with combined or visual approaches. Table 6 shows the results of the medGIFT runs for the case based retrieval task. In this task, cr8 achieved the highest MAP (0,169) among all submitted runs. It demonstrates that a Lucene baseline still obtains very good results with relatively low effort. MedGIFT was the only lab that submitted purely visual and mixed runs for the case–based task. Although the results of our visual runs are lower than textual runs, cr9, a mixed approach, performs better than the average of all submitted runs in this task.
As all of the best results are from text retrieval runs it becomes clear that visual techniques need to be used in different ways than was the case for obtaining good results. Most likely a matching of image occurrences in articles in a more complex way would be necessary for obtaining good results for the case–based tasks using visual data. 3.5
Modality Classi cation Evaluation Finally, Table 7 presents the classification accuracy of the submitted medGIFT runs for the modality classification task. The runs c8, mc6, mc7 achieved the three best accuracies in the mixed run category. The visual runs achieved an average performance with the inclusion of BoC as a global descriptor to improving the classification accuracy. It can be observed that the runs mc4, mc8 using the non–balanced expanded training sets and k = 14 are outperforming the runs mc6, mc2 that use the original training set. These runs also perform better than the runs mc3, mc7 that use k = 7, confirming our hypothesis that using a larger k can improve results. Moreover, in experiments not submitted as official runs using larger values of k leads to a better performance, with the mixed run reaching an accuracy of 68.5% using the triple value k = 21. 4
This article describes the methods and results of the the medGIFT group for the ImageCLEF 2012 medical tasks. We submitted ten runs each for the ad– hoc image retrieval and the modality classification tasks and nine runs for the case–based retrieval task.
In ImageCLEFmed 2012 we concentrated on fusion methods of the visual and textual features and training set expansion strategies. We included the BoC approach, which significantly improves classification performance.
In the image–based retrieval task, our strategy leads to limited results obtained by our submitted runs. However, we still need to do a more detailed analysis to try to understand what is hurting performance in these runs. Our Lucene baseline achieves the best MAP for the case–based retrieval task. Finally, in the modality classification task we submitted the three runs with the best accuracies in the mixed run category and with only one group delivering better results with a visual approach.
Future work will to go beyond in the work of fusion methods, as well as further research in new visual features. Visual information can be used in better ways, for example to classify images by modality and if modality names appear in a query then filtering by this modality. If the same modalities occur in a query and an article this can also give evidence for a higher relevance. 5
The research leading to these results has received funding from the European Union’s Seventh Framework Programme under grant agreement 257528 (KHRESMOI), 249008 (Chorus+) and 258191 (Promise). 19. Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Text REtrieval
Conference. (1993) 243–252 20. Cormack, G.V., Clarke, C.L.A., Bu¨ttcher, S.: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA, ACM (2009) 758–759