This article presents the participation of the medGIFT group in ImageCLEFmed 2013. Since 2004, the group has participated in the medical image retrieval tasks of ImageCLEF each year. There are four types of tasks for ImageCLEFmed 2013: modality classi cation, image{ based retrieval, case{based retrieval and a new task on compound gure separation. The medGIFT group participated in all four tasks. MedGIFT is developing a system named ParaDISE (Parallel Distributed Image Search Engine), which is the successor of GIFT (GNU Image Finding Tool). The alpha version of ParaDISE was used to run the experiments in the competition. The focus was on the use of multiple features in combinations with novel strategies, i.e, compound gure separation for modality classi cation or modality ltering for ad{hoc image and case{ based retrieval.
ImageCLEF [
This article describes the participation of the medGIFT2 research group in
ImageCLEFmed 2013. The medGIFT group has participated in ImageCLEFmed
since 2004 and is currently developing ParaDISE (Parallel Distributed Image
Search Engine), which is the successor of GIFT3 (GNU Image Finding Tool)
that was used for many years as the baseline in ImageCLEF. As in 2012 [
{ modality filtering was used for image– and case–based retrieval; { compound figure detection was performed.
A combination of these strategies was used for the run submissions. For the modality classification task, the best medGIFT submission achieved a classification accuracy of 69.63% and was ranked in the fourth position. In its first year, the compound figure separation task only attracted three groups. MedGIFT achieved the best results, which could be expected since the data had been used by the group and the ground truth was created ahead of time. In the ad–hod image and case–based retrieval, the medGIFT group was ranked second for the visual and mixed submissions. The text runs submitted acquired average results with a simple Lucene baseline.
The rest of the paper is organized as following. In Section 2, the datasets and the techniques used are described. The runs submitted to the ImageCLEFmed 2013 benchmark are described and evaluated in Section 3. Finally, conclusions are presented in Section 4. 2
This section describes the basic techniques used in ImageCLEFmed 2013 by the
medGIFT group. More detail on the setup of ImageCLEFmed 2013 can be found
in [
Ten runs (three textual, three visual and four mixed) were submitted to the image–based retrieval task, five runs (one textual, three visual and one mixed) to the case–based retrieval task, ten runs (one textual, four visual and five mixed) to the modality classification task and two visual runs to the compound figure separation task.
The database provided for ImageCLEFmed 2013 [
For text retrieval, the Apache Lucene framework was used. The built–in text analyser class named EnglishAnalyzer was used to apply lowercasing, stopword removal and stemming in their standard settings. The full text and the captions were indexed separately as the text to use depends on the exact goal of the search.
For modality classification, a step was included where a classifier using the
Radlex ontology described in [
images classified as containing radiology modalities. A mistake prevented this from being used in the submitted runs. Post–submission experiments show that the inclusion of semantics can improve the results. 2.3
In 2012, the bag–of–visual–words (BoVW) features using local descriptors were
a focus using the scale–invariant feature transform (SIFT) and the CIELab color
descriptor (namely bag–of colors, BoC [
These features were extracted from the test set (see Table 1). Then, the fusion of
the best features was performed to obtain a good feature set (see Table 2). Due
to time and resource limitations not all the possible combinations were tested
and only the features performing well alone were combined in a simple linear
way.
As seen in the ImageCLEFmed 2012 data set [
For this reason, the compound figure separation application detailed in [
This section details the techniques that were used to produce the runs for ImageCLEFmed 2013 and then evaluates the runs. 3.1
This year, medGIFT submitted 10 runs using the techniques described in Section 2. Two baseline runs were submitted: a visual and a textual. The visual baseline uses the features described in Section 2.3. The textual baseline is described in Section 2.2. The remaining runs are a combination of the baselines, the compound figure detection (Section 2.4), and the training set expansion (Section 2.5). The run IDs correspond to: { Run1–medgift2013 mc 5f : this run uses only visual information as a baseline. { Run2–medgift2013 mc 5f separate: this run first classifies the images as compound or non–compound (see Section 2.4). Then, the non–compound images are classified using only visual information. { Run3–medgift2013 mc 5f exp k8: this run uses the same techniques as
Run1 but over an expanded training set (see Section 2.5). { Run4–medgift2013 mc 5f exp separate k21: this run uses the same techniques as Run2 but over an expanded training set. { Run5–medgift2013 mc text k8: the images are classified using the textual information from the captions including semantic information. { Run6–medgift2013 mc mixed k8: this run uses visual and textual information combined. { Run7–medgift2013 mc mixed exp k21: this run uses the same techniques as Run6 but over an expanded training set. { Run8–mc mixed sem k8: this run uses the same techniques as Run6 including semantic information (see Section 2.2). { Run9–medgift2013 mc mixed exp sem k21: for this run visual and textual features are extracted as well as the semantic information. An expanded training set is used. { Run10–medgift2013 mc mixed exp sep sem k21: for this run the images are first classified as compound or non–compound. Secondly, the non– compound images are as in Run9. 3.2
This year marked the introduction of a new subtask in ImageCLEF, the
separation of compound figures. MedGIFT submitted two compound figure separation
runs. Run11 simply serves as a point of reference, since it was also used in [
{ Run11–... HESSO CFS: this run uses the MATLAB figure separation script mentioned above, which was also used as a first step in the manual generation of the ground truth. { Run12–... HESSO ... SCALE50 STANDARD: run that uses a bidimensional version of the region detector mentioned above, at a scale of 50 pixels. 3.3
This year the effect of modality filtering on the retrieval quality was investigated.
For this purpose, the full image dataset was classified using the method of the
best mixed run of the 2012 modality classification task [
Three approaches of modality filtering were tested. In the first one named ”exact” only the modality detected by the KNN classifier for each query image of the topic was put into the query modality set. The second named ”close” puts all the modalities detected by the KNN classifier of any query image into the topic. The third one named ”prefix”, is similar to the first but the broadest modality (diagnostic, general, compound) was used instead of the exact modality for boosting the image score in the retrieved set. Due to the limited number of submissions the ”exact” approach was not submitted as it had a low performance in preliminary tests on the ImageCLEF 2012 collection.
Multiple features were used for visually indexing the dataset (see Section 2.3).
Apart from the features used last year (SIFT–based BoVW and BoC features),
more features were added for the image retrieval task. The CEDD and FCTH
descriptors were also used, due to their good performance on ImageCLEF 2012
challenge. BoVW and BoC features that contain spatial information were used.
For BoVW a spatial pyramid matching (SPM) approach was used [
For the text runs, a late fusion of full text search with caption search was followed. Moreover, for each of these searches three different queries were fused for each topic. The first one queried the topic query for exact matching. The second connected the query terms with ’AND’, while the third used ’OR’ as a connector.
In the mixed runs, two different approaches were submitted. In the first
approach, a linear weighted late fusion was used. The weights were tuned using
the ImageCLEF 2012 benchmark. The second one used a late fusion (combMNZ)
of the visual and text runs to rerank the images in the result set that was
retrieved by the text run, similar to [
No modality filtering is used. 3.4
For the case–based retrieval task, similar techniques to the ones in Image–based retrieval task were submitted. For the mixed run, the visual features and the captions were used for retrieving a list of images that was then mapped to a list of associated articles. The new list was then fused with the article list returned by the baseline full text search. Below the characteristics of the 5 submitted runs are described: { Run23–medgift visual no lter casebased: visual run that uses the 6 features (BoVW, BoC, SPM BoVW, Grid BoC, CEDD and FCTH) and combMNZ fusion. No modality filtering is used. { Run24–medgift visual close casebased: same as Run23 but the ”close” modality filtering approach is used. { Run25–medgift visual pre x casebased: same as Run23 but the ”prefix” modality filtering approach is used. { Run26–HES-SO-VS FULLTEXT LUCENE ENGLISH: baseline text run using full text search. No modality filtering is used. { Run27–medgift mixed no lter casebased: mixed run using linear weighted fusion for Run23 and Run25. Weights were visual: 0.2, text: 0.8. No modality filtering is used. 3.5
This year medGIFT was the fourth group in the three types of run submissions. Table 3 shows the results achieved by the submitted runs. The baselines runs achieve the best results showing that the other techniques applied were not improving the accuracy. The use of multiple features indicate an improvement on the accuracy, achieving a better performance than last year’s “easier” test data set. More work is necessary in the compound figure detection. In post– submission experiments it was found that the compound figure detection step did not achieve a better compound/non–compound separation accuracy ( 70%) than the medGIFT baseline classifier ( 79%). This caused a worse performance of the runs containing this step. Additionally, after the success of the expansion of the database in 2012, the results show that it is deteriorating the approach. Probably it is due to the fact that only textual information was used to expand the database. Finally, the semantic step did not modify the results due to the bug in the code. The two runs submitted by medGIFT were the best and the worst in the list, respectively, with only four runs being submitted. Table 4 shows the results achieved by the runs submitted by medGIFT. Predictably, the run that was used in previous work on the same data gave the best results, whereas the run using the 2D region detector, which is not optimized for compound figure separation, yielded mediocre results. The results of the medGIFT runs are presented in Table 5. There are situations when ”nofilter” or ”close” filter perform better depending on the type of run (visual, textual or combined). Second best results were achieved in the visual runs using the baseline run (”nofilter”). On mixed techniques medGIFT was the second best group and on the textual runs medGIFT was the fourth group when using ”close” filtering. In several queries the average precision was strongly improved by the modality filtering, while other queries had much lower performance, for example when the modality was not correctly detected. This indicates that more accurate modality classification should further improve retrieval performance. Results also show that the use of reranking in place of weighting for the fusion achieves better results. This is important for large–scale retrieval as visual search on the whole dataset is computationally costly. By having text retrieval as a first step, the image–based retrieval search subspace is significantly (magnitude of two orders) smaller. In 2013, medGIFT submitted five runs in the case–retrieval task. The results are listed in Table 6. This year medGIFT obtained the second position in both visual and mixed runs. The results do not show significant differences with the use or without modality filtering. 4
This article describes the methods and results of the the medGIFT group for the ImageCLEF 2013 medical tasks. Ten runs were submitted each for the ad–hoc image retrieval and the modality classification tasks, five runs for the case– based retrieval task and two runs in the new compound figure separation task. In ImageCLEFmed 2013 medGIFT worked on runs based on multiple feature combinations. Several strategies were explored and not all techniques obtained improvements. Low results were obtained applying a database expansion strategy. This performance may be due to the use of only textual information for the indexing. Run ID Run type MAP GM{MAP bpref P10 P30 Best ImageCLEF run Visual 0.0281 0.0009 0.0335 0.0429 0.0238 medgift visual close casebased Visual 0.0029 0.0001 0.0036 0.0086 0.0076 medgift visual no lter casebased Visual 0.0029 0.0001 0.0035 0.0086 0.0067 medgift visual pre x casebased Visual 0.0029 0.0001 0.0036 0.0086 0.0067 Best ImageCLEF run Textual 0.2429 0.1163 0.2417 0.2657 0.1981 HES-SO-VS FULLTEXT LUCENE Textual 0.1791 0.1107 0.1630 0.2143 0.1581 Best ImageCLEF run Mixed 0.1608 0.0779 0.1426 0.18 0.1257 medgift mixed no lter casebased Mixed 0.1467 0.0883 0.1318 0.1971 0.1457
Among the 2013 results, it is possible to observe that the compound figure separation is still a field to investigate in order to improve modality classification accuracy.
Future work of the medGIFT group aims at optimizing the use of the modality classification to help the retrieval. A correct incorporation of the semantic resources into the modality classification pipeline will be explored. MedGIFT also plan to further investigate the compound figure separation task. 5
The research leading to these results has received funding from the European Union’s Seventh Framework Programme under grant agreement 257528 (KHRESMOI) and 258191 (PROMISE).