INTRODUCTION

Searching for images is a daily task for many medical professionals, especially in image-oriented fields such as radiology. However, the huge amount of visual data in hospitals and the medical literature is not always easily accessible and physicians have generally little time for information search as they are charged with many tasks. Therefore, medical image retrieval systems need to return information adjusted to the knowledge level and expertise of the user in a quick and precise fashion. A well known technique trying to improve search results by user interaction is relevance feedback [13]. Relevance feedback allows the user to mark results returned in a previous search step as relevant or irrelevant to refine the initial query. The concept behind relevance feedback is that though user may have difficulties in formulating a precise query for a specific task, they generally see quickly whether a returned result is relevant to the information need or not. This technique found use in image retrieval particularly with the emerge of content-based image retrieval (CBIR) systems [18,19,20]. Following the CBIR mentality, the visual content of the marked results is used to refine the initial image query. With the result images represented as a grid of thumbnails, relevance feedback can be applied quickly to speed up the search iterations and refine results. Recent user-tests with radiologists on a medical image search system also showed that this method is intuitive and straightforward to learn [7].

Depending on whether the user manually provides the feedback to the system (e.g. by marking results) or the system obtains this information automatically (e.g. by log analysis) relevance feedback can be categorized as explicit or implicit. Moreover, the information obtained by relevance feedback can be used to affect the general behaviour of the system (long-term learning). In [11] a market basket analysis algorithm is applied in image retrieval of long-term learning. A recent review of short-term and long-term learning relevance feedback techniques in CBIR can be found in [6]. An extensive survey of relevance feedback in text-based retrieval systems is presented in [15] and for CBIR in [14].

In the medical informatics field, [1] applies CBIR with relevance feedback on mammography retrieval. In [12], an image retrieval framework using relevance feedback is evaluated on a dataset of 5000 medical images that uses support vector machines to compute the refined queries.

In this paper we evaluate different explicit, short-term relevance feedback techniques using visual content or text for medical image retrieval. We propose a technique that combines visual and text-based relevance feedback and show that it achieves a competitive performance to the state-ofthe-art approaches.

METHODS

Rocchio algorithm

One of the most well known relevance feedback techniques is Rocchio's algorithm [13]. Its mathematical definition is given below:

qm = α qo + β 1 |Dr| d j ∈Dr dj − γ 1 |Dnr| d j ∈Dnr dj (1)

where qm is the modified query, qo is the original query, Dr is the set of relevant images, Dnr is the set of non-relevant images and α, β and γ are weights. Typical values for the weights are α = 1, β = 0.8 and γ = 0.2. Rocchio's algorithm is typically used in vector models and also for CBIR. Intuitively, the original query vector is moved towards the relevant vectors and away from the irrelevant ones. By giving a weight to the positive and negative parts a problem of CBIR can be avoided that when more negative than positive feedback exists that also many relevant images disappear from the results set.

Late fusion

Another technique that showed potential in image retrieval [5] is late fusion. Late fusion [2] is used in information retrieval to combine result lists. It can be applied for fusing multiple features, multiple queries and in multi-modal techniques. The concept behind this method is to merge the result lists into a single list while boosting common occurrences using a fusion rule.

For example, the fusion rule of the score-based late fusion method CombMNZ [17] is defined as:

S combMNZ (i) = F (i) * S combSUM (i)( 2 )

where F (i) is the number of times an image i is present in retrieved lists with a non-zero score, and S(i) is the score assigned to image i. CombSUM is given by

S combSUM (i) = N j j=1 Sj(i)( 3 )

where Sj(i) is the score assigned to image i in retrieved list j.

Multi-modal relevance feedback

Most of the techniques use vectors either from the text or the visual models. However, it has been shown that approaches that use both text and visual information can outperform single-modal ones in image retrieval. We propose the use of multi-modal information for relevance feedback to enhance the retrieval performance. This is, to the extend of our knowledge, the first time that such a technique is proposed in image retrieval. As late fusion is applied on result lists, it is straightforward to use for combining results from visual and text queries.

Experimental setup

For evaluating the relevance feedback techniques the following experimental setup was followed: The n search iterations are initiated with a text query in iteration 0. The relevant results from the top k results of iteration i were used in the relevance feedback formula of the iteration i + 1 for i = 0...n − 2.

The image dataset, topics and ground truth of Image-CLEF 2012 medical image retrieval task [9] were used in this evaluation. The dataset contains more than 300'000 images from the medical open access literature.

The image captions were accessed by the text-based runs and indexed with the Lucene 1 text search engine. Vector space model was used along with tokenization, stopword removal, stemming and Inverse document frequency-Term frequency weighting. The Bag-of-visual-words model described in [3] and the bag-of-colors model appearing in [4] 1 http://lucene.apache.org/ were used for the visual modelling of the images. In multimodal runs, the fusion of the visual and text information is performed only for the text 1000 top results as in the evaluation of ImageCLEF only the top 1000 documents are taken into account in any case. Five techniques were evaluated in this study:

1. text: text-based RF using vector space model. Word stemming, tokenization and stopword removal is performed in both text and multi-modal runs.

2. visual rocchio: visual RF using Rocchio to fuse the relevant image vectors and CombMNZ fusion to fuse the original query's results with the visual ones.

3. visual lf : visual RF using late fusion (and the CombMNZ fusion rule) to fuse the relevant image results and the original query results with the visual ones.

4. mixed rocchio: multimodal RF using Rocchio to fuse the relevant image vectors and CombMNZ fusion to fuse the original query results with the relevant caption results and relevant visual results.

5. mixed lf : multimodal RF using late fusion (and the CombMNZ fusion rule) to fuse the relevant image results and the original query results with the captions' results and relevant visual results.

RESULTS

The evaluation of the five techniques was performed for k = 5, 20, 50, 100 and n = 5. Results of the mean average precision (mAP) of each technique per iteration are shown in Figures 1, 2, 3, 4.

Table 1 gives the best mAP scores of each run. The numbers in parentheses are the number of the iteration when this score was achieved. For scores that were the same in multiple iterations of the same run, the iteration closer to the first is used.

DISCUSSION

All of the evaluated techniques improve retrieval after the initial search iteration. This demonstrates the potential of relevance feedback for refining medical image search queries.

Relevance feedback using only visual appearance models, even though improving the retrieval performance after the first iteration, performed worse than the text-based runs in most cases. Visual features still suffer from the semantic gap between the expressiveness of visual features and our human interpretation. Still, this shows their usefulness in image datasets where no or little text meta-data are available. Moreover, when combined with the text-information in the proposed method, they improve the text-only baseline.

The proposed multi-modal runs provide the best results in all the cases except for case k = 5. Surprisingly, the visual runs perform slightly better than the text and the multi-modal approaches for this case. However, assuming independent and normal distributed average precision values the significance tests show that the difference is not statistically significant.

We consider the case k = 20 as the most realistic scenario since users do not often inspect more than 2 pages of results. Especially for grid-like result interface views, where each page can contain 20 to 50 results, we consider k = 20 more realistic than k = 5. In this case the proposed methods achieve the best performance with 0.2606 and 0.2635 respectively. Again, the significance tests do not find any significance difference between the three best approaches. However, applying different fusion rules for combining visual and text information (such as linear-weighting) could further improve the results of the mixed approaches.

It can be noted that as the k increases, the performance improvement also increases, highlighting the added value of relevance feedback. Larger values of k were not explored as this scenarios were judged as unrealistic.

In the visual runs using Rocchio for combining the visual queries is performing worse than late fusion. This comes in accordance with the findings in [3]. The reason behind this could be that the large visual diversity of relevant images in medicine and the curse of dimensionality cause the modified vector to behave as an outlier in the high dimensional visual feature space. In the mixed runs the difference between the two methods is not statistically significant with Rocchio performing slightly better than the late fusion.

Irrelevant results were ignored, as they often have little or no impact on the retrieval performance [10,16]. More importantly, the ground truth of the dataset used contains a much larger portion of annotated irrelevant results than relevant ones. This was considered to potentially simulate an unrealistic scenario, as users do not usually mark many results as negative examples. Having too many negative examples could also cause the modified vector to follow an outlier behaviour. Preliminary results confirmed this hypothesis, where the use of negative results for relevance feedback can decrease performance after the first iteration.

It should be noted that this is an automated relevance feedback experiment of positive only feedback and that in selective relevance feedback situations the retrieval performance is expected to perform even better. A larger number of steps could be investigated but this might be unrealistic, given the fact that physicians have little time and stop after a few minutes of search [8]. Often users will only test a few steps of relevance feedback at the most.

CONCLUSIONS

This paper proposes the use of multi-modal information when applying relevance feedback to medical image retrieval. An experiment was set up to simulate the relevance feedback of a user on a number of medicine-related topics from Im-ageCLEF 2012.

In general, all the techniques evaluated in this study improve the performance, which shows the added value of rele-vance feedback. Text-based relevance feedback showed consistently good results. Visual-based techniques showed competitive performance for small shortlist sizes, underperforming in the rest of the cases. The proposed multi-modal approaches showed promising results slightly outperforming the text-based one but without statistical significance.

More fusion techniques are going to be evaluated in the future. Comparison to manual query refinement by users is considered in future plans, to assess relevance feedback as a concept in medical image retrieval. The addition of semantic search is also of interest, to take advantage of the structured knowledge of the medical ontologies such as RadLex (Radiology Lexicon) and MeSH (Medical Subject Headings).