We describe our experiments for the Image CLEF medical retrieval task. Our e orts were focused on the initial visual search. A content based approach was followed using global features that would perform well with the medical image collection. We used structure, texture, localisation and colour features that have been proven by previous experiments. The retrieval results showed that this simple approach was successful.
The initial visual search task was very straightforward, with 26 single image queries. With no training data it was not possible to use any learning classi ers. We therefore used a simple system to tackle the retrieval task, using the features described in the next section. The following steps were carried out: 1. Features were generated for the test collection and query images; 2. For each feature the Manhattan distance between the query images and the test set was calculated; 3. The set of distances from each feature was normalised by dividing by their median. This ensured that each feature would have an equal weighting; 4. The distances for each query were summed over all features. This gave the overall distance from each query image to the test set. These were then sorted to produce a ranked list of retrieval results.
The initial step in our work was to look at the collection and determine its characteristics. As a relatively speci c domain it displayed a large degree of homogeneity. We realised this could be exploited by choosing features that would di erentiate the image types.
The collection contained a large number of monochrome images, such as x-rays and CT scans, with very speci c layout. The patients are positioned very precisely to show the area under investigation at the centre of the image. The layout can be used to indicate both modality and anatomic region. For this reason a localisation feature, thumbnail, was used to detect images with similar layouts. Within the modalities the images could be discriminated by structure and texture. We therefore chose to use a convolution feature to discriminate structure and two texture features, co-occurrence matrices and Gabor lters. The two texture features were applied to nonoverlapping image tiles. This gives an additional structural element to the feature. Finally, for the relatively small number of colour images we deployed a colour structure descriptor. 3.1
This is perhaps the simplest feature in our feature set, yet it is highly e ective in detecting images with a near identical layout. Each image is converted to grey scale and then scaled down to a thumbnail of xed size. For these experiments we used 40 30 pixels. The pixel values of this new image then make up the feature vector. 3.2
This feature is based on Tieu and Viola's method [
The feature generation process starts with a set of 25 primitive features that are applied to the grey level image. This generates 25 feature maps. Each of these is recti ed and down-sampled before being ltered again by each of the 25 primitive lters. This gives 625 feature maps. The second stage of the process 'discovers' arrangements of features in the previous levels. The values of each feature map are summed to give a single number. These are combined into a feature vector of 625 values. 3.3
Haralick [
Using the results of our recent evaluation [
The original image was split into 7 7 non-overlapping tiles and the feature run for each of these; The colour image was quantised into 64 grey levels; 16 GLCMs were created for each image tile using vectors of length 1, 2, 3, and 4 pixels and orientations 0, =4, =2 and 3 =4; For each normalised co-occurrence matrix P (i; j) we calculated a homogeneity feature Hp, Hp = X X i j
P (i; j) 1 + ji jj :
This feature was chosen as it had performed consistently well in previous evaluations. 3.4
One of the most popular signal processing based approaches for texture feature extraction has
been the use of Gabor lters. These enable ltering in the frequency and spatial domain. It has
been proposed that Gabor lters can be used to model the responses of the human visual system.
Turner [
Our implementation is based on that of Manjunath et al [
Filtering an image I(x; y) with Gabor lters gmn designed according to [
Z Wmn(x; y) =
I(x; y)gmn(x x1; y y1)dx1dy1 : The mean and standard deviation of the magnitude jWmnj are used for the feature vector. The outputs of lters at di erent scales have di erent ranges. For this reason each element of the feature vector is normalised using the standard deviation of that element across the entire database.
From our evaluation [
For the colour images in the collection we used a feature that is good at capturing local colour
image structure. It is de ned in the HMMD (hue, min, max di ) colour space. This is used in the
MPEG-7 standard and is derived from both RGB and HSV spaces. The hue component is taken
from HSV and the min and max components are from the maximum and minimum values in the
RGB space. The di component is the di erence between min and max. We follow the MPEG-7
standard and quantise this space non-uniformly into 184 bins in the 3 dimensional hue, sum and
di colour space, see Manjunath and Ohm [
To calculate the colour structure descriptor an 8 8 window is slid over the image. Each of the 184 bins of the HMMD histogram contains the number of window positions for which there is at least one pixel falling into the bin under consideration. This feature is capable of discriminating between images with the same global colour distribution but di ering local colour structures. For this reason it is suited to colour medical images which tend to have similar overall colour but di ering structure depending on the detail of the photograph. 4
Our system had 34.5% mean average precision retrieval across all queries. This put us in third place for the visual retrieval task. The two runs performing better than ours, SUNY and KIDS, achieved maps of 34.88% and 34.57% respectivley. Fig 1 shows the precision recall graph for our run. These results showed that our simple approach, using good quality visual features, produced results comparable with the top systems. 4.1
With the relevance judgements available it was possible to look at how individual features had performed. Fig. 1 shows the precision recall graph for the individual features together with that for the combined features of the submitted run. Table 1 shows the mean average precisions for the same features. From these results it is clear that all features performed reasonably well. The Gabor feature on its own slightly outperformed the submitted run. Some additional feature combinations were tested, including adding the Gabor feature to each of the others in turn. However, none of these improved the mean average precision above that of Gabor on its own.
Reviewing the performance for individual queries showed that there was a wide variation.
Di erent features performed better on queries where they had better discrimination. An example
of this was query 7, a colour image, where HDS gave a mean average precision of 91.5% and Gabor
33.6%. This variation indicates that it could be possible to improve performance by weighting
features di erently depending on the query.
In addition to the search task we also used put the data set into a novel browsing network, NNk,
developed in our group [
This system can be accessed through our group demo page at http://km.doc.ic.ac.uk; open the iBase application and then select the CasImage collection on the settings page. A screen shot of the application is shown in Fig 2.
The experiments carried out used very e ective features. However, they were combined in a simple way, by summing the distances obtained from each feature. As shown shown in the postmortem analysis there was a wide variation in performance of di erent features for di erent queries. This would indicate that when querying for certain modalities or anatomic regions di erent combinations of features may perform better. By varying the weights applied to features we can introduce a degree of plasticity into our system and then use machine learning techniques to improve retrieval performance.
Given training data we would like to train a support vector machine as a meta classi er. We
have deployed this technique in other contexts, see Yavlinsky et al. [
These results show that CBIR can be used as a useful rst step in medical image retrieval.
Proceedings of the IEEE,