In this article, we present the algorithms and results of our participation in the medical image annotation and retrieval tasks of ImageCLEFmed 2006. We exploit both global features and local features to describe medical images in the annotation task. We examine di®erent kinds global features and extract the most descriptive ones, which e®ectively capture the intensity, texture and shape characters of the image content, to represent the radiographs. We also evaluate the descriptive power of local features, i.e. local image patches, for medical images. A newly developed spatial pyramid matching algorithm is applied to measure the similarity between images represented by sets of local features. Both descriptors use multi-class SVM to classify the images. The error rate is 17.6% for global description and 18.2% for the local one, which rank sixth and ninth respectively among all the submissions. For the medical image retrieval task, we only use visual features to describe the images. No textual information is considered. Di®erent features are used to describe gray images and color images. Our submission achieves a mean average precision (MAP) of 0.0681, which ranks second in the 11 runs that also only use visual features.
Due to the rapid development of biomedical informatics, medical images have become an indispensable investigation tool for medical diagnosis and therapy. A single average size radiology department may produce tens of tera-bytes of data annually. The ever-increasing amount of digitally produced images require e±cient methods to archive and access this data. Therefore, the application of general image classi¯cation and retrieval techniques in this specialized domain has obtained increasing research interest recently.
ImageCLEF, which conducts evaluation of cross-language image retrieval has come up with a medical image retrieval task since 2004. And an automatic medical image annotation task was added in 2005. It provides a benchmark to evaluate the performance of di®erent algorithms on the same tasks using the same dataset. The tasks in 2006 are similar to those in the last year. The dataset and the task description are almost the same. However, the topics are more challenging than last year's. More categories are de¯ned for the annotation task and more semantic queries are issued for the retrieval task.
In this paper, we describe our participation in the automatic medical image annotation and medical image retrieval tasks of ImageCLEF 2006. We submitted two runs for the annotation task, which exploited the e®ectiveness of two di®erent kinds of features to describe and classify medical images. The ¯rst run examined di®erent kinds of global features and extracted the most descriptive ones to represent the radiographs. It achieved an error rate of 17.6%, which ranked sixth among all the submissions. In the second run, we applied a newly developed spatial pyramid matching scheme to this task, which e®ectively measured the similarity between images represented by sets of local features. It achieved an error rate of 18.2%, and ranked ninth in the submissions. We submitted one run for the medical image retrieval task. We evaluated the e®ectiveness of visual features for medical image retrieval. Our submission yielded a mean average precision (MAP) of 0.0681, which ranked second in the 11 runs that also only used visual features.
The rest of the paper is organized as follows. We describe the details of our runs for the automatic annotation task in Section 2. The medical image retrieval task is presented in Section 3. Experimental results are discussed in Section 4. Finally, we conclude this paper in Section 5. 2
The automatic image annotation task is to classify images into a set of prede¯ned categories. It provides a dataset consisting of 10,000 fully classi¯ed radiographs for participants to train a classi¯cation system. These images are classi¯ed into 116 categories this year according to image modality, body orientation, body region and the biological system examined. 1000 additional radiographs whose classi¯cation labels are unavailable to participants are used to evaluate the performance of various algorithms.
We developed two di®erent schemes for this task. In the ¯rst algorithm, traditional global features, such as intensity, texture and shape descriptors were used to describe medical images. In the second one, we exploited using local features to represent the images. And a spatial pyramid matching scheme was then applied to measure the similarity between two images. Both methods used SVM to classify the images into di®erent categories. 2.1
When designing image features, we should consider two issues. First, the features should be representative for the images. Second, the complexity of calculating the features should be relatively low. Medical images have their particular characteristics in appearance. For example, radiographs are usually grayscale images and the spatial layouts of the anatomical structures in the radiographs of the same category are quite similar. The texture, shape and local features are valuable and discriminative for describing medical images.
According to these observations, we select several di®erent visual features to represent the radiographs. We extract gray-block feature and block wavelet feature from the original images. Shape-related features are exacted from the corresponding binary images. Then, they are combined into a 382-dimensional feature vector. The detail descriptions of the features are as follows: Gray-block feature The original images are uniformly divided into 8 £ 8 = 64 blocks. The average gray value in each block is calculated and a 64-dimensional gray-block feature is obtained. The `2¡norm of the feature vector is set to 1. The normalization could reduce the in°uence of illumination variance across di®erent images to some extent. According to the experiments, this is the most e®ective feature although it is straight forward and very simple.
Block-wavelet feature The wavelet coe±cients could characterize the texture of the images at di®erent scales. We divide the images into 4 £ 4 = 16 blocks and extract multi-scale wavelet features in each block. We implement 3-level wavelet transforms on the image blocks using Daubechies ¯lter (db8). Then, the mean and the variance of the wavelet coe±cients in the HL, LH and HH sub-bands are computed. Therefore, we get a 288(6 £ 3 £ 4 £ 4)-dimensional feature vector.
Features for the binary image We ¯rst convert the images into binary images. Otsu's method
[
Choosing suitable parameters for above features is very di±cult in theory. Therefore, we tune the parameters through experiments. The parameters, such as the size of the image block and the dimension of the features for the binary image, are determined through cross-validation on the training set. The same parameter settings are used in both of the annotation task and the retrieval task.
The classi¯er is trained using SVM, which is a classic machine learning technique that has
strong theoretical foundation and excellent empirical successes. The basic idea of SVM is to map
the data into a high dimensional space and then ¯nd a separating hyperplane with the maximal
margin. In the experiment, we use the multi-class SVM implemented by the LIBSVM tool[
Recently, a class of local descriptor based methods, which represent an image with an collection
of local photometric descriptors, have demonstrated impressive level of performance for object
recognition and classi¯cation. And this kind of algorithms have also been explored for
medical image classi¯cation, considering that most information in medical images is local [
Although SIFT descriptor [
In order to measure the similarity between two images represented by orderless collections of local patches, we ¯rst partition the scaled images into increasingly ¯ne sub-regions. Then we compute the histograms of cluster frequencies inside each sub-region by counting the number of patches that belong to each cluster (Fig. 1). The histograms from two images are compared using a weighted histogram intersection measure. Let X and Y be two sets of feature vectors representing two images. Their histograms in the ith sub-region at level l are denoted by HXli and HYli with HXli (j) and HYli(j) indicating the number of feature vectors from X and Y that fall into the jth bin of the histograms. The histogram intersection function is given by (1) (2) I(HXli ; HYli) =
M X min(HXli (j); HYli(j)) ; j=1 which measures the \overlap" between two histograms' bins. It implicitly ¯nds the correspondences between feature vectors falling into that sub-region. The similarity between X and Y is de¯ned as the weighted sum of the number of matches found in each sub-region:
K(X; Y ) =
L 4(l¡1) M X wl X X min(HXli (j); HYli(j)) ; l=1
i=1 j=1 where L refers to the max level. As shown in Fig. 1, the weight wl is inversely proportional to region size: the smaller the region the larger the weight, i.e. matches made within smaller regions are weighted more than those made in larger regions.
Actually, K can be implemented as a single histogram intersection of \long" vectors which are formed by concatenating the appropriately weighted histograms in all sub-regions. For L levels and M clusters, although the index of the single histogram may be as high as M PlL=1 4l¡1, the histogram of each image is actually very sparse. The number of non-zero bins is at most mL.
elvel1 *# # + *+ # + + *
Another implementation issue is normalization. In order not to favor large feature sets, which would always yield high similarity due to the intersection operation, we should normalize the histograms by the total weight of all features in the images before conducting matching.
K has been proved to satisfy the Mercer's condition, i.e. it is positive semi-de¯nite [
The dataset for the medical image retrieval task consists of images from the Casimage, MIR, PEIR and PathoPIC datasets. There are totally 50,026 images with di®erent modalities, such as photographs, radiographs, ultrasonic images, and scans of illustrations used for teaching etc. Query topics are formulated with example images and a short textual description, which denotes the exact information need such as the illness, the body region or the modalities shown in the images. Therefore, this task is much more challenging than the annotaion task. We only exploit the e®ectiveness of visual features for this task. No textual information is utilized in our experiment.
As general image retrieval systems, the whole retrieval procedure contains three steps: image preprocessing, feature extraction and relevance ranking based on similarity measure. For image preprocessing, we ¯rst resize the images so that the long sides are 512 pixels and their aspect ratios are maintained. As the characters of gray images and color images are quite di®erent, we examine whether an image is gray or color before extracting features from it. Note that the images in Fig. 2 are regarded as gray images because the color information in them are very limited and also useless for retrieval. Feature extraction is carried out according to the type of the image, i.e. the features for gray image and color image are di®erent: Features for gray images The global features used to describe radiographs in the annotation
task are used here to describe the gray images.
Features for color images We use band-correlogram, color histogram and block-wavelet
features to describe the color images:
- Band-correlogram We ¯rst quantize the RGB values into 64 bins. Then the general
autocorrelogram features are extracted within four square neighborhoods, whose radius are
1,3,5,7 pixels respectively. The ¯nal features used are the average of the corresponding
elements in the four square neighborhoods. It is a 64-dimensional feature vector.
- Color histogram We quantize the RGB values into 36 bins, and calculate the
36dimensional color histogram as described in [
L = 0:299 £ R + 0:587 £ G + 0:114 £ B :
Then the block-wavelet feature are calculated as introduced in Sect.2.1.
The last step is ranking the images in the dataset according to their relevance to the query images. As each topic contains multiple query images, the distance between a dataset image Z and a set of query images belonging to the same topic is de¯ned as the minimun distance between Z and each query image: d(Z; Q) = min d(Z; Qi) :
i The top 1000 images are returned for evaluation. 4 4.1
For the annotation task ,we submitted two runs named \msra wsm gray" and \msra wsm patch" for global feature and local feature methods respectively. The submission using global features achieved an error rate of 17.6%, which ranked sixth among all the submissions. And the error rate of the run using local features is 18.2%, which ranked ninth.
Fig. 3 illustrates the classi¯cation precisions of each category on the test dataset. The results for the run using global features are denoted by blue bars, and the local feature based method is denoted by red bars. In Fig. 4 we calculate the average precisions across di®erent categories, for which the numbers of training images are larger than a speci¯ed number given by the X axis. Through analyzing these experimental results, we could get some valuable information. Firstly, all the categories with zero precisions are corresponded to the categories whose training images are less than 20. Secondly, when the number of training images is larger than 20, our (3) (4) methods could have more stable performance on average precision. Thirdly, our two methods achieved comparable performances. As they are complementary for describing images, we could expect better performance if we combine these two descriptions together. However, we haven't implemented the combination so far. We will explore it in our future work. 4.2
In the medical image retrieval task, the parameters for gray images are the same with the annotation task. The parameters for color images are determined empirically. The details have been discussed in Section 3. We employ these features in our automatic \visual retrieval" system and submit only one run named \msra wsm". We achieved a MAP of 0.0681, which ranks second among the 11 runs that also only use visual features. The MAP of the best run is 0.0753.
The MAP values for each query are shown in Fig. 5. We use di®erent color bars to indicate the di®erent performances on visual, mixed and semantic topics. The average MAP on these three kinds of topics are 0.1324, 0.0313 and 0.0406 respectively. It is obvious that the performance on visual topics is the best. The performance is relatively poor on other topics with more semantic considerations. The di®erences between the performances on di®erent kinds of topics are reasonable considering the design of the topics. The MAP for the 23rd topic which is a semantic topic is strangely high. It is because the number of images that are similar with the query images of this topic is quite large. In this paper, we present our work on the medical image annotation and retrieval tasks of ImageCLEFmed 2006. Due to the special characteristics of medical images, we explored using global and local features respectively to describe the radiographs in the annotation task. Then we use the multi-class SVM to classify the images. We achieved an error rate of 17.6% for the global feature based method and 18.2% for the local feature method. For the medical image retrieval task, we distinguished gray images from color images and used di®erent kinds of visual features to describe them. Our submission ranked second among the 11 runs which also only used visual features.
This is our ¯rst participation in the tasks concerning medical images. We ¯nd this task quite interesting and very challenging. In our future work, we will investigate some more descriptive features and more suitable similarity measure for comparing images. We didn't utilize the textual information in our experiment. We will incorporate it into the retrieval framework in the future.