Combining Multilevel Visual Features for Medical Image Retrieval in ImageCLEFmed 2005

Abstract

In this paper we report our work on the fully automatic medical image retrieval task in ImageCLEFmed 2005. First, we manually identify visually similar sample images by visual perception for each query topic. These help us understand the variations of the query topic and form templates for similarity measure. To achieve higher performance, two similarity measuring channels are used with each using di®erent sets of features and operating in parallel. Their results are then combined to form a ¯nal score for similarity ranking. To improve e±ciency, a pre-¯ltering process using other features is utilized to act as a coarse topic image ¯ltering before the two similarity measures for ¯ne topic retrieval. During retrieval, no relevance feedback is used. Only visual features are used in our experiments for all the topics including visually possible queries (topics 1{11), mixed visual/semantic queries (topics 12-22) and semantic (rather textual) queries (topics 23-25). Over 50,000 medical images our approach achieved a mean average precision of 14.6% for all 25 topics, ranked as the best-performance run for the automatic medical image retrieval task in the ImageCLEFmed 2005.

Keywords

• H
• 3 [Information Storage and Retrieval]
• H
• 3
• 1 Content Analysis and Indexing
• H
• 3
• 3 Information Search and Retrieval

-

annotation for medical images. This paper reports our work in the sub-task of medical image retrieval. We con¯ne our e®orts in automatic runs using visual features only. The test data for this sub-task consist of four components: Casimage (containing 8725 radiology pathology images), MIR (containing 1177 nuclear medicine images from Mallinckrodt Institute of Radiology), PEIR (containing 32319 pathology and radiology images from Pathology Education Instructional Resource) and PathoPIC (containing 7805 pathology images). Besides the di®erent imaging modalities and anatomic regions, these 50k+ images are of various sizes and image qualities. Some of them use 1color channel while others 3-color channels. Results of ImageCLEFmed 2004 [ 4 ] with 26 retrieval topics on the Casimage collection are provided as training data for ImageCLEFmed 2005. 25 query topics are provided in this track. Each of them contains topic statements in English, French and German, and a collection of images for each topic. Normally one or two example images for the desired result for the topic are supplied. One query also contains a negative example as a test. These queries are divided into visually possible queries (topics 1{11), mixed visual/semantic queries (topics 12-22) and semantic (rather textual) queries (topics 23-25). Since we will use visual characteristics alone, it would be very hard for us to handle topics 12-25. This has been proved by the submitted results of this forum: the best run of the mixtures of textual and visual retrievals is almost twice good as that of runs using visual-only retrievals.

Content-based image retrieval (CBIR) retrieves images in terms of their visual contents [ 8, 9 ]. We apply it into this medical image retrieval campaign. The low-level raw features we used include pixel-level features such as color (which are very local), regional-level features such as regional color, shape and texture (which combine local and global information), and image-level features such as color and texture statistical property (which are global). Before retrieval, we manually identify more visually similar sample images by visual perception for each query topic. We can also generate some synthetic images from these chosen images with reasonable visual varieties. This helps us understand the visual variations of the query topic. Take the example of topic 3 provided in this track which asks to show pathology images of an entire kidney. Figure 1 shows the example image provided for this query topic together with three of many other instances of the query. They all contain an entire kidney but they di®er in almost all visual properties. In fact, there are a lot of variations of an entire kidney in size, shape, color, and/or skin texture. It shows that one should not choose visual features from a single example alone. It also shows that the retrieval task in this campaign is rather challenging. In some sense, retrieval of each topic is like a task of face recognition which has been proven to be very di±cult [ 14 ].

The sample images chosen for each topic are used as training data to design similarity measuring functions. We have used three functions ´1, ´2 and ´3 for each topic with three di®erent sets of features, respectively. The ¯rst function ´1 serves as a ¯lter to remove those dissimilar images. This improves e±ciency as many images can be excluded in the next comparison stage. Next, ´2 and ´3 operates in parallel [ 12 ] to yield two similarity measures. Then they are combined to produce a ¯nal score for image ranking. During retrieval, no relevance feedback is used anymore.

We submitted seven runs of experiments. All use visual features only but cover all the 25 topics including the visually possible queries (topics 1{11), mixed visual/semantic queries (topics 12-22) and semantic (rather textual) queries (topics 23-25). Five runs achieved top performance in this sub-task. We covered the top ¯ve runs in this subtask.

In the following sections, we introduce our work in more detail. The features used are explained in Section 2, followed by the retrieval methodology presented in Section 3. Section 4 explains our experiments. Conclusions are then drawn. 2

Multiple Feature Descriptions

In this section, we describe the visual features used in our work. A survey of visual features useful for general CBIR can be found in [ 13 ]. We have employed color, shape, texture characteristics at the pixel level, the region level and the entire image level. 2.1

Global Color and Layout Property

We have noticed that some of the query examples provided are colorful while others are gray. Those color images use three color channels. Most of gray images use one channel only. However there are some that still employ three channels. This channel information can be used directly to classify images. Furthermore, the layouts of images are di®erent consistently. For example, the ultrasound images are almost triangles. These features form description set F1. 2.2

Low Resolution Pixel Map at Pixel Level

Images in the database, even in the same class, vary in size and may have translations. Resizing them into a thumbnail [ 2, 5 ] of a ¯xed size, through introducing distortions, may overcome the above di±culties in representing the same class of images in the database. It is a reduced and low-resolution version from the original image in the database ignoring its original size. Thus this can also remove noise in the high-frequency band. A 16-by-16 image pixel map, called an \icon", is used. Examples are shown in Figure 2. Here the left three images are original ones in the same class and the right three images are their respective reduced versions. They look more similar visually than their original versions. These so-called \icons" are extensively used in face recognition and have proven to be e®ective [ 2 ]. We have also applied them to medical image retrieval [ 12 ]. We call them description set F2. We consider both regional and image-wide color, texture and shape features. Besides the pixel color, local contrast, anisotropy and polarity are captured to form a joint color-texture-spatial feature vector space. Gaussian-mixture models are built locally. The purpose is to de¯ne homogeneous regions in the feature space quantitatively. Numbering less than 5, the number of Gaussians for each region may be di®erent from each other. All the model parameters are found using the EM algorithm. These regions are extracted by a region merging process. The merged regions are segmented and referred to as meaningful local objects. The color, texture and shape properties of the regions are computed. The largest 10 regions are identi¯ed and obtained separately. Mean values and statistical properties of color, texture features and area are counted. The contour of each region is represented by elliptical Fourier expansion descriptors. We use the ¯rst 20 coef¯cients. Each region can be represented by the ellipse reconstructed from the ¯rst order of the expansion. These are the so-called blob representation [ 3 ]. Three pairs of examples are shown in Figure 3 where for each pair the left one is the original image and the right one is its blob representation. We have also included the global color histogram and texture histogram over the whole image. The above regional features and the global features form feature set F3, called as \blob" in this paper. The feature vector is of 352-dimension. For a more detailed description of the speci¯c usage, see our previous work [ 11, 12 ]. The section presents the retrieval methodology used in this work. The basic processing °ow of our approach is illustrated in Figure 4. Above the dashed horizontal line are processing procedures for any given query and below the line are procedures for the images in the test databases.

Before retrieval, we browse the four databases provided especially the CASImage database used for training. For j-th query topic, j=1,. . . ,25, some more semantically similar images (which may be visually di®erent) are chosen to form a set Qjn of n training images. For each image, some raw features, such as color, geometrical and texture properties, are extracted to form a p £ 1 feature vector xi = (xi1; xi2; ¢ ¢ ¢ ; xip) for image i.

Principal components are analyzed upon a set of such features. (We will provide more details later.) An eigenspace Ej is set up for each query topic j, j = 1; : : : ; 25. Feature dimension may also be reduced, which is illustrated as a dashed box. The feature vector of a test image xi = (xi1; xi2; : : : ; xip) is then projected to Ej . The similarity is measured in Ej .

This procedure is repeated for all test images to generate a similarity ranked list for them. For the test image, a pre-¯ltering is introduced using F1. Those images that are impossible to be similar are excluded earlier (denoted by \N"). Only those which can pass (indicated by \Y") will go to the ¯nal comparison stage. In this ¯nal stage, two parallel engines are introduced for similarity measures. They use independent sets of features \icon" and \blob", representing local and global characteristics, respectively. More speci¯cally, we analysis principal components and utilize them in two ways. The ¯rst is for feature dimension reduction. The second is used to design similarity measuring functions [ 10, 12 ]. Given a training dataset Qjn with n images: X = (x1; x2; ¢ ¢ ¢ ; xn)T , the generating matrix can be constructed as [ 10, 12 ]

C1= XXT; or,

C2= XTX: Here C1 is of n£n and C2 is of p£p. As mentioned before, n is the number of images/vectors and p is the number of features. C1 is used to generate templates of this dataset and C2 is used to reduce the dimension of the feature space when necessary. Supposing m out of n eigenvalues (¸i) and their eigenvectors (ui) are chosen based on C1. From the eigenvector matrix U = (u1; u2; ¢ ¢ ¢ ; um)T , the template vectors V = (v1; v2; ¢ ¢ ¢ ; vm)T are given by using vi = p 1 ¸i

XT ui; i = 1; :::; m: Given a test image set, its feature matrix Y is reconstructed by (1) (2) (3) (4) (5) where s1 and s2 are the similarity computed above using di®erent feature sets F1 and F2 whereas w1 and w2 are weighted coe±cients subject to 0 · w1; w2 · 1, w1 + w2 = 1. The resulting distance d serves as the ¯nal score for ranking: the larger the score is, the less similar the query and the test image are. 4

Experiments

In this campaign, we use visual features alone for all topic retrieval tasks including those that require text information. Experiments start from choosing training data. For each topic, we manually choose some images in the test database to represent the visual varieties of the query topic. Three undergraduate engineering students without medical background select these images. The only criteria are the visual appearance of the images. Consequently, it is no doubt there are many images wrongly chosen and the numbers of images are larger. The more correct the visual varieties of the query topic we can collect into the training set, the better the representation is semantically. This is done o²ine and before the retrieval.

We have also referred to the results from the baseline work from medGIFT [ 7 ]. Table 2 lists the number of images collected for each query topic. Here \q", \a" and \b" refers to the query topic and the number of images for two sets of training data, respectively. Total number of images in the training set \a" is 4789 with a mean 191.56 for each topic. In other words, 9.573% of the 50026 test images are used for training, which is a small portion. For training set \b", total there are 3874 images (i.e., 7.744% of the 50026 test images) with a mean 154.96 for each topic.

Next, we compute all the three feature sets F1, F2 (\icon") and F3 (\blob") for all images including those for training and testing. The similarity measuring function ´1 is a unit function for binary classi¯cation in terms of F1. Design ´2 and ´3 according to Equations (1) to (4) using F2 and F3 respectively. We combine their results according to Equation (5) with the same coe±cients (w1 = w2 = 0:5).

We submitted seven runs of retrievals. Table 2 lists their labels and their performance in terms of mean average precision (MAP). They are divided into 3 groups as shown in Table 2. Only Group 3 \I2Rfus" utilizes all techniques mentioned above. Other runs in Groups 1 and 2 use parts of the techniques for comparison. In Group 1, two subgroups are further divided in terms of the feature sets used. Subgroup 1 uses \blob" and Subgroup 2 uses \icon". In each subgroup, we have two members, one using pre-¯ltering (\I2RbPBcf" and \I2RcPBcf") while others (\I2RbPBnf" and \I2RcPBnf") do not.

We observe that using the \icon" feature set gives normally slightly higher MAP than using the \blob" feature set. This is clear by comparing \I2RbPBnf" (0.1067) against \I2RcPBnf" (0.1068), and \I2RbPBcf" (0.1114) against \I2RcPBcf" (0.1188), respectively. The binary classi¯er in Stage 1 improves the entire system performance. To see this e®ect, we can compare \I2RbPBnf" (0.1067) against \I2RbPBcf" (0.1114) and \I2RcPBnf" (0.1068) against \I2RcPBcf" (0.1188), respectively. The improvement is more signi¯cant when using the \icon" feature set (11.24%) than using the \blob" set (with 4.4%). Group 2 is the fusion of \I2RcPBnf" and \I2RcPBcf" where the weights are equal. It achieves the best results (MAP=14.55%).

It is important to select more examples to form a training set for each query topic before retrieval. To have a comparison, some of images (the underlined numbers in Table 1) are removed from the representation sets of some topics. We repeat experiments \I2RbPBnf" (0.1067) against \I2RcPBnf" (0.1068) but using these new training sets (Set \b"). This results in \I2RbP1nf" (using \blob" with MAP=0.0928) and \I2RcP1nf" (using \icon" with MAP=0.0934) in Group 3. Again, \icon" features have slightly better precision performance. Comparing experiments vertically using the two training sets, one ¯nds that performance of Group 3 drops down using either feature set. This shows that the representation of the query topic using the training set is indeed important. 5

Discussion and conclusion

We have reported our e®orts to the medical image retrieval task in the ImageCLEFmed 2005. We analyzed the contents of images and employ three sets of visual features at di®erent levels to represent each image. We start from manual selecting some training images for each topic before retrieval. These images construct a training set for us to span an eigenspace for the topic and to de¯ne similarity metrics for it. A pre-¯ltering process is used to act as a coarse topic image ¯ltering before the two similarity measures for ¯ne topic retrieval. The features are simple and the comparison is easy and fast. Many test images can be simply classi¯ed into impossible class reliably. To achieve higher performance, two similarity measuring channels are used. They use di®erent sets of features and operate in parallel. Their results are then combined to form a ¯nal score for similarity ranking. We have not used relevance feedback during the retrieval.

In our experiments, only visual features are applied to not only the 11 visual-retrieval-possible topics, but also those 13 topics needing rather textual information. We have submitted seven runs in this track. Our best approach utilizes multiple sets of features with pre-¯ltering and fusing strategies, which enables us to achieve a very good performance in the visual-only group.

It should be noted that our work is based on the pre-selection of more example images for the query topic. The training data were chosen o²ine and it may be inconvenient for online applications. Both the qualities and the number of these images in°uence the retrieval performance. In our current e®orts, they are still large for some topics and yet we have not re¯ned them. Our future e®orts would re¯ne the sets and combine some machine learning techniques to facilitate the selection.

References

[1] F. Alkoot and J. Kittler . Experimental evaluation of expert fusion strategies . Pattern Recognition Letters , 20 : 1361 { 1369 , 1999 .

[2] Peter N. Belhumeur , J. P. Hespanha , and David J. Kriegman . Eigenfaces vs. ¯sherfaces: Recognition using class speci¯c linear projection . IEEE Transactions on pattern analysis and machine intelligence , 19 ( 7 ): 711 { 720 , 1997 .

[3] C. Carson , S. Belongie , H. Greenspan , and J. Malik . Recognition of images in large databases using color and texture . IEEE Transactions on pattern analysis and machine intelligence , 24 ( 8 ): 1026 { 1038 , 2002 .

[4] Paul Clough , Mark Sanderson, and Henning MuÄller. The clef 2004 cross language image retrieval track . In C. Peters, J. Gonzalo P. Clough , G. Jones, M. Kluck , and B. Magnini, editors, Lecture Notes in Computer Science (LNCS) , Multilingual Information Access for Text, Speech and Images: Results of the Fifth CLEF Evaluation Campaign , Heidelberg, Germany, 2005 . Springer. in print.

[5] P. Howarth , A. Yavlinsky , D. Heesch , and S. RuÄger. Visual features for content-based medical image retrieval . In Proceedings of Cross Language Evaluation Forum (CLEF) Workshop 2004 , Bath, UK, 2004 .

[6] L. I. Kuncheva . A theoretical study on six classi¯er fusion strategies . IEEE Transactions on pattern analysis and machine intelligence , 24 ( 2 ): 281 { 286 , Feburary 2002 .

[7] Henning MuÄller , A. GeissbMuÄhler, and P. Ruch . Report on the clef experiments: Combining image and multi-lingual search for medical image retrieval . In C. Peters, J. Gonzalo P. Clough , G. Jones, M. Kluck , and B. Magnini, editors, Lecture Notes in Computer Science (LNCS) , Multilingual Information Access for Text, Speech and Images: Results of the Fifth CLEF Evaluation Campaign , Heidelberg, Germany, 2005 . Springer. in print.

[8] Henning MuÄller , N. Michoux , D. Bandon , and A. Geissbuhler . A review of content-based image retrieval systems in medicine - clinical bene¯ts and future directions . International Journal of Medical Informatics , 73 :1{ 23 , 2004 .

[9] A. W. M. Smeulders , M. Worring , S. Santini , A. Gupta , and R. Jain . Content-based image retrieval at the end of the early years . IEEE Transactions on Pattern Analysis and Machine Intelligence , 22 ( 12 ): 1349 { 1380 , 2000 .

[10] Wei Xiong , Bo Qiu, Qi Tian, , Changsheng Xu , Sim Heng Ong, and Kelvin Foong . Contentbased medical image retrieval using dynamically optimized regional features . In The IEEE International Conference on Image Processing 2005 , Genoa, Italy, September 2005 . to appear.

[11] Wei Xiong , Bo Qiu, Qi Tian, Henning MuÄller, and Changsheng Xu . A novel content-based medical image retrieval method based on query topic dependent image features (QTDIF) . Proceedings of SPIE , 5748 : 123 { 133 , 2005 .

[12] Wei Xiong , Bo Qiu, Qi Tian, Changsheng Xu, Sim Heng Ong, Kelvin Foong, and Jean-Pierre Chevallet . Multipre : A novel framework with multiple parallel retrieval engines for contentbased image retrieval . In ACM Multimedia 2005 , Hilton, Singapore, November 2005 . to appear.

[13] Zijun Yang and C.-C. Jay Kuo. Survey on image content analysis, indexing, and retrieval techniques and status report of mepg-7 . Tamkang Journal of Science and Engineering , 2 ( 3 ): 101 { 118 , 1999 .

[14] W. Zhao , R. Chellappa , A. Rosenfeld , and P. Phillips . Face recognition: A literature survey , 2000 .