Medical Image Retrieval and Automatic Annotation: VPA-SABANCI at ImageCLEF 2009

Abstract

Advances in the medical imaging technology has lead to an exponential growth in the number of digital images that needs to be acquired, analyzed, classi¯ed, stored and retrieved in medical centers. As a result, medical image classi¯cation and retrieval has recently gained high interest in the scienti¯c community. Despite several attempts, such as the yearly-held ImageCLEF Medical Image Annotation Competition, the proposed solutions are still far from being su±ciently accurate for real-life implementations. In this paper we summarize the technical details of our experiments for the ImageCLEF 2009 medical image annotation task. We use a direct and two hierarchical classi¯cation schemes that employ support vector machines and local binary patterns, which are recently developed low-cost texture descriptors. The direct scheme employs a single SVM to automatically annotate X-ray images. The two proposed hierarchical schemes divide the classi¯cation task into sub-problems. The ¯rst hierarchical scheme exploits ensemble SVMs trained on IRMA sub-codes. The second learns from subgroups of data de¯ned by frequency of classes. Our experiments show that hierarchical annotation of images by training individual SVMs over each IRMA sub-code dominates its rivals in annotation accuracy with increased process time relative to the direct scheme.

Introduction

Digital medical images, such as standard radiographs (X-Ray) and computed tomography (CT) images, represent a huge part of the data that need to be stored, archived, retrieved, and shared among medical centers. Manual labeling of this data is not only time consuming, but also errorprone due to inter/intra-observer variations. In order to realize an accurate classi¯cation of digital medical images one needs to develop tools that allow high performance automatic image annotation, i.e. a given image is automatically labeled with a text or a code without any user interaction.

Several attempts in the ¯eld of medical images have been performed in the past. For example, the WebMRIS system [ 3 ] aims at retrieving cervical spinal X-Ray images, whereas the ASSERT system [ 8 ] focuses on retrieving CT images of lung. While these e®orts consider retrieving a speci¯c body part only, other initiatives have been taken in order to retrieve multiple body parts.

The ImageCLEF Medical Image Annotation task, run as part of the Cross-Language Evaluation Forum (CLEF) campaign, is a yearly held medical image annotation challenge that aims in automatic classi¯cation of an X-Ray image archive containing more than 12,000 images randomly taken from the medical routine. The ImageCLEF Medical Annotation dataset contains images of di®erent body parts of people from di®erent ages, of di®erent genders, under varying viewing angles and with or without pathologies.

A potent classi¯cation system requires the image data to be translated into a more compact and more manageable representation containing descriptive features. Several feature representations have been investigated in the past for such a classi¯cation task. Among others, image features, such as average value over the complete image or its sub-regions [ 7 ] and color histograms [ 4 ], have been investigated. Recently in [ 2 ], texture features like local binary patterns (LBP) [ 6 ] have been shown to outperform other types of low-level image features in classi¯cation of X-Ray images. Subsequently in [ 10 ], it has been shown that retaining only the relevant features by applying attribute selection on local binary patterns achieves comparable classi¯cation accuracies with smaller feature sets, thus leading to reduced processing time and storage space requirements.

A less investigated path is to exploit from hierarchical organization of medical data, such as the ImageCLEF data labeled by the IRMA coding system, using ensemble classi¯ers. Accordingly, in this paper we explore the annotation performance of two hierarchical classi¯cation schemes based on IRMA sub-codes and frequency of classes, and compare them to the well-known single-classi¯er scheme over the ImageCLEF-2009 Medical Annotation dataset.

The paper is organized as follows. Section 2 presents our feature extraction and classi¯cation steps in detail. Then, in Section 3 we introduce the image database and the experimental evaluation process performed. And ¯nally, Sections 4 and 5, present corresponding results and our conclusions, respectively. 2 2.1

Method

Feature Extraction

We extract spatially enhanced local binary patterns as features from each image in the database. LBP [ 6 ] is a gray-scale invariant local texture descriptor with low computational complexity. The LBP operator labels image pixels by thresholding a neighborhood of each pixel with the center value and considering the results as a binary number. The neighborhood is formed by a symmetric neighbor set of P pixels on a circle of radius R. Formally, given a pixel at (xc,yc), the resulting LBP code can be expressed in the decimal form as follows :

LBPP;R(xc; yc) =

P ¡1 X s(in ¡ ic)2n n=0 (1) where n runs over the P neighbors of the central pixel, ic and in are the gray-level values of the central pixel and the neighbor pixel, and s(x) is 1 if x ¸ 0 and 0 otherwise. where L is the number of di®erent labels produced by the LBP operator, and I(A) is 1 if A is true and 0 otherwise.

The derived LBP histogram contains information about the distribution of local micro-patterns, such as edges, spots and °at areas, over the image. Following [ 6 ], not all LBP codes are informative, therefore we use the uniform version of LBP and reduce the number of informative codes from 256 to 59 (58 informative bins + one bin for noisy patterns). As in [ 2 ], we divide the images into 4x4 non-overlapping sub-regions and concatenate the LBP histograms extracted from each region into a single, spatially enhanced feature histogram (Figure 1). This step aims at obtaining a more local description of the image.

Finally, we obtain a total of 944 features per image. In order to avoid domination of attributes with greater numeric ranges over small ones, we linearly scale each feature to [-1,+1] range before presenting them to the classi¯er. 2.2

Image Annotation

In this work we use a support vector machine (SVM) based learning framework to automatically annotate the images. SVM [ 1 ] is a popular machine learning algorithm that provide good results for general classi¯cation tasks in the computer vision and medical domains: e.g. nine of the ten best models in ImageCLEFmed 2006 competition were based on SVM [ 5 ]. In a nutshell, SVM maps data to a higher-dimensional space using kernel functions and performs linear discrimination in that space by simultaneously minimizing the classi¯cation error and maximizing the geometric margin between the classes.

Among all available kernel functions for data mapping in SVM, Gaussian radial basis function is the most popular choice, and therefore it is used here.In this work we used LibSVM1 library (version 2.89) for SVM and empirically found its optimum parameters on the dataset. 2.2.1

Direct Annotation Scheme In the direct annotation scheme, we classify images by using a single SVM with one versus all multi-class model.

1Available at http://www.csie.ntu.edu.tw/ cjlin/libsvm 2005 88.0 88.0 83.3 3

Experimental Setup

Image Data

The database released for the ImageCLEF-2009 Medical Annotation task includes 12677 fully classi¯ed (2D) radiographs for training and a separate test set consisting of 2000 radiographs. The aim is to automatically classify the test set using four di®erent label sets including 57 to 193 distinct classes. A more detailed explanation of the database and the tasks can be found in [ 9 ]. 3.2

Evaluation

We evaluate our SVM-based learning using two schemes depending on the availability of test data labels: 1)5-fold cross validation if test data labels are missing, and 2)ImageCLEF error counting scheme, otherwise. In the former scheme, the training database is partitioned into ¯ve subsets. Each subset is used once for testing while the rest are used for training, and the ¯nal result is assigned as the average of the ¯ve validations. Note that for each validation all classes were equally divided among the folds. We measure the overall classi¯cation performance using accuracy, which is the number of correct predictions divided by the total number of images. To the contrary, the error counting scheme is introduced by the contest organizers to compare all runs submitted. Further details on this scheme can be found in [ 9 ]. 3.3

Runs Submitted

As Computer Vision and Pattern Analysis (VPA) Laboratory of Sabanci University, we submitted three di®erent runs to the ImageCLEF 2009 medical image annotation task. One obtained by the direct scheme (VPA-SABANCI-1), and two with the hierarchical schemes (VPA-SABANCI-2 and -3). For each run, the optimum parameter setting was realized by trial-and-error. 4

Results

In this section, we present the results obtained by the proposed annotation schemes. In Table 1 we observe the results realized on the training database with 5-fold cross-validation. Hierarchical scheme based on IRMA sub-codes clearly outperforms the others, especially in terms of the 2007, 2008 and overall accuracies.

Table 2 provides a detailed performance comparison of the direct scheme and the IRMA subcodes based hierarchical one over 2007 and 2008 labels. Simplifying the classi¯cation task by training a separate SVM over each sub-code, considerably improves the ¯nal accuracy relative to the usage of a single SVM. Furthermore, 2008 accuracies of individual SVMs excel those of 2007 despite higher number of classes (thus a more di±cult classi¯cation problem). The underlying reason for this observation may be attributed to the more realistic labels of 2008.

In Table 3 we present the results achieved on the test dataset in terms of prediction errors. As observed, IRMA sub-codes based hierarchical scheme (H-1) outperforms its rivals again. With this performance, VPA-SABANCI-2 run is ranked 7th among 18 runs submitted to the competition. 2007 accuracy (%) 2008 accuracy (%) In this paper we have introduced a classi¯cation work with the aim of automatically annotating X-Ray images. We have explored the annotation performances of two hierarchical classi¯cation schemes based on individual SVMs trained on IRMA sub-codes and frequency of classes, and compared the results with the popular single-classi¯er scheme. Our experiments on the ImageCLEF2009 Medical Annotation database revealed that breaking the annotation problem down to subproblems by training individual SVMs over each IRMA sub-code outperforms its rivals in terms of annotation accuracy with the compromise of increased computational expense. Run VPA-SABANCI-1 VPA-SABANCI-2 VPA-SABANCI-3

Type D H-1 H-2

CPU Time

T 4T kT

Memory Usage

M M M

of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, volume 3312, pages 392{403, December 1997.

References

[1] C. J. C. Burges . A tutorial on support vector machines for pattern recognition . Data Mining and Knowledge Discovery , 2 ( 2 ): 121 { 167 , 1998 .

[2] V. Jacquet , V. Jeanne , and D. Unay . Automatic detection of body parts in x-ray images . In Mathematical Methods in Biomedical Image Analysis , 2009 . MMBIA 2009 . IEEE Computer Society Workshop on, 2009 .

[3] L. R. Long , S. R. Pillemer , R. C. Lawrence , G. -H. Goh , L. Neve , and G. R. Thoma . WebMIRS: web-based medical information retrieval system. In I. K. Sethi and R. C. Jain, editors, Society

[4] A. Mueen , M. Sapian Baba , and R. Zainuddin . Multilevel feature extraction and x-ray image classi¯cation . J. Applied Sciences , 7 ( 8 ): 1224 { 1229 , 2007 .

[5] H. MuÄller , T. Deselaers, T. Deserno , P. Clough , E. Kim, and W. Hersh . Overview of the imageclefmed 2006 medical retrieval and medical annotation tasks . In Evaluation of Multilingual and Multi-modal Information Retrieval , pages 595 { 608 . 2007 .

[6] T. Ojala , M. Pietikainen , and T. Maenpaa . Multiresolution gray-scale and rotation invariant texture classi¯cation with local binary patterns . Pattern Analysis and Machine Intelligence , IEEE Transactions on, 24 ( 7 ): 971 { 987 , 2002 .

[7] Md. M. Rahman , B. C. Desai , and P. Bhattacharya . Medical image retrieval with probabilistic multi-class support vector machine classi¯ers and adaptive similarity fusion . Computerized Medical Imaging and Graphics , 32 ( 2 ): 95 { 108 , 2008 .

[8] C.-R. Shyu , C. E. Brodley , A. C. Kak , A. Kosaka , A. M. Aisen , and L. S. Broderick . Assert: a physician-in-the-loop content-based retrieval system for hrct image databases . Comput. Vis. Image Underst ., 75 ( 1-2 ): 111 { 132 , 1999 .

[9] T. Tommasi , B. Caputo , P. Welter , and T. M. Deserno . Overview of the CLEF 2009 medical image annotation track , CLEF working notes 2009 . Corfu, Greece, 2009 .

[10] D. Unay , O. Soldea , A. Ekin , M. Cetin , and A. Ercil . Automatic Annotation of X-ray Images: A Study on Attribute Selection. In Medical Content-based Retrieval for Clinical Decision Support (MCBR-CDS ) Workshop in conjunction with MICCAI' 09 , 2009 .