Automatic Classi cation of Body Parts X-ray Images

Abstract

The development of automatic analysis and classi cation methods for large databases of X-ray images is a pressing need that may have a great impact on clinical practice. To advance this objective the ImageCLEF-2015 clustering of body part X-ray images challenge was created. The aim of the challenge is to group digital X-ray images into ve structural groups: head-neck, upper-limb, body, lower-limb, and other. This paper presents the results of an experimental evaluation of X-ray images classi cation in the ImageCLEF-2015 challenge. We apply state-of-the-art classi cation and feature extraction methods for image classi cation and optimize them for the challenge task with emphasis on features indicating bone size and structure. The best classi cation results were obtained using the intensity, texture and HoG features and the KNN classi er. This combination has an accuracy of 86% and 73% for the 500 training images and 250 test images, respectively.

Keywords

• Classi cation
• X-ray images
• ImageCLEF-2015

-

The increasing amount of medical imaging data acquired in clinical practice constitutes a vast database of untapped diagnostically-relevant information, millions of images are acquired worldwide each year. Clinicians are struggling under the burden of diagnosis and follow up of such an immense amount of images. This phenomenon gave rise to a plethora of methods to improve and assist clinicians using e cient search capabilities.

Content-Based Image Retrieval (CBIR) is a popular growing research topic [ 1 ]. The goal of CBIR is to assist physicians with diagnosis by nding similar cases to the case at hand. Therefore, CBIR requires e cient search capabilities in a vast database of medical images. The problem is emphasized in X-ray imaging, the most widely used medical imaging modality today as many clinical home health-care centers are equipped with X-ray scanners and maintain their own database of images.

This paper elucidates the problem of classi cation of digital X-ray image into ve groups: head-neck, upper-limb, body, lower-limb and other (Fig 1).

A variety of methods exist for medical image feature extraction and classication, Haralick et al. [ 14 ] suggest feature extraction based on gray level cooccurrences matrices, whereas Weszka et al.[23] perform a classi cation based on local binary patterns (LBP). Another strategy is to combine local and global features presented by Rublee et al. [ 18 ], using pixel values and shape features extracted with the Canny edge detection method. The pixel values and shape features are then used as a unique multi-feature vector used for classi cation.

Advanced methods include image classi cation based on the IRMA code [ 16 ]. In this method, features are extracted from the modality, body orientation, anatomic region and biological system. More recently, the Bag of Visual Words model (BoVW) [ 4 ] was used for X-ray images. In the BoVW approach, a visual word vocabulary is created from local image patches to represent an image, which is obtained by extracting feature descriptors around interest points.

Ghofrani et al. recently proposed the classi cation-based fuzzy set theory [ 12 ] They performed a fuzzy set classi cation with feature extraction based on a combination of shape and texture using the Canny Edge Detector and the Discrete Gabor Transform. Zare et al. [24] present three techniques for image annotation: the probabilistic latent semantic analysis (PLSA) image annotation, binary classi cation annotation, and annotation based on similar images. In their approach, semantic information is captured from textual and visual modalities and the correlation between them is learned.

This paper presents the results of an experimental evaluation of X-ray images classi cation [ 3 ] in the ImageCLEF-2015 challenge [ 21 ]. The goal of the challenge is to group digital X-ray images into ve groups: head-neck, upper-limb, body, lower-limb, and other. In the context of the challenge, we apply state-of-theart classi cation and feature extraction methods for image classi cation and optimize them for the challenge task with an emphasis on features indicating bone size and structure. 2

Method

The aim of our method is to group digital X-ray images into ve groups: headneck, upper-limb, body, lower-limb, and other (see Fig 1) . Our objective is to apply state-of-the-art classi ers and feature selection methods and to optimize them for the challenge task with an emphasis on features indicating bone size and structure.

The input to our method is a set of (1) label X-ray images from ve groups, (2) features extraction techniques and; (3) classi ers. The output of our method is a combination of 10 features-classi er pairs that achieve the highest classi cation accuracy on the given ve groups classi cation task.

Our method consists of two steps. (1) a two-class experiment was used in order to select the features-classi er pair that best distinguished between X-ray images containing big and long bones (e.g. skull, arm and leg) against small and short bones (e.g. chest and abdomen bones). (2) The features-classi er pairs that that provide an average accuracy of grater then 90% in the rst step evaluate on the ve groups of X-ray images (head-neck, upper-limb, body, lower-limb and other Fig 1) set to nd 10 best combination of features-classi er. Those 10 best features-classi er pairs were submitted to the challenge evaluation.

Fig 2 illustrates the ow of our method. Next, we describe each step in details. 2.1

Features Extraction

Nine features extraction methods were evaluated in this study. Below we describe the various features that were examined in our study.

{ Color Extracting. A gray-scale histogram [ 22 ] is used to represent the color distribution of the image. We divide the image into equal patches (7x7) and compute an 8 bit histogram for each region. Then we add all patch based histogram into a single vector that serves as the color feature of our method. { Texture Extracting. Texture features are examined using Local Binary Pattern (LBP) [ 13 ] which provides highly discriminative texture information and is used to provide robust pattern-related information. We divide each image into equal (10x10) patches and extract LBP values for each patch. The patches are then represented in a single vector to serve as the texture feature of our method.

{ HoG This is a histogram of neighborhood pixels according to their gradient orientation, weighted by their gradient magnitude. HoG features were shown to be particularly discriminative of people and body shapes. We extract the HoG values for each 10x10 patch in the image. These values are then represented as a single vector to serve as the HoG feature of our method. [ 8 ]. { BoVW [ 9 ] This method produces a visual vocabulary. The method descriptors were extracted from detected key points using the following algorithms: Scale invariant feature transform (SIFT) [ 17 ].

Speded up robust features (SURF) [ 5 ].

Binary robust independent elementary features Brief (BRIEF) [ 6 ].

Oriented fast and rotated BRIEF (ORB) [ 20 ].

These descriptors were then clustered using the k-means algorithm. The cluster centers act as the BoVW feature of our method. Applying this scheme using the mentioned descriptor algorithms provides four additional methods.

Thus, we have 4 di erent types of BoVW features. { Color+Texture A combination of the color and texture values represented as as a multi-feature vector. { Color+Texture+HoG A combination of the color, texture and HoG values represented as a multi-feature vector. 1. KNN assigns a label according to the majority labels of the K-nearest neighbor in space [ 7 ]. 2. SVM is a linear classi cation of the points in space into two distinct classes [ 11 ] . 3. LR is a probability model that predicts a binary output based on the model predictor variables [ 10 ]. 4. DBN constructs deep hierarchical layers based on a representation of the training data. The DBN performs an unsupervised pre-training learning and then sets the weights of the network in order to successfully use a supervised learning for classi cation [ 2 ] . Given a set of feature extraction and classi er methods our goal is to nd the 10 best combinations of feature-classi er that will provide the highest classi cation accuracy for the ve groups of X-ray images.

To reduce the number of combinations and the complexity of the problem, a two-class selection is rst applied to distinguish between X-ray images containing big and long bones (e.g., skull, arm and leg) and those with small and short bones (e.g., chest and abdomen bones). An additional motivation is to identify the features that will isolate di erent bone structures.

Next, we select the feature-classi er pairs that provide and average accuracy of greater than 90% in the two-class experiment and train them on the ve groups of X-ray images set to nd the 10 best combinations of feature-classi ers. We test each feature-classi ers pair in leave-one-out cross-validation, in which training is learned based on all cases besides a single case that is not part of the training process and used for testing.

The 10 best feature-classi er combination were then submitted to the challenge to be tested on an unlabeled test set of images released by the ImageCLEF organization [ 21 ] [ 3 ]. We use the OpenCV-Python library [ 15 ] for the feature extraction and Python scikit-Learn Machine learning tool [ 19 ] to examine the four selected classi ers. 3

Results

The data set released by ImageCLEF[ 21 ] [ 3 ] consists of 750 X-ray images: (a) 500 images were labeled images (100 images from each group) and were released for training purposes. The ve image groups are Head-Neck, Body, Upper-Limb, Lower-Limb and other (Fig 1). and (b) 250 unlabeled images released for the challenge evaluation and benchmarking.

We rst present the results of the 500 labeled images training set of X-ray images obtained in our two-step approach. Then, we present the results of the 250 unlabeled images as validated by the challenge organizers. 3.1

Training

In the two-class experiment to we use all four classi ers and nine sets of features, thus creating combination of 36(4 9) feature-classi ers. In the rst step we select feature-classi er pairs that provide an average accuracy greater than 90%. This selection reduced the number of combination from 36 to 16. The results for all 36 feature-classi ers are shown in Table 1:

The BoVW and HoG features exhibit low accuracy regardless of the classi er tested. The combination of color and texture yield high accuracy rate of 89-93%. Using color, texture and HoG features all together yields the highest average classi cation accuracy in all classi ers.

In the second step, 5-class, classi cation was preformed on all ve group. We select 16 combinations of features-classi er that yield an accuracy greater than 90%. The goal of this second step is to investigate the performance of the methods and to reduce the number of feature-classi ers to the best 10 .

Table 2 presents the results of the second step on the training set.

To sum-up, the 10 best features-classi er pairs are: 1. Color+Texture+HoG and KNN 2. Color+Texture+HoG and SVM 3. Color+Texture + HoG and DBN 4. Color+Texture+HoG and LR 5. Texture+HoG and KNN 6. Texture+HoG and SVM 7. Texture+HoG and DBN 8. Color+Texture and LR 9. Color and KNN 10. Texture and KNN. These 10 best combinations are marked in Table 2 with (*) were sent to the challenge organizer for evaluation.

Note that our best submission (73.2% accuracy) is lower than the best accuracy obtained in the ve group training set (80-85%) using the same combination of color texture and HoG features. The challenge results are similar to the average results achieved on the training set. This may indicate that the variability of the training dataset does not fully re ect the images variability of the challenge dataset. 4

Discussion

In this work we have evaluated four state-of-the-art classi er and nine sets of features, resulting in 36 combinations of feature-classi ers. Surprisingly, the simplest classi er, in terms of implementation and computational complexity, KNN, exhibited the best results. Moreover, despite the major trend of using Deep Belief Network (DBN) methods for many image-based classi cation problems, the use of DBN in our study exhibited reliable results only when applied to a large-scale dataset. However, suboptimal results were obtained when used on small-scale datasets (see Table 2). This is in line with the theory of DBN, which requires large-scale databases for reliable performance. The BoVW method was the least e cient method among all feature extraction schemes that have been tested.

From all feature extraction methods we evaluated, the color feature yielded the highest accuracy. This is surprising, considering that X-ray images are graylevel based images. This could be explained by the algorithm implementation, which extracts the gray-scale histogram features from di erent regions of the image and provides more speci c information and a better perspective on the distribution of the color. Another advantage of using the color features is low computational complexity as compared to the texture, HoG and BoVW. 5

Conclusions

This paper presents research on medical X-ray image classi cation. We analyze state-of-the-art classi ers and feature extraction methods for image classi cation. The image features that have been used include the color, texture, HoG and BoVW, which were used by our tested classi ers: SVM, KNN, LR and DBN. We used the datasets of the ImageCLEF-2015 [ 19 ] clustering of body part Xray challenge [ 3 ]: 500 X-ray images were used for training and 250 for testing. The highest classi cation accuracy results were obtained when using the intensity, texture and HoG features and the KNN classi er. This combination has an accuracy of 86% and 73% for the 500 training images and 250 test images, respectively.

Future work consists of examining an additional set of classi ers and extending the completeness of our algorithm to estimate the partitioning of the initial clusters into sub-clusters. For example, the upper-limb cluster can be further divided into the following categories: clavicle, scapula, humerus, radius, ulna and hand.

Future work consists of examining an additional set of classi ers and extending the completeness of our algorithm to estimate the partitioning of the initial clusters into sub-clusters, for example the upper-limb cluster can be farther divided into: Clavicle, Scapula, Humerus, Radius, Ulna, and Hand. 23. Weszka, J.S., Dyer, C.R., Rosenfeld, A.: A comparative study of texture measures for terrain classi cation. IEEE Transactions on Systems, Man and Cybernetics (4), 269{285 (1976) 24. Zare, M.R., Mueen, A., Seng, W.C.: Automatic medical x-ray image classi cation using annotation. Journal of digital imaging 27(1), 77{89 (2014)

References

1. Akgul, C.B., Rubin , D.L. , Napel , S. , Beaulieu , C.F. , Greenspan , H. , Acar , B. : Content-based image retrieval in radiology: Current status and future directions . Journal of Digital Imaging 24 ( 2 ), 208 { 222 ( 2011 )

2. Ali , K.H. , Wang , T. : Learning features for action recognition and identity with deep belief networks . In: 2014 International Conference on Audio, Language and Image Processing (ICALIP) ,. pp. 129 { 132 . IEEE ( 2014 )

3. Amin , M.A. , Mohammed , M.K. : Overview of the ImageCLEF 2015 medical clustering task . In: CLEF2015 Working Notes. CEUR Workshop Proceedings , CEURWS.org, Toulouse, France (September 8-11 2015 )

4. Avni , U. , Goldberger , J. , Sharon , M. , Konen , E. , Greenspan , H.: Chest x-ray characterization: from organ identi cation to pathology categorization . In: Proceedings of the international conference on Multimedia information retrieval . pp. 155 { 164 . ACM ( 2010 )

5. Bay , H. , Ess , A. , Tuytelaars , T. , Van Gool , L. : Speeded-up robust features (surf) . Computer vision and image understanding 110(3) , 346 { 359 ( 2008 )

6. Calonder , M. , Lepetit , V. , Strecha , C. , Fua , P. : Brief: Binary robust independent elementary features . In: Computer Vision{ECCV 2010 , pp. 778 { 792 . Springer ( 2010 )

7. Cunningham , P. , Delany , S.J. : k-nearest neighbour classi ers . Multiple Classi er Systems pp. 1 { 17 ( 2007 )

8. Dalal , N. , Triggs , B. : Histograms of oriented gradients for human detection . In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005 . vol. 1 , pp. 886 { 893 ( 2005 )

9. Deselaers , T. , Pimenidis , L. , Ney , H.: Bag-of-visual-words models for adult image classi cation and ltering . In: Pattern Recognition , 2008 . ICPR 2008 . 19th International Conference on. pp. 1 { 4 ( 2008 )

10. Dreiseitl , S. , Ohno-Machado , L. : Logistic regression and arti cial neural network classi cation models: a methodology review . Journal of biomedical informatics 35(5) , 352 { 359 ( 2002 )

11. Fan , Y. , Shen , D. , Davatzikos , C. : Classi cation of structural images via highdimensional image warping, robust feature extraction, and svm . In: Medical Image Computing and Computer-Assisted Intervention{MICCAI 2005 , pp. 1 { 8 . Springer ( 2005 )

12. Ghofrani , F. , Helfroush , M.S. , Rashidpour , M. , Kazemi , K. : Fuzzy-based medical x-ray image classi cation . Journal of medical signals and sensors 2 ( 2 ), 73 ( 2012 )

13. Guo , Z. , Zhang , L. , Zhang , D.: Rotation invariant texture classi cation using lbp variance (lbpv) with global matching . Pattern recognition 43(3) , 706 { 719 ( 2010 )

14. Haralick , R.M. , Shanmugam , K. , Dinstein , I.H. : Textural features for image classication . IEEE Transactions on Systems, Man and Cybernetics (6), 610 { 621 ( 1973 )

15. Howse , J.: OpenCV Computer Vision with Python . Packt Publishing Ltd ( 2013 )

16. Lehmann , T.M. , Schubert , H. , Keysers , D. , Kohnen , M. , Wein , B.B. : The irma code for unique classi cation of medical images . In: Medical Imaging 2003 . pp. 440 { 451 . International Society for Optics and Photonics ( 2003 )

17. Liu , X. , Shao , Z. , Liu , J.: Ontology-based image retrieval with sift features . In: First International Conference on Pervasive Computing Signal Processing and Applications (PCSPA) . pp. 464 { 467 . IEEE ( 2010 )

18. Mueen , A. , Zainuddin , R. , Baba , M.S.: Automatic multilevel medical image annotation and retrieval . Journal of digital imaging 21(3) , 290 { 295 ( 2008 )

19. Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M. , Prettenhofer , P. , Weiss , R. , Dubourg , V. , et al.: Scikit-learn: Machine learning in python . The Journal of Machine Learning Research 12 , 2825 { 2830 ( 2011 )

20. Rublee , E. , Rabaud , V. , Konolige , K. , Bradski , G.: Orb: an e cient alternative to sift or surf . In: IEEE International Conference on Computer Vision (ICCV). pp. 2564 { 2571 ( 2011 )

21. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , de Herrera , A.G.S. , Bromuri , S. , Amin , M.A. , Mohammed , M.K. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , del Mar Roldan Garc a, M.: General Overview of ImageCLEF at the CLEF 2015 Labs . Lecture Notes in Computer Science, Springer International Publishing ( 2015 )

22. Wang , S.L. , Liew , A. : Information-based color feature representation for image classi cation . In: International Conference on Image Processing (ICIP) . vol. 6 , pp. VI{353 ( 2007 )