Overview of the ImageCLEF 2015 Medical Classi cation Task

Abstract

This articles describes the ImageCLEF 2015 Medical Classi cation task. The task contains several subtasks that all use a data set of gures from the biomedical open access literature (PubMed Central). Particularly compound gures are targeted that are frequent in the literature. For more detailed information analysis and retrieval it is important to extract targeted information from the compound gures. The proposed tasks include compound gure detection (separating compound from other gures), multi{label classi cation (de ne all sub types present), gure separation ( nd boundaries of the sub gures) and modality classi cation (detecting the gure type of each sub gure). The tasks are described with the participation of international research groups in the tasks. The results of the participants are then described and analysed to identify promising techniques.

Keywords

• ImageCLEFmed
• compound gure detection
• multi{label classi cation
• gure separation
• modality classi cation

-

The amount and availability of biomedical literature has increased considerably due to the advent of the Internet [ 1 ]. The task of medical doctors has on the other hand not become simpler as the amount of information to review for taking decisions has become overwhelming. Despite this growing complexity, physicians would use services that improve their understanding of an illness even if these involve more cognitive e ort than in standard practice [ 2 ]. Images in biomedical articles can contain highly relevant information for a speci c information need and can accelerate the search by ltering out irrelevant documents [ 3 ]. As a consequence image{based retrieval has been proposed as a way of improving access to the medical literature and complement text search [ 4, 5 ].

Image classi cation can play an important role in improving the image{ based retrieval of the biomedical literature, as this helps to lter out irrelevant information from the retrieval process. Many images in the biomedical literature (around 40% [ 6 ]) are compound gures (see Figure 1), so determining the gure type is not clear as several di erent types of gures can be present in a single compound gure. ? Alba G. Seco de Herrera is currently working at the National Library of Medicine (NLM/NIH), Bethesda, MD, USA

Information retrieval systems for images should be capable of distinguishing the parts of compound gures that are relevant to a given query, as usually queries are limited to a single modality [ 7 ]. Compound gure detection and multi{label classi cation are therefore a required rst step to focus retrieval of images. Some le formats, such as DICOM (Digital Imaging and Communications in Medicine), contain metadata that can be used to lter images by modality, but this information is lost when using images from the biomedical literature where images are stored as JPG, GIF or PNG les. In this case caption text and visual appearance are key to understanding the content of the image and whether or not it is a compound gure. Both types of information, text and visual, are complementary to each other and can help managing the multi{label classi cation [ 8 ]. From the standpoint of information retrieval and classi cation of compound images and associated text, the current systems could greatly bene t from the use of multi{label classi cation approaches [ 9 ] by a) de ning models that can use the dependencies between the extracted images; b) de ning models that can express the importance of a label in a compound gure. In addition, compound gures are naturally redundant sources of information with natural dependencies occurring between the di erent regions of the image.

Retrieval systems can fail if they are not speci cally designed to work with compound gures and partial relevance. Identi cation of each subpart of the gures can improve retrieval accuracy by enabling comparison of gures with lower noise levels [ 10 ].

To promote research on this eld a medical classi cation task is proposed in the context of the ImageCLEF 2015 lab [ 11 ]. This paper describes this benchmark in detail.

This article is structured as follows: Section 2 presents an overview of the participants and of the datasets used in the competition. Section 3 discusses the results with respect to the selected datasets. Finally, Section 4 concludes the paper and presents relevant future work for the next edition of ImageCLEF.

Tasks, Data Sets, Ground Truth, Participation

This section describes the main scenario of the benchmark including the data used, the tasks, ground truthing and participation. 2.1

The Tasks in 2015

There were four subtasks in 2015:

{ compound gure detection; { compound gure separation; { multi{label classi cation; { sub gure classi cation.

This section gives an overview of each of the four subtasks.

Compound Figure Detection Compound gure identi cation is a required rst step to make compound gures from the literature accessible for further analysis. Therefore, the goal of this subtask is to identify whether a gure is a compound gure or not. The task makes training data available containing compound and non{compound gures from the biomedical literature. Figure 2 shows an example of a compound and a non{compound gure.

(a) Compound gure.

(b) non{compound gure.

Figure Separation This task was rst introduced in 2013 and the same evaluation methodology is used in 2015 [ 6 ]. The goal of this task is to separate the compound gures into sub gures using separation lines. Figure 3 shows a compound gure which is separated into sub gures by blue lines. In 2015, a larger number of compound gures was distributed compared to the previous years.

(b) Compound gure with separation lines. Multi{label Classi cation The fundamental di erence with respect to compound gure separation resides in the fact that the compound gure is not separated into sub gures, but it is rather used entirely to perform a scene classication task. The intuition behind this approach resides in the fact that sub gures in medical papers are usually assembled because they add complementary information concerning the aeticle topic (see Figure 4). In this sense, much work was performed in the multi{label classi cation community [ 12 ] and many algorithms already exist to classify multi{label problems. A multi{label dataset in medical imaging was never considered before to the best of our knowledge. More formally, this problem can be expressed as follows:

Let X be the domain of observations and let L be the nite set of labels. Given a training set T = f(x1; Y1); (x2; Y2); :::; (xn; Yn)g (xi 2 X; Yi L) i.i.d. drawn from an unknown distribution D, the goal is to learn a multi{label classi er h : X ! 2L. However, it is often more convenient to learn a real{valued scoring function of the form f : X L ! R. Given an instance xi and its associated label set Yi, a working system will attempt to produce larger values for labels in Yi than those that are not in Yi, i.e. f (xi; y1) > f (xi; y2) for any y1 2 Yi and y2 2= Yi. By the use of the function f ( ; ), a multi{label classi er can be obtained: h(xi) = fyjf (xi; y) > ; y 2 Lg, where is a threshold to infer from the training set. The function f ( ; ) can also be adapted to a ranking function rankf ( ; ), which maps the outputs of f (xi; y) for any y 2 L to f1; 2; :::; jLjg such that if f (xi; y1) > f (xi; y2) then rankf (xi; y1) < rankf (xi; y2).

Multi{label performance measures di er from single label ones. Following the same approach presented in [ 12 ], the Hamming Loss is proposed as the evaluation measure for multi{label learning in ImageCLEF.

More formally let a testing set S = f(x1; Y1); (x2; Y2); :::; (xm; Ym)g.

Hamming loss: evaluates how many times an observation{label pair is misclassi ed. The score is normalized between 0 and 1, where 0 is the best: hlossS (h) = 1 Xm jh(xi)4Yij : m i=1 jLj (1) where 4 represents the symmetric di erence.

Sub gure Classi cation The sub gure classi cation task is a variation of the multi{label classi cation task in which the sub gures contained in the multi{ label gures are provided separately for classi cation. The main reason to proceed in this way is to provide two matched dataset that researchers can use to compare multi{label classi cation of the full compound image versus taking each single image in the compound image and classify it. 2.2

Datasets

In 2015, the dataset was a subset of the of the full ImageCLEF 2013 dataset [ 6 ], which is a part of PubMed Central1 containing in total over 1,700,000 images 1 http://www.ncbi.nlm.nih.gov/pmc/ in 2014. The distributed subset contains a total of 20,867 gures. The training set contains 10,433 gures and the test set 10,434 gures. Each of these two sets contains 6,144 compound gures and 4289{4290 non{compound gures. The entire dataset is used for the compound gure detection task.

6,784 of the compound gures are used for the gure separation task. 3,403 gures are distributed in the training set and 3,381 in the test set.

A subset of these images containing 1,568 images are labelled for the multi{ label learning task. These images are also distributed as a training set (containing 1,071 gures) and a test set (containing 497 gures). The labels were assigned using the same class hierarchy as the one used for the ImageCLEF 2012 [ 13 ] and 2013 [ 6 ] modality classi cation task. A slight di erence is that in 2015 the class \compound" is not included because only the non{compound parts can be labels with all compound images being split.

Figure 5 shows the ImageCLEF 2015 class hierarchy where the class codes with descriptions are the following ([Class code] Description): [GGEN ] Gene sequence [GGEL] Chromatography, Gel [GCHE] Chemical structure [GM AT ] Mathematics, formula [GN CP ] Non{clinical photos [GHDR] Hand{drawn sketches

Finally, each gure from the multi{label classi cation task is separated into sub gures and each of the sub gures is labelled. As a result, 4,532 sub gures were released in the training set and 2,244 in the test set. To link the multi{label classi cation and the sub gure separation tasks, the gure IDs were related. If the gure ID is \1297-9686-42-10-3", then the corresponding sub gure IDs are \1297-9686-42-10-3-1", \1297-9686-42-10-3-2", \1297-9686-42-10-3-3" and \1297-9686-42-10-3-4".

In addition to the gures, the articles of the gures are provided to allow for the use of textual information. 2.3

Participation

Over seventy groups registered for the medical classi cation tasks and obtained access to the data sets. Eight of the registered groups submitted results to the medical classi cation tasks.

7 runs were submitted to the compound gure detection task, 12 runs to the multi{label classi cation task, 5 runs to the gure separation task and 16 runs to the sub gure separation task.

The following groups submitted at least one run: { AAUITEC (Institute of Information Technology, Alpen{Adria University of

Klagenfurt, Austria); { FHDO BCSG (FHDO Biomedical Computer Science Group, University of

Applied Science and Arts, Germany); { BMET (Institute of Biomedical Engineering and Technology, University of

Sydney, Australia) { CIS UDEL (Computer & Information Sciences, University of Delaware Newark,

USA) { CMTECH (Cognitive Media Technologies Research Group, Pompeu Fabra

University, Spain) { IIS (Institute of Computer Science, University of Innsbruck, Austria) { MindLab (Machine Learning, Perception and Discovery Lab, National University of Colombia, Colombia); { NLM (National Library of Medicine, USA). 3

Results

This section describes the results obtained by the participants for each of the subtasks. 3.1

Compound Figure Detection

Very good results were obtained for the compound gure detection task, reaching up to 85% for HDO BCSG as seen in Table 1. Table 1 contains the results obtained by the two participants of the compound gure detection task.

Group Run Run Type Accuracy FHDO BCSG task1 run2 mixed sparse1 mixed 85.39 FHDO BCSG task1 run1 mixed stemDict mixed 83.88 FHDO BCSG task1 run3 mixed sparse2 mixed 80.07 FHDO BCSG task1 run4 mixed bestComb mixed 78.32 FHDO BCSG task1 run6 textual sparseDict textual 78.34 CIS UDEL exp1 visual 82.82

FHDO BCSG task1 run5 visual sparseSift visual 72.51

FHDO BCSG [ 14 ] achieved best results with an accuracy of 85:39% using a multi{modal approach. FHDO BCSG applied a combination of visual features and text. With respect to visual features they focused on features detecting the border of the gures and a bag{of{keypoints. The bag{of{words approach is used for text classi cation using the provided gure caption. They also proposed two runs applying only either visual or text information obtaining in general lower results that applying multi{modal approaches.

CIS UDEL [ 15 ] obtained best results when using only visual information achieving an accuracy of 82:82%. A combination of connected component analysis of sub gures and peak region detection is used. 3.2 In 2015, two groups participated in the gure separation task. Table 2 shows the results achieved. Best results were obtained by NLM [ 16 ]. NLM distinguished two types of compound images: stitched multipanel gure and multipanel gures with a gap. A manual selection of stitched multipanel gures from the whole dataset is rst carried out. Then, two approaches are used. Best results are obtained by \run2 whole" where stitched multipanel gure separation is combined with both image panel separation and label extraction. \run2 whole" achieved an accuracy of 79:85% by combining stitched multipanel gure separation with panel separation.

AAUITEC [ 17 ] submitted three runs. Each run used a speci c separator line detection based on \bands" (run \aauitec gsep band") or \edges" (run \aauitec gsep edge"). Best results achieved an accuracy of 49:40% when using a combination of both detection types. A recursive algorithm is used starting by classifying the images as illustrations or not. Depending on the type of image a speci c separator line detection is used, based on \bands" or \edges", respectively. 3.3

Multi{label Classi cation

With respect to the multi{label classi cation task, there were two participating groups, IIS [ 18 ] and Mindlab2. Quite interestingly, none of the two participants decided to apply standard multi{label classi cation algorithms [ 12 ] such ask Multi{Label K{nearest neighbours (MLKNN) or Binary Relevance Support 2 https://sites.google.com/a/unal.edu.co/mindlab/ Vector Machines (BR{SVM), but rather decided to come up with two new solutions to the problem. Table 3 presents the results of the runs submitted by the two groups.

ISS applied Khronker decomposition to nd a set of lters of the gure as features for a maximum margin layer classi er in which the multi{label task is mapped on a dual problem of the standard margin optimization with SVMs. To achieve this the authors consider the possibility of modelling the problem by introducing an additional kernel matrix calculated starting from the vector of labels associated to the compound gures.

The Mindlab approach is based on building a visual representation by means of deep convolutional neural networks, by relying on the theory of transfer learning which is based in the ability of a system to recognize and apply knowledge learned in previous domains to novel domains, which share some commonality. For this task, Mindlab used the Yangqing Jia et al. (Ca e) [ 19 ] pretrained network to represent gures. Ca e is an open source implementation of the winning convolutional network architecture of the ImageNet challenge. For the label assignment the authors proceeded as follows: Once the prediction is made a distribution of the classes is obtained and used to annotate only those with a score above 0.5. Furthermore, in the second run when a sample concept that scores above 0.5 does not exist, the two top labels are assumed as relevant.

The scores of the two presented approaches are quite close in the result, but the best result was achieved by Mindlab with a Hamming Loss of 0.5 as seen in Table 3. Three groups participated in the sub gure classi cation task and the results can be seen in Table 4. The FHDO BCSG group achieved the best classi cation accuracy (67.60%) by using textual and visual features as described in [ 14 ]. FHDO BCSG also achieved the best result when only using visual features (60.91%). The method reuses existing techniques and fuses visual and textual features. Given the high dimensionality of the data, principal component analysis (PCA) is used to reduce the dimensionality to a subset of components explaining the variance of the dataset. SVMs and Random Forests are used together for the classi cation.

The CMTECH group used a descriptor based on covariance of the visual features associated with the sub gures [ 20 ]. The advantage of the proposed descriptor, being based on covariance, is to be robust with respect to noise. In this sense, the feature vector provided has 11 features. As claimed by the authors, this is particularly interesting because the matrix de ned with this approach lies within the Riemann manifold, where images providing similar features are close in the Riemann Geometry. From this standpoint the authors also speci ed the conditions under which two images can be considered close.

The paper proposes a new approach to classifying images and the approach performs well without using a complicated classi er, which demonstrates the need to de ne good features.

The last approach was proposed by the BMET group [ 21 ]. The article present a convolutional neural networks (CNN) used for the sub gure classi cation. As speci ed by the authors, the CNN selected is a simpli ed version of the CNN used in LeNet{5. The major claim of the paper concerns the ability of the network to extract features in an unsupervised way and without the aid of domain knowledge. The main drawback of the paper is that the method used for the evaluation takes a long time to converge and the results were calculated on only partly optimized models. The classi cation results re ect this fact. Despite the relatively low results, the BMET contribution presents an interesting experimentation concerning the task proposed in ImageCLEF and it would certainly be interesting to see how these methods can be extended to improve the preliminary results obtained.

Conclusions

In this paper the results of the medical classi cation task ImageCLEF 2015 competition are presented. As in 2013, the challenge involved a subtask on the separation of compound gures from the biomedical literature. This year, three new subtasks were introduced: a subtask on detection of compound gures to identify whether a gure is compound or not; a multi{label subtask in which the sub gure labels of a compound gure need to be determined without separating the sub gures; and a sub gure classi cation challenge in which the separated sub gures are classi ed, following a traditional modality classi cation approach.

In the rst year of this task, eight groups participated submitting forty runs. The participants present a variety of techniques for the problems on compound gure analysis. This is a wide problem than can make a large number of subgures available for image search applications. The large number of di erent techniques leading to good results shows that many techniques can be used for the problem and the detailed optimization is most often responsible for obtaining a very good performance.

Acknowledgements

We would like to thank the support received from the European Science Foundation (ESF) via the ELIAS project.

References

1. Lu , Z. : PubMed and beyond: a survey of web tools for searching biomedical literature . Database (Oxford) 2011 ( 2011 )

2. Kannampallil , T.G. , Franklin , A. , Mishra , R. , Almoosa, Khalid F.and Cohen , T. , Patel , V.L. : Understanding the nature of information seeking behavior in critical care: Implications for the design of health information technology . Arti cial Intelligence in Medicine 57(1) ( 2013 ) 21 { 29

3. Muller, H., Foncubierta-Rodr guez, A ., Lin , C. , Eggel , I. : Determining the importance of gures in journal articles to nd representative images . In: SPIE Medical Imaging . Volume 8674 . ( 2013 ) 86740I{ 86740I { 9

4. Muller, H., Michoux , N. , Bandon , D. , Geissbuhler , A. : A review of content{based image retrieval systems in medicine{clinical bene ts and future directions . International Journal of Medical Informatics 73 ( 1 ) ( 2004 ) 1 { 23

5. Akgul, C. , Rubin , D. , Napel , S. , Beaulieu , C. , Greenspan , H. , Acar , B. : Content{ based image retrieval in radiology: Current status and future directions . Journal of Digital Imaging 24 ( 2 ) ( 2011 ) 208 { 222

6. Garc a Seco de Herrera, A. , Kalpathy-Cramer , J. , Demner Fushman , D. , Antani , S. , Muller, H.: Overview of the ImageCLEF 2013 medical tasks . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

7. Markonis , D. , Holzer , M. , Dungs , S. , Vargas , A. , Langs , G. , Kriewel , S. , Muller, H.: A survey on visual information search behavior and requirements of radiologists . Methods of Information in Medicine 51 ( 6 ) ( 2012 ) 539 { 548

8. Clinchant , S. , Ah-Pine , J. , Csurka , G.: Semantic combination of textual and visual information in multimedia retrieval . In: Proceedings of the 1st ACM International Conference on Multimedia Retrieval. ICMR '11 , ACM ( 2011 ) 44 : 1 { 44 : 8

9. Zhang, M.L. , Zhou , Z.H.: A review on multi{label learning algorithms . IEEE Transactions on Knowledge and Data Engineering 99 ( 2013 )

10. Muller, H., Kalpathy-Cramer , J. , Demner-Fushman , D. , Antani , S. : Creating a classi cation of image types in the medical literature for visual categorization . In: SPIE Medical Imaging . ( 2012 )

11. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Garc a Seco de Herrera, A. , Bromuri , S. , Amin , M.A. , Kazi Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , Roldan Garc a , M.d.M.: General overview of ImageCLEF at the CLEF 2015 labs . In: Working Notes of CLEF 2015. Lecture Notes in Computer Science . Springer International Publishing ( 2015 )

12. Zhang , M.L. , Zhou , Z.H.: A review on multi{label learning algorithms . IEEE Transactions on Knowledge and Data Engineering 26 ( 8 ) ( 2014 ) 1819 { 1837

13. Muller, H., Garc a Seco de Herrera, A. , Kalpathy-Cramer , J. , Demner Fushman , D. , Antani , S. , Eggel , I. : Overview of the ImageCLEF 2012 medical image retrieval and classi cation tasks . In: Working Notes of CLEF 2012 ( Cross Language Evaluation Forum) . (September 2012 )

14. Pelka , O. , Friederich , C.M. : FHDO Biomedical Computer Science Group at Medical Classi cation Task of ImageCLEF 2015 . In: Working Notes of CLEF 2015 . ( 2015 )

15. Wang , X. , Shatkay , H. , Kambhamettu , C. : CIS UDEL working notes on ImageCLEF 2015 : Compound gure detection task . In: Working Notes of CLEF 2015 . ( 2015 )

16. Santosh , K. , Xue , Z. , Antani , S. , Thoma , G.: NLM at ImageCLEF2015: Biomedical multipanel gure separation . In: Working Notes of CLEF 2015 . ( 2015 )

17. Taschwer , M. , Marques , O. : AAUITEC at ImageCLEF 2015: Cmpound gure separation . In: Working Notes of CLEF 2015 . ( 2015 )

18. Rodr guez-Sanchez, A. , Fontanella , S. , Piater , J. , Szedmak , S.: IIS at ImageCLEF 2015: Multi{label classi cation task . In: Working Notes of CLEF 2015 . ( 2015 )

19. Jia , Y. , Shelhamer , E. , Donahue , J. , Karayev , S. , Long , J. , Girshick , R.B. , Guadarrama , S. , Darrell , T.: Ca e: Convolutional architecture for fast feature embedding . In: Proceedings of the ACM International Conference on Multimedia, MM'14 . ( 2014 ) 675 { 678

20. Cirujeda , P. , Binefa , X. : Medical image classi cation via 2D color feature based covariance descriptors . In: Working Notes of CLEF 2015 . ( 2015 )

21. Lyndon , D. , Kumar , A. , Kim , J. , Leong , P.H.W. , Feng , D. : Convolutional neural networks for sub gure classi cation . In: Working Notes of CLEF 2015 . ( 2015 )