Overview of the ImageCLEF 2016 Medical Task

Abstract

ImageCLEF is the image retrieval task of the Conference and Labs of the Evaluation Forum (CLEF). ImageCLEF has historically focused on the multimodal and language{independent retrieval of images. Many tasks are related to image classi cation and the annotation of image data as well. The medical task has focused more on image retrieval in the beginning and then retrieval and classi cation tasks in subsequent years. In 2016 a main focus was the creation of meta data for a collection of medical images taken from articles of the the biomedical scienti c literature. In total 8 teams participated in the four tasks and 69 runs were submitted. No team participated in the caption prediction task, a totally new task. Deep learning has now been used for several of the ImageCLEF tasks and by many of the participants obtaining very good results. A majority of runs was submitting using deep learning and this follows general trends in machine learning. In several of the tasks multimodal approaches clearly led to best results.

Keywords

• ImageCLEFmed
• compound gure detection
• multi{label classi cation
• gure separation
• modality classi cation
• caption detection

-

ImageCLEF has organized image retrieval evaluation campaigns since 2003 and a medical task was added in 2004 [ 1, 2 ]. With a focus on multimodal and language{ independent retrieval of images, the databases used have evolved strongly over the years and many of the datasets have gotten larger as well. Several medical tasks have been organized over the years [3{5], ranging from the classi cation of medical images to retrieval of single images or entire cases. This year's tasks are an evolution from the tasks that were organized in 2015 [ 6 ]. The main objective with the current data sets and tasks is to make the large amount of visual content that is shared in the biomedical open access literature available in an easier way by generating meta data. PubMed Central4 (PMC) makes a large 4 http://www.ncbi.nlm.nih.gov/pmc/ amount of currently over 4 million articles available including text and gures in a structured form. The collection is growing strongly with over 200'000 articles being added in 2014 alone and with a quickly increasing tendency. Basically no metadata are available for the gures besides the gure captions and global information on the articles including global MeSH (Medical Subject Headings) terms. A major problem is that about half of the available gures contain more than one sub gure, so are compound or multi{pane gures. The tasks in 2016 aims at rst detecting, whether a gure is a compound gure, then trying to separate the compound gures into their parts or extract image type information for all sub gures of a compound gure. Then, a modality classi cation tries to detect the image type, that ranges from medical modalities (e.g. X{ray, MRI, CT) to general image types such as graphs and ow charts. All these tasks can help to generate metadata for the almost 4 million images available via PMC. Including the extracted sub gures this will likely amount to over 10 million medical gures that are available and currently only little exploited.

This article rst describes the ve tasks that were organized in 2016, then describes the data sets, ground truth and participation. The conclusions summarize the main lessons learned from the evaluation campaign. Finally, a little outlook is given into the limitations and a possible future of the task. 2 2.1

Tasks, Data Sets, Ground Truth, Participation

The Tasks in 2016

Five subtasks were organized in 2016: { compound gure detection; { compound gure separation; { multi{label classi cation with image types; { sub gure classi cation into image types; { caption prediction.

This section gives an overview of each of the ve subtasks.

Compound Figure Detection As a rst step for the retrieval of compound gures and its sub gures, compound gure detection is necessary. This subtask was introduced in 2015 and the goal is to identify whether a gure is a compound gure or not (see Figure 1). The task is not easy, as compound gures can or not have dominant sub gures and do not always have clear separating lines. The subtask provides a set of compound and non{compound gures from the biomedical literature of PMC.

Compound Figure Separation The goal of this subtask is to separate the compound gures into sub gures to be able to work with the sub gures independently. This subtask was introduced in 2013. Figure 2 shows a simple example of a compound gure separated by blue lines. There are many more challenging examples where the separating lines are not straight or where a large number of sub gures is put into a single gure.

(a) Compound gure.

(b) Non{compound gure.

Multi{label Classi cation Compound gures are often an aggregate belonging to multiple classes, as they can show multiple perspectives concerning a medical problem. This aggregation is not random, a compound gure is an aggregation of sub{ gures representing a relationship with a clear semantic meaning. Goal of the task was to see whether it is possible to learn the components of a gure without learning from single images representing the image types, but from other labelled compound gures. Techniques such as deep learning should work well on such tasks.

Figure 3 is an example of such an image in which multiple perspectives of the same set of cells are shown.

Bromuri et al. [ 7 ] formulates the general multi{label problem as follows: Let X be the domain of observations and let L be the nite set of labels. Given a training set T = f(x1; Y1); (x2; Y2); :::; (xn; Yn)g (xi 2 X; Yi L) i.i.d. drawn from an unknown distribution D, the goal is to learn a multi-label classi er h : X ! 2L. However, it is often more convenient to learn a real{valued scoring function of the form f : X L ! R. Given an instance xi and its associated label set Yi, a working system will attempt to produce larger values for labels in Yi than those that are not in Yi, i.e. f (xi; y1) > f (xi; y2) for any y1 2 Yi and y2 2= Yi. By the use of the function f ( ; ), we can obtain a multi{label classi er: h(xi) = fyjf (xi; y) > ; y 2 Lg, where is a threshold to infer from the training set. The function f ( ; ) can also be adapted to a ranking function rankf ( ; ), which maps the outputs of f (xi; y) for any y 2 L to f1; 2; :::; jLjg such that if f (xi; y1) > f (xi; y2) then rankf (xi; y1) < rankf (xi; y2).

Multi{label performance measures are generally di erent from those used in the single label tasks. In [ 6 ], we introduced the Hamming loss. The Hamming loss evaluates how many times an observation{label pair is misclassi ed. The score lies between 0 and 1, where 0 is the best: hlossS (h) = 1 Xm jh(xi)4Yij : m i=1 jLj (1) (2) 4 represents the symmetric di erence.

In 2016, we also introduce the mean of the F{Measures of all the labels belonging to a gure, where for each i label we de ne the F{Measure to be: F {M easurei = 2 precisioni recalli precisioni + recalli This allows us to understand if there is an unbalanced distribution of the labels that leads to the classi er over tting to the majority class.

Sub gure Classi cation The gure classi cation task was already run in a slightly di erent con guration from 2011 to 2013. In 2015 and 2016 the subtask focuses on the modality classi cation of sub gures extracted from the compound gures distributed for the multi{label classi cation subtask. This subtask aims to classify gures into the 30 classes of the hierarchy shown in Figure 4. The class codes with descriptions are the following ([Class code] Description): { [Dxxx] Diagnostic images: [DRxx] Radiology (7 categories): [DRU S] Ultrasound [DRM R] Magnetic Resonance [DRCT ] Computerized Tomography [DRXR] X{Ray, 2D Radiography [DRAN ] Angiography [DRP E] PET [DRCO] Combined modalities in one image { [DV xx] Visible light photography (3 categories): [DV DM ] Dermatology, skin [DV EN ] Endoscopy [DV OR] Other organs { [DSxx] Printed signals, waves (3 categories): [DSEE] Electroencephalography [DSEC] Electrocardiography [DSEM ] Electromyography { [DM xx] Microscopy (4 categories): [DM LI] Light microscopy [DM EL] Electron microscopy [DM T R] Transmission microscopy [DM F L] Fluorescence microscopy { [D3DR] 3D reconstructions (1 category) { [Gxxx] Generic biomedical illustrations (12 categories): [GT AB] Tables and forms [GP LI] Program listing [GF IG] Statistical gures, graphs, charts [GSCR] Screenshots [GF LO] Flowcharts [GSY S] System overviews [GGEN ] Gene sequence [GGEL] Chromatography, Gel [GCHE] Chemical structure [GM AT ] Mathematics, formula [GN CP ] Non{clinical photos [GHDR] Hand{drawn sketches

(a) Light microscopy sub gure.

(b) Fluorescence microscopy sub gure. (c) Light microscopy sub gure.

(d) Fluorescence microscopy sub gure.

Caption Prediction Thanks to the technical advances of cloud computing many large and data intensive applications have become possible. Modern GPUs (Graphical Processing Units) have made massively parallel computing of simple operations possible and lead to a revival of methods based on neural networks with more complex and deeper architectures. There has been a strong hype around such Deep Learning techniques [ 9 ]. One of the most successful application of Deep Learning is that of deep captioning [ 10 ].

The purpose of the caption prediction task is to mimic the ability of a medical professional to recognize gures in a medical text and provide a description of these gures. We believe that this is be an important task that can lead to future applications in medical image information retrieval. 2.2

Datasets

The dataset used in this task is a subset of images contained in articles from the biomedical literature extracted from the PMC. The trainining sets were obtained merging the training and test sets of the ImageCLEFmed 2015 subtasks [ 6 ]. Therefore, in 2016 a larger number of gures were distributed than in 2015. Image captions were also provided in addition to all images. For the compound gure detection subtask 21,000 gures were labelled as compound gures or non{compound for the training set and 3,456 for the test set. A subset of the compound gures of the compound gure detection subtask was distributed to be separated into sub gures for the gure separation subtask. 6,783 and 1,614 were distributes as training and test sets, respectively. In 2016, more stitched compound gures were added making the subtask more complicated. For the multi{label classi cation, a subset of the compound gures were distributed containing 1,568 in the training set and 1,083 in the test set. These compound gures were separated into sub gures and distributed for the subgure classi cation subtask. The naming of the sub gures was done in a way that if the compound gure ID is "1297-9686-42-10-3", then the corresponding sub gure IDs are "1297-9686-42-10-3-1", "1297-9686-42-10-3-2", "1297-9686-4210-3-3" and "1297-9686-42-10-3-4" on the case of four sub gures. This resulted in 6,776 sub gures in the training set and 4,166 sub gures in the test set.

The data distributed to the participants for the caption prediction subtask involved 10,000 images from diagnostic imaging category and relative captions. We gured that diagnostic images might be of the highest relevance in this context. The test set comprised another 10,000 diagnostic images but the captions were not included for these images. 2.3

Participation

72 groups registered and obtained access to the data. The same number of groups as in 2015 submitted results to the medical task (8 groups in total). Groups participated from four continents, so the regional spread was high.

Despite that the number of groups that registered in 2016 being smaller than in 2015, the number of submitted runs increased. 15 runs were submitted to the compound gure detection task, 3 runs to the multi{label classi cation task, 9 runs to the gure separation task and 42 runs to the sub gure separation task. There were unfortunately no participants in the new caption prediction task.

The following groups submitted at least one run: { BMET (Institute of Biomedical Engineering and Technology, University of

Sydney, Australia); { CIS UDEL (Computer & Information Sciences, University of Delaware, Newark,

USA); { DUTIR (Department of Computer Science and Engineering, Dalian University of Technology, China.)*; { FHDO BCSG (FHDO Biomedical Computer Science Group, University of

Applied Science and Arts, Dortmund, Germany); { IPL (Athens University of Economics and Business, Greece); { MLKD (Department of Informatics, Aristotle University of Thessaloniki,

Greece)*; { NOVASearch (NOVA LINCS, Department of Computer Science Faculty of

Science and Technology, University NOVA of Lisbon, Portugal)*; { NWPU (Northwestern Polytechnical University, China)*; Participants marked with a star had not participated in the medical task in 2015.

Results

This section provides the results obtained by the participants in each of the subtasks. groups participated in the sub gure detection subtask obtaining an accuracy of up to 92:70% using a multi{modal approach, followed by a visual approach submitted for the same group, DUTIR. DUTIR applied deep convolutional neural networks on vectors trained on the words of all captions using Word2Vec. Five deep convolutional neural networks were also applied on the resized images.

CIS UDEL [ 11 ], also achieved its best results using a multi{modal approach. In the textual approach the group extracted a set of delimiters from the captions. In the visual approach, CIS UDEL applied the output of a gure separation approach to classify the gures into compound and non{compound. Finally, the results were fused using several methods, such as a logical union and a decision tree classi er.

MLKD submitted a single run in this subtask achieving the best results using only text information. The textual approach is based on the caption and on the text citing the gure inside the article followed by the use of a random forest classi er. Table 2 shows the results for the gure separation subtask. In 2016 only one group participated in the compound gure separation task, CIS UDEL [ 11 ], achieving very good results up to an accuracy of 84:43%. Similar results to 2015 were obtained although the di culty of the subtask was increased in 2016. CIS UDEL applied a connected component analysis to separate the compound gures. A post{processing step was applied to avoid over{fragmentation. This year two groups submitted runs for the multi{label classi cation task. The BMET group achieved the best Hamming loss (0.0131) and both groups achieved a F-Measure of 0.32. Table 3 summarises these results. The BMET group also submitted a working notes article [ 12 ], highlighting the use of Deep Learning and CNNs to classify the images with multiple labels. 3.4

Sub gure Classi cation

This subtask was the most popular task in 2016 with seven groups participating. The results achieved by the participants are shown in Table 4. As in the compound detection task best results were obtained by a multi{modal approach, followed by visual and textual approaches. BCSG [ 13 ] achieved the best accuracy of 88:43% by applying multiple visual features and deep convolutional neural networks (CNN). Figure captions and paper full text were also used for the classi cation. To remove unimportant visual words information gain is used for feature selection. MLKD achieved the best results using a text analysis approach. Similar approaches as in the compound gure detection subtask were applied. Best results on visual approaches were obtained by BCSG followed by IPL [ 14 ]. IPL adopted various state{of{the{art visual features, such as, Bag{of{ Visual|Words computed with pyramid{histogram{of{visual{word descriptors and quad{treebag{of{colors. BMET [ 12 ] applied a method similar to the one they used for the multi{label classi cation task based on CNNs. NWPU also based its method on deep CNNs. A hierarchical approach was used that rst classi ed the gures into diagnostic images and generic biomedical illustrations through a deep CNN. Then, two other deep CNNs were trained to nish the classi cation of diagnostic images and generic biomedical illustrations,respectively. CIS UDEL [ 11 ] also applied a hierarchical classi er using multiple visual descriptors. Neural networks were used as a classi er. Finally, NovaSearch [ 15 ] also applied three di erent CNN models in their approaches. 4

Conclusions

In 2016, the ImageCLEF medical task proposed 5 subtasks. One of the subtasks was organized for the rst time, the caption prediction subtask. Unfortunately, no participants nally submitted results to the task. This year, more gures were added to the database in the other four subtasks that had already been run in the past. In total, there were eight participants who submitted results, the same number as in 2015 but more runs were submitted in 2016 compared to 2015. The best accuracy obtained was very good in three of the tasks: 92:70% in the compound detection subtask using a multi{modal approach; 84:43% in the gure separation subtask using a visual approach; and 88:43% in the sub gure detection subtask using a multi{modal approach. For the multi{label subtask, the BMET group obtained 0:0135 Hamming loss and a F{Measure of 0:32 using a deep learning approach based on CNNs.

The clear novelty and trend in 2016 is the use of neural network models or deep learning for classi cation subtasks obtaining very good results in general. CIS UDEL was the only participant of the 2016 gure separation subtask separating the images using a connected component analysis. The main novelty concerning the multi{label task in 2016 was the use of ne{tuned CNNs to perform the multi{label classi cation.

Acknowledgements

This research was supported in part by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC).

References

1. Clough , P. , Muller, H., Sanderson , M.: The CLEF 2004 cross{language image retrieval track . In Peters, C. , Clough , P. , Gonzalo , J. , Jones , G.J.F. , Kluck , M. , Magnini , B., eds.: Multilingual Information Access for Text, Speech and Images: Result of the fth CLEF evaluation campaign . Volume 3491 of Lecture Notes in Computer Science (LNCS)., Bath , UK, Springer ( 2005 ) 597 { 613

2. Villegas , M. , Muller, H., Gilbert , A. , Piras , L. , Wang , J. , Mikolajczyk , K. , Garc a Seco de Herrera, A. , Bromuri , S. , Amin , M.A. , Kazi Mohammed , M. , Acar , B. , Uskudarli , S. , Marvasti , N.B. , Aldana , J.F. , Roldan Garc a , M.d.M.: General overview of ImageCLEF at the CLEF 2015 labs . In: Working Notes of CLEF 2015. Lecture Notes in Computer Science . Springer International Publishing ( 2015 )

3. Muller, H., Garc a Seco de Herrera, A. , Kalpathy-Cramer , J. , Demner Fushman , D. , Antani , S. , Eggel , I. : Overview of the ImageCLEF 2012 medical image retrieval and classi cation tasks . In: Working Notes of CLEF 2012 ( Cross Language Evaluation Forum) . (September 2012 )

4. Garc a Seco de Herrera, A. , Kalpathy-Cramer , J. , Demner Fushman , D. , Antani , S. , Muller, H.: Overview of the ImageCLEF 2013 medical tasks . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

5. Kalpathy-Cramer , J. , Garc a Seco de Herrera, A. , Demner-Fushman , D. , Antani , S. , Bedrick , S. , Muller, H.: Evaluating performance of biomedical image retrieval systems: Overview of the medical image retrieval task at ImageCLEF 2004{2014 . Computerized Medical Imaging and Graphics 39 ( 0 ) ( 2015 ) 55 { 61

6. Garc a Seco de Herrera, A. , Muller, H., Bromuri , S. : Overview of the ImageCLEF 2015 medical classi cation task . In: Working Notes of CLEF 2015 ( Cross Language Evaluation Forum) . (September 2015 )

7. Bromuri , S. , Zu erey, D., Hennebert , J. , Schumacher , M.I.: Multi{label classi - cation of chronically ill patients with bag of words and supervised dimensionality reduction algorithms . Journal of Biomedical Informatics 51 ( 2014 ) 165 { 175

8. Muller, H., Kalpathy-Cramer , J. , Demner-Fushman , D. , Antani , S. : Creating a classi cation of image types in the medical literature for visual categorization . In: SPIE Medical Imaging . ( 2012 )

9. Bengio , Y. : Deep learning and cultural evolution . In: Genetic and Evolutionary Computation Conference , GECCO, Vancouver, BC, Canada ( 2014 ) 1{ 2

10. Mao , J. , Xu , W. , Yang , Y. , Wang , J. , Yuille , A.L. : Deep captioning with multimodal recurrent neural networks (m-rnn) . CoRR abs/1412 .6632 ( 2014 )

11. Li , P. , Sorensen , S. , , Kolagunda , A. , Jiang , X. , Wang , X. , Kambhamettu , C. , Shatkay , H.: UDEL CIS working notes in ImageCLEF 2016 . In: CLEF2016 Working Notes. CEUR Workshop Proceedings, Evora, Portugal, CEUR-WS.org ( September 5-8 2016 )

12. Kumar , A. , Lyndon , D. , Kim , J. , Dagan , F. : Sub gure and multi{label classi cation using a ne{tuned convolutional neural network . In: CLEF2016 Working Notes. CEUR Workshop Proceedings , Evora, Portugal, CEUR-WS.org ( September 5-8 2016 )

13. Koitka , S. , Friedrich , C.M.: Traditional feature engineering and deep learning approaches at medical classi cation task of ImageCLEF 2016 . In: CLEF2016 Working Notes. CEUR Workshop Proceedings, Evora, Portugal, CEUR-WS.org ( September 5-8 2016 )

14. Valavanis , L. , Kalamboukis , T. : IPL at CLEF 2016 medical task . In: CLEF2016 Working Notes. CEUR Workshop Proceedings , Evora, Portugal, CEUR-WS.org ( September 5-8 2016 )

15. Semedo , D. , Magalha~es, J.a.: NovaSearch at ImageCLEFmed 2016 sub gure classication task . In: CLEF2016 Working Notes. CEUR Workshop Proceedings , Evora, Portugal, CEUR-WS.org ( September 5-8 2016 )