Overview of the ImageCLEF 2013 medical tasks

Abstract

In 2013, the tenth edition of the medical task of the ImageCLEF benchmark was organized. For the rst time, the ImageCLEFmed workshop takes place in the United States of America at the annual AMIA (American Medical Informatics Association) meeting even though the task was organized as in previous years in connection with the other ImageCLEF tasks. Like 2012, a subset of the open access collection of PubMed Central was distributed. This year, there were four subtasks: modality classi cation, compound gure separation, image{based and case{based retrieval. The compound gure separation task was included due to the large number of multipanel images available in the literature and the importance to separate them for targeted retrieval. More compound gures were also included in the modality classi cation task to make it correspond to the distribution in the full database. The retrieval tasks remained in the same format as in previous years but a larger number of tasks were available for image{based and case{based tasks. This paper presents an analysis of the techniques applied by the ten groups participating 2013 in ImageCLEFmed.

Keywords

• ImageCLEFmed
• modality classi cation
• compound gure separation
• image{based retrieval
• case{based retrieval

-

ImageCLEF1 [ 1 ] is the image retrieval track of the Cross Language Evaluation Forum (CLEF). ImageCLEFmed is part of ImageCLEF focusing on medical images [2{7].

In the 10th edition of the medical task, the workshop is for the rst time organized outside of Europe at the annual AMIA2 (American Medical Informatics Association) meeting. The same format as in 2012 was followed and a new task was added, the compound gure separation. Characterisation of compound gures is often di cult, as they can contain features of various image types. Focusing search on the sub gures can lead to better results. In 2013, the modality 1 http://www.imageclef.org/ 2 http://www.amia.org/amia2013/ classi cation task also included a larger amount of compound gures to make the task more realistic and correspond to the distribution in the database. The four tasks of 2013 are: { modality classi cation; { compound gure separation; { image{based retrieval; { case{based retrieval.

The paper is organized as follows. Section 2 describes the ImageCLEFmed tasks in more detail as well as the participation in each of the tasks. Section 3 presents the main results of the tasks and compares results within the participating groups and the techniques employed. Section 4 concludes the paper. 2

Participation, Data Sets, Tasks, Ground Truth

This section describes the four tasks organized in ImageCLEFmed 2013. The datasets and the ground truth provided for the evaluation campaign are explained in detail. 2.1

Participation

Like 2012, over sixty groups registered for the medical tasks and obtained access to the data sets. Ten of the registered groups submitted results to the medical tasks compared to 17 in 2012 with a total of 166 valid runs submitted, slightly fewer runs than in 2012. The smaller number of participants and submitted runs can be due to a change in the evaluation schedule of CLEF 2013 and may also be due to the fact that the event will be organized outside of Europe.

51 runs were submitted to the modality classi cation task, 4 runs to the compound gure separation task, 9 runs to the image retrieval task and 45 runs to the case-based retrieval task. As in previous years, the number of runs per group was limited to ten per subtask. The following groups submitted at least one run: { AAUITEC (Institute of Information Technology, Alpen{Adria University of

Klagenfurt, Austria)*; { CITI (Center of Informatics and Information Technology, Portugal)*; { DEMIR (Dokuz Eylul University, Turkey); { FCSE (Faculty of Computer Sciences and Engineering, University of Ss Cyril and Methodius, Macedonia); { IBM Multimedia Analytics (United States); { IPL (Athens University of Economics and Business, Greece); { ITI (Image and Text Integration Project, NLM, United States); { medGIFT (University of Applied Sciences Western Switzerland, Switzerland); { MiiLab (Medical Image Information Laboratory, Shanghai Advanced Research Institute,China)*; { SNUMedInfo (Medical Informatics Laboratory, Seul National University, Republic of Korea)*; Participants marked with a star had not participated in the medical task in 2012. 2.2

Datasets

In ImageCLEFmed 2013, the same database as in 2012 was supplied to the participants. The database contains over 300,000 images of 75,000 articles of the biomedical open access literature that allow free redistribution of the data. The ImageCLEFmed database is a subset of PubMed Central3 containing in total over 1.5 million images of over 600,000 articles. 2.3

Modality Classi cation

The modality classi cation task was rst introduced in 2010. The goal of this task is to classify the images into medical modalities and other images types, such as Computed Tomography, xray or general graphs. A modality hierarchy of 38 classes of which 31 appear in the data was used [ 8 ]. Using the modality information, the retrieval results could often be improved in the past by ltering our non{relevant image types [ 9 ]. The same hierarchy as in ImageCLEFmed 2012 was used (see Figure 1). In 2013 a larger number of compound gures than in ImageCLEFmed 2012 were provided in the training and test data sets. The current distribution corresponds to that in the PubMed Central data set, much closer to reality than in previous years.

The class codes with descriptions are the following ([Class code] Description): { [COM P ] Compound or multipane images (1 category) { [Dxxx] Diagnostic images: [DRxx] Radiology (7 categories): [DRU S] Ultrasound [DRM R] Magnetic Resonance [DRCT ] Computerized Tomography [DRXR] X{Ray, 2D Radiography [DRAN ] Angiography [DRP E] PET [DRCO] Combined modalities in one image { [DV xx] Visible light photography (3 categories): [DV DM ] Dermatology, skin [DV EN ] Endoscopy [DV OR] Other organs { [DSxx] Printed signals, waves (3 categories): [DSEE] Electroencephalography [DSEC] Electrocardiography [DSEM ] Electromyography

3 http://www.ncbi.nlm.nih.gov/pmc/

In the ImageCLEFmed 2012 data set [ 7 ] between 40% and 60% of the gures are compound or multipanel gures. Making the content of the compound gures accessible for targeted search can improve retrieval accuracy. For this reason the detection of compound gures and their separation into sub gures is considered an important task. Examples for compound gures can be seen in Figure 2. (a) Mixed modalities in a single gure (b) Graphs and microscopy images in a single gure The data set used in the ImageCLEF 2013 compound gure separation task are all gures of the data set from the biomedical litertature. 2,967 compound gures were selected from the complete data set after a manual classi cation of images into compound and other gures. This subset was randomly split into two parts: a training set containing 1,538 images and a testing set with 1,429 images.

The ground truth for the dataset was generated in a semi{automatic way, using a two{step approach: rst, an automated separation process (using the technique described in [ 10 ]) was run on both image sets in order to obtain a general overview of the sub gures. The automatic results were then manually corrected. Missing lines were added and incorrect lines removed, whereas often the lines were only slightly changed. Separating lines rather than bounding boxes were used to separate sub gures. The evaluation then required to have a minimum overlap between the ground truth and the data supplied by the groups in their runs.

The terminology used in the evaluation is: { The term gure, F , refers to a compound gure as a whole. { A sub gure, fi, represents a part (or panel) of a gure. The ground truth for the gure F consists of a set of KGFT sub gures f1; : : : ; fKGFT . { The word candidate, cj , refers to the data being evaluated against the ground truth. Separation of gure F consists of a set of KCF candidates c1; : : : ; cKCF . A brief summary of the evaluation algorithm for a given gure F is as follows: (a) Perfect score (b) Not enough candidates (c) Too many candidates { The score SF is computed based on the number of correct candidates,

CcForrect. { For each sub gure fi de ned in the ground truth the best matching candidate sub gure will be determined. Only one candidate is used in case there are several matches. { The main metric used to compare sub gures is the overlap between a candidate sub gure and the ground truth. To be considered a valid match the overlap between a candidate sub gure and a sub gure from the ground truth must correspond to at least 66% of the candidate's size. If the best candidate is an acceptable match, the number of correctly matched gures CcForrect will be incremented. Since only one candidate sub gure can be assigned to each of the sub gures from the ground truth, CcForrect KGFT . { The maximum score for the gure is 1 and the normalisation factor used to compute the score will be the maximum between the number of sub gures in the ground truth KGFT and the number of candidate sub gures KCF .

SF =

CcForrect max(KGFT ; KCF ) :

CcForrect = KCF = KGFT : Therefore the maximum score is obtained only when the number of candidates KCF is equal to the number of sub gures in the ground truth KGFT and all of them are correctly matched: The image{based retrieval task is the classical medical retrieval task, similar to those organized each year since 2004 with the target unit being the image. In 2013, 35 queries were given to the participants so more than in previous years. The 22 queries used in 2012 [ 7 ] were part of the 35 queries that all contain text (in English, Spanish, French and German) with 1{7 sample images for each query. As in previous years, the queries were classi ed into textual, mixed and semantic, based on the methods that are expected to yield the best results. 2.6

Case{Based Retrieval

The case{based retrieval task has been running since 2009. In this task, a case description, with patient demographics, limited symptoms and test results including imaging studies, is provided (but not the nal diagnosis). As in previous years, the goal is to retrieve cases including images that might best suit the provided case description. This year the 26 topics distributed in 2012 were also part of the 35 nal topics. Each of the topics was accompanied by one or two images. 3

Results

This section describes the results of ImageCLEF 2013. Runs are ordered based on the tasks (modality classi cation, compound gure separation, image{based and case{based retrieval) and the techniques used (visual, textual, mixed). In 2013, several groups used the ImageCLEF 2012 [ 7 ] database to optimize the parameters [11{13]. 3.1

Modality Classi cation Results

Techniques Used for Visual Classi cation The IBM team achived the best results in the visual classi cation.FCSE [ 12 ] was the second best group (77.14%) using a spatial pyramid in combination with dense sampling using an opponentSIFT descriptor for each image patch. Finally, Support Vector Machines (SVM) with 2 kernel were used as a classi er. As in 2012, multiple features were extracted from the images, most frequently color and edge directivity descriptors (CEDD) [ 11, 13, 14, 16, 17 ], fuzzy color and texture histogram (FCTH) [ 11, 13, 14, 16, 17 ] and scale{invariant feature transform (SIFT) variants [ 11, 12, 15 ]. Several classi ers were explored by the participants such as SVM [ 12, 14, 15, 17 ], k{nearest neighbour (k{nn) [ 11, 15 ] or class{centroid{based techniques [ 17 ].

Techniques Used for Classi cation Based on Text In 2012, only the ITI

team [ 18 ] submitted runs for the textual modality classi cation task. In 2013, seven groups submitted textual results. A variety of techniques was employed using systems as Terrier IR4 [ 12, 13 ], Lucene5 [ 11, 16 ] or Essie [ 14 ].

Techniques Used for Multimodal Classi cation Eight groups submitted

multimodal runs, ve more than in 2012. The groups fused the techniques described above for visual and textual classi cation with a variety of fusion techniques, leading to the best results overall with multimodal techniques. 3.2

Compound Figure Separation Results

Three groups participated in the rst year of the compound gure separation task (see Table 2). MedGIFT [ 11 ] achieved the best result in one of its runs but it simply serves as a point of reference, since it was also used when the separating lines were drawn [ 10 ] and thus has an advantage over other techniques. ITI [ 14 ] achived 69.27% using a combination of gure caption analysis, panel border detection and panel label recognition. FCSE [ 12 ] got 68.59% using an unsupervised algorithm based on a breadth{ rst search strategy using only visual information. Finally, medGIFT [ 11 ] submitted a second run which was not strictly designed for gure separation but provided a point of comparison. The run used a region detection algorithm mainly focused on volumetric medical image retrieval [ 19 ] with 46.82% of accuracy showing the possibility to use such techniques. Nine groups submitted image{based runs in 2013. The best results in terms of mean average precision (MAP) were obtained by ITI [ 14 ] using multimodal methods. The same group also obtained best results in 2012. The best textual run achieved the same MAP than the best multimodal run (0.3196). As in previous years, visual approaches achieved much lower results than the textual and multimodal techniques. Most of the techniques used in the retrieval task were also used for the modality classi cation task and are described in Section 3.1.

4 http://terrier.org/

5 http://lucene.apache.org/

Visual Retrieval Eight groups submitted 28 visual runs (see Table 3). DEMIR [ 13 ] achieved the best position in terms of MAP applying a classi cation algorithm. In addition to the techniques used in the modality classi cation task, some participants split and rescaled the images [ 17, 16 ]. Borda{fuse methods were also used [ 20 ]. textual retrieval task (see Table 4). ITI [ 14 ] achieves the best results with a combination of two queries using Essie. The participants explored a variety of retrieval techniques mostly described in Section 3.1. FCSE [ 12 ] proposed a concept{scape approach matching the text data to medical concepts. Multimodal Retrieval Only three groups submitted runs in the multimodal task (see Table 5). As in 2012, ITI [ 14 ] submitted the run with the highest MAP. For this run the group used the same method as the best textual run achieving exactly the same results. Mixed approaches combined the above textual and visual approaches using early [ 11, 14, 17 ] and late [ 11, 13, 14, 16 ] fusion strategies. In 2013, the case{based retrieval task became more popular with seven groups submitting 42 runs. More groups than in previous years used visual and multimodal techniques. Textual runs achived the best results and visual runs obtained lower results than the textual and multimodal runs.

Visual Retrieval The results using visual retrieval on the case{based task are shown in Table 6. CITI [ 16 ] achived the best result outperforming the second best result by a factor of ten in terms of MAP. This group extracted a set of descriptors for 6 6 image grid. Run Name Group MAP GM-MAP bpref P10 P30 FCT SEGHIST 6x6 LBP CITI 0.0281 0.0009 0.0335 0.0429 0.0238 medgift visual no lter casebased medGIFT 0.0029 0.0001 0.0035 0.0086 0.0067 medgift visual close casebased medGIFT 0.0029 0.0001 0.0036 0.0086 0.0076 medgift visual pre x casebased medGIFT 0.0029 0.0001 0.0036 0.0086 0.0067 nlm-se-case-based-visual ITI 0.0008 0.0001 0.0044 0.0057 0.0057 Textual Retrieval Table 7 shows that SNUMedInfo [ 20 ] team achieved the best MAP (0:2429) in its rst participation. SNUMedInfo used an external corpus (MEDLINE6) for robust and e ective expansion term inference. CITI [ 16 ] achieved close results using MeSH expansion. ITI [ 14 ] and FCSE [ 12 ] incorporate UMLS (Uni ed Medical Language System) concepts. In general, the groups used the same techniques or very similar techniques compared to the ad{hoc image retrieval task.

Multimodal Retrieval Three groups submitted multimodal runs, combining of visual and textual techniques. As in the visual case{based task, the CITI [ 16 ] team achieved the best results in terms of MAP (see Table 8). A rank{based fusion was applied in their approach improving existing algorithms by a small margin. 4

Conclusions

After one decade of running the ImageCLEF medical task, in 2013 ImageCLEFmed is organized at the annual AMIA meeting in the form of a workshop. The task had 10 groups submitting 166 valid runs to the four subtasks. The main novelty in 2013 was the inclusion of a new task, the compound gure separation

6 http://www.nlm.nih.gov/bsd/pmresources.html

Run Name Group MAP GM-MAP bpref P10 P30 FCT CB MM rComb CITI 0.1608 0.0779 0.1426 0.1800 0.1257 medgift mixed no lter casebased medGIFT 0.1467 0.0883 0.1318 0.1971 0.1457 nlm-se-case-based-mixed ITI 0.0886 0.0303 0.0926 0.1457 0.0962 FCT CB MM MNZ CITI 0.0794 0.0035 0.0850 0.1371 0.0810 task. In its rst year three groups joined this complex task. More compound gures were included into the modality classi cation, so the training and test set are more di cult and correspond to the reality of the database, now. The other two tasks, image and case based retrieval, remained in the same format as in previous years but had a larger number of retrieval topics.

As in previous years, visual, textual or multimodal techniques can all perform best depending on the situation. For the modality classi cation, a mixed run achieved the best accuracy. For the image{based retrieval task, the highest MAP was achieved by a multimodal run. In the case{based retrieval task, textual techniques obtained the best results. Finally, for the compound gure separation task only visual and mixed techniques were explored, with visual techniques leading to best results.

In 2013, many groups used ImageCLEFmed 2012 database to optimize the parameters. Many of the techniques used had already been employed in previous years. This shows the utility of past campaigns, which provide databases as well as information regarding tools used by other participants. ImageCLEF conducts participative research and experimentation among free and reusable collections and has shown an important impact in visual medical information retrieval. 5

Acknowledgements

We would like to thank the EU FP7 projects Khresmoi (257528) and PROMISE (258191) for their support.

References

1. Caputo , B. , Muller, H., Thomee , B. , Villegas , M. , Paredes , R. , Zellhofer , D. , Goeau , H. , Joly , A. , Bonnet , P. , Martinez Gomez , J. , Garcia Varea , I. , Cazorla , C. : Imageclef 2013: the vision, the data and the open challenges . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

2. Hersh , W. , Muller, H., Kalpathy-Cramer , J. , Kim , E. , Zhou , X. : The consolidated ImageCLEFmed medical image retrieval task test collection . Journal of Digital Imaging 22 ( 6 ) ( 2009 ) 648 { 655

3. Muller, H., Clough , P. , Deselaers , T. , Caputo , B., eds.: ImageCLEF { Experimental Evaluation in Visual Information Retrieval . Volume 32 of The Springer International Series On Information Retrieval. Springer, Berlin Heidelberg ( 2010 )

4. Muller, H., Kalpathy-Cramer , J. , Jr. , C.E.K. , Hatt , W. , Bedrick , S. , Hersh , W. : Overview of the ImageCLEFmed 2008 medical image retrieval task . In Peters, C. , Giampiccolo , D. , Ferro , N. , Petras , V. , Gonzalo , J. , Pen~as, A. , Deselaers , T. , Mandl , T. , Jones , G. , Kurimo , M., eds. : Evaluating Systems for Multilingual and Multimodal Information Access { 9th Workshop of the Cross-Language Evaluation Forum . Volume 5706 of Lecture Notes in Computer Science (LNCS)., Aarhus , Denmark ( September 2009 ) 500 { 510

5. Muller, H., Kalpathy-Cramer , J. , Eggel , I. , Bedrick , S. , Radhouani , S. , Bakke , B. , Kahn , J.C.E. , Hersh , W. : Overview of the clef 2009 medical image retrieval track . In: Proceedings of the 10th international conference on Cross-language evaluation forum: multimedia experiments . CLEF'09 , Berlin, Heidelberg, Springer{ Verlag ( 2010 ) 72 { 84

6. Kalpathy-Cramer , J. , Muller, H., Bedrick , S. , Eggel , I. , Garc a Seco de Herrera, A. , Tsikrika , T. : The CLEF 2011 medical image retrieval and classi cation tasks . In: Working Notes of CLEF 2011 ( Cross Language Evaluation Forum) . (September 2011 )

7. Muller, H., Garc a Seco de Herrera, A. , Kalpathy-Cramer , J. , Demner Fushman , D. , Antani , S. , Eggel , I. : Overview of the ImageCLEF 2012 medical image retrieval and classication tasks . In: Working Notes of CLEF 2012 ( Cross Language Evaluation Forum) . (September 2012 )

8. Muller, H., Kalpathy-Cramer , J. , Demner-Fushman , D. , Antani , S. : Creating a classi cation of image types in the medical literature for visual categorization . In: SPIE medical imaging . ( 2012 )

9. Tirilly , P. , Lu , K. , Mu , X. , Zhao , T. , Cao , Y. : On modality classi cation and its use in text{based image retrieval in medical databases . In: Proceedings of the 9th International Workshop on Content{Based Multimedia Indenxing. CBMi2011 ( 2011 )

10. Chhatkuli , A. , Markonis , D. , Foncubierta-Rodr guez , A., Meriaudeau , F. , Muller, H.: Separating compound gures in journal articles to allow for sub gure classi - cation . In: SPIE, Medical Imaging. ( 2013 )

11. Garc a Seco de Herrera, A. , Markonis , D. , Schaer , R. , Eggel , I. , Muller, H.: The medGIFT group in ImageCLEFmed 2013 . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

12. Kitanovski , I. , Dimitrovski , I. , Loskovska , S.: FCSE at medical tasks of ImageCLEF 2013 . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

13. Ozturkmenoglu , O. , Ceylan , N.M. , Alpkocak , A. : DEMIR at ImageCLEFmed 2013: The e ects of modality classi cation to information retrieval . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

14. Simpson , M.S. , You , D. , Rahman , M.M. , Demner-Fushman , D. , Antani , S. , Thoma , G.: ITI's participation in the 2013 medical track of ImageCLEF . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

15. Zhou , X. , Han, M. , Song , Y. , Li , Q. : Fast ltering techniques in medical image classi cation and retrieval . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

16. Mour~ao, A. , Martins , F. , Magalha~es, J.a.: NovaSearch on medical ImageCLEF 2013 . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

17. Stathopoulos , S. , Lourentzou , I. , Kyriakopoulou , A. , Kalamboukis , T. : IPL at CLEF 2013 medical retrieval task . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )

18. Simpson , M.S. , You , D. , Rahman , M.M. , Demmer-Fushman , D. , Antani , S. , Thoma , G.: ITI's participation in the ImageCLEF 2012 medical retrieval and classi cation tasks . In: Working Notes of CLEF 2012 . ( 2012 )

19. Foncubierta-Rodr guez , A., Muller, H., Depeursinge , A. : Region{based volumetric medical image retrieval . In: SPIE Medical Imaging: Advanced PACS{based Imaging Informatics and Therapeutic Applications . ( 2013 )

20. Sungbin , C. , Lee , J. , Cho , J.: SNUMedinfo at ImageCLEF 2013: Medical retrieval task . In: Working Notes of CLEF 2013 ( Cross Language Evaluation Forum) . (September 2013 )