Overview of the ImageCLEF 2012 medical image retrieval and classi cation tasks

Abstract

The ninth edition of the ImageCLEF medical image retrieval and classi cation tasks was organized in 2012. A subset of the open access collection of PubMed Central was used as the database in 2012, using a larger number of over 300'000 images than in 2011. As in previous years, there were three subtasks: modality classi cation, image{based and case{based retrieval. A new hierarchy for article gures was created for the modality classi cation task. The modality detection could be one of the most important lters to limit the search and focus the results sets. The goal of the image{based and the case{based retrieval tasks were similar compared to 2011 adding mainly complexity. The number of groups submitting runs has remained stable at 17, with the number of submitted runs remaining roughly the same with 202 (207 in 2011). Of these, 122 were image{based retrieval runs, 37 were case{based runs while the remaining 43 were modality classi cation runs. Depending on the exact nature of the task, visual, textual or multimodal approaches performed better.

-

The CLEF 20121 labs continue the CLEF tradition of community–based benchmarking and complement it with workshops on emerging topics on information retrieval evaluation methodologies. Following the format introduced in 2010, two forms of labs were offered: labs could either be run as benchmarking activities campaign–style during the ten month period preceding the conference, or as workshop–style labs that explore possible benchmarking activities and provide a means to discuss information retrieval evaluation challenges from various perspectives.

ImageCLEF2 [ 1–4 ] is part of CLEF and focuses on cross–language and language– independent annotation and retrieval of images. ImageCLEF has been organized since 2003. Four tasks were offered in 2012: 1 http://www.clef2012.org/

2 http://www.imageclef.org/

{ medical image classification and retrieval; { photo annotation and retrieval (large–scale web, Flickr, and personal photo tasks); { plant identification; { robot vision.

The medical image classification and retrieval task in 2012 is a use case of the PROMISE3 network of excellence and is supported by the project. This task covers image modality classification and image retrieval with visual, semantic and mixed topics in several languages using a data collection from the biomedical literature. This year, there are three types of tasks in the medical image classification and retrieval task: { modality classification; { image–based retrieval; { case–based retrieval.

This article presents the main results of the tasks and compares results between the various participating groups and the techniques employed. 2

Participation, Data Sets, Tasks, Ground Truth

This section describes the details concerning the set–up and the participation in the medical retrieval task in 2012. 2.1

Participation

In total over 60 groups registered for the medical tasks and obtained access to the data sets. ImageCLEF in total had over 200 registrations in 2012, with a bit more than 30% of the groups submitting results. 17 of the registered groups submitted results to the medical tasks, the same number as in previous years. The following groups submitted at least one run: { Bioingenium (National University of Colombia, Colombia)*; { BUAA AUDR (BeiHang University, Beijing, China); { DEMIR (Dokuz Eylul University, Turkey); { ETFBL (Faculty of Electrical Engineering Banja Luka, Bosnia and Herzegovina)*; { FINKI (University in Skopje, Macedonia)*; { GEIAL (General Electric Industrial Automation Limited, United States)*; { IBM Multimedia Analytics (United States)*; { IPL (Athens University of Economics and Business, Greece); { ITI (Image and Text Integration Project, NLM, United States)*; { LABERINTO (Universidad de Huelva, Spain); { lambdasfsu (San Francisco State University, United States)*;

3 http://www.promise-noe.eu/

{ medGIFT (University of Applied Sciences Western Switzerland, Switzerland); { MIRACL (Higher Institute of Computer Science and Multimedia of Sfax,

Tunisia)*; { MRIM (Laboratoire d’Informatique de Grenoble, France); { ReDCAD (National School of Engineering of Sfax, Tunisia)*; { UESTC (University of Electronic Science and Technology, China); { UNED–UV (Universidad Nacional de Educacion a Distancia and Universitat de Val`encia, Spain); Participants marked with a star had not participated in the medical retrieval task in 2011.

A total of 202 valid runs were submitted, 43 of which were submitted for modality detection, 122 for the image–based topics and 37 for the case–based topics. The number of runs per group was limited to ten per subtask and case– based and image–based topics were seen as separate subtasks in this view. 2.2

Datasets

In ImageCLEFmed 2012, a larger database than 2011 was provided using the same types of images and the same journals. The database contains over 300,000 images of 75’000 articles of the biomedical open access literature that allow free redistribution of the data. The ImageCLEF database is a subset of the PubMed Central4 database containing in total over 1.5 million images. PubMedCentral contains all articles in PubMed that are open access but the exact copyright for redistribution varies among the journals. 2.3

Modality Classi cation

Previous studies [ 5, 6 ] have shown that imaging modality is an important information on the image for medical retrieval. In user–studies [ 7 ], clinicians have indicated that modality is one of the most important filters that they would like to be able to limit their search by. Many image retrieval websites (Goldminer, Yottalook) allow users to limit the search results to a particular modality [ 8 ]. Using the modality information, the retrieval results can often be improved significantly [ 9 ].

An improved ad–hoc hierarchy with 31 classes in the sections compound or multipane images, diagnostic images and generic biomedical illustrations was created based on the existing data set [ 10 ]. The following hierarchy was used for the modality classification, more complex than the classes in ImageCLEF 2011.

The class codes with descriptions are the following ([Class code] Description): { [COM P ] Compound or multipane images (1 category) { [Dxxx] Diagnostic images:

[DRxx] Radiology (7 categories):

4 http://www.ncbi.nlm.nih.gov/pmc/

[GP LI] Program listing [GF IG] Statistical figures, graphs, charts [GSCR] Screenshots [GF LO] Flowcharts [GSY S] System overviews [GGEN ] Gene sequence [GGEL] Chromatography, Gel [GCHE] Chemical structure [GM AT ] Mathematics, formulae [GN CP ] Non–clinical photos [GHDR] Hand–drawn sketches For this hierarchy 1,000 training images and 1,000 test images were provided to the participants. Labels for the training images were known whereas labels for the test images were distributed after the results submission, only. 2.4

Image{Based Topics

The image–based retrieval task is the classic medical retrieval task, similar to the tasks organized from 2004 to 2011 where the query targets are single images. Participants were given a set of 22 textual queries (in English, Spanish, French and German) with 1–7 sample images for each query. The queries were classified into textual, mixed and semantic queries, based on the methods that are expected to yield the best results.

The topics for the image–based retrieval task were based on a selection of queries from search logs of the Goldminer radiology image search system [ 11 ]. Only queries occurring 10 times or more (about 200 queries) were considered as candidate topics for this task. A radiologist assessed the importance of the candidate topics, resulting in 50 candidate topics that were checked for at least occurring a few times in the database. The resulting 22 queries were then distributed among the participants and example query images were selected from a past collection of ImageCLEF [ 12 ]. 2.5

Case{Based Topics

The case–based retrieval task was first introduced in 2009. This is a more complex task but one that we believe is closer to the clinical workflow. In this task, 30 case descriptions with patient demographics, limited symptoms and test results including imaging studies were provided (but not the final diagnosis). The goal was to retrieve cases including images that a physician would judge as relevant for differential diagnosis. Unlike the ad–hoc task, the unit of retrieval here was a case, not an image. The topics were created form an existing medical case database. Topics included a narrative text and several images. 2.6

Relevance Judgements

The relevance judgements were performed with the same on–line system as in 2008–2011 for the image–based topics as well as case–based topics. For the case– based topics, the system displays the article title and several images appearing in the text (currently the first six, but this can be configured). Judges were provided with a protocol for the process with specific details on what should be regarded as relevant versus non–relevant. A ternary judgement scheme was used again, wherein each image in each pool was judged to be “relevant”, “partly relevant”, or “non–relevant”. Images clearly corresponding to all criteria were judged as “relevant”, images for which relevance could not be accurately confirmed were marked as “partly relevant” and images for which one or more criteria of the topic were not met were marked as “non–relevant”. Judges were instructed in these criteria and results were manually verified during the judgement process. As in previous years, judges were recruited by sending out an email to current and former students at OHSU’s (Oregon Health and Science University) Department of Medical Informatics and Clinical Epidemiology. Judges, primarily clinicians, were paid a small stipend for their services. Many topics were judged by two or more judges to explore inter–rater agreements and its effects on the robustness of the rankings of the systems. 3

Results

This section describes the results of ImageCLEF 2012. Runs are ordered based on the tasks (modality classification, image–based and case–based retrieval) and the techniques used (visual, textual, mixed).

17 teams submitted at least one run in 2012, the same number than in 2011. 3.1

Modality Classi cation Results

The results of the modality classification task are compared using classification accuracy. With a higher number of classes, this task was more complex than in previous years. As seen in Table 1, the best result were obtained by the IBM Multimedia Analytics [ 13 ] group using visual methods (69.6%). In previous years combining visual and textual methods most often provided the best results. The best run using visual methods had a slightly better accuracy than the best run using mixed methods (66.2%) by the medGIFT group [ 14 ]. Only a single group submitted text–based results that performed worse than the average of all runs. The best run using textual methods alone obtained a much lower accuracy (41.3%).

Techniques Used for Visual Classi cation The IBM Multimedia Analytics

team used multiple features extracted from a set of image granularities with Kernel approximation fusion in the best run [ 13 ]. A variety of image processing techniques were explored by the other participants. Multiple features were extracted from the images, most frequently scale–invariant feature transform (SIFT) variants [ 13–17 ], GIST (gist is not an acronym) [ 13 ], local binary patterns (LBP) [ 13, 17 ], edge and color histograms [ 13, 16–19 ] and gray value histograms [ 16 ]. Several texture features were also explored such us Tamura [ 13, 16–18 ], Gabor filters [ 16–18 ], Curvelets [ 13 ], a granulometric distribution function [ 19 ] and spatial size distribution [ 19 ]. For recognizing compound images ITI used an algorithm that detects sub–figure labels and the border of each sub–figure within a compound image [ 17 ].

k–Nearest Neighbors (kNN) [ 14 ], a logistic regression model [ 19 ] or multi– class support vector machines (SVMs) [ 13, 16–18 ] were employed to classify the images into the 32 categories. Only one group used hierarchical classification [ 17 ].

Three groups augmented the training data with additional examples for the categories [ 13, 14, 18 ]. Not all details of the training data expansion are clear and it needs to be assured that purely visual runs such as the best–performing run only use visual features for the training data set expansion.

Techniques Used for Classi cation Based on Text ITI [17] was the only

group submitting a run for the textual modality classification task. They extracted the unified medical language system (UMLS) synonyms using the Essie system [ 20 ] and used it for term expansion when indexing enriched citations with Lucene/SOLR5.

Techniques Used for Multimodal Classi cation Three groups submitted

multimodal runs for the classification task. The medGIFT team obtained the best results [ 14 ] (66.2%). The approach fuses Bag–of–Visual–Words (BoVW) features based on SIFT and Bag–of–Colors (BoC) representing local image colors using reciprocal rank fusion. All three groups used techniques based on the Lucene search engine for the textual part and simple fusion techniques. 3.2 Image{Based Retrieval Results 13 teams submitted 36 visual, 54 textual and 32 mixed runs for the image– based retrieval task. The best result in terms of mean average precision (MAP) was obtained by ITI [ 17 ] using multimodal methods. The second best run was a purely textual run submitted by Bioingenium [ 21 ]. As in previous years, visual approaches achieved much lower results than the textual and multimodal techniques.

Visual Retrieval 36 of the 122 submitted runs used purely visual techniques. As seen in Table 2, DEMIR [ 22 ] achieved the best MAP, 0:0101, performing explicit grade relevance feedback. The second best run (M AP = 0:0092) was achieved also by DEMIR without applying relevance feedback. They combined color and edge directivity (CEDD) using combSUM [ 17, 21–23 ]. Bioingenium [ 21 ]

5 http://lucene.apache.org/

submitted the third best run (M AP = 0:0073). They used a spatial pyramid extension for the CEDD.

In addition to the techniques used in the modality classification task, participants used visual features such as visual MPEG–7 features [ 22, 24 ], scalable color [ 24 ] and brightness/texture directionality histograms (BTDH) [ 22, 23 ]. Other techniques used are fuzzy color and texture histograms (FCTH) [ 17, 22, 23 ] and color layout (CL) [ 22, 24 ]. To extract these features most participants used tools such as Rummager [ 22, 23 ] or LIRE (Lucene Image Retrieval Engine) [ 24 ]. Textual Retrieval Table 3 shows that the Bioingenium [ 21 ] team achieved the best MAP using textual techniques (0:2182). They developed their own implementation of Okapi–BM25. The BUAA AUDR [ 18 ] team achieved the second best textual result (0:2081) with a run indexed with MeSH for query expansion and modality prediction. The remaining participants explored a variety of retrieval techniques such as stop word and special character removal, tokenization and stemming (e. g. Porter stemmer) [ 19, 22–25 ]. For text indexing many groups used Terrier [ 22, 23, 26, 27 ]. In 2012, some groups included concept features [ 16, 25, 26 ] using tools such as MetaMap or MeSHUP. Query expansion [ 22, 28 ] was also explored.

Multimodal Retrieval The run with the highest MAP in the image retrieval task was a multimodal run submitted by the ITI team [ 17 ] (0:2377), see also Table 4. For this run various low–level visual descriptors were extracted to create the BoVW. This BoVW was combined with words taken from the topic description to form a multimodal query appropriate for Essie. ITI also submitted the second best mixed run (M AP = 0:2166) that has a slightly worse MAP than the best textual run (M AP = 0:2182).

Several late fusion strategies were used by the participants such as the product fusion algorithm [ 19 ], a linear weighed fusion strategy [ 23 ], reciprocal rank fusion [ 14 ], weighted combSUM [ 22 ] and combMNZ [ 14 ]. 3.3

Case{based Retrieval Results

In 2012, 37 runs were submitted in the case–based retrieval task. As in previous years most of them were textual runs. Only the medGIFT team [ 14 ] submitted visual and multimodal case–based retrieval runs. Although textual runs achieved the best results, a mixed approach performs better than the average of all submitted runs in this task. Visual runs do not perform as well as most of the textual retrieval runs.

Visual Retrieval Table 5 shows the results using visual retrieval on the case– based task. The medGIFT team [ 14 ] is the only group that submitted a multimodal run in this task, using a combination of BoVW and BoC and obtaining the best accuracy in the multimodal classification task. The results also show that there can be an enormous difference combining the two base feature sets. Textual Retrieval The medGIFT team [ 14 ] achieved the highest MAP, 0:169, among all submitted runs. For this run only the standard Lucene baseline was used. The second best run was submitted by MRIM (M AP = 0:1508) [28]. MRIM proposed a solution to the frequency shift thrugh a new counting strategy.

In addition to the techniques used in other tasks, the participants used semantic similarity [ 13, 16 ] measures. Moreover, three of the six groups participating used concept–based approaches [ 16, 25, 28 ]. The ITI team [ 17 ] used the Google Search API6 to determine relevant disease names to correspond to signs and symptoms found in a topic case.

6 https://developers.google.com/custom-search/v1/overview

Run Name Group MAP GM-MAP bpref P10 P30 medgift-lf-boc-bovw-reci-IMAGES-cb medGIFT 0,0366 0,0014 0,0347 0,0269 0,0141 medgift-lf-boc-bovw-mnz-IMAGES-cb medGIFT 0,0302 0,001 0,0293 0,0231 0,009 baseline-sift-early-fusion-cb medGIFT 0,0016 0 0,0032 0,0038 0,0013 baseline sift late fusion cb medGIFT 0,0008 0 0 0,0038 0,0013 medgift-ef-boc-bovw-reci-IMAGES-cb medGIFT 0,0008 0,0001 0,0007 0 0,0013 medgift-ef-boc-bovw-mnz-IMAGES-cb medGIFT 0,0007 0 0 0 0,0013 Multimodal Retrieval As in the visual case–based task, only the medGIFT team [ 14 ] submitted multimodal case–based runs. The runs combine the visual approach based on BoVW and BoC with a Lucene baseline and obtained averaged results when using the combMNZ fusion.

Run Name Group MAP GM-MAP bpref P10 P30 medgift-ef-mixed-mnz-cb medGIFT 0,1017 0,0175 0,0857 0,1115 0,0679 medgift-ef-mixed-reci-cb medGIFT 0,0514 0,009 0,0395 0,0654 0,0564 4

Conclusions

As in previous years, the largest number of runs submitted for the image–based retrieval task. However, in 2012 there were 122 runs in this task, eight less than in 2011. For the case–based retrieval task the number of runs also decreased to 37 (43 in 2011). On the other hand, the number submitted runs at the modality classification task increased to 43 (34 in 2011).

There are still different situations as to whether visual, textual or combined techniques perform better depending on the task. For the modality classification, a visual run achieved the best accuracy using training data extension. In the case of the image–based retrieval task, multimodal runs obtained best results. Finally, for the case–based retrieval task textual runs obtained the best results.

In 2011, the Xerox team [29] that did not participate in 2012 explored the expansion of the training set. This approach achieved the best accuracy for the modality classification task. In 2012, three teams applied expansion of the training set and also obtained good results. This evolution of techniques is a good example of the added value of evaluation campaigns such as ImageCLEF showing the improvements due to specific techniques.

Many groups explored the same or similar descriptors obtaining often quite differing results. This shows that particularly the tuning of existing techniques and the intelligent combination of results fusion can lead to optimal results. Often, the differences in techniques are quite small and more on intelligent feature combinations might be necessary to reach conclusive results. 5

Acknowledgements

We would like to thank the EU FP7 projects Khresmoi (257528), PROMISE (258191) and Chorus+ (249008) for their support as well as the Swiss national science foundation with the MANY project (number 205321–130046). 25. Majdoubi, J., Loukil, H., Tmar, M., Gargourri, F.: Medical case{based retrieval by using a language model: MIRACL at ImageCLEF 2012. In: Working Notes of CLEF 2012. (2012) 26. Gasmi, K., Torjmen-Khemakhem, M., Ben Jemaa, M.: Word indexing versus conceptual indexing in medical image retrieval (ReDCAD participation at ImageCLEF medical image retrieval 2012). In: Working Notes of CLEF 2012. (2012) 27. Crespo, M., Mata, J., Man~a, M.J.: LABERINTO at ImageCLEF 2012 medical image retrieval tasks. In: Working Notes of CLEF 2012. (2012) 28. Abdulahhad, K., Chevallet, J.P., Berrut, C.: MRIM at ImageCLEF2012. from words to concepts: A new couting approach. In: Working Notes of CLEF 2012. (2012) 29. Csurka, G., Clinchant, S., Jacquet, G.: XRCE's participation at medical image modality classi cation and ad{hoc retrieval task of ImageCLEFmed 2011. In: Working Notes of CLEF 2011. (2011)

References

1. Muller, H., Clough , P. , Deselaers , T. , Caputo , B., eds.: ImageCLEF { Experimental Evaluation in Visual Information Retrieval . Volume 32 of The Springer International Series On Information Retrieval. Springer, Berlin Heidelberg ( 2010 )

2. Clough , P. , Muller, H., Sanderson , M.: The CLEF cross{language image retrieval track (ImageCLEF) 2004 . In Peters, C. , Clough , P. , Gonzalo , J. , Jones , G.J.F. , Kluck , M. , Magnini , B., eds.: Multilingual Information Access for Text, Speech and Images: Result of the fth CLEF evaluation campaign . Volume 3491 of Lecture Notes in Computer Science (LNCS)., Bath , UK, Springer ( 2005 ) 597 { 613

3. Muller, H., Deselaers , T. , Kim , E. , Kalpathy-Cramer , J. , Deserno , T.M. , Clough , P. , Hersh , W. : Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks . In: CLEF 2007 Proceedings. Volume 5152 of Lecture Notes in Computer Science (LNCS) ., Budapest, Hungary, Springer ( 2008 ) 473 { 491

4. Kalpathy-Cramer , J. , Muller, H., Bedrick , S. , Eggel , I. , Garc a Seco de Herrera, A. , Tsikrika , T. : The CLEF 2011 medical image retrieval and classi cation tasks . In: Working Notes of CLEF 2011 ( Cross Language Evaluation Forum) . (September 2011 )

5. Kalpathy-Cramera , J. , Hersh , W. : Automatic image modality based classi cation and annotation to improve medical image retrieval . Studies in Health Technology and Informatics 129 ( 2007 ) 1334 { 1338

6. Csurka , G. , Clinchant , S. , Jacquet , G. : Medical image modality classi cation and retrieval . In: 9th International Workshop on Content-Based Multimedia Indexing , IEEE ( 2011 ) 193 { 198

7. Markonis , D. , Holzer , M. , Dung , S. , Vargas , A. , Langs , G. , Kriewel , S. , Muller, H.: A survey on visual information search behavior and requirements of radiologists . Methods of Information in Medicine ( 2012 ) Forthcoming

8. Zhang , D. , Lu , G. : Review of shape representation and description techniques . Pattern Recognition (1) ( 2004 ) 1 { 19

9. Tirilly , P. , Lu , K. , Mu , X. , Zhao , T. , Cao , Y. : On modality classication and its use in text{based image retrieval in medical databases . In: Proceedings of the 9th International Workshop on Content{Based Multimedia Indenxing. CBMi2011 ( 2011 )

10. Muller, H., Kalpathy-Cramer , J. , Demner-Fushman , D. , Antani , S. : Creating a classi cation of image types in the medical literature for visual categorization . In: SPIE medical imaging . ( 2012 )

11. Tsikrika , T. , Muller, H., Kahn Jr ., C.E.: Log analysis to understand medical professionals' image searching behaviour . In: Proceedings of the 24th European Medical Informatics Conference. MIE2012 ( 2012 )

12. Hersh , W. , Muller, H., Kalpathy-Cramer , J. , Kim , E. , Zhou , X. : The consolidated ImageCLEFmed medical image retrieval task test collection . Journal of Digital Imaging 22 ( 6 ) ( 2009 ) 648 { 655

13. Cao , L. , Chang , Y.C. , Codella , N. , Merler , M.: IBM t .j. watson research center, multimedia analytics: Modality classi cation and case{based retrieval task of ImageCLEF2012 . In: Working Notes of CLEF 2012 . ( 2012 )

14. Garc a Seco de Herrera, A. , Markonis , D. , Eggel , I. , Muller, H.: The medGIFT group in ImageCLEFmed 2012 . In: Working Notes of CLEF 2012 . ( 2012 )

15. Collins, J. , Okada , K. : A comprative study of similarity measures for content{based medical image retrieval . In: Working Notes of CLEF 2012 . ( 2012 )

16. Wu , H. , Sun , K. , Deng , X. , Zhang , Y. , Che , B. : UESTC at ImageCLEF 2012 medical tasks . In: Working Notes of CLEF 2012 . ( 2012 )

17. Simpson , M.S. , You , D. , Rahman , M.M. , Demmer-Fushman , D. , Antani , S. , Thoma , G.: ITI's participation in the ImageCLEF 2012 medical retrieval and classi cation tasks . In: Working Notes of CLEF 2012 . ( 2012 )

18. Song , W. , Zhang, D. , Luo , J.: BUAA AUDR at ImageCLEF 2012 medical retrieval task . In: Working Notes of CLEF 2012 . ( 2012 )

19. Castellanos , A. , Benavent , J. , Benavent , X. , Garc a-Serrano, A. : Using visual concep features in a multimodal retrieval system for the medical collection at ImageCLEF2012 . In: Working Notes of CLEF 2012 . ( 2012 )

20. Ide , N.C. , Loane , R.F. , Demner-Fushman , D. : Application of information technology: Essie: A concept{based search engine for structured biomedical text . Journal of the American Medical Informatics Association 14 ( 3 ) ( 2007 ) 253 { 263

21. Vanegas , J.A. , Caicedo , J.C. , Camargo , J. , Ramos , R. , Gonzalez , F.A. : Bioingenium at ImageCLEF: Textual and visual indexing for medical images . In: Working Notes of CLEF 2012 . ( 2012 )

22. Vahid , A.H. , Alpkocak , A. , Hamed , R.G. , Caylan , N.M. , Ozturkmenoglu , O. : DEMIR at ImageCLEFmed 2012: Inter{modality and intra{modality integrated combination retrieval . In: Working Notes of CLEF 2012 . ( 2012 )

23. Kitanovski , I. , Dimitrovski , I. , Loskovska , S. : FCSE at ImageCLEF 2012: Evaluating techniques for medical image retrieval . In: Working Notes of CLEF 2012 . ( 2012 )

24. Stathopoulos , S. , Sakiotis , N. , Kalamboukis , T. : IPL at CLEF 2012 medical retrieval task . In: Working Notes of CLEF 2012 . ( 2012 )