This paper describes the ImageCLEFmed 2020 Concept Detection Task. After rst being proposed at ImageCLEF 2017, the medical task is in its 4th edition this year, as the automatic detection from medical images still remains a challenging task. In 2020, the format remained the same as in 2019, with a single sub-task. The concept detection task is part of the medical tasks, alongside the tuberculosis and visual question and answering tasks. Similar to the 2019 edition, the data set focuses on radiology images rather than biomedical images, however with an increased number of images. The distributed images were extracted from the biomedical open access literature (PubMed Central). The development data consists of 65,753 training and 15,970 validation images. Each image has corresponding Uni ed Medical Language System (UMLS R ) concepts, that were extracted from the original article image captions. In this edition, additional imaging acquisition technique labels were included in the distributed data, which were adopted for pre- ltering steps, concept selection and ensemble algorithms. Most applied approaches for the automatic detection of concepts were deep learning based architectures. Long short-term memory (LSTM) recurrent neural networks (RNN), adversarial auto-encoder, convolutional neural networks (CNN) image encoders and transfer learning-based multi-label classi cation models were adopted. The performances of the submitted models (best score 0.3940) were evaluated using F1-scores computed per image and averaged across all 3,534 test images.
1
In this paper, the approaches for the detection of Uni ed Medical Language
System (UMLS R ) concepts present in radiology images are presented. The task is
part of the ImageCLEF1 bench-marking campaign, that is part of the Cross
Language Evaluation Forum2 (CLEF). Since 2003, the ImageCLEF bench-marking
campaign has been proposing several image understanding tasks from di erent
domains every year [
The concept detection task in this year is the fourth edition. At
ImageCLEFmed Caption 2017 [
As there is an increasing number of medical images available without metadata, for example in the scienti c literature, there is an essential need to create systems that can automatically generate such information, hence making the content of these data sets more useful. The purpose of the ImageCLEFmed 2020 concept detection task was to create a platform for the evaluation of systems capable of automatically creating UMLS R concepts of a given radiology image. These predicted information is applicable for data sets that either not labeled or structured, but also for medical data sets lacking textual metadata, as multi-modal approaches prove to obtain better results regarding several image classi cation tasks [18, 19].
The manual interpretation and generation of knowledge from medical images is not only time-consuming and prone to error, but also impractical. Therefore, the modeling systems that can automatically map visual content present in the images to concise textual representations is a necessity, in regards to e cient information retrieval and image classi cation.
For development data, both the development and test sets from the ImageCLEFmed Caption 2019 [16] was distributed. This data set is a subset of the Radiology Object in COntext data set (ROCO) [17] and contains solely radiology images that originate from the PubMed Central (PMC) Open Access Subset3 [20]. Several UMLS R Concept Unique Identi ers (CUIs) are included to each image. The test set used for o cial evaluation was created in the same manner as proposed in Pelka et al. [17], for generalization purposes. 1 http://imageclef.org/ [last accessed: 28.07.2020] 2 http://www.clef-initiative.eu/ [last accessed: 28.07.2020] 3 https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/[last accessed: 28.07.2020]
This paper presents an overview of the ImageCLEFmed 2020 Concept Detection Task. Section 2 contains the task description and lists the participating teams. An explorative analysis computed on the distributed development and test data sets is described in Section 3. The framework used to evaluate the submission runs is explained in Section 4. Section 5 displays the modeling approaches applied by the participating teams and the obtained scores, and is followed by discussion and conclusions in Section 6. 2
Similar to the ImageCLEF caption task in 2019 [16], in ImageCLEF Caption 2020 the focus is on the automatic detection of concepts in a large corpus of radiology images. The proposed task aims to interpret and summarise insights gained from medical images and therefore provide tools for radiology image understanding. The distributed images in both development and evaluation data sets originate from biomedical articles extracted from the PubMed Central (PMC) Open Access Subset[20]. To each radiology image in the distributed data sets, UMLS R CUIs are included. These concepts are generated from the the original image captions found in the articles. Figure 1 displays an example of an image in the distributed data sets. In comparison to the previous tasks, the following improvements were made: { The imaging modality was included. { The focus remained on radiology images as in ImageCLEF 2019 . { The number of concepts was decreased by preprocessing the captions prior to concept extraction. The automatic detection of concepts present in images is a fundamental step towards scene understanding and hence image captioning, as the presence of applicable biomedical concepts can be detected and located. As the usage of multi-modal representations (visual and textual) for image classi cation tasks helps to achieve good performance [19], the automatically generated concepts can be adopted for this purpose. In addition, the concepts can also be used for context-based image analysis, as well as for information retrieval. The detected concepts are evaluated image-wise with precision and recall scores from the ground truth, which is described in Section 4.
In the ImageCLEF 2020 concept detection task a total of 23 unique teams registered in AICrowd and downloaded the End-User-Agreement. This license is needed to obtain access to both development and evaluation data. 57 graded runs were submitted for evaluation by 7 teams from the following countries: Germany, United Kingdom, India, Greece and United States of America, which is listed in Table 2. Each of the groups was allowed 10 graded runs and 5 faulty runs altogether. 10 of the submitted runs were faulty and were not used for the o cial evaluation. 3
As in previous editions, the data set distributed for the task originates from biomedical articles of the PMC Open Access subset [20]. The development data set contains training and validation sets with 65,753 and 15,970 images, respectively. These images are subsets of the multi-modal image data set Radiology Objects in COntext (ROCO), which is presented in Pelka et al. [17]. ROCO has two classes: Radiology and Out-Of-Class. The rst contains 81,825 radiology images and was adopted for the proposed task. It includes several medical imaging modalities such as, Computed Tomography (CT), Ultrasound, X-Ray, Fluoroscopy, Positron Emission Tomography (PET), Mammography, Magnetic Resonance Imaging (MRI), Angiography and PET-CT.
The development data of the 2020 task includes the ImageCLEF caption 2019 development data set (archiving date: until 31.01.2018) and the o cial evaluation set (archiving date: 01.02.2018 - 01.02.2019). To avoid an overlap with images distributed in previous ImageCLEF medical tasks, the test set for ImageCLEF 2020 was created with a subset of PMC Open Access (archiving date: 01.02.2019 - 01.02.2020). The same procedures applied for the creation of the ROCO data set were applied for the test set as well. An analysis of the distributed data can be seen in Table 2.
Imaging Technique Train Validation Test Sum DRAN: Angiography 4,713 1,132 325 6,170 DRCO: Combined modalities in one image 487 73 49 609 DRCT: Computerized Tomography 20,031 4,992 1,140 26,163 DRMR: Magnetic Resonance 11,447 2,848 562 14,857 DRPE: Positron emission tomography 502 74 38 614 DRUS: Ultrasound 8,629 2,134 502 11,265 DRXR: X-Ray, 2D radiography 18,944 4,717 918 24,579 Sum 65,753 15,970 3534 84,257 From the PMC Open Access subset [20], a total of 6,031,814 image - caption pairs were extracted in January 2018. Compound gures, which are images with more than one sub gure, were removed using deep learning as proposed in Koitka et al. [13]. The non-compound images were further split into radiology and nonradiology, as the focus was on radiology. Semantic knowledge of object interplay present in the images were extracted in the form of UMLS R Concepts using the QuickUMLS library [23]. The image captions from the biomedical articles served as basis for the extraction of the concepts. The text pre-processing steps applied are described in Pelka et al. [17]. Using deep learning systems as proposed in Koitka et al. [13], the radiology images were further split into seven (7) imaging modality classes. This information can be used for ltering steps prior to model training, as well as for model ne-tuning.
An additional UMLS R CUI denoting the imaging technique modality was
added to each image. Figure 2 shows example images from the development
data set, according to image modality and additional UMLS R CUI. Similarly to
the caption task in 2019 [16], concepts with very high frequency (>13,000), as
well as redundant synonyms were removed. This lead to a reduction of concepts
per image in comparison to the previous years, from 5,528 in 2019 [16] to 3,047
in 2020. Not all concepts in the ground truth can be visually seen, for example
the concept 'Hole Finding' in Fig. 2 can not be detected from the image. Images
in the training, validation and test sets have [
The maximum number of concepts allowed per image was set to 150. This limitation was chosen as the training, validation and test set contain a maximum of 140, 142 and 95 concepts per image. Each group could have a maximum of 15 submission, with 10 valid and 5 faulty. Faulty submissions may include: { Same image id more than once { Wrong image id { Too many concepts { Same concept more than once { Not all test images included
All submission runs were uploaded by the participating teams and evaluated with AICrowd4. The source code of the evaluation tool is available on the ImageCLEF web page5. 5
The overall performance achieved by the concepts detection models submitted
by the 7 participating teams are listed and discussed in this section. In Table 4,
the submission run with best performance per team is shown. An additional
evaluation regarding the imaging modality was done internally, after the o cial
concept detection evaluation process. The accuracy (%) across all images in the
test set was computed and is listed in Table 6. Compared to the previous
editions, there is an improvement regarding the F1-Score of the submitted concept
detection models, from 0.1583 in ImageCLEF 2017 [
The AUEB NLP Group [12] from the Athens University of Economics achieved the overall highest F1-Score of 0.3940 for the detection of concepts for the images in the o cial evaluation test set. Their three (3) submission runs ranked 1st, 2nd and 6th of all 47 submitted runs. The submitted systems are a variation of CheXNet [26] with DenseNet-121 [8] and followed by a feed-forward Neural Network (FFNN), which acts as the classi er layer on the top [12]. The system was rst pre-trained on the ImageNet data set [21] and then ne-tuned using the ImageCLEF 2020 concept detection development data set. Several ensemble methods such as the intersection and union of predicted concepts were experimented. The system with the intersection of concepts achieved the overall highest F1-Score.
The overall 2nd ranked participating team is PwC Healthcare group from PricewaterhouseCoopers with a total number of nine (9) submitted runs. The adopted approaches range from Convolutional Neural Network (CNN) architectures, to Natural Language Processing techniques, as well as clustering algorithms [24]. The group's three (3) best systems ranked 3rd, 4th and 5th. Several pre-processing approaches such as range and intensity normalization and 4 https://www.aicrowd.com/challenges/imageclef-2020-caption-concept-detection [last accessed: 26.07.2020] 5 https://www.imageclef.org/system/ les/ImageCLEF-ConceptDetection
Evaluation.zip [last accessed: 26.07.2020] data augmentation were adopted prior to training the models [24]. Multi-modal approaches were experimented to incorporate the concept imbalanced distribution and a novel approach of band classi cation was applied. This classi cation method rst clusters the vocabulary of concepts into bands and then creates for each band a classi cation architecture [24].
The third best participating team was from the University of Essex, with an overall F1-Score of 0.381. The proposed approach adopts pre-trained DenseNet models [8] for the extraction of relevant features. The additional information on the imaging modality was used for ne-tuning by adding a fully connected layer to the DenseNet-121 model and thereby transforming it into a multi-label classi cation model [6]. Several concept selection strategies, such as distance and ranked based methods, were applied to a given query image from the test set. The group's ve best runs of the nine submitted runs ranked 6th to 10th among all submissions.
Five runs were submitted by the IML group from the German Research
Center for Arti cial Intelligence, with the best F1-Score of 0.3745, and the 4th
best team. Multiple deep learning systems such as VGG16 [22], ResNet50 [
TUC MC, a media computing group from the Chemnitz University of
Technology ranked 5th best participating team. The highest F1-Score from the ten
submitted runs was 0.3745. The adopted deep learning model was based on the
Xception architecture [
Ten runs were submitted by Morgan CS, a group from the computer science department at the Morgan State University. The best achieved F1-Score was 0.1673, by approaching the concept detection task as a multi-label classi cation problem [14]. Classi ers were trained with deep features extracted with the deep learning system DenseNet169 and ResNet50 and pre-trained on ImageNet. Other methods experimented include a recurrent concept sequence generator that was modelled using a multimodal technique of fusing text and image features for recurrent sequence prediction.
CSE SSN from the department of computer science of the SSN College of
Engineering Chennai submitted one (1) run for o cial evaluation and achieved the
average F1-Score of 0.1347 on all images in the test set. Similar to several
participating teams, the concept detection task was addressed as a convolution neural
network multi-label classi cation problem [
An ex-post evaluation was computed on all submitted runs. The aim was to compute the performance on correctly predicting the imaging modality. All images in the development and test set were assigned concepts that denote the acquisition technique, as shown in Figure 2. The images belonging to the imaging modality 'DRCO: Combined modalities in one image' were not considered for evaluation. For all images in the test set, we computed the presence of these concepts in the submission runs using this additional information. The best performance grouped per team is listed in Table 6 and the complete evaluation in Table 7.
Institution PricewaterhouseCoopers US Advisory, Mumbai, India Department of Informatics, Athens University of Economics and Business, Athens, Greece School of computer Science and Electronic Engineering, University of Essex, Essex, United Kingdom Technische Unversitat Chemnitz, Chemnitz, Germany Interactive Machine Learning Group, German Research Center for Arti cial Intelligence (DFKI), Saarbrucken, Germany Computer Science Department, Morgan State University, Baltimore, Maryland, United States of America Department of Computer Science and Engineering, SSN College of Engineering, Chennai, India This paper presents an overview of applied approaches and their performance, as well as the task description, participation and distributed data set for the ImageCLEF 2020 concept detection task. Similar to the 2019 edition, the results this year show that there is an improvement in the achieved F1-scores (best score 0.3940). In this edition, not only does the dataset contain an increased number of images, the number of concepts were reduced to be more precise and additional modality information was distributed. In the previous editions, the overall best F1-Scores were 0.2823 in Image-med Caption 2019, 0.1108 in ImageCLEFmed Caption 2018 and 0.1583 in ImageCLEFmed Caption 2017. Almost all participating groups were new to the task, with only one team that participated in ImageCLEF caption 2019. The seven participating teams are a liated to institutions from 5 countries, which shows the continuing research interest to this challenging task.
Most of the submitted runs are based on deep learning architectures. The pre-trained models DenseNet-121, ResNet50 and VGG16 on the ImageNet and CheXNet were used to extract relevant visual representation for the images. Multiple pre-processing steps such as concept ltering, data augmentation and image enhancement were applied to optimize the input for the predicting systems. Long short-term memory (LSTM) recurrent neural networks (RNN), adversarial auto-encoders, CNN image encoders and transfer learning-based multi-label classi cation models were the frequently used approaches.
As the focus in the caption task 2019 was reduced from biomedical images to solely radiology images, a reduction of the extracted concepts from 111,155 to 5,528 was observed. We added this year an additional label denoting the imaging modality of the images. This extra information was used by several teams for pre- ltering steps prior to training the models, concept selection and for ensemble algorithms. The class imbalance in the distributed data set proved to be challenging for several teams. However, medical data and diseases are also usually unbalanced with a few conditions happening very frequently and most being very rare.
In future work, an extensive review of the clinical relevance for the concepts in the development data should be explored. As the concepts originate from the natural language captions, not all concepts have high clinical utility. Medical journals also have very di erent policies in terms of checking gure captions. We believe this will assist in creating more e cient systems for automated medical data analysis. nition, CVPR, Las Vegas, USA, June 26 - July 1, 2016. pp. 770{778 (06 2016). https://doi.org/10.1109/CVPR.2016.90 6. de Herrera, A.G.S., Andrade, F.P., Bentley, L., Compean, A.A.: Essex at ImageCLEFcaption 2020 task. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 7. de Herrera, A.G.S., Eickho , C., Andrearczyk, V., Muller, H.: Overview of the ImageCLEF 2018 Caption Prediction Tasks. In: Working Notes of CLEF 2018 Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. (2018), http://ceur-ws.org/Vol-2125/invited paper 4.pdf 8. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely Connected Convolutional Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, USA, July 22-25, 2017. pp. 2261{2269 (July 2017). https://doi.org/10.1109/CVPR.2017.243 9. Ionescu, B., Muller, H., Peteri, R., Abacha, A.B., Datla, V., Hasan, S.A., DemnerFushman, D., Kozlovski, S., Liauchuk, V., Cid, Y.D., Kovalev, V., Pelka, O., Friedrich, C.M., de Herrera, A.G.S., Ninh, V.T., Le, T.K., Zhou, L., Piras, L., Riegler, M., l Halvorsen, P., Tran, M.T., Lux, M., Gurrin, C., Dang-Nguyen, D.T., Chamberlain, J., Clark, A., Campello, A., Fichou, D., Berari, R., Brie, P., Dogariu, M., Stefan, L.D., Constantin, M.G.: Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), vol. 12260. LNCS Lecture Notes in Computer Science, Springer, Thessaloniki, Greece (September 22-25 2020) 10. Kalimuthu, M., Nunnari, F., Sonntag, D.: A Competitive Deep Neural Network Approach for the ImageCLEFmed Caption 2020 Task. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 11. Kalpathy-Cramer, J., de Herrera, A.G.S., Demner-Fushman, D., Antani, S.K., Bedrick, S., Muller, H.: Evaluating performance of biomedical image retrieval systems - An overview of the medical image retrieval task at ImageCLEF 2004-2013. Comp. Med. Imag. and Graph. 39, 55{61 (2015). https://doi.org/10.1016/j.compmedimag.2014.03.004, https://doi.org/10.1016/j.compmedimag.2014.03.004 12. Karatzas, B., Pavlopoulos, J., Kougia, V., Androutsopoulo, I.: AUEB NLP Group at ImageCLEFmed Caption 2020. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 13. Koitka, S., Friedrich, C.M.: Optimized Convolutional Neural Network Ensembles for Medical Sub gure Classi cation. In: Jones, G.J., Lawless, S., Gonzalo, J., Kelly, L., Goeuriot, L., Mandl, T., Cappellato, L., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction at the 8th International Conference of the CLEF Association, Dublin, Ireland, September 11-14, 2017, Lecture Notes in Computer Science (LNCS) 10456. pp. 57{68. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1 5 14. Lyode, O., Rahman, M.: Concept Detection in Biomedical Images with Deep Learning Based Multilabel Classi cation. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 15. Muller, H., Clough, P.D., Deselaers, T., Caputo, B. (eds.): ImageCLEF, Experimental Evaluation in Visual Information Retrieval. Springer (2010). https://doi.org/10.1007/978-3-642-15181-1 16. Pelka, O., Friedrich, C.M., de Herrera, A.G.S., Muller, H.: Overview of the ImageCLEFmed 2019 Concept Detection Task. In: Cappellato, L., Ferro, N., Losada, D.E., Muller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org (2019), http://ceur-ws.org/Vol2380/paper 245.pdf 17. Pelka, O., Koitka, S., Ruckert, J., Nensa, F., Friedrich, C.M.: Radiology Objects in COntext (ROCO): A Multimodal Image Dataset. In: Intravascular Imaging and Computer Assisted Stenting - and - Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings. pp. 180{189 (2018). https://doi.org/10.1007/978-3-030-01364-6 20, https://doi.org/10.1007/978-3-030-01364-6 20 18. Pelka, O., Nensa, F., Friedrich, C.M.: Adopting Semantic Information of Grayscale Radiographs for Image Classi cation and Retrieval. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 2: BIOIMAGING, Funchal, Madeira, Portugal, January 19-21, 2018. pp. 179{187 (2018). https://doi.org/10.5220/0006732301790187 19. Pelka, O., Nensa, F., Friedrich, C.M.: Variations on Branding with Text Occurrence for Optimized Body Parts Classi cation. In: Proceedings of the 41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBC 2019, Berlin, Germany, July 23-27, 2019. pp. 890{894 (2019). https://doi.org/10.1109/EMBC.2019.8857478 20. Roberts, R.J.: PubMed Central: The GenBank of the published literature. Proceedings of the National Academy of Sciences of the United States of America 98(2), 381{382 (Jan 2001). https://doi.org/10.1073/pnas.98.2.381 21. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115 (09 2014). https://doi.org/10.1007/s11263-015-0816-y 22. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale
Image Recognition. arXiv 1409.1556 (09 2014) 23. Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016) 24. Sonker, R., Mishra, A., Bansal, P., Pattnaik, A.: Techniques for Medical Concept Detection from Multi-Modal Images. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 25. Udas, N., Beuth, F., Kowerko, D.: TUC MC group at ImageCLEFmed 2020 concept detection task using Xception models. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece (September 22-25 2020) 26. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classi cation and Localization of Common Thorax Diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, USA, July 22-25, 2017. pp. 3462{3471 (2017)