This paper describes the ImageCLEF 2019 Concept Detection Task. This is the 3rd edition of the medical caption task, after it was rst proposed in ImageCLEF 2017. Concept detection from medical images remains a challenging task. In 2019, the format changed to a single subtask and it is part of the medical tasks, alongside the tuberculosis and visual question and answering tasks. To reduce noisy labels and limit variety, the data set focuses solely on radiology images rather than biomedical gures, extracted from the biomedical open access literature (PubMed Central). The development data consists of 56,629 training and 14,157 validation images, with corresponding Uni ed Medical Language System (UMLS R ) concepts, extracted from the image captions. In 2019 the participation is higher, regarding the number of participating teams as well as the number of submitted runs. Several approaches were used by the teams, mostly deep learning techniques. Long short-term memory (LSTM) recurrent neural networks (RNN), adversarial auto-encoder, convolutional neural networks (CNN) image encoders and transfer learning-based multi-label classi cation models were the frequently used approaches. Evaluation uses F1-scores computed per image and averaged across all 10,000 test images.
The concept detection task presented in this paper is part of the ImageCLEF1
benchmarking campaign, that is part of the Cross Language Evaluation Forum2
(CLEF). ImageCLEF was rst held in 2003 and in 2004 a medical task was
added that has been held every year [
The caption task was rst proposed in 2016 as a caption prediction task. In
2017, the caption task was split into two subtasks: concept detection and caption
prediction and ran in that format at ImageCLEFcaption 2017 [
The motivation for this task is that an increasing number of images has become available without metadata, so obtaining some metadata is essential to make the content usable. The objective is to develop systems capable of predicting concepts automatically for radiology images, or possibly for other clinical images. These predicted concepts enable order for unlabeled and unstructured radiology images and for data sets lacking metadata, as multi-modal approaches prove to obtain better results regarding image classi cation [12]. As the interpretation and summarization of knowledge from medical images such as radiology output is time-consuming, there is a considerable need for automatic methods that can approximate this mapping from visual information to condensed textual descriptions. The more image characteristics are known, the more structured are the radiology scans and hence, the more e cient are the radiologists regarding interpretation.
For development data, a subset of the Radiology Object in COntext data set (ROCO) [11] is used. ROCO contains radiology images originating from the PubMed Central (PMC) Open Access Subset3 [14], with several Uni ed Medical Language System (UMLS R ) Concept Unique Identi ers (CUIs) per image. The test set used for o cial evaluation was created in the same manner as proposed in Peltka et al. [11].
This paper presents an overview of the ImageCLEFmed Concept Detection Task 2019 with task description and participating teams in Section 2, an exploratory analysis on the data set and ground truth described in Section 3 and the evaluation framework explained in Section 4. The approaches applied by the participating teams are listed in Section 5, which is followed by discussion and conclusions in Section 6. 2
Succeeding the previous subtasks in ImageCLEFcaption 2017 [
In Table 2, the 11 participating teams of the ImageCLEFmed Concept Detection task are listed. There were 49 registered participants out of 99 teams, who downloaded the End-User-Agreement. Altogether, 77 runs were submitted for evaluation. Out of the 77 submitted runs, 60 were graded and 17 were faulty submission. The majority of the participating teams are new to the task, as only three groups participated in the previous years. Equivalently to previous editions, the data set distributed for the ImageCLEFmed 2019 Concept Detection task originates from biomedical articles of the PMC Open Access subset.
The training and validation sets containing 56,629 and 14,157 images were subsets of the ROCO data set presented in Peltka et al. [11]. ROCO has two classes: Radiology and Out-Of-Class. The rst contains 81,825 radiology images, which was used for the presented work. It includes several medical imaging modalities such as, Computed Tomography (CT), Ultrasound, X-Ray, Fluoroscopy, Positron Emission Tomography (PET), Mammography, Magnetic Resonance Imaging (MRI), Angiography and PET-CT, and can be seen in Figure 2.
From the PMC Open Access subset [14], a total of 6,031,814 image - caption
pairs were extracted. Compound gures, which are images with more than one
sub gure, were removed using deep learning as proposed in Koitka et al. [
Examples of concepts in the training set are listed in descending order of occurence in Table 2. A few concepts were labelled only once, as can be seen in Figure 3.
ROCO contains images from the PMC archive extracted in January 2018, which makes up the training set for the ImageCLEF Concept Detection Task. To avoid an overlap with images distributed at previous ImageCLEF medical tasks, the test set for ImageCLEF 2019 was created with a subset of PMC Open Access (archiving date: 01.02.2018 - 01.02.2019). The same procedures applied for the creation of the ROCO data set were applied for the test set as well.
Concepts with very high frequency (>13,000), such as \Image", as well as
redundant synonyms were removed. This lead to reduction of concepts per image
in comparison to the previous years. All images in the training, validation and
test sets have [
UMLS R CUIs need to be automatically predicted by the participating teams for
all 10,000 test images. As in previous editions [
As the training, validation and test set contain a maximum of 72, 77 and 34 concepts per image, the maximum number of concepts allowed in the submission runs was set to 100. Each participating group could submit altogether 10 valid and 7 faulty submission runs. Faulty submissions include: { Same image id more than once { Wrong image id { Too many concepts { Same concept more than once { Not all test images included All submission runs were uploaded by the participating teams and evaluated with CrowdAI4. The source code of the evaluation tool is available on the task Web page5. 5
This section details the results achieved by all 11 participating teams for the
concept detection task. The best run per team is shown in Table 3. Table 4
contains the complete list of all graded submission runs. There is an improvement
compared to both previous editions, from 0.1583 in ImageCLEF 2017 [
Best results were achieved by the AUEB NLP Group [8] by applying convolutional neural network (CNN) image encoders that were combined either with image retrieval methods or feed-forward neural networks to predict the concepts for images in the test set. On the test set, this CheXNet-based system [13] achieved better results in terms of F1-score, while an ensemble of an k-NN image retrieval system with CheXNet performed better on the development data. AUEB NLP ranked 1st to 3rd place with 3 out of the 4 submitted runs. 4 https://www.crowdai.org/challenges/imageclef-2019-caption-concept-detection6812fec9-8c9e-40ad-9fb9-cc1721c94cc1 [last accessed: 02.06.2019] 5 https://www.imageclef.org/system/ les/ImageCLEF-ConceptDetection
Evaluation.zip [last accessed: 02.06.2019]
Damo [19] was the second ranked group with 9 runs and applied two distinct methods to address the concept detection task. The latest deep learning system ResNet-101 was used for a multi-label classi cation approach, as well as a CNN-RNN model framework with attention mechanisms. Due to the imbalanced concept distribution, the group applied several data ltering methods. This proved to be positive, as the best run was a combination of multi-label classi cation with a ltered and reduced data set.
A two-stage concept detection approach was presented by the third ranked
group: ImageSem [
The pri2si17 team [16] participated for the rst time in the concept detection task. They addressed the task as a multi label classi cation problem and limited the concepts to the most frequent 25 labels.
UA.PT BioInformatics [
A recurrent neural network (RNN) architecture was proposed by Sam Maksoud [9]. Soft attention and visual gating mechanisms are used to enable the network to dynamically regulate \where" and \when" to extract visual data for concept generation. Two runs were submitted for grading, with the score of 0.1749 ranked 22nd out of all submissions and the group was ranked 6th overall.
The 7th overall ranked group is AI600 with 7 graded submission runs. Multilabel classi cation based on Bag-of-Visual-Words model with color descriptors and logistic regression, using di erent SIFT (Scale-Invariant Feature Transform) descriptors as visual features were applied for the concept detection task. The best run with a combination of SIFT, C-SIFT, HSV-SIFT and RGB-SIFT visual descriptors achieved 0.1656261, which is the 26th out of all submissions.
MS-CSIRO [15] submitted 1 run for o cial evaluation. Relevant concepts were predicted with an approach based on a multi-label classi cation model using CNN. MS-CSIRO ranked as the 8th best team and their submitted run with the score 0.1435 ranked 36th.
Similar to team Damo, the deep learning system ResNet-101 was utilized as base model. pri2si17 are the ninth best ranked team. Three runs were submitted for grading, of which the best run achieved the score 0.0497 ranking 45th. 6
The results of the task in 2019 show that there is an improvement in the F1scores in this 3rd edition (best score 0.2823) in comparison to ImageCLEF 2017 and ImageCLEF 2018. In the previous years, the best scores were 0.1583 in 2017 and 0.1108 in 2018. There were several new teams participating for the 1st time, as well as 3 teams, who participated in all editions. In addition, an increased number of participating teams and submitted runs was noticed in 2019. This shows the interest in this challenging task.
Most submitted runs are based on deep learning techniques. Several methods such as concept ltering, data augmentation and image normalization were applied to optimize the input for the predicting systems. Long short-term memory (LSTM) recurrent neural networks (RNN), adversarial auto-encoder, CNN image encoders and transfer learning-based multi-label classi cation models were the frequently used approaches.
The focus this year was reduced from biomedical images to solely radiology images, which led to the reduction of extracted concepts from 111,155 to 5,528. However, there is still an unbalanced distribution of concepts, which shows to be challenging to most teams. This can be due to the di erent imaging modalities, as well as several body parts included in the data set. Medical data and diseases are also usually unbalanced with a few conditions happening very frequently and most being very rare.
In future work, an extensive review of the clinical relevance for the concepts in the development data should be explored. As the concepts originate from the natural language captions, not all concepts have high clinical utility. Medical journals also have very di erent policies in terms of checking gure cations. We believe this will assist in creating more e cient systems for automated medical data analysis. of the CLEF Association, Dublin, Ireland, September 11-14, 2017, Lecture Notes in Computer Science (LNCS) 10456. pp. 57{68. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-65813-1 5 8. Kougia, V., Pavlopoulos, J., Androutsopoulo, I.: Aueb nlp group at imageclefmed caption 2019. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019) 9. Maksoud, S., Wiliem, A., Lovell, B.: Recurrent attention networks for medical concept prediction. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019) 10. Muller, H., Clough, P.D., Deselaers, T., Caputo, B. (eds.): ImageCLEF, Experimental Evaluation in Visual Information Retrieval. Springer (2010). https://doi.org/10.1007/978-3-642-15181-1 11. Pelka, O., Koitka, S., Ruckert, J., Nensa, F., Friedrich, C.M.: Radiology objects in context (ROCO): A multimodal image dataset. In: Intravascular Imaging and Computer Assisted Stenting - and - Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings. pp. 180{189 (2018). https://doi.org/10.1007/978-3-030-01364-6 20, https://doi.org/10.1007/978-3-030-01364-6 20 12. Pelka, O., Nensa, F., Friedrich, C.M.: Adopting semantic information of grayscale radiographs for image classi cation and retrieval. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 2: BIOIMAGING, Funchal, Madeira, Portugal, January 19-21, 2018. pp. 179{187 (2018). https://doi.org/10.5220/0006732301790187 13. Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., Lungren, M.P., Ng, A.Y.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. CoRR abs/1711.05225 (2017), http://arxiv.org/abs/1711.05225 14. Roberts, R.J.: PubMed Central: The GenBank of the published literature. Proceedings of the National Academy of Sciences of the United States of America 98(2), 381{382 (Jan 2001). https://doi.org/10.1073/pnas.98.2.381 15. Singh, S., Karimi, S., Ho-Shon, K., Hamey, L.: Biomedical concept detection in medical images: Mq-csiro at 2019 imageclefmed caption task. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019) 16. Sinha, P., Purkayastha, S., Gichoya, J.: Full training versus ne tuning for radiology images concept detection task for the imageclef 2019 challenge. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019) 17. Soldaini, L., Goharian, N.: QuickUMLS: a fast, unsupervised approach for medical concept extraction. In: MedIR Workshop, SIGIR (2016) 18. Wang, X., Liu, N.: Ai600 lab at imageclef 2019 concept detection task. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019) 19. Xu, J., Liu, W., Liu, C., Wang, Y., Chi, Y., Xie, X., Hua, X.: Concept detection based on multi-label classi cation and image captioning approach - damo at imageclef 2019. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, (CEUR-WS.org), ISSN 1613-0073, http://ceur-ws.org/Vol-2380/, Lugano, Switzerland (September 09-12 2019)