This paper presents an overview of the ImageCLEF 2017 caption tasks on the analysis of images from the biomedical literature. Two subtasks were proposed to the participants: a concept detection task and caption prediction task, both using only images as input. The two subtasks tackle the problem of providing image interpretation by extracting concepts and predicting a caption based on the visual information of an image alone. A dataset of 184,000 gure-caption pairs from the biomedical open access literature (PubMed Central) are provided as a testbed with the majority of them as trainign data and then 10,000 as validation and 10,000 as test data. Across two tasks, 11 participating groups submitted 71 runs. While the domain remains challenging and the data highly heterogeneous, we can note some surprisingly good results of the di cult task with a quality that could be bene cial for health applications by better exploiting the visual content of biomedical gures.
Interpreting and summarizing the insights gained from medical images such as
radiography or biopsy samples is a time-consuming task that involves highly
trained experts and often represents a bottleneck in clinical diagnosis pipelines.
As a consequence, there is a considerable need for automatic methods that can
approximate the mapping from visual information to condensed textual
descriptions. ImageCLEF4 is an evaluation campaign that has being organized as part
of the CLEF initiative labs since 2003 [
This paper gives an overview of the caption task at ImageCLEF 2017. Section 2 introduces the two subtasks and Section 3 the data set and ground truth. A description of the evaluation methodology is provided in Section 4. Subsequently, the participant submissions are analysed in Section 5 and Section 6 brie y discusses their respective strengths and weaknesses as well as their implications for academic research and medical practice. Finally, we conclude with an outlook to the possible future of the evaluation campaign in Section 7. 2
This rst edition of the biomedical image captioning task at ImageCLEF comprises two sub tasks: (1) Concept Detection and (2) Caption Prediction. Figure 1 shows an example image of a tomographic angiography reconstruction along with its relevant concepts as well as the reference caption.
Concept Detection As a rst step towards automatic image caption understanding, participating systems are tasked with identifying the presence of relevant biomedical concepts in medical images. Based on the visual image content, this subtask provides the building blocks for the image understanding step by identifying the individual components from which full captions can be composed. Caption Prediction On the basis of the concept vocabulary detected in the rst subtask as well as the visual information of their interaction in the image, participating systems are tasked with composing coherent natural language captions for the entirety of an image. In this step, rather than the mere coverage of visual concepts, detecting the interplay of visible elements is crucial for recreating the original image caption. 5 PubMed Central (PMC) is a free fulltext archive of biomedical and life sciences journal literature at the U.S. National Institute of Healths National Library of Medicine (NIH/NLM) (see http://www.ncbi.nlm.nih.gov/pmc/). 6 https://www.nlm.nih.gov/research/umls
Concept detection: { C0002940: Aneurysm { C0002978: angiogram { C0027530: Neck { C0087111: Therapeutic procedure { C0524425: inside the blood vessel { C0524865: Reconstructive Surgical Procedures { C3887704: treatment - ActInformationManagementReason
Caption prediction:
Preoperative computed tomographic angiography reconstruction showing hostile neck
anatomy amenable to treatment with endovascular aneurysm sealing (EVAS).
The experimental corpus is derived from scholarly biomedical articles on PMC
from which we extract gures and their corresponding captions. In total, the
collection is comprised of 184,614 image-caption pairs. This overall set is further
split into disjunct training (164,614 pairs), validation (10,000 pairs) and test
(10,000 pairs) sets. For the concept detection sub task, we used the QuickUMLS
library [
The subset of PMC was created using an automated method to classify all 3
million images of PMC from early 2016 into image types [
The evaluation of both sub tasks is conducted separately. For the concept detection task, we measure the balanced precision and recall trade-o in terms of F1 scores. To this end, we use Python's scikit-learn (v0.17.1-2) library. We compute micro F1 per image and average across all test images. A total of 393 reference captions in the test set do not contain any UMLS concepts. The respective images are excluded from the evaluation.
Caption prediction performance is assessed on the basis of BLEU scores [
The source code of both evaluation scripts is available on the task Web page7. 5
We received a total of 71 submissions by 11 individual teams. Table 1 gives an overview of all participants and their runs. There was a limit of at most 10 runs per team and sub task and the submissions are roughly evenly split between tasks. The call for contributions did not initially make any assumptions about the kinds of strategies and external data that participants would rely on. As a consequence, in this rst edition of the task, we see a broad range of performance scores as well as methods being applied. Evaluation of the results showed that some teams employed methods that were at least partially trained on external resources including PMC articles. Since such approaches cannot be guaranteed to have respected our division into training, validation and test folds and might subsequently leak test examples into the training process, we separately list runs relying exclusively on the o cial collection as well as those making use of external information. 5.1
Concept Detection The concept detection task received 37 runs from 9 participating groups. Table 2 lists the performance of all o cial (no external information used) runs. The global overview of all runs, including those using external information, can be found in Table 3.
The vast majority of runs was purely automatic (A) in nature with only few submissions relying on some form of manual intervention (M). There was no noticeable advantage of relying on manual interventions as all manual runs lie well in the center of the performance score range. 7 http://imageclef.org/2017/caption # Runs T1 # Runs T2 1 0 10 3 0 3 3 0 2 1 4 10 4 10 0 4 0 10 0 0 0 6
While most teams rely on some form of convolutional neural networks to
represent visual information (NLM [18], PRNA [9], BMET [12], AAI [
The use of very deep residual networks (AAI [
Several groups included retrieval-based methods that would identify highly visually related images in the o cial training set (IPL [13], MSU [17]) or an external collection of images (NLM [18]). The captions of such related images are then scanned for bio-medical concepts to be assigned to the candidate image. This approach generally resulted in very good results, among them several of the best-performing submissions for the task. The harder caption prediction task received 34 runs from 5 participating groups. Table 4 lists the performance of all o cial (no external information used) runs. The global overview of all runs, including those using external information, can be found in Table 5. For this task, no manual runs were submitted.
Most submitted runs are based on the teams' respective contributions to the concept detection task expanded by language modeling capabilities. Often this takes the form of recurrent neural networks (ISIA [14], BCSG [10], PRNA [9], BMET [12]), making the CNN + LSTM combination a frequently-used setup.
As for the rst sub task, the use of retrieval-based methods to identify highly visually related images and using their captions as a starting point for candidate caption generation (ISIA [14] MSU [17], NLM [18]) resulted in highly competitive performance. There are several observations that should be taken into account when analyzing the results presented in the previous section. Most notably, as a consequence of the data source (scholarly biomedical journal articles), the collection contains a considerable amount of noise in the form of compound gures with potentially highly heterogeneous content. In future editions of this task, we will consider using a less diverse source of images such as radiology/pathology in order to reduce the amount of variation in the data.
Secondly, the UMLS concept extraction employed here is a probabilistic process that introduces its own errors. As a consequence, there are several training captions that do not contain any UMLS concepts, making such examples di cult to use for concept detection purposes. In the future, we will rely on more rigorous ltering to ensure good concept coverage across training, validation and test data.
Finally, there should have been a clearer speci cation of what external material, if any, is permissible for use. The teams employed a wide number of third-party material ranging from general academic collections such as ImageNet, mainly in the form of pre-trained networks, to corpora of scholarly articles. While the former do not represent a major problem, the latter could, conceivably contain the exact image and caption pairs of our test set, the use of which would create a strong advantage and a non-realistic setting for really novel data. The experimental overview shows some evidence of this happening when methods using PubMed Central images in the training step vastly outperform all competitors on both tasks. For this reason, we made the conservative decision to separate between o cial runs using no external information at all and those that used third-party material. In the future, we will more carefully specify which kind of external material is safe to use. It does make sense to allow for external data to be used but it needs to be made clear that no test data are included.
This paper presents an overview of the ImageCLEF 2017 biomedical image captioning task. We consider the sub tasks of concept detection and full caption prediction. The participating groups investigated the use of a wide range of image understanding techniques. Especially neural network methods are highly popular and delivered convincing performance on these hard problems. The individually relatively low scores motivate further homogenization of tasks and collection in future editions of the challenge.
This research was supported in part by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC). 9. Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T., Datla, V.V., Lee, K., Qadir, A., Swisher, C., Farri, O.: PRNA at ImageCLEF 2017 caption prediction and concept detection tasks. (2017) 10. Pelka, O., Friedrich, C.M.: Keyword generation for biomedical image retrieval with recurrent neural networks. (2017) 11. Pinho, E., Figueira Silva, J.a., Ferreira Silva, J.M., Costa, C.: Towards representation learning for biomedical concept detection in medical images: UA.PT bioinformatics in ImageCLEF 2017. (2017) 12. Lyndon, D., Kumar, A., Kim, J.: Neural captioning for the ImageCLEF 2017 medical image challenges. (2017) 13. Valavanis, L., Stathopoulos, S.: IPL at ImageCLEF 2017 concept detection task.
CLEF working notes, CEUR (2017) 14. Liang, S., Li, X., Zhu, Y., Li, X., Jiang, S.: ISIA at ImageCLEF 2017 image caption task. (2017) 15. Mothe, J., Ny Hoavy, N., Randrianarivony, M.I.: IRIT & MISA at ImageCLEF 2017 - multi label classi cation. (2017) 16. Stefan, L.D., Ionescu, B., Muller, H.: Generating captions for medical images with a deep learning multi-hypothesis approach: ImageCLEF 2017 caption task. (2017) 17. Rahman, M., Lagree, T., Taylor, M.: A cross-modal concept detection and caption prediction approach in ImageCLEFcaption track of ImageCLEF 2017. (2017) 18. Ben Abacha, A., Garc a Seco de Herrera, A., Gayen, S., Demner-Fushman, D., Antani, S.: NLM at ImageCLEF 2017 caption task. CLEF working notes, CEUR (2017)