This paper presents an overview of the ImageCLEF 2017 caption tasks on the analysis of images from the biomedical literature. Two subtasks were proposed to the participants: a concept detection task and caption prediction task, both using only images as input. The two subtasks tackle the problem of providing image interpretation by extracting concepts and predicting a caption based on the visual information of an image alone. A dataset of 184,000 figure-caption pairs from the biomedical open access literature (PubMed Central) are provided as a testbed with the majority of them as trainign data and then 10,000 as validation and 10,000 as test data. Across two tasks, 11 participating groups submitted 71 runs. While the domain remains challenging and the data highly heterogeneous, we can note some surprisingly good results of the difficult task with a quality that could be beneficial for health applications by better exploiting the visual content of biomedical figures.
Interpreting and summarizing the insights gained from medical images such as radiography or biopsy samples is a time-consuming task that involves highly trained experts and often represents a bottleneck in clinical diagnosis pipelines. As a consequence, there is a considerable need for automatic methods that can approximate the mapping from visual information to condensed textual descriptions. ImageCLEF 4 is an evaluation campaign that has being organized as part of the CLEF initiative labs since 2003 [1,2]. The campaign offers several research tasks that welcome participation from teams around the world and change from year to year [3]. In 2017, the caption task of ImageCLEF 2017 addresses the problem of image understanding as a cross-modality matching scenario in which visual content and textual descriptors need to be aligned and concise textual interpretations of medical images are generated. A similar task was proposed in 2016 but without any submission [4], as it is a very challenging task. The task is based on a large-scale collection of figures from open access biomedical journal articles from PubMed Central (PMC) 5 . Each image is accompanied by its original caption and a set of extracted UMLS R (Unified Medical Language System R ) 6 Concept Unique Identifiers (CUIs), constituting a natural testbed for this image captioning task. A subset of PMC concentrating on clinical images and limiting the number of compound figures is used.
This paper gives an overview of the caption task at ImageCLEF 2017. Section 2 introduces the two subtasks and Section 3 the data set and ground truth. A description of the evaluation methodology is provided in Section 4. Subsequently, the participant submissions are analysed in Section 5 and Section 6 briefly discusses their respective strengths and weaknesses as well as their implications for academic research and medical practice. Finally, we conclude with an outlook to the possible future of the evaluation campaign in Section 7.
This first edition of the biomedical image captioning task at ImageCLEF comprises two sub tasks: (1) Concept Detection and (2) Caption Prediction. Figure 1 shows an example image of a tomographic angiography reconstruction along with its relevant concepts as well as the reference caption.
Concept Detection As a first step towards automatic image caption understanding, participating systems are tasked with identifying the presence of relevant biomedical concepts in medical images. Based on the visual image content, this subtask provides the building blocks for the image understanding step by identifying the individual components from which full captions can be composed.
Caption Prediction On the basis of the concept vocabulary detected in the first subtask as well as the visual information of their interaction in the image, participating systems are tasked with composing coherent natural language captions for the entirety of an image. In this step, rather than the mere coverage of visual concepts, detecting the interplay of visible elements is crucial for recreating the original image caption.
The experimental corpus is derived from scholarly biomedical articles on PMC from which we extract figures and their corresponding captions. In total, the collection is comprised of 184,614 image-caption pairs. This overall set is further split into disjunct training (164,614 pairs), validation (10,000 pairs) and test (10,000 pairs) sets. For the concept detection sub task, we used the QuickUMLS library [5] to identify all UMLS concepts mentioned in the caption text.
The subset of PMC was created using an automated method to classify all 3 million images of PMC from early 2016 into image types [6] fully automatically. We keep clinical image types and remove compound figures. As PMC contains many compound figures and as the method was fully automatic we have approximately 10-20% of the images that are either compound or non-clinical, which creates noise in the data set and makes the task even more challenging.
The evaluation of both sub tasks is conducted separately. For the concept detection task, we measure the balanced precision and recall trade-off in terms of F 1 scores. To this end, we use Python's scikit-learn (v0.17.1-2) library. We compute micro F 1 per image and average across all test images. A total of 393 reference captions in the test set do not contain any UMLS concepts. The respective images are excluded from the evaluation.
Caption prediction performance is assessed on the basis of BLEU scores [7] using the Python NLTK (v3.2.2) default implementation. Candidate captions are lower cased, stripped of all punctuation and English stop words. Finally, to increase coverage, we apply Snowball stemming. BLEU scores are computed per reference image, treating each entire caption as a sentence, even though it may contain multiple natural sentences. We report average BLEU scores across all 10,000 test images.
The source code of both evaluation scripts is available on the task Web page7 .
We received a total of 71 submissions by 11 individual teams. Table 1 gives an overview of all participants and their runs. There was a limit of at most 10 runs per team and sub task and the submissions are roughly evenly split between tasks. The call for contributions did not initially make any assumptions about the kinds of strategies and external data that participants would rely on. As a consequence, in this first edition of the task, we see a broad range of performance scores as well as methods being applied. Evaluation of the results showed that some teams employed methods that were at least partially trained on external resources including PMC articles. Since such approaches cannot be guaranteed to have respected our division into training, validation and test folds and might subsequently leak test examples into the training process, we separately list runs relying exclusively on the official collection as well as those making use of external information.
The concept detection task received 37 runs from 9 participating groups. Table 2 lists the performance of all official (no external information used) runs. The global overview of all runs, including those using external information, can be found in Table 3.
The vast majority of runs was purely automatic (A) in nature with only few submissions relying on some form of manual intervention (M). There was no noticeable advantage of relying on manual interventions as all manual runs lie well in the center of the performance score range. While most teams rely on some form of convolutional neural networks to represent visual information (NLM [18], PRNA [9], BMET [12], AAI [8], MAMI [15], MUPB [16]), some chose more traditional bag-of-visual-words representations (IPL [13], MSU [17]) or even relied on mixtures of both representation types (UAPT [11]). While, on average, CNN-based models seem to deliver more robust results, some of the most competitive submissions are purely based on traditional features. The use of very deep residual networks (AAI [8], MAMI [15], MUPB [16]), on average, did not introduce significant improvements over shallower CNN versions. On top of the basic image representation approaches, we see a broad range of affiliate techniques used for recognizing bio-medical concepts. PRNA [9] successfully rely on attention models for image understanding which seems to introduce a considerable relative advantage over other model variants. The use of convolutional de-noising auto-encoders (UAPT [11]) for unsupervised representation learning did not seem to lead to considerable improvements.
Several groups included retrieval-based methods that would identify highly visually related images in the official training set (IPL [13], MSU [17]) or an external collection of images (NLM [18]). The captions of such related images are then scanned for bio-medical concepts to be assigned to the candidate image. This approach generally resulted in very good results, among them several of the best-performing submissions for the task.
The harder caption prediction task received 34 runs from 5 participating groups. Table 4 lists the performance of all official (no external information used) runs.
The global overview of all runs, including those using external information, can be found in Table 5. For this task, no manual runs were submitted. Most submitted runs are based on the teams' respective contributions to the concept detection task expanded by language modeling capabilities. Often this takes the form of recurrent neural networks (ISIA [14], BCSG [10], PRNA [9], BMET [12]), making the CNN + LSTM combination a frequently-used setup.
As for the first sub task, the use of retrieval-based methods to identify highly visually related images and using their captions as a starting point for candidate caption generation (ISIA [14] MSU [17], NLM [18]) resulted in highly competitive performance.
There are several observations that should be taken into account when analyzing the results presented in the previous section. Most notably, as a consequence of the data source (scholarly biomedical journal articles), the collection contains a considerable amount of noise in the form of compound figures with potentially highly heterogeneous content. In future editions of this task, we will consider This paper presents an overview of the ImageCLEF 2017 biomedical image captioning task. We consider the sub tasks of concept detection and full caption prediction. The participating groups investigated the use of a wide range of image understanding techniques. Especially neural network methods are highly popular and delivered convincing performance on these hard problems. The individually relatively low scores motivate further homogenization of tasks and collection in future editions of the challenge.
This research was supported in part by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC).
using a less diverse source of images such as radiology/pathology in order to reduce the amount of variation in the data.
Secondly, the UMLS concept extraction employed here is a probabilistic process that introduces its own errors. As a consequence, there are several training captions that do not contain any UMLS concepts, making such examples difficult to use for concept detection purposes. In the future, we will rely on more rigorous filtering to ensure good concept coverage across training, validation and test data.
Finally, there should have been a clearer specification of what external material, if any, is permissible for use. The teams employed a wide number of third-party material ranging from general academic collections such as Ima-geNet, mainly in the form of pre-trained networks, to corpora of scholarly articles. While the former do not represent a major problem, the latter could, conceivably contain the exact image and caption pairs of our test set, the use of which would create a strong advantage and a non-realistic setting for really novel data. The experimental overview shows some evidence of this happening when methods using PubMed Central images in the training step vastly outperform all competitors on both tasks. For this reason, we made the conservative decision to separate between official runs using no external information at all and those that used third-party material. In the future, we will more carefully specify which kind of external material is safe to use. It does make sense to allow for external data to be used but it needs to be made clear that no test data are included.