Digital medical imaging has opened new advances in clinical decision support and treatment procedures since its inception. This leads to the creation of huge amounts of data that are often not fully exploited. The development and evaluation of representation learning techniques for automatic detection of concepts in medical images can make way for improved indexing, processing and retrieval capabilities in medical imaging archives. This paper discloses several independent approaches for multi-label classification of biomedical concepts, in the context of the ImageCLEFmed Caption challenge of 2019. We emphasize the use of threshold tuning to optimize the quality of sample retrieval, as well as the diferences between training a convolutional neural network end-to-end for supervised image classification, and training unsupervised learning models before linear classifiers. In the test results, the best mean F1-score of 0.206 was obtained with the supervised approach, albeit with images of a larger resolution than for the dual-stage approaches.
Medical imaging modalities are an essential and well established medium, and as the amount of medical images is dramatically growing, automatic and semiautomatic algorithms are quite pertinent for the extraction of information from biomedical image data [14]. Therefore, deep learning techniques are becoming increasingly useful and necessary for this aim, posing as a valuable key for the development of representation learning techniques, and ultimately for improving the quality of systems in healthcare.
The process of annotating images with useful information in this context is
time-consuming and usually requires medical expertise. The development of
powerful representations of images could enable the automatic detection of
biomedical concepts in a medical imaging data set. The ImageCLEFmed initiative,
inserted in ImageCLEF [
This paper presents our solution proposal for the concept detection task, describing our methodology and evaluating its performance under the ImageCLEF 2019 challenge. 2
For this task, a data set with a total of 70,786 radiology images of several medical
imaging modalities was provided from Radiology Objects in Context (ROCO)
[
The ImageCLEF Caption 2019 data set includes an overwhelming number of 5,216 unique concepts, not all of which can be reasonably considered due to the very small number of positive samples in the training and validation splits. In all of the methods described next, we have admitted only the 1,100 concepts with the highest number of samples with a positive occurrence of that concept (henceforth named positive samples). The label vectors were built based on a direct mapping from the UMLS concept unique identifier (CUI) to an index in the vector. The reverse mapping was kept for producing the textual list of concepts.
Also contributing to this decision, was the observed imbalance in the number of positives of each label, as discerned in Figure 1, making them dificult to train classifiers and evaluate them. By considering the 1,100 most frequent concepts, one could ensure, in the extreme case, a minimum number of 29 positive samples in the training set and 2 positive samples in the validation set. We admit that attempting to detect any less frequent concepts is unlikely to result in useful classifiers.
At ImageCLEF 2018, the highest mean F1-score was obtained through
unsupervised methods, namely by using the features of an adversarial auto-encoder,
followed by logistic regression [
Convolutional neural networks (CNNs) are considered one of the best approaches for image classification [ 15]. Unlike a 2-stage approach, where the extracted feature descriptors are served as input to a trainable classifier, the images themselves are used in the learning process of the CNN. This could lead to a more focused guidance of the feature learning process across the multiple layers of the network. It was of our interest to compare this common supervised image classification method with the 2-stage pipeline involving unsupervised learning methods and simple classifiers.
Therefore, we have addressed the concept detection task with multiple independent approaches, which can be divided in two major groups: – Through image representations, obtained by the implementation of several feature extraction methods: • Color and edge directivity descriptors, that were used as image descriptors. • An auto-encoder and an adversarial auto-encoder was trained and features were extracted from its bottleneck vector. • The ensemble of features obtained from the previous point was used for classification. – An end-to-end approach, using two deep learning architectures: • A simple convolutional neural network model was assumed.
• A residual neural network.
In every case, some form of optimum threshold tuning was employed, to overcome the classifier’s focus on accuracy rather than F-measure. Further details are given in Sections 2.4 and 2.5.
Neural network training, feature extraction, and logistic regression were conducted using TensorFlow on one of the GPUs of an NVIDIA Tesla K80 graphics card in an Ubuntu server machine. 2.1
As traditional visual feature extraction algorithms are still very often considered
in medical image recognition, these techniques contribute to a baseline, which
we expect modern deep learning methods to surpass. For this purpose, we have
extracted Color and Edge Directivity Descriptors (CEDDs) [
For the unsupervised extraction of visual features from the medical images, an
adversarial auto-encoder (AAE) [
x′ = G(E(x)) Lrec(x, x′) = 1 2N
N ∑ (xi − x′i)2
i V (E, D) = min max Ez∼pz [log D(z)] + Ex∼p(x)[log (1 − D(E(x)))]
E D
Both encoder and decoder architecture are based on the ResNet19 [
(1) (2)
The AAE was trained for 30 epochs on the training set, with images resized
to a minimum dimension of 72 pixels and then randomly cropped to a 64 x 64
square. All three RGB channels were kept with their values normalized to the
[
Moreover, in order to better understand the influence of the adversarial autoencoder’s regularization phase, a separate auto-encoder (AE) was trained with the same encoder and decoder characteristics as in the adversarial one, but without the adversarial loss. In this case, each iteration is only composed of the reconstruction step, mainly influenced by the loss Lrec. 2.3
The combination of information from diferent techniques seems intuitively
appealing for improving the performance of the learning process. More recently,
there is an attempt of combining traditional image characteristics with high-level
features [
Although it is known that the combination of low-level features, such as color and texture in CEDD, is important to the quality of visual features, it is not as clear whether the combination of high-level features obtained from two autoencoders can benefit from an early fusion. Hence, we hereby took the features obtained from the AE and the AAE, and concatenated them to form a 1024dimensional feature set, for its subsequent use in the logistic regression step alongside the other feature sets. 2.4
For each of the previously described set of features, logistic regression
classiifers were trained for the chosen labels, for a set of predefined operating point
thresholds: 0.075, 0.1, 0.125, and 0.15. The linear classifiers were trained with
the Adam optimizer [
Aware of the presence of concepts with a very low number of positive samples, one may wonder whether certain labels were not well trained or resulted in uninformative classifiers. To mitigate this, we calculated the area under the curve (AUC) of each binary classifier’s receiver operating characteristic curve (ROC). Afterwards, we have tested whether ignoring the predictions of concepts where the AUC was lower that 0.5 would potentially improve the overall performance. Testing this hypothesis on the validation set, it is revealed that this would improve the mean F1-score in most cases, albeit only slightly. For the features of the AAE, as an example, this tweak has only increased the score by 5 × 10−5. With the features of the simple AE, the score was only improved by 7.3 × 10−4. We held this mechanism away from the classifiers trained with the CEDD feature set.
Probabilistic classifiers minimizing binary cross-entropy inherently optimize
for the accuracy of predictions. However, accuracy is overoptimistic when labels
have a very low number of positives, as is the case in this task, making a poor
metric for the classifiers’ usefulness. As recognized by past work in the scope of
ImageCLEF concept detection, adjusting the operating point thresholds to
optimize the F1-score provides significant improvements in the final metric values,
in spite of the known implications of this practice [
In the event that a sample was predicted to have more than 100 concepts before submission, the list of concepts was trimmed by the less frequent concepts. In practice, this has only happened to the linear classifiers trained using CEDD. 2.5
A simple CNN was designed (Table 1) and trained for multi-label classification, thus once again treating concepts as labels. Conv2D stands for 2D convolution layer, GAP for global average pooling and FC for fully connected layer. Training samples were augmented using random square random crops, experimented with diferent sized squares. In one approach, denoted as CNN-A-256px, we used 256 pixel-wide and excluded the layer Conv2D-5. In CNN-B, the full CNN was used with 128 (CNN-B-128px) and 64 (CNN-B-64px) pixel-wide images. Validation and test samples were simply resized to fit these dimensions, according to the training process.
Moreover, to serve as a more intuitive means of comparison, the same
architecture as the encoder in the AAE and AE, based on ResNet19 [
Both end-to-end deep learning models were trained with the AMSGrad optimizer [16], with a batch size of 32 and a learning rate of 10−4 with a decay of 2x10−5 over each update and parameters β1 = 0.9 and β2 = 0.999. For each model, threshold fine tuning was performed by evaluating the F1-score performance for multiple thresholds, using the validation data set. Thereafter, we determined the threshold which would yield the optimal mean F1-score on the validation set. 3
Alongside with the metrics obtained from our submissions, we also present a brief qualitative analysis as part of our results. 3.1
The visualizations of the features for five of our models were obtained by
training dimensionality reduction algorithms, namely principal component analysis
(PCA) and uniform manifold approximation and projection (UMAP) [
and the algorithm was configured with 15 as the number of neighbors and 0.1 as the minimum distance.
For the CNN, the features were extracted at the GAP layer, whereas in the remaining the visualizations depict the features extracted before the classification process. The points associated with the concepts C0441633 (diagnostic scanning) , C0817096 (thoracics) and C0935598 (sagittal planes set) are labeled in red, green and blue, respectively, each painted in an additive fashion.
Comparing the two types of dimensionality reduction algorithms, we notice that the representations obtained with PCA have more outliers. In a good representation, the samples will be linearly separable based on their associated concepts. In both types of representations, we can identify regions in the manifold in which points of one of the chosen labels are mostly gathered, with this being more perceptible in the representations obtained with UMAP and for the CNN trained end-to-end. In fact, the clustering of representations with common labels is highly expected, even more so for the CNN, since the feature learning in this case was uniquely guided by the target labels. In general, these observations are a rough approximation of the efective performance of each method. 3.2
The metrics on the validation and test set for each submission are depicted in Table 4. Both Val F1-score and Test F1-score represent the F1-scores averaged sample-wise. When applied to the validation set, all concepts of the ground truth were considered, even though the predictions were made only assuming the 1,100 most frequent concepts on the training set, and so always predicting ”negative” for the remaining concepts.
The scores obtained from end-to-end CNN models (CNN-A-256px, CNN-B128px, CNN-B-64px) was highly varied, which demonstrates the impact of the input shape, as well as neural network architecture, in the performance of the model. With an image resolution of 64×64, this approach did not perform better than any of the 2-stage procedures. On the other hand, higher resolutions have contributed to significantly better scores.
Concerning the unsupervised methods, the mean F1-score obtained with
CEDDs (CEDD) was lower than with the deep learning architectures,
probably because they lack representation ability for high-level problems, an efect
that was also observed in prior work [
The early fusion of the features obtained from the two auto-encoders (AE + AAE) was also beneficial, resulting in a higher score than any of the two forms independently(AE and AAE), suggesting that this aggregation was not entirely redundant, thus providing another useful distribution.
It is also worth noting that, much unlike in our previous participations in the same task, the instance-wise mean F1-scores on the testing set were higher than on the validation set. This efect is even more noteworthy, since these methods relied on the validation set for threshold optimization, and as such the classifiers were fit for both the training and validation sets. This consistent discrepancy is due to the fact that the test set did not include any new concepts that were not present in the overall data set provided at the beginning of the challenge, whereas the validation set contained some concepts which were not present in the training set.
In the context of the ImageCLEFmed Caption challenge, we did an assessment of feature learning techniques for concept detection from radiology images of several medical imaging modalities. The extraction of informative – and intriguing – visual features can yield great potential for multiple use cases in medical imaging systems, including automated image labelling and content-based retrieval.
We had confirmed the greater potential of deeper architectures for the construction of more powerful representations, in comparison with low-level feature extraction algorithms. With the data set size being significantly smaller in this edition of the challenge, this was seen as an opportunity to compare end-to-end classification models with the use of unsupervised learning methods. The outperforming CNN model had a larger image size as input, making this factor a counterbalance to obtain a better F1-score than with the unsupervised models. In fact, at a late stage of these experiments, we have identified that the attempted resolution of 64 × 64 is insuficient to attain better results. In the end, a simple CNN with a higher resolution showed the best performance among these submissions. Time constraints have not enabled us to combine the two ideas together in our submissions.
The quality of the results obtained in this edition may also be attributed to
the use of threshold tuning to optimize the F1-score. Without an adjustment
of the classifiers’ operating point, these methods would have a focus towards
the highest accuracy of a prediction, which is not as useful in the context of
information retrieval. When the number of positive samples is low, the potential
retrieval of less relevant entries is compensated by a significantly greater chance
of receiving relevant images. Nevertheless, we understand that the focus of a
single metric can distort the perception of quality among multiple methods in
the challenge, such that a change of performance metric could result in diferent
rankings [
This year presented an increase in participants engagement in the challenge, which might echo the interest in solving timely situations in medical information retrieval and automated medical data analysis. We believe that further investment in the challenge, both from participants and organizers, will enable the implementation of these solutions in real-world scenarios.
This work was supported by the Integrated Programme of SR&TD “SOCA” (Ref. CENTRO-01-0145-FEDER-000010), co-funded by Centro 2020 program, Portugal 2020, European Union, through the European Regional Development Fund. 14. Pinho, E., Costa, C.: Unsupervised learning for concept detection in medical images: A comparative analysis. Applied Sciences 8(8) (2018). https://doi.org/10.3390/app8081213 15. Rawat, W., Wang, Z.: Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation 29(9), 2352 – 2449 (2017). https://doi.org/10.1162/neco_a_00990 16. Reddi, S.J., Kale, S., Kumar, S.: On the Convergence of Adam and Beyond. In: International Conference on Learning Representations (2018), https://openreview. net/forum?id=ryQu7f-RZ