ImageCLEF is an ongoing evaluation event launched in 2003 as part of the Cross Language Evaluation Forum (CLEF) [1]. In 2024, the ImageCLEFmedical Lab presents the 8th edition of the automatic image captioning task, which consists of two subtasks: concept detection and caption prediction [2]. The objective of the concept detection subtask is to identify the Unified Medical Language System (UMLS) concepts of each image. These concepts are unique identifiers assigned to different medical-related terms. The training, validation, and test datasets for this subtask comprise 70,108, 9,972, and 17,237 images, respectively. This subtask is considered a multi-label classification problem, where 1,945 different concepts must be detected and a single medical image can be associated with multiple labels. The dataset is highly imbalanced, with four of the most prevalent concepts having a frequency of occurrence in the training set of 24,227, 19,363, 11,296, and 9,870, in contrast to 306 classes that have ten or fewer images. For these reasons, this dataset is particularly challenging and complex, providing an ideal setting for the development of new deep learning (DL) approaches where robust solutions must be capable of identifying the different concepts for each medical image.
In this work, we present our approach, which we submit as part of the UACH-Vision Lab group for the ImageCLEFmedical Concept Detection subtask. This proposal consists of an ensemble of five deep learning models based on the EfficientNet B0 (ENB0) architecture [3]. An initial ENB0 associates each image with 1,945 possible medical concepts. Given the high imbalance of the dataset, an additional four ENB0 models were trained to identify specific concepts and improve the performance of our proposal.
The rest of this paper is organized as follows: Section 2 presents a general description of the ImageCLEFmedical dataset, Section 3 introduces our approach, and Sections 4 and 5 provide results and conclusions.
The multimodal data utilized in the ImageCLEFmedical Lab is derived from the Radiology Object in Context version 2 (ROCOv2) dataset [4]. This dataset consists of radiological images accompanied by their respective medical concepts and captions. It is comprised of three distinct subsets: the training set, the validation set, and the test set. The training and validation datasets are accompanied by commaseparated value (CSV) files, which contain the medical image identifiers and the corresponding Concept Unique Identifiers (CUIs). The objective of the concept detection task is to automatically assign the corresponding CUIs to the different images of the dataset. Figure 1 shows a visual representation of the medical concepts associated with the different CUIs. In this case, the size of each word is related to its frequency. Among the most frequently occurring concepts are X-Ray Computed Tomography, Plain x-ray, Ultrasonography, Magnetic Resonance Imaging, and Chest, to mention some.
The task of assigning the 1,945 possible medical concepts to each image in the ImageCLEFmedical dataset is highly challenging, given the high level of complexity involved. For instance, images obtained from the same image modality may describe different conditions affecting different parts of the body. This is exemplified in Figure 2, where images corresponding to the same modality, X-Ray Computed Tomography, show different parts of the body emphasizing different medical concepts.
Another case is presented in Figure 3, where different image modalities present the same medical CUI. In this case, an angiogram, plain x-ray, and magnetic resonance imaging are associated with the CUI heart. Therefore, it is possible that one CUI can be present in different image modalities.
Figure 4 shows an additional challenging scenario where images that appear to be highly similar may, in fact, have different CUIs.
Our proposal is based on the baseline model provided by the ImageCLEFmedical 2024 organizers, an EfficientNet B0 (ENB0) neural architecture. Our team evaluates different neural architecture models, such as ResNet [5], DenseNet [6], the Vision Transformer (ViT) [7], and Convolutional vision Transformer (CvT) [8]. However, the one proposed by the organizers yielded the best F1-scores with the validation set. The results of the ENB0 model indicate that certain CUIs exhibit highly accurate F1 performance while others exhibit zero performance. This discrepancy is primarily attributed to the multi-label class imbalance issue inherent in real-world application datasets [9,10,11,12,13]. Table 1 presents the top eight best F1-score performances. Based on these results, we select specific CUIs to create four multi-label subgroups to train and validate separate ENB0 models. The number of support samples and visual similarities in the images were considered when selecting these CUIs. For example, the categories bone structure of cranium, lower extremity and angiogram exhibit a comparable number of samples. In contrast, ultrasonography is a particularly interesting image modality, given the homogeneity of the images within this subgroup.
Figure 5 shows a block diagram of the proposed approach. First, an initial ENB0 model is trained to classify all the images of the training dataset on all the possible CUIs of the challenge. The output of this model is a vector of dimensionality 1,945. Then, four subgroups are defined based on the classification results of the ultrasonography, bone structure of cranium, lower extremity and angiogram CUIs. If an image is classified within any of the four aforementioned concepts, it is considered to be part of a specific subgroup. Once the subgroups have been defined, they are trained with a separate ENB0 model to identify the possible medical concepts they contain. During training, we consider it appropriate to eliminate those CUIs with a very high or low-frequency appearance to avoid severe class imbalance issues. For example, the concept plain x-ray is a very common concept. Therefore, it is eliminated from all the subgroups. For low-frequency concepts, we consider those CUIs with a support set of at least 50 samples and a maximum of 20 concepts to predict for each model.
Then, the proposed methodology is as follows. If the initial ENB0 identifies that the input medical image contains a CUI associated with the concepts of ultrasonography, bone structure of cranium, lower extremity or angiogram, then the ENB0 model trained with the specific subgroup will also analyze this input image and will produce an output prediction. All possible predictions identified by the second ENB0 will be included in the initial prediction. In other words, four ENB0 neural architectures are employed to enhance the outcome of the initial model. To ensure a precise final prediction, it is essential to exercise caution in determining the location of the CUI, as the output dimensionality of these models differs. Figure 6 illustrates this procedure. In this example, the angiogram concept is identified, and the prediction of the model trained with this specific subgroup is utilized to generate the final prediction result. In this case, the second ENB0 model detects four new concepts included in the final prediction.
Once we define the four subgroups, we proceed to analyze the relationship between the different CUIs they contain. Figure 7 shows the chord diagram of the angiogram concept. This figure illustrates the relationship between the CUIs within this subgroup. The nodes represent the different concepts, and the width of the edges is proportional to the relationship between the two nodes. Table 2 provides a more detailed overview of the different concepts within this subgroup and the support set of each of them. The most frequent concepts are the anterior descending branch of left coronary artery, stent device, right coronary artery structure and stenosis. As can be observed in Figure 7, the anterior descending branch of left coronary artery has a strong relationship with stenosis, pulmonary artery structure, and structure of circumflex branch of left coronary artery. Furthermore, it is noteworthy that the right coronary artery structure is a frequent medical concept in this subgroup that exhibits a constant relationship with the majority of other concepts, with the exception of pseudoaneurysm.
Figure 8 shows the chord diagram of the medical concept bone structure of cranium. Table 3 shows the specific canonical names of this subgroup and their support set. As can be observed, mandible is the more common medical concept. It has a strong relationship with permanent premolar tooth, and maxilla but also, the concepts tooth structure, tooth root structure and structure of wisdom tooth are related to it. On the contrary, X-Ray Computed Tomography is only slightly related to maxilla and the CUI C1266909 (this CUI does not present a canonical name associated with it). Figure 9 and Table 4 show the chord diagram and CUIs, canonical names, and support set of the lower extremity subgroup. Femur is the most frequent concept with a strong relationship with cerebral cortex, axis vertebra, and head of femur. We would like to point out that we are not sure if the cerebral cortex should be the correct canonical name of C0007776. Furthermore, it can be observed that the medical concepts of bone plates and screw are closely related.
Ultrasonography is our last subgroup. Figure 10 shows its relationship chord diagram, and Table 5 presents the canonical names and support set of this subgroup. Left ventricular structure and right ventricular structure are the more common concepts and present a high relationship between them. Right atrial structure is another common concept, and it can be observed that it is associated with the concepts left ventricular structure, right ventricular structure and left atrial structure.
This working note paper presents the approach and results of the UACH-VisionLab team on the ImageCLEFmedical 2024 Concept Detection subtask. An analysis of the results yielded by the baseline code provided by the organizers reveals a significant imbalance issue in the context of multi-label classification. Therefore, we consider it appropriate to define subgroups with the aim of reducing this class imbalance problem. The medical concepts of ultrasonography, bone structure of the cranium, lower extremity and angiogram are identified as appropriate for use in the construction of these subgroups. Each subgroup is trained separately, and their results are merged with those produced by an initial ENB0 neural model. Upon examination of the validation results obtained in the various iterations experiments, we observe an increase in the recall metric. This indicates that our approach has reduced the number of false negative detections, which is the behavior we are looking for in class imbalance datasets. However, it has also resulted in an increase in the number of false positives, decreasing the precision metric. The only subgroup that does not produce an improvement in the metric results is the bone structure of cranium. Further investigation is required in order to gain an understanding of this behavior.
A chord diagram of the formed subgroups provides a more comprehensive understanding of the diverse concepts within them and their interconnections. Unfortunately, due to time constraints, we were unable to incorporate this crucial knowledge into the training of the models. However, we consider it to be of paramount importance, and we intend to incorporate this information into future approaches.
All the neural models were trained on an NVIDIA GeForce RTX 3080 Ti 12GB GPU using the PyTorch framework and the Adam Optimizer, with an initial learning rate of 1e-3 using a batch size of 64.
Table 6 shows the results of our team, UACH-VisionLab, with the test partition dataset. These results were provided by the ImageCLEFmedical Lab 2024 organizers. The F1-score is a measure of the harmonic mean of precision and recall. A secondary F1-score was calculated using a subset of concepts that was manually curated. Two runs were submitted by our team. The first run use a drop path rate of 0.2 while the second a drop path rate of 0.3, with a weight decay factor of 1e-5.
The results presented in Table 6 demonstrate that the first run achieves a superior performance. The increase in the drop path rate and the use of the L2 regularization method affect the performance of the model, reducing its generalization ability with test data.
In order to gain a deeper understanding of the manner in which the incorporation of the four ENB0 models enhances the performance of our approach, Table 7 presents the results of the precision, recall, and F1-score metrics on randomly selected CUIs. The first three columns show the results obtained when only one ENB0 model is employed, defined as the "Base" model. Subsequently, the approach was further enhanced by incorporating the training of the lower extremity (LE) subgroup defining the "Base+LE" approach. The "Base+LE+Angio" approach was created by additionally including the angiogram subgroup. The "Base+LE+Angio+Ultrasono" approach was constructed by combining the LE and angiogram subgroups with ultrasonography. Finally, the "Base+LE+Angio+Ultrasono+Cranium" approach integrates the bone structure of cranium subgroup.
A green highlight in Table 7 indicates a metric improvement, whereas a yellow highlight indicates a metric decrease. It is important to note that the improvements in the F1-score are mainly related to an increase in the recall score. The recall metric measures how often a true positive image is identified, whereas the precision metric considers how many positive predictions are true positive samples. Consequently, if the model detects only one true positive sample with a specific CUI, the precision metric will be high. In contrast, the recall metric will exhibit a low performance (as observed, for example, in the third row of Table 7 where many false negative samples are detected). Consequently, with fewer false negative detections but more false positives, the precision metric will decrease (highlighted in yellow), while the recall metric will increase, resulting in an improved F1-score metric (highlighted in green).
The improvements in the F1-score metric resulting from the incorporation of the lower extremity subgroup (Base+LE apporach) are structure of left hip, femur, joint capsule, screw, and head of femur. All of these medical concepts are considered in the training of this subgroup.
The improvement in the concepts detection resulting from the incorporation of the angiogram subgroup (Base+LE+Angio approach) includes the stent device, caudal, structure of circumflex branch of left coronary artery, collateral branch of vessel, pseudoaneurysm and vessel positions. It should be noted that all the aforementioned improvements, which had been reported in the previous approach (Base+LE), are maintained in this one, but only those that are new are highlighted in these three columns. This same reporting strategy is used in the remaining approaches.
The training and incorporation of the ultrasonography subgroup results in the Base+LE+Angio+Ultrasono approach. The concepts that demonstrate an improvement in the F1-score metric are liver, heart atrium, right atrial structure, aorta, mitral valve, right ventricular structure, uterus, heart ventricle, thrombus, and pericardial effusion. The medical concept heart atrium was also slightly modified with the training and incorporation of the bone structure of cranium subgroup. However, this is the only concept that was modified. No additional improvements could be identified with the Base+LE+Angio+Ultrasono+Cranium approach.