This paper describes the approach proposed by the UACH-VisionLab team for the ImageCLEFmedical Concept Detection subtask 2024. The objective of this subtask is to assign medical concepts to images automatically. In particular, 1,945 distinct Clinical Concepts of Unique Identifiers (CUIs) must be associated with medical images representing a multi-label classification (MLC) problem. In this context, the ImageCLEFmedical Concept Detection subtask provides a multi-label dataset in which a medical image may contain multiple descriptive labels. The class imbalance problem in MLC poses a challenge where the samples and their corresponding labels are not uniformly distributed over the dataset. To address this challenge, our approach employs an ensemble of five EficientNet B0 (ENB0) neural architectures. An initial neural network, ENB0, classifies each image into all possible labels. Based on the classification results, we create subgroups of multi-label datasets considering specific CUIs, such as ultrasonography, bone structure of the cranium, angiogram, and lower extremity. A separate ENB0 architecture is trained for each of these subgroups. Finally, the outputs of these five neural architectures are combined to generate the final prediction results. Our proposal ranks 5th place in the ImageCLEFmedical Concept Detection subtask, achieving an F1-score of 0.59. The code to implement our proposal can be found in https://github.com/axelm11/CLEF-ImageCLEF-2024.
image with 1,945 possible medical concepts. Given the high imbalance of the dataset, an additional four ENB0 models were trained to identify specific concepts and improve the performance of our proposal.
The rest of this paper is organized as follows: Section 2 presents a general description of the ImageCLEFmedical dataset, Section 3 introduces our approach, and Sections 4 and 5 provide results and conclusions.
The multimodal data utilized in the ImageCLEFmedical Lab is derived from the Radiology Object in
Context version 2 (ROCOv2) dataset [
The task of assigning the 1,945 possible medical concepts to each image in the ImageCLEFmedical dataset is highly challenging, given the high level of complexity involved. For instance, images obtained from the same image modality may describe diferent conditions afecting diferent parts of the body. This is exemplified in Figure 2, where images corresponding to the same modality, X-Ray Computed Tomography, show diferent parts of the body emphasizing diferent medical concepts.
Another case is presented in Figure 3, where diferent image modalities present the same medical CUI. In this case, an angiogram, plain x-ray, and magnetic resonance imaging are associated with the CUI heart. Therefore, it is possible that one CUI can be present in diferent image modalities.
Figure 4 shows an additional challenging scenario where images that appear to be highly similar may, in fact, have diferent CUIs.
Our proposal is based on the baseline model provided by the ImageCLEFmedical 2024 organizers, an
EficientNet B0 (ENB0) neural architecture. Our team evaluates diferent neural architecture models, such
as ResNet [
Figure 5 shows a block diagram of the proposed approach. First, an initial ENB0 model is trained to classify all the images of the training dataset on all the possible CUIs of the challenge. The output of this model is a vector of dimensionality 1,945. Then, four subgroups are defined based on the classification results of the ultrasonography, bone structure of cranium, lower extremity and angiogram CUIs. If an image is classified within any of the four aforementioned concepts, it is considered to be part of a specific subgroup. Once the subgroups have been defined, they are trained with a separate ENB0 model to identify the possible medical concepts they contain. During training, we consider it appropriate to eliminate those CUIs with a very high or low-frequency appearance to avoid severe class imbalance issues.
For example, the concept plain x-ray is a very common concept. Therefore, it is eliminated from all the subgroups. For low-frequency concepts, we consider those CUIs with a support set of at least 50 samples and a maximum of 20 concepts to predict for each model.
Then, the proposed methodology is as follows. If the initial ENB0 identifies that the input medical image contains a CUI associated with the concepts of ultrasonography, bone structure of cranium, lower extremity or angiogram, then the ENB0 model trained with the specific subgroup will also analyze this input image and will produce an output prediction. All possible predictions identified by the second ENB0 will be included in the initial prediction. In other words, four ENB0 neural architectures are employed to enhance the outcome of the initial model. To ensure a precise final prediction, it is essential to exercise caution in determining the location of the CUI, as the output dimensionality of these models difers. Figure 6 illustrates this procedure. In this example, the angiogram concept is identified, and the prediction of the model trained with this specific subgroup is utilized to generate the final prediction result. In this case, the second ENB0 model detects four new concepts included in the final prediction.
Once we define the four subgroups, we proceed to analyze the relationship between the diferent CUIs they contain. Figure 7 shows the chord diagram of the angiogram concept. This figure illustrates the relationship between the CUIs within this subgroup. The nodes represent the diferent concepts, and the width of the edges is proportional to the relationship between the two nodes. Table 2 provides a more detailed overview of the diferent concepts within this subgroup and the support set of each of them. The most frequent concepts are the anterior descending branch of left coronary artery, stent device, right coronary artery structure and stenosis. As can be observed in Figure 7, the anterior descending branch of left coronary artery has a strong relationship with stenosis, pulmonary artery structure, and structure of circumflex branch of left coronary artery . Furthermore, it is noteworthy that the right coronary artery structure is a frequent medical concept in this subgroup that exhibits a constant relationship with the majority of other concepts, with the exception of pseudoaneurysm.
Figure 8 shows the chord diagram of the medical concept bone structure of cranium. Table 3 shows the specific canonical names of this subgroup and their support set. As can be observed, mandible is the more common medical concept. It has a strong relationship with permanent premolar tooth, and maxilla but also, the concepts tooth structure, tooth root structure and structure of wisdom tooth are related to it. On the contrary, X-Ray Computed Tomography is only slightly related to maxilla and the CUI C1266909 (this CUI does not present a canonical name associated with it).
Figure 9 and Table 4 show the chord diagram and CUIs, canonical names, and support set of the lower extremity subgroup. Femur is the most frequent concept with a strong relationship with cerebral cortex, axis vertebra, and head of femur. We would like to point out that we are not sure if the cerebral cortex should be the correct canonical name of C0007776. Furthermore, it can be observed that the medical concepts of bone plates and screw are closely related.
Ultrasonography is our last subgroup. Figure 10 shows its relationship chord diagram, and Table 5 presents the canonical names and support set of this subgroup. Left ventricular structure and right ventricular structure are the more common concepts and present a high relationship between them. Right atrial structure is another common concept, and it can be observed that it is associated with the concepts left ventricular structure, right ventricular structure and left atrial structure.
All the neural models were trained on an NVIDIA GeForce RTX 3080 Ti 12GB GPU using the PyTorch framework and the Adam Optimizer, with an initial learning rate of 1e-3 using a batch size of 64.
Table 6 shows the results of our team, UACH-VisionLab, with the test partition dataset. These results were provided by the ImageCLEFmedical Lab 2024 organizers. The F1-score is a measure of the harmonic mean of precision and recall. A secondary F1-score was calculated using a subset of concepts that was manually curated. Two runs were submitted by our team. The first run use a drop path rate of 0.2 while the second a drop path rate of 0.3, with a weight decay factor of 1e-5.
The results presented in Table 6 demonstrate that the first run achieves a superior performance. The increase in the drop path rate and the use of the L2 regularization method afect the performance of the model, reducing its generalization ability with test data.
In order to gain a deeper understanding of the manner in which the incorporation of the four ENB0 models enhances the performance of our approach, Table 7 presents the results of the precision, recall, and F1-score metrics on randomly selected CUIs. The first three columns show the results obtained when only one ENB0 model is employed, defined as the “Base" model. Subsequently, the approach was further enhanced by incorporating the training of the lower extremity (LE) subgroup defining the “Base+LE" approach. The “Base+LE+Angio" approach was created by additionally including the angiogram subgroup. The “Base+LE+Angio+Ultrasono" approach was constructed by combining the LE and angiogram subgroups with ultrasonography. Finally, the “Base+LE+Angio+Ultrasono+Cranium" approach integrates the bone structure of cranium subgroup.
A green highlight in Table 7 indicates a metric improvement, whereas a yellow highlight indicates a metric decrease. It is important to note that the improvements in the F1-score are mainly related to an increase in the recall score. The recall metric measures how often a true positive image is identified, whereas the precision metric considers how many positive predictions are true positive samples. Consequently, if the model detects only one true positive sample with a specific CUI, the precision metric will be high. In contrast, the recall metric will exhibit a low performance (as observed, for example, in the third row of Table 7 where many false negative samples are detected). Consequently, with fewer false negative detections but more false positives, the precision metric will decrease (highlighted in yellow), while the recall metric will increase, resulting in an improved F1-score metric (highlighted in green).
The improvements in the F1-score metric resulting from the incorporation of the lower extremity subgroup (Base+LE apporach) are structure of left hip, femur, joint capsule, screw, and head of femur. All of these medical concepts are considered in the training of this subgroup.
The improvement in the concepts detection resulting from the incorporation of the angiogram subgroup (Base+LE+Angio approach) includes the stent device, caudal, structure of circumflex branch of left coronary artery, collateral branch of vessel, pseudoaneurysm and vessel positions. It should be noted that all the aforementioned improvements, which had been reported in the previous approach (Base+LE), are maintained in this one, but only those that are new are highlighted in these three columns. This same reporting strategy is used in the remaining approaches.
The training and incorporation of the ultrasonography subgroup results in the Base+LE+Angio+Ultrasono approach. The concepts that demonstrate an improvement in the F1-score metric are liver, heart atrium, right atrial structure, aorta, mitral valve, right ventricular structure, uterus, heart ventricle, thrombus, and pericardial efusion . The medical concept heart atrium was also slightly modified with the training and incorporation of the bone structure of cranium subgroup. However, this is the only concept that was modified. No additional improvements could be identified with the Base+LE+Angio+Ultrasono+Cranium approach.
This working note paper presents the approach and results of the UACH-VisionLab team on the ImageCLEFmedical 2024 Concept Detection subtask. An analysis of the results yielded by the baseline code provided by the organizers reveals a significant imbalance issue in the context of multi-label classification. Therefore, we consider it appropriate to define subgroups with the aim of reducing this class imbalance problem. The medical concepts of ultrasonography, bone structure of the cranium, lower extremity and angiogram are identified as appropriate for use in the construction of these subgroups. Each subgroup is trained separately, and their results are merged with those produced by an initial ENB0 neural model.
Upon examination of the validation results obtained in the various iterations of our experiments, we observe an increase in the recall metric. This indicates that our approach has reduced the number of false negative detections, which is the behavior we are looking for in class imbalance datasets. However, it has also resulted in an increase in the number of false positives, decreasing the precision metric. The only subgroup that does not produce an improvement in the metric results is the bone structure of cranium. Further investigation is required in order to gain an understanding of this behavior.
A chord diagram of the formed subgroups provides a more comprehensive understanding of the diverse concepts within them and their interconnections. Unfortunately, due to time constraints, we were unable to incorporate this crucial knowledge into the training of the models. However, we consider it to be of paramount importance, and we intend to incorporate this information into future approaches.