Decision making in medical diagnosis is tedious and very rigorous task, hence the requirement to use more advanced and intelligent medical imaging diagnostic support systems. The automation of the recognition of medical imaging modalities and human anatomical organs gives these systems the possibility of processing, in an automatic and adapted manner, different types of images in consideration of different medical imaging modalities. It also offers better support to clinicians and patients allowing them to access to more effective image analysis and diagnostic tools. In this context, three deep learning approaches were developed and tested on six different CNN models (VGG16, VGG19, ResNet-50, Xcpetion, Inception and NASNet). Two deep transfer learning modes and an ensemble deep learning algorithm based on stacking were used. The experiments carried out on two datasets of medium and high challenges show very interesting results with F-score reaching 99% for the classification of image modalities and 98% for the classification of anatomical organs. Anatomy organs, medical imaging modalities, deep transfer learning, ensemble deep IDDM'2023: 6th International Conference on Informatics & Data-Driven Medicine, November 17 - 19, 2023, Bratislava, Slovakia Proceedings
learning, medical image processing, computer-aided diagnosis.
The medical imaging field has undergone spectacular evolution in recent decades, offering
unprecedented opportunities for the early and accurate diagnosis of various pathologies. However,
manual or conventional interpretation of medical images and segmentation of anatomical organs or
lesions remain complex and time-consuming tasks for radiologists and clinicians. In this context, the
use of machine learning techniques, and in particular deep learning, has shown promise in improving
the efficiency and accuracy of these processes. Today, the scientific community uses deep learning
algorithms to improve diagnosis and help doctors in their work [
A large number of deep learning methods use deep convolutional neural networks (CNN). They are successfully applied in medical image analysis, giving promising results. The application area
2023 Copyright for this paper by its authors. CEUR
ceur-ws.org
covers the entire spectrum of medical image analysis, including detection, segmentation, classification
and computer-aided diagnosis [
The automatic classification of medical imaging modalities and anatomical organs will enable the development of medical diagnosis support systems and the appropriate automatic processing of a large corpus of images. It also facilitates the research work of doctors and healthcare professionals, by automatically knowing the image modality, clinicians can correctly interpret the image and make precise medical decisions. For example, it allows clinicians to quickly find the medical images they need to make comparisons, diagnoses, plan treatments and track the progression of diseases.
This work presents several original contributions by addressing the problem of recognition of medical imaging modalities and anatomical organs. A problem that is rarely addressed in the literature where public datasets are very rare. The main contributions of this work are as follows: - construction of a new dataset with different challenges, created from several sources, - comparisons of six existing CNN models with simple and complex architecture according to different transfer learning strategies (VGG 16, VGG 19, ResNet-50, Xcpetion, Inception and NASNet networks). We also studied the ability to transfer knowledge and features, learned initially on the basis of ImageNet dataset containing more than 14 million non-medical images to medical images constituting a set of images of relatively small size. Therefore, we chose to explore different levels of fine tuning allowing partial transfer of the learned features from ImageNet dataset to the images from our targeted dataset. This type of transfer can be very useful when deep neural networks are pretrained on datasets very different from the subject area and when we have a very small and insufficient image dataset.
- Application of ensemble deep learning using the stacking of different CNN sub-models, for the combination of knowledge of different models and the pooling of their complementarity.
The rest of this paper is organized as follows: Section 2 presents related works on classifications of medical image modalities and anatomical organs using deep learning. In the third section, we present the different proposed approaches. The experimental results are presented and discussed in the fourth section.
The classification of medical imaging modalities is a very important preliminary step to bring
more autonomy to intelligent medical diagnostic support systems and to help clinicians access the
required medical imaging in the system. Among the existing works that focus on deep learning, we
find the work of Yu et al. [
Kim et al. [
In another study, Remedios et al. [
Chiang et al. [
Recent work of Atrey and al. [
According to the literature, we notice the intensive need to explore new ways and approaches based on the transfer learning and on the ensemble deep learning in order to achieve better results in medical diagnostic support systems and especially in the context of classification of medical imaging modalities as well as for the classification of anatomical organs which represents the object of our study.
Automated classification of anatomical organs is an important step and a prerequisite for many
medical diagnostic support systems. Spatial complexity and variability of anatomy throughout the
human body make classification difficult. In the literature we can find the review of Jiang and al. [
Takiyama et al. [
Deep learning-based approaches have yielded promising results in the classification of anatomical organs. However, they often require large amounts of data, which can be difficult to obtain in the medical field.
As part of this work, we present different approaches based on deep transfer learning that we have developed for the classification of medical imaging modalities and anatomical organs. Such classifications present several challenges that should not be overlooked during development. The main difficulties are often posed by the intra-class variability of medical images, the diversity of imaging modalities used, the complexity of anatomical structures and the unavailability of datasets of sufficient size in the medical field. The use of deep transfer learning is a better choice to design a more robust approach to these constraints. This involves the use of pre-trained CNN models on generic image datasets of sufficient volume, to benefit from representations that focus on generic and low-level image features, learned on massive and diverse data. These models are thus re-trained (on our small dataset) and refined by seeking the best level of fine tuning, making it possible to complete the initial low-level representation, valid for all kinds of images, with a second high-level representation, specific to our problem of classification of image modalities and anatomical organs.
In this context, we chose to develop six very popular CNN models (VGG16, VGG19, ResNet-50, Inception, Xception and NASNet) with performance already demonstrated in the medical field. Based on our previous study on CNN models combination [15] and it’s benefits, we also developed an ensemble deep learning which involves combining several CNNs to take advantage of their complementarities. With the stacking mechanism, we propose to use a softmax meta-model which learns the best weighting and combination of these sub-models.
modalities approaches for the classification of medical imaging
At this level, we have developed three different approaches to determine the best behavior to follow, the first two approaches concern the development of six CNN models using two transfer learning modes (features extractor and fine-tuning modes). Transfer learning makes it possible to solve the problem of the reduced size of a dataset. It consists of reusing a pre-trained model on another large dataset (even outside the medical field), preserving part of it for relevant extraction of generic characteristics and fine tuning the remaining part on our small target dataset to extract specific characteristics. This approach allows for faster learning and a more reliable model from a very small dataset.
The third approach consists of seeking the best combination of different CNN models with the stacking technique in order to take advantage of the complementarity between them. This combination has the advantage of being able to aggregate very different classifiers and significantly improve the quality of the final prediction. The use of ensemble deep learning methods is necessary when we want to take a step forward in obtaining better prediction results of medical imaging modalities.
• Approach 1: is based on transfer learning in features extractor mode. The convolutional part (features extractor) of the pre-trained CNN is completely frozen in order to preserve all the knowledge already acquired on the initial (very large) dataset. With this mode, only the classifier part (Softmax) will be adapted to the new image modalities classification task. The use of pretrained CNN models, aims to extract high-level characteristics from medical images. Then, these features were used to train modality-specific classifiers. This approach allows us to benefit from knowledge learned from large generic databases. Transfer learning with this mode is faster than that based on fine tuning, however, it requires the presence of certain similarities between the original dataset images and the target dataset images. Six CNNs are compared using this approach (see Figure 1).
Approach 2: is based on transfer learning in fine-tuning mode. In this mode, a pre-trained model is used as a starting point, but unlike the features extractor mode, a set of the last layers of the convolutional part of the model are fine-tuned during training on the new dataset, specific to medical imaging. The layers which are not refined are frozen (this concerns the layers closest to the input) to preserve certain generic knowledge of the pre-trained CNN model which has already learned on a large dataset to extract low level features (which concerns all kinds of images). For the convolution part, the number of frozen blocks must be fixed empirically in order to have a better score. Since the number of classes is different in the target dataset compared to that of the original dataset, the structure of the classifier part (Full connected layers) must be adapted to recognize the new classes of different image modalities. This method is composed of three steps, the first one concern the finetuning of the parameters of the non-fixed layers, secondly extraction of image features, finally, the last step consists of the generation of predications for the classification using the Softmax classifier.
Using fine tuning
mode of TL • Approach 3: is based on the combining mode of different CNN models with the stacking technique. In this mode, several pre-trained models are used as a starting point, we train the models on a database which only contains MRI images of different types (ex. Flair, T1w, T1wCE and T2w), we will thus obtain trained models, then we combined them with stacking.
Stacking process consists of training several CNN models independently on the same dataset. Each sub-model may have a different architecture with different settings. Once the sub-models are trained individually, a meta-model is added to the output of these models. It receives predictions from different CNNs as input and learns to combine these predictions to produce a final prediction. This meta-model can be based on any machine learning model. In our method, a softmax classifier is used as a stacking meta-model. It receives as input the different prediction probabilities coming from the output of different stacked CNN sub-models. The objective of this step is to train a new model (softmax) to learn how to best combine the contributions from each CNN sub-model.
Stacking process allows you to take advantage of the diversity of individual models by combining their strengths and mitigating their weaknesses. This approach can result in better predictive performance than any single contributing model.
It is also important to note that it is possible to distribute this approach in a real-time framework and make the CNN sub-models work in multitasking, multiprocessors, multicores or parallelism. This allows better management of the complexity generated by the stacking of sub-models. 3.2.
For the classification of anatomical organs, we will use the transfer learning in feature extractor mode. We used it by the same idea as that of the approach 1 to the classification of medical imaging modalities.
In this section, we will present the experiments that we carried out as part of our study, as well as the results obtained. First, we will present the evaluation metrics used. Then we will present in detail the datasets used with samples of each of them. After that, a description of the data augmentation technique used. Then, the results of our experiments for each approach illustrated in tables followed by comparisons and comments. 4.1.
In the literature, there are several evaluation metrics, in our case, and to evaluate the different proposed approaches, we used the following metrics: Accuracy, Precision, Recall and F1-score. • Confusion matrix: Confusion matrix or error matrix is one of the key concepts when we talk about classification problems. This matrix is a two-dimensional array (“actual” and “predicted”) and sets of “classes” in both dimensions. Our actual classifications are columns and the predicted ones are rows as shown in the table below: • Accuracy: Number of correct predictions divided by the total number of samples. Is a good measure when the classes of target variables in the data are almost balanced.
=
= 1 − 2 × + (1) (2) (3) (4)
The difference between Precision and Recall in the classification problem is that Recall gives us information about the performance of a classifier against false negatives (how many did we miss), while precision gives us information about its performance versus false positives (how many did we catch). 4.2.
Our experiments were carried out on two different datasets with different sizes and challenges. • First dataset: called MC4_Dataset (Size and average challenges): This image dataset was created by us from nine public image datasets (size of 35150 images). Then, we divided this dataset into 4 classes depending on the imaging modality used (MRI 7023 images, ultrasound 2116 images, CT-Scan 12988 images, X-ray: 12023 images). Each class is divided into subclasses according to the different anatomical organs (see Table 2). This dataset is created from different datasets and image sources presenting different degradations, resolutions, complexities, etc. This increases the challenges of our dataset which will be confronted with the developed models.
This table contains an unbalanced dataset, ex. the Ultrasound subset (2116 images) is a minority class compared to the CT-scan class (12988 images). The significant difference between the sizes of the classes can destabilize the model, limit its generalization and/or cause overfitting. To reduce these problems, we balanced the dataset by oversampling each minority class to give it the same weight as the majority class, using data augmentation: image rotation, zooming, shifting, scaling and shearing (see section 4.3).
Number of images 50682 55440 68012 67722 3000 3000 3000 3000
224x224x3
• Second RSNA-MICCAI dataset (high size and challenges)
The second used dataset, presents more challenges and more diversity compared to the first dataset. This is the public dataset “RSNA-MICCAI Brain Tumor Radio-genomic Classification” (size = 290923 images) [22]. It is used to test CNN combination in approach 3. The latter is divided into four classes of modalities (FLAIR, T1w, T1wCE and T2w). The following table summarizes the image distribution in this dataset according to the different classes. This dataset is balanced like the first dataset using data augmentation.
T1w T1wCE T2w Total
To improve deep learning on small datasets, there is a technique called data augmentation. It consists of creating new training data samples by applying simple transformations to existing data, which increases the size of the available dataset in the aim to improve model training. It has several advantages for deep learning, it allows generalization of models, by exposing the model to a variety of transformations, it becomes more resilient to variations such. The transformations used in our study for the data augmentation are rotation, zooming, shifting, scaling, cropping, etc. By using data augmentation, it is possible to improve the performance of deep learning models in an extraordinary way. Data augmentation makes it possible to avoid overfitting on complex CNN architectures (with a large number of parameters) applied to a small dataset. It also makes it possible to offer better stability and model generalization on an unbalanced dataset like the MC4_Dataset. 4.4.
In order to study the relevance of deep learning for the classification of medical imaging modalities, we tested the three proposed approaches using different CNN models with transfer learning on our datasets.
This approach was applied on each of six CNN models (VGG16, VGG19, ResNet-50, Inception 3, Xception and NASNet) already pre-trained on the ImageNet dataset. The dataset used is “MC4_Dataset” with data augmentation technique to overcome the problem of the small size in this dataset. The features extraction mode is applied with the adaptation of the Softmax classifier part to classification in 4 modalities (MRI, ultrasound, CT-scan and X-ray).
The following table summarizes the results obtained by the six CNNs:
According to this table, we see the effectiveness of the NASNet model compared to all the other models with a value of 0.99 for each of the evaluation metrics, then comes the Xception model with values less than the NASNet but very convincing values.
The goal of this experiment is to determine whether the performance of the first five CNNs (VGG 16, VGG 19, ResNet50, Inception 3, XCeption) will be enhanced by the fine-tuning mode and make them also competitive compared to the NASNet CNN.
Still working on the MC4_Dataset dataset with four output classes (MRI, ultrasound, CT-Scan and x-ray), we trained these five CNN models (already pre-trained on ImageNet) in fine-tuning mode on our medical image dataset. Each model is trained in 5 epochs with a batch-size of 100, where we have frozen a certain percentage of layers (the leftmost layers which are closer to the initial image), this process can be described as follows: • VGG16: By default, this model contains 5 convolution blocks, but in our case, we generated two models from the base model, the first was fixed at 80% (4 blocks out of 5) and the second was fixed at 60 % (3 blocks out of 5). • VGG19: In the same way as the VGG16 model, we generated two models from this model where the first was fixed at 80% and the second one at only 60%. • ResNet-50: This model contains 50 layers, while we generated only one single model from it, with a percentage of 80%. • Inception 3: We thus generated two models from this model which contains 48 layers, the first one was fixed at 90% (43 layers) and the other at 80% (38 layers). • XCeption: Based on its number of layers which is 71 layers, we thus generated two models from the base model, the first one we frozen 90% (64 layers) of its layers and the other at 80% (57 layers).
The obtained results are displayed in this table:
According to these results, we can see the effectiveness of the fine-tuning approach, and especially for the two models Inception 3 and XCeption which gave very high values in terms of accuracy, precision, recall and F1-Score (0.99) and which are also competitive with the NASNet architecture.
Given that the performance of the CNNs (approaches 1 and 2) on the MC4_Dataset dataset (average challenges) was 0.99, we considered that it was unnecessary to use the third approach (combination of CNNs) which was dedicated on all to higher challenge datasets. This is why we tested approach 3 only on the second dataset ("RSNA-MICCAI Brain" with 4 classes Flair, T1w, T1wCE, T2w) which presents greater diversity and difficulties.
Initially we tested each of the six CNNs separately (using approach 2). Subsequently we combined three best architectures (Inception 3, XCeption and NASNet), relying on Stacking mode in order to achieve a more efficient model with very promising and convincing results which are displayed in the following table: Table 6
As we see in the table, the combination of models is efficient and gave us better results comparing to other models separately. 4.5.
For the classification of anatomical organs, we tested on the six CNN models, which are already pre-trained based on ImageNet dataset. The transfer learning technique was adopted in features extraction mode in order to increase the learning results, and to take advantage of the power of the ImageNet dataset. In this case, the MC4_Dataset dataset is subdivided into 8 classes of anatomical organs: human torso acquired by scanner, human kidney acquired by scanner, brain acquired by MRI, torso acquired by MRI, female chest acquired by ultrasound, fetal head acquired by ultrasound and finally of the torso and hand both acquired by x-ray.
Table 7
The obtained results mentioned in this table, showing the effectiveness of the three models (NASNet, Inception 3 and XCeption) well recognized in the literature, giving very convincing values.
In order to select the best architecture among the six architectures seen precisely, we adopted the F1-Score as the best performance comparison criterion, due to the costs of false positives and false negatives which differ in number, which leads to obtaining additional false positives (false alerts) rather than saving false negatives.
In the first approach, we saw that the NASNet model outperformed the other models with an F1score of 0.99, but it should be noted that the NASNet in features extraction mode did not have a high challenge dataset.
In the second approach, we noticed that the results increased or decreased, depending on the number of layers frozen by the fine-tuning mode. The XCeption architecture beat the other models in fine-tuning by 80% where it improved its F1-score compared to the first approach in features extraction mode, this shows that the choice of fine-tuning levels had a significant impact to overcome the problem of the small size of the image database.
The third approach which serves to combine the three CNN architectures (Inception 3, XCeption and NASNet) where the choice was justified by the best results obtained by these models on the basis of medical imaging of different types of MRI, this approach gave us very good results on a very high challenge dataset. The model that combines the three models improved the F1-score of the best performing model (XCeption) by 5%. This leads us to the conclusion that the deep learning stacking approach is very powerful on high challenge datasets.
For the classification of anatomical organs, we noted the effectiveness of the two models (XCeption and NASNet) with the softmax classifier which had a score of 0.98. Considering the sufficient results, the application of approach three was not necessary.
This work focuses on the classification of medical imaging modalities and the classification of anatomical organs. For this, six CNN architectures were tested and compared according to three different approaches, which we proposed. The objective was to explore deep transfer learning in two modes (features extraction and fine-tuning) and ensemble deep learning using the stacking technique which combines and complements several models (the best) CNNs. The experiments were carried out on two datasets with different challenges: unbalanced MC4_Dataset (created from nine existing datasets, medium size and challenges) and RSNA-MICCAI Brain (very high size and challenges). The experimental results showed that the NASNet architecture is very powerful compared to the other five models on small or medium challenge datasets. Its performance on challenges datasets with larger sizes is significantly increased when using combinations with other models.
Overall, our approach represents a significant advancement in the classification of medical image modalities and anatomical organs via the use of deep transfer learning. These results open new perspectives for the automation and improvement of medical image analysis tools, thus contributing to the improvement of healthcare and medical decision-making.
In future work, we plan to test stacking on higher challenge datasets by combining CNN models with ViT model.