Pneumonia is a life-threatening disease that occurs in the lungs and is caused by a bacterial or viral infection. However, it is very difficult to diagnose it by simply looking at chest x-rays, because it is necessary to improve diagnostic accuracy. This study aims to simplify the process of detecting and classifying pneumonia for both experts and patients, using a dataset containing 5247 CXR images. Five different pretrained CNNs: AlexNet, VGG-16, ResNet50, DenseNet-121 and InceptionV3 were used separately or together for transfer learning in a progressive way. Firstly, they are pretrained on ImageNet dataset, and secondly, on a radiographic images which concerns another disease (available in medium size with a nature close to our base). These models are refined according to different fine-tuning levels and strategies. A weighted classifier-based approach is introduced to combine their weighted prediction. The results obtained show the possibility of moving easily from the classification of a disease to another using a progressive transfer learning, which has a limited number of images by taking advantage of the knowledge already acquired on another very large base.
Medical imaging has a key role in the detection and classification of diseases. Although chest xrays (CXR) have lower resolution than magnetic resonance imaging (MRI) or computed tomography (CT), and they adopt low-cost and easy to-use technology. Pneumonia is an acute respiratory infection that affects the lungs. This disease is considered the leading cause of infant mortality worldwide. About 1.4 million children die of pneumonia every year, or 18% of the total number of children who die under the age of five. There are mainly two types of pneumonia: viral and bacterial. Generally, viral pneumonia tends to be mild while bacterial pneumonia is more severe, especially in children. Inflammations seen as white patches on chest x-ray are due to various abnormalities such as pneumonia, tuberculosis, pneumothorax, pleural effusion, etc. hence the ambiguity in the diagnosis of pneumonia from chest X-ray images, either by radiologists or by computer-aided diagnosis (CAD) systems. This article studies the application of deep transfer learning in medical imaging to help diagnose pneumonia, it is organized as follows: Section 2 expose some related works; Section 3 explains the proposed method; section 4 details the experiments and results performed on the system, and Section 5 presents the discussion and conclusions.
One of the primary use cases for machine learning in healthcare is in the early detection and effective diagnosis of disease. Certain diseases such as hereditary or genetic disorders and certain types of cancer are difficult to identify at an early stage by a specialist. However, the use of intelligent computing solutions through machine learning can well detect these. These solutions have continued to evolve using innovative learning algorithms on datasets representing different diseases with varying complexity. In general, these systems consist of two main parts: a discriminant feature extractor (which must be chosen by computer vision and health experts, or dedicated algorithms) and a classifier (based on machine learning using a representative image base).
The authors of [
The detection of pneumonia by chest X-ray has been an open problem for many years, the main
limitation being the scarcity of publicly available data. Yao and al. [
Deep learning has quickly established itself as a standard in several fields, often with better
performance compared to conventional machine learning algorithms (presented previously). Recently,
a number of researchers have come up with different artificial intelligence (AI) based solutions for
different medical problems. Convolutional Neural Networks (CNNs) have enabled researchers to
achieve positive results in broad medical problems such as detection of thyroid nodules, breast cancer,
detection and segmentation of brain tumors, classification of diseases in x-ray images, etc. The
authors of [
In the article [18], the authors carried out a comparative study of different CNN networks in the diagnosis of pulmonary diseases. They collected a sample of 357 images of the different diseases, and a healthy sample (100 images), and having analyzed 38 non-image related features, including complaints of cough, weakness, chest pain and high body temperature. The neural network with the Levenberg Marquardt algorithm achieved the highest accuracy for the detection of pneumonia (91.67%). In 2018, some researchers highlighted a visualization technique in conjunction with CNNs to locate and detect Regions Of Interest (ROIs) that can be used to identify pneumonia and distinguish between bacterial and viral types in pediatrics. In this sense, we find the work of the authors of [19] who evaluated the performance of different CNN architectures: Sequential CNN, Inception CNN, Residual CNN and VGG16 on a dataset of 5232 pediatric chest radiographs. Their approach was based on a visualization technique to define the ROI. The customized VGG16 achieved 96.2% accuracy in pneumonia detection and 91.8% in its classification. Xianghong and al. [20] used a 08layer FCN model with transfer learning to segment anatomical lung regions. The model was trained and tested on both JSRT (241 images) and MC (138 images) datasets. After segmentation, an AlexNet DCNN model was used to classify the lung regions. A binary classification [21] was performed using the SVM-RBF kernel by combining DCNN and manual functionalities. Features extraction by DCNN with transfer learning achieved accuracy (0.8048 ± 0.0202) and better sensitivity (0.7755 ± 0.0296). Rajpurka and al. [22] proposed the Chexnet algorithm for pneumonia detection and tested its accuracy against 4 radiologists, using an F1-score metric. The algorithm worked as the same as an experienced radiologist; however, it exceeded the performance of an average radiologist. Jaiswal and al. [23] proposed an identification model inspired by the Mask-RCNN model that included critical modifications to the training process and a new post-processing step that merges the bounding boxes of multiple models. Ayan and al. [24] used two CNN network models: Xception and VGG16 for the diagnosis of pneumonia on the Kermany dataset (5856 images). Relying on transfer learning and finetuning on the Xception model, hence the use of pretrained ImageNet weights before the start of training and the last 10 first layers were frozen. The test results showed that the Vgg16 network outperforms the Xception network. Varshani and al. [25] used ResNet50, DenseNet-121 and DenseNet169 as the optimal CNN models for the features extraction step and different classifiers such as RF, SVM, etc. The best results favored the SVM classifier. In 2020, Vikash and al. [26], used AlexNet, DenseNet121, Inception V3, GoogLeNet and Res-Net18 pe-trained on the ImageNet basis. Then they came up with an ensemble model that combines the outputs of all these models into a prediction vector, and a majority vote is used to choose the final prediction. This combination achieved a test accuracy of 96.39%, with an area under the ROC curve of 99.34% and a sensitivity of 99.62%. Rahman and al. [27] attempted to automatically diagnose different classes of pneumonia (bacterial and viral), on the Kaggle Chest X-Ray pneumonia dataset, it includes 5247 X-ray images.
Data training is done using the pretrained algorithms: AlexNet, ResNet18, DenseNet201 and SqueezeNet. They observed that DenseNet201 outperforms the other three CNNs, achieving 98% accuracy in pneumonia detection and accuracy of 93.3% in the differentiation between the two etiological variants. Hashmi and al. [28], presented a method combining Five CNNs: ResNet18, Dense-Net121, InceptionV3, Xception and MobileNetV2 for the automatic pneumonia detection, using data augmentation, after that the five CNNs were refined to pneumonia classification. Then the predictions were combined, using a weighted classifier to calculate the final prediction. Their model achieved an accuracy of 98.857%, and furthermore, a high F1-score of 99.002 and an AUC score of 99.809.
In 2021, the authors of [29], proposed a novel end-to-end Deep Transfer Learning framework using deep convolutional neural network that detects and classifies three types of pneumonia from chest X-ray scans. In the same year, Alqudah and al. [30] constructed two hybrid models, namely CNN-KNN and CNN-SVM, while using a 10-fold cross-validation methodology. The proposed CNN has been trained, validated and tested using a large chest X-ray image dataset of 5852 images. The hybrid CNN-KNN model achieved an accuracy of 94.03%, while the CNN-SVM model achieved an accuracy of 93.9%. In the work of [31], we meet a deep learning-based approach to diagnose and classify pneumonia from Chest X-ray images using transfer learning based on three pre-trained architectures (ResNet50, InceptionV3 and InceptionResNetV2).
Polat and al. [32] proposed a classification approach based on extracting relevant features from digital chest radiographs. They used a binary CNN and a three-class CNN to detect pneumonia in 5840 pediatric CXR images. They also used a minimum distance classifier for classification. They conducted three different parametric studies and found that the proposed method achieved 100% accuracy in detecting pneumonia, 92% in distinguishing between two types of pneumonia, and 90% in distinguishing normal, bacterial, or viral pneumonia.
CNNs perform better on large datasets; however, our target dataset is very small. This requires the use of data augmentation step to enrich the small base and the transfer learning strategy to take advantage of the consistency of a pretrained network on a largest dataset. This strategy can improve significantly the classification rates.
The proposed approach is based on transfer learning of five CNN architectures (AlexNet, ResNet50V2, DenseNet-121, VGG-16 and InceptionV3) for pneumonia detection from chest x-rays. To solve the dataset size issue, we also propose a new strategy of transfer learning, called progressive transfer learning, that adopts a pretrained CNN with a large amount of acceptable medical images (See Figure 1, 2). In order to evaluate the performance of the proposed approach, three transfer learning approaches have been developed, using different strategies (See Figure 1, 2, 3 and 4). First, the five models are pretrained on the ImageNet dataset, and second, are pretrained and refined on an intermediate Tuberculosis image database (Kermany dataset). Then we reused the pretrained model progressively on these two datasets to learn how to classify pneumonia on our dataset (Curated Xray dataset). The third approach consists of combining the five CNN models, to take advantage of the performance of each approach.
Approach 1: Direct transfer learning from ImageNet to Curated Xray dataset:
2. Approach 2: Progressive transfer learning, form ImageNet to Kermany dataset, then to Curated Xray dataset 3. Approach 3: The combined architecture (called AVRDIS) is based on the combination by weighting of the five CNN models (AlexNet, VGG-16, ResNet-50, DenseNet-121 and InceptionV3) using the Softmax classifier (See Figure 3).
Alexnet VGG-16
ResNet DenseNet InceptionV3
P1 P2 P3 P4 P5 w1 w2 w3 w4 w5
E Weighted classifier
After feature extraction, different supervised classifiers (SVM, RF, Gaussian Naïve Bayes : GNB and softmax) were used for the pneumonia classification task in combination with CNN part of feature extraction. The proposed method is carried out in three steps: · the first consists in extracting the features of the radiological images of pneumonia; such that the latter are passed as inputs to the five CNN models, the output is a feature vector which depends on this image. · The second step consists in passing this feature vector as input of one of the four classifiers : SVM, RF, GNB and softmax. · For the last step, the predictions of these models were combined, using a weighted classifier to calculate the final prediction.
In summary, this strategy consists in passing as input for each architecture preprocessed radiological images of pneumonia, then providing as output the following predictions: PNEUMONIA or NORMAL for the first base (binary classification); Bacterial Pneumonia, Viral Pneumonia, Normal and Covid-19 for the second base (4-class multiple classification).
Weighted average assembly is a powerful classifier fusion mechanism. However, the choice of weights to be assigned to the respective basic learners plays a central role in ensuring the success of the whole. In this work, we adopted a weighted mean ensemble technique for better classification. To find the best combination of weights that gives the maximum precision, we have implemented a method that consists of calculating the final precision for each combination of weights, and the weights with the best precision found will be returned.
A weight (wk) takes a value between 0 and 1, each model, after being refined, returned the probabilities for each class label, i.e. 2 classes in the form of a matrix (P). The weights (wk) are multiplied by the corresponding basic learners probabilities Pk to calculate the weighted average probability ensemble, as shown in Equation (1).
The convolution layers of the five CNNs can be trained according to the following three transfer learning strategies (See Figure 4).
· Strategy 1: Using a pretrained model as a feature extractor, and for that one has to freeze all the convolution blocks to keep the convolution part of CNN in its original form, and then use its outputs to feed an updated classifier, this strategy is generally used in the case of small databases where there is a lack of calculation. · Strategy 2: Building on the idea that the lower layers of a CNN refer to general features (problem independent), while the upper layers refer to specific features (problemdependent), it serves to replace the last layer Fully-Connected by the new randomly initialized classifier, and we fix the parameters of the last convolution block of the pretrained network. · Strategy 3: It has the same principle as the previous strategy, except that instead of training only one block, the last two convolution blocks are trained. The AlexNet architecture is not affected by this strategy because it contains only 5 convolution layers.
To evaluate the proposed approach, we used two different datasets: 1) Kermany dataset containing 5840 images labeled in two categories (Normal: healthy, and pneumonia: viral or bacterial). This dataset is subdivided into 90% for training task and 10% for test task. 2) Curated Xray dataset with a total of 18417 images labeled in 4 categories (Normal: healthy, Bacterial Pneumonia, viral Pneumonia, COVID19), subdivided into 91% for training task and 9% for test task.
TP, TN, FN, FP represents the number of true positive, true negative, false negative, false positive respectively.
dataset
Tables 1 and 2 summarize the results obtained by the 5 models without and with image augmentation respectively using the Kermany database, including the transfer of learning based on ImageNet dataset and refined with the three fine-tuning strategies quoted previously. Expressed in terms of accuracy, precision, recall and F1-score. And we notice that the rates obtained with image augmentation surpass the rates obtained without data augmentation. These results confirm that this methodology is relevant to apply to our problem. The parameters used in data augmentation are: (Rotation = 20°, Width shift = 0.1, Shear = 0.2, Zoom=0.2).
Accuracy Accuracy
Table 5 displays the found metric values of the five models for the three strategies, with dataaugmentation. This experiment was conducted with the aim of evaluating our 5 models on a broader basis and with more classes (4 classes including viral, bacterial, Covid and normal pneumonia). The results obtained by the models (using previous experience) with image augmentation using the Curated X-ray image dataset are summarized in Table 5. The DenseNet model outperformed all other models with an F1-score equal to 92.49%.
The tables presented above allow us to see how the right choice of the training dataset and the transfer of learning helps to improve performance. As well as the increase in data that has played a crucial role in this improvement and confirms all its performance in the medical field where available data is scarce. Table 7 shows a comparison of the two approaches (progressive TL and direct TL) using F1-score metric. These results show that progressive transfer learning (Approach 2) offers better performance compared to direct transfer learning from ImageNet (Approach 1).
Approach 3 consists of combining the 5 models. This makes it possible to assess the presence of the complementarities of different models in the pneumonia classification. The following table (Table 8 and Figure 6), shows a comparison of the results of approach 3 (called AVRDIS, See Figure 2) and the results of each of the 5 models (with its best configuration). This comparison showed the superiority of the approach based on the combination of models. This confirms the presence of complementarities between the 5 models. This combination gave the best performance and achieved a test accuracy of 96.79%, and an F1-score equal to 97.44%. Table 8 shows the results for each model as well as for the proposed method.
Table 9 gives us a comparison of the combined model (AVRDIS) with some existing methods using the metrics mentioned above. It confirms the superiority of the proposed approach. This final model is the most relevant to apply it to our problem with an accuracy of 96.76%, a sensitivity of 97.47% and an F1-score of 97.44%.
This paper deals with the problem of the classification of pneumonia on radiographic images, since they are considered the main cause of infant mortality in the world. The goal was to develop a robust and automatic approach, based on deep learning. In this context, we adopted the transfer learning approach and used a combination of five pre-trained architectures: AlexNet, VGG-16, ResNet, DenseNet121 and InceptionV3, initially trained on the ImageNet dataset which consists of 14 million natural images. Our experiments revealed that the scenario of progressive transfer learning from the tuberculosis image dataset to classify pneumonia improves the performance of the studied models. In addition, a comparative study of supervised softmax classifier with different classifiers showed that these latter offered better results thanks to the power of RF, SVM and Naive Bayes classifiers combined with CNN models. Finally, we used a model combining five pretrained CNN models. The latter outperformed all other models with a good performance in terms of F1-score (97.44%) and accuracy (96.76%). Although many methods have been developed to work on this dataset, the proposed methodology has obtained better results. We observed that performance could be further improved by increasing the size of the databases using a data augmentation approach and and by using other more advanced models, such as Vision Transformers (ViT) without or with combination with a CNN. The coupling with a metaheuristic approach (like PSO) would allow better piloting and optimization of the combination of classifiers. This will pave the way for our future works to explore this line of research.