The diagnosis of tuberculosis is challenging. We present our approach for classifying whether a patient has high or low severity tuberculosis and for detecting which lung is involved, if there is decreased capacity, and if there are pleurisies, calcifications or cavities present. Our best results for the CT report task were obtained by converting volume images into an 8x4 montage of sagittal or coronal images and ensembling the results of separate networks trained separately on sagittal and coronal montage images. The best results for the severity scoring were obtained by ensembling the results from the CT report with the provided metadata.
Tuberculosis is a common disease where fast diagnosis using CT images can often
improve treatment results. An accurate and automatic method for classifying tuberculosis
from CT images may be especially useful in regions of the world with few radiologists.
The
Data.
The data set provided for both the CT report subtask and severity scoring subtask of the
For the CT report task, the training set distribution of pathology was somewhat unbalanced with lung involvement being very common, and calcifications and pleurisy rare.
Caverns
Pleurisy
Calcification Lung Capacity Decreased
Right Lung Affected Left Lung Affected 0 50 100 150 200 250 For the severity scoring task, the training set distribution of high and low severity was balanced. See Figure 2
Present
Absent
Score 0 20 40 60 80 100
120 LOW
HIGH
Reviewing the metadata shows that some factors are a strong predictor of high severity score. See Table 1
The images for the ImageCLEF tuberculosis task were provided as NIfTI 3D datasets. We used two different approaches for preprocessing images. For the first run (SVT_5, CTR_3) we used a method similar to what we employed for the ImageCLEF 2018 challenge [4]. We converted the images using med2image, a Python3 utility that converts medical image formatted files to more visual friendly ones, such as png and jpg. After reconstructing them in all three planes, we decided to use the coronal plane images, since they had the most images containing areas of abnormal lung. Although we did not visually verify the images of this data set, tuberculosis usually involves the upper lobes with relatively unaffected lung bases. As a result, axial images through the lung bases could possibly be normal even in patients with severe disease in the upper lobes. As med2image did not take in consideration slice thickness, the reconstructed coronal images were deformed and of different height. To correct this problem, all images were resized to a 512 x 512 matrix. Image masks for the lungs were available[3], and were used to select the 200 images with the largest area of lung in the image. For the first run, all image equalization and data augmentation was done at the time of training using the fastai library [5].
For further runs (SVR_1, SVR_2, SVR_3, CTR_1, CTR_2 we used a different approach. We use nibabel library [6] to convert the NIfTI 3D datasets into numpy 3D arrays, using the provided lung masks [3], we cropped the 3D arrays to the smallest parallelogram that includes mostly the lungs. We equalized the array. We reshaped the array to have 31-32 slices in either the sagittal or coronal plane with a 256x256 matrix. Using montage, we combined the images into a single image. We did not correct for difference in slice thickness. See Figure 1 and 2. Data augmentation was done at the time of the training using the fastai library.
For training the neural network, we used a workstation with an AMD Ryzen Threadripper 1950X CPU with 16 CPU cores and 32 threads, a Nvidia Quadro P6000 GPU, 64 GB RAM, and a 1 TB solid state drive. We took advantage of the fastai library to perform transfer learning of convolutional neural networks. We tried the following architectures that were available in the fastai library: resnet18, resnet34, resnet50, resnet101, resnet152, squeezenet1_0, squeezenet1_1, densenet121, densenet161, densenet169, densenet201, vgg16_bn, vgg19_bn, and alexnet. Resnet50, resnet101, densenet121, densenet161, and densenet169 gave the best results, so we decided to ensemble them.
For training the CNN, image sizes of 224x224, 299x299, and 384x384 were utilized. The learning rate was determined after running the learning rate finder function and plotting the learning rate vs. loss. Orange [7] was used to create a prediction based on metadata only (SVR_4), and to combine metadata results with neural network results (SVR_1, SVR_2). See Figure 5. For the CTR_3 submission, for each patient we took the 200 images with the largest lung surface, scored each of those images separately using all pre-trained CNNs available in the fastai library, and averaged those results. Both mean AUC and minimum AUC were low, probably because only a few images of each patient have pathology, and averaging results decreased the probability of positive results.
For the CTR_1 and CTR_2 submissions we created a 4x8 montage of sagittal or coronal images for each patient. W separately scored sagittal and coronal images with 6 neural networks. For the CTR_2 submission, we ensembled all results, and for the CTR_1 submission, we ensembled the 3 best results.
Run CTR_Cor_32_montage.txt CTR_ReportsubmissionEnsemble2.csv TB_ReportsubmissionLimited1.csv
For the SVR_5 submission, we once again took the 200 images with the largest lung surface of each patient. For each patient, we scored each of those 200 images separately using all pretrained neural networks available in the fastai library and averaged those results. Both AUC metrics were low, for similar reasons to the CT Report Task.
Fig. 6. ROC curves of different models trained using only the metadata of the training set, based on 10-fold cross validation, calculated with Orange3 workflow from Figure 3 For the SVR_4 submission, we trained different machine learning models available in Orange3 (Constant, AdaBoost, Tree, CN2 rule inducer, Random Forest, SVM, kNN, Logistic Regression, Neural Network, Naive Bayes) and based on validation results we selected the top 4 to ensemble for the submission. See Figure 6.
For SVR_3 we took the results of classifying 4x8 montages of sagittal or coronal images as high or low severity, and ensembled them. For each 4x8 montage, we scored each sagittal and coronal image separately by ensembling the results of 6 neural networks.
For SVR_2 we ensembled SVR_3 with the metadata.
For SVR_1 we used Orange3 to create a model from the metadata using (Comorbidity, Disability, Symptoms of TB, Relapse, Drug Resistance, Bacillary, Higher Education, Alcoholic, Ex-Prisoner, Smoking) and training data (Left Lung Affected, Right Lung Affected, Lung Capacity Decrease, Calcification, Cavity, Pleurisy) and for the prediction we used the test metadata and the results from CTR_1 (Left Lung Affected, Right Lung Affected, Lung Capacity Decrease, Calcification, Cavity, Pleurisy). Although we tried Constant, AdaBoost, Tree, CN2 rule inducer, Random Forest, SVM, kNN, Logistic Regression, Neural Network, and Naive Bayes models, after evaluating the validation results, we used only SVM, Logistic Regression, Neural Network and Naive Bayes models to ensemble for the final submission. 4
In this paper, we presented the use of transfer learning to quickly train a CNN to classify the severity of tuberculosis and different pathological manifestations of tuberculosis. 5
The training data set for the CT Report was imbalanced with only a few cases of calcification or pleurisy, but we did not try to compensate for this imbalance. Trying to compensate for this imbalance may improve results. We trained the neural network as a multilabel task on the same set of equalized images. Using images with different windows to enhance calcifications, training neural networks to detect just calcifications or just cavities, and using windows set to visually enhance air within the lungs, may improve results. Using Hounsfield units from the original images instead of values in the png files may also be more accurate. As our best results for the Severity Task came from combining the results of the CT Report Task with the metadata, improving results of the CT Report should improve results for the Severity Task too. 1.