This paper presents an approach for automated analysis of 3D Computed Tomography (CT) images based on representing the 3D CT data as a set of 2D projection images along all three axes. Such approach reduces the dimensionality of the input data and therefore allows using less complicated models for image classi cation tasks. Deep Learning methods were used to predict most of the features of CT images of patients with lung tuberculosis (TB). For part of the features, conventional methods were used. Two di erent methods of segmentation of lungs were employed including the registration-based scheme. The proposed image analysis approach was utilized by United Institute of Informatics Problems (UIIP BioMed) participant for accomplishing the two subtasks of ImageCLEF Tuberculosis task of ImageCLEF 2019 international competition. Employing the proposed approach allowed achieving the best performance in both CT Report Generation and TB Severity Scoring subtasks. Source codes implementing the proposed methods are available on Github1.
Automated analysis of 3D CT images is an important step in many tasks
connected with the development of Computer-Aided Diagnosis systems, screening
of lung diseases, early detection of pathology and development of the dedicated
web-portals2 [
The approach for analysis of 3D CT images proposed with this study employs the idea of representing each 3D CT scan as a set of 2D projection images along all three (X, Y and Z) axes: sagittal, frontal and axial projections. The advantage of such an approach over the conventional slice-wise representation consists in the fact that each projection obtained via averaging of voxel intensity values contains information about all the slices present in the image. On the one hand, such approach signi cantly reduces the complexity of the input data which eases the task of training the Convolution Neural Networks on a limited amount of training data. On the other hand, generating projections along di erent axes (X, Y , Z) provides additional "native" augmentation of the input data. 2
The tuberculosis task [
The subtask #1 (SVR subtask) was dedicated to the problem of categorizing TB cases into one of the two classes: high severity and low severity. In contrast to the previous year's challenge, the task was only to predict TB Severity class ("HIGH"/"LOW") rather than to provide a Severity score from 1 to 5. Also, along with the CT images of TB patients an additional information about the patients was provided. The metadata included information about presence of disability, relapse, presence of TB symptoms, co-morbidity, bacillarity, drug resistance status, patient's education, being ex-prisoner, smoking status and alcohol addiction. The task was aimed at automatic classi cation of TB Severity into "HIGH" and "LOW" classes. The frequencies of occurrence of each metadata label are listed in Table 1.
The subtask #2 (CTR subtask) was the newly-introduces subtask which was dedicated to automated generation of CT reports which indicated presence of several types of abnormalities in lungs. Such automated annotation of CT scans is important for development of the dedicated image databases. The task was to predict the presence of six types of ndings in CT scans. Information about the corresponding labels is listed in Table 2. The key idea of the approach proposed with this study consists in converting 3D CT scans into 2D projections followed by analysis of the obtained 2D projection images with use of Deep Learning and conventional methods.
For generation of the projections, two di erent versions of automatically
extracted lung masks were used: the lung masks provided by the competition
organizers [
The process of generation of CT projection images considered performing the following major procedures. The CT image voxel intensity values were increased by the value of 1024 Houns eld Units (HU) to ensure only positive intensity values. To exclude the in uence of image segmentation faults on the lung borders, the lung masks were eroded using ellipsoidal structure elements of radius 10 along XY plane. The radius along Z-axis was calculated respectively to the inter-slice distance of the speci c CT image. Image intensity values outside of the eroded lung masks were zeroed. Optionally, an intensity threshold can be applied to lter noisy voxels with low intensities.
Each 2D projection image was represented as a pseudo-RGB image and had three channels. The rst (red) channel contained mean values of CT image intensities along the speci ed axis. The mean values were nally divided on their maximum value along the projection image. The second (green) channel contained maximum intensity values along the speci ed axis divided by the value of 1500 which corresponds to 1500 1024 = 476 HU in terms of the original voxel intensities. The third (blue) channel was composed of the corresponding Standard Deviation intensity values. As in the case of red channel, blue channel values were divided by their maximum. Finally, the resultant 2D projection images were cropped using the bounding boxes of non-zero regions and resized to 256 256 pixels size. Projections were generated for each lung separately. The general scheme of generation of 2D projections from 3D CT images is shown in Fig. 2
Using this scheme, six 2D projection images were generated for each CT scan in the dataset: sagittal, frontal and axial projections for each of the two lungs. Three versions of projection images were generated with use of di erent lung masks and intensity thresholds. Details on the versions are shown in Table 3. For correct validation of the developed image analysis algorithms, the Development dataset provided by the organizers was split into training and validation subsets. With this study, each 4-th CT image from the Development dataset was labeled as validation image (54 cases), whereas all other CT cases were used for training the algorithms (164 cases). This training/validation data split was used at every stage for both subtasks without changes. 3.3
Convolutional Neural Networks (CNNs) were used to predict most of CT
image characteristics. A deeply modi ed version of a popular VGG16 [
Conventional VGG16 architecture includes 13 convolutional layers in ve blocks and three fully-connected layers (13 + 3 = 16). Considering the limited amount of the available data, the employed neural network model was simpli ed by (i) reducing the number of convolutional layers to one per block (5 layers), (ii) using global maximum pooling after the last convolutional block instead of attening, and (iii) reducing the number of nodes in the fully-connected layers to 128. All this signi cantly reduces the number of trained parameters of the network model which minimizes the e ect of over- tting. The resultant network model included eight trainable layers: 5 convolutional and 3 fully-connected. 3.4
Training CNN for lung-wise detection of abnormalities. Most of the metadata labels provided with this challenge (TB Severity class, presence of certain lesions) were assigned at the level of CT scans. On the other hand, LeftLungA ected and RightLungA ected labels from the CTR subtask metadata can be used to train image classi cation models to detect presence of abnormalities at single lung level.
At this step, a Convolutional Neural Network was trained to classify a 2D lung projection image as either healthy ("normal") or a ected by TB ("abnormal"). The training and validation data was composed of lung projection images along X, Y and Z axes, left and right lungs were considered separately. Thus, the total number of input samples was 164 2 (lungs) 3 (projections) = 984 for training and 54 2 3 = 324 for validation. The size of the network input was 256 256 3 which corresponded to a projection image of a single lung. Categorical cross-entropy was used as the cost function. Additional on-the- y data augmentation was applied which included random rotations with 10 degree range, width and height shifts and random re-scaling.
The network training for lung-wise "normal"/"abnormal" classi cation was performed with use of lung projections v1.0, Adam optimizer and learning rate set to 10 5 within 120 training epochs. Network weights were initialized randomly. Here and further the training was performed with use of Keras framework with Tensor ow backend on a personal computer equipped with a GPU of Nvidia TITAN X type with 3072 CUDA Cores and 12 GB of GDDR5 on-board memory.
The trained CNN was evaluated on the lung projection images generated from the validation CT scans to produce the con dence scores. Projection-wise comparison to the ground truth data resulted in 0.865 Area Under ROC-Curve (AUC). However, 2D projections generated along di erent axes from the same 3D image must have the same label ("normal" or "abnormal"). Therefore, the con dence scores obtained for the three projections of each lung were combined to increase the lung-wise classi cation quality. The best performance of lung-wise "normal"/"abnormal" classi cation was achieved by means of using maximum among the three con dence values which corresponded to X, Y and Z axes (AUC = 0.920).
Detection of caverns and lung capacity decrease. Similar approach was used for detection of caverns and lung capacity decrease. In this case, the corresponding labels were speci ed at CT image level, therefore the input data consisted of projections of both lungs. The corresponding neural network model had the input of size 256 512 3 which corresponded to two concatenated projection images. Lung projections v1.1 were used.
Weights of the convolutional layers were initialized with the corresponding weights of the CNN previously trained for lung-wise "normal"/"abnormal" classi cation. The networks for detection of lung capacity decrease and caverns were trained for 50 and 120 epochs respectively. Evaluation of the trained CNNs on the validation data resulted in 0.832 AUC for lung capacity decrease and 0.809 AUC for caverns detection at projection level. Averaging the con dence values obtained for X, Y and Z projections gave the CT-wise detection performance of 0.856 AUC for lung capacity decrease and 0.879 AUC for caverns. Detection of calci cation and pleurisy. Scores for prediction of calci cation were calculated using the mean intensity values of v1.2 projection images. These values re ect the number of voxels in lung regions of the original CT images with intensities exceeding 1000 HU threshold. Evaluation of such scores on the Development dataset resulted in 0.726 AUC value.
Since the lung masks provided by the organizers tend to exclude pleurisy from lung segmentation (see Fig.1), the scores for detection of pleurisy were calculated as di erence between the volume of lung masks obtained via registration-based scheme and the volume of lung masks provided by the organizers. Evaluation of such scores gave 0.776 AUC value on the Development dataset.
Summary on the CTR subtask. The results of evaluation of the prediction algorithms for the CTR subtask are shown in Table 4. All the AUC values were assessed on the validation dataset. The approach used with this study for prediction of TB severity consisted of two major stages.
The rst stage considered training a CNN for prediction of severity class with use of only CT projection images. At this stage, the VGG16 network was trained to classify TB severity into "HIGH" and "LOW" classes. Similarly to the detection of caverns and lung capacity decrease in the CTR subtask, input for the CNN for severity classi cation was composed of projections of both left and right lungs. Weights of all trainable layers (convolutional and fullyconnected) of the network were initialized with the corresponding weights of the CNN previously trained for lung-wise "normal"/"abnormal" classi cation. The network was trained on v1.0 projection images during 60 epochs. Evaluation of the trained network on the validation set resulted in 0.768 AUC value.
At the second stage, the output of a trained CNN (con dence score) was combined with the available metadata features for prediction of TB severity with use of Linear Regression classi er. Several more classi ers were tested including Random Forests, Logistic Regression and Support Vector Machine, but Linear Regression gave the best AUC on the validation subset. Two sets of the metadata features were used: all available metadata features and the subset of four selected features which gave the same AUC on the validation subset. The subset of four features included DrugResistance, HigherEducation, ExPrisoner and Alcoholic labels.
AUC values assessed on the validation data using di erent combinations of metadata features and CNN outputs obtained from di erent projection images are shown in Table 5. The testing results presented with the table suggest that 2D projection images generated along X axis were the most informative for prediction of TB Severity class by the neural network. Utilizing projections along Z axis did not improve the results compared to the prediction using only X and Y projections.
Among the combinations which used all available metadata features, the one which used CNN outputs from X projections showed the best performance with 0.844 AUC on the validation subset. The overall best performance on the validation data was achieved using the subset of four metadata features along with CNN outputs from X and Y projections. These two combinations were used in the two submitted runs for the SVR subtask. 4
As the result of this study, three runs were submitted by UIIP BioMed for the CTR subtask and two for the SVR subtask. Full list of the submitted results for both subtasks is available at the task web page4. Network models and classi ers
used for predicting the Test data were trained on the training subset of Development data (see subsection 3.2). All the computed prediction scores were scaled to the range from 0 to 1 to t the submission requirements.
Subsection 3.4 describes the methods used for the last ("CTR run3 ...") submission for the CTR subtask. The previous submissions di ered with the ways of predicting LungCapacityDecrease, Pleurisy and Caverns and demonstrated poorer performance on the validation data. In "CTR run1 ...", these three ndings were predicted by a single neural network trained for multi-class multi-label classi cation (0.844 mean AUC on validation). In "CTR run2 ...", LungCapacityDecrease and Caverns labels were predicted using separate CNNs trained for binary classi cation which gave better results (0.852 mean AUC on validation). Table 6 shows results achieved by the participants in the CTR subtask, one best run for each participating group. The run submitted by UIIP BioMed achieved the highest values of both mean and minimum AUC for prediction of presence of lung abnormalities.
The methods used for submissions in the SVR subtask are described in subsection 3.5. The rst submitted run ("SRV run1 ...") used all metadata features and X projections for TB Severity classi cation and achieved the overall best performance in both AUC (0.7877) and Accuracy (0.7179). The second submission "SRV run2 ..." used the subset of four metadata features along with X and Y projections. This approach demonstrated better performance on the validation data. However, on the Test data it resulted in lower AUC (0.7636 vs. 0.7877) but higher Accuracy (0.7350 vs. 0.7179). Results of the participants' submissions with the highest AUC values are shown in Table 7. 5
The results of this study allow to draw the following conclusions: { 2D projections of lungs generated from 3D CT scans preserve basic information about the lungs structure which is su cient for detection of abnormalities of di erent sort. { TB Severity can be assessed based on the analysis of CT scans, however additional information on clinical data, drug resistance and patient's social status are helpful for more accurate scoring of TB Severity. { Usefulness of di erent methods of lungs segmentation may vary depending of the speci c task being solved. Combining multiple lung segmentation methods may provide additional information which can be useful for detection of certain abnormalities in lungs.
This study was partly supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-18-64818-1 "Year 7: Belarus TB Database and TB Portals".