This paper presents a possible approach for the automated analysis of 3D Computed Tomography (CT) images based on the usage of feature vectors extracted by a deep convolutional 3D autoencoder network. Conventional classi cation models were used on top of the "autoencoded" feature vectors as well as vectors of meta-information paired with the images. The proposed CT image analysis approach was used by participant UIIP (Siarhei Kazlouski) for accomplishing the two subtasks of the ImageCLEF Tuberculosis task of the ImageCLEF 2019 international competition. Employing the proposed approach allowed to achieve the 2nd best performance on the TB Severity Scoring subtask and the 6th best performance in the TB CT Report subtask.
Automated analysis of 3D CT images is an example of a task that can be solved during the development of computer assisted diagnosis systems which may be used for lung disease screening for the early detection of pathology. While promising results have been shown in automated analysis of medical images of some modalities [1, 4, 7{9], the task of CT image analysis remains challenging due to the complexity and scarcity of data. A CT image is 3D data which can often be represented as a set of 2D slices with the inter-slice distance varying between 0.5 and 5 mm. Variability in the sizes and shapes of CT image voxels implies di culties in the application of many image analysis algorithms, while low availability of CT imaging data makes it di cult to use data-greedy approaches, for example, deep learning.
Despite the lack of data available, the approach for the analysis of 3D CT images proposed with this study employs the idea of trying to get 3D image descriptors by utilizing a 3D autoencoder network [6]. The motivation for this idea is the potential for maximum information usage as soon as the network is able to work with the entire 3D image. Extracted features were further analyzed using conventional classi cation models which were ensembled with models trained on image metadata. One can note, that pure 3D classi cation networks could be used instead of conventional models on top of autoencoded features. The advantage of the used approach is its generality: once we got autoencoded descriptors, a conventional classi cation model could be easily and quickly trained for arbitraty labelling (either TB severity, or one provided in CT report ndings, or some arbitrary ndings classes which are not mentioned in the competition), as well as research interest. 2
The Tuberculosis task [2] of ImageCLEF 2019 Challenge [5] included two subtasks: TB Severety (SVR) and CT Report (CTR), both dealing with 3D CT images. The same CT imaging data was used in both subtasks and included 218 images in the training dataset and 117 in the test dataset. Along with the CT images, the lungs masks [3] and additional information about the patients was provided. The metadata included information about the presence of disability, relapse, presence of TB symptoms, co-morbidity, bacillarity, drug resistance status, patient's education, being an ex-prisoner, smoking status and alcohol addiction. The frequencies of occurrence of each metadata label are listed in Table 1.
The subtask #1 (SVR subtask) was dedicated to the problem of categorizing TB cases into one of two classes: high severity and low severity. The task was to predict TB Severity class ("HIGH"/"LOW")
The subtask #2 (CTR subtask) was dedicated to the automated generation of CT reports which indicate the presence of several types of abnormalities in the lungs. Such automated annotation of CT scans is important for the development of the dedicated image databases. The task was to predict the presence of six types of ndings in CT scans. Information about the corresponding labels is listed in Table 2. Since the key idea of the approach proposed is based on using an autoencoder network, the extremely small amount of data samples we have becomes the main challenge, especially keeping in mind the sample dimensionality. The second issue is the sample dimensionality itself, which restricts possible model architectures due to GPU memory limitations.
In order to overcome the mentioned issues the following concepts were used: a) image size was reasonably decreased, b) the autoencoder was trained not on per-CT, but on per-lung basis which simultaneously doubled the training dataset size and decreased the sample dimensionality two times. Data augmentation was also used. Detailed steps of data preprocessing during the training stage were as follows:
1) Each CT image was split into two parts, each containing one lung. The split was performed roughly, into equal parts by splitting on the middle Y coordinate. The part containing the left lung was used as it is, and the part containing the right lung was re ected along Y axis in order to make the right lung oriented similar to left one. Lung images were normalized to 0-1 scale. The provided lungs masks were treated the same way (except for normalization).
2) For each lung a random transformation was generated and applied to the lung and its mask. The mask was binarized after the transformation and applied to the lung image (all voxels outside of the masked lung area were set to zero). The resulting image was resized to 128 x 128 x 128 pixels and normalized once again. Transformations included 3D shift, rotation, scale, crop and shear, and were applied sequentially with a probability of 50% for each transform. The ranges of parameters used for the transformation are presented in Table 3.
The training dataset provided by the organizers was split into training and validation subsets for models development.
Two splits were used, one for autoencoder model training, and another one for classi cation models training. For autoencoder training, a randomly sampled 90% of training data was used for training and the remaining 10% was used for validation. For classi cation models the xed random 5-fold cross-validation split was used. Smaller validation size in the autoencoder case is motivated by maximizing training set size, while validation loss for the autoencoder model is less important. For classi cation tasks scoring is crucial, so cross-validation was used. 3.3
A custom convolutional architecture was used for the autoencoder model. Several
trials with di erent hyperparmeters (number of layers, kernel size, lter number)
were exectuted and the model with the best achieved validation loss was selected.
The selected architecture is presented in Table 4. Each convolutional layer had a
kernel size of (
The autoencoder model was trained using Adam optimizer and was performed in three stages. On the rst stage the model was trained on the mixture of left and right lung images using data augmentation as described before. The training was performed until any signi cant improvements in validation loss were observed. Speci cally, model trained for 72 epochs was selected on this stage. On the second stage the retrieved pretrained model was netuned with a 10 times smaller learning rate separately for the left and right lungs using data augmentation, resulting in two di erent models, one for the left and one for the right lungs. Training stop criteria was the same, resulting in 20 epochs of training. So, on the rst two stages 92 randomly augmented versions of each original CT were used for model training. Finally, learning rate was decreased 10 times again and each of the two models were netuned for 2 epochs on competition data without data augmentation.
On the inference stage both autoencoder models were used to generate feature vectors for the left and right lung of each CT image, so the resulting feature vectors of CT were a concatenation of the left and right lung encoded descriptors, with 512 components in total.
Analysis of encoded vectors showed that around half of the components of the vector are very close to zero for all images, which probably re ects the nonoptimal architecture and/or weights of model. As soon as no better model could be retrieved in experiments, it was decided just to drop the "zero" components of the encoded vector, which resulted in the nal version of the encoded CT images descriptors containing 220 components each. 3.4
Since all of the predictions in both subtasks are binary classi cation problems, they were treated and solved in the same way. The idea may be formulated as follows: for each of the requested binary class labels (which are: SeverityHigh, LeftLungA ected, RightLungA ected, LungCapacityDecrease, Calci cation, Pleurisy, Caverns) build a binary classi cation model which will take encoded image descriptors and available meta information about patients as input features.
Because of the di erent nature of encoded image descriptors and image meta information it was decided to train separate classi cation models for each feature type and then ensemble the models' output rather than concatenating feature vectors.
Conventional classi cation models were used for working with both types of features and included scikit-learn package implementation of SVM, K-neighbors classi er (kNN), random forest classi er (RF) and AdaBoost classi er. Hyper parameters for each of the models were tuned by cross-validation using random parameter search.
In case of meta information features were used "as is", while for autoencoded image features their PCA-transformed presentation with 3,5,10,50 components were used as well.
The resulting algorithm can be described as follows.
1) For each of 5 folds, take the autoencoded features and t PCA using the training set and transform validation set using 3, 5, 10, 50 components. Thus six alternative feature vectors were presented for each image in each crossvalidation split: meta information (META), encoded features (AEC), and PCAtransformed encoded features with 3, 5, 10, 50 components (PCA3, PCA5, PCA10, PCA50).
2) Sample random hyper parameters for each of 4 classi er types. Parameter ranges are presented in Table 5 (only valid parameters combinations were used).
For each of the target labels: 3) Get the mean AUC-ROC score at cross-validation for all models and sampled hyper parameters.
4) Select the best models according to the achieved scores. Models selection was performed not just by the top score value, but using the following heurisctic: 1) classi er-feature type combination were used only once, 2) in the case that scores are reasonably close, more simple models have priority (examples: a) if the kNN model with 11 neighbors scores 0.80 and with 2 neighbors scores 0.78, the second one is used; b) if the same model scores 0.8 on the 50 component PCA features and 0.77 on the 3-component, the second one is used).
The application of the described algorithm resulted in the selection of from 1 to 4 best models for each target class prediction, which were ensembled during nal prediction by the simple averaging of class probabilities. A summary on the selected models and their validation performance is presented in Table 6. 4
As the result of this study, the described method was applied to generate predictions for the competition test dataset. Predictions were submitted by participant UIIP for the CTR and SVR subtasks. The full list of the submitted results for both subtasks is available at the task web page1. 1 https://www.imageclef.org/2019/medical/tuberculosis/
Label LeftLungA ected RightLungA ected Calci cation Caverns Pleurisy LungCapacityDecr.
Severity *estimators number **max depth
Before generating the nal submission, the autoencoder model and selected classi ers used for predicting the test data were trained on the whole available training dataset. Final probabilities of each target class were calculated as the average probability of the selected classi cation models.
Table 7 shows the best results achieved by the participants in the CTR subtask. The run submitted by UIIP achieved the 6th best mean AUC, while also demostrating the worst minimum AUC for the prediction of the presence of lung abnormalities. The average mean result on the contrary to the worst minimum AUC demonstrate, that selected models might work pretty well for some of the target labels in the report, while failing for other labels. Since all targets demonstrate similar performances on the validation set, test results may be caused by over tting to validation and the di erent distribution of train and test sets in the competition, or by some errors in validation set generation (in particular, the uneven distribution of meta information and targets classes itself was not carefully treated in experiments). { Despite data scarcity, deep autoencoder networks may be used for extracting reasonable descriptors of 3D CT data if some tricks for training set extension are used. { Meta information about patients are helpful for the more accurate predictions of TB characteristics. { Although the used approach demonstrated good performance in the SVR subtask, it was not very reliable for the generation of the CT report, which means the suggested method is not very stable or at least needs more careful validation.
This study was partly supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-18-64818-1 "Year 7: Belarus TB Database and TB Portals".