The paper presents image description and classi cation method which was used by United Institute of Informatics Problems (UIIP BioMed) group for accomplishing the three subtasks of ImageCLEFtuberculosis task. The image description method employed is based on automated detection of tuberculosis (TB) lesions of di erent types in 3D lung Computed Tomography (CT) scans. The lesion detection method is based on Coder-Decoder Convolutional Neural Network trained on a third-party dataset of 149 CT scans with lesions labeled by a quali ed radiologist. It was shown that combination of lesion-based TB-descriptor and Random Forests classi er allows achieving the best performance in TB type classi cation and TB severity scoring subtasks.
The tuberculosis task [
The subtask #2 of ImageCLEFtuberculosis task is aimed at automatic categorization of CT images into one of ve types of tuberculosis: In ltrative, Focal, Tuberculoma, Miliary and Fibro-cavernous. Compared to 2017, the datasets were extended by adding new CT scans of the patients involved earlier, and also by introducing CT images of some new patients. However, in this study only the rst CT scan of each patient was used.
The newly represented subtask #3 was dedicated to assessment of severity of TB based on a single CT image of a patient. The severity score has meaning of a cumulative score of severity of TB case assigned by a medical doctor. Originally, the severity scores were assigned using natural numbers between 1 ("critical/very bad") and 5 ("very good"). Additionally, for the case of binary classi cation the scores were converted to binary values where scores from 1 to 3 corresponded to "high severity" and the remaining 4 and 5 corresponded to "low severity". In the process of scoring, the medical doctors considered many factors like patterns of lung lesions, results of microbiological tests, duration of treatment, patient's age and some other. One of the goals of this subtask is to distinguish "low severity" from "high severity" based solely on the CT scan. 2
In this section, a method for automated detection of lung lesions in 3D CT images
is described. The method is based on training the Deep Convolutional Neural
Network (CNN) on a set of data derived from 3D CT images with manually
labeled lesions of di erent types. The method utilizes slice-wise image
segmentation technique previously described in [
The developed software tool allows labeling of 10 di erent types of TB
lesions. Some types of lesions were well represented in the dataset whilst lesions
of some other types (Plevritis, Atelectasis, Pneumathorax) were present only in
few images in the dataset. List of lesion types and the corresponding frequencies
of occurrence in dataset images are shown in Table 1. In the result of labeling
process, 3D masks with the corresponding lesion indexes were obtained.
For extraction of lung regions for both lesion detection and
ImageCLEFtuberculosis subtasks, a domestic implementation of a conventional
segmentation-byregistration approach [
From the total amount of 198 labeled 3D scans, 149 were used for training the algorithms and the rest 49 were used for validation. Lesion types with indexes 1{5 were merged together into one class "Foci" as having similar nature and/or being mixture of classes. From the 149 training CT images, 268,278 2D image tiles were extracted. For each tile a corresponding label image was composed using manually labeled lesion data (see Fig. 5). Image regions which lay beyond the lung segmentation masks are marked with a special "don't care" label. Neural network omits these regions at both training and validation stages which allows to better focus the available computational facilities on the actual regions of interest. On the label images such regions are marked with gray color.
For segmentation of lesions in 2D slice regions a Fully Convolutional
Network Alexnet [
Training was performed on a personal computer equipped with Intel i7-6700K CPU and dedicated GPU of Nvidia TITAN X type with 3072 CUDA Cores and 12 GB of GDDR5 onboard memory. NVIDIA DIGITS interface and Ca ee framework were used. The network training parameters were set to the following values: Number of epochs=60, Activation function=ReLu, Batch size=64, Solver type=SGD Ca e solver. Learning Rate was set to 0.001 for the rst 20 epochs, 0.0001 for the next 20 and 0.00001 for the last 20 ones. 2.4
Once the training process is nished, the trained network model can be used for detection of lesions in an arbitrary 3D CT scan. In this case the CT image undergoes the same procedures as for the training images: { segmentation of lung regions; { extraction of 2D tiles; { processing the tiles with the trained CNN and obtaining probability maps for each lesion type considered; { collecting the obtained 2D probability maps into 3D probability maps for each lesion type separately;
Additionally, probability maps can be smoothed to reduce the number of falsely detected lesions in images, or thresholded so that all probability values below minimum allowed value are zeroed. Fig. 6 demonstrates the detected lesions on test CT scans. Lesion regions were obtained from the corresponding probability maps by means of thresholding with Pthres = 0:5. The resultant lesion regions are marked with colors with correspondence to the colormap from Fig. 5. 2.5
Once the probability maps are built, the TB-descriptor proposed with this study is built as follows. The lungs region on CT image is divided into 6 parts as it is shown on Fig. 7. Height of the parts along Z axis is taken equal. For every type of lesion its presence in each of six parts is calculated as the sum of probabilities in the corresponding voxels divided by the number of lung voxels within the considered part. Since all the probabilities are ranged from 0 to 1, the lesion presence score for each part is also a number from 0 to 1. Finally, the presence scores obtained for each lesion type and each lung part are concatenated into a single TB-descriptor of size Nlesion types Nparts.
Thus, the proposed TB-descriptor indicates presence of lesions of certain types in di erent parts of lungs: upper left, middle right, etc. Portion of the a ected lung volume is considered as well. Such TB-descriptor was used for recognition of drug resistance status, type and severity of tuberculosis in the ImageCLEF challenge subtasks. 3
For all the ImageCLEFtuberculosis subtasks the following prediction scheme was used: { segmentation of lung regions for each CT image; { detection of lesions; { calculation of TB-descriptors for each image; { prediction of the desired values using a valid classi er.
Subtasks of ImageCLEFtuberculosis considered di erent types of predictions: multiple-class prediction where only the index of predicted class must be provided, two-class prediction where probability of belonging to positive class must be provided as well, and regression where the corresponding method needs to predict value of a continuous variable as precise as possible. For all three subtasks, Random Forests classi er was used which is capable of handling all the above-mentioned tasks. Assessment of the algorithms performance was carried out on the Training data using k-fold cross-validation procedure with k = 5. 3.1
Following the above-mentioned prediction scheme, TB-descriptors were calculated for all the available CT images. Random Forests classi er was trained on the set of TB-descriptors with concatenated meta-data values: patients' age and gender. Based on a series of experiments, number of trees in the classi er was chosen to be 150 for this subtask. Accuracy assessment within 5-fold crossvalidation demonstrated Area Under ROC-Curve (AUC) value of 0.6385. One run was submitted as the result of prediction of test data.
A total number of 39 runs were submitted by 7 di erent participating groups for MDR detection subtask. Table 2 shows top-15 best participants' results in terms of AUC value. Utilizing lesion-based TB-descriptor resulted in 0.5558 AUC and ranked 14-th place among the 39 runs. The best acheived result by VISTA@UEvora team with 0.6178 AUC value outperforms previous year's result with 0.5825 AUC. However, MDR detection performance still remains at a level close to random classi cation. Increase of prediction performance might be caused by adding a number of more severe cases with XDR TB into the dataset and also by utilizing information about patients' age and gender. For TB type classi cation subtask, a similar procedure was carried out with the di erence that Random Forests classi er was trained for the case of multiple image classes. Number of trees for this subtask was chosen to be 150. Instead of using all the available data, only the rst CT scan of each patient was used both for algorithms training and for nal prediction of patient's TB class.
In total, 39 runs were submitted by 8 participating groups for TB type
classi cation subtask. The results were evaluated and ranked by accuracy and
Cohen's Kappa coe cient [
In contrast to the two previous subtasks, the TB severity scoring subtask was evaluated in two principally di erent ways.
One way of evaluation used the original severity scores from 1 to 5 as provided by the doctors and the task for participants was to predict those numerical scores as precise as possible. Here, Root Mean Square Error (RMSE) was computed between ground truth and predicted severity scores provided by participants. The goal was to achieve lowest possible RMSE value.
The other way of evaluation considered binary classi cation problem. The original severity index was transformed into two class values: cases with scores from 1 to 3 were labeled as "high severity" cases and the other cases with scores 4 and 5 corresponded to "low severity" class. With this way of evaluation the participants were to provide probabilities of TB cases belonging to "high severity" class. The results were ranked using AUC value. Top-10 runs for both evaluation methods are shown in Tables 4 and 5.
In total, 36 runs were submitted by 7 participants for this subtask. As it can be seen from the tables, lesion-based TB-descriptor appeared to be extremely useful for assessing TB severity with the best result in terms of regression (minimum RMSE among all runs) and 6-th best result in terms of "low severity"/"high severity" classi cation. Number of trees for this experiments was set to 100. The highest binary classi cation performance with AUC value of 0.7708 was achieved by MedGIFT group. 4
The results of this study allows to draw the following conclusions: { Combination of lesion-based TB-descriptor and Random Forests classi er allowed achieving the best performance in TB type classi cation and TB severity scoring subtasks. { Similar to 2017 results, image-based MDR TB detection performance remains low (AUC 0.6178, accuracy 55.93%) despite the addition of XDR TB cases into the dataset and utilizing information about patients' age and gender. { Lesion-based TB-descriptor derived from lung CT scans conveys valuable information on patient's state and is worth to consider in CT image analysis of TB patients. { Extending the training data for lesion detection is desirable for further improvements of computerized TB diagnosis.
In this paper, image description and analysis method based on automatic detection of TB lesions in lungs and composing TB-descriptor is presented. The method was employed by UIIP BioMed group in all three subtasks of ImageCLEFtuberculosis 2018 challenge.
This study was supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, U.S. Department of Health and Human Services, USA through the CRDF project DAA3-17-63599-1 "Year 6: Belarus TB Database and TB Portals".