According to the World Health Organization (WHO), tuberculosis remains as one of the top 10 causes of death worldwide [1]. Particularly when the disease is multi-drug-resistant (MDR), current treatment can be toxic and patient outcome is often poor [2]. An important challenge is still to correctly detect these MDR cases, as it has been estimated that only 20% of MDR tuberculosis cases are actually detected and treated accordingly [3]. Therefore, a better detection and classification of tuberculosis types can result in a more targeted treatment for the patients [4].
ImageCLEF4 is an image retrieval and analysis evaluation campaign (part of CLEF, the Cross Language Evaluation Forum) where many algorithms can be compared on the same basis [5]. ImageCLEF started in 2003, with a medical task included since 2004 and held every year since then [6]. In ImageCLEF 2017, the topic of one of the two medical tasks was related to tuberculosis data analysis from chest CT (Computed Tomography) images [7]. Two independent subtasks were proposed to the participants: (1) Multi-drug resistance (MDR), including 230 3D CT scans from only HIV-negative patients, no relapses and classified according to their tuberculosis treatment: drug sensitive or multi-drug resistant, and (ii) Tuberculosis type (TBT), containing 500 3D CT scans of TB patients, classified into five disease types: infiltrative, focal, tuberculoma, miliary and fibro-cavernous. For the MDR subtask the performance evaluation was done using ROC-curves generated from the probabilities provided by the contestants. On the other hand, for the TBT subtask the evaluation of performance was done using the unweighted Cohens Kappa. [8] gives more details on the dataset, task setup and full participant results, also for the other tasks of ImageCLEF 2017.
The MedGIFT-UBP group, submitted 3 runs in total within the ImageCLEF 2017 tuberculosis task, with participation in both subtasks. The runs were based on visual features extracted from the 2D axial slices of the patient CT scans, learned using a very deep Convolutional Neural Network (CNN). Two main research questions were addressed while training and testing algorithms with a heterogeneous and challenging dataset: 1) how to design an effective and efficient data transformation to enhance small lesions in the full image space?; 2) how to learn the ConvNet models given the limited training samples?.
The remainder of this paper is organized as follows: In Section 2.1, the data transformation method used to enhance small lesions in the full image space is described. Then, in Section 2.2, we investigate the network from a model-selection and optimization perspective. Section 3 reports the experimental setup. In Section 4 we report the results of our runs with respect to the top scores. Finally, in Section 5 we present conclusions and discuss current and future directions.
The proposed methods were built on top of the successful deep learning architecture described in [9] while tackling the problems mentioned above. In terms of volume structure modeling, a key observation is that consecutive slices are highly redundant. Therefore, a sparse structural sampling strategy was favorable in this case. To unleash the full potential of ConvNets with the provided data, several training practices were implemented to overcome the aforementioned difficulties resulting from a limited number of training samples. These practices include cross-modality pre-training and enhanced data augmentation.
We apply for both of the subtasks the provided mask of the lungs [10] and extract the slices using a sparse structural sampling strategy described later in the paper. Then, we use a very deep CNN to perform the feature extraction from each slice of the CT volume of the patients. The parameters for each subtask are described in Section 3. Finally, we take the average of scores and use a Softmax classification to achieve the training-classification of new instances.
In this section, we first briefly describe our image representation method sampled using a sparse structural sampling strategy followed by several good practices for improving the size of the dataset for learning.
The range of gray scale values in a natural image is of 256, but when considering color images, the possible values are 256 3 , i.e. more than 16M. The range of Hounsfield Units (HU) in a CT slice could be grater than 4000 containing negative values (from -1024 to more than 3000), so a direct usage of a pre-trained GoogleNet network may not work. However, if we see each HU as a color code, then it should be possible to use a pre-trained GoogleNet network. Following this idea, we transformed each HU-based slice into a 2D RGB (Red, Green, Blue) image. Since we were interested only in the lung region containing HU in the range [-1024,300], we thresholded the image to this range, assigning the values out of the range to the respective limits. Then, each HU was mapped to a uniform distribution of the HSV space, varying from red, passing by yellow, green, cyan, blue, magenta, and red again. This mapping set to red any value outside the specified range. Furthermore, a key observation is that consecutive slices are highly redundant and therefore we used a sparse structural sampling strategy to enhance the quality of the dataset. Examples of a few 2D RGB images and their respective slices are depicted in Figure 1.
To reduce the over-fitting problem due to the reduced training set we used a 10-crops data augmentation scheme to generate diverse training samples. We only crop 4 corners and 1 center of the images as well as their horizontal reflections (hence ten crops in all). We fix the input image size as 256 × 256 and randomly sample the cropping width and height from {256, 224, 192, 168}. After that, we resize the cropped regions to 224 × 224. At test time we average the predictions made by the network's Softmax layer on the 10 crops.
GoogLeNet, the winner of the ILSVRC 2014 challenge, is a 22-layer deep convolutional network with layers stacked upon each other with different sizes. In order to speed up the computational efficiency it uses 1×1 convolutional operations for dimension reduction. Furthermore, it uses Average Pooling instead of Fully Connected layers at the top of the ConvNet, eliminating a large amount of parameters that do not seem to matter much. More details can be found in its original paper [9].
We choose this architecture due to its improved utilization of the computing resources that allows to increase the depth and width of the network while keeping the computational budget constant that can therefore be trained effectively within a reasonable amount of time.
In this section, we give a detailed description of our proposed methods. We first present the network input and the training details. After that, we describe our testing strategies for both of the subtasks.
Training Run 1: TBT T GNet: In this run, the network weights are learned using the mini-batch stochastic gradient descent with momentum set to 0.9. At each iteration, a mini-batch of 32 samples is constructed by sampling 32 training slices from the dataset using a batch accumulation of 2. We did not use any data augmentation techniques in this run. The learning rate is initially set to 10 −5 , and then the rate is changed to 10 −6 after 1/3 of the iterations, then to 10 −7 after 2/3 of the iterations, and finally 10 −8 after which the training is stopped.
As the network in this run takes RGB images as input, we pre-trained it using the ImageNet [11] model as initialization.
Run 2: TBT TEST RUN 2 GoogleNet 10crops at (different scales): In this run, the network weights are learned using the mini-batch stochastic gradient descent with momentum set to 0.9. At each iteration, a mini-batch of 32 samples is constructed by sampling 32 training slices from the dataset. In net training, a sub-image is randomly cropped from the selected slice using the techniques described in the previous section. The learning rate is initially set to 10 −5 scheduled with a polynomial decreasing policy at a power of 5. As the network in this run takes gray level images as input, we pretrained it using the ImageNet model as initialization. First, we discretize the slices into the interval from 0 to 255 by a linear transformation. This step makes the range to be the same with RGB images. Then, we modify the weights of first convolution layer of the ImageNet RGB model to handle the input of grayscale images. Specifically, we average the model filters of first layer across the channel. This initialization method works pretty well and reduce the effect of overfitting in experiments.
Testing Run 1: TBT T GNet: At test time, given a volume, we extract all the slices (samples). The class scores for the whole volume are then obtained by averaging the scores across the slices. Finally, we use a softmax classification to achieve the training-classification of new instances. This yields an accuracy of 46% on the validation set.
Run 2: TBT TEST RUN 2 GoogleNet 10crops at (different scales): At test time, given a volume, we extract all the slices. The class scores for the whole volumes are then obtained by averaging the scores across the slices and crops therein. Finally, we use a Softmax classification to achieve the training-classification of a new volume. This yields an accuracy of 41% on the validation set.
In this run we simultaneously enriched the dataset both in quality and quantity, but this resulted in performance deterioration due to the noise introduced. We also tried the data enrichment approach using the RGB modality, but we did not notice any improvement.
Training MDR TST RUN 1 : The network weights are learned using the minibatch stochastic gradient descent with momentum set to 0.9. At each iteration, a mini-batch of 32 samples is constructed by sampling 32 training slices from the dataset using a batch accumulation of 2. In net training, a 224×224 sub-image is randomly cropped from the selected frame. The learning rate is initially set to 10 −5 scheduled with a polynomial decreasing policy at a power of 5.
Testing MDR TST RUN 1 : In the test phase, given a volume, we extract all the slices. The class scores for the whole volume is then obtained by averaging the scores across the slices.
For the TBT subtask the submitted runs that obtained the 3rd and 8th position out of 23 runs submitted for this task, with a top Kappa value of 0.2329.
In Table 1, the performance of the proposed and the top techniques for the TBT subtask can be seen. The Table reports scores that correspond to the test set, from the metric proposed by the organizers. Our best score is reported in run1, which uses the RGB modality without data augmentation techniques. On the other hand, the run2 reports scores obtained by using the grayscale modality with data augmentation techniques. In the latter result, we consider that the representation is affected by noise when increasing the number of samples. In the MDR subtask, the proposed approach obtained the 7th position according to the accuracy (0.5352) out of 28 participant runs. In Table 2, the performance of the proposed and the top techniques for the MDR subtask can be seen. The table reports scores that correspond to the test set using the metrics proposed by the organizers.
Overall, our group obtained good positions both in the TBT and MDR tasks.
In this work we propose two fully automatic tuberculosis classification methods and one automatic predictor for assessing the probability of TB patients having a multi-drug resistant TB. These approaches were evaluated in the TBT and